Data on Noetic Nought

HT interactives

Thu, 28 Sep 2017 00:46:00 +0530

I recently discovered the interactives by HindustanTimes, and thoroughly loved some of them. It felt like a fresh breath of air amongst all the other Indian data driven journalism that I have been coming across in the recent past.

The visualizations are a lot more interactive than some of the other Indian publications. These remind me of NYT’s interactives, which are amongst the best you’d see, while more often than not exploring data in the Indian context. I really like that they explore a variety of ways of doing these visualizations, and don’t just fit the data into one of the many standard options available in a visualization library.

It feels like they are willing to take their time to do these visualizations, and are willing to collect data from multiple sources, and combine multiple datasets to tell a story. I also like the fact that the visualizations and the articles are more open-ended, allowing users to interact with and explore the data, rather than trying to come up with an article full of conclusions with a click-baity title. The fact that one of the sections is called Just for Fun speaks to me.

The data and the code used in most of these interactives is open and available on GitHub, which is nice!

On the whole, I think they are doing pretty good work, and I shall be on the lookout for even better stuff from them. I’m sure I’ll pick up a thing or two from their work, in my projects. Hopefully, other teams doing similar work will also learn from there and produce more good stuff for greedy readers like me.

Blog trends from word clouds

Mon, 11 Sep 2017 23:23:00 +0530

I came across a couple of fun word clouds, and felt like generating a word cloud for my blog content to get a sense of the major themes on my blog, over the years.

With some simple Python code, I was able to parse the blog and get the word frequency over the years. I then used a modified version of this d3 example to generate a word cloud.

Using all the words used in each year to generate the word-cloud, made it very noisy. So, I switched to using only the top 50 words for each year.

The word cloud doesn’t seem very useful or insightful, but was fun to generate. Each year’s cloud seems to have some words that gives me a sense of some major events/themes for that year, though it may not be very apparent to anybody other than me.

The years which have a lot of posts have clear winners, but the winning words are quite generic. For example, 2007 has words like “life”, “time”, etc., as winners. To try to get rid of the generic words in the word cloud, I tried a quick and dirty tf-idf based word-cloud, but it didn’t really seem to help.

I might get back to this later, to try and improve the tf=idf word cloud. There are also other problems, like code-blocks in posts contributing variable names, urls contributing domain names, etc.

Also, a simple line chart of the usage of tags vs. year might give a better sense of the themes in the blog by year, even though it may not look as fancy as a word-cloud.

30 years of floods in India

Sun, 03 Sep 2017 19:01:00 +0530

Inspired by this post showing the major flooding events in the US, I created a similar graphic for India. You can find an interactive version here – hovering over each flooding, shows some more information about the event.

The graphic uses flooding related data from the Dartmouth Flood Observatory
The data for 2017 events is not up-to date.
It is very likely that there is some missing data, and some inaccuracies in the data. 1987, for instance, doesn’t show the Bihar flood.
The flood severity is indicated by the color of each shape
Each shape represents the geographic flood extents - based on information obtained from news sources.
The data for India map shape is obtained from this topojson collection

It is interesting to look at the severity definitions here – the extreme class floods, for instance, are defined to be those that have an estimated recurrence interval of over 100 years. In a span of 30 odd years, there are a whole bunch of regions which have been affected by extreme floods. Yet another case in point showing that the climate change shit has really hit the roof!

Code

I used ogr2ogr to convert the shape file obtained from the Dartmouth Flood Observatory

ogr2ogr -f geoJSON data/floods.json FloodArchive_region.shp

This file turned out to be about 6MB. I created a file with only Indian floods by parsing the json file.

import json

with open('floods.json', encoding='latin-1') as f:
    data = json.load(f)

india_features = [
    feature for feature in data['features']
    if feature['properties']['COUNTRY'] == 'India'
]
data['features'] = india_features

# FIX some names in the data
NAME_FIXES = [
    ('Tropical Storm K', 'Tropical Storm Komen'),
    ('Tropical Storm Hudhug', 'Tropical Storm Hudhud'),
]
for feature in india_features:
    for name, fix in NAME_FIXES:
        cause = feature['properties']['MAINCAUSE']
        if name in cause:
            feature['properties']['MAINCAUSE'] = cause.replace(name, fix)

with open('india-floods.json', 'w', encoding='latin-1') as f:
    json.dump(data, f)

The visualization code itself is about a hundred odd lines of d3 code.

Google Trends, Clickbaits, and PV Sindhu

Sat, 20 Aug 2016 22:59:13 +0530

TL;DR

Am I saying India doesn’t have a caste problem?

No, it’s definitely a HUGE problem!
Are these articles on Google Trends and Sindhu’s caste useful at all?

Just more click-baits and no useful contribution to the discussion on caste.

First of all, congratulations to PV Sindhu on making it to the finals and giving the world number one a hard fight!

This post, though, is not a celebratory one. I read this article – While PV Sindhu fought hard for a medal, many Indians googled her caste – by the News Minute thanks to a re-tweet by TM Krishna. I probably wouldn’t have paid much attention to it, if not for the retweet by TMK.

Caste is a huge problem for the country, and if you still like to pretend it isn’t – read this for a start. I’m not surprised that some people searched for Sindhu’s caste while she was battling it out at Rio. But, some people also searched for Badminton rules, and others for Sania Mirza! What surprises me is that the article doesn’t try to put the data in a context at all. Nor do I understand the contribution this article makes to the discussion on caste.

I played around a little more with these search terms and Google Trends, and here are a few things that I thought were worth sharing.

Indian trends

First of all, the article says “Many Indians”, and gives off cumulative month-wise numbers (UPDATE [2016-08-21 Sun 12:15]: which are so wrong! See Sankrant Sanu’s excellent article on India Facts). But, what does it really mean?

I don’t know if anybody actually wrote about how many people actually followed the match, but Sindhu actually managed to get a whole lot of people interested!

More people searched for PV Sindhu than the daily average number of searches (not hits) for Facebook and very close to the daily number of searches for Porn – two of the biggest things Internet is allegedly used for. Can you see the red line at the bottom? That is the number of people who actually searched for Sindhu’s caste – a tiny blip.

Let’s take a closer look. Here’s a plot of some of the other auto suggest terms + badminton rules & courts compared to Sindhu’s name.

Nothing other than badminton rules (which is at about 1% of max. searches) is even visible, clearly.

So, these other terms (wiki & caste being auto-suggested ones) are nowhere near the actual number of searches being made for PV Sindhu’s name alone.

Let’s get rid of the name graph to get a closer look at the other terms.

The maximum number of people searching for Sindhu’s caste is at about a fifth of the maximum number of people searching for badminton rules. Interpreting it is upto you. But, I’m not surprised knowing how significant caste has been in our society. Also, notice that searches for wiki and caste fall off more gradually than the searches for the rules which peak while the game is being played and watched.
Unsurprisingly, a bulk of the searches appear to be coming from Andhra and Telangana. Notice that the other states are searching more for the rules or her wiki page.

Do you notice that the red (caste), blue (wiki) and violet (images) lines following a similar trend, until the last few hours? What do you think is happening there? It’s the shitty click baits working! Look at the trends from ~8:30AM to ~12:30AM below – the red has spread to other states as well!

And this is how it looks by the evening (17:00-21:00). Well done, folks! The click bait articles totally succeeded!

If you are wondering how much effect the auto suggestions are having in suggesting people to try searching for caste, I don’t think that is much because wiki and images are the other auto-suggest terms, with no such surges in the number of searches.

Trends from the Telugu states

We could see that the two Telugu states were more worried about the caste, than the rest of the country. Let’s see how abnormal they are.

You can definitely see the red line more prominently. In case of Andhra the peak searches for caste are at about 5% of the name searches. A statistically significant difference from the rest of the country.

Let’s zoom in at the auto-suggest terms alone.

Woah, Andhra is definitely quite an aberration from the rest of the country. From about 1/5th of the maximum number of searches for rules, the maximum number of caste searches jumps to about twice. Telangana also is different, but much less so.

Tamil Nadu’s trends below for comparison. You can’t help noticing the last few hours, can you?

NOTE: The trends and screen-shots for the states are slightly more recent than the others.

Conclusion

Without a context, I don’t really see the point of these “news” articles. It seems more click-bait-y than useful to me.

If you still feel many Indians were searching for Sindhu’s caste, while she was fighting it out, go ahead and play with Google trends here. Add/remove terms, change the time and location filters. Some terms that could be interesting to add to the comparison are: Kashmir, Independence day, Facebook.

Later, I looked at the search results and then found that this article wasn’t the only one like it. I didn’t bother to read the others, but I really hope at least some of them are talking sense. Is any of them really trying to contribute to the discussion on caste, meaningfully?

Also, the actual search results don’t seem to give useful answers to the folks so curious about Sindhu’s caste. I hope none of these articles are trying to help them out.

Finally, if you are using Google Trends to write an article, Danny Page has some really good advice.

Thanks to Kamal, baali and 9 for reading and reviewing drafts of this. Errors mine, obviously.

Thinking about Data Ethics

Wed, 25 May 2016 00:00:00 +0530

Earlier this month, a researcher made a dataset containing the profiles of about 70,000 users public. He didn’t really see a problem in doing this because he felt he was only presenting already publicly available data in a more usable form. was only presenting it in a more usable form.

Yesterday, I came across this quote in the very first chapter of Allen Downey’s book Think Stats which I liked a lot, and reminded me of this incident.

I hadn’t looked at the OKCupid data release and the discussion around it much, but I went back and read this article by a social media researcher who thinks a lot about these things.

She puts forth a lot of interesting ideas to think about ethics. Some things that stood out to me are:

Ask yourself how the person whose data you are using feels about the data.
Taking a ‘what if’ impact approach to thinking about data and ethics.

Also, you needn’t really call yourself a researcher to be actually doing experiments with (or analyzing) “big-data” and discovering and putting out facts that have an impact – however big or small. You should really go read the article, whether or not you are a researcher using data.

Incidentally, there is a meet-up on Data Ethics this weekend in Bangalore. I’m excited to learn and think more about this, and talk to others who care.