Visualization on Noetic Nought

Preview: A visualization git repository viewer

Wed, 22 Nov 2017 10:18:00 +0530

A few weeks ago, I was watching a talk title Design is a Search Problem by @mbostock of d3 fame, which he gave a couple of years ago while he was still working at the New York Times.

Primarily, he’s trying to drive home the point that the process of coming up with a good design (for a data visualization) is a hard problem – there are a few thumb rules and philosophical guidelines on how to come up with one, but there are no silver bullets to come up with a good design for every scenario.

He makes the case for a process that lets you efficiently try out a lot of designs to see what works and what doesn’t. In the latter part of his talk he goes on to demo an internal tool at NYT called Preview, that they use view, explore and contribute to each others’ work. I hoped that in the two years since the talk was given, this tool would’ve been open-sourced, but it was not to be.

So, I started working on my own project, also called Preview, that has a similar set of features to those described in the talk. Even if I am not working in a visualization team, I think it would be pretty useful to have such a tool to explore the work of other people to learn more about their process, when coming up with a visualization/design.

Currently, Preview lets you keep track of all the repositories of a single user/organisation on GitHub. Any repository with an index.html file is assumed to be a repository of interest, and tracked.

It also lets you view the screenshots for a repository over all the commits in the repository. This is intended to be some sort of a visual version of git log.

You can also view any repository at the latest commit on any branch.

The tool is pretty new and quite brittle right now, but already quite functional and useful, I think. If you do give it a spin, I’d love to hear your comments and feedback.

HT interactives

Thu, 28 Sep 2017 00:46:00 +0530

I recently discovered the interactives by HindustanTimes, and thoroughly loved some of them. It felt like a fresh breath of air amongst all the other Indian data driven journalism that I have been coming across in the recent past.

The visualizations are a lot more interactive than some of the other Indian publications. These remind me of NYT’s interactives, which are amongst the best you’d see, while more often than not exploring data in the Indian context. I really like that they explore a variety of ways of doing these visualizations, and don’t just fit the data into one of the many standard options available in a visualization library.

It feels like they are willing to take their time to do these visualizations, and are willing to collect data from multiple sources, and combine multiple datasets to tell a story. I also like the fact that the visualizations and the articles are more open-ended, allowing users to interact with and explore the data, rather than trying to come up with an article full of conclusions with a click-baity title. The fact that one of the sections is called Just for Fun speaks to me.

The data and the code used in most of these interactives is open and available on GitHub, which is nice!

On the whole, I think they are doing pretty good work, and I shall be on the lookout for even better stuff from them. I’m sure I’ll pick up a thing or two from their work, in my projects. Hopefully, other teams doing similar work will also learn from there and produce more good stuff for greedy readers like me.

Prime Ministers of India - Simple d3 timeline

Fri, 22 Sep 2017 10:48:00 +0530

I have been working on a simple time-line showing the terms of Prime Ministers of India. Currently, the time-line is not very useful and only shows the duration of the term in the time0line, with some pictures, with the hover tool-tip linking to a Wikipedia page. The interactive version is here.

A simple extension of this would be to add historical events of different types over this time-line, with links to more information on these events. I’m not entirely sure, where I’m going with this, though.

Any ideas and suggestions are more than welcome.

Blog trends from word clouds

Mon, 11 Sep 2017 23:23:00 +0530

I came across a couple of fun word clouds, and felt like generating a word cloud for my blog content to get a sense of the major themes on my blog, over the years.

With some simple Python code, I was able to parse the blog and get the word frequency over the years. I then used a modified version of this d3 example to generate a word cloud.

Using all the words used in each year to generate the word-cloud, made it very noisy. So, I switched to using only the top 50 words for each year.

The word cloud doesn’t seem very useful or insightful, but was fun to generate. Each year’s cloud seems to have some words that gives me a sense of some major events/themes for that year, though it may not be very apparent to anybody other than me.

The years which have a lot of posts have clear winners, but the winning words are quite generic. For example, 2007 has words like “life”, “time”, etc., as winners. To try to get rid of the generic words in the word cloud, I tried a quick and dirty tf-idf based word-cloud, but it didn’t really seem to help.

I might get back to this later, to try and improve the tf=idf word cloud. There are also other problems, like code-blocks in posts contributing variable names, urls contributing domain names, etc.

Also, a simple line chart of the usage of tags vs. year might give a better sense of the themes in the blog by year, even though it may not look as fancy as a word-cloud.

30 years of floods in India

Sun, 03 Sep 2017 19:01:00 +0530

Inspired by this post showing the major flooding events in the US, I created a similar graphic for India. You can find an interactive version here – hovering over each flooding, shows some more information about the event.

The graphic uses flooding related data from the Dartmouth Flood Observatory
The data for 2017 events is not up-to date.
It is very likely that there is some missing data, and some inaccuracies in the data. 1987, for instance, doesn’t show the Bihar flood.
The flood severity is indicated by the color of each shape
Each shape represents the geographic flood extents - based on information obtained from news sources.
The data for India map shape is obtained from this topojson collection

It is interesting to look at the severity definitions here – the extreme class floods, for instance, are defined to be those that have an estimated recurrence interval of over 100 years. In a span of 30 odd years, there are a whole bunch of regions which have been affected by extreme floods. Yet another case in point showing that the climate change shit has really hit the roof!

Code

I used ogr2ogr to convert the shape file obtained from the Dartmouth Flood Observatory

ogr2ogr -f geoJSON data/floods.json FloodArchive_region.shp

This file turned out to be about 6MB. I created a file with only Indian floods by parsing the json file.

import json

with open('floods.json', encoding='latin-1') as f:
    data = json.load(f)

india_features = [
    feature for feature in data['features']
    if feature['properties']['COUNTRY'] == 'India'
]
data['features'] = india_features

# FIX some names in the data
NAME_FIXES = [
    ('Tropical Storm K', 'Tropical Storm Komen'),
    ('Tropical Storm Hudhug', 'Tropical Storm Hudhud'),
]
for feature in india_features:
    for name, fix in NAME_FIXES:
        cause = feature['properties']['MAINCAUSE']
        if name in cause:
            feature['properties']['MAINCAUSE'] = cause.replace(name, fix)

with open('india-floods.json', 'w', encoding='latin-1') as f:
    json.dump(data, f)

The visualization code itself is about a hundred odd lines of d3 code.