Software on Noetic Nought

Made in 2022

Wed, 08 Mar 2023 23:35:00 +0530

Keeping with last year’s tradition, this post comes when we are almost 1/5th through this year.

2022 projects

I built a personal Expense Tracker over the month of October using SQLite and Streamlit, because I wasn’t comfortable sharing my financial data with a 3rd party. I’ve tried to keep it simple, so it’s not a pain to use. But, I’ve also tried to keep it extensible, for others to try and use.

The last few months have been quite busy personally, and I haven’t been able to use it as much as I would like to, but hopefully things will settle down soon. A couple of things that I’d like to improve:

The Sqlite file is stored locally, and the data is not synced between different computers.
The tagging/categorization of expenses is very manual, and a little bit more automation would be handy.

artful-dodger

A Next.js project for hosting an art/photo gallery with metadata stored in a Google Spreadsheet and images in a Google Drive (with an optional CDN, recommended). I built this for a friend, who is experimenting with creating “a more equitable model for curating, buying, and selling art”. The gallery is currently in “private beta”, and I can’t link to it. But, here’s a sample gallery here, whose configuration lives here.

ox-gist

I released my first Emacs MELPA package – ox-gist – an Orgmode backend to export and update sub-trees and buffers to GitHub gists. It was a great experience contributing to MELPA. I wrote a blog post about it too.

Termux scripts

I’ve started using Termux on my Android phone and have found the Text-to-Speech (TTS) API quite handy. I’ve a couple of scripts on my phone that lets me use TTS to read out contents on the clipboard or an article from my browser.

Fantasy League Site

With help from a bunch of volunteers collecting and validating data, Jonny and I built a website for the Fantasy League for Indian Ultimate State Championships. There was nothing technically challenging about the project, but it just involved a lot of data cleaning and co-ordination between the volunteers. The data was collected using pen and paper, instead of a digital app for collecting the stats. This made the data cleaning and validation significantly harder. I’d definitely push for digital data collection if people try to do something like this again.

Bookmarklets

A web-page with a bunch of Bookmarklets from my browser’s Bookmarks bar. I’ve had most of them around for a while, but putting them on a web page makes it easy to share and find myself on other browsers/computers.

Music

Unlike 2021, I’ve only been able to record a couple of songs. And this year (2023) hasn’t been any better, so far. I hope I’ll be able to make amends to this, soon, though.

Review of 2021 projects

Earworm

I didn’t make any updates to the package, not because it’s perfect but because I didn’t use it much in 2022 – thanks to how little I’ve been able to record.

While, writing this blog post I realized that workflow had a couple of annoyances. I improved the code to fix them.

I need to move an audio file back-and-forth between my phone and my laptop, one full round trip to add ID3 tags and cover image. This makes me put off uploading recorded files to the site. Also, the new media file needs to be named in a specific format. I’ve added a new CLI subcommand to automate all of this for me.
I always need the full collection of the music files on the computer, where I’m updating the site from. I can just scp the existing site onto my computer and merge the media directories.

Ukulele Tutorials Aggregator

The job of manually updating the site is a lot of work, and I haven’t had the interest to keep doing it.

RPi Server and Phone Backup

The physical setup was accidentally dismantled by folks I share my physical space with, and I haven’t been able to put back everything in place, yet. I’m hoping I’ll be able to get back to doing this at some point this year.

Outro

I’ve already built a couple of small things in 2023, and I hope I am able to find some interesting things to do this year.

Link to the 2021 post.

Made in 2021

Thu, 24 Feb 2022 02:23:00 +0530

Some things I made in 2021:

Earworm: A static site generator to listen to audio (and video?) files in a directory. I use it to host a couple of simple websites for music files. It also happens to be the first package I uploaded to PyPI despite writing Python for many years.
Ukulele Tutorials Aggregator: Inspired by Ukealong.com, I set out to create an aggregator for Ukulele Tutorials for Indian songs. I’ve, with help from Kamal, populated about 300 songs on the site. I haven’t been able to keep up with maintaing it, after an initial bunch of work. I’ve wanted to automate more of the workflow, but haven’t yet.
RPi Server and Phone Backup: I’ve started to use a Raspberry Pi to backup data from my phone and laptop to a personal Hard disk. I’ve ansible scripts to reproduce the RPi’s setup, and some scripts to “sync” my phone’s data every few hours, when it’s on the same network. There’s a lot more that I need here, before I can ditch cloud backups of important things.
Music: I made covers of about 10 songs on the Ukulele, that I shared with some friends. I’ve ways to go before I’m comfortable sharing them on the web, publicly.

I’m glad to have had the energy, time and resources to be able to spend time working on some fun things despite everything that 2021 had in store for us. Let’s see what 2022 has in store!

PS: This post is quite late into 2022, but it’s an excuse to post something to a blog that’s not seen any updates since the middle of 2020.

Hat tip to Bryan Braun whose post inspired me to write this

Instagram's user-friendliness

Wed, 24 Jun 2020 20:13:00 +0530

TL; DR: I’m a video content creator on Instagram, and don’t tell me Instagram is user friendly. Being able to preview posts before making them viewable for everyone will go a long way, though.

Introduction

I’ve been using Instagram quite regularly from around the beginning of this year, posting about one video a week on an average. Previously, I used it primarily as a content consumer for about a year, before getting off it for quite long since it started to turn into a big time-suck. In the current stint of using Instagram as a content-creator, I’ve had quite a few frustrating user experiences. On a platform that comes with a giant reputation of being user-friendly, this is a surprise.

A lot of my posts this year have been video content. While posting them, I ran into a bunch of things that didn’t “just work”. For instance, in the first 3 or 4 posts of the Humans of TIKS series, every post had some thing broken, leading to a poor viewing experience.

Some of my friends, though, found it surprising that I was frustrated with Instagram. “Isn’t it very user-friendly?”, I was asked by many. I think Instagram is just riding on their reputation from the past. The newer features, especially around video content, aren’t exactly “user-friendly”. Instagram does focus on giving the content consumer a nice (read: addictive) experience. But, as a creator, there are a bunch of hoops to jump. Content creators probably put up with it just for the reach Instagram gives.

In this post, I’ve listed down everything that I see as broken. I’ve also tried to suggest improvements to each broken feature that would make the experience more friendly. I start with the more common problem scenarios and progress towards the edge cases.

Caption formatting is needlessly hard

I’ve always had to fiddle with the caption of a post, after publishing it, if it had multiple paragraphs. There’s a magic algorithm that Instagram uses to decide if multiple line-breaks in the text actually mean separate paragraphs or not.

After some trial and error, I’ve figured out a few things about it. For instance, leaving a trailing space at the end of a paragraph causes paragraphs to be joined together. But, when the second “paragraph” starts with a non-text character, it doesn’t seem to matter what the previous paragraph ends with, they always are joined together as a single one.

I don’t want to find out these details by trial and error. Adding previews for captions would make this so much nicer. Some documentation for the “magic” would probably help too. Instagram could even look at WhatsApp, its messaging cousin, for some lessons on how to make writing text easier. Switching to a more common markup language would be a nice bonus.

Albums are restrictive

Posting multiple images in a single post (an album) is a pain. All images are cropped to the same orientation – square, portrait or landscape.

Until a few years ago, Instagram required every image to be square. If you want to keep things dead simple for your users, the square image restriction is as simple as it can get. At least it’s easy to understand and remember, even though I can’t post my pictures exactly the way I want.

The current same orientation restriction is worse on both fronts. I’m still left with not being able to post my pictures exactly the way I want. But, also the restriction is not easy to infer from the UI. I had to fiddle with the UI and my pictures for a while, to understand that there’s no way around this, and that Instagram is actually enforcing this restriction.

I don’t want to be thinking about Instagram’s restrictions while clicking my pictures. I want to take the best pictures I can, and upload them with the best possible orientation/crop.

Instagram may have these restrictions in place to ensure the best possible “viewing” experience. But, for the creators the user-friendly thing to do here would be to remove these restrictions. I think, even going back to the universal square image restriction would feel more user-friendly!

Public profiles, really?

I think, a public profile should be viewable by anybody without having to login, or having to use a specific kind of device. But, Instagram’s idea of “public” profiles is quite different.

If you are not logged into Instagram, and are viewing a public profile, you are prompted to login after scrolling down through a few posts on the profile. If you are on a desktop browser, you can’t open up any post from a public profile. You are prompted to login, when you try to view a post. Even on a mobile browser, you are prompted to login after opening 3 or 4 posts. Opening direct links (or opening posts in a new tab) works, though.

Instagram obviously wants to grow as much as it can, and it doesn’t really care about people not on the platform. As a content creator, though, I care about my audience, irrespective of whether they have an Instagram account or not. A user friendly platform would keep their users’ interests ahead of their growth goals and not do such shady stuff. Or at least, tell us the truth and stop calling it “public” profiles.

So many video types!

When I want to post a video on Instagram, I’ve to first decide what kind of a “post” to make. If I want to post a “short” video, it can go into a normal post/album if it is under a minute. If not, I’ll either need to split it up into smaller segments, or post it as an IGTV video. Stories can only be 15 seconds long, or they get chopped into multiple shorter stories.

IGTV supports longer videos – from 1 minute up to 60 minutes long. The videos need to be portrait videos with an aspect ratio of 4:5 or smaller, 9:16 is recommended, while normal posts prefer square (1:1) videos.

Just like with pictures, I don’t want to care about Instagram’s restrictions when capturing a video. I want to shoot my video, and let Instagram do what is required to share that video. I find it strange that there are hundreds of blog posts documenting how to upload an already recorded video to IGTV.

A user-friendly platform would let users upload the videos as they shot them, and prompt about adding some padding, or allowing users to crop and select a part of the video, to match the aspect ratio requirements.

9:16 IGTV videos are broken

The documented aspect ratio for an IGTV video is 9:16, but guess what! They aren’t displayed correctly in the mobile app.

They are displayed correctly in mobile and desktop browsers, though. But, given how Instagram nudges everyone to use the app, I would guess that 90-95% of viewers would be using the app, if not more.

After a lot of trial and error, I figured out that I need to upload my videos at a ratio of 9:21 or lower to be able to prevent the sides from being chopped off. Instagram’s UI or docs don’t mention this at all. Also, this workaround only works when uploading from a destkop browser. I haven’t yet figured out how to prevent mobile uploads from looking broken.

For a visual platform that is so bullish on everything looking perfect, this is horrendous! I don’t know where to place this - is this a bug in Instagram’s processing pipeline? Or is this a part of attempts to provide the best viewing experience? Just display my videos as advertised, please!

Video post-processing is magic

Any video I upload goes through a processing pipeline, that is undocumented, before it gets displayed to the users. This is possibly done to optimize for space, etc., but it has led to broken user experience for at least a couple of my videos.

On one occassion, it made the audio and video go out of sync. To be fair, the video did have some issues with the audio stream data missing for the first 3 seconds of the video, due to a bug in my video processing pipeline. The same video worked perfectly fine in MPlayer, mpv, VLC, and on YouTube. Noticing that Instagram’s post-processing made the audio and video go out of sync, after about 50 people watched it, is such a shame!

To avoid such problems, we decided to make a demo account and upload every video there, before posting it from the real account. Needing to do this seems super broken to me. I still did it, since I wanted everything to work just right. But, that wasn’t to be.

We had a video where we added some background music on the video, and it ended up having a 4-channel audio stream. It turns out the Instagram app on iOS won’t play such videos, and we only detected that after some people in our audience let us know. Instagram has this documented on their IGTV documentation under additional technical details. But, I only read that because things weren’t working. Again, our followers had a broken experience because of this.

It can be hard to “document” all the technical details of the post-processing pipeline. But, it should be fairly easy to detect things like 4-channel audio, and warning users about it!

Even simpler, but user-friendlier thing to do here would be to allow users to view the video after it has been uploaded and processed, before making it viewable by everyone. Currently, draft posts cannot be shared with friends. I’m not sure if they can even be found and viewed from multiple devices – it wasn’t intuitive to do this, when I tried. Just making it easy to preview posts before sharing it publicly would make things a lot lot more user-friendly. It feels quite backward to be using a private account for this to ensure no broken content is published.

This may sound too extreme, but just give me the exact ffmpeg command being run on the uploaded videos somewhere in the docs! I would then be able to verify that my videos are Instagram compliant, without having to upload a dozen videos and checking on multiple devices. With a lot of big brands using Instagram, I think there’d be other “expert” users who would appreciate these details.

Published content is uneditable

I would find the brokenness more tolerable if I could go back and change the uploaded videos. But, Instagram doesn’t let you edit published posts. It does seem like a reasonable restriction, since this content may have been referenced elsewhere. This seems to be Twitter’s take too, for instance.

For something as “heavy” as a video, though, it seems a little too restrictive. It may not be a great idea to allow changing content behind people’s backs. But, I’m not happy about my followers having to watch a broken post. Deleting a broken post and making a new one doesn’t really cut it – it feels spammy.

I’d like to see some kind of versioning added to the videos. Let user upload a newer version, and make it possible for users to see older versions too, if they chose to. Google Drive and Dropbox seem to do this, for instance.

Outro

Instagram started off by allowing users to only post square pictures. It was very easy to undestand and use, even if not very accomodating of different use cases. Everyone and their dog could use it!

A whole bunch of new features have been added in the last few years, trying to compete with the other platforms. Doing the simplest thing - posting a single picture - may still be easy to do, but trying to do anything more than that leads to a frustrating experience quite often. I think the reputation of being user-friendly is now just a vestige of the past.

People who use it regularly get used to its quirks and probably put up with the annoyances, because it gives them a way to reach their audiences with whom Instagram is popular. Instagram is “friendly” to the content consumer, and tries its best to keep the content extremely consumable to keep them coming back for more. The burden of figuring out and making the content fit the restrictions is left on the content-creators. This can often be frustrating.

Thanks to Kartik, Meghana, Nitin, Pratiksha, Ranjan, Ritesh, Shantanu and Vivek for reading drafts (or parts) of this post and helping by boucing off ideas.

An innocent query that caused Zulip downtime

Thu, 28 May 2020 11:34:00 +0530

Introduction

Recently, a change that I contributed caused some downtime on the Zulip cloud (zulipchat.com). Here’s a user tweeting at us, about this.

I shared this with some friends (who use Zulip) and they asked me to share more about what happened. I began writing an explanation, and it eventually turned into a blog post. The post tries to explain what caused the down-time, without assuming too much knowledge of the Zulip DB schema, but it would help if you used Zulip a little.

The goal of my change was to change how Zulip backend calculates the furthest_read_time for a user – the time stamp indicating when the last message was read.

Previously, the code used simply kept track of the message ID of the messages being read by a user in a field called the “pointer”. My change was to move away from this pointer approach to using the actual timestamp of when a user last read a message.

What is the pointer?

Put very simply, the pointer is just a message ID¹ – ideally, the ID of the “last message” a user has read. Message IDs in Zulip are incremental – a message that has arrived after another one, would have a higher ID than the older message. So, it would’ve been a great way to keep track of the last read message, if all the messages were in “one continuous feed”. But, guess what, Zulip has streams and topics which are precisely what make Zulip such a valuable communication tool!

The pointer can be set to the ID of the message that is currently being read, but only when we are in the “All messages” view. In this view, all messages are ordered chronologically, so any message that is after another message would have arrived after it, and would have an ID greater than the other message. So reading a message in this view means, any message with a lower ID must’ve already been read.

When you are reading messages in a stream/topic (narrowed) view, the pointer is not updated. We can’t be sure that all messages with ID lower than the current message’s ID have already been read. There could be a whole bunch of unread messages in other streams/topics than the current one you are reading. So, the pointer doesn’t get updated until you go to the “All messages” view and scroll over the messages, even if you have already read them in a narrowed view.

Why change this?

Using the pointer to compute the furthest_read_time doesn’t work very well, because it is only updated in the “All messages” view. In other words, furthest_read_time turned out to actually mean something like furthest_read_time_in_all_messages_view.

The furthest_read_time is used in the UI to display the bankruptcy banner – a prompt shown to a user to mark all messages as read, if they have “too many” unreads, and haven’t read new messages in the last 48 hours.

But, this is a problem, especially in Zulip instances with a lot of traffic, where people leave a lot of messages unread until they have the time to catch-up. It is common for people to communicate only in a handful of streams/topics, and leave a lot of messages unread in other streams or topics, that they are hoping to catch-up on, when they have more time at hand. Such users would avoid using the “All messages” view at any cost.

Even in not-so-high-traffic realms, some users just prefer not to use the “All messages” view, and just stick to narrowed views.

So, for these users the furthest_read_time would get updated very rarely, and could potentially remain at a date way back in the past. This meant the users would see the bankruptcy banner despite actively participating in the Realm, even if only in a few topics/streams. This is annoying UX, and what something we wanted to fix.

What did we change this to?

We decided to use the last read message time as the message to consider for the furthest_read_time. The following “innocent looking” query should do that for us, we thought…

UserMessage.objects.filter(user_profile=user_profile, flags=UserMessage.flags.read).last()

Well, it does, but not without bringing down the whole of zulipchat.com after a deploy! 🙈 But, why?!

Some background on how data is stored would be useful to see why.

UserMessage table is huge, my friend

Zulip’s data model has a Message table that stores the content of each message, when it was sent, by whom, to which topic, etc. The UserMessage table is then used to keep track of what the status of this message is, for each user that has received the message. A 32 bit flag field is used to keep track of things like, whether the user has read a message, starred a message, has been mentioned in a message, etc. So, if a messages is sent to a stream with N users, it creates 1 row in the Message table and N rows in the UserMessage table. If M such messages are sent, it would create M*N rows in the UserMessage table. So, pretty soon, the UserMessage table becomes huge! ²

To make common queries on this table efficient, the DB has indexes for them. Some of the indexes on the UserMessage table are documented here, though the documentation was created in the migration context for production deployments. There are indexes for various things, including unread messages.

And here’s our innocent looking query, as a reminder.

UserMessage.objects.filter(user_profile=user_profile, flags=UserMessage.flags.read).last()

This query searches through the read messages for a particular user, and tries to figure out the last message they read. Note that the DB only has indexes for unread messages, not read messages! So, this query isn’t using DB indexes to search for this needle in the haystack, and ends up being pretty damn slow!

Additionally, this query is run for every user when the “home page” is loaded since we want to calculate their furthest_read_time to decide if should show the bankruptcy banner.

Also, Zulip’s architecture is designed to allow using the same app server and DB to host multiple different realms. And zulipchat.com is such a deployment where a few servers are used to serve all the realms. (I don’t know too much about the deployment setup, though).

When a new change is deployed to zulipchat.com and the server is restarted, all the clients of all the users reload. Having an expensive query in that path, would just kill the DB. And this innocent looking query didn’t disappoint!

How did we fix this?

Well, this change was reverted immediately by Tim, and things were kicked back into a usable state.

Later, we decided to take a different approach. We decided to look at a different table (UserActivity) that tracks user activity, mainly for the purpose of analytics. This table keeps track of the last time a user performed an action based on the API calls made.

Among other things, changing any of the flags in the 32-bit flag field is one of the actions that is tracked. Marking a message as read was being tracked too. We use the last time this action was performed to compute the last time a user was “active”, and based on that decide if we should show the bankruptcy banner or not.

Here’s the actual commit that makes this change. But, it turns out that needed a follow-up fix too. 🙈

Thanks to Tim Abbott and Shantanu for reading drafts of this post.

The pointer was used to do a bunch of useful things, but we are currently in the process of removing it. ↩︎
Zulip’s code base uses a concept called soft-deactivation to mitigate this problem on Realms where there are a lot of users who log in very infrequently, or don’t login after an initial period of activity. So, the number of rows are not exactly M*N in all the cases, but in the worst case they are. ↩︎

My Favorite Git Resources

Fri, 22 May 2020 16:07:00 +0530

I often help friends who are new to using git with using it. This blog post is going to serve as a single link to share with them, instead of trying to find individual links over and over.

Understanding Git

I got started with using git after using svn and hg for a few months. But, I would often be lost in the myriad of command line options, putting myself in situations I couldn’t come out of without some scars. I didn’t really understand how git worked, and that was the real problem.

Improving my mental model of how git works, and then trying to fit the commands into this mental model is what made git finally click for me. The Git Parable really helped me understand git from the inside out, and gave me a much clearer mental model of git. There was no looking back from there.

If parables are not your thing, and you’d rather read something that takes a less flowery and more straighforward approach to explaining these concepts, I really liked John Wiegley’s Git from the Bottom Up.

Day-to-day git

Zulip (an open source group chat application that I spend a lot of time contributing to) uses a rebase-heavy workflow, where good commits are merged from different PRs, as and when they are reviewed, instead of waiting for all the commits in a PR to be good to be merged. Developers often have to pull from branches that have been force-pushed to, rebase their commits, squash commits, learn to change git history, etc.

Zulip’s Git Guide is top-class documentation that gives guidelines on using git from a practical stand-point. A lot of new developers use it to get upto speed with Zulip’s workflow that involves a lot of things that I’ve seen devs who have been using git for many years struggle with.

The section on pull request workflow is quite helpful, and should work reasonably well for contributing to other FOSS projects too.

I also like the sections with recipes for common git history edit actions, and the troubleshooting section with recipes to get out of common troublesome situations. Oh shit, git! is a very similar resource to this, that has a bunch of useful “recipes” to recover from different scenarios.

Julia Evans’s exercises approach to learning to explore a git repository is quite interesting, though I haven’t run through all the exercises myself.

Git as a communication tool

I like to look at git history as telling the story of evolution of a code base. The advice that is sometimes given to writers about doing the hardwork, so that the readers can do less of it, holds true for git history too. Communicate the story as clearly as possible for your readers to understand, and don’t just publish a “stream of consciousness” piece.

What does this mean in practical terms, though?

I think editing git history to present a clearer story is completely acceptable. If one tries a handful of different things, and then finally finds the solution to do something, it’s not necessary to have all the failed attempts as commits. Just the final approach can be a commit/pull-request, while documenting the other approaches the commit message or as a comment in the code, if that makes sense.

Expect your collaborators (including future you) to look at not just older versions of code, but the commit history for understanding how a piece of code evolved, for undertsanding the history. Make your git commit messages as useful as you can. Writing a good summary line goes a long way in understanding a change. In most cases a body would be useful to explain the how and the why of the change, while the summary line explains the what.

I like this commit discipline that the Git project itself uses, and was adopted by the Zulip project: “Each commit is a minimal coherent idea”. Simply put, you want each commit to be such that they can be can be deployed (and/or reverted by itself), without depending on any other commits alongside them. You can read more about what minimal and coherent mean in this context in the Zulip documentation.

Tools for Git

Some of the confusion in using git even after understanding how it works is the inconsitent CLI API that it often exposes. There has been some work going into this, to make this more consistent. But, using a nice client can help with this. I use the excellent Emacs client for git called magit. It is an intuitive UI on top of git’s data model, that allows me to do things that I commonly want to do, with just a few hot-keys, instead of typing out elaborate recipes to do things. I highly recommend it, if you are an Emacs user.

It is also helpful to change you shell’s prompt to tell you more about the status of the current directory’s repository. There are a bunch of configurations out there (Stackoverflow, Gists, etc.) that let you do this. The git repo itself comes with a prompt customisation. I like being able to see at least the current branch and dirty/clean state of the repo.

Here are some simple changes to the global git config, off the top of my head. These could make your git workflow more pleasant.

[push]
  # Push to the branch to the same remote branch as the one we are working on,
  # locally. This is the default in Git >= 2.0, so you may not need to set it.
  default = simple

[diff]
  # You could configure a GUI tool to use here. (I haven't used one in a long
  # time, and I don't have recommendations on what to use)
  tool = icdiff

[merge]
  # Shows common ancestor in a merge conflict. Makes it easier to understand
    # the conflict and merge, for me.
  conflictstyle = diff3

# Never garbage collect commits/blobs that are unreachable
# The cost of keeping this data around is negligble compared losing data
[gc]
  reflogExpire = never
  reflogExpireUnreachable = never

Outro

If you have other recommendations or tools that you’d like to share with me, please do! Using git better, and understanding it better, is something that I’m always excited about. I’d also be happy to answer questions, or help you with specific things that you are trying to do with git.

The tools we build change us

Wed, 22 Apr 2020 12:54:00 +0530

A couple of weeks ago, I spent some time building a couple of “automation tools” for myself. It is still early days to talk about their impact, but I have already noticed some changes, and have some thoughts to share.

Thanks to the lock-down, I felt like I needed to do something to get more fresh-air and sunlight. I figured out a nice setup to work from my balcony, when I feel like it. It’s nice to sit in the balcony, since there’s not as much traffic, and there’s enough fresh-air and sunlight.

I usually work with dark themes in all my tools - code editor, terminal, OS, chat clients, etc. But, switching to the balcony during the day makes it harder to read text in dark mode. I have to switch all my tools to light mode. And then switch back, when I come back in. It takes me about a minute or so, each time to make this switch. In a few days, this started to seem like work! And I could see myself needing to push myself to step out, or step back in.

A few days later, I spent about half a day writing a tool that could switch modes for all of these programs with one command. If I was doing the switches manually each time, it would take 2-4 months for the switching time to add up to the 4-6 hours, I spent on getting everything working correctly. So, it’s not really a bad investment. But, what matters more is that I don’t feel a mental block to alternate between indoors and outdoors because of all the extra work that manual theme switching felt like. I am getting some fresh air and sunlight everyday. :)

The other tool I wrote recently was for generating a daily summary of what I worked on from my org-mode based clock entries. I add more detail to the bulleted-list summary that gets generated, and then post it on our team’s Zulip. I noticed a couple of changes because of this.

First, I clock things more diligently than I did before. If I forget to clock into something, I go back and add some manual entries, to maintain a proper log. Second, a couple of other people on my team also have started posting a summary of their day’s work, which is quite nice. This tool may have triggered a change in my teammates’ behavior, though I didn’t really intend it to.

We are creatures of habit, often going through the motions, without needing to think about what we are doing. This is evolution helping us save energy for the more critical tasks. But, our environments can have a big impact on our habits. Having tools around us, that are easy to use, can help us nudge ourselves in directions that we want to be nudged in.

The tools we build can change us; if we let them.

Thanks to Rajesh, Ritesh, Tejaa and Shantanu for reading drafts of this post.

Django ModelFormSet and multiple saves

Wed, 08 Jan 2020 09:45:00 +0530

A friend and I were trying to use the Django ModelFormSets in a project, and we ran into a subtle bug that took a little big of digging into the Django code to identify and work around.

The Bug

We weren’t really familiar with Django’s FormSet or ModelFormSet, when we started. We began by reading the documentation to get started.

The Django docs had an example that looked straight-forward, and showed how to use ModelFormSet.

  def manage_authors(request):
      AuthorFormSet = modelformset_factory(Author, fields=('name', 'title'))
      if request.method == 'POST':
          formset = AuthorFormSet(request.POST, request.FILES)
          if formset.is_valid():
              formset.save()
              # do something.
      else:
          formset = AuthorFormSet()
      return render(request, 'manage_authors.html', {'formset': formset})

You create a ModelFormSet class using the modelformset_factory using the required model, and the required fields.
Use the data from a POST to create an instance of the class from above.
Save the data if the formset validates, and we are done!

We tweaked the example to our needs, and it worked for us. Mostly.

A user reported that after saving the form once with some data, trying to save more data didn’t work! The form would try to save the data already saved in the last save, and fail validation that required some of the fields to be unique.

On the second save, the data which was saved during the last save was still being treated as newly filled in data!

This also caused newly filled-in data during the second save to be lost forever. The user was forced to reload the page and re-enter the data.

The Why

The way Django keeps track of newly filled-in data in the form along with multiple levels of caching causes this bug to manifest.

`ModelFormSet` `queryset` argument

AuthorFormSet (or any ModelFormSet created using the modelformset_factory) takes an additional optional argument queryset. This argument can be used to choose specific rows in the DB that need to be used to populate the initial form rows, if any. If no queryset is specified, all the objects in the DB are used.

`FormSet` metadata

First, metadata about the form is saved in a few hidden input fields (called the ManagementForm). This hidden form keep tracks of metadata like the total number of forms to show, initial form count and the minimum & maximum number of forms to display.

  <input type="hidden" name="form-TOTAL_FORMS" value="19" id="id_form-TOTAL_FORMS">
  <input type="hidden" name="form-INITIAL_FORMS" value="7" id="id_form-INITIAL_FORMS">
  <input type="hidden" name="form-MIN_NUM_FORMS" value="0" id="id_form-MIN_NUM_FORMS">
  <input type="hidden" name="form-MAX_NUM_FORMS" value="1000" id="id_form-MAX_NUM_FORMS">

Initial form count is the count of the number of forms that are pre-filled using data from the DB – essentially, the length of the queryset

Metadata does not change

When the ModelFormSet is used as described in the documentation, the metadata in the ManagementForm doesn’t get updated when the form is re-rendered in the response to a successful POST request. If one form with new data was successfully saved, the number of initial forms in the re-rendered form must go up by 1, but it doesn’t.

Queryset caching problem

Django has caching at the queryset level which means that the queryset doesn’t get updated with the newly created instances, even if they match the query that the queryset uses.

  formset = AuthorFormSet(request.POST, request.FILES, queryset=existing_authors)

existing_authors is a queryset that looks for a specific set of authors in the DB, or may be all the authors in the DB, if that’s what the form needs to do.

So, when a formset has been submitted, and some new rows have been created in the model’s table, the new rows don’t automatically get added to the queryset since it the results fetched from the DB are cached by the queryset. We need to create a new queryset for it to include the newly created rows.

These two levels of caching combined make the bug manifest, as it does. I’m not yet sure what the correct “fix” for this is, but we ended up using a work-around – create a new formset when returning from a successful POST.

The Work-Around

We instantiate a new ModelFormSet instance when returning the form, on a successful POST. This means, the ManagementForm is recreated for the new formset and has the correct metadata about the form, even when being rendered in the response of a successful POST.

The code in the example in Django docs, would look something like the one below:

  def manage_authors(request):
      AuthorFormSet = modelformset_factory(Author, fields=('name', 'title'))
      if request.method == 'POST':
          formset = AuthorFormSet(request.POST, request.FILES)
          if not formset.is_valid():
              return render(request, 'manage_authors.html', {'formset': formset})
          else:
              formset.save()

      formset = AuthorFormSet()
      return render(request, 'manage_authors.html', {'formset': formset})

This blog post should probably become a PR or an issue, but I’m not familiar with how contributing to Django works. This blog post is an attempt to understand the issue better, and potentially help others who get bit by it.

Thanks to Shantanu for reading drafts of this post.

Been there, done that, and still doing it!

Thu, 05 Sep 2019 12:23:00 +0530

I was talking to a friend a few days ago about a new internship she got. She wasn’t sure if and how much she was going to get paid for it. Unpaid internships aren’t cool, and experience doesn’t pay the bills, as they say. Eventually, it turned out that she was indeed going to get paid for it.

But, when she found out that she’ll be paid, she felt way more pressure to “not screw up”. I think working in this mode of “don’t screw up” isn’t very productive. Staying in the mindset of “learn as much as I can” would still be the best way to go about the internship.

Screwing up is a part of the job. People make mistakes and learn from them. Screwing up is a part of gaining experience, and learning. To let people feel comfortable and safe while they are learning, it’s important to have better fallback and recovery mechanisms when things go wrong. The productivity gains are worth investing time into such systems.

I’m going to write down a couple of times I screwed up – one of them in my very first job after college, and the other just a month or so ago. People screw up every day, irrespective of how much experience they have.

Corrupted a server’s hard-disk

In my first job, I had screwed up big time in the first couple of weeks of my job. It’s been a while, and a lot of the details escape me now.

I was supposed to resize partitions on a server’s hard disk so that there’s more space available for the media that we were going to store on the server. The server was pretty newly setup, and there wasn’t all that much data on it, but my bosses (Asokan and Prabhu) had spent some time on configuring it, and setting up user accounts, etc., for all the people that were going to join the project.

I tested the resize operation on a machine locally in the office that I worked in, and it seemed to work. I replicated this on the server, and it went kaput! All the data was lost and I could no longer SSH into it. I was totally freaking out!

But my boss ended up being really calm about it, and asked me to just write to the server hosting service to get a new machine with a bigger media disk. And no, I didn’t get fired from the job. I worked with him for 5-6 years after that, including being hired again at a company he started working for.

Almost Deleted 7-8 years of email

A few weeks ago, I screwed up again. It could’ve been pretty bad, but luckily for me it wasn’t.

I administer a legacy Google Suite for UPAI. We have only 10 email IDs to allocate, and we often recycle them. Recently, I had to merge 3 accounts into a single one to free up 2 new accounts. I transferred data from two of them to the third, and went ahead and deleted them. I did this with a couple of other email accounts before, but they were barely used and nobody really cared about them.

I hadn’t paid attention to what data gets transferred, and what doesn’t. It turns out emails don’t get transferred, and only data in the Google Drive gets transferred. This meant I had lost years of important email communication for the organisation. I freaked out!

My “boss” was again pretty calm, and asked me to write to Google to see if I could get them to restore the data. I looked around, and found that Google allowed restoring deleted accounts for 20 days! I restored those accounts, transferred the email and then deleted the accounts again.

Learning from screw-ups

Learning and improving the systems in-place to prevent such screw-ups is the best thing that we can do from such screw-ups. Just like a piece of software gets hardened by bug reports that are acted upon, over the course of time, other systems can and should too.

As described in this blog post by GitLab on how to take advantage of screw-ups, acknowledging mistakes, fixing them, and improving the systems in-place to prevent such mistakes is a good way to go about it.

Systems needn’t always be super complicated things, but just a little bit of automation, and even simple checklists go a long way in improving work-flows and making processes less error prone. Google’s recovery mechanism definitely helped, and gives me confidence to continue working with these user accounts. Our server had a system admin pretty soon, and he had automated back-ups and the usual stuff in place, for us!

The way a team deals with mistakes sets the tone for how the team works, takes responsibility, and feels about working together. I still screw up after working for about a decade as a developer, and being paid for it. I’m sure I’ll continue to have my screw-ups, even though I’d like to have none. What was nice in both the cases was the way in which my mentors/bosses reacted, and how calm they were about the whole situation.

May you have screw-ups that you learn and grow from!

PS: Google’s Data Migration Service doesn’t support legacy Google Suite accounts, and I ended up using Got Your Back to move email from the accounts I wanted to delete to the other accounts. It worked pretty smoothly!

Thanks to Aishwarya, Meghana, Tejaa and Shantanu for reading drafts of this post.

Thoughts on Technical Debt

Tue, 23 Apr 2019 07:52:00 +0530

I recently watched this short video by Ward Cunningham where he reflects on the history, motivation for, and common misunderstandings of the term Technical debt.

This post is just some of my reflections on Technical debt, based on ideas and thoughts from Ward Cunningham, Martin Fowler and Uncle Bob.

What is Technical Debt?

Ward Cunningham coined the “Techincal Debt” metaphor – he used the metaphor to explain the idea of shipping something with a limited understanding of the problem, in the hope of gaining a better understanding as more people interact with the shipped code.

The difference between the understanding of the problem and how the code models it is going to slow down the developers. This “slowing down” is equivalent to paying interest, until we take out the time to refactor the code to accurately reflect the current understanding of the problem.

Why take on Technical Debt?

When Cunningham says debt, he means shipping code with a partial, unclear or even incorrect understanding of the problem and how it should be modeled. You are willing to get the code out of the door, to improve this understanding.

With borrowed money, sometimes you can do something sooner than you would otherwise. Similarly, rushing software out the door can help us get some experience with it, which would help gain a better understanding and improve the software.

A mess is not a debt, or is it?

Cunningham says that people confuse messily written code with technical debt. He is of the opinion that messy code isn’t technical debt. He’s never in favor of shipping bad or messy code. He emphasizes that code that’s shipped should always be clear enough to refactor easily, when required, even if it doesn’t model our current understanding of the problem very accurately.

Uncle Bob also has a post, on similar lines, where he says, a mess is not a technical debt.

Martin Fowler, on the other hand, argues that the debt metaphor is a useful metaphor to think about and to communicate problems in code — especially to non-technical people — irrespective of the reason for the problem.

Not shipping messy code makes sense. But, the debt metaphor is a useful tool to talk about existing “messy” code, compared to just calling it messy code. Looking at it as a debt lets you think about whether or not it’s worth paying off, and if so when, etc. The Technical Debt Quadrant (explained below) further classified this debt into 4 classes, which makes it even more useful to identify bad code as debt.

Technical Debt Quadrant

Martin Fowler came up with a Technical Debt Quadrant which incorporates other kinds of debt, along with the one Ward Cunningham originally defined.

Fowler comes up with two kinds of divisions of debt — Reckless vs Prudent and Inadvertent vs Deliberate.

When Ward says he’s against writing bad code, I think he means he’s never for taking on reckless debt. Uncle Bob’s mess is also what Fowler would call Reckless debt.

Reckless-Deliberate debt is code written out of not caring enough about the code, or not stopping to think about everything that’s going on with the code, or just being mentally lazy. Code-reviews seem like one good way to avoid this kind of debt.

Reckless-Inadvertent debt is something that’s hard to prevent. It sounds like unknown-unknowns territory to me. The developers or the team would probably benefit from improving their general design skills and/or getting some external help as consultants, etc.

Prudent-Deliberate debt is what Ward talks about, I think, in his video. You know your understanding is not good enough, but you aren’t yet sure what’s the right design is. You may gather more knowledge about the problem while the software is out there in the wild and being used.

Ward talks about better understanding. But, you may not always realize that your understanding is incomplete, when you are shipping the code. This I think would put it in the 4th quadrant - Prudent-Inadvertent debt.

Is everything tech debt?

The Tech Debt quadrant seems to make a lot of things seem to fit into the debt metaphor, but there are some things that I’m not sure about.

For example, it can be a significant time investment to update dependencies. Often, teams don’t update dependencies until it becomes absolutely essential — “don’t fix what ain’t broken”. This strategy of postponing of updates, to use the time for other things, seems like taking on a debt. But, I’m not sure which quadrant to put it into.

Why should I care?

Just calling code messy isn’t really qualifying the mess, and helping us talk about how to tackle it. Thinking about debt while taking decisions seems helpful, especially with further classification of debt into the 4 quadrants.

It helps think about whether we’d like to pay off debts we have taken, and to communicate with non-technical people the kind of “maintenance” or refactoring work we plan to take up, and why it is important to do so. The deliberate debt concept can be used to think and communicate about things we aren’t dealing with, right now, in the interest of shipping software quicker.

Keeping track and repaying debt

If a debt is something we can take on deliberately, and we can take on multiple such debts, what would be a good way to keep track of these debts?

Keeping a list of these debts that we periodically review, seems like a good idea. We check if our understanding has improved from the time we took on a debt, and take out time to incorporate it back into the code.

This list could just be a bunch of FIXMEs in the code with some detailed explanation of the debt, and why it was required, or just a list of issues in the issue tracker with a special tag to make it easy to find them.

Thanks to Shantanu Choudhary and Vivek Krishnaswamy for reading drafts of this post and giving helpful suggestions.

Self-care tips for new developers

Wed, 24 Oct 2018 20:14:00 +0530

A couple of weeks ago, I was at PyCon India – the 10th edition, the 8th PyCon I was attending. It was great to see so many enthusiastic young developers, attending their first PyCon. With each year, as the event gets bigger, you see more and more of them. Lot of curiosity, and enthusiasm to become better programmers.

The conference had a whole bunch of talks, catering to people with varied interests, at different stages in their career – war stories, tips and tricks, helpful advice on getting started with a library or tool, thoughts on the community culture, etc. With each passing year, there are more and more proposals for talks, and a lot of good talks have to be left out due to constraints on the number of talks that the conference can accommodate.

Having said that, I’d have liked to see a talk on self-care tips for developers – advice on staying sane and productive in the long run. It is quite common for programmers to get burnt-out. I’m sure there are some useful lessons to be shared by everyone who has been programming professionally for a while – ranging from about a couple of years to many decades!

In a hallway conversation, Noufal was telling some of us about the Okinawa martial art lessons he’s been taking - it was inspiring to see how seriously he takes the lessons. A lot of programmers that I know, who have been programming for a while, have other interests that they pursue quite seriously, to take time off from programming. It would be nice to have someone speak about mental health and self care in a talk, rather than hope that people pick it up in hallway conversations. Explicit is better than implicit.

I was considering doing a lightning talk on this, but I didn’t get in touch with the organizers early enough to get a slot. So, here’s a blog post.

Disclaimer: Not all of these are things I’m good at doing consistently, but they are things that have at some point helped me. This post is also a way for me to write down things that I (or others) can come back to whenever necessary.

Take breaks

We are often so engrossed in getting something to work, that we lose track of time – slumping in our chairs for hours on end, forgetting to eat, hydrate or move around.

Our bodies weren’t made for this – we were built to move around. Also, stepping away from the problem for a bit, after wrestling with it for a while, brings a fresh perspective and new ideas.

You could use a tool like workrave to remind you to take regular breaks and stretch. Or use a Pomodoro timer. Another trick is to drink keep drinking water regularly, so you are forced to get up and take a piss, every once in a while.

Hydrate, Eat and Sleep

Being dehydrated can make one feel tired and sleepy, making it hard to focus well. Same goes for being hungry and sleep-deprived. Not having enough glucose in your blood has been shown to cause people to make bad decisions.

Mental work uses up a lot of energy, even if we seem to be just sitting around all day, and not expending much more energy than sleeping. It is important to feel good physically to work effectively.

You could have a bottle of water next to you, that you frequently drink from and keep re-filling at breaks. Don’t skip on your lunch to use that extra half an hour being productive. Keeping some snack accessible for you to eat towards the end of the day, when you are tired, helps. Use a sleep tracker or just roughly track sleep each night manually. Make sure you are getting enough of it.

Invest in a good set-up

We spend around half of our waking time at work - over a life time, that’s a lot of time! Sitting in a bad posture, or doing a only-slightly-uncomfortable action over and over, can easily grow into bigger problems in the long run.

You deserve good quality equipment, that “fits” you well. Get yourself a good keyboard and mouse, an external monitor, or whatever it is that you think will make your body thank you. If you end-up working a lot from your home office, invest in a good chair and a table.

Sunlight & Exercise

Being locked up in our office buildings all day doesn’t help with how much exposure we get to the sun. Not getting enough sunlight regularly can effect us both mentally and physically. Same goes for exercise. Getting some exercise everyday has been shown to have significant mental and physical health benefits.

You can start playing a sport, if hitting the gym or doing something alone seems like a chore. Playing a team sport helps even more since there are other people you are accountable to. Try to get some sunlight every morning, even if it is just standing out in the sun, while you drink your coffee.

Take Vacations

Switch off completely, and give your body and mind the rest they deserve. The break not only lets you rejuvenate physically, but also gives your brain the time to synthesize all the inputs it has, and come back with surprising ideas or solutions to problems that have been troubling you for a while.

The importance of taking a break is often forgotten, when we have so much to do. Honestly, I’m not very good at taking regular vacations, myself.

But, having a fixed time every year to take a vacation is something that I’ve seen working. For example, 2 weeks around your birthday every year, you can decide to take a vacation.

Do other things

We get into the game because we are really excited by the things we can build by flipping bits. We get so obsessed with it, that we lose ourselves in it, oblivious to the world outside. All the people we hangout with are people very similar to us.

Often, we are building stuff for people very unlike us. It’s very helpful to see the world from a different perspective than ours. Also, there is only so much we can express through code. There is a whole lot of us that can be expressed in other ways.

Pick up a team sport, do improv/stand-up, draw or paint, write, learn a martial art, join a cross fit gym – anything that you find interesting, other than writing code. Hangout with people who do very different things everyday than what you do. Their completely different perspective at life will make you a better programmer and a better person.

Avoid distractions

The internet can be so distracting! But, as developers, we tend to be constantly connected, leaving ourselves vulnerable to all this distraction. All these distractions can lead to very expensive context switches.

I’ve found getting into, and working in flow-states to be enormously satisfying. What I’m actually working on, doesn’t matter very much when I’m to deeply focus on something I want to be doing.

Creating an environment for yourself, where its easy to get into these states, is totally worth it. I’ve found using Pomodoros to be quite helpful, when I’m feeling distracted. Shutting off the Internet, or using a tool like get-shit-done helps with stopping myself from getting distracted by cat videos on the internet. Turning off all notifications on my phone and computer is another thing that is quite effective I also find it helpful to not be able to constantly have sight of the clock on my computer - so I work with all my windows full-screen. Leaving the phone in another room or in my bag works great, too. You can find a lot more tips on the internet, if these tricks aren’t up your alley.

If you see your team over-working themselves, and holding exhaustion as a badge of honour, please be the person to strike conversations around self care. Your team will thank you.

I’ve only listed down some of the things that I’ve found helpful. But, I hope this serves as a good starting point. Also, do take a look at selfcare.tech for more resources and advice.

Thanks Akilesh Mani and Shantanu Choudhary for reading drafts of this and giving helpful suggestions.

Automation and Habits

Wed, 15 Aug 2018 18:44:00 +0530

We usually see automation as a thing we do to save time. But, there’s more to it than just saving time.

Several years ago, I built a tool called org2blog that let me write blog posts in my text-editor, and publish them with just a couple of key-strokes. I spent several man-weeks getting it to work well – adding lots of features including image uploads. The time I spent on developing and maintaining it dwarfs the time saved by automation.

But, in the three months following this, I published 30 blog posts – as many as I would’ve otherwise published during the whole year. org2blog wasn’t just saving time – it drastically reduced the friction to publish new posts. On some occasions where I would’ve previously avoided writing a new post, I ended up writing one because org2blog made it a breeze. I could focus on actually writing the post and not let my attention be hijacked by the repetitive and boring steps that were previously required to start writing a new post and publish it.

Automation abstracts away the details that we would’ve otherwise had to spend our attention on. This reduced mental overload encourages us to “do the right thing”, when we would’ve otherwise been lazy and taken the easier route of not doing something. No wonder, Douglas Adams confessed to happily spending a day on automating something that would’ve taken him ten seconds.

When extended to a team setting this would mean our (automation) tools nudging everyone on the team towards better habits and behavior. We must recognize this and treat our tools as an integral part of our team, and not just time-savers. The Zulip community is a good example of this. Tim, the lead developer of Zulip, pays particular attention to developer workflows in Zulip and is constantly on the lookout to automate tasks and/or eliminate unnecessary tasks, to make it easier for everyone to “do things right”.

The tools we build aren’t just saving us time, but are changing us back. Building better tools helps build better teams. Go forth and automate!

PS: This post is inspired from this talk by Jessica Kerr where, among other things, she talks about Automation and some reasons for doing it.

PPS: Jason Silva talks about humans as being a technology creating species, and how the tools we create change us, and how we change them to change us further. This seems a lot like the idea in this post, except at a much larger scale.

Thanks to Shantanu for reviewing this blog post.

Error messages and new users

Tue, 17 May 2016 00:00:00 +0530

I was helping a friend of mine setup his blog and we were trying to use Hexo – a static site generator. We chose a Javascript based tool since he’s trying to learn Javascript. I skimmed through active Javascript projects in this list and finally zeroed down upon Hexo based on its popularity. I promised to help my friend to set this up, but he first tried to do it on his own and got back to me after an hour or so, quite frustrated and almost on the verge of giving up setting it up. I didn’t expect this from a tool that had so many stars, forks, plugins and so much active development.

We finally got it working, but we found that the error messages were horrendous – even for someone who has been using free and open-source tools for a while now. Printing out errors from compiler or interpreter directly along with the stack trace is almost always the worst thing to do for a tool/utility (as opposed to an API or library). The stack trace is definitely useful, for developers trying to build upon or improve your tool. Have a debug or development mode where developers can get all the information they need.

If you care about your users, especially new users, make sure you spend sufficient time on showing human-readable messages. If possible list the possible causes for every error along with tips for troubleshooting.

Learning about Spaced Repetition, SuperMemo, Org-drill, et al.

Fri, 24 Oct 2014 00:00:00 +0530

I have been reading a bunch of articles on the SuperMemo site (including the original thesis of P. A. Wozniack, the creator of SuperMemo). I initially started off trying to understand the algorithms SM2,5 and 8 which org-drill implements, but reading the articles has given me a lot of general background and theory on how memory works, and why SuperMemo is as effective as it is (for those who stick with it). This blog post is an attempt to summarize and capture what I have read in all those articles, and not get lost in them. I would recommend reading the thesis and the suggested reading there, yourself, but if you really don’t want to – this blog post + the summary in here, and here would be a good alternative.

Memory works by repetition. Each time you recall a fact, the time it takes for the memory to fade increases. Spaced repetition is the idea of repeating items, just when you are about to forget them – trying to optimize the time spent on learning, as well as the retention of learned material.
Prioritization of the items we need to learn is key to being able to stick to Spaced repetition. Beginners tend to be over-enthusiastic about the technique and fill up their system with stuff that they don’t really care about. Applicability of the knowledge you are trying to gain, is a good test for whether or not it goes into your system.
Each version of SuperMemo comes with a slightly modified (on most occassions, improved!) algorithm. SM2, SM5, … refer to these algorithms. Org-drill currently supports only sm2,5 and 8. It may be worth looking at sm15/16 and seeing if it can be implemented and is worth doing. In any case, using any of these is way better than not using Spaced repetition at all.
Understanding what we are trying to learn goes a long way in helping memory. It is, therefore, good to start with basic material. People have been surprised by how easily the advanced stuff fell into place, once they had the basics covered.
Keep items in the system simple. Stick to the minimum information principle. Keeping it simple, doesn’t mean you leave out on learning the complex stuff; instead break it into simpler components. Simpler items has the following advantages:
- Simple is easier to remember.
- Simpler items are easier to schedule. You get more resolution to figure out what the problem areas are.
- Simple makes brings in redundancy automatically. And redundancy is good for memory, just like it is for managing hardware failure.
Keep learning fun! Don’t make review sessions a chore. Think about the material you are reviewing. Edit/delete/improve items based on your reviews, then and there. Stay active.
It is useful to add a link to the source of the items added to your system, considering that you wish to use the system over a long period of time (ideally, the rest of your life!)
Prioritize, add examples, appeal to your emotional state, link to existing knowledge, use images. Anything else that works for you! Use all of these techniques, that work well and you probably already use in everyday life, in the items you add to your system, to make them easier to recall.
Being physically fit and healthy is important for being so mentally. Exercise, sleep, eat well, avoid caffeine.
Try incremental reading, as an input source of items into your memory system.

Learning to use Org-drill

Tue, 14 Oct 2014 00:07:04 -0400

Org-drill is an Org-mode extension that provides spaced-repetition and flash-card functionality. It has a wonderful documentation on Worg, but somehow I couldn’t get myself to read the whole document, and setup org-drill, until now.

The setup is quite straight forward, once you have org-mode along with the contrib packages. Just (require 'org-drill), and you are all set! To add a new card, all you need to do is add a :drill: tag to the items you wish to “Org-drill”. You can start a review session with simply M-x org-drill. You will be shown flash cards, and you can rate how correct and comfortable you were, in answering the questions. Based on your responses, the cards are scheduled for review. Start another review session, whenever you need one!

What I could only understand once I got myself to read the whole document was that:

The default scope of a drill session is the current file. But, you can start sessions with scopes like current tree, current directory, or a specified list of files. This is super-useful!
The review sessions are not automatically scheduled, based on when you schedule failed flash-cards. Scheduling the review for a card only sets a due-date for them, and effects, what you are asked in your next session.

Cram mode and incremental reading are also things I want to try, as I go along.

Happy Learning!

Best Practices for Scientific Computing

Mon, 24 Dec 2012 00:00:00 +0530

Shantanu and I gave a short talk titled “Software Carpentry for Scientists” for the graduate students of Chemical Engineering department, IISc, this Friday. We gave a short introduction to Git, TDD, Numpy/Scipy, etc and mentioned a few things from Greg Wilson et al’s paper.

I promised to revert to them with links to a few resources. I figured it would be more beneficial, if I just put it in a publicly available place.

A summary of the paper by Greg Wilson et. al., is below.

Useful resources

Software Carpentry

Paper by Greg Wilson et. al.
Software Carpentry

Git & version control

TDD

SciPy

http://scipy-lectures.github.com

Python

http://docs.python.org/tutorial

GUI tools in Python

Summary of paper by Greg Wilson et. al.

Write programs for people, not computers
- a program should not require its readers to hold more than a handful of facts in memory at once.
- names should be consistent, distinctive and meaningful.
- code style and formatting should be consistent.
- all aspects of software development should be broken down into tasks roughly an hour long
Automate repetitive tasks
- rely on the computer to repeat tasks
- save recent commands in a file for re-use
- use a build to automate scientific work-flows
Use the computer to record history
- software tools should be used to track computational work automatically.
Make incremental changes
- work in small steps with frequent feedback and course correction
Use version control
- use a version control system
- everything that has been created manually should be put in version control
Don’t repeat yourself (or others)
- every piece of data must have a single authoritative representation in the system
- code should be modularized rather than copied and pasted
- re-use code instead of rewriting it
Plan for mistakes
- add assertions to programs to check their operation
- use an off-the-shelf unit testing library
- use all available oracles when testing programs
- turn bugs into test cases
- use a symbolic debugger
Optimize software only after it works correctly
- use a profiler to identify bottlenecks
- write code in the highest-level language possible
Document design and purpose, not mechanics
- document interfaces and reasons, not implementations
- refactor code instead of explaining how it works
- embed the documentation for a piece of software in that software
Collaborate
- use pre-merge code reviews
- use pair programming when bringing someone new up to speed and when tackling particularly tricky problems
- use an issue tracking tool

3 tips for those shipping (commercial) apps

Wed, 28 Nov 2012 00:00:00 +0530

Here are some very generic (and paraphrased) notes from a short talk today, by Deepankar Sharma.

Whenever you release a new major version, make sure you keep a copy of the whole “ecosystem” to be able to run it whenever you want. At any point in time, you should be able to run any version of your software.
When writing benchmarks/tests/etc., try and ensure that you cover a broad spectrum of test data, to try and replicate the different types of data that users could possibly have.
Don’t develop applications with modes. Be very careful before you add a new mode to your application, effectively adding one more code path to maintain.
(Bonus) Beware of too much extensibility

GetHub: Chrome Notifications for Github updates

Wed, 02 Mar 2011 00:00:00 +0530

Over the last two days, I hacked up my first Chrome extension. I’ve been using Chrome only for the past couple of weeks or so and I begin to like it, though some of the extensions aren’t as mature as I would’ve liked.

The original idea was floated by my friend, Madhu, and Lee helped me quite a bit, while I was working on it.

What does it do?

It is a simple extension, that shows pop-ups, whenever there is an update in your GitHub “Wall” (yes, this is a Facebook world) or News Feed as they call it.

After installation, you will need to save your username and token for the extension to work.

Where to get it?

Presently, you will need to get it from GitHub.

I might add it to the Chrome Web Store, once I see more people using it. I couldn’t justify, to myself, paying the initial one-time verification fee that Google asks for.

UPDATE: [2011-03-04 Fri] Added GetHub to the store

Comments and Feedback

Feel free to write to me at punchagan+gethub@jeemail.com
Or file issues at GitHub.

Happy GitHubbing!

Richard Stallman in IIT-Bombay

Wed, 01 Sep 2010 00:00:00 +0530

RMS will be in IIT-Bombay on the 6th of this month. He will “speak about the goals and philosophy of the Free Software Movement, and the status and history of the GNU operating system, which in combination with the kernel Linux is now used by tens of millions of users world-wide. "

Date: 6th September, 2010
Time: 8:30 P.M
Venue: PC Saxena Auditorium, IIT Bombay

Sage Days 25, Mumbai, India

Wed, 21 Jul 2010 00:00:00 +0530

What is ‘Sage Days’?

Sage Days is a confluence of present and prospective SAGE Users and Developers. It is an opportunity to come together to share ideas, brainstorm and hack on Sage. Sage Days 25 is the 25th version of Sage Days, and is being organized in Mumbai, India. In order to cater to an Indian audience and scenario, this version has been tweaked slightly. Sage Days 25 has beginner level tutorials, in addition to the usual talks and sprints, to help new users get started with Sage and help promote the use of Sage in India.

What is Sage?

Sage is a free, open-source mathematics software system licensed under the GPL. It combines the power of numerous existing open-source packages into a common Python-based interface. It’s mission is to create a “viable free open source alternative to Magma, Maple, Mathematica and Matlab”. Sage has tools for a broad range of mathematical areas like Linear Algebra, Calculus, Symbolic Math, Plotting, Rings & Groups, Graph Theory, Number Theory and Cryptography. Essentially, “it can do anything from mapping a 12-dimensional object to calculating rainfall patterns under global warming” - as Science Daily puts it . Eager to get started? Start here. Apart from being feature rich, it’s usability is one of it’s greatest strengths. Sage Notebook, a web-interface for all the math you’ll ever want to do, is really the killer feature! As the Sage Marketing page says, “The SAGE GUI surely works on your computer box, because it just runs in Firefox!”. Try it Now!

Why should you attend?

Sage Days 25 is being attended by the creator and lead developer of Sage, Prof. William Stein. It will also be attended by other developers of Sage. This would be a great opportunity to meet and interact with them! The conference will be attended by a plethora of enthusiastic people from all over the country who use Sage or are interested in doing so. The conference will also see the presence of many mathematicians interested in software. Who knows, you may run into someone you’d want to collaborate with, for your future work! This event will be a great learning experience, if you are even remotely interested in math and software for it!

When and Where?

Venue: IIT-Bombay, Mumbai, India
Dates: August 9-12, 2010
Tentative Schedule
Register Here
Click here for more info

Software Freedom Day

Sun, 21 Sep 2008 00:00:00 +0530

Three Cheers to ‘Free Software’!

A toast for GNU on its 25th Birthday!¹

If you intend to ask, what I did on this day, I have nothing to show. I haven’t done anything that’s tangible but yes, I have re-dedicated myself to the idea of Free Software. This post intends to shed some light on a few things (if not for the benefit of others, just as a reminder for myself)

Free Software is a matter of liberty, not price. I’ve often been in a position that required me to correct people. ‘Free Software’ is software that’s free as in free speech and not free beer. For the lack of a better word in english, the word ‘free’ which also means gratis has been used. Using the term ‘Libre’ sometimes helps and if you are in this part of the world, “mukt” is the best word to use.
Free Software² comes along with four fundamental freedoms. To put it simply, the freedom to use, study, share and modify any software.
Free Software may have the advantage of being ’technically sounder’, but the philosophy is what matters the most to me.
I will do whatever is possible within my capacity to spread the philosophy and the associated freedom

Be Free, My Friend!

Here is an extract from one of Stallman’s ³ essays:

We must talk about Freedom

Estimates today are that there are ten million users of GNU/Linux systems such as Debian GNU/Linux and Red Hat Linux. Free software has developed such practical advantages that users are flocking to it for purely practical reasons.

The good consequences of this are evident: more interest in developing free software, more customers for free software businesses, and more ability to encourage companies to develop commercial free software instead of proprietary software products.

But interest in the software is growing faster than awareness of the philosophy it is based on, and this leads to trouble. Our ability to meet the challenges and threats described above depends on the will to stand firm for freedom. To make sure our community has this will, we need to spread the idea to the new users as they come into the community.

But we are failing to do so: the efforts to attract new users into our community are far outstripping the efforts to teach them the civics of our community. We need to do both, and we need to keep the two efforts in balance.

http://www.gnu.org/fry/ ↩︎
The Free Software Definition ↩︎
The GNU Project by Richard Stallman ↩︎

Software on Noetic Nought

Made in 2022

2022 projects

Review of 2021 projects

Outro

Made in 2021

Instagram's user-friendliness

Introduction

Caption formatting is needlessly hard

Albums are restrictive

Public profiles, really?

So many video types!

9:16 IGTV videos are broken

Video post-processing is magic

Published content is uneditable

Outro

An innocent query that caused Zulip downtime

Introduction

What is the pointer?

Why change this?

What did we change this to?

UserMessage table is huge, my friend

How did we fix this?

My Favorite Git Resources

Understanding Git

Day-to-day git

Git as a communication tool

Tools for Git

Outro

The tools we build change us

Django ModelFormSet and multiple saves

The Bug

The Why

ModelFormSet queryset argument

FormSet metadata

Metadata does not change

Queryset caching problem

The Work-Around

Been there, done that, and still doing it!

Corrupted a server’s hard-disk

Almost Deleted 7-8 years of email

Learning from screw-ups

Thoughts on Technical Debt

What is Technical Debt?

Why take on Technical Debt?

A mess is not a debt, or is it?

Technical Debt Quadrant

Is everything tech debt?

Why should I care?

Keeping track and repaying debt

Self-care tips for new developers

Take breaks

Hydrate, Eat and Sleep

Invest in a good set-up

Sunlight & Exercise

Take Vacations

Do other things

Avoid distractions

Automation and Habits

Error messages and new users

Learning about Spaced Repetition, SuperMemo, Org-drill, et al.

Learning to use Org-drill

Best Practices for Scientific Computing

Useful resources

Software Carpentry

Git & version control

TDD

SciPy

Python

GUI tools in Python

Summary of paper by Greg Wilson et. al.

3 tips for those shipping (commercial) apps

GetHub: Chrome Notifications for Github updates

What does it do?

Where to get it?

Comments and Feedback

Richard Stallman in IIT-Bombay

Sage Days 25, Mumbai, India

What is ‘Sage Days’?

What is Sage?

`ModelFormSet` `queryset` argument

`FormSet` metadata