Programming on Noetic Nought

Why my Elfeed index was 0 bytes

Fri, 30 Jan 2026 16:58:00 +0530

My Linux machine crashed, while I was in the middle of a video call. I restarted quickly, and continued the discussion. Later, when I was trying to sync Elfeed updates onto the Elfeed Offline app on my phone, I found it acting weird. All the bazillion things needed to get it working were in place, but it was still complaining about the Emacs Elfeed server not being accessible. I tried opening Elfeed inside Emacs and failed! The index file in Elfeed’s DB was empty! Gone! Poof!

Frustrating, but I didn’t have time to look into what happened. I would’ve been furious if I had been using Elfeed for longer and had a lot more metadata saved. But, it was only a few weeks of lost metadata - posts I read, starred, etc. I quickly setup a Git based backup to prevent future losses and moved on.

Later, I found time to dig into what might have happened…

The breadcrumbs

Elfeed stores all the metadata for all the posts in an index file with a content addressed store of the contents of each of the posts. The index file is simply a dump of the hash-table containing the metadata for the subscribed feeds, their entries and metadata like tags, read/unread status, etc.

The DB save happens in elfeed-db-save, which simply dumps the hash-table to disk inside a call to the with-temp-file macro.

(defun elfeed-db-save ()
  ; <snip>
  (with-temp-file (expand-file-name "index" elfeed-db-directory)
    ; ...
    (princ (format ";;; Elfeed Database Index (version %s)\n\n"
                       elfeed-db-version))
    ; ...
    (prin1 elfeed-db)
    ;...
    ))

with-temp-file, as its documentation says, lets you create a new buffer, evaluate the body there, and write the buffer to file. For a moment, I thought there was some temporary file involved, but nope! I guess the name comes as an extension from with-temp-buffer which does create a temporary buffer where the body of the macro gets evaluated. Stripped to its core, this function is:

`(let ((,temp-file ,file)
       (,temp-buffer (generate-new-buffer " *temp file*" t)))
   (prog1
       (with-current-buffer ,temp-buffer
         ,@body)
     (with-current-buffer ,temp-buffer
       (write-region nil nil ,temp-file nil 0))))

`write-region` and `O_TRUNC`

So, write-region is the workhorse which writes the contents of the temporary buffer to disk. It’s a roughly 300 line long C function that essentially opens the file with the flags O_WRONLY | O_CREAT | O_TRUNC (in this case) and then does something to write the contents to the file, etc. Honestly, I didn’t look at anything else too carefully after I spotted the O_TRUNC.

The man page of open explains O_TRUNC as follows:

O_TRUNC :: If the file already exists and is a regular file and the access mode allows writing (i.e., is O_RDWR or O_WRONLY) it will be truncated to length 0.

Voilà!

The write is not atomic. The file first gets truncated to length 0, and then we hope that the new contents get correctly written before something goes wrong.

It now makes sense why the file got truncated to 0 bytes. The crash happened after the open, but before the write. Somehow the crash happened in this short window, and poof!

Emacs has backup files, doesn’t it?

Temporary files with ~ in their file extensions have definitely annoyed me in the past when Emacs created them where I didn’t want them. So, I do know that Emacs has back-up mechanisms out of the box. But, it turns out that the backups occur in code paths that are more interactive, like save-buffer, write-file, etc. And not via the programmatic APIs like write-region or the higher level with-temp-file. save-buffer calls backup-buffer before writing, but write-region is much more low-level and doesn’t deal with backups.

The fix

My “fix” for this is to backup the Elfeed data in a git repository to be able to recover from any such corruptions of data, which I already wrote about here.

I know that any code that uses with-temp-file (or write-region) could be affected by this, and the right fix for this may be to write to a temporary file and rename it. Maybe next time I lose data I’ll actually fix it properly.

GitHub Gists from Emacs Orgmode

Tue, 26 Apr 2022 12:37:00 +0530

TL;DR: I released my first Emacs MELPA package – ox-gist – An Orgmode backend to export and update sub-trees and buffers to GitHub gists. It was a great experience contributing to MELPA.

Motivation

I often share bits and pieces of my journal.org file with others, mostly for reading, some times for comments. The file contains different subtrees for each day with subtrees for different topics, meetings, articles, etc.

Orgmode lets you export subtrees to specific file names using the EXPORT_FILE_NAME property (and FILE_NAME buffer export option when exporting full buffers). It is easy to create these exported files on a publicly accessible webserver using the power of Emacs TRAMP mode. For instance, I often set the property to some thing like /ssh:muse-amuse.in:~/public_html/<filename> to publish stuff.

But, exporting to HTML is a distraction some times because I start fiddling with CSS and styling. Exporting to a simple org file doesn’t seem to cut it because there’s no styling at all, and making longer exports hard to read in a web-browser. Exporting to a GitHub Gist is a nice middle ground because GitHub renders Orgmode files reasonably well. Additionally, the comment functionality is pretty handy.

I could also directly use @defunkt’s excellent gist.el to share stuff as Gists. I’ve used it in the past, and it works well when I don’t care about having the contents of the Gist locally on my machine. It’s also quite easy to update existing/old Gists in this scenario, since there’s a single “source of truth” and gist.el makes it quite easy to update Gists.

But, this isn’t good enough when I want to have a “synced” copy of the text on my machine. Also, it doesn’t work very well when I want to share just a subtree from my notes file, and not the entire file. I’ve to export the subtree to a temporary Orgmode buffer and then create a Gist from that buffer.

Hackish wrapper around `gist.el`

A couple or so years ago, I wrote a wrapper around gist.el to let me post Orgmode subtrees as a Gist and also update them when I edited these subtrees locally. There was no support for exporting entire buffers since that wasn’t my use-case. This wrapper code was just a part of my .emacs and worked reasonably well for me.

A few weeks ago, I ended up procrastinating on writing and sharing notes for a discussion with a friend, and extracted this wrapper as a separate package – org2gist – to make it easy for other people to use.

MELPA submission and feedback

I submitted this as a package to MELPA while being skeptical about whether it would be accepted. I wasn’t sure if my thin wrapper around gist.el would meet MELPA’s criteria for a “Reasonably innovative package”:

MELPA provides a curated set of Emacs Lisp packages, not an exhaustive list of every single Emacs Lisp file ever created. By default, MELPA maintainers will reject packages that duplicate functionality provided by existing packages. Please try to improve existing packages instead of creating new ones when possible.

It was a pleasant surprise to get very helpful reviews from the MELPA maintainers. They not only thought the package was good enough to include in MELPA, but also looked at it’s code and gave great feedback — not just basic quality checks before accepting the package into MELPA.

I’m pretty sure mainting a package repository like MELPA is a lot of work, but as an end-user it mostly just worked for me and I never really gave it much thought. It was eye-opening to submit the package to MELPA and see first-hand the care and effort the maintainers put into what goes into the repository.

@riscy (a MELPA maintainer) suggested a different approach that I hadn’t thought of — turning the wrapper into an Orgmode Export backend.

Current version: `ox-gist`

It turned out to be a great idea which I ended up implementing. The user interface became much cleaner, since it used the Orgmode Export UI. And it also forced me to “complete” the feature-set by adding support for exporting entire Orgmode buffers to GitHub Gists, not just subtrees.

The package now adds an Orgmode export backend that lets you export a subtree or the whole buffer as a GitHub Gist, similar to how you’d export to HTML or LaTeX from Orgmode. The GitHub Gist ID is saved for each exported buffer or subtree and is to update a Gist when re-exporting an already exported subtree or buffer.

Outro

The package has now been added to MELPA as ox-gist. Feel free to give it a spin and give feedback directly to me or on the issue tracker!

I don’t know if anyone else would find the package useful and use it, though, MELPA tells me that 20-odd people atleast downloaded it. I’m happy this code may be to a handful of people.

Also, I now have a new found appreciation for MELPA and the work put into it by the maintainers. I’m glad I took the time out to submit this package to MELPA.

Thanks to Shantanu for reading drafts and helping me restructure this post.

Thoughts on Technical Debt

Tue, 23 Apr 2019 07:52:00 +0530

I recently watched this short video by Ward Cunningham where he reflects on the history, motivation for, and common misunderstandings of the term Technical debt.

This post is just some of my reflections on Technical debt, based on ideas and thoughts from Ward Cunningham, Martin Fowler and Uncle Bob.

What is Technical Debt?

Ward Cunningham coined the “Techincal Debt” metaphor – he used the metaphor to explain the idea of shipping something with a limited understanding of the problem, in the hope of gaining a better understanding as more people interact with the shipped code.

The difference between the understanding of the problem and how the code models it is going to slow down the developers. This “slowing down” is equivalent to paying interest, until we take out the time to refactor the code to accurately reflect the current understanding of the problem.

Why take on Technical Debt?

When Cunningham says debt, he means shipping code with a partial, unclear or even incorrect understanding of the problem and how it should be modeled. You are willing to get the code out of the door, to improve this understanding.

With borrowed money, sometimes you can do something sooner than you would otherwise. Similarly, rushing software out the door can help us get some experience with it, which would help gain a better understanding and improve the software.

A mess is not a debt, or is it?

Cunningham says that people confuse messily written code with technical debt. He is of the opinion that messy code isn’t technical debt. He’s never in favor of shipping bad or messy code. He emphasizes that code that’s shipped should always be clear enough to refactor easily, when required, even if it doesn’t model our current understanding of the problem very accurately.

Uncle Bob also has a post, on similar lines, where he says, a mess is not a technical debt.

Martin Fowler, on the other hand, argues that the debt metaphor is a useful metaphor to think about and to communicate problems in code — especially to non-technical people — irrespective of the reason for the problem.

Not shipping messy code makes sense. But, the debt metaphor is a useful tool to talk about existing “messy” code, compared to just calling it messy code. Looking at it as a debt lets you think about whether or not it’s worth paying off, and if so when, etc. The Technical Debt Quadrant (explained below) further classified this debt into 4 classes, which makes it even more useful to identify bad code as debt.

Technical Debt Quadrant

Martin Fowler came up with a Technical Debt Quadrant which incorporates other kinds of debt, along with the one Ward Cunningham originally defined.

Fowler comes up with two kinds of divisions of debt — Reckless vs Prudent and Inadvertent vs Deliberate.

When Ward says he’s against writing bad code, I think he means he’s never for taking on reckless debt. Uncle Bob’s mess is also what Fowler would call Reckless debt.

Reckless-Deliberate debt is code written out of not caring enough about the code, or not stopping to think about everything that’s going on with the code, or just being mentally lazy. Code-reviews seem like one good way to avoid this kind of debt.

Reckless-Inadvertent debt is something that’s hard to prevent. It sounds like unknown-unknowns territory to me. The developers or the team would probably benefit from improving their general design skills and/or getting some external help as consultants, etc.

Prudent-Deliberate debt is what Ward talks about, I think, in his video. You know your understanding is not good enough, but you aren’t yet sure what’s the right design is. You may gather more knowledge about the problem while the software is out there in the wild and being used.

Ward talks about better understanding. But, you may not always realize that your understanding is incomplete, when you are shipping the code. This I think would put it in the 4th quadrant - Prudent-Inadvertent debt.

Is everything tech debt?

The Tech Debt quadrant seems to make a lot of things seem to fit into the debt metaphor, but there are some things that I’m not sure about.

For example, it can be a significant time investment to update dependencies. Often, teams don’t update dependencies until it becomes absolutely essential — “don’t fix what ain’t broken”. This strategy of postponing of updates, to use the time for other things, seems like taking on a debt. But, I’m not sure which quadrant to put it into.

Why should I care?

Just calling code messy isn’t really qualifying the mess, and helping us talk about how to tackle it. Thinking about debt while taking decisions seems helpful, especially with further classification of debt into the 4 quadrants.

It helps think about whether we’d like to pay off debts we have taken, and to communicate with non-technical people the kind of “maintenance” or refactoring work we plan to take up, and why it is important to do so. The deliberate debt concept can be used to think and communicate about things we aren’t dealing with, right now, in the interest of shipping software quicker.

Keeping track and repaying debt

If a debt is something we can take on deliberately, and we can take on multiple such debts, what would be a good way to keep track of these debts?

Keeping a list of these debts that we periodically review, seems like a good idea. We check if our understanding has improved from the time we took on a debt, and take out time to incorporate it back into the code.

This list could just be a bunch of FIXMEs in the code with some detailed explanation of the debt, and why it was required, or just a list of issues in the issue tracker with a special tag to make it easy to find them.

Thanks to Shantanu Choudhary and Vivek Krishnaswamy for reading drafts of this post and giving helpful suggestions.

Heroku apps with custom-domain and Cloudflare SSL

Thu, 14 Mar 2019 13:03:00 +0530

Motivation

I have a bunch of small (toy?) apps hosted on Heroku. They are probably used by a few dozen people, at most. For these apps, I’d like the following setup:

Use a custom domain for the apps
- Heroku provides domains of the form appname.herokuapp.com for all apps.
- But, it also has the option to add one or more custom domains
- I usually already have a domain under which I want to add a sub-domain, where these apps would run.
Use HTTPS for all connections
- The domains or the sub-domains usually don’t already already have an SSL certificate. With lets-encrypt, I guess this can change in future.
- Heroku provides SSL certificates for the default domain (appname.herokuapp.com) for free.
- They also have an option to buy SSL certificates for custom domains, but they are expensive!
- I’m going to use Cloudflare’s free SSL service instead. I’d like to have SSL for the full domain, and not just the app’s subdomain.
Redirect all the requests to the Heroku domain to the custom domain
- Nobody should really be using the appname.herokuapp.com domain, in case I’d like to move away from it.
- A few lines of app (Flask) code can do this for us.

Setup

Setup DNS to use Cloudflare

Since we plan to use Cloudflare for the SSL certificates, we need to change DNS settings at our domain registrar to use Cloudflare. We first need to create a Cloudflare account, and let Cloudflare do a DNS scan to add all the existing domain settings automatically. Next we point to the Cloudflare DNS servers on our domain registrar.

Enable HTTPS always in Cloudflare

Using page rules in Cloudflare, add a rule for the whole domain to “Always use HTTPS”. For example, use the url http://*example.com/* and select the Always use HTTPS option.

Select the SSL option to use

Cloudflare provides different options for the SSL setting which changes whether or not traffic is encrypted between Cloudflare and the Heroku app.

Choose Full (strict) to enable encryption between Cloudflare and Heroku.

If you are not so concerned about encryption between Cloudflare and Heroku, you could select the Flexible option too.

Add a custom domain in Heroku

Add a custom domain – app.example.com – in Heroku’s settings.

Heroku provides a DNS Target that needs to be used as the destination for a CNAME setting in the DNS provider (Cloudflare). The DNS Target looks something like foo-bar-123abcdef.herokudns.com

If you chose Full (strict) SSL option, this DNS target cannot be used and can be ignored.

Add a CNAME for the subdomain

Add a new CNAME setting in Cloudflare for app.example.com and use the app’s Heroku domain name (appname.herokuapp.com) as the destination value/IP address.

It is important to use the appname.herokuapp.com value if the SSL settings you chose above was Full (strict). Using the DNS Target provided by Heroku instead (foo-bar-123abcdef.herokudns.com) would give an SSL Handshake error since the SSL certificate provided by Heroku only works for the appname.herokuapp.com domain, and not for foo-bar-123abcdef.herokudns.com.

If you chose Flexible, though, you can use the DNS Target provided by Heroku – foo-bar-123abcdef.herokudns.com.

Also, ensure that the Cloudflare Proxy Toggle is toggled on – the cloud icon is orange, not grey!

Redirect all requests

We redirect all the requests coming to the old domain to the new one.

@app.before_request
def redirect_heroku():
    """Redirect herokuapp requests."""
    urlparts = urlparse(request.url)
    if urlparts.netloc == "appname.herokuapp.com":
        urlparts_list = list(urlparts)
        urlparts_list[1] = "app.example.com"
        return redirect(urlunparse(urlparts_list), code=301)

Enforce HTTPS on Heroku

If you don’t care about a custom domain and just wish to enforce SSL for the Heroku domain (appname.herokuapp.com), Flask-SSLify is the way to go. The app can be hosted on Heroku and we can use the certs provided for free.

from flask_sslify import SSLify
app = Flask(__name__)
if "DYNO" in os.environ:
    # Always use SSL if the app is running on Heroku (not locally)
    sslify = SSLify(app)

Conclusion

I’ve done this setup, or some parts of it quite a few times, but each time I seem to need to look up the documentation. Every time, it seems to take longer than it needs to! Hopefully, this post will make it quick and reproducible.

2018 in Review

Thu, 17 Jan 2019 15:02:00 +0530

Here’s my review of 2018 - my first successful annual review. I remember wanting to do something like this, a couple of times before too. This time, I managed to get started and wrap it up!

I’ve been inspired to write one by a some reviews that I read this year – some of them do it every year.

Sher Minn’s visual retorospective
Nat’s month-wise review, with a review of 2018 goals and a bunch of goals for 2019.
Julia Evans has an annual review with a bunch of “conclusions” listing what worked in 2018.
James Clear answers three questions every year - what went well, what didn’t, and what were the learnings.
Sacha Chua’s reviews were one of the earliest reviews that I read, and they inspired so much awe! I may not be able to do as much quantified-self type posts as her, any time soon, but I love them!
Buster Benson does annual reviews on his birthday every year. He gives himself a motto to live-by, each year.

My review isn’t as systematic as any of these. But, I have some inspiring examples in front of me to aspire to. Hope this will start something that I can keep doing annually from now, if not more frequently.

I’ve divided my review into a few different areas - based mostly the time I have spent on them, how important I feel they are, and the extent of notes I have in those areas.

Work

I continue to work on RSR at Akvo.
The most challenging project was to wrangle a legacy permissions system that worked at an organization level to give some kind of user level permissions. The project was plagued with incomplete communication of requirements from our partners, failure to understand the complexity of the legacy code, and weird bugs manifesting in even weirder ways. We eventually delivered something, but I’m not at all happy with the solution, how long it took and how the whole project felt.
The company has been in a cost-cutting mode, and has been rethinking the way the products are positioned - transitioning more into a single solution/experience for our partners. This makes a lot of sense – the focus shifts from building tools to solving problems, which is great! But, I wasn’t particularly happy about how much RSR features in these conversations. Towards the end of the year, our team has managed to get more conversations going on involving RSR, but we need more.
A couple of friends asked me if I’d want to work for their companies, but I decided not to. But, these were opportunities to think more about what I want out of my job and the work I do. This is a work in progress.
I did some mentoring at one friend’s company, and got to know the team and their work pretty well.
I gave an interview at a company that I came across at PyCon which went horribly, partially because I didn’t prepare well and partially because of interview jitters. I was expecting a better interview experience from them, though, given how much I liked their team culture, etc.
I also happened to take my first ever front-end interviews, along with a team mate. I really liked how he led the interviews, and made the interview candidates feel comfortable. I learned a thing or two, including the fact that I don’t know a lot of things that people with a more structured and systematic approach to learning front-end development would know.
I didn’t attend too many events this year, like always. But, I really enjoyed PyCon - especially the Keynotes by Sidu Ponnappa and Armin Ronacher.

Side projects

I worked on a bunch of side-projects this year, some of them abandoned, some completed and found useful, some abandoned after the first version turned out to be not so useful. Most of the stuff was “webby”, and unsurprisingly a bunch of my side-projects were Ultimate related.

Zulip

I spent a quite a bit of my spare time between about March and September contributing to Zulip – mainly as a mentor on the zulip-terminal project, apart from some minor contributions around improving setting up of the dev environment for the zulip-server project. I also facilitated a Zulip sprint during PyCon at Hyderabad.

I also got to attend the Zulip India Summit! It was great to meet a whole bunch of Zulip contributors. I especially enjoyed the conversations with and the presentations by Tim and Greg, mostly around developer productivity.

I’m really impressed and inspired by the amount of care and effort that is put into stream-lining the process of contributing to the project, and the emphasis on developer productivity.

I played around with a little bit of React Native development, but wasn’t able to devote much time to it. I’d like to play around some more with it, this year.

Find Playo Venue

I wrote a simple map based tool to find the most suitably located Playo venue with good ratings, when some of my friends and I were playing a lot of badminton together. I still use it sometimes, but we’ve mostly figured out our “favorite” venues.

moditweetarchive

I built a site inspired by this Trump tweet archive for Modi’s tweets. I used Dash to build it, and it was quite fun, despite a few annoyances. This tweet archive, though, isn’t as interesting as the Trump archive, since the tweets are more calculated and strategic. But, a few things could be added to make it more useful.

The repo is private as of now, but I’m happy to share the code, if anyone is interested.

Blaggregator

I spent a couple of weekends maintaining Blaggregator. I updated Django to 1.11.x which is also something I did on Akvo RSR (my work project), later during the year. I also worked on making the crawl faster by handling dead blogs better, etc.

weblogviz

I tried to get started with learning some Rust, and as an exercise, I started building a tool to analyze Apache logs and show some useful stats.

Org-mode to Zulip helpers

I wrote some Emacs helpers to post to Zulip. I’m quite happy with being able to post longer messages from inside Emacs - this lets me retain a copy of things like my checkins or interesting links in my notes/journal file.

aradhana.org

I tried to help a friend of mine build a newer website for aradhana.org, and spent about a couple of days on it. But, there were other bigger problems which needed attention from my friends, and this site just fell by the wayside.

Abandoned projects

There are bunch of other projects that I’ve abandoned. I may end up spending some time looking into them, this year.

SOTG Calculator

I had the chance to look at the Spirit scoring sheets for a couple of tournaments, in the beginning of 2018, and was utterly disappointed. I always knew that it was a lot of work to get the top scores just a few minutes after the final game, but I found a lot more mistakes than I expected – a lot of them were easy to avoid with just a little automation.

To help improve the situation, I built a simple webapp to compute the scores from a Google Spread Sheet. Finding the initial adopters turned out to be very hard, despite the app making life a lot easier for people. It was disappointing to see people saying they didn’t have enough time to try the app, and instead chose to manually compute everything that the app could’ve done with a couple of clicks!

But, eventually it started to be pushed as the official tool starting this season, and has found a bunch of happy users.

Team RSVP

We used Whatsapp to co-ordinate team practices and to keep track of the players attending, etc. But, this started to get quite spammy, and it was a hassle to have other conversations on the group, while folks were calling in for a practice. It was a nightmare to have people call in for two different events at the same time.

To see if people were willing to switch away to something less spammy, a friend and I started with a simple RSVP app and most people seemed to not mind using it.

Unsurprisingly, the app eventually got a lot of features - to make it easy to manage team practices and team resources like photos and training material. It is being used regularly by the team, without too many problems.

As an after thought, may be we have an over-engineered solution, and just asking everyone to use (Google) Calender & Email would’ve worked.

Huddle magazine archive

The Huddle was a good Ultimate related magazine run by Ben & Andy a few years ago. They eventually stopped running it, and the archives disappeared from the original site but are available on the USA Ultimate website which is super hard to read! I wrote some code to scrape the site, and convert the articles into markdown posts and made a more readable version of the magazine here.

I tried to get in touch with Ben & Andy, to ask if they were interested to replace the “coming soon” message on the original site with these archives, but didn’t succeed. Posting it on Reddit and sharing it with my team got some people interested, and hopefully they keep using it.

vquiz

To help my team with understanding rules better, I built a light weight tool to easily create video based quizzes - embed video and ask related questions, use videos to show answers.

To build a question bank, I tried to get some people on the team to volunteer, but it didn’t really take off.

Ultimate Frisbee

2018 was filled with a lot of Ultimate - not just playing, but also thinking, talking and planning.

Our team had a couple of the best tournaments that we’ve since I’ve been playing with the team - not just in terms of the results but in how much fun we all seemed to have together.
I captained the team in a bunch of tournaments. I’m not sure how exactly I got to be the captain – I guess it was just stepping forward for a tournament and offering to spend the time and effort required. It was a great learning for me, and I’ve learned some things about people and leading them that I would’ve taken much longer to learn. I also think I did get better over the course of the year – I got more confident and the team also trusted me much more.
In the past, game time at tournaments has been a big concern. We experimented with a bunch of simple rotation systems – essentially just systems to keep track of how much each person has been playing, and trying to prevent too many imbalances. People were encouraged to bring forth their concerns and ideas during the games, rather than keep everything until the end of the tournament when it is not fixable until the next time!

We had the luxury of being able to try this since everyone on the team had more or less the same skill level, and everyone trusted the system. It helped take players’ minds off playing time, and helped them focus on actually making the most of the opportunities on the field. Most of the complaints about game-time seem to have gone away.
Communicating the reasoning behind a decision is as important as communicating the plan. Often, I’d hear surprising responses to a plan or just generally disappointing comments like how decisions were being made arbitrarily, when we were actually spending a lot of time and energy to try and make decisions. On most occasions, I felt the reason was the lack of an understanding of the reasons behind a decision.
Team communication was another tricky thing that I’ve had to wrestle with. The team still needs to find a balance between communicating over a WhatsApp group or discussing things in person, and the fact that most people are very busy to make in-person meetings with a good attendance happen. We are experimenting with a few things to see what works and what doesn’t.
At tournaments, having a positive environment where everyone is supportive and has each others’ back is the most important thing for me. Irrespective of how much practice we had together and what kind of strategies we practiced, just having a team where everyone is positive and helpful goes a long way.

On the other hand, this positivity is also such a fragile thing that needs to be carefully protected and nurtured. It’s very easy for one negative comment or thought to spread negativity and discontent like wild-fire.

I think as a captain, I should’ve strived harder to safe guarding and cultivating this positivity in the team.
Thanks to some discussions I had with my teammates, I spent a lot of time thinking about what it means to feel like a valued member of a team, and what it means to feel like you are contributing and making a difference. Being chosen an MVP during a game is external validation that you made a difference in a game, but only one or two people get that validation in each game. What about the rest of the dozen or so other people? How can we as a team recognize and acknowledge each others’ contributions to each point or game?

Some of these questions will continue to take up some of my mindspace this year, whether or not I continue to captain the team.

I also took up the role of UPAI’s Director of Technology (aka sys-admin) along with AK. It’s mostly been minor email or slack related change requests, and nothing more. But, it’s been fun to get to know some of the people in the Ultimate community a little better. I also helped organize the UPAI conference.

Writing

I wrote only 8 blog posts, but 6 of them in the second half of the year!
Some friends and I started a Writing Club in July, and it’s been going strong so far. It has not only helped me publish at least one blog post a month, but based on the responses I’ve received, I feel it has also improved the quality of my writing. A lot of credit goes to my friends who read initial drafts of my writing, and gave me really good suggestions on how to improve them.
My journaling and personal note taking improved slightly as compared to last year - I have more notes for 2018, than 2017. And that’s how I was able to write this review post. But, I think the notes are restricted to a few areas of my life, and I can take some actions to write about other things that are important too.

Relationships

I started a couple of Zulip instances with my friends on it – one with my college friends and another with my Ultimate team mates on it. My friends from college went back to WhatsApp, but the other Zulip realm has had some pretty conversations, especially on #beanbags. It hasn’t been super active towards the end of the year, but I’m really happy it exists, and hope it continues to grow this year.
It was good to have some more teammates turn into friends. Thanks to all the Ultimate stuff I’ve been doing, I got to spend a lot more time with some of the people that I previously hadn’t interacted much with.
Tarle, a teammate and friend, passed away. 4 months and he’s still dead. We all miss him a lot! But, we’ve also come together stronger, in some ways.
I’ve tried scheduling regular check-ins with some friends who live far away. They worked for a bit, but seem to be easily swallowed by higher priority things. When they did happen, though, they were quite nice!
Towards the end of the year, I moved houses. The new house is well furnished and has neighbors who mostly mind their business – I had a lot of friends and family visiting, which has been good.

2019 goals/ideas

I want to give having a motto for an year a shot, a la Buster Benson. My motto for 2019 is “Be Deliberate”.

I’m not sure I know exactly what this entails, and I’m sure I’ll get a better understanding of it, as the year goes by. But, here are few things that I think would be good starting points:

Choose what I read more deliberately, and read for understanding rather than just information. Keep better notes of the books I read.
Go back to using Howdy or creating check-ins with more people, and making sure that they happen. Prefer meeting in person or audio/video calls, rather than chat.
Journal more about feelings and thoughts, especially when making decisions, rather than just about events. Journal about more areas of my life rather than just “work” related stuff.
Consider hiring an editor to help write better blog posts, and improve writing style. Getting reviews and suggestions from my Writing Club friends has been good, but this would be a level up.
Look for mentors in other areas of life - especially those where I feel like I’m plateauing.
Pick side projects more carefully. One thing that I would be deliberately thinking more about would be working on side projects that pay.
Write quarterly posts to document my improved understanding of the theme – Be Deliberate – and to track progress I’m making on these goals.

Thanks Kartik Krovvidi and Shantanu Choudhary for reading drafts of this post, being excited about it, and useful suggestions to make it better.

Software development practices for sports teams

Sun, 23 Sep 2018 11:31:00 +0530

Motivation

A few weeks ago, I came across a talk on Mob programming — a technique for working on a software project as a team, that was accidentally discovered by a team trying to figure out how to work well together. The speaker lets us in on how they stumbled on it, and how they fine-tuned it to make it more effective while being relaxed and enjoyable.

A few days before that, I shared a post on “getting feedback (as a software developer)” with my Ultimate Frisbee team-mates and it seemed to resonate with them. I’m often looking at things happening in my sports team from a software guy’s perspective, since I spend a lot of time being the software guy. But, I rarely, if ever, directly share these thoughts and ideas with my team-mates.

But, the mob programming talk inspired me to make a conscious effort to look for software development patterns that could be adopted by our (sports) team.

Most players on our team have been playing Ultimate for a while now (2-5 years). So, we have more problems due to miscommunication and everyone not being on the same page, than we have due to lack of skills or knowledge. When someone on the team does not understand the plan, the whole plan goes for a toss.

How do we make sure that everyone understands what the team is trying to do and is able to contribute their bit to the team’s plans and success? How do we make playing with each other as enjoyable as it can be?

“The object isn’t to make art, it’s to be in that wonderful state which makes art inevitable.” — Robert Henri

Some ideas for adoption

Here are some ideas that, I think, could be adopted by our team. Some of these are things that teams, including ours, do in some shape or form. But, identifying and naming would make it easier for people to talk about them and develop a common understanding of why and how to do them.

Ubiquitous Language

Eric Evans, in his book Domain Driven Design, talks about building a Ubiquitous language in the team. It grows as the team’s understanding of the problem grows, and helps the team communicate clearly throughout the development cycle.

Our team plays a lot of zone defense, and it usually works well for us But, it wasn’t working particularly well, during one of the tournaments last year. Luckily, we had Moby playing with us, and he asked us some probing questions around the objective of the play and the roles of each of the players. With just a short chat and a few small adjustments, the zone became way more effective. Essentially, we built a common understanding of the objective of the defense and brought everyone onto the same page.

Ultimate teams use a lot of lingo – a lot. New players just pick it up by osmosis, at practice, from the senior folks. This works most of the time, except when it doesn’t. Each person on the team has a slightly different understanding of what a term means, and the difference is sometimes big enough to cause a plan to fail due to miscommunication. Spending time building a common understanding would make the team communicate and operate more effectively. Practicing game scenarios is essentially a way of doing this, but it would be worth thinking about and discussing the terms and the language being used.

Off-field lingo should also be clarified using a similar process – for instance, what does “good sideline support” mean? What does giving or not giving “feedback on the field” mean?

Sprints

Agile development recommends continuous development of software. Often software teams use 2-4 week development cycles, called sprints. For each sprint, the team sets itself a goal, and the whole team prioritizes tasks and makes decisions in-line with this goal.

Sports teams would also benefit from sprints – short and focused practice cycles, where everyone on the team is aware of the team’s goal. This would make it much easier for players to help each other out. Also, players can align their individual goals with the team goal, making practices much more effective.

Retrospectives

At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly. — The Agile Manifesto

Most Ultimate teams do some kind of post-practice retrospectives, though, they don’t call it that. These retrospectives are a discussion of the good, bad and the ugly. Usually, no conscious decisions are made about what practices are working well for the team, and what should be taken forward to upcoming ones.

Slightly longer term retrospectives would be quite useful too. If a team adopts the idea of sprints, each new sprint should start with a retrospective of the last one and planning for the next.

A retrospective’s participants are expected to follow the prime directive.

Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand. — Norm Kerth, Project Retrospectives: A Handbook for Team Review

Internalising this would be really helpful especially during post-tournament retrospectives, where emotions can run high – it would keep the discussion positive and focused on identifying problems and finding solutions, rather than spiraling into blame games, defensive arguments, etc.

Daily Stand-ups

Agile teams gather each day at the start of the work-day, to let each know about the progress of the work-items. The team tries to help each other get rid of the impediments to progress.

Ultimate teams could also benefit operating in this style – everyone shares with each other what they are working on, and the problems in their learning. This would make it easier for others to look out for things to help with. Team-mates would limit themselves to giving feedback to the thing that a player is working on.

Players setting themselves a small number of clear focus points would also force them to consciously think about and work on their game more effectively.

Face-to-face communication

The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. — The Agile Manifesto

As our team has grown larger over the years, more and more communication has moved to the WhatsApp group. Keeping everyone informed via face-to-face communication, at this team size needs a lot of co-ordination and planning, and the time at practice is considered to be too precious to be spent talking and not practicing.

But, clearly WhatsApp doesn’t seem like the best medium to have discussions. Discussions rarely, if ever, happen on our groups. It often just seems to be information dissemination. Using a better communication tool could mitigate a few of these problems, but face-to-face discussions have other benefits. They bring about a greater sense of belonging to the team and increase buy-in into the team’s plans.

Unit tests vs. Integration tests

Practice games are helpful to gauge if a team is playing well together and if its plans are working. This seems analogous to integration tests — checking that we have functional software, end-to-end.

But, in the early stages of development, it’s common to write unit-tests – tests to ensure that each of the individual components works well in isolation. An analogous thing would be to have individual evaluations to help players identify weak areas, and to measure progress in those areas. It would also prevent players from slipping on important “indicators”.

Other ideas to think about

Some ideas that I find interesting, but don’t yet know if and how they can be adopted by sports teams are below.

Mob programming

In a mob programming session, everyone on the team is working on the same computer, in the same room, at the same time. This seems tremendously ineffective at the face of it, but turns out to be a good way to produce high quality software, quickly.

Hammock driven development

Bugs are incredibly expensive to be found and fixed in production software. They are cheaper to fix in the development phase. But, the design phase is the best place to fix them.

Rich Hickey recommends feeding a lot of information about the problem you are trying to solve to your waking mind, so that your background mind can feed off of it while you are asleep, and make useful connections.

Conclusion

Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done. — The Agile manifesto

This sounds exactly like what players in our team need too. The captains and the mentors in the team should work towards providing the environment and the support everyone on the team needs, and trust them to do the job.

Thanks Meghana Iyer, Ravitheja Tetali, Shantanu Choudhary, Vandith PSR, Varun Rangarajan and Vivek Krishnaswamy for reading drafts of this post and giving helpful suggestions.

Python's globals

Sun, 29 Jul 2018 09:09:00 +0530

TL; DR

The global statement is a parser directive to indicate that globals are being used. When looking up globals, the globals of the current scope are fetched.
Python’s globals are really just module-level.
Every function has an associated __globals__ dictionary, which is the same as the module’s __dict__ for the module where it was defined. This __globals__ dict is the name-space that is looked up when trying to fetch globals within a function.
Avoid globals to make it easier to test functions

Motivation

During a mentoring session I was doing recently, we ran into a problem with globals in Python.

Here is a toy example to describe what we were seeing.

We had code in a module foo.py where the function f was using a global defined in main, and we were trying to write tests for the function f.

# foo.py
def f():
    print(a)


def main():
    global a
    a = 5
    f()

if __name__ == '__main__':
    main()

Running this file foo.py prints out 5 as expected.

We would now like to import f into a different module and use it.

Running `f` in a different module

# bar.py
from foo import f

def main():
    f()

main()

Running the module bar.py gives us a NameError, as expected. The name a was defined in the main in foo.py which is never being run when the code is imported in bar.py

Traceback (most recent call last):
  File "bar.py", line 10, in <module>
    main()
  File "bar.py", line 7, in main
    f()
  File "foo.py", line 5, in f
    print(a)
NameError: global name 'a' is not defined

Setting a global value for `a`

We’d like to be able to run f without running main and the first fix that comes to mind is to set the value of a in bar, and let f use that.

# bar.py
def main():
    global a
    a = 4
    f()

Surprise! Nothing changes.

Traceback (most recent call last):
  File "/tmp/example/bar.py", line 13, in <module>
    main()
  File "/tmp/example/bar.py", line 9, in main
    f()
  File "/tmp/example/foo.py", line 5, in f
    print(a)
NameError: global name 'a' is not defined

Why doesn’t this work?!

Function `globals` and the `global` statement

The global statement is a directive to the parser, that specifies that the variable being assigned to is a global variable.

This can be seen by looking at the disassembled code for f

import dis
from foo import f

dis.dis(f)

5           0 LOAD_GLOBAL              0 (print)
            2 LOAD_GLOBAL              1 (a)
            4 CALL_FUNCTION            1
            6 POP_TOP
            8 LOAD_CONST               0 (None)
           10 RETURN_VALUE

globals for the current frame are fetched, and the value is updated in that dict. Each function in Python has an associated __globals__ dict which is a reference to that module’s __dict__ in which the function was defined. So, in the case where we try to set the a = 4 in bar.main, the main function’s __globals__ dict is being updated.

# bar.py
def main():
    global a
    a = 4
    print(main.__globals__.keys())
    print(main.__globals__['a'])

dict_keys(['__name__', '__doc__', '__package__', '__loader__', '__spec__', '__annotations__', '__builtins__', '__file__', '__cached__', 'foo', 'f', 'dis', 'main', 'a'])
4

As you can see a is being set to 4, but f still doesn’t see value, since it has it’s own __globals__ dictionary. Printing the globals dictionary for f should make that clear.

from foo import f
print(f.__globals__)

If the variable a was declared in the module foo outside of any of the functions, it would be in f’s __globals__ dict when it is imported, and hence the name error would go away, but setting it still would not work.

# foo.py
a = 3

def f():
    print(a)


def main():
    global a
    a = 5
    f()

if __name__ == '__main__':
    main()

# bar.py
from foo import f

def main():
    global a
    a = 4
    f()

main()

Running bar.py would print the value 3 which has been defined in foo.py, and not 4.

Updating `globals`

To update the value of a for f, we could modify it’s globals dict.

# bar.py
from foo import f

def main():
    f.__globals__['a'] = 4
    f()

main()

Module `dict` and monkey-patching

As mentioned previously, a function’s __globals__ dict is a reference to the module’s __dict__ for the module where the function was defined. So, we could achieve the same result as above by updating foo.__dict__. And setting an attribute on the module foo is the same as updating this dict.

# bar.py
import foo
from foo import f


def main():
    foo.a = 4
    f()

If you have used a library like mock to patch some code while running tests, this is essentially what is happening. The target module’s dict is looked up for the specified object/function and replaced with a mock object.

Use an argument to make it testable

The function f would’ve been much easier to test, if it took a as an argument, instead of using a global value. This functional approach would make the code easier to reason about too.

# foo.py
def f(a):
    print(a)


def main():
    a = 5
    f(a)

if __name__ == '__main__':
    main()

# bar.py
from foo import f

def main():
    a = 3
    f(a)

Thanks to Akshaya and Shantanu for reviewing this blog post.

Assist time

Tue, 14 Jun 2016 00:00:00 +0530

I sometimes hang out on #org-mode or #nikola and answer questions. I usually am not able to answer off the top of my head and I look up docs or dig through the code. Sometimes, we find ready-made documented answers, but other times we end up contributing some documentation, filing an issue, submitting a patch or hacking up something for that super-special use-case.

Until now, I looked at this time as IRC time, which loosely translates to distraction time. But, I’m usually learning about the tools I use a little bit more. Even when I’m not, I’m helping someone do something they want to do. Sometimes empowering them to fix future problems on their own. And indirectly making the user community a wee-bit happier, and possibly the software a tiny bit better.

This isn’t limited just to helping someone on IRC. Helping out a co-worker do something that they are new to, or just adding a comment or editing a Stack-overflow answer may end up in the “distraction time” bin, just because you weren’t doing something on your TODO list. It needn’t be.

Taking cue from scorers in Football, I decided to call this time assist time and to try to start seeing this time as (semi-)productive. Naming helps.

Reading-time based scheduling

Mon, 06 Jun 2016 00:00:00 +0530

I had posted a link to an poem written on Medium on a Slack channel that I use with friends. A friend said that she liked the fact that the Slack article preview had the reading time from Medium in it. She could decide whether or not she wanted to read the poem or any other article at that moment.

This gave me the idea for a reading time extension for my browser, or my feed reader or my bookmarks – my reading list. The first version should be able to compute or extract the reading time for an article or a tab in my browser, and index them. I want to be able to specify the amount of time I will be able to spend reading, and be presented with something from my reading list. I think this would help with scheduling the reading of longer articles, and also to actually help me get through my reading list.

Reading time estimates that use heuristics based on word-count may not really work, and may do more harm than good. But, it may still be worth a try to see if it helps my reading habits in any way. A quick search pointed me to this extension, that can give the reading time for any page but doesn’t really do what I want.

Stepping AFK

Fri, 03 Jun 2016 00:00:00 +0530

In the past few weeks, I noticed three instances where I was forced to take a break exactly when I was ready to jump in and write some code to fix a bug or add a small feature. I had to step out of the house and take a walk to meet someone, etc.

I ended up getting ideas during the walks, which significantly changed and simplified how I would’ve implemented things, if not for those breaks. Even if I did end up zeroing down to those solutions, I am pretty sure it would’ve taken a couple of not-so-good attempts and much longer than it did.

Context switches are usually considered to be expensive for programmers, but taking a break at the exact time when I had all the required context loaded into my head seemed to help. It was also probably helpful that I was taking a walk, and there wasn’t really any other inputs or outputs competing for the space in my head.

This got me thinking about doing this more deliberately – I’d love to hear about any experiences or any experiments any of you have done with this. Also, I’m reminded of Rich Hickey’s Hammock Driven Development talk and I wonder if this is a smaller/different version of it, and makes me want to try out the things he suggests more deliberately. If any of you has thoughts and suggestions, I’d love to hear from you!

Thinking about Data Ethics

Wed, 25 May 2016 00:00:00 +0530

Earlier this month, a researcher made a dataset containing the profiles of about 70,000 users public. He didn’t really see a problem in doing this because he felt he was only presenting already publicly available data in a more usable form. was only presenting it in a more usable form.

Yesterday, I came across this quote in the very first chapter of Allen Downey’s book Think Stats which I liked a lot, and reminded me of this incident.

I hadn’t looked at the OKCupid data release and the discussion around it much, but I went back and read this article by a social media researcher who thinks a lot about these things.

She puts forth a lot of interesting ideas to think about ethics. Some things that stood out to me are:

Ask yourself how the person whose data you are using feels about the data.
Taking a ‘what if’ impact approach to thinking about data and ethics.

Also, you needn’t really call yourself a researcher to be actually doing experiments with (or analyzing) “big-data” and discovering and putting out facts that have an impact – however big or small. You should really go read the article, whether or not you are a researcher using data.

Incidentally, there is a meet-up on Data Ethics this weekend in Bangalore. I’m excited to learn and think more about this, and talk to others who care.

Level-up Tools

Mon, 23 May 2016 00:00:00 +0530

Thanks to a friend I got an upgrade to our still-being-setup kitchen. I now have a non-stick pan along with a few more new additions. I would previously use a bowl that people usually use to boil milk etc. for making whatever I did. The non-stick pan feels so great! It has made it a lot simpler to make some of the things I used to, because its non-stick. And it has vastly expanded the possibilities of things I can make, by virtue of being flat and wide based. The pan is such a great addition to my kitchen paraphernalia, and it adds a new dimension to the kind of things I can make. I’m not here to write a user review for it, though.

What are such tools in other things that you do, that drastically changed the way you did something, or added a new dimension to the kinds of things you could do, tools that make you feel like you have a new super-power? Learning to write Python (after starting off with C) seemed to give me so much power allowing me to focus on the problem, rather than fussing over the low level details. Sasha mentions in this post how using a Spaced Repetition System like Anki drastically improved her efficiency because she could focus on thinking about higher level things rather than trying to recall or search for what method or function to use to do something.

What are some such level-up tools for you? Is there a systematic approach to discovering tools?

Tedium in work-flows

Thu, 19 May 2016 00:00:00 +0530

I use Nikola for generating this blog. When creating a new post, it prompts for a title, and creates a file for the post.

Often I’m starting off with only a vague idea that needs to be fleshed out before it can be published (or discarded). It is quite difficult to come up with a title at this stage. I just want to start a draft and write things down!

I could use a “draft-title” and change it after finishing a post, but this feels tedious – requires 3 steps – change the title, post filename and post slug. The last two steps are optional, really, but I feel they are important especially when the original title is very different from the new one.

Being forced to come up with a title before anything else, feels tedious and, adds to the effort required to start off a new post. I shouldn’t really be worrying about the effort required to change the title of an unwritten post, but it happens subconsciously.

To work around this, I now have a “re-title utility” in my editor that takes care of all the tedious details. I can start with a random title, like Draft-1, and change it when I’m done with the post. I feel this is going to lead to a lot more drafts, at the very least, if not published posts.

Another work-flow related thing I came across recently was @Malabarba’s issue on CIDER (an IDE for Clojure in Emacs). The REPL takes a while to startup and this caused him to not use CIDER for running tests, if there wasn’t an already open REPL.

The tedium that people feel effects how they use the tool. Not surprisingly, making tedious-feeling tasks a breeze with the tool also effects how and how much they use it. Subtle variations in a work-flow could make or break it. How do you discover such potential work-flow make-or-break-ers? I think, these things would help:

Use the tool yourself (dog-food)
Talk to (or watch!) people using your tool
Look at work-flows in other similar tools
Thinking explicitly about various scenarios and simplifying or improving work-flows

I’d love to hear examples of this, and any ideas or thoughts you may have on identifying and fixing such things!

Error messages and new users

Tue, 17 May 2016 00:00:00 +0530

I was helping a friend of mine setup his blog and we were trying to use Hexo – a static site generator. We chose a Javascript based tool since he’s trying to learn Javascript. I skimmed through active Javascript projects in this list and finally zeroed down upon Hexo based on its popularity. I promised to help my friend to set this up, but he first tried to do it on his own and got back to me after an hour or so, quite frustrated and almost on the verge of giving up setting it up. I didn’t expect this from a tool that had so many stars, forks, plugins and so much active development.

We finally got it working, but we found that the error messages were horrendous – even for someone who has been using free and open-source tools for a while now. Printing out errors from compiler or interpreter directly along with the stack trace is almost always the worst thing to do for a tool/utility (as opposed to an API or library). The stack trace is definitely useful, for developers trying to build upon or improve your tool. Have a debug or development mode where developers can get all the information they need.

If you care about your users, especially new users, make sure you spend sufficient time on showing human-readable messages. If possible list the possible causes for every error along with tips for troubleshooting.

How I learnt to use Emacs' profiler

Sat, 03 Jan 2015 02:01:17 +0530

I learnt to use Emacs’ profiler yesterday, after many hours of yak-shaving, trying to get Memacs working. Memacs is a memory extension system for Emacs written by Karl Voit, that I have been meaning to try out for a long time now. Seeing lots of review posts at the turn of the year and watching Karl’s recent Emacs Chat with Sacha Chua pushed me to try and finally set it up.

I started writing a module to create a Memacs file – an org archive file – from my browser history. It was pretty easy to write, and I had it spitting out a huge file with 22k entries after about a couple of hours of work. Then I excitedly pulled up my agenda, and turned on the option to view archived entries, only to be super-disappointed. It turned out to be extremely slow! Actually, the agenda never came up with the 22k entries file that I had. At least not in 5 or so minutes, before I got impatient. The performance was unacceptable even when I reduced it to 5k entries.

I was pretty sure it wasn’t that slow for Karl in his demo and tweeted to him, asking for a workaround. Meanwhile, I looked at his dot-emacs, but wasn’t able to dig out what was needed to speed up things. He confirmed that his performance was way better than what I was getting.

First, I ruled out the possibility of it being because of the SSD, since clearly my CPU usage was peaking, and the task was CPU bound and not I/O. Next, I tried using the same file on a different machine (with a different version of Emacs and org-mode), and it worked blazingly fast. So, it was either the version of Emacs or org-mode that I was using.

I should have stopped, thought clearly, and started experimenting with org version, but hindsight is 20-20. I tried Ubuntu’s pre-built Emacs and agendas were fast! I suspected my Emacs build, since I recently started building Emacs from git. I built two or three other versions of Emacs, and wasted a lot of time, before realizing that I wasn’t using the org-mode source bundled inside Emacs for the tests, and there were two “independent” variables.

Finally, I began bisecting org-mode’s source and found that all hell broke loose with an inconspicuous change around release 8.2.6. It turns out that org-overview was broken before this, and collapsing all the trees in a newly opened org-buffer (default option) wasn’t working. Once this bug was fixed, opening huge org files would slow down by a great deal, in turn causing agenda generation to be unbearably slow.

All I had to do was add a #+STARTUP: showeverything to the top of the file. This speeded up things by about 50 times! It turns out, I later found out, that all of this is documented on Worg. I did try a few search engine queries, but sadly none of them brought this up. Adding the following to my config, speeded up agenda generation by about 150-200 times!

(setq org-agenda-inhibit-startup t) ;; ~50x speedup
(setq org-agenda-use-tag-inheritance nil) ;; 3-4x speedup

In the course of all this debugging, I learnt how to use Emacs’ profiler. The profile reports along with git bisect, eventually helped me figure out what the problem was.

To profile the CPU usage, all you have to do is add a call like

(profiler-start 'cpu)  ;; or M-x profiler-start

at the place where you wish to start it. Emacs will then start collecting information about where time is being spent, by sampling every sampling-interval seconds (default 10^6 nanoseconds = 1 milli second).

You can view the information being collected, at any point of time using

(profiler-report) ;; or M-x profiler-report

The report is a nice, interactive tree with the percentage of time spent in each call. You can stop profiling by calling (profiler-stop). If you have more than one report, you can compare them by hitting = in one of the report buffers. I’m definitely going to use this for other things! (like speeding up my startup?)

Now that I have Memacs working with reasonably fast agenda views, I’m looking forward to collecting as much personal information as I can! Thanks Karl for writing Memacs. I am going to be a pretty heavy user, I think! There seem to be a few rough edges, though, and I hope to help smoothen them out a little bit, over the next few weeks.

Martin Fowler on Refactoring @ RubyRogues

Sat, 25 Oct 2014 00:00:00 +0530

I stumbled on a Ruby Rogues podcast yesterday, which had Martin Fowler as the guest. I really enjoyed the discussion on Refactoring (the noun, the verb and the book!)

Martin clarified in the podcast that refactoring (the verb/process) is a sequence of very small refactorings, while you keep making sure that you can run the test suite always. A refactoring (noun) is a change where you change the structure of the code without any externally observable changes, with the intent of making it easier to understand and cheaper to change in future.

I also really liked the metaphor of a ‘healthy code base’. Refactoring is, then, the process of keeping healthy – exercise, speaking metaphorically. You can stack up all the exercise you need to do, until you get really unfit. Refactoring, similarly, needs to be done regularly, to keep the code base healthy. This lets you go faster, in the future.

I also really liked the advise about trying to push back the mental contexts you build, while trying to debug/understand some code that is not very clear, by refactoring the code to make it clearer. Code needn’t be one big chunk of cryptographic text. Code is writing. It should be clear and understandable. Or, at least we should strive to make it so!

The discussion, as always on this podcast, was very lively, pleasant and enjoyable! Enjoy!

Recurse Center, 2014-07-07

Tue, 08 Jul 2014 10:07:20 -0400

As preparation for a one-on-one this week with one of the facilitators, I was wondering if I was really getting better as a programmer, by doing what I am doing.
I have heard at numerous places that reading and reflection are keys to getting better. I feel like I haven’t been giving these things much attention in the past couple of weeks. I don’t catch up on reading all the awesome reading material shared on Zulip and I switched from writing this blog post first thing in the morning, to any-time-after-lunch. I don’t think this worked out very well. Writing the post worked as a way to reflect on what I had done yesterday, and what I should be doing today. So, I am back to writing the blog post, first thing in the morning!
Yesterday, I worked on indexing the Python sources in a way that the inspection code can look up, later. During this process, I found that my code to use libclang’s AST wasn’t generic enough, and I had to clean it up to be able to extract useful information from any file in the cpython sources.
We also got to attend a super-awesome talk by Steve Labnik! He talked about his progression from being an application developer, to writing libraries, to working on languages (as a professional developer). He made a lot of interesting and inspiring points during his talk. Some of those that stuck with me are:
- None of these is particularly harder than any of the other. Depending on each person’s personality, or the way their brain works, they are good at doing one or the other.
- Getting good at programming is a matter of showing up, more than about the “genes”. He repeated quite a few times that he disliked the idea of “baby hacker”, and left out the story of his childhood and college programming days! I’m totally stealing his idea of meeting every saturday at 1pm, with a bunch of friends and working until it was 10pm or so, when they could get cheap beer and food! And he did this all through his college! It is interesting that this idea is so similar to Hacker School!
It was a very enjoyable and inspiring talk on the whole.
The plan for today is to actually have the parsed information dumped into some persistent format, and modify the inspect code to actually use it.
I will also be pairing with Kyle for a few hours on working through some of http://mitpress.mit.edu/books/audio-programming-book

Best Practices for Scientific Computing

Mon, 24 Dec 2012 00:00:00 +0530

Shantanu and I gave a short talk titled “Software Carpentry for Scientists” for the graduate students of Chemical Engineering department, IISc, this Friday. We gave a short introduction to Git, TDD, Numpy/Scipy, etc and mentioned a few things from Greg Wilson et al’s paper.

I promised to revert to them with links to a few resources. I figured it would be more beneficial, if I just put it in a publicly available place.

A summary of the paper by Greg Wilson et. al., is below.

Useful resources

Software Carpentry

Paper by Greg Wilson et. al.
Software Carpentry

Git & version control

TDD

SciPy

http://scipy-lectures.github.com

Python

http://docs.python.org/tutorial

GUI tools in Python

Summary of paper by Greg Wilson et. al.

Write programs for people, not computers
- a program should not require its readers to hold more than a handful of facts in memory at once.
- names should be consistent, distinctive and meaningful.
- code style and formatting should be consistent.
- all aspects of software development should be broken down into tasks roughly an hour long
Automate repetitive tasks
- rely on the computer to repeat tasks
- save recent commands in a file for re-use
- use a build to automate scientific work-flows
Use the computer to record history
- software tools should be used to track computational work automatically.
Make incremental changes
- work in small steps with frequent feedback and course correction
Use version control
- use a version control system
- everything that has been created manually should be put in version control
Don’t repeat yourself (or others)
- every piece of data must have a single authoritative representation in the system
- code should be modularized rather than copied and pasted
- re-use code instead of rewriting it
Plan for mistakes
- add assertions to programs to check their operation
- use an off-the-shelf unit testing library
- use all available oracles when testing programs
- turn bugs into test cases
- use a symbolic debugger
Optimize software only after it works correctly
- use a profiler to identify bottlenecks
- write code in the highest-level language possible
Document design and purpose, not mechanics
- document interfaces and reasons, not implementations
- refactor code instead of explaining how it works
- embed the documentation for a piece of software in that software
Collaborate
- use pre-merge code reviews
- use pair programming when bringing someone new up to speed and when tackling particularly tricky problems
- use an issue tracking tool

3 tips for those shipping (commercial) apps

Wed, 28 Nov 2012 00:00:00 +0530

Here are some very generic (and paraphrased) notes from a short talk today, by Deepankar Sharma.

Whenever you release a new major version, make sure you keep a copy of the whole “ecosystem” to be able to run it whenever you want. At any point in time, you should be able to run any version of your software.
When writing benchmarks/tests/etc., try and ensure that you cover a broad spectrum of test data, to try and replicate the different types of data that users could possibly have.
Don’t develop applications with modes. Be very careful before you add a new mode to your application, effectively adding one more code path to maintain.
(Bonus) Beware of too much extensibility

Advice - Programming in Elisp

Fri, 06 Aug 2010 00:00:00 +0530

Below is a mail sent by Eric Schulte to the org-mode mailing list answering a query on how to write elisp for org-mode. I am reproducing it here, since it is useful advice for me. The actual thread is here.

The way that I learned how to program in emacs lisp was mainly using two commands `elisp-index-search’ bound to `C-h e’ on my system, and most importantly `describe-function’ bound to `C-h f’. With `describe-function’ you can look at the source code of functions whose behavior you are familiar with, you can then copy portions of the code to your scratch buffer where they can be edited and evaluated with `eval-defun’ bound to `C-M-x’. Now with Babel, instead of doing this in the scratch buffer you could do this in emacs-lisp code blocks in an org file, enabling notes and hierarchical organization – it can be nice to have your noodling all collected in one file for later reference.

If you are going to do any serious work with lisp, I would emphatically recommend using paredit-mode, and becoming friends with the Sexp movement functions

C-M-f runs the command paredit-forward C-M-b runs the command paredit-backward C-M-u runs the command backward-up-list C-M-k runs the command kill-sexp C-y runs the command yank

They allow you to manipulate lisp code on the level of logical expressions, the utility of which can not be over stated.

As for working with Org-mode in particular, I’d recommend looking at the documentation and source-code of Org-mode functions with `describe-function’, and then looking for how these functions are actually used in the Org-mode code base with `rgrep'.

For a more structured learning experience, I’ve heard very good things about http://www.gnu.org/software/emacs/emacs-lisp-intro/, although I haven’t used it myself.

Hope this helps. Happy Hacking – Eric

Programming on Noetic Nought

Why my Elfeed index was 0 bytes

The breadcrumbs

write-region and O_TRUNC

Emacs has backup files, doesn’t it?

The fix

GitHub Gists from Emacs Orgmode

Motivation

Hackish wrapper around gist.el

MELPA submission and feedback

Current version: ox-gist

Outro

Thoughts on Technical Debt

What is Technical Debt?

Why take on Technical Debt?

A mess is not a debt, or is it?

Technical Debt Quadrant

Is everything tech debt?

Why should I care?

Keeping track and repaying debt

Heroku apps with custom-domain and Cloudflare SSL

Motivation

Setup

Setup DNS to use Cloudflare

Enable HTTPS always in Cloudflare

Select the SSL option to use

Add a custom domain in Heroku

Add a CNAME for the subdomain

Redirect all requests

Enforce HTTPS on Heroku

Conclusion

2018 in Review

Work

Side projects

Ultimate related projects

Ultimate Frisbee

Writing

Relationships

2019 goals/ideas

Software development practices for sports teams

Motivation

Some ideas for adoption

Ubiquitous Language

Sprints

Retrospectives

Daily Stand-ups

Face-to-face communication

Unit tests vs. Integration tests

Other ideas to think about

Mob programming

Hammock driven development

Conclusion

Python's globals

TL; DR

Motivation

Running f in a different module

Setting a global value for a

Function __globals__ and the global statement

Updating __globals__

Module __dict__ and monkey-patching

Use an argument to make it testable

Assist time

Reading-time based scheduling

Stepping AFK

Thinking about Data Ethics

Level-up Tools

Tedium in work-flows

Error messages and new users

How I learnt to use Emacs' profiler

Martin Fowler on Refactoring @ RubyRogues

Recurse Center, 2014-07-07

Best Practices for Scientific Computing

Useful resources

Software Carpentry

Git & version control

TDD

SciPy

Python

GUI tools in Python

Summary of paper by Greg Wilson et. al.

`write-region` and `O_TRUNC`

Hackish wrapper around `gist.el`

Current version: `ox-gist`

Running `f` in a different module

Setting a global value for `a`

Function `globals` and the `global` statement

Updating `globals`

Module `dict` and monkey-patching