Noetic Nought

Why my Elfeed index was 0 bytes

Fri, 30 Jan 2026 16:58:00 +0530

My Linux machine crashed, while I was in the middle of a video call. I restarted quickly, and continued the discussion. Later, when I was trying to sync Elfeed updates onto the Elfeed Offline app on my phone, I found it acting weird. All the bazillion things needed to get it working were in place, but it was still complaining about the Emacs Elfeed server not being accessible. I tried opening Elfeed inside Emacs and failed! The index file in Elfeed’s DB was empty! Gone! Poof!

Frustrating, but I didn’t have time to look into what happened. I would’ve been furious if I had been using Elfeed for longer and had a lot more metadata saved. But, it was only a few weeks of lost metadata - posts I read, starred, etc. I quickly setup a Git based backup to prevent future losses and moved on.

Later, I found time to dig into what might have happened…

The breadcrumbs

Elfeed stores all the metadata for all the posts in an index file with a content addressed store of the contents of each of the posts. The index file is simply a dump of the hash-table containing the metadata for the subscribed feeds, their entries and metadata like tags, read/unread status, etc.

The DB save happens in elfeed-db-save, which simply dumps the hash-table to disk inside a call to the with-temp-file macro.

(defun elfeed-db-save ()
  ; <snip>
  (with-temp-file (expand-file-name "index" elfeed-db-directory)
    ; ...
    (princ (format ";;; Elfeed Database Index (version %s)\n\n"
                       elfeed-db-version))
    ; ...
    (prin1 elfeed-db)
    ;...
    ))

with-temp-file, as its documentation says, lets you create a new buffer, evaluate the body there, and write the buffer to file. For a moment, I thought there was some temporary file involved, but nope! I guess the name comes as an extension from with-temp-buffer which does create a temporary buffer where the body of the macro gets evaluated. Stripped to its core, this function is:

`(let ((,temp-file ,file)
       (,temp-buffer (generate-new-buffer " *temp file*" t)))
   (prog1
       (with-current-buffer ,temp-buffer
         ,@body)
     (with-current-buffer ,temp-buffer
       (write-region nil nil ,temp-file nil 0))))

`write-region` and `O_TRUNC`

So, write-region is the workhorse which writes the contents of the temporary buffer to disk. It’s a roughly 300 line long C function that essentially opens the file with the flags O_WRONLY | O_CREAT | O_TRUNC (in this case) and then does something to write the contents to the file, etc. Honestly, I didn’t look at anything else too carefully after I spotted the O_TRUNC.

The man page of open explains O_TRUNC as follows:

O_TRUNC :: If the file already exists and is a regular file and the access mode allows writing (i.e., is O_RDWR or O_WRONLY) it will be truncated to length 0.

Voilà!

The write is not atomic. The file first gets truncated to length 0, and then we hope that the new contents get correctly written before something goes wrong.

It now makes sense why the file got truncated to 0 bytes. The crash happened after the open, but before the write. Somehow the crash happened in this short window, and poof!

Emacs has backup files, doesn’t it?

Temporary files with ~ in their file extensions have definitely annoyed me in the past when Emacs created them where I didn’t want them. So, I do know that Emacs has back-up mechanisms out of the box. But, it turns out that the backups occur in code paths that are more interactive, like save-buffer, write-file, etc. And not via the programmatic APIs like write-region or the higher level with-temp-file. save-buffer calls backup-buffer before writing, but write-region is much more low-level and doesn’t deal with backups.

The fix

My “fix” for this is to backup the Elfeed data in a git repository to be able to recover from any such corruptions of data, which I already wrote about here.

I know that any code that uses with-temp-file (or write-region) could be affected by this, and the right fix for this may be to write to a temporary file and rename it. Maybe next time I lose data I’ll actually fix it properly.

Elfeed DB backup hooks

Tue, 20 Jan 2026 23:13:00 +0530

I had a bunch of things running on my laptop – video call with screenshare, my Windows VM, Firefox with a lot of tabs, etc. And my laptop crashed! I didn’t have the time to dig into what, why and how.

Later in the day, I discovered my Elfeed’s DB was gone – blown away. :( I’m guessing the crash happened in the middle of elfeed-db-save, and the data was lost.

I’ve now added some back-up for the DB, since I intend to use Elfeed regularly.

(defvar pc/elfeed-db-save-timer nil
  "Timer for debounced elfeed database saves.")

(defun pc/elfeed-db-save-and-backup ()
  "Save the elfeed database and commit to git."
  (when (and (boundp 'elfeed-db) elfeed-db)
    (elfeed-db-save)
    (let ((default-directory elfeed-db-directory))
      (when (file-exists-p ".git")
        (call-process "git" nil "*elfeed-db-backup*" nil "add" "-A")
        (call-process "git" nil "*elfeed-db-backup*" nil "commit" "-m" "auto-backup")
        (call-process "git" nil "*elfeed-db-backup*" nil "push" "origin" "main")))))

(defun pc/elfeed-db-save-soon ()
  "Schedule a database save after 10 seconds of idle."
  (interactive)
  (when pc/elfeed-db-save-timer
    (cancel-timer pc/elfeed-db-save-timer))
  (setq pc/elfeed-db-save-timer
        (run-with-idle-timer 10 nil #'pc/elfeed-db-save-and-backup)))

;; Save and backup when tags change (elfeed-web usage)
(add-hook 'elfeed-tag-hooks   (lambda (&rest _) (pc/elfeed-db-save-soon)))
(add-hook 'elfeed-untag-hooks (lambda (&rest _) (pc/elfeed-db-save-soon)))

;; Save and backup when new entries are added
(add-hook 'elfeed-db-update-hook #'pc/elfeed-db-save-soon)

Safari and invalid HTTP/2 headers

Fri, 16 Jan 2026 21:37:00 +0530

Elfeed-offline currently has a Dream web server which acts as a proxy server in front of Elfeed’s Emacs simple-httpd server.

simple-httpd supports HTTP/1.1 protocol, while Dream provides transparent upgrading of connections to HTTP/2 — if the client can handle HTTP/2 and the connection is using HTTPS, it is transparently upgraded to HTTP/2.

My proxying code was too simplistic in forwarding the headers too along with the content received from the simple-httpd server. Some of the HTTP/1 headers are no longer valid in HTTP/2. And, Safari (and curl) strictly adhere to the protocol and fail if there are invalid headers. Curl, for instance, fails with the following error:

< HTTP/2 200
< server: simple-httpd (Emacs 30.1)
< date: Fri, 16 Jan 2026 10:57:25 GMT
* Invalid HTTP header field was received: frame type: 1, stream: 1, name: [connection], value: [keep-alive]
* [HTTP2] [1] received invalid frame: FRAME[HEADERS, len=77, hend=1, eos=0], error -531: Invalid HTTP header field was received
* HTTP/2 stream 1 was not closed cleanly: unknown (err 4294966765)
* Connection #0 to host 192.168.1.5 left intact
curl: (92) Invalid HTTP header field was received: frame type: 1, stream: 1, name: [connection], value: [keep-alive]

This was causing issues for @Feyorsh who was trying out elfeed-offline with Safari. Thanks for taking the time to debug the problem and for suggesting a fix!

An offline-friendly Elfeed web UI

Wed, 07 Jan 2026 02:31:00 +0530

I want to read articles from my RSS subscriptions on my phone without signing up to a hosted service or running a public server.

Elfeed, the Emacs feed reader, already lets me manage my subscriptions and read from within Emacs. It comes with an experimental web UI, but needs the server (running on my laptop) to be accessible whenever I want to read. My phone becomes useless the moment my laptop sleeps.

So, I built an alternate web UI with a service worker that caches content on the client side for a smooth offline reading experience. It’s written in OCaml and compiled to JavaScript using js_of_ocaml.

Why bother with RSS feeds?

It is annoyingly easy to fall into the trap of short form videos or other algorithmically “curated” feeds and get sucked into scrolling mindlessly. I fear this is going to get even worse with all the generative AI stuff.

I want to be more deliberate about what I consume. RSS feels calmer and gives me a greater sense of control. Or it may just be nostalgia from good old Google Reader days.

Why Emacs and Elfeed?

Since Google Reader was killed, I’ve bounced between many readers but none of them has really stuck with me. I’ve had different phases of using Elfeed, over the years, though.

I like the fact that everything is local with Elfeed – no hosted services, no public servers. And since it lives inside Emacs, everything from tagging to how entries are displayed can be tweaked.

Why Elfeed offline?

Elfeed also ships a basic web UI that lets me read my feeds from a different device, but only when my laptop is online and reachable. I’d like to be able to read these posts even when I’m “on the road”, say, while waiting for a train or while taking a cab ride.

I knew it should be possible to do this with some client side caching. And, I considered improving the web UI of Elfeed itself but it doesn’t seem to be actively maintained in the last couple of years. Creating a separate project would also give me more freedom to experiment.

How does it work?

Elfeed offline comes with a tiny Dream based webserver that acts as a proxy in front of the Emacs Elfeed webserver for the API requests. The Dream web server also serves the static assets like the HTML, JavaScript, stylesheets and the Service Worker for the web app. The diagram below shows how data flows between the different components.

+---------------+     +-----------------+
|  Emacs Elfeed |<--->|     Dream       |
|     Server    |     |   Web-server    |
+---------------+     +-----------------+
                               ^
                               |
                               v
 +------------+        +----------------+       +---------+
 |    Web     |        |    Service     |       | Browser |
 |  Client    |<------>|     Worker     |<----->|  Cache  |
 +------------+        +----------------+       +---------+

The Service Worker intercepts all the GET requests from the client and responds using a cache-first-with-cache-refresh strategy. This makes the reading experience feel very responsive. The web client also implements a “good enough” clone of the original search functionality of Elfeed to allow searching and filtering content while offline.

The web app also lets me mark entries as read or star them when I’m offline. The Service Worker caches these operations and updates the server when the server becomes reachable. There’s no smart conflict resolution here, and the API requests just overwrite and update the state on the server.

I have been using this for a couple of weeks now, and am quite happy with how responsive the UI feels and the number of posts I managed to read so far.

But, there are definitely some rough edges to smoothen out.

It’s a pain to make sure that both the servers are running when I’m online and want to sync new updates to my phone. I’m considering adding a tiny Emacs helper to manage this.
I need to remember to open the app on my phone to trigger a cache update before stepping away from my laptop. I want to explore using the Background Sync API (not available on Firefox) to possibly make this easier.
The UI could use some improvements to indicate pending syncs when offline and also indicate the last sync with the server to help ensure everything is synced before stepping away from my laptop.
Service workers need the server to be accessible via https. I have a wrapper script around mkcert to make this easy, but some apps on my phone refuse to work correctly when there are user installed certificates.

Why OCaml?

At work, I write OCaml every day and enjoy it. This project seemed like a good one to try out js_of_ocaml since everything is local and I don’t need to worry about bundle sizes, etc.

js_of_ocaml makes it quite convenient to share code between the server and the client side. And given there’s also a “third” Service Worker component in the mix, I thought it would make life easier. For instance, the message types and the serialization code for the messages between the client and the service worker are shared.

I’m also using this project to experiment with Dune’s “soon to be released”™ package management that our team at Tarides is working on building.

How can you use it?

You can try out Elfeed offline’s UI here as a static site (with some posts from Planet Emacslife and The OCaml Planet). You should be able to read posts, search and filter for posts, mark them as read or star them. Any changes you make should be preserved between reloads.

If you are an Emacs user but don’t use Elfeed, you can checkout the Elfeed’s README for instructions to set it up. It also has links to a bunch of blog posts and videos that show off the features of Elfeed.

If you are already using Elfeed, you should be able to set it up quite easily if you’re happy installing some OCaml tooling. The installations instructions are available in the README.

Outro

I have been enjoying using Elfeed offline for the past couple of weeks. And I’m hoping I’ll end up spending much more time reading in it, than fixing and building it. It currently definitely feels like something I built for myself, but I’d love to hear from others who try it out.

Some useful Git configuration for Windows

Fri, 24 Jan 2025 01:25:00 +0530

I’ve recently been working on Windows with a relatively involved git repository and ran into a bunch of issues. Setting these configuration values turned out to be very helpful!

# Allow symlinks
git config --global core.symlinks true

# Don't automatically change file endings to \r\n (carriage return + line feed)
git config --global core.autocrlf false

# Use line feed for line endings
git config --global core.eol lf

# Allow long paths in the repo
git config --global core.longpaths true

It’s also useful to set these as configuration values in .gitattributes in a repository to share this configuration with other people working on it.

Git resources : A post with some resources to better understand git, along with some useful git configuration.

Restoring a broken Ubuntu upgrade

Wed, 20 Nov 2024 15:17:00 +0530

A couple of weeks ago, Shantanu and I were discussing Ubuntu upgrades – his upgrade was borked and he ended up re-installing his system. I was celebrating the fact that I never really had to throw away my OS installation entirely since I switched to Ubuntu after getting tired of Arch Linux in 2011-12.

And on cue, I had a really broken update on my dad’s laptop! It was really broken – no WiFi, no Ethernet connection, no GUI packages, etc.

The crash

I started the update but forgot about it, promptly. My dad was semi-supervising it, since he was using the laptop on-and-off. I came back to it a couple of days later, and I ran an autoremove, assuming everything else in the install went ok. Everything continued to work alright until some time later, we had to restart the X server because it had hung up. And it never came back. And on a full restart, no network – both Ethernet and Wifi.

`chroot` to the rescue!

Thankfully these are not the days when I had access to a single computer, and crashing it meant running to a friend and using their computer to download stuff. I quickly made a bootable USB on my laptop.

I boot my dad’s laptop using the live distro and chroot to the installed OS using the live distro. Live Ubuntu has Ethernet working, thankfully.

For apt-get install and dist-upgrade to work correctly with the chroot, I had to mount a bunch of volumes.

Due to some network manager changes in the update, the chroot system still wasn’t connecting to the internet. I probably should’ve looked at the resolv.conf file first, but ended up going the offline package install route…

Offline updates

I generate a list of missing packages to be downloaded using the --print-uris flag to apt-get on the chroot system; And use wget to download the packages on the “live” system; Then copying these packages to /var/cache/apt/archives/ gets the update to finish.

But more missing packages!

I rebooted into the installed OS, once apt-get dist-upgrade seems to have everything resolved. But, I still didn’t have a network! I had to chroot again into the system, and this time I figured out that there was an issue with resolv.conf being a broken soft-link which seemed to cause the broken network (with Ethernet).

For WiFi, I had to install proprietary drivers for the Broadcom wireless card (BCM43142 802.11b/g/n).

I had to also install the ubuntu-desktop and related packages that seem to have gone missing, and on reboot everything was up and running!

Outro

When I told Shantanu, I didn’t have to reinstall my Ubuntu from scratch doesn’t mean that all my updates were flawless, and things never broke. I have had such broken updates a few times – network updates leaving the system in a broken state after network connections died, Ubuntu upgrade being in a borked state after the xz vulnerability was discovered, etc.

Updates do fail. But, more often than not I’ve been able to restore things thanks to dpkg ’s robustness and awesomness! And in some extreme cases, reaching for chroot to save myself.

There’s probably nothing new to learn or discover in this post, but just some help forum links that might come in handy, in future. In this age of AI answers, it seemed nice to look through some old forum QAs and work things out using them.

Thanks to Shantanu and Kamal for reading drafts of this post.

Responsive Auto Export for Org Hugo

Wed, 13 Nov 2024 00:17:00 +0530

I use ox-hugo to write blog posts in org-mode and publish them using Hugo. I enjoy using org-mode for any writing that I do, including blog posts (when I am able to get myself to write them). After a long hiatus, I’ve been trying to get back to blogging, as this post might have given away. But, ox-hugo’s auto export seemed much slower than I remember it being.

Each time I hit save on my blog-posts.org file, Emacs gets busy for about 10ish seconds exporting the post to Hugo markdown. And this is annoying because I tend to hit save-buffer multiple times while writing in Emacs – thanks, muscle memory!

Enter Emacs profiler

I was on a flight and had some time to dig into this. If I was online, I probably would have looked through the README and/or the issue tracker, but I jumped in with the handy Emacs profiler.

The profiler-report showed that a big chunk of time was being spent in org-id-update-id-locations even before the actual export started. And then, during the export, a bulk of the time was spent in org-hugo--get-pre-processed-buffer. Once I knew the problem areas, I started poking around in the ox-hugo code to “fix” these issues.

Turn off Org ID location update

org-id-update-id-locations scans a bunch of org-mode files and stores a mapping of all the IDs of subtrees to their filenames. If you have a lot of org subtrees this can take a while, even if none of them actually have IDs. It turns out that I didn’t have any ID properties set, and this caused the update function to run before every export!

I simply added a new ID property on one of the subtrees in my blog-posts file to prevent the ID updation from running on every export! I’m not sure how a stale org-id-locations value affects cross links, but at this stage of writing I don’t care about the cross links. (spoiler: The next hack actually nullifies any impact a stale value might have had!)

Cutting down auto export time by 4-5 seconds is great! But, I’m still not happy to wait for 5-6 seconds in the middle of writing my posts. Let’s look at the other hotspot – the buffer preprocessing!

Turn off buffer preprocessing, maybe?

Currently, I don’t have any cross links between posts in my org-mode source. So, I can turn off this feature completely by setting org-hugo--preprocess-buffer to nil. Viola! Hitting save-buffer doesn’t freeze my Emacs any more. I can compose “100s of blog posts”™ in a flurry! ;)

But, if I’m going to have these “100s of blog posts”, wouldn’t it be better to have cross links? But, with preprocessing turned off when there are cross-links, the “auto-export and build” workflow breaks. The variable org-hugo--preprocess-buffer MUST be non-nil to produce posts with valid cross-links. If not, the exported markdown file processed by hugo ends up having broken cross-links, which crashes hugo serve and/or hugo build.

Unsetting and setting the org-hugo--preprocess-buffer variable for the writing vs publishing phase, respectively, isn’t an ergonomic workflow. It’s not an improvement over disabling and enabling the auto-export mode as needed. I want to enjoy auto export with Hugo’s live reload feature.

Moar workaround!

Looking through the code some more, I learnt org-hugo-link first uses a custom export handler, if one exists for a link’s protocol. I decided to piggy back on this functionality and made up hugo: protocol for cross-links.

The hugo link simply contains the EXPORT_FILE_NAME of the linked blog post i.e., name of the exported markdown file (without the .md extension) as the ‘path’ of the link. The custom protocol export handler can then generates a relref shortcode for Hugo to process in the exported markdown file.

This nicely works around the need to preprocess my entire blog-posts.org buffer to generate valid cross-links!

(org-link-set-parameters "hugo" :export #'pc/org-hugo-link-export-to-md "Export Hugo blog link to markdown file" )

(defun pc/org-hugo-link-export-to-md (path desc backend &optional info)
  "Export a link to a Hugo blog link in markdown format."
  (message (format "path: %s, desc: %s, backend: %s" path desc backend))
  (cond
   ((eq backend 'md)
    (if (equal org-export-current-backend 'hugo)
        (format "[%s]({{< relref \"%s\" >}})" desc path)
      (error "Cannot export Hugo link to non-Hugo backend")))
   (t (error "Cannot export Hugo link to non-Hugo backend"))))

Outro

I made a simple helper to make it easier to insert these cross-links with the hugo: protocol. It simply looks through all the headlines with an EXPORT_FILE_NAME property, and allows it to be inserted as a link with the hugo: protocol.

Now, cross-linking posts and publishing posts with cross-links are both a breeze. Stay tuned for the “100s of blog posts” ™ I’m going to write!

Thanks to Shantanu and Kamal for reading drafts of this post.

Made in 2022

Wed, 08 Mar 2023 23:35:00 +0530

Keeping with last year’s tradition, this post comes when we are almost 1/5th through this year.

2022 projects

Expense Tracker

I built a personal Expense Tracker over the month of October using SQLite and Streamlit, because I wasn’t comfortable sharing my financial data with a 3rd party. I’ve tried to keep it simple, so it’s not a pain to use. But, I’ve also tried to keep it extensible, for others to try and use.

The last few months have been quite busy personally, and I haven’t been able to use it as much as I would like to, but hopefully things will settle down soon. A couple of things that I’d like to improve:

The Sqlite file is stored locally, and the data is not synced between different computers.
The tagging/categorization of expenses is very manual, and a little bit more automation would be handy.

artful-dodger

A Next.js project for hosting an art/photo gallery with metadata stored in a Google Spreadsheet and images in a Google Drive (with an optional CDN, recommended). I built this for a friend, who is experimenting with creating “a more equitable model for curating, buying, and selling art”. The gallery is currently in “private beta”, and I can’t link to it. But, here’s a sample gallery here, whose configuration lives here.

ox-gist

I released my first Emacs MELPA package – ox-gist – an Orgmode backend to export and update sub-trees and buffers to GitHub gists. It was a great experience contributing to MELPA. I wrote a blog post about it too.

Termux scripts

I’ve started using Termux on my Android phone and have found the Text-to-Speech (TTS) API quite handy. I’ve a couple of scripts on my phone that lets me use TTS to read out contents on the clipboard or an article from my browser.

Fantasy League Site

With help from a bunch of volunteers collecting and validating data, Jonny and I built a website for the Fantasy League for Indian Ultimate State Championships. There was nothing technically challenging about the project, but it just involved a lot of data cleaning and co-ordination between the volunteers. The data was collected using pen and paper, instead of a digital app for collecting the stats. This made the data cleaning and validation significantly harder. I’d definitely push for digital data collection if people try to do something like this again.

Bookmarklets

A web-page with a bunch of Bookmarklets from my browser’s Bookmarks bar. I’ve had most of them around for a while, but putting them on a web page makes it easy to share and find myself on other browsers/computers.

Music

Unlike 2021, I’ve only been able to record a couple of songs. And this year (2023) hasn’t been any better, so far. I hope I’ll be able to make amends to this, soon, though.

Review of 2021 projects

Earworm

I didn’t make any updates to the package, not because it’s perfect but because I didn’t use it much in 2022 – thanks to how little I’ve been able to record.

While, writing this blog post I realized that workflow had a couple of annoyances. I improved the code to fix them.

I need to move an audio file back-and-forth between my phone and my laptop, one full round trip to add ID3 tags and cover image. This makes me put off uploading recorded files to the site. Also, the new media file needs to be named in a specific format. I’ve added a new CLI subcommand to automate all of this for me.
I always need the full collection of the music files on the computer, where I’m updating the site from. I can just scp the existing site onto my computer and merge the media directories.

Ukulele Tutorials Aggregator

The job of manually updating the site is a lot of work, and I haven’t had the interest to keep doing it.

RPi Server and Phone Backup

The physical setup was accidentally dismantled by folks I share my physical space with, and I haven’t been able to put back everything in place, yet. I’m hoping I’ll be able to get back to doing this at some point this year.

Outro

I’ve already built a couple of small things in 2023, and I hope I am able to find some interesting things to do this year.

Link to the 2021 post.

Moving from Heroku to Fly.io

Sun, 06 Nov 2022 12:19:00 +0530

A couple of months ago Heroku announced their plan to “Focus on Mission Critical” and shut down the free product plan. The ideas is to free up their developer time to focus on the important stuff instead of fighting abuse of the free plans. I’m surprised they invested in the free plans for so many years!

My Heroku Apps

I have a bunch of apps on the free (Hobby dev) plan, and only app on the Hobby Basic plan because it needed a bigger DB. All of these apps are small enough that I could easily host all of them on my shared Hetzner server, where this blog is hosted. But, I like using Heroku for services that I run for other people. It makes access control and deployment much easier, and lets me invite other people to help maintain apps easily, despite constraints like 10k rows in the Postgres DB and Dyno sleeping.

Over the years of using Heroku, I ended up accumulating a bunch of small apps on the service. Most of these apps aren’t actively being used any more and didn’t feel worth the effort of moving.

I decided to let most of the apps die, since they aren’t actively being used. Only a couple of apps are used regularly enough to be worth keeping alive – SOTG Calculator and RSVP app. I turned on maintenance mode on the RSVP app for a couple of days, as an experiment, and I had to bring back the services due to “popular” demand.

Why not pay Heroku?

I did consider just moving up to the Hobby basic plan and paying for running the app. I wasn’t keen on investing numerous hours to get these apps to run again. Akilesh convinced me that the amount to pay Heroku could be collected from the team and wouldn’t amount to much per person. I almost took this route.

But, it bugged me that the amount we’d be paying Heroku could fund a couple of turf practice sessions for the team. I could host the apps on my Hetzner server – we already pay enough for it. But, giving other people maintainer access to apps is important and this was not straightforward with Hetzner.

Also, the SOTG app couldn’t be funded this way.

Why Fly.io?

I ranted about this to my friends on Zulip and Rajesh offered to help me as a way to learn some devops stuff. I was more than happy to have someone to discuss this stuff with and get some help.

We decided to give Fly.io a try despite being skeptical about using yet another free service. Their free Hobby plan includes 3GB of persistent storage space (which neither of the apps currently use, but potentially could). They also built Turboku – a tool to move apps from Heroku. It seemed worth giving a try, and we could always fall back on Hetzner if it didn’t work out.

We had a couple of months before the Heroku shutdown date, when I ranted about this. Rajesh and I exchanged a few links and messages on how to potentially go about doing this, when we got down to it. But, he got busy with other things and now the deadline was only a month away. I decided to use the notes and do the migration myself. I was glad I had the notes to refer to. The initial “research” made the task seem less daunting.

Migrating a simple app

I started with the simpler app first – the SOTG app. I used the Turboku web page and it worked pretty seemlessly! It copied over the environment variables from Heroku and deployed the app.

But, using the Turboku web page meant that I didn’t have a fly.toml file to do any future deployments. I used the fly launch command to create a new app and edited the fly.toml file to point to the migrated app.

Setting up the custom domain was pretty simple too, using the CLI. I just had to run fly certs add sotg.indiaultimate.org after changing the DNS entries to point to sotg-calculator.fly.dev.

The whole thing felt quite simple and I was done in under an hour – reading the docs, installing the tool, trying out different things, getting it all working. Everything. Quite impressive!

Migrating a bigger app

Encouraged by this, I started with the RSVP app. I didn’t think it would be much harder than this, but I guess when there are a few more moving parts, it’s hard to tell what could pop-up.

I used the same approach for the migration – Turboku web page to migrate from Heroku and then ran the launch command to generate a fly.toml file that I edited to point to the deployed app.

The deployment seemed to go fine, but the app wouldn’t come up. I looked at the logs to find out that the app was getting OOM killed. Fly.io provides half the memory (256MB) that Heroku provided. But, I wasn’t expecting this tiny app to need much more.

I removed some default imports of large packages, but it didn’t really help that much. I did consider reducing the Gunicorn’s WEB_CONCURRENCY setting to 1, but didn’t want to take that route unless absolutely required. I looked around the forums a little, and found that using a Dockerfile instead of the buildpacks might improve the memory situation. And it did!

Next, I moved to setting up the custom domain and it worked from the Fly.io side seemlessly. But, the flask_dance library was generating http URLs in it’s OAuth redirect URLs. The app already uses FlaskSSLify, but that didn’t seem to be sufficient. After reading some Flask dance and Flask docs, I found that Werkzeug’s ProxyFix exists exactly for this!

After dealing with these two unexpected problems, I got to the problem that I was aware I had to tackle – Fly.io doesn’t have support for cron jobs. The RSVP app uses some scheduled jobs to periodically sync some metadata from Google Drive, Calendar, etc.

I took a hackish route to work around this. Using the schedule library, I wrote a “cron” script that would run the jobs on the desired schedule. I created a duplicate app to run the cron service, even though it could potentially be run as a different process on the same app. I didn’t want to risk any more OOM kills on the web app. But, it can be a little bit more work to keep the environment variables and other such things in sync between the two apps.

I was quite excited about Fly.io having a datacenter in Madras and was swayed by Fly.io’s marketing about running the app close to the users. I tried this but the performance was terrible since the Mongo DB was in North Virginia. I quickly switched back to using an East Coast region for Fly.io app too.

But, while writing this blog post, I realized that I could move the DB to India. I moved the DB to the AWS Mumbai datacenter, and the Fly.io app to the Madras datacenter.

All’s well that ends well

The app has been chugging along for a few days without any new hiccups. I’m hoping things stay this way for the weeks and months to come, if not years.

On the whole, I found the whole experience more beta-ish than deploying something to Heroku. But, I’m quite happy with how the whole thing turned out. I think things will get smoother with time – with my growing familarity with Fly.io and more people using their tools suggesting improvements.

I am wary of depending on yet another Hobby plan, but neither of these apps make any revenue and I’m not (yet) comfortable charging users for the small set of features they provide.

I’d like to hear stories of other people moving away from Heroku! Feel free to reach out!

Old Talk: Write Prose, Not Just Programs

Wed, 15 Jun 2022 09:54:00 +0530

I’ve recently signed up for the Art of Clear Writing course by Amit Verma. It is still early days and I can’t comment on the course itself. But, the discussion after the first class reminded me of a talk I had given a couple of years ago – Write Prose, Not Just Programs.

A friend, Vivek, had nudged me to do a talk on “Writing for Developers” and I ended up talking to his team about a week later. The PyCon India CFP was open a couple of months later, where I submitted this talk and it was accepted.

Even if a couple of years late, I thought it might be helpful to leave links to the talk video and slides on my blog. So, here they are:

Slides
Video

PS: Ironically, my blog hasn’t seen much activity after giving that talk due to various reasons. I’ve been meaning and trying to change that off-late, though.

GitHub Gists from Emacs Orgmode

Tue, 26 Apr 2022 12:37:00 +0530

TL;DR: I released my first Emacs MELPA package – ox-gist – An Orgmode backend to export and update sub-trees and buffers to GitHub gists. It was a great experience contributing to MELPA.

Motivation

I often share bits and pieces of my journal.org file with others, mostly for reading, some times for comments. The file contains different subtrees for each day with subtrees for different topics, meetings, articles, etc.

Orgmode lets you export subtrees to specific file names using the EXPORT_FILE_NAME property (and FILE_NAME buffer export option when exporting full buffers). It is easy to create these exported files on a publicly accessible webserver using the power of Emacs TRAMP mode. For instance, I often set the property to some thing like /ssh:muse-amuse.in:~/public_html/<filename> to publish stuff.

But, exporting to HTML is a distraction some times because I start fiddling with CSS and styling. Exporting to a simple org file doesn’t seem to cut it because there’s no styling at all, and making longer exports hard to read in a web-browser. Exporting to a GitHub Gist is a nice middle ground because GitHub renders Orgmode files reasonably well. Additionally, the comment functionality is pretty handy.

I could also directly use @defunkt’s excellent gist.el to share stuff as Gists. I’ve used it in the past, and it works well when I don’t care about having the contents of the Gist locally on my machine. It’s also quite easy to update existing/old Gists in this scenario, since there’s a single “source of truth” and gist.el makes it quite easy to update Gists.

But, this isn’t good enough when I want to have a “synced” copy of the text on my machine. Also, it doesn’t work very well when I want to share just a subtree from my notes file, and not the entire file. I’ve to export the subtree to a temporary Orgmode buffer and then create a Gist from that buffer.

Hackish wrapper around `gist.el`

A couple or so years ago, I wrote a wrapper around gist.el to let me post Orgmode subtrees as a Gist and also update them when I edited these subtrees locally. There was no support for exporting entire buffers since that wasn’t my use-case. This wrapper code was just a part of my .emacs and worked reasonably well for me.

A few weeks ago, I ended up procrastinating on writing and sharing notes for a discussion with a friend, and extracted this wrapper as a separate package – org2gist – to make it easy for other people to use.

MELPA submission and feedback

I submitted this as a package to MELPA while being skeptical about whether it would be accepted. I wasn’t sure if my thin wrapper around gist.el would meet MELPA’s criteria for a “Reasonably innovative package”:

MELPA provides a curated set of Emacs Lisp packages, not an exhaustive list of every single Emacs Lisp file ever created. By default, MELPA maintainers will reject packages that duplicate functionality provided by existing packages. Please try to improve existing packages instead of creating new ones when possible.

It was a pleasant surprise to get very helpful reviews from the MELPA maintainers. They not only thought the package was good enough to include in MELPA, but also looked at it’s code and gave great feedback — not just basic quality checks before accepting the package into MELPA.

I’m pretty sure mainting a package repository like MELPA is a lot of work, but as an end-user it mostly just worked for me and I never really gave it much thought. It was eye-opening to submit the package to MELPA and see first-hand the care and effort the maintainers put into what goes into the repository.

@riscy (a MELPA maintainer) suggested a different approach that I hadn’t thought of — turning the wrapper into an Orgmode Export backend.

Current version: `ox-gist`

It turned out to be a great idea which I ended up implementing. The user interface became much cleaner, since it used the Orgmode Export UI. And it also forced me to “complete” the feature-set by adding support for exporting entire Orgmode buffers to GitHub Gists, not just subtrees.

The package now adds an Orgmode export backend that lets you export a subtree or the whole buffer as a GitHub Gist, similar to how you’d export to HTML or LaTeX from Orgmode. The GitHub Gist ID is saved for each exported buffer or subtree and is to update a Gist when re-exporting an already exported subtree or buffer.

Outro

The package has now been added to MELPA as ox-gist. Feel free to give it a spin and give feedback directly to me or on the issue tracker!

I don’t know if anyone else would find the package useful and use it, though, MELPA tells me that 20-odd people atleast downloaded it. I’m happy this code may be to a handful of people.

Also, I now have a new found appreciation for MELPA and the work put into it by the maintainers. I’m glad I took the time out to submit this package to MELPA.

Thanks to Shantanu for reading drafts and helping me restructure this post.

Made in 2021

Thu, 24 Feb 2022 02:23:00 +0530

Some things I made in 2021:

Earworm: A static site generator to listen to audio (and video?) files in a directory. I use it to host a couple of simple websites for music files. It also happens to be the first package I uploaded to PyPI despite writing Python for many years.
Ukulele Tutorials Aggregator: Inspired by Ukealong.com, I set out to create an aggregator for Ukulele Tutorials for Indian songs. I’ve, with help from Kamal, populated about 300 songs on the site. I haven’t been able to keep up with maintaing it, after an initial bunch of work. I’ve wanted to automate more of the workflow, but haven’t yet.
RPi Server and Phone Backup: I’ve started to use a Raspberry Pi to backup data from my phone and laptop to a personal Hard disk. I’ve ansible scripts to reproduce the RPi’s setup, and some scripts to “sync” my phone’s data every few hours, when it’s on the same network. There’s a lot more that I need here, before I can ditch cloud backups of important things.
Music: I made covers of about 10 songs on the Ukulele, that I shared with some friends. I’ve ways to go before I’m comfortable sharing them on the web, publicly.

I’m glad to have had the energy, time and resources to be able to spend time working on some fun things despite everything that 2021 had in store for us. Let’s see what 2022 has in store!

PS: This post is quite late into 2022, but it’s an excuse to post something to a blog that’s not seen any updates since the middle of 2020.

Hat tip to Bryan Braun whose post inspired me to write this

Instagram's user-friendliness

Wed, 24 Jun 2020 20:13:00 +0530

TL; DR: I’m a video content creator on Instagram, and don’t tell me Instagram is user friendly. Being able to preview posts before making them viewable for everyone will go a long way, though.

Introduction

I’ve been using Instagram quite regularly from around the beginning of this year, posting about one video a week on an average. Previously, I used it primarily as a content consumer for about an year, before getting off it for quite long since it started to turn into a big time-suck. In the current stint of using Instagram as a content-creator, I’ve had quite a few frustrating user experiences. On a platform that comes with a giant reputation of being user-friendly, this is a surprise.

A lot of my posts this year have been video content. While posting them, I ran into a bunch of things that didn’t “just work”. For instance, in the first 3 or 4 posts of the Humans of TIKS series, every post had some thing broken, leading to a poor viewing experience.

Some of my friends, though, found it surprising that I was frustrated with Instagram. “Isn’t it very user-friendly?”, I was asked by many. I think Instagram is just riding on their reputation from the past. The newer features, especially around video content, aren’t exactly “user-friendly”. Instagram does focus on giving the content consumer a nice (read: addictive) experience. But, as a creator, there are a bunch of hoops to jump. Content creators probably put up with it just for the reach Instagram gives.

In this post, I’ve listed down everything that I see as broken. I’ve also tried to suggest improvements to each broken feature that would make the experience more friendly. I start with the more common problem scenarios and progress towards the edge cases.

Caption formatting is needlessly hard

I’ve always had to fiddle with the caption of a post, after publishing it, if it had multiple paragraphs. There’s a magic algorithm that Instagram uses to decide if multiple line-breaks in the text actually mean separate paragraphs or not.

After some trial and error, I’ve figured out a few things about it. For instance, leaving a trailing space at the end of a paragraph causes paragraphs to be joined together. But, when the second “paragraph” starts with a non-text character, it doesn’t seem to matter what the previous paragraph ends with, they always are joined together as a single one.

I don’t want to find out these details by trial and error. Adding previews for captions would make this so much nicer. Some documentation for the “magic” would probably help too. Instagram could even look at WhatsApp, its messaging cousin, for some lessons on how to make writing text easier. Switching to a more common markup language would be a nice bonus.

Albums are restrictive

Posting multiple images in a single post (an album) is a pain. All images are cropped to the same orientation – square, portrait or landscape.

Until a few years ago, Instagram required every image to be square. If you want to keep things dead simple for your users, the square image restriction is as simple as it can get. At least it’s easy to understand and remember, even though I can’t post my pictures exactly the way I want.

The current same orientation restriction is worse on both fronts. I’m still left with not being able to post my pictures exactly the way I want. But, also the restriction is not easy to infer from the UI. I had to fiddle with the UI and my pictures for a while, to understand that there’s no way around this, and that Instagram is actually enforcing this restriction.

I don’t want to be thinking about Instagram’s restrictions while clicking my pictures. I want to take the best pictures I can, and upload them with the best possible orientation/crop.

Instagram may have these restrictions in place to ensure the best possible “viewing” experience. But, for the creators the user-friendly thing to do here would be to remove these restrictions. I think, even going back to the universal square image restriction would feel more user-friendly!

Public profiles, really?

I think, a public profile should be viewable by anybody without having to login, or having to use a specific kind of device. But, Instagram’s idea of “public” profiles is quite different.

If you are not logged into Instagram, and are viewing a public profile, you are prompted to login after scrolling down through a few posts on the profile. If you are on a desktop browser, you can’t open up any post from a public profile. You are prompted to login, when you try to view a post. Even on a mobile browser, you are prompted to login after opening 3 or 4 posts. Opening direct links (or opening posts in a new tab) works, though.

Instagram obviously wants to grow as much as it can, and it doesn’t really care about people not on the platform. As a content creator, though, I care about my audience, irrespective of whether they have an Instagram account or not. A user friendly platform would keep their users’ interests ahead of their growth goals and not do such shady stuff. Or at least, tell us the truth and stop calling it “public” profiles.

So many video types!

When I want to post a video on Instagram, I’ve to first decide what kind of a “post” to make. If I want to post a “short” video, it can go into a normal post/album if it is under a minute. If not, I’ll either need to split it up into smaller segments, or post it as an IGTV video. Stories can only be 15 seconds long, or they get chopped into multiple shorter stories.

IGTV supports longer videos – from 1 minute up to 60 minutes long. The videos need to be portrait videos with an aspect ratio of 4:5 or smaller, 9:16 is recommended, while normal posts prefer square (1:1) videos.

Just like with pictures, I don’t want to care about Instagram’s restrictions when capturing a video. I want to shoot my video, and let Instagram do what is required to share that video. I find it strange that there are hundreds of blog posts documenting how to upload an already recorded video to IGTV.

A user-friendly platform would let users upload the videos as they shot them, and prompt about adding some padding, or allowing users to crop and select a part of the video, to match the aspect ratio requirements.

9:16 IGTV videos are broken

The documented aspect ratio for an IGTV video is 9:16, but guess what! They aren’t displayed correctly in the mobile app.

They are displayed correctly in mobile and desktop browsers, though. But, given how Instagram nudges everyone to use the app, I would guess that 90-95% of viewers would be using the app, if not more.

After a lot of trial and error, I figured out that I need to upload my videos at a ratio of 9:21 or lower to be able to prevent the sides from being chopped off. Instagram’s UI or docs don’t mention this at all. Also, this workaround only works when uploading from a destkop browser. I haven’t yet figured out how to prevent mobile uploads from looking broken.

For a visual platform that is so bullish on everything looking perfect, this is horrendous! I don’t know where to place this - is this a bug in Instagram’s processing pipeline? Or is this a part of attempts to provide the best viewing experience? Just display my videos as advertised, please!

Video post-processing is magic

Any video I upload goes through a processing pipeline, that is undocumented, before it gets displayed to the users. This is possibly done to optimize for space, etc., but it has led to broken user experience for at least a couple of my videos.

On one occassion, it made the audio and video go out of sync. To be fair, the video did have some issues with the audio stream data missing for the first 3 seconds of the video, due to a bug in my video processing pipeline. The same video worked perfectly fine in MPlayer, mpv, VLC, and on YouTube. Noticing that Instagram’s post-processing made the audio and video go out of sync, after about 50 people watched it, is such a shame!

To avoid such problems, we decided to make a demo account and upload every video there, before posting it from the real account. Needing to do this seems super broken to me. I still did it, since I wanted everything to work just right. But, that wasn’t to be.

We had a video where we added some background music on the video, and it ended up having a 4-channel audio stream. It turns out the Instagram app on iOS won’t play such videos, and we only detected that after some people in our audience let us know. Instagram has this documented on their IGTV documentation under additional technical details. But, I only read that because things weren’t working. Again, our followers had a broken experience because of this.

It can be hard to “document” all the technical details of the post-processing pipeline. But, it should be fairly easy to detect things like 4-channel audio, and warning users about it!

Even simpler, but user-friendlier thing to do here would be to allow users to view the video after it has been uploaded and processed, before making it viewable by everyone. Currently, draft posts cannot be shared with friends. I’m not sure if they can even be found and viewed from multiple devices – it wasn’t intuitive to do this, when I tried. Just making it easy to preview posts before sharing it publicly would make things a lot lot more user-friendly. It feels quite backward to be using a private account for this to ensure no broken content is published.

This may sound too extreme, but just give me the exact ffmpeg command being run on the uploaded videos somewhere in the docs! I would then be able to verify that my videos are Instagram compliant, without having to upload a dozen videos and checking on multiple devices. With a lot of big brands using Instagram, I think there’d be other “expert” users who would appreciate these details.

Published content is uneditable

I would find the brokenness more tolerable if I could go back and change the uploaded videos. But, Instagram doesn’t let you edit published posts. It does seem like a reasonable restriction, since this content may have been referenced elsewhere. This seems to be Twitter’s take too, for instance.

For something as “heavy” as a video, though, it seems a little too restrictive. It may not be a great idea to allow changing content behind people’s backs. But, I’m not happy about my followers having to watch a broken post. Deleting a broken post and making a new one doesn’t really cut it – it feels spammy.

I’d like to see some kind of versioning added to the videos. Let user upload a newer version, and make it possible for users to see older versions too, if they chose to. Google Drive and Dropbox seem to do this, for instance.

Outro

Instagram started off by allowing users to only post square pictures. It was very easy to undestand and use, even if not very accomodating of different use cases. Everyone and their dog could use it!

A whole bunch of new features have been added in the last few years, trying to compete with the other platforms. Doing the simplest thing - posting a single picture - may still be easy to do, but trying to do anything more than that leads to a frustrating experience quite often. I think the reputation of being user-friendly is now just a vestige of the past.

People who use it regularly get used to its quirks and probably put up with the annoyances, because it gives them a way to reach their audiences with whom Instagram is popular. Instagram is “friendly” to the content consumer, and tries its best to keep the content extremely consumable to keep them coming back for more. The burden of figuring out and making the content fit the restrictions is left on the content-creators. This can often be frustrating.

Thanks to Kartik, Meghana, Nitin, Pratiksha, Ranjan, Ritesh, Shantanu and Vivek for reading drafts (or parts) of this post and helping by boucing off ideas.

Deploying Hugo Drafts, simplified

Sun, 31 May 2020 17:19:00 +0530

This is a follow-up post to my last post, Deploying Hugo Drafts. Kaushal Modi, the author of ox-hugo (a tool that lets you compose in org-mode and use Hugo to export), let me know about the build options in Hugo.

Have you looked at the new _build options: https://t.co/Nj8Uefis6Q. I haven't used them yet, but looks like they fit your use case.
— Kaushal (@kaushalmodi) May 30, 2020

The build options in Hugo let you configure building a post for different use cases like render a post but hide it in lists, or don’t render separate pages for posts and only show them in lists, etc.

To use these build options for my use case – being able to share draft post links with friends, without them appearing in the index page or RSS feed – I just need to add the following to my draft blog posts!

_build:
  render: true
  list: false

But, how do I do this with ox-hugo? I know ox-hugo doesn’t yet have this feature, for sure, because Kaushal mentioned that he hasn’t tried it out yet.

I checked to see if I could use the EXPORT_HUGO_CUSTOM_FRONT_MATTER property to get it to generate this extra metadata for each blog post. But that didn’t work. The property assumes all the arguments to it are top level front matter keys and values, which is good enough for “normal” use cases.

EDIT [2020-05-31 Sun]: I previously manually created the drafts directory with the _index.md, but Kaushal pointed me to more docs, again! :) There’s also an issue for adding an easier way to add these build options from ox-hugo.

I created a subtree like this in my org file, that creates the drafts/_index.md file with the required front matter and build options.

* Drafts
:PROPERTIES:
:EXPORT_FILE_NAME: _index
:EXPORT_HUGO_SECTION: drafts
:EXPORT_HUGO_CUSTOM_FRONT_MATTER: yaml
:END:

#+begin_src yaml :front_matter_extra t
  _build:
    render: false
    publishResources: false
    list: false
  cascade:
    _build:
      render: true
#+end_src

The build options for the index page turn off publishing and showing it in lists. With the cascade option, we can get the options to all the children – any files created in the drafts directory, but also override the render option to be true. In other words, the configuration above would translate to adding the following to every draft post:

title: A How-to on Hugo Drafts
_build:
  render: true
  publishResources: false
  list: false

Now, I just need to make sure that all my draft posts are created in this drafts directory. ox-hugo allows doing this by simply using the EXPORT_HUGO_SECTION property on my post subtree. I added this additional property to my helper function that I use to create hugo blog posts from any subtree.

* DRAFT A How-to on Hugo Drafts :blogging:hack:
:PROPERTIES:
:EXPORT_FILE_NAME: how-to-hugo-drafts
:EXPORT_HUGO_SECTION: drafts
:EXPORT_DATE: [2020-05-31 Sun 16:51]
:END:

I’m now all set to publish draft posts without requiring to use rsync and two hugo builds. :) Publishing a draft post would need me to remove the DRAFT todo keyword, like before, but additionally also remove the :EXPORT_HUGO_SECTION: property on the subtree. This should be quite quick to do, or pretty easy to put into an interactive function if required.

Deploying Hugo Drafts

Sat, 30 May 2020 19:07:00 +0530

Update [2020-05-31 Sun 17:23]: I published a simpler workflow for this, using Hugo’s build options that I wasn’t aware of, because they are relatively new.

A few friends and I run a weekly writing club similar to this. Once we have our drafts ready, we usually share them for reviews, etc.

Previously, I used to use Nikola, and it’s default behaviour is what worked well for this workflow – share the posts URL with reviewers, but the post shouldn’t appear in the feed, post list, etc.

If you set the status metadata field of a post to draft, it will not be shown in indexes and feeds. It will be compiled, and if you deploy it it will be made available, so use with care. If you wish your drafts to be not available in your deployed site, you can set DEPLOY_DRAFTS = False in your configuration.

I moved to using Hugo a few years ago. By default, it doesn’t publish the draft posts, but you could ask it to do so using the --buildDrafts option. But, that doesn’t do what I want – draft posts are treated just like any other post, and they appear in the feeds, post list, etc.

Given how fast hugo builds are, though, it is pretty easy to hack-up the behaviour I want.

Run a build with drafts, say to dev directory.

DEV_DIR="dev"
# Build drafts to dev dir
hugo --cleanDestinationDir -D -d "${DEV_DIR}" > /dev/null

Run a build without drafts, to public directory.

PUBLIC_DIR="public"
hugo --cleanDestinationDir -d "${PUBLIC_DIR}"

Copy the drafts from the dev build to the public build

rsync -ri --ignore-existing "${DEV_DIR}"/ "${PUBLIC_DIR}"

Deploy the public build

The code above is here as a single script. Here is the full deployment script for my blog.

An innocent query that caused Zulip downtime

Thu, 28 May 2020 11:34:00 +0530

Introduction

Recently, a change that I contributed caused some downtime on the Zulip cloud (zulipchat.com). Here’s a user tweeting at us, about this.

I shared this with some friends (who use Zulip) and they asked me to share more about what happened. I began writing an explanation, and it eventually turned into a blog post. The post tries to explain what caused the down-time, without assuming too much knowledge of the Zulip DB schema, but it would help if you used Zulip a little.

The goal of my change was to change how Zulip backend calculates the furthest_read_time for a user – the time stamp indicating when the last message was read.

Previously, the code used simply kept track of the message ID of the messages being read by a user in a field called the “pointer”. My change was to move away from this pointer approach to using the actual timestamp of when a user last read a message.

What is the pointer?

Put very simply, the pointer is just a message ID¹ – ideally, the ID of the “last message” a user has read. Message IDs in Zulip are incremental – a message that has arrived after another one, would have a higher ID than the older message. So, it would’ve been a great way to keep track of the last read message, if all the messages were in “one continuous feed”. But, guess what, Zulip has streams and topics which are precisely what make Zulip such a valuable communication tool!

The pointer can be set to the ID of the message that is currently being read, but only when we are in the “All messages” view. In this view, all messages are ordered chronologically, so any message that is after another message would have arrived after it, and would have an ID greater than the other message. So reading a message in this view means, any message with a lower ID must’ve already been read.

When you are reading messages in a stream/topic (narrowed) view, the pointer is not updated. We can’t be sure that all messages with ID lower than the current message’s ID have already been read. There could be a whole bunch of unread messages in other streams/topics than the current one you are reading. So, the pointer doesn’t get updated until you go to the “All messages” view and scroll over the messages, even if you have already read them in a narrowed view.

Why change this?

Using the pointer to compute the furthest_read_time doesn’t work very well, because it is only updated in the “All messages” view. In other words, furthest_read_time turned out to actually mean something like furthest_read_time_in_all_messages_view.

The furthest_read_time is used in the UI to display the bankruptcy banner – a prompt shown to a user to mark all messages as read, if they have “too many” unreads, and haven’t read new messages in the last 48 hours.

But, this is a problem, especially in Zulip instances with a lot of traffic, where people leave a lot of messages unread until they have the time to catch-up. It is common for people to communicate only in a handful of streams/topics, and leave a lot of messages unread in other streams or topics, that they are hoping to catch-up on, when they have more time at hand. Such users would avoid using the “All messages” view at any cost.

Even in not-so-high-traffic realms, some users just prefer not to use the “All messages” view, and just stick to narrowed views.

So, for these users the furthest_read_time would get updated very rarely, and could potentially remain at a date way back in the past. This meant the users would see the bankruptcy banner despite actively participating in the Realm, even if only in a few topics/streams. This is annoying UX, and what something we wanted to fix.

What did we change this to?

We decided to use the last read message time as the message to consider for the furthest_read_time. The following “innocent looking” query should do that for us, we thought…

UserMessage.objects.filter(user_profile=user_profile, flags=UserMessage.flags.read).last()

Well, it does, but not without bringing down the whole of zulipchat.com after a deploy! 🙈 But, why?!

Some background on how data is stored would be useful to see why.

UserMessage table is huge, my friend

Zulip’s data model has a Message table that stores the content of each message, when it was sent, by whom, to which topic, etc. The UserMessage table is then used to keep track of what the status of this message is, for each user that has received the message. A 32 bit flag field is used to keep track of things like, whether the user has read a message, starred a message, has been mentioned in a message, etc. So, if a messages is sent to a stream with N users, it creates 1 row in the Message table and N rows in the UserMessage table. If M such messages are sent, it would create M*N rows in the UserMessage table. So, pretty soon, the UserMessage table becomes huge! ²

To make common queries on this table efficient, the DB has indexes for them. Some of the indexes on the UserMessage table are documented here, though the documentation was created in the migration context for production deployments. There are indexes for various things, including unread messages.

And here’s our innocent looking query, as a reminder.

UserMessage.objects.filter(user_profile=user_profile, flags=UserMessage.flags.read).last()

This query searches through the read messages for a particular user, and tries to figure out the last message they read. Note that the DB only has indexes for unread messages, not read messages! So, this query isn’t using DB indexes to search for this needle in the haystack, and ends up being pretty damn slow!

Additionally, this query is run for every user when the “home page” is loaded since we want to calculate their furthest_read_time to decide if should show the bankruptcy banner.

Also, Zulip’s architecture is designed to allow using the same app server and DB to host multiple different realms. And zulipchat.com is such a deployment where a few servers are used to serve all the realms. (I don’t know too much about the deployment setup, though).

When a new change is deployed to zulipchat.com and the server is restarted, all the clients of all the users reload. Having an expensive query in that path, would just kill the DB. And this innocent looking query didn’t disappoint!

How did we fix this?

Well, this change was reverted immediately by Tim, and things were kicked back into a usable state.

Later, we decided to take a different approach. We decided to look at a different table (UserActivity) that tracks user activity, mainly for the purpose of analytics. This table keeps track of the last time a user performed an action based on the API calls made.

Among other things, changing any of the flags in the 32-bit flag field is one of the actions that is tracked. Marking a message as read was being tracked too. We use the last time this action was performed to compute the last time a user was “active”, and based on that decide if we should show the bankruptcy banner or not.

Here’s the actual commit that makes this change. But, it turns out that needed a follow-up fix too. 🙈

Thanks to Tim Abbott and Shantanu for reading drafts of this post.

The pointer was used to do a bunch of useful things, but we are currently in the process of removing it. ↩︎
Zulip’s code base uses a concept called soft-deactivation to mitigate this problem on Realms where there are a lot of users who log in very infrequently, or don’t login after an initial period of activity. So, the number of rows are not exactly M*N in all the cases, but in the worst case they are. ↩︎

My Favorite Git Resources

Fri, 22 May 2020 16:07:00 +0530

I often help friends who are new to using git with using it. This blog post is going to serve as a single link to share with them, instead of trying to find individual links over and over.

Understanding Git

I got started with using git after using svn and hg for a few months. But, I would often be lost in the myriad of command line options, putting myself in situations I couldn’t come out of without some scars. I didn’t really understand how git worked, and that was the real problem.

Improving my mental model of how git works, and then trying to fit the commands into this mental model is what made git finally click for me. The Git Parable really helped me understand git from the inside out, and gave me a much clearer mental model of git. There was no looking back from there.

If parables are not your thing, and you’d rather read something that takes a less flowery and more straighforward approach to explaining these concepts, I really liked John Wiegley’s Git from the Bottom Up.

Day-to-day git

Zulip (an open source group chat application that I spend a lot of time contributing to) uses a rebase-heavy workflow, where good commits are merged from different PRs, as and when they are reviewed, instead of waiting for all the commits in a PR to be good to be merged. Developers often have to pull from branches that have been force-pushed to, rebase their commits, squash commits, learn to change git history, etc.

Zulip’s Git Guide is top-class documentation that gives guidelines on using git from a practical stand-point. A lot of new developers use it to get upto speed with Zulip’s workflow that involves a lot of things that I’ve seen devs who have been using git for many years struggle with.

The section on pull request workflow is quite helpful, and should work reasonably well for contributing to other FOSS projects too.

I also like the sections with recipes for common git history edit actions, and the troubleshooting section with recipes to get out of common troublesome situations. Oh shit, git! is a very similar resource to this, that has a bunch of useful “recipes” to recover from different scenarios.

Julia Evans’s exercises approach to learning to explore a git repository is quite interesting, though I haven’t run through all the exercises myself.

Git as a communication tool

I like to look at git history as telling the story of evolution of a code base. The advice that is sometimes given to writers about doing the hardwork, so that the readers can do less of it, holds true for git history too. Communicate the story as clearly as possible for your readers to understand, and don’t just publish a “stream of consciousness” piece.

What does this mean in practical terms, though?

I think editing git history to present a clearer story is completely acceptable. If one tries a handful of different things, and then finally finds the solution to do something, it’s not necessary to have all the failed attempts as commits. Just the final approach can be a commit/pull-request, while documenting the other approaches the commit message or as a comment in the code, if that makes sense.

Expect your collaborators (including future you) to look at not just older versions of code, but the commit history for understanding how a piece of code evolved, for undertsanding the history. Make your git commit messages as useful as you can. Writing a good summary line goes a long way in understanding a change. In most cases a body would be useful to explain the how and the why of the change, while the summary line explains the what.

I like this commit discipline that the Git project itself uses, and was adopted by the Zulip project: “Each commit is a minimal coherent idea”. Simply put, you want each commit to be such that they can be can be deployed (and/or reverted by itself), without depending on any other commits alongside them. You can read more about what minimal and coherent mean in this context in the Zulip documentation.

Tools for Git

Some of the confusion in using git even after understanding how it works is the inconsitent CLI API that it often exposes. There has been some work going into this, to make this more consistent. But, using a nice client can help with this. I use the excellent Emacs client for git called magit. It is an intuitive UI on top of git’s data model, that allows me to do things that I commonly want to do, with just a few hot-keys, instead of typing out elaborate recipes to do things. I highly recommend it, if you are an Emacs user.

It is also helpful to change you shell’s prompt to tell you more about the status of the current directory’s repository. There are a bunch of configurations out there (Stackoverflow, Gists, etc.) that let you do this. The git repo itself comes with a prompt customisation. I like being able to see at least the current branch and dirty/clean state of the repo.

Here are some simple changes to the global git config, off the top of my head. These could make your git workflow more pleasant.

[push]
  # Push to the branch to the same remote branch as the one we are working on,
  # locally. This is the default in Git >= 2.0, so you may not need to set it.
  default = simple

[diff]
  # You could configure a GUI tool to use here. (I haven't used one in a long
  # time, and I don't have recommendations on what to use)
  tool = icdiff

[merge]
  # Shows common ancestor in a merge conflict. Makes it easier to understand
    # the conflict and merge, for me.
  conflictstyle = diff3

# Never garbage collect commits/blobs that are unreachable
# The cost of keeping this data around is negligble compared losing data
[gc]
  reflogExpire = never
  reflogExpireUnreachable = never

Outro

If you have other recommendations or tools that you’d like to share with me, please do! Using git better, and understanding it better, is something that I’m always excited about. I’d also be happy to answer questions, or help you with specific things that you are trying to do with git.

The tools we build change us

Wed, 22 Apr 2020 12:54:00 +0530

A couple of weeks ago, I spent some time building a couple of “automation tools” for myself. It is still early days to talk about their impact, but I have already noticed some changes, and have some thoughts to share.

Thanks to the lock-down, I felt like I needed to do something to get more fresh-air and sunlight. I figured out a nice setup to work from my balcony, when I feel like it. It’s nice to sit in the balcony, since there’s not as much traffic, and there’s enough fresh-air and sunlight.

I usually work with dark themes in all my tools - code editor, terminal, OS, chat clients, etc. But, switching to the balcony during the day makes it harder to read text in dark mode. I have to switch all my tools to light mode. And then switch back, when I come back in. It takes me about a minute or so, each time to make this switch. In a few days, this started to seem like work! And I could see myself needing to push myself to step out, or step back in.

A few days later, I spent about half a day writing a tool that could switch modes for all of these programs with one command. If I was doing the switches manually each time, it would take 2-4 months for the switching time to add up to the 4-6 hours, I spent on getting everything working correctly. So, it’s not really a bad investment. But, what matters more is that I don’t feel a mental block to alternate between indoors and outdoors because of all the extra work that manual theme switching felt like. I am getting some fresh air and sunlight everyday. :)

The other tool I wrote recently was for generating a daily summary of what I worked on from my org-mode based clock entries. I add more detail to the bulleted-list summary that gets generated, and then post it on our team’s Zulip. I noticed a couple of changes because of this.

First, I clock things more diligently than I did before. If I forget to clock into something, I go back and add some manual entries, to maintain a proper log. Second, a couple of other people on my team also have started posting a summary of their day’s work, which is quite nice. This tool may have triggered a change in my teammates’ behavior, though I didn’t really intend it to.

We are creatures of habit, often going through the motions, without needing to think about what we are doing. This is evolution helping us save energy for the more critical tasks. But, our environments can have a big impact on our habits. Having tools around us, that are easy to use, can help us nudge ourselves in directions that we want to be nudged in.

The tools we build can change us; if we let them.

Thanks to Rajesh, Ritesh, Tejaa and Shantanu for reading drafts of this post.

Django ModelFormSet and multiple saves

Wed, 08 Jan 2020 09:45:00 +0530

A friend and I were trying to use the Django ModelFormSets in a project, and we ran into a subtle bug that took a little big of digging into the Django code to identify and work around.

The Bug

We weren’t really familiar with Django’s FormSet or ModelFormSet, when we started. We began by reading the documentation to get started.

The Django docs had an example that looked straight-forward, and showed how to use ModelFormSet.

  def manage_authors(request):
      AuthorFormSet = modelformset_factory(Author, fields=('name', 'title'))
      if request.method == 'POST':
          formset = AuthorFormSet(request.POST, request.FILES)
          if formset.is_valid():
              formset.save()
              # do something.
      else:
          formset = AuthorFormSet()
      return render(request, 'manage_authors.html', {'formset': formset})

You create a ModelFormSet class using the modelformset_factory using the required model, and the required fields.
Use the data from a POST to create an instance of the class from above.
Save the data if the formset validates, and we are done!

We tweaked the example to our needs, and it worked for us. Mostly.

A user reported that after saving the form once with some data, trying to save more data didn’t work! The form would try to save the data already saved in the last save, and fail validation that required some of the fields to be unique.

On the second save, the data which was saved during the last save was still being treated as newly filled in data!

This also caused newly filled-in data during the second save to be lost forever. The user was forced to reload the page and re-enter the data.

The Why

The way Django keeps track of newly filled-in data in the form along with multiple levels of caching causes this bug to manifest.

`ModelFormSet` `queryset` argument

AuthorFormSet (or any ModelFormSet created using the modelformset_factory) takes an additional optional argument queryset. This argument can be used to choose specific rows in the DB that need to be used to populate the initial form rows, if any. If no queryset is specified, all the objects in the DB are used.

`FormSet` metadata

First, metadata about the form is saved in a few hidden input fields (called the ManagementForm). This hidden form keep tracks of metadata like the total number of forms to show, initial form count and the minimum & maximum number of forms to display.

  <input type="hidden" name="form-TOTAL_FORMS" value="19" id="id_form-TOTAL_FORMS">
  <input type="hidden" name="form-INITIAL_FORMS" value="7" id="id_form-INITIAL_FORMS">
  <input type="hidden" name="form-MIN_NUM_FORMS" value="0" id="id_form-MIN_NUM_FORMS">
  <input type="hidden" name="form-MAX_NUM_FORMS" value="1000" id="id_form-MAX_NUM_FORMS">

Initial form count is the count of the number of forms that are pre-filled using data from the DB – essentially, the length of the queryset

Metadata does not change

When the ModelFormSet is used as described in the documentation, the metadata in the ManagementForm doesn’t get updated when the form is re-rendered in the response to a successful POST request. If one form with new data was successfully saved, the number of initial forms in the re-rendered form must go up by 1, but it doesn’t.

Queryset caching problem

Django has caching at the queryset level which means that the queryset doesn’t get updated with the newly created instances, even if they match the query that the queryset uses.

  formset = AuthorFormSet(request.POST, request.FILES, queryset=existing_authors)

existing_authors is a queryset that looks for a specific set of authors in the DB, or may be all the authors in the DB, if that’s what the form needs to do.

So, when a formset has been submitted, and some new rows have been created in the model’s table, the new rows don’t automatically get added to the queryset since it the results fetched from the DB are cached by the queryset. We need to create a new queryset for it to include the newly created rows.

These two levels of caching combined make the bug manifest, as it does. I’m not yet sure what the correct “fix” for this is, but we ended up using a work-around – create a new formset when returning from a successful POST.

The Work-Around

We instantiate a new ModelFormSet instance when returning the form, on a successful POST. This means, the ManagementForm is recreated for the new formset and has the correct metadata about the form, even when being rendered in the response of a successful POST.

The code in the example in Django docs, would look something like the one below:

  def manage_authors(request):
      AuthorFormSet = modelformset_factory(Author, fields=('name', 'title'))
      if request.method == 'POST':
          formset = AuthorFormSet(request.POST, request.FILES)
          if not formset.is_valid():
              return render(request, 'manage_authors.html', {'formset': formset})
          else:
              formset.save()

      formset = AuthorFormSet()
      return render(request, 'manage_authors.html', {'formset': formset})

This blog post should probably become a PR or an issue, but I’m not familiar with how contributing to Django works. This blog post is an attempt to understand the issue better, and potentially help others who get bit by it.

Thanks to Shantanu for reading drafts of this post.

Been there, done that, and still doing it!

Thu, 05 Sep 2019 12:23:00 +0530

I was talking to a friend a few days ago about a new internship she got. She wasn’t sure if and how much she was going to get paid for it. Unpaid internships aren’t cool, and experience doesn’t pay the bills, as they say. Eventually, it turned out that she was indeed going to get paid for it.

But, when she found out that she’ll be paid, she felt way more pressure to “not screw up”. I think working in this mode of “don’t screw up” isn’t very productive. Staying in the mindset of “learn as much as I can” would still be the best way to go about the internship.

Screwing up is a part of the job. People make mistakes and learn from them. Screwing up is a part of gaining experience, and learning. To let people feel comfortable and safe while they are learning, it’s important to have better fallback and recovery mechanisms when things go wrong. The productivity gains are worth investing time into such systems.

I’m going to write down a couple of times I screwed up – one of them in my very first job after college, and the other just a month or so ago. People screw up every day, irrespective of how much experience they have.

Corrupted a server’s hard-disk

In my first job, I had screwed up big time in the first couple of weeks of my job. It’s been a while, and a lot of the details escape me now.

I was supposed to resize partitions on a server’s hard disk so that there’s more space available for the media that we were going to store on the server. The server was pretty newly setup, and there wasn’t all that much data on it, but my bosses (Asokan and Prabhu) had spent some time on configuring it, and setting up user accounts, etc., for all the people that were going to join the project.

I tested the resize operation on a machine locally in the office that I worked in, and it seemed to work. I replicated this on the server, and it went kaput! All the data was lost and I could no longer SSH into it. I was totally freaking out!

But my boss ended up being really calm about it, and asked me to just write to the server hosting service to get a new machine with a bigger media disk. And no, I didn’t get fired from the job. I worked with him for 5-6 years after that, including being hired again at a company he started working for.

Almost Deleted 7-8 years of email

A few weeks ago, I screwed up again. It could’ve been pretty bad, but luckily for me it wasn’t.

I administer a legacy Google Suite for UPAI. We have only 10 email IDs to allocate, and we often recycle them. Recently, I had to merge 3 accounts into a single one to free up 2 new accounts. I transferred data from two of them to the third, and went ahead and deleted them. I did this with a couple of other email accounts before, but they were barely used and nobody really cared about them.

I hadn’t paid attention to what data gets transferred, and what doesn’t. It turns out emails don’t get transferred, and only data in the Google Drive gets transferred. This meant I had lost years of important email communication for the organisation. I freaked out!

My “boss” was again pretty calm, and asked me to write to Google to see if I could get them to restore the data. I looked around, and found that Google allowed restoring deleted accounts for 20 days! I restored those accounts, transferred the email and then deleted the accounts again.

Learning from screw-ups

Learning and improving the systems in-place to prevent such screw-ups is the best thing that we can do from such screw-ups. Just like a piece of software gets hardened by bug reports that are acted upon, over the course of time, other systems can and should too.

As described in this blog post by GitLab on how to take advantage of screw-ups, acknowledging mistakes, fixing them, and improving the systems in-place to prevent such mistakes is a good way to go about it.

Systems needn’t always be super complicated things, but just a little bit of automation, and even simple checklists go a long way in improving work-flows and making processes less error prone. Google’s recovery mechanism definitely helped, and gives me confidence to continue working with these user accounts. Our server had a system admin pretty soon, and he had automated back-ups and the usual stuff in place, for us!

The way a team deals with mistakes sets the tone for how the team works, takes responsibility, and feels about working together. I still screw up after working for about a decade as a developer, and being paid for it. I’m sure I’ll continue to have my screw-ups, even though I’d like to have none. What was nice in both the cases was the way in which my mentors/bosses reacted, and how calm they were about the whole situation.

May you have screw-ups that you learn and grow from!

PS: Google’s Data Migration Service doesn’t support legacy Google Suite accounts, and I ended up using Got Your Back to move email from the accounts I wanted to delete to the other accounts. It worked pretty smoothly!

Thanks to Aishwarya, Meghana, Tejaa and Shantanu for reading drafts of this post.

Isolating flaky Django tests

Wed, 05 Jun 2019 18:37:00 +0530

The Zulip repository had a bunch of flaky tests that were fixed a few weeks ago. I learnt a couple of tricks that I wasn’t aware of during this.

Django’s test runner has an option to run tests in the reverse order. The documentation of this option explicitly mentions that this option is useful for debugging side-effects of tests that are not properly isolated. I wasn’t aware of this option, and it’s a pretty neat tool.

The test runner also has a --parallel option to run tests in parallel, and the Zulip server uses this option when running the back-end tests. I never really bothered to look into how test cases are assigned to different processes. The Django docs explain:

Django distributes test cases — unittest.TestCase subclasses — to subprocesses. If there are fewer test cases than configured processes, Django will reduce the number of processes accordingly.

Each process gets its own database. You must ensure that different test cases don’t access the same resources. For instance, test cases that touch the filesystem should create a temporary directory for their own use.

When running tests in parallel, the set of tests running in the same process can vary quite a bit based on the execution speed of tests in different processes. To add more variation to the order in that tests are run, the number of processes being used could also be changed.

These two tools combined together can get tests to run in different orders than the usual order in which they usually get run, and can help find tests which are not well isolated.

It could be a good idea to run tests in parallel (with a randomly chosen, reasonable number of processes) and with the --reverse flag, once in every few CI runs, or as a cron job periodically.

Thanks to Shantanu Choudhary for nudging me to write this post, and reading drafts of this post. Thanks also to Tim Abbott for making me aware of these tricks.

Thoughts on Technical Debt

Tue, 23 Apr 2019 07:52:00 +0530

I recently watched this short video by Ward Cunningham where he reflects on the history, motivation for, and common misunderstandings of the term Technical debt.

This post is just some of my reflections on Technical debt, based on ideas and thoughts from Ward Cunningham, Martin Fowler and Uncle Bob.

What is Technical Debt?

Ward Cunningham coined the “Techincal Debt” metaphor – he used the metaphor to explain the idea of shipping something with a limited understanding of the problem, in the hope of gaining a better understanding as more people interact with the shipped code.

The difference between the understanding of the problem and how the code models it is going to slow down the developers. This “slowing down” is equivalent to paying interest, until we take out the time to refactor the code to accurately reflect the current understanding of the problem.

Why take on Technical Debt?

When Cunningham says debt, he means shipping code with a partial, unclear or even incorrect understanding of the problem and how it should be modeled. You are willing to get the code out of the door, to improve this understanding.

With borrowed money, sometimes you can do something sooner than you would otherwise. Similarly, rushing software out the door can help us get some experience with it, which would help gain a better understanding and improve the software.

A mess is not a debt, or is it?

Cunningham says that people confuse messily written code with technical debt. He is of the opinion that messy code isn’t technical debt. He’s never in favor of shipping bad or messy code. He emphasizes that code that’s shipped should always be clear enough to refactor easily, when required, even if it doesn’t model our current understanding of the problem very accurately.

Uncle Bob also has a post, on similar lines, where he says, a mess is not a technical debt.

Martin Fowler, on the other hand, argues that the debt metaphor is a useful metaphor to think about and to communicate problems in code — especially to non-technical people — irrespective of the reason for the problem.

Not shipping messy code makes sense. But, the debt metaphor is a useful tool to talk about existing “messy” code, compared to just calling it messy code. Looking at it as a debt lets you think about whether or not it’s worth paying off, and if so when, etc. The Technical Debt Quadrant (explained below) further classified this debt into 4 classes, which makes it even more useful to identify bad code as debt.

Technical Debt Quadrant

Martin Fowler came up with a Technical Debt Quadrant which incorporates other kinds of debt, along with the one Ward Cunningham originally defined.

Fowler comes up with two kinds of divisions of debt — Reckless vs Prudent and Inadvertent vs Deliberate.

When Ward says he’s against writing bad code, I think he means he’s never for taking on reckless debt. Uncle Bob’s mess is also what Fowler would call Reckless debt.

Reckless-Deliberate debt is code written out of not caring enough about the code, or not stopping to think about everything that’s going on with the code, or just being mentally lazy. Code-reviews seem like one good way to avoid this kind of debt.

Reckless-Inadvertent debt is something that’s hard to prevent. It sounds like unknown-unknowns territory to me. The developers or the team would probably benefit from improving their general design skills and/or getting some external help as consultants, etc.

Prudent-Deliberate debt is what Ward talks about, I think, in his video. You know your understanding is not good enough, but you aren’t yet sure what’s the right design is. You may gather more knowledge about the problem while the software is out there in the wild and being used.

Ward talks about better understanding. But, you may not always realize that your understanding is incomplete, when you are shipping the code. This I think would put it in the 4th quadrant - Prudent-Inadvertent debt.

Is everything tech debt?

The Tech Debt quadrant seems to make a lot of things seem to fit into the debt metaphor, but there are some things that I’m not sure about.

For example, it can be a significant time investment to update dependencies. Often, teams don’t update dependencies until it becomes absolutely essential — “don’t fix what ain’t broken”. This strategy of postponing of updates, to use the time for other things, seems like taking on a debt. But, I’m not sure which quadrant to put it into.

Why should I care?

Just calling code messy isn’t really qualifying the mess, and helping us talk about how to tackle it. Thinking about debt while taking decisions seems helpful, especially with further classification of debt into the 4 quadrants.

It helps think about whether we’d like to pay off debts we have taken, and to communicate with non-technical people the kind of “maintenance” or refactoring work we plan to take up, and why it is important to do so. The deliberate debt concept can be used to think and communicate about things we aren’t dealing with, right now, in the interest of shipping software quicker.

Keeping track and repaying debt

If a debt is something we can take on deliberately, and we can take on multiple such debts, what would be a good way to keep track of these debts?

Keeping a list of these debts that we periodically review, seems like a good idea. We check if our understanding has improved from the time we took on a debt, and take out time to incorporate it back into the code.

This list could just be a bunch of FIXMEs in the code with some detailed explanation of the debt, and why it was required, or just a list of issues in the issue tracker with a special tag to make it easy to find them.

Thanks to Shantanu Choudhary and Vivek Krishnaswamy for reading drafts of this post and giving helpful suggestions.

Heroku apps with custom-domain and Cloudflare SSL

Thu, 14 Mar 2019 13:03:00 +0530

Motivation

I have a bunch of small (toy?) apps hosted on Heroku. They are probably used by a few dozen people, at most. For these apps, I’d like the following setup:

Use a custom domain for the apps
- Heroku provides domains of the form appname.herokuapp.com for all apps.
- But, it also has the option to add one or more custom domains
- I usually already have a domain under which I want to add a sub-domain, where these apps would run.
Use HTTPS for all connections
- The domains or the sub-domains usually don’t already already have an SSL certificate. With lets-encrypt, I guess this can change in future.
- Heroku provides SSL certificates for the default domain (appname.herokuapp.com) for free.
- They also have an option to buy SSL certificates for custom domains, but they are expensive!
- I’m going to use Cloudflare’s free SSL service instead. I’d like to have SSL for the full domain, and not just the app’s subdomain.
Redirect all the requests to the Heroku domain to the custom domain
- Nobody should really be using the appname.herokuapp.com domain, in case I’d like to move away from it.
- A few lines of app (Flask) code can do this for us.

Setup

Setup DNS to use Cloudflare

Since we plan to use Cloudflare for the SSL certificates, we need to change DNS settings at our domain registrar to use Cloudflare. We first need to create a Cloudflare account, and let Cloudflare do a DNS scan to add all the existing domain settings automatically. Next we point to the Cloudflare DNS servers on our domain registrar.

Enable HTTPS always in Cloudflare

Using page rules in Cloudflare, add a rule for the whole domain to “Always use HTTPS”. For example, use the url http://*example.com/* and select the Always use HTTPS option.

Select the SSL option to use

Cloudflare provides different options for the SSL setting which changes whether or not traffic is encrypted between Cloudflare and the Heroku app.

Choose Full (strict) to enable encryption between Cloudflare and Heroku.

If you are not so concerned about encryption between Cloudflare and Heroku, you could select the Flexible option too.

Add a custom domain in Heroku

Add a custom domain – app.example.com – in Heroku’s settings.

Heroku provides a DNS Target that needs to be used as the destination for a CNAME setting in the DNS provider (Cloudflare). The DNS Target looks something like foo-bar-123abcdef.herokudns.com

If you chose Full (strict) SSL option, this DNS target cannot be used and can be ignored.

Add a CNAME for the subdomain

Add a new CNAME setting in Cloudflare for app.example.com and use the app’s Heroku domain name (appname.herokuapp.com) as the destination value/IP address.

It is important to use the appname.herokuapp.com value if the SSL settings you chose above was Full (strict). Using the DNS Target provided by Heroku instead (foo-bar-123abcdef.herokudns.com) would give an SSL Handshake error since the SSL certificate provided by Heroku only works for the appname.herokuapp.com domain, and not for foo-bar-123abcdef.herokudns.com.

If you chose Flexible, though, you can use the DNS Target provided by Heroku – foo-bar-123abcdef.herokudns.com.

Also, ensure that the Cloudflare Proxy Toggle is toggled on – the cloud icon is orange, not grey!

Redirect all requests

We redirect all the requests coming to the old domain to the new one.

@app.before_request
def redirect_heroku():
    """Redirect herokuapp requests."""
    urlparts = urlparse(request.url)
    if urlparts.netloc == "appname.herokuapp.com":
        urlparts_list = list(urlparts)
        urlparts_list[1] = "app.example.com"
        return redirect(urlunparse(urlparts_list), code=301)

Enforce HTTPS on Heroku

If you don’t care about a custom domain and just wish to enforce SSL for the Heroku domain (appname.herokuapp.com), Flask-SSLify is the way to go. The app can be hosted on Heroku and we can use the certs provided for free.

from flask_sslify import SSLify
app = Flask(__name__)
if "DYNO" in os.environ:
    # Always use SSL if the app is running on Heroku (not locally)
    sslify = SSLify(app)

Conclusion

I’ve done this setup, or some parts of it quite a few times, but each time I seem to need to look up the documentation. Every time, it seems to take longer than it needs to! Hopefully, this post will make it quick and reproducible.

Catch of the Year Poll on Ultiworld: A pointless exercise

Thu, 07 Mar 2019 20:23:00 +0530

Ultiworld, a popular Ultimate related magazine did a poll for it’s “readers” to pick the Catch of the Year. I wouldn’t have noticed it normally, but this time around Pranav Rajan, a player from India, featured in it. A lot of people in the Indian Ultimate community tried to ensure the whole community took notice and voted in large numbers!

[…] we studied many hours of tape and solicited your submissions to create a list of 16 finalists. Over the next week, you the Ultiworld readers will have a chance to weigh in and select the Catch of the Year.

Studying all these hours of tape and picking some of the best catches seems like a great contribution to the Ultimate community. It gives these players a good platform of recognition and it also gives all of us people to look up to.

But, I find this whole exercise of popular voting quite pointless. What exactly is Ultiworld trying to achieve here? It seems like it is just a competition to see how much popularity and support each player can garner.

Perhaps unsurprising to those who have followed along with our brackets in the past, Pranav managed by far the most votes in the first round, buoyed by a continent’s worth of support. If past brackets are any indication, it will take massive domestic support to take the young Indian star down in this competition. Can Sovell-Fernandez muster it?

In one of the tighter matchups we’ve seen in this year’s competition, Pranav’s U24 World Championships snag captured 56% of the vote to surpass another young star in the making, Allyn Suzuki of California Current. With the margin always close, Suzuki led as late as with 18 hours to go, but Pranav saw a late surge to claim victory.

The poll is unauthenticated. People can vote multiple times, with simple cookie tweaking. Even if we are relying on people to vote in the “Spirit of the Game”, I don’t see what the community gains out of this exercise.

The players, who featured in the poll, themselves are left wondering if they deserved to win given the poll wasn’t fool-proof and given that humans with all their biases easily vote for players they know, or players from their country, etc.

Ultiworld gets lots of page views and the like. The link to the vote was shared with lots of people who probably hadn’t heard of Ultiworld before or don’t care to read it. May be they found a few new readers?

I would’ve liked the whole exercise a lot more if Ultiworld just stopped at picking the 8 or 16 top catches, and putting them all in a post. But, may be that wouldn’t have got so many page views!

I refuse to participate in any such future popularity contests. And I think we all should.

Thanks to Meghana Iyer for sharing her strong views on this Ultiworld poll and for reading a draft of this post. Thanks to Shantanu Choudhary for encouraging me to write this post.

Glue Highlight Reels

Wed, 06 Feb 2019 14:33:00 +0530

As a leader on my Ultimate team, I spend a lot of time talking to my team mates. I get to hear out their complaints, their joys and everything in between. All these conversations got me thinking about what makes someone feel like an integral part of the team, where they are making a meaningful contribution.

Metrics to measure contribution

How do we measure our contributions to the team. What do we look at, to tell ourselves and others, whether we have made a contribution or not? I’m still trying to make sense of this, but here are a few metrics I found.

Highlight or Debacle reels

A lot of us tend to remember things as highlight or debacle reels - we make stories out of these events. If you got that layout grab or D, you tend to remember it and that leaves you with a feeling of satisfaction. If you’ve more highlights to remember than the number of debacles, you feel like you made a positive contribution.

But, I think games are not as “eventful” as we think [citation needed!] – especially when a team is playing good Ultimate. When everyone is making good decisions and executing them well, the game is quite uneventful. The credit for the success is distributed over the team, rather than one or two individuals. Are we capable of identifying this, acknowledging it as a meaningful personal contribution?

Tiredness

When we are done with a day’s play, we only have a few moments stuck in our memory. What remains as a physical manifestation is how tired we are, physically and mentally. A lot of us tend to use this metric, probably even subconsciously.

But, with a big enough squad that is being rotated well, I’m not sure this is a good metric.

MVP awards

At the end of every game, teams have a spirit circle where they discuss each others’ play, and pick a player in the opposition who made a difference to the game - Most Valuable Player. Being identified the MVP is definitely one of the easiest things that is used as a metric for contribution. Only one person gets it in a game, and often, the same person gets it in a few matches over the course of a tournament.

Sure, the MVP played well, but what about the dozen others? If you weren’t the best player, doesn’t mean you didn’t make a difference. Would not being identified by the opposition make you forget those contributions? Is there something the team can do to help recognize and identify them?

Being Challenged

When a game was very challenging, win or loss, people tend to feel happy and satisfied about it.

Just being challenged to their limits seems to make people feel like they have contributed. Or may be just makes them forget about asking this question.

Contributing in other ways

If I’m not having a good day, or I’m injured, I don’t mind filling water bottles and taking it easy. If I’ve been able to alert someone from the sideline and they make a difference, that’s a contribution I’m making.

Better metrics

Stats

Stats might help in a small way. For instance, just using a rotating scheme for calling lines, a lot of the not-enough game time problems seemed to go away. Just being able to assure people that there’s a system in place to ensure they get enough game time, took their minds away from it.

The usual kinds of stats - scores, assists, drops, etc. - might actually only end-up reinforcing the outlook of highlight and debacle reels. But, it may help with triggering some faded memories, which might alter the story we make up from the events recorded.

I wonder, though, if there can be other kinds of metrics that might help bring out some of the “grind” aspects of the game, rather than events. These stats will be harder to collect, but would there be a way to reward someone on how well they made space for the next cutter, or how they shutdown the person they were defending and took them out of the game?

Culture

I am beginning to think the problem is primarily cultural. In a world where we care so much about the number of likes and retweets, I think it’s just hard to remember the “grind”.

Sarah Griffith shared an interesting idea that she calls Glue players - players who just do the basics right, and aren’t necessarily involved in spectacles like a layout D or a super difficult throw.

Matty’s shout outs are for glue plays - reliable fill cuts for big yards, or somebody who committed to fronting their matchup because that was the game plan even though it’s scary as heck to give that cutter the chance to go deep on you.

And anyone who has ever played for Matty knows that these shout outs are a coveted thing, because you know that what you’ve done was selfless and helped the team succeed, and that’s an honor.

Matty Tsang is a great coach who has some revolutionary ideas on how to play and coach Ultimate. I’m not surprised the idea of glue plays goes back to him.

But, as a team without a non-playing coach, I wonder how to bring about this culture of identifying and encouraging glue players. May be having a handful of players who make it their job to do this for a few weeks or months, will set the ball rolling.

Sarah has some advice on how to go about doing this in her article.

When you’re holding tryouts for your team, look for who is facilitating scores even if they’re not always catching them. Look for who’s eliminating their person from the offense with great positioning rather than bidding all over the place on D. Look for who shows up, and who you can count on. And then let other people know. Celebrate it. Glue playing is contagious - make people aware of when it’s happening by calling it out and focusing on it, instead of the glory plays.

The idea of “Glue highlights” replacing the (figurative) highlight reels seems like a very promising idea. I’d like to give this a shot, as a team!

Thanks to Tariq, Tejaa and Shantanu for being excited reading drafts of this post, and encouraging me to publish it.

Noetic Nought

Why my Elfeed index was 0 bytes

The breadcrumbs

write-region and O_TRUNC

Emacs has backup files, doesn’t it?

The fix

Elfeed DB backup hooks

Safari and invalid HTTP/2 headers

An offline-friendly Elfeed web UI

Why bother with RSS feeds?

Why Emacs and Elfeed?

Why Elfeed offline?

How does it work?

Why OCaml?

How can you use it?

Outro

Some useful Git configuration for Windows

Related posts

Restoring a broken Ubuntu upgrade

The crash

chroot to the rescue!

Offline updates

But more missing packages!

Outro

Responsive Auto Export for Org Hugo

Enter Emacs profiler

Turn off Org ID location update

Turn off buffer preprocessing, maybe?

Moar workaround!

Outro

Made in 2022

2022 projects

Review of 2021 projects

Outro

Moving from Heroku to Fly.io

My Heroku Apps

Why not pay Heroku?

Why Fly.io?

Migrating a simple app

Migrating a bigger app

All’s well that ends well

Old Talk: Write Prose, Not Just Programs

GitHub Gists from Emacs Orgmode

Motivation

Hackish wrapper around gist.el

MELPA submission and feedback

Current version: ox-gist

Outro

Made in 2021

Instagram's user-friendliness

Introduction

Caption formatting is needlessly hard

Albums are restrictive

Public profiles, really?

So many video types!

9:16 IGTV videos are broken

Video post-processing is magic

Published content is uneditable

Outro

Deploying Hugo Drafts, simplified

Deploying Hugo Drafts

An innocent query that caused Zulip downtime

Introduction

What is the pointer?

Why change this?

What did we change this to?

UserMessage table is huge, my friend

How did we fix this?

My Favorite Git Resources

Understanding Git

Day-to-day git

Git as a communication tool

Tools for Git

Outro

The tools we build change us

Django ModelFormSet and multiple saves

The Bug

The Why

ModelFormSet queryset argument

FormSet metadata

`write-region` and `O_TRUNC`

`chroot` to the rescue!

Hackish wrapper around `gist.el`

Current version: `ox-gist`

`ModelFormSet` `queryset` argument

`FormSet` metadata