Writing on Noetic Nought

Responsive Auto Export for Org Hugo

Wed, 13 Nov 2024 00:17:00 +0530

I use ox-hugo to write blog posts in org-mode and publish them using Hugo. I enjoy using org-mode for any writing that I do, including blog posts (when I am able to get myself to write them). After a long hiatus, I’ve been trying to get back to blogging, as this post might have given away. But, ox-hugo’s auto export seemed much slower than I remember it being.

Each time I hit save on my blog-posts.org file, Emacs gets busy for about 10ish seconds exporting the post to Hugo markdown. And this is annoying because I tend to hit save-buffer multiple times while writing in Emacs – thanks, muscle memory!

Enter Emacs profiler

I was on a flight and had some time to dig into this. If I was online, I probably would have looked through the README and/or the issue tracker, but I jumped in with the handy Emacs profiler.

The profiler-report showed that a big chunk of time was being spent in org-id-update-id-locations even before the actual export started. And then, during the export, a bulk of the time was spent in org-hugo--get-pre-processed-buffer. Once I knew the problem areas, I started poking around in the ox-hugo code to “fix” these issues.

Turn off Org ID location update

org-id-update-id-locations scans a bunch of org-mode files and stores a mapping of all the IDs of subtrees to their filenames. If you have a lot of org subtrees this can take a while, even if none of them actually have IDs. It turns out that I didn’t have any ID properties set, and this caused the update function to run before every export!

I simply added a new ID property on one of the subtrees in my blog-posts file to prevent the ID updation from running on every export! I’m not sure how a stale org-id-locations value affects cross links, but at this stage of writing I don’t care about the cross links. (spoiler: The next hack actually nullifies any impact a stale value might have had!)

Cutting down auto export time by 4-5 seconds is great! But, I’m still not happy to wait for 5-6 seconds in the middle of writing my posts. Let’s look at the other hotspot – the buffer preprocessing!

Turn off buffer preprocessing, maybe?

Currently, I don’t have any cross links between posts in my org-mode source. So, I can turn off this feature completely by setting org-hugo--preprocess-buffer to nil. Viola! Hitting save-buffer doesn’t freeze my Emacs any more. I can compose “100s of blog posts”™ in a flurry! ;)

But, if I’m going to have these “100s of blog posts”, wouldn’t it be better to have cross links? But, with preprocessing turned off when there are cross-links, the “auto-export and build” workflow breaks. The variable org-hugo--preprocess-buffer MUST be non-nil to produce posts with valid cross-links. If not, the exported markdown file processed by hugo ends up having broken cross-links, which crashes hugo serve and/or hugo build.

Unsetting and setting the org-hugo--preprocess-buffer variable for the writing vs publishing phase, respectively, isn’t an ergonomic workflow. It’s not an improvement over disabling and enabling the auto-export mode as needed. I want to enjoy auto export with Hugo’s live reload feature.

Moar workaround!

Looking through the code some more, I learnt org-hugo-link first uses a custom export handler, if one exists for a link’s protocol. I decided to piggy back on this functionality and made up hugo: protocol for cross-links.

The hugo link simply contains the EXPORT_FILE_NAME of the linked blog post i.e., name of the exported markdown file (without the .md extension) as the ‘path’ of the link. The custom protocol export handler can then generates a relref shortcode for Hugo to process in the exported markdown file.

This nicely works around the need to preprocess my entire blog-posts.org buffer to generate valid cross-links!

(org-link-set-parameters "hugo" :export #'pc/org-hugo-link-export-to-md "Export Hugo blog link to markdown file" )

(defun pc/org-hugo-link-export-to-md (path desc backend &optional info)
  "Export a link to a Hugo blog link in markdown format."
  (message (format "path: %s, desc: %s, backend: %s" path desc backend))
  (cond
   ((eq backend 'md)
    (if (equal org-export-current-backend 'hugo)
        (format "[%s]({{< relref \"%s\" >}})" desc path)
      (error "Cannot export Hugo link to non-Hugo backend")))
   (t (error "Cannot export Hugo link to non-Hugo backend"))))

Outro

I made a simple helper to make it easier to insert these cross-links with the hugo: protocol. It simply looks through all the headlines with an EXPORT_FILE_NAME property, and allows it to be inserted as a link with the hugo: protocol.

Now, cross-linking posts and publishing posts with cross-links are both a breeze. Stay tuned for the “100s of blog posts” ™ I’m going to write!

Thanks to Shantanu and Kamal for reading drafts of this post.

2018 in Review

Thu, 17 Jan 2019 15:02:00 +0530

Here’s my review of 2018 - my first successful annual review. I remember wanting to do something like this, a couple of times before too. This time, I managed to get started and wrap it up!

I’ve been inspired to write one by a some reviews that I read this year – some of them do it every year.

Sher Minn’s visual retorospective
Nat’s month-wise review, with a review of 2018 goals and a bunch of goals for 2019.
Julia Evans has an annual review with a bunch of “conclusions” listing what worked in 2018.
James Clear answers three questions every year - what went well, what didn’t, and what were the learnings.
Sacha Chua’s reviews were one of the earliest reviews that I read, and they inspired so much awe! I may not be able to do as much quantified-self type posts as her, any time soon, but I love them!
Buster Benson does annual reviews on his birthday every year. He gives himself a motto to live-by, each year.

My review isn’t as systematic as any of these. But, I have some inspiring examples in front of me to aspire to. Hope this will start something that I can keep doing annually from now, if not more frequently.

I’ve divided my review into a few different areas - based mostly the time I have spent on them, how important I feel they are, and the extent of notes I have in those areas.

Work

I continue to work on RSR at Akvo.
The most challenging project was to wrangle a legacy permissions system that worked at an organization level to give some kind of user level permissions. The project was plagued with incomplete communication of requirements from our partners, failure to understand the complexity of the legacy code, and weird bugs manifesting in even weirder ways. We eventually delivered something, but I’m not at all happy with the solution, how long it took and how the whole project felt.
The company has been in a cost-cutting mode, and has been rethinking the way the products are positioned - transitioning more into a single solution/experience for our partners. This makes a lot of sense – the focus shifts from building tools to solving problems, which is great! But, I wasn’t particularly happy about how much RSR features in these conversations. Towards the end of the year, our team has managed to get more conversations going on involving RSR, but we need more.
A couple of friends asked me if I’d want to work for their companies, but I decided not to. But, these were opportunities to think more about what I want out of my job and the work I do. This is a work in progress.
I did some mentoring at one friend’s company, and got to know the team and their work pretty well.
I gave an interview at a company that I came across at PyCon which went horribly, partially because I didn’t prepare well and partially because of interview jitters. I was expecting a better interview experience from them, though, given how much I liked their team culture, etc.
I also happened to take my first ever front-end interviews, along with a team mate. I really liked how he led the interviews, and made the interview candidates feel comfortable. I learned a thing or two, including the fact that I don’t know a lot of things that people with a more structured and systematic approach to learning front-end development would know.
I didn’t attend too many events this year, like always. But, I really enjoyed PyCon - especially the Keynotes by Sidu Ponnappa and Armin Ronacher.

Side projects

I worked on a bunch of side-projects this year, some of them abandoned, some completed and found useful, some abandoned after the first version turned out to be not so useful. Most of the stuff was “webby”, and unsurprisingly a bunch of my side-projects were Ultimate related.

Zulip

I spent a quite a bit of my spare time between about March and September contributing to Zulip – mainly as a mentor on the zulip-terminal project, apart from some minor contributions around improving setting up of the dev environment for the zulip-server project. I also facilitated a Zulip sprint during PyCon at Hyderabad.

I also got to attend the Zulip India Summit! It was great to meet a whole bunch of Zulip contributors. I especially enjoyed the conversations with and the presentations by Tim and Greg, mostly around developer productivity.

I’m really impressed and inspired by the amount of care and effort that is put into stream-lining the process of contributing to the project, and the emphasis on developer productivity.

I played around with a little bit of React Native development, but wasn’t able to devote much time to it. I’d like to play around some more with it, this year.

Find Playo Venue

I wrote a simple map based tool to find the most suitably located Playo venue with good ratings, when some of my friends and I were playing a lot of badminton together. I still use it sometimes, but we’ve mostly figured out our “favorite” venues.

moditweetarchive

I built a site inspired by this Trump tweet archive for Modi’s tweets. I used Dash to build it, and it was quite fun, despite a few annoyances. This tweet archive, though, isn’t as interesting as the Trump archive, since the tweets are more calculated and strategic. But, a few things could be added to make it more useful.

The repo is private as of now, but I’m happy to share the code, if anyone is interested.

Blaggregator

I spent a couple of weekends maintaining Blaggregator. I updated Django to 1.11.x which is also something I did on Akvo RSR (my work project), later during the year. I also worked on making the crawl faster by handling dead blogs better, etc.

weblogviz

I tried to get started with learning some Rust, and as an exercise, I started building a tool to analyze Apache logs and show some useful stats.

Org-mode to Zulip helpers

I wrote some Emacs helpers to post to Zulip. I’m quite happy with being able to post longer messages from inside Emacs - this lets me retain a copy of things like my checkins or interesting links in my notes/journal file.

aradhana.org

I tried to help a friend of mine build a newer website for aradhana.org, and spent about a couple of days on it. But, there were other bigger problems which needed attention from my friends, and this site just fell by the wayside.

Abandoned projects

There are bunch of other projects that I’ve abandoned. I may end up spending some time looking into them, this year.

SOTG Calculator

I had the chance to look at the Spirit scoring sheets for a couple of tournaments, in the beginning of 2018, and was utterly disappointed. I always knew that it was a lot of work to get the top scores just a few minutes after the final game, but I found a lot more mistakes than I expected – a lot of them were easy to avoid with just a little automation.

To help improve the situation, I built a simple webapp to compute the scores from a Google Spread Sheet. Finding the initial adopters turned out to be very hard, despite the app making life a lot easier for people. It was disappointing to see people saying they didn’t have enough time to try the app, and instead chose to manually compute everything that the app could’ve done with a couple of clicks!

But, eventually it started to be pushed as the official tool starting this season, and has found a bunch of happy users.

Team RSVP

We used Whatsapp to co-ordinate team practices and to keep track of the players attending, etc. But, this started to get quite spammy, and it was a hassle to have other conversations on the group, while folks were calling in for a practice. It was a nightmare to have people call in for two different events at the same time.

To see if people were willing to switch away to something less spammy, a friend and I started with a simple RSVP app and most people seemed to not mind using it.

Unsurprisingly, the app eventually got a lot of features - to make it easy to manage team practices and team resources like photos and training material. It is being used regularly by the team, without too many problems.

As an after thought, may be we have an over-engineered solution, and just asking everyone to use (Google) Calender & Email would’ve worked.

Huddle magazine archive

The Huddle was a good Ultimate related magazine run by Ben & Andy a few years ago. They eventually stopped running it, and the archives disappeared from the original site but are available on the USA Ultimate website which is super hard to read! I wrote some code to scrape the site, and convert the articles into markdown posts and made a more readable version of the magazine here.

I tried to get in touch with Ben & Andy, to ask if they were interested to replace the “coming soon” message on the original site with these archives, but didn’t succeed. Posting it on Reddit and sharing it with my team got some people interested, and hopefully they keep using it.

vquiz

To help my team with understanding rules better, I built a light weight tool to easily create video based quizzes - embed video and ask related questions, use videos to show answers.

To build a question bank, I tried to get some people on the team to volunteer, but it didn’t really take off.

Ultimate Frisbee

2018 was filled with a lot of Ultimate - not just playing, but also thinking, talking and planning.

Our team had a couple of the best tournaments that we’ve since I’ve been playing with the team - not just in terms of the results but in how much fun we all seemed to have together.
I captained the team in a bunch of tournaments. I’m not sure how exactly I got to be the captain – I guess it was just stepping forward for a tournament and offering to spend the time and effort required. It was a great learning for me, and I’ve learned some things about people and leading them that I would’ve taken much longer to learn. I also think I did get better over the course of the year – I got more confident and the team also trusted me much more.
In the past, game time at tournaments has been a big concern. We experimented with a bunch of simple rotation systems – essentially just systems to keep track of how much each person has been playing, and trying to prevent too many imbalances. People were encouraged to bring forth their concerns and ideas during the games, rather than keep everything until the end of the tournament when it is not fixable until the next time!

We had the luxury of being able to try this since everyone on the team had more or less the same skill level, and everyone trusted the system. It helped take players’ minds off playing time, and helped them focus on actually making the most of the opportunities on the field. Most of the complaints about game-time seem to have gone away.
Communicating the reasoning behind a decision is as important as communicating the plan. Often, I’d hear surprising responses to a plan or just generally disappointing comments like how decisions were being made arbitrarily, when we were actually spending a lot of time and energy to try and make decisions. On most occasions, I felt the reason was the lack of an understanding of the reasons behind a decision.
Team communication was another tricky thing that I’ve had to wrestle with. The team still needs to find a balance between communicating over a WhatsApp group or discussing things in person, and the fact that most people are very busy to make in-person meetings with a good attendance happen. We are experimenting with a few things to see what works and what doesn’t.
At tournaments, having a positive environment where everyone is supportive and has each others’ back is the most important thing for me. Irrespective of how much practice we had together and what kind of strategies we practiced, just having a team where everyone is positive and helpful goes a long way.

On the other hand, this positivity is also such a fragile thing that needs to be carefully protected and nurtured. It’s very easy for one negative comment or thought to spread negativity and discontent like wild-fire.

I think as a captain, I should’ve strived harder to safe guarding and cultivating this positivity in the team.
Thanks to some discussions I had with my teammates, I spent a lot of time thinking about what it means to feel like a valued member of a team, and what it means to feel like you are contributing and making a difference. Being chosen an MVP during a game is external validation that you made a difference in a game, but only one or two people get that validation in each game. What about the rest of the dozen or so other people? How can we as a team recognize and acknowledge each others’ contributions to each point or game?

Some of these questions will continue to take up some of my mindspace this year, whether or not I continue to captain the team.

I also took up the role of UPAI’s Director of Technology (aka sys-admin) along with AK. It’s mostly been minor email or slack related change requests, and nothing more. But, it’s been fun to get to know some of the people in the Ultimate community a little better. I also helped organize the UPAI conference.

Writing

I wrote only 8 blog posts, but 6 of them in the second half of the year!
Some friends and I started a Writing Club in July, and it’s been going strong so far. It has not only helped me publish at least one blog post a month, but based on the responses I’ve received, I feel it has also improved the quality of my writing. A lot of credit goes to my friends who read initial drafts of my writing, and gave me really good suggestions on how to improve them.
My journaling and personal note taking improved slightly as compared to last year - I have more notes for 2018, than 2017. And that’s how I was able to write this review post. But, I think the notes are restricted to a few areas of my life, and I can take some actions to write about other things that are important too.

Relationships

I started a couple of Zulip instances with my friends on it – one with my college friends and another with my Ultimate team mates on it. My friends from college went back to WhatsApp, but the other Zulip realm has had some pretty conversations, especially on #beanbags. It hasn’t been super active towards the end of the year, but I’m really happy it exists, and hope it continues to grow this year.
It was good to have some more teammates turn into friends. Thanks to all the Ultimate stuff I’ve been doing, I got to spend a lot more time with some of the people that I previously hadn’t interacted much with.
Tarle, a teammate and friend, passed away. 4 months and he’s still dead. We all miss him a lot! But, we’ve also come together stronger, in some ways.
I’ve tried scheduling regular check-ins with some friends who live far away. They worked for a bit, but seem to be easily swallowed by higher priority things. When they did happen, though, they were quite nice!
Towards the end of the year, I moved houses. The new house is well furnished and has neighbors who mostly mind their business – I had a lot of friends and family visiting, which has been good.

2019 goals/ideas

I want to give having a motto for an year a shot, a la Buster Benson. My motto for 2019 is “Be Deliberate”.

I’m not sure I know exactly what this entails, and I’m sure I’ll get a better understanding of it, as the year goes by. But, here are few things that I think would be good starting points:

Choose what I read more deliberately, and read for understanding rather than just information. Keep better notes of the books I read.
Go back to using Howdy or creating check-ins with more people, and making sure that they happen. Prefer meeting in person or audio/video calls, rather than chat.
Journal more about feelings and thoughts, especially when making decisions, rather than just about events. Journal about more areas of my life rather than just “work” related stuff.
Consider hiring an editor to help write better blog posts, and improve writing style. Getting reviews and suggestions from my Writing Club friends has been good, but this would be a level up.
Look for mentors in other areas of life - especially those where I feel like I’m plateauing.
Pick side projects more carefully. One thing that I would be deliberately thinking more about would be working on side projects that pay.
Write quarterly posts to document my improved understanding of the theme – Be Deliberate – and to track progress I’m making on these goals.

Thanks Kartik Krovvidi and Shantanu Choudhary for reading drafts of this post, being excited about it, and useful suggestions to make it better.

HT interactives

Thu, 28 Sep 2017 00:46:00 +0530

I recently discovered the interactives by HindustanTimes, and thoroughly loved some of them. It felt like a fresh breath of air amongst all the other Indian data driven journalism that I have been coming across in the recent past.

The visualizations are a lot more interactive than some of the other Indian publications. These remind me of NYT’s interactives, which are amongst the best you’d see, while more often than not exploring data in the Indian context. I really like that they explore a variety of ways of doing these visualizations, and don’t just fit the data into one of the many standard options available in a visualization library.

It feels like they are willing to take their time to do these visualizations, and are willing to collect data from multiple sources, and combine multiple datasets to tell a story. I also like the fact that the visualizations and the articles are more open-ended, allowing users to interact with and explore the data, rather than trying to come up with an article full of conclusions with a click-baity title. The fact that one of the sections is called Just for Fun speaks to me.

The data and the code used in most of these interactives is open and available on GitHub, which is nice!

On the whole, I think they are doing pretty good work, and I shall be on the lookout for even better stuff from them. I’m sure I’ll pick up a thing or two from their work, in my projects. Hopefully, other teams doing similar work will also learn from there and produce more good stuff for greedy readers like me.

Emacs frame as a pop-up input

Thu, 14 Sep 2017 22:26:00 +0530

I wanted to try using a dialog box/pop-up window as a prompt to remind me to periodically make journal entries. I had the following requirements:

Simple, light-weight dialog box that allows text of arbitrary length
Ability to launch the dialog from the shell
Ability to have some placeholder or template text, each time the dialog is shown
Save the input text to a specific org-mode file
Write as little code of my own, as possible, to do this

I had initially thought about using a tool like zenity, or write a simple dialog box in Python using Qt, wx or even tk, and then yank the input text at the desired location. This probably wouldn’t have turned out to be too hard, but getting things to look and work exactly the way I wanted would have required more code than I was willing to write or maintain.

After avoiding doing this for a while, I finally realized that I could simply use Emacs with a new frame with the appropriate dimensions, and with the correct file/buffer open to the desired location. This would

eliminate the need for me to write the UI myself
eliminate the need to do text manipulation in code, to yank it at the right place, in the right form. By directly opening up the editor at the required location, the onus is on me (as a text inputting user) to put it in, the way I want it.
additionally provide me the comfort of being able to write with the full power of Emacs - keybindings and all that jazz.
let me leverage elisp to do essentially whatever I want with the buffer being displayed as the dialog box.

I ended up with a command that looks something like this

emacsclient -c -n\
            -F '((title . "Title") (left . (+ 550)) (top . (+ 400)) (width . 110) (height . 12))'\
            -e '(pc/open-journal-buffer)'

This worked pretty nicely, except for the fact that with gnome-shell, the pop-up frame doesn’t always appear raised. It often gets hidden in the Emacs windows group, and the whole idea of the pop-up acting as a reminder goes for a toss! But, thanks to this Ask Ubuntu post, I could fix this pretty easily.

emacsclient -c -n\
            -F '((title . "Title") (left . (+ 550)) (top . (+ 400)) (width . 110) (height . 12))'\
            -e '(progn (pc/open-journal-buffer) (raise-frame) (x-focus-frame (selected-frame)))'

Back, Hopefully

Fri, 01 Sep 2017 22:47:00 +0530

I haven’t written anything here for almost a year. I needed to break the silence. So, here we go with a not-so-useful post showing how frequently I have been posting to this blog, to get a sense of how long this break has been in comparison to other silences in the past.

Neither the code below, nor the plots are very insightful. But, I just hope this will get me started on the path to blogging more regularly. See you around!

Parsed post content

I wrote some code to parse the content of the blog, and each post object looks something like this:

{'date': datetime.datetime(2010, 3, 17, 18, 30, tzinfo=<UTC>),
 'draft': False,
 'tags': ['blab', 'life', 'poem'],
 'title': 'Just another bunch'}
Post count: 190

Post frequency by year

import pandas
posts = pandas.DataFrame(posts)
counts = posts['date'].groupby(posts['date'].dt.year).count()
plot = counts.plot(kind='bar', figsize=(8, 6))
plot.set_xlabel('Years')
plot.set_ylabel('# of posts')

<matplotlib.text.Text at 0x7f7d99ee20f0>

<matplotlib.figure.Figure at 0x7f7d99ea7390>

Post frequency by month

# Add a DatetimeIndex to the Dataframe
posts.index = pandas.DatetimeIndex(posts['date'].values)
counts = posts['date'].groupby(pandas.TimeGrouper('M')).count()
ax = counts.plot(kind='bar', figsize=(12, 8))

n = 5
ticks = ax.xaxis.get_ticklocs()
labels = counts.index.strftime('%Y-%m')
labels = ax.xaxis.set_ticklabels(labels[::n])
ticks = ax.xaxis.set_ticks(ticks[::n])

ax.set_xlabel('year-month')
ax.set_ylabel('# of posts')

<matplotlib.text.Text at 0x7f7d99e2f240>

<matplotlib.figure.Figure at 0x7f7d99e2cf98>

Work-flow

I jumped onto the hugo bandwagon too.

I was totally impressed by how fast it is, and have been meaning to try it out for a while, but wasn’t impressed with the built-in org-mode support it came with. This changed when I finally came across the ox-hugo package that does a wonderful job of exporting blog posts from an org file to hugo’s markdown format. I have contributed a couple of patches to it, to make it work better for myself and hopefully for others too.

Also, for this post, I used ob-ipython with the enhancements from scimax and it has really made the whole experience quite enjoyable.

Among other things, I think one of the reasons for those peaks in the second half of 2010, was having a smooth work-flow. My current work-flow feels pretty nice too, and I hope it’ll reduce some of the friction in writing more posts.

Onwards!

Recurse Center, 2014-07-07

Tue, 08 Jul 2014 10:07:20 -0400

As preparation for a one-on-one this week with one of the facilitators, I was wondering if I was really getting better as a programmer, by doing what I am doing.
I have heard at numerous places that reading and reflection are keys to getting better. I feel like I haven’t been giving these things much attention in the past couple of weeks. I don’t catch up on reading all the awesome reading material shared on Zulip and I switched from writing this blog post first thing in the morning, to any-time-after-lunch. I don’t think this worked out very well. Writing the post worked as a way to reflect on what I had done yesterday, and what I should be doing today. So, I am back to writing the blog post, first thing in the morning!
Yesterday, I worked on indexing the Python sources in a way that the inspection code can look up, later. During this process, I found that my code to use libclang’s AST wasn’t generic enough, and I had to clean it up to be able to extract useful information from any file in the cpython sources.
We also got to attend a super-awesome talk by Steve Labnik! He talked about his progression from being an application developer, to writing libraries, to working on languages (as a professional developer). He made a lot of interesting and inspiring points during his talk. Some of those that stuck with me are:
- None of these is particularly harder than any of the other. Depending on each person’s personality, or the way their brain works, they are good at doing one or the other.
- Getting good at programming is a matter of showing up, more than about the “genes”. He repeated quite a few times that he disliked the idea of “baby hacker”, and left out the story of his childhood and college programming days! I’m totally stealing his idea of meeting every saturday at 1pm, with a bunch of friends and working until it was 10pm or so, when they could get cheap beer and food! And he did this all through his college! It is interesting that this idea is so similar to Hacker School!
It was a very enjoyable and inspiring talk on the whole.
The plan for today is to actually have the parsed information dumped into some persistent format, and modify the inspect code to actually use it.
I will also be pairing with Kyle for a few hours on working through some of http://mitpress.mit.edu/books/audio-programming-book

Why I like Org as a markup

Wed, 12 Jan 2011 18:00:00 +0530

Org-mode, as a mode, is obviously awesome. The ability to move around whole blocks of text, for re-organizing documents and the usability of tables are just two brilliant features.

But, the org markup itself is such a cool thing. I’ve been looking at rst all day, and it’s such a pain to find out where your headings are, what level your current sub-heading is at, etc. (I’ll probably do a bit better, with some getting used to. I’m an org-addict. ;)). With org-mode it’s so intuitive! It’s so easy to find out, with just one quick look! I don’t think any other markup has this coolness! Org-mode FTW!