Python on Noetic Nought

Moving from Heroku to Fly.io

Sun, 06 Nov 2022 12:19:00 +0530

A couple of months ago Heroku announced their plan to “Focus on Mission Critical” and shut down the free product plan. The ideas is to free up their developer time to focus on the important stuff instead of fighting abuse of the free plans. I’m surprised they invested in the free plans for so many years!

My Heroku Apps

I have a bunch of apps on the free (Hobby dev) plan, and only app on the Hobby Basic plan because it needed a bigger DB. All of these apps are small enough that I could easily host all of them on my shared Hetzner server, where this blog is hosted. But, I like using Heroku for services that I run for other people. It makes access control and deployment much easier, and lets me invite other people to help maintain apps easily, despite constraints like 10k rows in the Postgres DB and Dyno sleeping.

Over the years of using Heroku, I ended up accumulating a bunch of small apps on the service. Most of these apps aren’t actively being used any more and didn’t feel worth the effort of moving.

I decided to let most of the apps die, since they aren’t actively being used. Only a couple of apps are used regularly enough to be worth keeping alive – SOTG Calculator and RSVP app. I turned on maintenance mode on the RSVP app for a couple of days, as an experiment, and I had to bring back the services due to “popular” demand.

Why not pay Heroku?

I did consider just moving up to the Hobby basic plan and paying for running the app. I wasn’t keen on investing numerous hours to get these apps to run again. Akilesh convinced me that the amount to pay Heroku could be collected from the team and wouldn’t amount to much per person. I almost took this route.

But, it bugged me that the amount we’d be paying Heroku could fund a couple of turf practice sessions for the team. I could host the apps on my Hetzner server – we already pay enough for it. But, giving other people maintainer access to apps is important and this was not straightforward with Hetzner.

Also, the SOTG app couldn’t be funded this way.

Why Fly.io?

I ranted about this to my friends on Zulip and Rajesh offered to help me as a way to learn some devops stuff. I was more than happy to have someone to discuss this stuff with and get some help.

We decided to give Fly.io a try despite being skeptical about using yet another free service. Their free Hobby plan includes 3GB of persistent storage space (which neither of the apps currently use, but potentially could). They also built Turboku – a tool to move apps from Heroku. It seemed worth giving a try, and we could always fall back on Hetzner if it didn’t work out.

We had a couple of months before the Heroku shutdown date, when I ranted about this. Rajesh and I exchanged a few links and messages on how to potentially go about doing this, when we got down to it. But, he got busy with other things and now the deadline was only a month away. I decided to use the notes and do the migration myself. I was glad I had the notes to refer to. The initial “research” made the task seem less daunting.

Migrating a simple app

I started with the simpler app first – the SOTG app. I used the Turboku web page and it worked pretty seemlessly! It copied over the environment variables from Heroku and deployed the app.

But, using the Turboku web page meant that I didn’t have a fly.toml file to do any future deployments. I used the fly launch command to create a new app and edited the fly.toml file to point to the migrated app.

Setting up the custom domain was pretty simple too, using the CLI. I just had to run fly certs add sotg.indiaultimate.org after changing the DNS entries to point to sotg-calculator.fly.dev.

The whole thing felt quite simple and I was done in under an hour – reading the docs, installing the tool, trying out different things, getting it all working. Everything. Quite impressive!

Migrating a bigger app

Encouraged by this, I started with the RSVP app. I didn’t think it would be much harder than this, but I guess when there are a few more moving parts, it’s hard to tell what could pop-up.

I used the same approach for the migration – Turboku web page to migrate from Heroku and then ran the launch command to generate a fly.toml file that I edited to point to the deployed app.

The deployment seemed to go fine, but the app wouldn’t come up. I looked at the logs to find out that the app was getting OOM killed. Fly.io provides half the memory (256MB) that Heroku provided. But, I wasn’t expecting this tiny app to need much more.

I removed some default imports of large packages, but it didn’t really help that much. I did consider reducing the Gunicorn’s WEB_CONCURRENCY setting to 1, but didn’t want to take that route unless absolutely required. I looked around the forums a little, and found that using a Dockerfile instead of the buildpacks might improve the memory situation. And it did!

Next, I moved to setting up the custom domain and it worked from the Fly.io side seemlessly. But, the flask_dance library was generating http URLs in it’s OAuth redirect URLs. The app already uses FlaskSSLify, but that didn’t seem to be sufficient. After reading some Flask dance and Flask docs, I found that Werkzeug’s ProxyFix exists exactly for this!

After dealing with these two unexpected problems, I got to the problem that I was aware I had to tackle – Fly.io doesn’t have support for cron jobs. The RSVP app uses some scheduled jobs to periodically sync some metadata from Google Drive, Calendar, etc.

I took a hackish route to work around this. Using the schedule library, I wrote a “cron” script that would run the jobs on the desired schedule. I created a duplicate app to run the cron service, even though it could potentially be run as a different process on the same app. I didn’t want to risk any more OOM kills on the web app. But, it can be a little bit more work to keep the environment variables and other such things in sync between the two apps.

I was quite excited about Fly.io having a datacenter in Madras and was swayed by Fly.io’s marketing about running the app close to the users. I tried this but the performance was terrible since the Mongo DB was in North Virginia. I quickly switched back to using an East Coast region for Fly.io app too.

But, while writing this blog post, I realized that I could move the DB to India. I moved the DB to the AWS Mumbai datacenter, and the Fly.io app to the Madras datacenter.

All’s well that ends well

The app has been chugging along for a few days without any new hiccups. I’m hoping things stay this way for the weeks and months to come, if not years.

On the whole, I found the whole experience more beta-ish than deploying something to Heroku. But, I’m quite happy with how the whole thing turned out. I think things will get smoother with time – with my growing familarity with Fly.io and more people using their tools suggesting improvements.

I am wary of depending on yet another Hobby plan, but neither of these apps make any revenue and I’m not (yet) comfortable charging users for the small set of features they provide.

I’d like to hear stories of other people moving away from Heroku! Feel free to reach out!

An innocent query that caused Zulip downtime

Thu, 28 May 2020 11:34:00 +0530

Introduction

Recently, a change that I contributed caused some downtime on the Zulip cloud (zulipchat.com). Here’s a user tweeting at us, about this.

I shared this with some friends (who use Zulip) and they asked me to share more about what happened. I began writing an explanation, and it eventually turned into a blog post. The post tries to explain what caused the down-time, without assuming too much knowledge of the Zulip DB schema, but it would help if you used Zulip a little.

The goal of my change was to change how Zulip backend calculates the furthest_read_time for a user – the time stamp indicating when the last message was read.

Previously, the code used simply kept track of the message ID of the messages being read by a user in a field called the “pointer”. My change was to move away from this pointer approach to using the actual timestamp of when a user last read a message.

What is the pointer?

Put very simply, the pointer is just a message ID¹ – ideally, the ID of the “last message” a user has read. Message IDs in Zulip are incremental – a message that has arrived after another one, would have a higher ID than the older message. So, it would’ve been a great way to keep track of the last read message, if all the messages were in “one continuous feed”. But, guess what, Zulip has streams and topics which are precisely what make Zulip such a valuable communication tool!

The pointer can be set to the ID of the message that is currently being read, but only when we are in the “All messages” view. In this view, all messages are ordered chronologically, so any message that is after another message would have arrived after it, and would have an ID greater than the other message. So reading a message in this view means, any message with a lower ID must’ve already been read.

When you are reading messages in a stream/topic (narrowed) view, the pointer is not updated. We can’t be sure that all messages with ID lower than the current message’s ID have already been read. There could be a whole bunch of unread messages in other streams/topics than the current one you are reading. So, the pointer doesn’t get updated until you go to the “All messages” view and scroll over the messages, even if you have already read them in a narrowed view.

Why change this?

Using the pointer to compute the furthest_read_time doesn’t work very well, because it is only updated in the “All messages” view. In other words, furthest_read_time turned out to actually mean something like furthest_read_time_in_all_messages_view.

The furthest_read_time is used in the UI to display the bankruptcy banner – a prompt shown to a user to mark all messages as read, if they have “too many” unreads, and haven’t read new messages in the last 48 hours.

But, this is a problem, especially in Zulip instances with a lot of traffic, where people leave a lot of messages unread until they have the time to catch-up. It is common for people to communicate only in a handful of streams/topics, and leave a lot of messages unread in other streams or topics, that they are hoping to catch-up on, when they have more time at hand. Such users would avoid using the “All messages” view at any cost.

Even in not-so-high-traffic realms, some users just prefer not to use the “All messages” view, and just stick to narrowed views.

So, for these users the furthest_read_time would get updated very rarely, and could potentially remain at a date way back in the past. This meant the users would see the bankruptcy banner despite actively participating in the Realm, even if only in a few topics/streams. This is annoying UX, and what something we wanted to fix.

What did we change this to?

We decided to use the last read message time as the message to consider for the furthest_read_time. The following “innocent looking” query should do that for us, we thought…

UserMessage.objects.filter(user_profile=user_profile, flags=UserMessage.flags.read).last()

Well, it does, but not without bringing down the whole of zulipchat.com after a deploy! 🙈 But, why?!

Some background on how data is stored would be useful to see why.

UserMessage table is huge, my friend

Zulip’s data model has a Message table that stores the content of each message, when it was sent, by whom, to which topic, etc. The UserMessage table is then used to keep track of what the status of this message is, for each user that has received the message. A 32 bit flag field is used to keep track of things like, whether the user has read a message, starred a message, has been mentioned in a message, etc. So, if a messages is sent to a stream with N users, it creates 1 row in the Message table and N rows in the UserMessage table. If M such messages are sent, it would create M*N rows in the UserMessage table. So, pretty soon, the UserMessage table becomes huge! ²

To make common queries on this table efficient, the DB has indexes for them. Some of the indexes on the UserMessage table are documented here, though the documentation was created in the migration context for production deployments. There are indexes for various things, including unread messages.

And here’s our innocent looking query, as a reminder.

UserMessage.objects.filter(user_profile=user_profile, flags=UserMessage.flags.read).last()

This query searches through the read messages for a particular user, and tries to figure out the last message they read. Note that the DB only has indexes for unread messages, not read messages! So, this query isn’t using DB indexes to search for this needle in the haystack, and ends up being pretty damn slow!

Additionally, this query is run for every user when the “home page” is loaded since we want to calculate their furthest_read_time to decide if should show the bankruptcy banner.

Also, Zulip’s architecture is designed to allow using the same app server and DB to host multiple different realms. And zulipchat.com is such a deployment where a few servers are used to serve all the realms. (I don’t know too much about the deployment setup, though).

When a new change is deployed to zulipchat.com and the server is restarted, all the clients of all the users reload. Having an expensive query in that path, would just kill the DB. And this innocent looking query didn’t disappoint!

How did we fix this?

Well, this change was reverted immediately by Tim, and things were kicked back into a usable state.

Later, we decided to take a different approach. We decided to look at a different table (UserActivity) that tracks user activity, mainly for the purpose of analytics. This table keeps track of the last time a user performed an action based on the API calls made.

Among other things, changing any of the flags in the 32-bit flag field is one of the actions that is tracked. Marking a message as read was being tracked too. We use the last time this action was performed to compute the last time a user was “active”, and based on that decide if we should show the bankruptcy banner or not.

Here’s the actual commit that makes this change. But, it turns out that needed a follow-up fix too. 🙈

Thanks to Tim Abbott and Shantanu for reading drafts of this post.

The pointer was used to do a bunch of useful things, but we are currently in the process of removing it. ↩︎
Zulip’s code base uses a concept called soft-deactivation to mitigate this problem on Realms where there are a lot of users who log in very infrequently, or don’t login after an initial period of activity. So, the number of rows are not exactly M*N in all the cases, but in the worst case they are. ↩︎

Django ModelFormSet and multiple saves

Wed, 08 Jan 2020 09:45:00 +0530

A friend and I were trying to use the Django ModelFormSets in a project, and we ran into a subtle bug that took a little big of digging into the Django code to identify and work around.

The Bug

We weren’t really familiar with Django’s FormSet or ModelFormSet, when we started. We began by reading the documentation to get started.

The Django docs had an example that looked straight-forward, and showed how to use ModelFormSet.

  def manage_authors(request):
      AuthorFormSet = modelformset_factory(Author, fields=('name', 'title'))
      if request.method == 'POST':
          formset = AuthorFormSet(request.POST, request.FILES)
          if formset.is_valid():
              formset.save()
              # do something.
      else:
          formset = AuthorFormSet()
      return render(request, 'manage_authors.html', {'formset': formset})

You create a ModelFormSet class using the modelformset_factory using the required model, and the required fields.
Use the data from a POST to create an instance of the class from above.
Save the data if the formset validates, and we are done!

We tweaked the example to our needs, and it worked for us. Mostly.

A user reported that after saving the form once with some data, trying to save more data didn’t work! The form would try to save the data already saved in the last save, and fail validation that required some of the fields to be unique.

On the second save, the data which was saved during the last save was still being treated as newly filled in data!

This also caused newly filled-in data during the second save to be lost forever. The user was forced to reload the page and re-enter the data.

The Why

The way Django keeps track of newly filled-in data in the form along with multiple levels of caching causes this bug to manifest.

`ModelFormSet` `queryset` argument

AuthorFormSet (or any ModelFormSet created using the modelformset_factory) takes an additional optional argument queryset. This argument can be used to choose specific rows in the DB that need to be used to populate the initial form rows, if any. If no queryset is specified, all the objects in the DB are used.

`FormSet` metadata

First, metadata about the form is saved in a few hidden input fields (called the ManagementForm). This hidden form keep tracks of metadata like the total number of forms to show, initial form count and the minimum & maximum number of forms to display.

  <input type="hidden" name="form-TOTAL_FORMS" value="19" id="id_form-TOTAL_FORMS">
  <input type="hidden" name="form-INITIAL_FORMS" value="7" id="id_form-INITIAL_FORMS">
  <input type="hidden" name="form-MIN_NUM_FORMS" value="0" id="id_form-MIN_NUM_FORMS">
  <input type="hidden" name="form-MAX_NUM_FORMS" value="1000" id="id_form-MAX_NUM_FORMS">

Initial form count is the count of the number of forms that are pre-filled using data from the DB – essentially, the length of the queryset

Metadata does not change

When the ModelFormSet is used as described in the documentation, the metadata in the ManagementForm doesn’t get updated when the form is re-rendered in the response to a successful POST request. If one form with new data was successfully saved, the number of initial forms in the re-rendered form must go up by 1, but it doesn’t.

Queryset caching problem

Django has caching at the queryset level which means that the queryset doesn’t get updated with the newly created instances, even if they match the query that the queryset uses.

  formset = AuthorFormSet(request.POST, request.FILES, queryset=existing_authors)

existing_authors is a queryset that looks for a specific set of authors in the DB, or may be all the authors in the DB, if that’s what the form needs to do.

So, when a formset has been submitted, and some new rows have been created in the model’s table, the new rows don’t automatically get added to the queryset since it the results fetched from the DB are cached by the queryset. We need to create a new queryset for it to include the newly created rows.

These two levels of caching combined make the bug manifest, as it does. I’m not yet sure what the correct “fix” for this is, but we ended up using a work-around – create a new formset when returning from a successful POST.

The Work-Around

We instantiate a new ModelFormSet instance when returning the form, on a successful POST. This means, the ManagementForm is recreated for the new formset and has the correct metadata about the form, even when being rendered in the response of a successful POST.

The code in the example in Django docs, would look something like the one below:

  def manage_authors(request):
      AuthorFormSet = modelformset_factory(Author, fields=('name', 'title'))
      if request.method == 'POST':
          formset = AuthorFormSet(request.POST, request.FILES)
          if not formset.is_valid():
              return render(request, 'manage_authors.html', {'formset': formset})
          else:
              formset.save()

      formset = AuthorFormSet()
      return render(request, 'manage_authors.html', {'formset': formset})

This blog post should probably become a PR or an issue, but I’m not familiar with how contributing to Django works. This blog post is an attempt to understand the issue better, and potentially help others who get bit by it.

Thanks to Shantanu for reading drafts of this post.

Isolating flaky Django tests

Wed, 05 Jun 2019 18:37:00 +0530

The Zulip repository had a bunch of flaky tests that were fixed a few weeks ago. I learnt a couple of tricks that I wasn’t aware of during this.

Django’s test runner has an option to run tests in the reverse order. The documentation of this option explicitly mentions that this option is useful for debugging side-effects of tests that are not properly isolated. I wasn’t aware of this option, and it’s a pretty neat tool.

The test runner also has a --parallel option to run tests in parallel, and the Zulip server uses this option when running the back-end tests. I never really bothered to look into how test cases are assigned to different processes. The Django docs explain:

Django distributes test cases — unittest.TestCase subclasses — to subprocesses. If there are fewer test cases than configured processes, Django will reduce the number of processes accordingly.

Each process gets its own database. You must ensure that different test cases don’t access the same resources. For instance, test cases that touch the filesystem should create a temporary directory for their own use.

When running tests in parallel, the set of tests running in the same process can vary quite a bit based on the execution speed of tests in different processes. To add more variation to the order in that tests are run, the number of processes being used could also be changed.

These two tools combined together can get tests to run in different orders than the usual order in which they usually get run, and can help find tests which are not well isolated.

It could be a good idea to run tests in parallel (with a randomly chosen, reasonable number of processes) and with the --reverse flag, once in every few CI runs, or as a cron job periodically.

Thanks to Shantanu Choudhary for nudging me to write this post, and reading drafts of this post. Thanks also to Tim Abbott for making me aware of these tricks.

Python's globals

Sun, 29 Jul 2018 09:09:00 +0530

TL; DR

The global statement is a parser directive to indicate that globals are being used. When looking up globals, the globals of the current scope are fetched.
Python’s globals are really just module-level.
Every function has an associated __globals__ dictionary, which is the same as the module’s __dict__ for the module where it was defined. This __globals__ dict is the name-space that is looked up when trying to fetch globals within a function.
Avoid globals to make it easier to test functions

Motivation

During a mentoring session I was doing recently, we ran into a problem with globals in Python.

Here is a toy example to describe what we were seeing.

We had code in a module foo.py where the function f was using a global defined in main, and we were trying to write tests for the function f.

# foo.py
def f():
    print(a)


def main():
    global a
    a = 5
    f()

if __name__ == '__main__':
    main()

Running this file foo.py prints out 5 as expected.

We would now like to import f into a different module and use it.

Running `f` in a different module

# bar.py
from foo import f

def main():
    f()

main()

Running the module bar.py gives us a NameError, as expected. The name a was defined in the main in foo.py which is never being run when the code is imported in bar.py

Traceback (most recent call last):
  File "bar.py", line 10, in <module>
    main()
  File "bar.py", line 7, in main
    f()
  File "foo.py", line 5, in f
    print(a)
NameError: global name 'a' is not defined

Setting a global value for `a`

We’d like to be able to run f without running main and the first fix that comes to mind is to set the value of a in bar, and let f use that.

# bar.py
def main():
    global a
    a = 4
    f()

Surprise! Nothing changes.

Traceback (most recent call last):
  File "/tmp/example/bar.py", line 13, in <module>
    main()
  File "/tmp/example/bar.py", line 9, in main
    f()
  File "/tmp/example/foo.py", line 5, in f
    print(a)
NameError: global name 'a' is not defined

Why doesn’t this work?!

Function `globals` and the `global` statement

The global statement is a directive to the parser, that specifies that the variable being assigned to is a global variable.

This can be seen by looking at the disassembled code for f

import dis
from foo import f

dis.dis(f)

5           0 LOAD_GLOBAL              0 (print)
            2 LOAD_GLOBAL              1 (a)
            4 CALL_FUNCTION            1
            6 POP_TOP
            8 LOAD_CONST               0 (None)
           10 RETURN_VALUE

globals for the current frame are fetched, and the value is updated in that dict. Each function in Python has an associated __globals__ dict which is a reference to that module’s __dict__ in which the function was defined. So, in the case where we try to set the a = 4 in bar.main, the main function’s __globals__ dict is being updated.

# bar.py
def main():
    global a
    a = 4
    print(main.__globals__.keys())
    print(main.__globals__['a'])

dict_keys(['__name__', '__doc__', '__package__', '__loader__', '__spec__', '__annotations__', '__builtins__', '__file__', '__cached__', 'foo', 'f', 'dis', 'main', 'a'])
4

As you can see a is being set to 4, but f still doesn’t see value, since it has it’s own __globals__ dictionary. Printing the globals dictionary for f should make that clear.

from foo import f
print(f.__globals__)

If the variable a was declared in the module foo outside of any of the functions, it would be in f’s __globals__ dict when it is imported, and hence the name error would go away, but setting it still would not work.

# foo.py
a = 3

def f():
    print(a)


def main():
    global a
    a = 5
    f()

if __name__ == '__main__':
    main()

# bar.py
from foo import f

def main():
    global a
    a = 4
    f()

main()

Running bar.py would print the value 3 which has been defined in foo.py, and not 4.

Updating `globals`

To update the value of a for f, we could modify it’s globals dict.

# bar.py
from foo import f

def main():
    f.__globals__['a'] = 4
    f()

main()

Module `dict` and monkey-patching

As mentioned previously, a function’s __globals__ dict is a reference to the module’s __dict__ for the module where the function was defined. So, we could achieve the same result as above by updating foo.__dict__. And setting an attribute on the module foo is the same as updating this dict.

# bar.py
import foo
from foo import f


def main():
    foo.a = 4
    f()

If you have used a library like mock to patch some code while running tests, this is essentially what is happening. The target module’s dict is looked up for the specified object/function and replaced with a mock object.

Use an argument to make it testable

The function f would’ve been much easier to test, if it took a as an argument, instead of using a global value. This functional approach would make the code easier to reason about too.

# foo.py
def f(a):
    print(a)


def main():
    a = 5
    f(a)

if __name__ == '__main__':
    main()

# bar.py
from foo import f

def main():
    a = 3
    f(a)

Thanks to Akshaya and Shantanu for reviewing this blog post.

Create a Public Jupyter Server, quickly!

Thu, 25 Aug 2016 00:00:00 +0530

I create public Jupyter notebooks once in a while, to collaborate with a friend, or to make it easier for myself to work with data on a remote machine.

Each time I need to look up the docs, and manually set-up a few things, before I can start using the notebook.

I just wrote a bash script that does the following, quickly -

Install Jupyter into a temporary virtualenv
Create certificate files
Start the server with https enabled and a password
Delete all temporary files, when the server is shutdown

You can get it here, if you’d like to use it.

Python reload and module dict

Thu, 23 Jun 2016 00:00:00 +0530

I was trying to play around with Nikola’s code today and learnt about a documented weirdness of Python’s reload.

Below are two versions of code – ORIGINAL and UPDATED respectively referring to the orginial code and the code after changes. The code only has a PLUGINS list, which is changed in each version of the code.

# Work in a temporary directory
import os
import tempfile
os.chdir(tempfile.mkdtemp())

# Module content, original and updated
ORIGINAL = "# PLUGINS = []"
UPDATED = "PLUGINS = ['rss']"

def create_conf_file(content):
    """Create a conf.py module with given content."""
    with open('conf.py', 'w') as f:
        f.write(content)

conf doesn’t have a PLUGINS attribute in the ORIGINAL code. It’s been commented out!

create_conf_file(ORIGINAL)
import conf
# PLUGINS is not defined in the module, originally.
print(conf.PLUGINS)

The code for conf has been updated, but the module doesn’t yet have a PLUGINS attribute, since the new module isn’t imported until we reload.
```
create_conf_file(UPDATED)
import conf
print(conf.PLUGINS)
```

PLUGINS has the expected value, after the reload

import importlib
importlib.reload(conf)
print(conf.PLUGINS)

What happens when we revert to the ORIGINAL code, and reload the module?

# We write back the original file. PLUGINS should be empty!
create_conf_file(ORIGINAL)
import conf
importlib.reload(conf)
print(conf.PLUGINS, "<---Whaaaat!")

The behavior is well documented, along with a reasoning of why it is the way it is, but you can trip over it if you don’t know. I hit a bug and was wondering if there was a race condition somewhere, until I read the docs for reload. On reload, the module dict is updated, instead of creating a new dict. Any values not redefined in the new code for the module remain unchanged.

Playing music using mpsyt from Emacs

Mon, 20 Apr 2015 00:00:00 +0530

I’ve started using the wonderful mpsyt to play any music from youtube, since I’m not really interested in the video. But, since I use emacs for chat/IRC, I end up getting youtube links into emacs and opening them opens them up in my browser. I ended up writing some elisp to play the songs from within an instance of mpsyt running inside an emacs buffer.

(defun pc/short-url-at-point ()
  "Gets the short url at point.

This function is required only because
`thing-at-point-url-at-point' ignores urls (without a scheme)
that don't start with www."
  (let ((bounds (thing-at-point-bounds-of-url-at-point t)))
    (when (and bounds (< (car bounds) (cdr bounds)))
      (buffer-substring-no-properties (car bounds) (cdr bounds)))))

(defun pc/mpsyt-url (url)
  (let ((buffer (current-buffer))
        (mpsyt-proc-name "*mpsyt*"))

    ;; Start a new term with *mpsyt* if there isn't one
    (unless (get-process mpsyt-proc-name)
      (when (get-buffer mpsyt-proc-name)
        (kill-buffer (get-buffer mpsyt-proc-name)))
      (ansi-term "mpsyt" "mpsyt"))

    ;; Play given url in mpsyt
    (let ((mpsyt-proc (get-process mpsyt-proc-name)))
      ;; If something is already playing, stop it and play this...
      (term-send-string mpsyt-proc "\n\n\n")
      ;; We wait for a bit, since looking for the prompt seems to fail, sometimes?
      (sleep-for 1)
      (term-send-string mpsyt-proc "\n")

      ;; Actually send the command to playurl
      (term-simple-send (get-process mpsyt-proc-name)
                        (format "playurl %s" url)))

    (switch-to-buffer buffer)))

(defun pc/mpsyt-url-at-point ()
  "Play the URL at point using mpsyt."
  (interactive)
  (let ((url (or (url-get-url-at-point) (pc/short-url-at-point))))
    (if (not url)
      (message "No URL found")
        (message (format "Playing %s with mpsyt" url))
      (pc/mpsyt-url url))))

The current version of mpsyt crashes when run from inside emacs due to a bug in the code to get the terminal size, which should be fixed once this patch is merged.

I would’ve expected thing-at-point-url-at-point to be able to find urls even when they don’t have a schema, but it tries to guess the schema from urls and fails to work when the url starts with youtube.com instead of www.youtube.com.

I started off using the command-line interface of mpsyt by running it using shell-command or start-process. But, it seemed useful to have a buffer of mpsyt to switch to – easier to search for new music, repeating songs, etc. Not all tasks/actions are achievable through mpsyt’s command line args.

I ended up writing more code than I thought I would have to[1]. But, I’m pretty happy with how this all works, right now.

[1] - Isn’t it true, more often than not?

Recurse Center, 2014-08-27

Thu, 28 Aug 2014 09:51:30 -0400

I spent the whole of yesterday just working on the LED-bot.
Spent some time cleaning up meta-stuff about the project, making it a real package, adding a setup.py, added LICENSE & AUTHOR files, etc.
I found it really frustrating that there’s no good/standard way to remove duplication of information between pip’s requirements.txt and install_requires
We got the bot running on the Beagle Bone, but it turned out to be too slow. After some playing around with trying to get rid of Python for-loops, I was able to use PIL’s image.tobytes and some image cropping to get the data to be sent to the LEDs, and it sped up the display quite a bit.

Recurse Center, 2014-08-26

Wed, 27 Aug 2014 10:23:27 -0400

I spent most of the day refactoring the code for the LED Bot, and am happy with the way it looks right now.
My talk at Hack and Tell went OK. I was distracted by live_reveal not displaying properly on the smaller resolution, and not being able to mirror screens!
There were a bunch of interesting talks presented by the others. http://comparea.org, Taxis and Rainbows were the most interesting ones for me.

Recurse Center, 2014-08-20

Fri, 22 Aug 2014 22:43:04 -0400

I finished going through the third of Dave Beazley’s tutorials on Generators and Co-routines.
Also refactored api-diff to use astroid, instead of the ast module. Kinda happy with what I have, for now. May be more, later…

Recurse Center, 2014-08-21

Fri, 22 Aug 2014 10:33:34 -0400

I added an additional view to xtab, to see all the pages that were killed, and reopen them, if you so choose.
Also, cleaned up the api-diff project and renamed it to what-changed.
Started thinking about building the GitHub organization dashboard, that I thought would be interesting to have, since the 1st or 2nd week of Hacker School.
A bunch of awesome thursday presentations, and job fair and awesome food by Chef Warren!

Recurse Center, 2014-08-19

Wed, 20 Aug 2014 18:06:53 -0400

I spent most of the day working through David Beazly’s tutorials on generators and coroutines, and it was mind bending.
I learnt about send and throw methods on generators! I hadn’t heard of them before, and was totally surprised to come across them.
I spent a little time late in the day getting class diffs working, and refactoring the api-diff tool to remove duplication of code between classes and modules. But, not very happy with the way I did it.

Recurse Center, 2014-08-18

Tue, 19 Aug 2014 13:11:49 -0400

I refactored the API-diff code, and am pretty happy with what I have right now, and need to add support for diffing Classes. I’m considering using Atstroid instead of Python’s ast module, since some of the work that I want to do, has already been done for me.
With the white-boarding group, we worked on a set of simple exercises for strings and arrays.
José Valim’s presentation on Elixir was interesting and got me reading up a stuff about concurrency, and related stuff. Thanks!

Recurse Center, 2014-08-17

Mon, 18 Aug 2014 21:59:07 -0400

Friday

Updated my HS profile page with projects that I worked on, and a short bio.
Joined the group discussing algorithmic complexity
Submitted a pull request to blaggregator to fix issues with broken profile images.

Saturday & Sunday

I almost didn’t write any code during the weekend.

Recurse Center, 2014-08-14

Fri, 15 Aug 2014 18:09:59 -0400

I spent the morning cleaning up the “repr” functions of the diff classes, so that the diff of functions between two modules can be printed out in a somewhat meaningful way.
I spent most of the afternoon, preparing to make a small presentation on how sys.path works in Python. I thought I didn’t do as good a job of it, as I would’ve liked, but Tom thought it was not too bad…
Looks like another of those days, where I didn’t get much done… I spent the evening finishing off Homeland. It’s a brilliant book, and everyone who cares about the Government and policies and technology MUST read it! Thanks Cory.

Recurse Center, 2014-08-13

Thu, 14 Aug 2014 11:11:59 -0400

I feel like I didn’t get much done yesterday.
I mostly worked on the API differ, but didn’t get much done. I have a few tests, and some code for diffing two functions, but I’m not very happy with it.
I helped Giorgio and Carlos with using my client only hs_oauth script/library, for their Zulip bot.
The white boarding group worked on some binary related problems, and it was fun.
I looked at how the HS OAuth backend works on blaggregator and thought about how to go about fixing the issue of broken images on the site. The problem happens because the image URLs are returned from the HS API calls, that are made whenever a user logs in(?), and the URLs are cached. Since, HS uses cloudfront for its assets, the URLs expire after a period of time, and the cached urls in blag’s database need to be updated. There could be two ways of doing it -
1. Check if the URL 404s, every time a URL is requested, on the server side.
2. Do it on the client side, with some javascript magic.
Intuitively, I feel like 2. would be better, but Madhu suggested that 1. wouldn’t be that bad either. We could try both out, and see which works better I suppose.

I’d be interested to try this out, sometime.

Recurse Center, 2014-08-12

Wed, 13 Aug 2014 10:30:30 -0400

An issue I was seeing with pip and a package installed in system-site-packages turned out to be non-reproducible. It was probably PEBKAC.
I was wondering around in the morning trying to find something to work on.
I decided to work on Pankaj’s idea to write a tool that gives you API differences between two versions of a library. It would be useful to have the list of public API changes in a library you use, to estimate the cost of updating to the newer version, without having to manually go through all the changes.
I spent some time talking to Sophia, and looking at her FTP server’s refactored code.

Recurse Center, 2014-08-11

Mon, 11 Aug 2014 11:16:08 -0400

I spent about an hour in the morning testing out the staging blaggregator instance, and things seemed to work as expected. Side note, I got added as a collaborator on the blaggregator repo!
We worked through the first chapter on getting video working, but we mostly ended up just copying code rather than actually understanding/carefully studying it. I’m not sure how I feel about it, but we got to display fancy stuff.
I wasn’t feeling like working on any of my ‘old’/‘ongoing’ projects, and since Mondays don’t feel very productive, I worked on writing a couple of simple scripts to create twitter lists of HSers, batch wise.

For the HS API, I just reused some old code I had lying around from my experiments with the OAuth2 API, though I had to tweak it a little bit to be able to actually login, using requests. Something about CSRF tokens seems to have changed.

In the process, I found that twitter’s API isn’t very pleasant to use. Or may be it’s the fact that I was using python twitter client without getting my own client id/key, but the whole experience of dealing with the twitter API wasn’t a pleasant one at all!
Amber, Daria and I also spent about an hour white-boarding some problems from the Cracking the Code Interview book.
I ended up fixing some minor issues with blaggregator and am hoping that the long pending PR will be merged today.

Recurse Center, 2014-08-08

Sat, 09 Aug 2014 20:56:22 -0400

I spent the morning doing some white-boarding interview practice with a Jorge and Brian.
I spent the rest of the day, playing with PyPy and working through a tutorial that teaches us to write a BF interpreter.
I spent the Saturday revamping my Emacs config. The main issue with my config was having a “package” list, of all the packages that the configuration uses, in case we are trying to duplicate it elsewhere. I noticed that this gets out of sync, because I install packages by hand and never update this list. I wrote up some code, that keeps this list in sync with all the packages I have installed. I configured el-get to play well with my config, and am pretty happy with my setup, though I still have to add some matching/searching features with helm/ido/whatever else.
I spent the Sunday writing a patch to xtab to be able to limit the number of tabs in Chromium by the memory it uses.

Recurse Center, 2014-08-03

Mon, 04 Aug 2014 16:10:41 -0400

Friday

On Friday, I didn’t do the exercises that everyone else doing job prep were doing. I wanted to continue writing C code, and continued to work on the bencode parser in C. I barely was able to finish adding support for lists.
I got comfortable using Valgrind, in the process of debugging a stupidity of not using realloc correctly.
What I was able to write with Python in an hour or so, I have been writing for about a day, in C, and haven’t yet finished it! It definitely doesn’t help that my C is very rusty…

Saturday & Sunday

I didn’t work much, and went around New York with Madhu. We went to quite a few places around here, the highlight being, the view of Manhattan’s skyline from the Staten Island Ferry.
NYC’s library also had an interesting exhibhition on the history of America, just before and during WW-I.
On Sunday, I tried out Mary TTS on the raspberry pi, and it is simply unusable.
I then, started writing a simple wrapper around Google’s tts that is used by Google translate.
The TTS part is more or less working right now, the next step would be to deploy it on the RaspberryPi and start using it as a radio, to see what my needs are like. Hooking it up to Jasper client seems like a good idea, right now… though I’m not sure if it would work very well, with the pi’s audio output drowning out the voice commands.

Recurse Center, 2014-07-28

Mon, 28 Jul 2014 22:23:39 -0400

There seemed to be excitement in the space to have Jessica around. We had coffee with her, and I chatted with her briefly. She encouraged me to try and send a patch to CPython, that involved debugging and fixing some C issues.
I spent some time with Tom, while he tried to get cinspect working for him, and it seemed like having the developer next to you, helped. As always. We didn’t make any fixes to the code, but it worked.
I spent the afternoon, adding tests for a simple issue, to see how patches are sent.
Later, I paired with Sophia for an hour to refactor the vcs code. It’s beginning to look much better than what it used to, and I like that.
Jessica’s talk on buffer overflows was enlightening and entertaining. It was very similar to a talk that Max gave on one of the thursdays, but it was exciting nevertheless.
After the talk, I am inspired to try and work through the Hacking: Art of exploitation, at some point. I’m beginning to feel like time at HackerSchool running out!

Recurse Center, 2014-07-25

Fri, 25 Jul 2014 09:25:50 -0400

We worked on writing our own (toy) version control system on Friday, as a part of job prep. I paired with Sophia, and we got upto the point where we had diffs working, when we decided to refactor the code with some tests and stuff.
The party was brilliant with a lot of activity. All the project installations looked so cool, and it was so much fun!
I spent some time during the weekend watching the “Anatomy of Matplotlib” lectures.
I started some work with poking around Matplotlib’s template backend, and trying to start writing and ascii backend.
Also, I was trying to help UC setup GrooveBasin on his raspberry pi, and helped him configure jasper on it.

Recurse Center, 2014-07-24

Thu, 24 Jul 2014 09:53:03 -0400

Glowing LEDs and me

It turned out that the SPI driver exposed a faster way to write bits, writing to all the LEDs at once, instead of one at a time, and this was sufficient to get things working!
Kyle and I started to have enough, of the glowing LEDs and decided to wrap up the project, but, Nick had different ideas for us! We had talked about having the music stream via airplay, but Nick got his DJing tools, and it would be too bad to not use it for the party, but doing that meant Airplay’s delay wouldn’t help the DJ. So, we connected a line-in, and thought we’d be done if we just read the input and used that data. But, it turns out that’s not so simple because sending data over the wire, added noise to it, and in a whole range of frequencies! I tried removing all the fourier coefficients which were lower than a threshold, but I felt like it didn’t work very well, until Nick reduced the gain of this output on the amp to the very minimum. Finally, Kyle and I decided that this was good enough for the party!
I also hacked up a quick script to read a font file, and use the grey values to get ascii characters to show up on the LED strip.

Miscellaneous

While trying to find the RPi on the network, I learnt that sudo nmap -sP 10.0.1.* gives us the hostname as well, instead of just IPs. sudo being the key, here.
Kyle and I went and grabbed dinner in a restaurant in west village. It was good to sit down and talk about various things, and eat. The glowing LEDs were totally out of my mind for that period, and the break was good!
There were a bunch of HSers doing mock-interviews and it was fun to watch, so I stayed on until quite late, when all of us headed out together.

Today

I’m excited about trying to build a small version control system from scratch today!
Though I have had quite a bit of glowing LEDs in my face for the past week, I may suck up to it, and figure out how to do scrolling text on the display.
Also, excited to see how all the cool projects will come together in the party tonight!

Recurse Center, 2014-07-23

Wed, 23 Jul 2014 10:51:51 -0400

Shairplay and WS2801

Kyle mounted the LEDs onto a cardboard, and packaged it to look cool.
We thought we were done with the project, until we found out we weren’t! The LEDs all lighted up correctly, when we played the audio from a file on disk, but streaming messed something up! Only the first few columns lighted up, when we were listening on the stream.
We spent the whole evening trying to debug the issue, and eliminated quite a few hypothesis, but weren’t able to fix it, yet.

Miscellaneous

rntz live coded a parser for us yesterday, and I learnt a few things about parsers. I’m glad I went to the discussion/talk.
I started working through the videos of Anatomy of Matplotlib, and hope to get new insights into how it works.

Today

Today will be spent mostly trying to get streaming audio working.
I feel like debugging this, will take my understanding of the hardware-software interface to a new level.

Python on Noetic Nought

Moving from Heroku to Fly.io

My Heroku Apps

Why not pay Heroku?

Why Fly.io?

Migrating a simple app

Migrating a bigger app

All’s well that ends well

An innocent query that caused Zulip downtime

Introduction

What is the pointer?

Why change this?

What did we change this to?

UserMessage table is huge, my friend

How did we fix this?

Django ModelFormSet and multiple saves

The Bug

The Why

ModelFormSet queryset argument

FormSet metadata

Metadata does not change

Queryset caching problem

The Work-Around

Isolating flaky Django tests

Python's globals

TL; DR

Motivation

Running f in a different module

Setting a global value for a

Function __globals__ and the global statement

Updating __globals__

Module __dict__ and monkey-patching

Use an argument to make it testable

Create a Public Jupyter Server, quickly!

Python reload and module dict

Playing music using mpsyt from Emacs

Recurse Center, 2014-08-27

Recurse Center, 2014-08-26

Recurse Center, 2014-08-20

Recurse Center, 2014-08-21

Recurse Center, 2014-08-19

Recurse Center, 2014-08-18

Recurse Center, 2014-08-17

Friday

Saturday & Sunday

Recurse Center, 2014-08-14

Recurse Center, 2014-08-13

Recurse Center, 2014-08-12

Recurse Center, 2014-08-11

Recurse Center, 2014-08-08

Recurse Center, 2014-08-03

Friday

Saturday & Sunday

Recurse Center, 2014-07-28

Recurse Center, 2014-07-25

Recurse Center, 2014-07-24

Glowing LEDs and me

Miscellaneous

Today

Recurse Center, 2014-07-23

Shairplay and WS2801

Miscellaneous

Today

`ModelFormSet` `queryset` argument

`FormSet` metadata

Running `f` in a different module

Setting a global value for `a`

Function `globals` and the `global` statement

Updating `globals`

Module `dict` and monkey-patching