Django on Noetic Nought

Django ModelFormSet and multiple saves

Wed, 08 Jan 2020 09:45:00 +0530

A friend and I were trying to use the Django ModelFormSets in a project, and we ran into a subtle bug that took a little big of digging into the Django code to identify and work around.

The Bug

We weren’t really familiar with Django’s FormSet or ModelFormSet, when we started. We began by reading the documentation to get started.

The Django docs had an example that looked straight-forward, and showed how to use ModelFormSet.

  def manage_authors(request):
      AuthorFormSet = modelformset_factory(Author, fields=('name', 'title'))
      if request.method == 'POST':
          formset = AuthorFormSet(request.POST, request.FILES)
          if formset.is_valid():
              formset.save()
              # do something.
      else:
          formset = AuthorFormSet()
      return render(request, 'manage_authors.html', {'formset': formset})

You create a ModelFormSet class using the modelformset_factory using the required model, and the required fields.
Use the data from a POST to create an instance of the class from above.
Save the data if the formset validates, and we are done!

We tweaked the example to our needs, and it worked for us. Mostly.

A user reported that after saving the form once with some data, trying to save more data didn’t work! The form would try to save the data already saved in the last save, and fail validation that required some of the fields to be unique.

On the second save, the data which was saved during the last save was still being treated as newly filled in data!

This also caused newly filled-in data during the second save to be lost forever. The user was forced to reload the page and re-enter the data.

The Why

The way Django keeps track of newly filled-in data in the form along with multiple levels of caching causes this bug to manifest.

`ModelFormSet` `queryset` argument

AuthorFormSet (or any ModelFormSet created using the modelformset_factory) takes an additional optional argument queryset. This argument can be used to choose specific rows in the DB that need to be used to populate the initial form rows, if any. If no queryset is specified, all the objects in the DB are used.

`FormSet` metadata

First, metadata about the form is saved in a few hidden input fields (called the ManagementForm). This hidden form keep tracks of metadata like the total number of forms to show, initial form count and the minimum & maximum number of forms to display.

  <input type="hidden" name="form-TOTAL_FORMS" value="19" id="id_form-TOTAL_FORMS">
  <input type="hidden" name="form-INITIAL_FORMS" value="7" id="id_form-INITIAL_FORMS">
  <input type="hidden" name="form-MIN_NUM_FORMS" value="0" id="id_form-MIN_NUM_FORMS">
  <input type="hidden" name="form-MAX_NUM_FORMS" value="1000" id="id_form-MAX_NUM_FORMS">

Initial form count is the count of the number of forms that are pre-filled using data from the DB – essentially, the length of the queryset

Metadata does not change

When the ModelFormSet is used as described in the documentation, the metadata in the ManagementForm doesn’t get updated when the form is re-rendered in the response to a successful POST request. If one form with new data was successfully saved, the number of initial forms in the re-rendered form must go up by 1, but it doesn’t.

Queryset caching problem

Django has caching at the queryset level which means that the queryset doesn’t get updated with the newly created instances, even if they match the query that the queryset uses.

  formset = AuthorFormSet(request.POST, request.FILES, queryset=existing_authors)

existing_authors is a queryset that looks for a specific set of authors in the DB, or may be all the authors in the DB, if that’s what the form needs to do.

So, when a formset has been submitted, and some new rows have been created in the model’s table, the new rows don’t automatically get added to the queryset since it the results fetched from the DB are cached by the queryset. We need to create a new queryset for it to include the newly created rows.

These two levels of caching combined make the bug manifest, as it does. I’m not yet sure what the correct “fix” for this is, but we ended up using a work-around – create a new formset when returning from a successful POST.

The Work-Around

We instantiate a new ModelFormSet instance when returning the form, on a successful POST. This means, the ManagementForm is recreated for the new formset and has the correct metadata about the form, even when being rendered in the response of a successful POST.

The code in the example in Django docs, would look something like the one below:

  def manage_authors(request):
      AuthorFormSet = modelformset_factory(Author, fields=('name', 'title'))
      if request.method == 'POST':
          formset = AuthorFormSet(request.POST, request.FILES)
          if not formset.is_valid():
              return render(request, 'manage_authors.html', {'formset': formset})
          else:
              formset.save()

      formset = AuthorFormSet()
      return render(request, 'manage_authors.html', {'formset': formset})

This blog post should probably become a PR or an issue, but I’m not familiar with how contributing to Django works. This blog post is an attempt to understand the issue better, and potentially help others who get bit by it.

Thanks to Shantanu for reading drafts of this post.

Isolating flaky Django tests

Wed, 05 Jun 2019 18:37:00 +0530

The Zulip repository had a bunch of flaky tests that were fixed a few weeks ago. I learnt a couple of tricks that I wasn’t aware of during this.

Django’s test runner has an option to run tests in the reverse order. The documentation of this option explicitly mentions that this option is useful for debugging side-effects of tests that are not properly isolated. I wasn’t aware of this option, and it’s a pretty neat tool.

The test runner also has a --parallel option to run tests in parallel, and the Zulip server uses this option when running the back-end tests. I never really bothered to look into how test cases are assigned to different processes. The Django docs explain:

Django distributes test cases — unittest.TestCase subclasses — to subprocesses. If there are fewer test cases than configured processes, Django will reduce the number of processes accordingly.

Each process gets its own database. You must ensure that different test cases don’t access the same resources. For instance, test cases that touch the filesystem should create a temporary directory for their own use.

When running tests in parallel, the set of tests running in the same process can vary quite a bit based on the execution speed of tests in different processes. To add more variation to the order in that tests are run, the number of processes being used could also be changed.

These two tools combined together can get tests to run in different orders than the usual order in which they usually get run, and can help find tests which are not well isolated.

It could be a good idea to run tests in parallel (with a randomly chosen, reasonable number of processes) and with the --reverse flag, once in every few CI runs, or as a cron job periodically.

Thanks to Shantanu Choudhary for nudging me to write this post, and reading drafts of this post. Thanks also to Tim Abbott for making me aware of these tricks.

Partial postgres db dumps for a Django app

Sun, 10 Jan 2016 00:00:00 +0530

Off late, I have been working with a large postgres database that is used by an app built in Django. I wanted a partial dump of the database to try out some experimental clean up scripts. I haven’t really used databases before, and the last time I had to do this I did it in a pretty ad-hoc fashion. This time around, I tried to do it more methodically and to document it.

The Django Route

I looked around for tools that let you do this, and found django-fixture-magic. I first tried it out on my older partial dump (10% as large as the original db) and it turned out to be reasonably fast and worked well, after making a few changes to get it working with Python 3.x. Its kitchensink flag to the dump_object seemed like a promising option, but didn’t really seem to get all the required tables for ManyToManyFields. I worked around it, by getting a dump of all the models which were related using Django’s dumpdata.

Get a dump with objects of interest

The dump_object command lets you run commands to select the objects that you want to have in the dump, and that is quite a useful thing.

python manage.py dump_object dataset.Product -k --query '{"subcategory_id__in": [1886, ...]}' > products.json

Also, get a dump of related tables.

# Dump of related fields
python manage.py dumpdata dataset.Attribute > attributes.json

Create the new empty db

Next, create a new database where this fixture can be loaded!

# Create the db
sudo su - postgres
createdb mydb

# Create a user, if required
createuser -P

Grant access to the user

In the psql prompt type the following to grant the user permissions for the database.

GRANT ALL PRIVILEGES ON DATABASE mydb TO myuser;

Fix settings.py and create tables for the models.

Make changes to settings.py to use the newly created database, and then create the tables used by the app, and then load the data.

python manage.py syncdb
python manage.py loaddata products.json

Too slow!

This method worked and was reasonably fast when I was trying to get 20k rows from a table with about 200k rows, with all the dependencies.

But, when I tried to get a dump of about 200k rows from a table with 2M rows, it was way too slow to be of any use. There could’ve been a couple of reasons for this, which I didn’t have the time to look into, and debug.

The web-server where the Django app was running, and the db server with the postgres database were on different machines in separate datacenters, which could’ve been adding a significant amount of latency.
Just the size of the database being much larger could be making it slower?

These are things I should be looking into and learning about, when I have more time at hand. For now, I needed a quicker way to get a dump. Even though the raw SQL route was more manual, it turned out to be much quicker.

Raw SQL dump route

Get a dump of the interesting tables

First, I had to get a dump of all the tables with the data I was interested in, one-by-one.

COPY (SELECT * FROM "dataset_product" WHERE ("dataset_product"."subcategory_id" IN (319557, 94589, 332, 406, 626, 1886) AND "dataset_product"."gender_id" = 1)) TO '/tmp/products.tsv'

COPY (SELECT * FROM "dataset_photo" WHERE "dataset_photo"."product_id" IN (SELECT U0."id" FROM "dataset_product" U0 WHERE (U0."subcategory_id" IN (319557, 94589, 332, 406, 626, 1886) AND U0."gender_id" = 1))) TO '/tmp/photos.tsv'

-- Copy a bunch of other tables!

Load the data from the dumps

-- syncdb
COPY dataset_product FROM '/tmp/products.tsv' ENCODING 'UTF8';
COPY dataset_photo FROM '/tmp/photos.tsv' ENCODING 'UTF8';
-- Copy a bunch of other tables!

Make tables writable

Some of the tables did not let me write anything to them, until I altered the sequence for these tables.

Automating

It would be pretty nice if all of this was automated – allow a user to enter exactly the same kind of a query that django-fixture-magic lets you run, and figure out the SQL copies that need to be done to get the requested dump. Its something that currently would qualify as yak-shaving, but may be a handy thing to have. Someone somewhere possibly already has something that does this.

Recurse Center, 2014-08-11

Mon, 11 Aug 2014 11:16:08 -0400

I spent about an hour in the morning testing out the staging blaggregator instance, and things seemed to work as expected. Side note, I got added as a collaborator on the blaggregator repo!
We worked through the first chapter on getting video working, but we mostly ended up just copying code rather than actually understanding/carefully studying it. I’m not sure how I feel about it, but we got to display fancy stuff.
I wasn’t feeling like working on any of my ‘old’/‘ongoing’ projects, and since Mondays don’t feel very productive, I worked on writing a couple of simple scripts to create twitter lists of HSers, batch wise.

For the HS API, I just reused some old code I had lying around from my experiments with the OAuth2 API, though I had to tweak it a little bit to be able to actually login, using requests. Something about CSRF tokens seems to have changed.

In the process, I found that twitter’s API isn’t very pleasant to use. Or may be it’s the fact that I was using python twitter client without getting my own client id/key, but the whole experience of dealing with the twitter API wasn’t a pleasant one at all!
Amber, Daria and I also spent about an hour white-boarding some problems from the Cracking the Code Interview book.
I ended up fixing some minor issues with blaggregator and am hoping that the long pending PR will be merged today.

Recurse Center, 2014-07-03

Sat, 05 Jul 2014 09:49:37 -0400

I got distracted trying to add a hack to Hacker School’s blaggregator, to enable posting to different channels on zulip.
I learnt a few things about Django. Commented out references to views that don’t exists breaks templating. Formsets didn’t seem very convenient to use, for editing data showing the user only partial forms.
I played around with Clang for a bit, to try and use libclang to parse the C code for inspection, instead of writing one myself. It sometimes feels like an overkill, but it feels like it’ll make the whole code more robust. I’m not sure. I’ll need to play around for a bit more to decide.
I also finally got around to fix the resolution of my tty shells. I was in a bus, and wanted my laptop’s battery to last longer, and decided to use a tty shell with just emacs running. But, the resolution sucked. So, I fixed. Essentially it involved removing a blacklist file that an old version of Nvidia drivers left behind in /usr/share/grub-gfxpayload-lists/blacklist/ and setting GRUB_GFXMODE=1920x1080 in /etc/default/grub.
Later in the evening I also bought the domain octo.cat, given there were a bunch of people buying cat domains, and talking about it on Zulip!

Update [7/7/14]: I forgot to write about the presentations. There were interesting presentations!