Elfeed on Noetic Nought

Why my Elfeed index was 0 bytes

Fri, 30 Jan 2026 16:58:00 +0530

My Linux machine crashed, while I was in the middle of a video call. I restarted quickly, and continued the discussion. Later, when I was trying to sync Elfeed updates onto the Elfeed Offline app on my phone, I found it acting weird. All the bazillion things needed to get it working were in place, but it was still complaining about the Emacs Elfeed server not being accessible. I tried opening Elfeed inside Emacs and failed! The index file in Elfeed’s DB was empty! Gone! Poof!

Frustrating, but I didn’t have time to look into what happened. I would’ve been furious if I had been using Elfeed for longer and had a lot more metadata saved. But, it was only a few weeks of lost metadata - posts I read, starred, etc. I quickly setup a Git based backup to prevent future losses and moved on.

Later, I found time to dig into what might have happened…

The breadcrumbs

Elfeed stores all the metadata for all the posts in an index file with a content addressed store of the contents of each of the posts. The index file is simply a dump of the hash-table containing the metadata for the subscribed feeds, their entries and metadata like tags, read/unread status, etc.

The DB save happens in elfeed-db-save, which simply dumps the hash-table to disk inside a call to the with-temp-file macro.

(defun elfeed-db-save ()
  ; <snip>
  (with-temp-file (expand-file-name "index" elfeed-db-directory)
    ; ...
    (princ (format ";;; Elfeed Database Index (version %s)\n\n"
                       elfeed-db-version))
    ; ...
    (prin1 elfeed-db)
    ;...
    ))

with-temp-file, as its documentation says, lets you create a new buffer, evaluate the body there, and write the buffer to file. For a moment, I thought there was some temporary file involved, but nope! I guess the name comes as an extension from with-temp-buffer which does create a temporary buffer where the body of the macro gets evaluated. Stripped to its core, this function is:

`(let ((,temp-file ,file)
       (,temp-buffer (generate-new-buffer " *temp file*" t)))
   (prog1
       (with-current-buffer ,temp-buffer
         ,@body)
     (with-current-buffer ,temp-buffer
       (write-region nil nil ,temp-file nil 0))))

`write-region` and `O_TRUNC`

So, write-region is the workhorse which writes the contents of the temporary buffer to disk. It’s a roughly 300 line long C function that essentially opens the file with the flags O_WRONLY | O_CREAT | O_TRUNC (in this case) and then does something to write the contents to the file, etc. Honestly, I didn’t look at anything else too carefully after I spotted the O_TRUNC.

The man page of open explains O_TRUNC as follows:

O_TRUNC :: If the file already exists and is a regular file and the access mode allows writing (i.e., is O_RDWR or O_WRONLY) it will be truncated to length 0.

Voilà!

The write is not atomic. The file first gets truncated to length 0, and then we hope that the new contents get correctly written before something goes wrong.

It now makes sense why the file got truncated to 0 bytes. The crash happened after the open, but before the write. Somehow the crash happened in this short window, and poof!

Emacs has backup files, doesn’t it?

Temporary files with ~ in their file extensions have definitely annoyed me in the past when Emacs created them where I didn’t want them. So, I do know that Emacs has back-up mechanisms out of the box. But, it turns out that the backups occur in code paths that are more interactive, like save-buffer, write-file, etc. And not via the programmatic APIs like write-region or the higher level with-temp-file. save-buffer calls backup-buffer before writing, but write-region is much more low-level and doesn’t deal with backups.

The fix

My “fix” for this is to backup the Elfeed data in a git repository to be able to recover from any such corruptions of data, which I already wrote about here.

I know that any code that uses with-temp-file (or write-region) could be affected by this, and the right fix for this may be to write to a temporary file and rename it. Maybe next time I lose data I’ll actually fix it properly.

Elfeed DB backup hooks

Tue, 20 Jan 2026 23:13:00 +0530

I had a bunch of things running on my laptop – video call with screenshare, my Windows VM, Firefox with a lot of tabs, etc. And my laptop crashed! I didn’t have the time to dig into what, why and how.

Later in the day, I discovered my Elfeed’s DB was gone – blown away. :( I’m guessing the crash happened in the middle of elfeed-db-save, and the data was lost.

I’ve now added some back-up for the DB, since I intend to use Elfeed regularly.

(defvar pc/elfeed-db-save-timer nil
  "Timer for debounced elfeed database saves.")

(defun pc/elfeed-db-save-and-backup ()
  "Save the elfeed database and commit to git."
  (when (and (boundp 'elfeed-db) elfeed-db)
    (elfeed-db-save)
    (let ((default-directory elfeed-db-directory))
      (when (file-exists-p ".git")
        (call-process "git" nil "*elfeed-db-backup*" nil "add" "-A")
        (call-process "git" nil "*elfeed-db-backup*" nil "commit" "-m" "auto-backup")
        (call-process "git" nil "*elfeed-db-backup*" nil "push" "origin" "main")))))

(defun pc/elfeed-db-save-soon ()
  "Schedule a database save after 10 seconds of idle."
  (interactive)
  (when pc/elfeed-db-save-timer
    (cancel-timer pc/elfeed-db-save-timer))
  (setq pc/elfeed-db-save-timer
        (run-with-idle-timer 10 nil #'pc/elfeed-db-save-and-backup)))

;; Save and backup when tags change (elfeed-web usage)
(add-hook 'elfeed-tag-hooks   (lambda (&rest _) (pc/elfeed-db-save-soon)))
(add-hook 'elfeed-untag-hooks (lambda (&rest _) (pc/elfeed-db-save-soon)))

;; Save and backup when new entries are added
(add-hook 'elfeed-db-update-hook #'pc/elfeed-db-save-soon)

An offline-friendly Elfeed web UI

Wed, 07 Jan 2026 02:31:00 +0530

I want to read articles from my RSS subscriptions on my phone without signing up to a hosted service or running a public server.

Elfeed, the Emacs feed reader, already lets me manage my subscriptions and read from within Emacs. It comes with an experimental web UI, but needs the server (running on my laptop) to be accessible whenever I want to read. My phone becomes useless the moment my laptop sleeps.

So, I built an alternate web UI with a service worker that caches content on the client side for a smooth offline reading experience. It’s written in OCaml and compiled to JavaScript using js_of_ocaml.

Why bother with RSS feeds?

It is annoyingly easy to fall into the trap of short form videos or other algorithmically “curated” feeds and get sucked into scrolling mindlessly. I fear this is going to get even worse with all the generative AI stuff.

I want to be more deliberate about what I consume. RSS feels calmer and gives me a greater sense of control. Or it may just be nostalgia from good old Google Reader days.

Why Emacs and Elfeed?

Since Google Reader was killed, I’ve bounced between many readers but none of them has really stuck with me. I’ve had different phases of using Elfeed, over the years, though.

I like the fact that everything is local with Elfeed – no hosted services, no public servers. And since it lives inside Emacs, everything from tagging to how entries are displayed can be tweaked.

Why Elfeed offline?

Elfeed also ships a basic web UI that lets me read my feeds from a different device, but only when my laptop is online and reachable. I’d like to be able to read these posts even when I’m “on the road”, say, while waiting for a train or while taking a cab ride.

I knew it should be possible to do this with some client side caching. And, I considered improving the web UI of Elfeed itself but it doesn’t seem to be actively maintained in the last couple of years. Creating a separate project would also give me more freedom to experiment.

How does it work?

Elfeed offline comes with a tiny Dream based webserver that acts as a proxy in front of the Emacs Elfeed webserver for the API requests. The Dream web server also serves the static assets like the HTML, JavaScript, stylesheets and the Service Worker for the web app. The diagram below shows how data flows between the different components.

+---------------+     +-----------------+
|  Emacs Elfeed |<--->|     Dream       |
|     Server    |     |   Web-server    |
+---------------+     +-----------------+
                               ^
                               |
                               v
 +------------+        +----------------+       +---------+
 |    Web     |        |    Service     |       | Browser |
 |  Client    |<------>|     Worker     |<----->|  Cache  |
 +------------+        +----------------+       +---------+

The Service Worker intercepts all the GET requests from the client and responds using a cache-first-with-cache-refresh strategy. This makes the reading experience feel very responsive. The web client also implements a “good enough” clone of the original search functionality of Elfeed to allow searching and filtering content while offline.

The web app also lets me mark entries as read or star them when I’m offline. The Service Worker caches these operations and updates the server when the server becomes reachable. There’s no smart conflict resolution here, and the API requests just overwrite and update the state on the server.

I have been using this for a couple of weeks now, and am quite happy with how responsive the UI feels and the number of posts I managed to read so far.

But, there are definitely some rough edges to smoothen out.

It’s a pain to make sure that both the servers are running when I’m online and want to sync new updates to my phone. I’m considering adding a tiny Emacs helper to manage this.
I need to remember to open the app on my phone to trigger a cache update before stepping away from my laptop. I want to explore using the Background Sync API (not available on Firefox) to possibly make this easier.
The UI could use some improvements to indicate pending syncs when offline and also indicate the last sync with the server to help ensure everything is synced before stepping away from my laptop.
Service workers need the server to be accessible via https. I have a wrapper script around mkcert to make this easy, but some apps on my phone refuse to work correctly when there are user installed certificates.

Why OCaml?

At work, I write OCaml every day and enjoy it. This project seemed like a good one to try out js_of_ocaml since everything is local and I don’t need to worry about bundle sizes, etc.

js_of_ocaml makes it quite convenient to share code between the server and the client side. And given there’s also a “third” Service Worker component in the mix, I thought it would make life easier. For instance, the message types and the serialization code for the messages between the client and the service worker are shared.

I’m also using this project to experiment with Dune’s “soon to be released”™ package management that our team at Tarides is working on building.

How can you use it?

You can try out Elfeed offline’s UI here as a static site (with some posts from Planet Emacslife and The OCaml Planet). You should be able to read posts, search and filter for posts, mark them as read or star them. Any changes you make should be preserved between reloads.

If you are an Emacs user but don’t use Elfeed, you can checkout the Elfeed’s README for instructions to set it up. It also has links to a bunch of blog posts and videos that show off the features of Elfeed.

If you are already using Elfeed, you should be able to set it up quite easily if you’re happy installing some OCaml tooling. The installations instructions are available in the README.

Outro

I have been enjoying using Elfeed offline for the past couple of weeks. And I’m hoping I’ll end up spending much more time reading in it, than fixing and building it. It currently definitely feels like something I built for myself, but I’d love to hear from others who try it out.

Elfeed hook to fetch full content

Sat, 19 Dec 2015 13:42:29 +0530

I have started to use Pinboard’s unread tag as my to-read list. It has a bookmark-let that works pretty well for adding stuff into my “to-read” list. I then catch up on this list using elfeed and subscribing to the unread items' RSS feed. The work-flow is pretty nice for adding stuff into the list, and finding items on the list. But, when it comes to the actual reading part, the entries in the feed don’t have the actual content I want to read, and I end up opening the links in a browser.

Inspired by a comment from FiloSottile, I realized it should be pretty easy to setup a hook that fetches the actual content to make my reading work-flow smoother. I wrote a small script, using python-readability, to fetch the page content, given a URL. This script is then hooked onto elfeed-new-entry-hook, to fetch content of for new entries as they are fetched. All the old entries can be easily fixed with a single call to elfeed-apply-hooks-now.

(defun pc/get-url-content (url)
  "Fetches the content for a url."
  (shell-command-to-string (format "~/bin/get_article.py %s" url)))

(defun pc/get-entry-content (entry)
  "Fetches content for pinboard entries that are not tweets."
  (interactive
   (let ((entry elfeed-show-entry))
     (list entry)))

  (let ((url (elfeed-entry-link entry))
        (feed-id (elfeed-deref (elfeed-entry-feed-id entry)))
        (content (elfeed-deref (elfeed-entry-content entry))))
    (when (and (s-matches? "feeds.pinboard.in/" feed-id)
               (not (s-matches? "twitter.com/\\|pdf$\\|png$\\|jpg$" url))
               (string-equal "" content))
      (setq content (pc/get-url-content url))
      (setf (elfeed-entry-content entry) (elfeed-ref content)))))

(add-hook 'elfeed-new-entry-hook #'pc/get-entry-content)