blog-admin and Nikola

Another post about blogging.

blog-admin now supports Nikola, thanks to yours truly. blog-admin is an Emacs package by CodeFalling that lets you view and manage your (static site generated) blog from within inside Emacs.

Nikola's command line utility is pretty nifty and does a bunch of useful things. I had a few utility functions to do common tasks like create new post and deploy blog. This worked well, but moment I came across this blog-admin's tabular view, I was sold!

org2blog (a blogging tool I used previously) had a tracking file that kept track of all the posts I made, and I used it quite a bit for navigation – thanks to org-mode's search functionality. The tabular view of blog-admin is even better! I really like the fact that the author has tried to keep the package generic enough to support any blog, and adding support for Nikola has been quite easy.

The filtering functionality is crude, but good enough for a start. One thing I want to add is a preview functionality for drafts. Showing some (writing) statistics would also be nice – No. posts in the last month, total published posts, etc. No promises, but you may see some of these things, soon. :)

Clock in and get-shit-done

I had setup a couple of hooks about an year ago that turn off all notifications while I'm clocking in. But, I find myself switching to the browser and jumping to twitter, out of habit. I've tried get-shit-done in the past to help myself break this habit. But enabling get-shit-done manually is step that quickly became a non-habit.

So, I hooked up get-shit-done into an org-clock-in-hook. The snippet below is what I added into a function that is added to this hook.

(with-temp-buffer
  (cd "/sudo::/")
  (shell-command "HOME=/home/punchagan get-shit-done work"))

get-shit-done needs to be run as root, since it does things like modifying /etc/hosts and restarting networking. Just calling get-shit-done as a shell command fails with the error sudo: no tty present and no askpass program specified. I found a couple of ways to fix this. The snippet above piggy-backs on tramp to allow for a way to enter the password for sudo to use. This also means that I don't need to enter the password, as long as the tramp connection is alive.

For someone worried about having such an easy way of running something as root, using something like gnome-ssh-askpass as the askpass program might work better.

(shell-command "SUDO_ASKPASS=\\"/usr/lib/openssh/gnome-ssh-askpass\\" get-shit-done work")

Elfeed hook to fetch full content

I have started to use Pinboard's unread tag as my to-read list. It has a bookmark-let that works pretty well for adding stuff into my "to-read" list. I then catch up on this list using elfeed and subscribing to the unread items' RSS feed. The work-flow is pretty nice for adding stuff into the list, and finding items on the list. But, when it comes to the actual reading part, the entries in the feed don't have the actual content I want to read, and I end up opening the links in a browser.

Inspired by a comment from FiloSottile, I realized it should be pretty easy to setup a hook that fetches the actual content to make my reading work-flow smoother. I wrote a small script, using python-readability, to fetch the page content, given a URL. This script is then hooked onto elfeed-new-entry-hook, to fetch content of for new entries as they are fetched. All the old entries can be easily fixed with a single call to elfeed-apply-hooks-now.

(defun pc/get-url-content (url)
  "Fetches the content for a url."
  (shell-command-to-string (format "~/bin/get_article.py %s" url)))

(defun pc/get-entry-content (entry)
  "Fetches content for pinboard entries that are not tweets."
  (interactive
   (let ((entry elfeed-show-entry))
     (list entry)))

  (let ((url (elfeed-entry-link entry))
	(feed-id (elfeed-deref (elfeed-entry-feed-id entry)))
	(content (elfeed-deref (elfeed-entry-content entry))))
    (when (and (s-matches? "feeds.pinboard.in/" feed-id)
	       (not (s-matches? "twitter.com/\\\\|pdf$\\\\|png$\\\\|jpg$" url))
	       (string-equal "" content))
      (setq content (pc/get-url-content url))
      (setf (elfeed-entry-content entry) (elfeed-ref content)))))

(add-hook 'elfeed-new-entry-hook #'pc/get-entry-content)

Say Howdy with Emacs!

Staying in touch with people is something I'm not very good at. Since I am not on popular (among my friends/family) networks – FB and Whatsapp – I don't even see random updates from people, to get some sense of being in touch.

I recently read some old posts by Sacha Chua and was inspired by how much code she had for contact management in her old blog posts. I was inspired by this post in particular to try and be more meticulous about how I stay in touch with people. Michael Fogleman blogged about his contact management work-flow using keepintouch. It seemed to do most of what I wanted, but I wanted this to be integrated with my org-contacts-db and I felt having native elisp code would make it easier to hook up email, chat, etc. to this.

I ended up writing a small utility called howdy to help me keep in touch with people. It currently has only a couple of features:

  • M-x howdy lets me update the last contacted timestamp for a contact.
  • Shows me contacts that I'm out of touch in the agenda, once I add the following snippet to an agenda file.
    * Howdy
      %%(howdy-howdy)
    

I also have a few hooks to hook up jabber messages and email to update the db. I've added them to howdy-hooks.el in case anybody else wants to use them. They can also be used as examples to write other hooks. Feel free to contribute other hooks or suggest improvements. The library also ships with a modest test suite, that will hopefully make it easier for others to contribute.

I'm looking forward to experimenting with this over the next few weeks and improving it. Hopefully, it'll help me keep in touch, better than I do now.

Playing music using mpsyt from Emacs

I've started using the wonderful mpsyt to play any music from youtube, since I'm not really interested in the video. But, since I use emacs for chat/IRC, I end up getting youtube links into emacs and opening them opens them up in my browser. I ended up writing some elisp to play the songs from within an instance of mpsyt running inside an emacs buffer.

(defun pc/short-url-at-point ()
  "Gets the short url at point.

This function is required only because
`thing-at-point-url-at-point' ignores urls (without a scheme)
that don't start with www."
  (let ((bounds (thing-at-point-bounds-of-url-at-point t)))
    (when (and bounds (< (car bounds) (cdr bounds)))
      (buffer-substring-no-properties (car bounds) (cdr bounds)))))

(defun pc/mpsyt-url (url)
  (let ((buffer (current-buffer))
	(mpsyt-proc-name "*mpsyt*"))

    ;; Start a new term with *mpsyt* if there isn't one
    (unless (get-process mpsyt-proc-name)
      (when (get-buffer mpsyt-proc-name)
	(kill-buffer (get-buffer mpsyt-proc-name)))
      (ansi-term "mpsyt" "mpsyt"))

    ;; Play given url in mpsyt
    (let ((mpsyt-proc (get-process mpsyt-proc-name)))
      ;; If something is already playing, stop it and play this...
      (term-send-string mpsyt-proc "\\n\\n\\n")
      ;; We wait for a bit, since looking for the prompt seems to fail, sometimes?
      (sleep-for 1)
      (term-send-string mpsyt-proc "\\n")

      ;; Actually send the command to playurl
      (term-simple-send (get-process mpsyt-proc-name)
			(format "playurl %s" url)))

    (switch-to-buffer buffer)))

(defun pc/mpsyt-url-at-point ()
  "Play the URL at point using mpsyt."
  (interactive)
  (let ((url (or (url-get-url-at-point) (pc/short-url-at-point))))
    (if (not url)
      (message "No URL found")
	(message (format "Playing %s with mpsyt" url))
      (pc/mpsyt-url url))))

The current version of mpsyt crashes when run from inside emacs due to a bug in the code to get the terminal size, which should be fixed once this patch is merged.

I would've expected thing-at-point-url-at-point to be able to find urls even when they don't have a schema, but it tries to guess the schema from urls and fails to work when the url starts with youtube.com instead of www.youtube.com.

I started off using the command-line interface of mpsyt by running it using shell-command or start-process. But, it seemed useful to have a buffer of mpsyt to switch to – easier to search for new music, repeating songs, etc. Not all tasks/actions are achievable through mpsyt's command line args.

I ended up writing more code than I thought I would have to1. But, I'm pretty happy with how this all works, right now.

Footnotes:

1

- Isn't it true, more often than not?

org-drill for making it stick!

Those who read the last few posts here, would know that I have been experimenting with org-drill (a spaced repetition extension to Org mode). I have been using the system (almost) religiously for the past 2 months, and I do find that it has helped a great deal! (in some respects). I have also spent a considerable amount of time trying to reduce the friction to put new stuff into the system, and am constantly on the look out for further improvements.

Using this system has definitely helped with retention, and I find that I can recall quite a few things I have read a few weeks ago, that I would normally have been unable to. Though, I can recall a lot of information, I have been having a feeling of "fragmentation": the feeling of just retaining individual bits/fragments of information, while losing out on actually internalizing the knowledge; not seeing the big picture, etc.

Wozniak (the author of super-memo) warns against learning without understanding, and memorizing before actually learning stuff. I haven't consciously added stuff into the system that I didn't understand (when I added it), but, later it does feel like I have lost some connections or the understanding, and am only holding onto the fragments of information.

The problems as explained in (read: as interpreted by me from) Make it Stick appear to be:

  1. The understanding (if any) at the time of adding stuff into the spaced-repetition system is untested. It may just be familiarity masquerading as understanding.
  2. The lack of any spaced repetitions for the overall concept/understanding and actual repetitions only for individual bits doesn't help retention of the understanding (even if there was any, in the first place).

To work around this, I'm going to try adding questions that test understanding, to the system. The Super-memo team strongly recommends keeping the drill items small and easy to answer. This may be helpful in keeping each drill session short, but I would really like to add conceptual questions to the system, and see how it goes. I hacked org-drill to allow me to type out answers, before looking at the "correct" ones. This is an adaptation of a system that a fellow Hacker Schooler uses, and shared. Also, hopefully forcing myself to type out the answer will help me get around the problem of sometimes saying "yeah I know that", then looking at the answer only to reaffirm the feeling of familiarity, rather than actually testing myself. I'm still going to continue adding quick and short questions that test "bits of information", though. But, hopefully the additional conceptual questions are going to tie things together and help fill in the gaps. Lets see how this goes!

For those interested, my hacks to org-drill below. The code is really a hack, and welcome any suggestions on cleaning up the code.

(advice-add 'org-drill-presentation-prompt :around 'pc/org-drill-presentation-prompt)

(defun pc/org-drill-presentation-prompt (old-fun &rest fmt-and-args)
  "A presentation prompt that allows capturing answers."

  (let ((cb (current-buffer))
	(heading (nth 4 (org-heading-components)))
	(entry-id (org-entry-get (point) "ID"))
	(input ""))
    (switch-to-buffer-other-window "*org-capture-drill-answer*")
    (org-mode)
    (insert "# Hit C-c C-c once you are done answering!\\n")
    (org-insert-heading-respect-content)
    (insert (format "Answer: %s" heading))
    (org-entry-put (point) "QUESTION_ID" entry-id)
    (goto-char (point-max))
    (insert "  ")
    (org-time-stamp-inactive '(16))
    (insert "\\n\\n  ")
    (while (not (and input (equal input "")))
      (ignore-errors
	(execute-kbd-macro input))
      (setq input (read-key-sequence nil)))
    (switch-to-buffer-other-window cb)
    (apply old-fun fmt-and-args)))

(advice-add 'org-drill-reschedule :around 'pc/org-drill-reschedule)

(defun pc/org-drill-reschedule (old-fun)
  "Calls the original reschedule, but also archives the answer"
  (prog1 (funcall old-fun)
    (let ((cb (current-buffer)))
      (switch-to-buffer-other-window "*org-capture-drill-answer*")
      (pc/org-refile-to-datetree "drill.org_archive")
      (message (buffer-name))
      (switch-to-buffer-other-window cb)
      (kill-buffer "*org-capture-drill-answer*"))))

(require 'org-datetree)
(defun pc/org-refile-to-datetree (journal)
  "Refile an entry to journal file's date-tree"
  (interactive "fRefile to: ")
  (let* ((journal (expand-file-name journal org-directory))
	 (date-string (or (org-entry-get (point) "TIMESTAMP_IA")
			  (org-entry-get (point) "TIMESTAMP")))
	 (dct (decode-time (or (and date-string (org-time-string-to-time date-string))
			       (current-time))))
	 (date (list (nth 4 dct) (nth 3 dct) (nth 5 dct))))
    (org-cut-subtree)
    (with-current-buffer (or (find-buffer-visiting journal)
			     (find-file-noselect journal))
      (org-mode)
      (save-excursion
	(org-datetree-file-entry-under (current-kill 0) date)
	(bookmark-set "org-refile-last-stored")))
    (message "Refiled to %s" journal)))

How I learnt to use Emacs' profiler

I learnt to use Emacs' profiler yesterday, after many hours of yak-shaving, trying to get Memacs working. Memacs is a memory extension system for Emacs written by Karl Voit, that I have been meaning to try out for a long time now. Seeing lots of review posts at the turn of the year and watching Karl's recent Emacs Chat with Sacha Chua pushed me to try and finally set it up.

I started writing a module to create a Memacs file – an org archive file – from my browser history. It was pretty easy to write, and I had it spitting out a huge file with 22k entries after about a couple of hours of work. Then I excitedly pulled up my agenda, and turned on the option to view archived entries, only to be super-disappointed. It turned out to be extremely slow! Actually, the agenda never came up with the 22k entries file that I had. At least not in 5 or so minutes, before I got impatient. The performance was unacceptable even when I reduced it to 5k entries.

I was pretty sure it wasn't that slow for Karl in his demo and tweeted to him, asking for a workaround. Meanwhile, I looked at his dot-emacs, but wasn't able to dig out what was needed to speed up things. He confirmed that his performance was way better than what I was getting.

First, I ruled out the possibility of it being because of the SSD, since clearly my CPU usage was peaking, and the task was CPU bound and not I/O. Next, I tried using the same file on a different machine (with a different version of Emacs and org-mode), and it worked blazingly fast. So, it was either the version of Emacs or org-mode that I was using.

I should have stopped, thought clearly, and started experimenting with org version, but hindsight is 20-20. I tried Ubuntu's pre-built Emacs and agendas were fast! I suspected my Emacs build, since I recently started building Emacs from git. I built two or three other versions of Emacs, and wasted a lot of time, before realizing that I wasn't using the org-mode source bundled inside Emacs for the tests, and there were two "independent" variables.

Finally, I began bisecting org-mode's source and found that all hell broke loose with an inconspicuous change around release 8.2.6. It turns out that org-overview was broken before this, and collapsing all the trees in a newly opened org-buffer (default option) wasn't working. Once this bug was fixed, opening huge org files would slow down by a great deal, in turn causing agenda generation to be unbearably slow.

All I had to do was add a #+STARTUP: showeverything to the top of the file. This speeded up things by about 50 times! It turns out, I later found out, that all of this is documented on Worg. I did try a few search engine queries, but sadly none of them brought this up. Adding the following to my config, speeded up agenda generation by about 150-200 times!

(setq org-agenda-inhibit-startup t) ;; ~50x speedup
(setq org-agenda-use-tag-inheritance nil) ;; 3-4x speedup

In the course of all this debugging, I learnt how to use Emacs' profiler. The profile reports along with git bisect, eventually helped me figure out what the problem was.

To profile the CPU usage, all you have to do is add a call like

(profiler-start 'cpu)  ;; or M-x profiler-start

at the place where you wish to start it. Emacs will then start collecting information about where time is being spent, by sampling every sampling-interval seconds (default 10^6 nanoseconds = 1 milli second).

You can view the information being collected, at any point of time using

(profiler-report) ;; or M-x profiler-report

The report is a nice, interactive tree with the percentage of time spent in each call. You can stop profiling by calling (profiler-stop). If you have more than one report, you can compare them by hitting = in one of the report buffers. I'm definitely going to use this for other things! (like speeding up my startup?)

Now that I have Memacs working with reasonably fast agenda views, I'm looking forward to collecting as much personal information as I can! Thanks Karl for writing Memacs. I am going to be a pretty heavy user, I think! There seem to be a few rough edges, though, and I hope to help smoothen them out a little bit, over the next few weeks.

Jabber message queue

I've always wanted to be able to queue up messages to send to friends, until I go online the next time. I tried using email instead of chat a few times, or just ended up staying online with a busy status.

Finally, now that I have started using jabber-mode for chatting from within Emacs, I took out the time to write a "queuing system" for sending chat messages, similar to the mail queue for smtpmail. Instead of persisting sexps, though, I persist the messages in a JSON format and the queue is flushed every time I connect to jabber, in a jabber-post-connect-hook.

To make the interface as similar to the interface available when I am online, I hacked completion for the to ID using email addresses in my address book (mu4e~contact-list). I really like the fact that the chat buffer opens up, and I can type and send messages like I usually do. Hitting RET after typing a message queues it up, instead of trying to send it. Smooth!

The code is in my .emacs

erc-notifications when Emacs not in focus

I have been trying to get ERC working with notifications. Julien Danjou's wonderful notifications module for ERC is great, but it is annoying to get notifications even when Emacs is in focus.

I had looked at circe-notifications, which has the feature but uses xdotool and xprop to do it. I was looking for something simpler, though… and it suddenly struck me that I have an auto-save hook in Emacs that is run when I focus out of it. I wondered if I could disable and enable notifications on focus, and it worked.

In case it is useful for somebody else -

(add-to-list 'erc-modules 'notifications)
(erc-notifications-mode)
(add-hook 'focus-out-hook 'erc-notifications-enable)
(add-hook 'focus-in-hook 'erc-notifications-disable)

I wonder if there are some corner cases where this doesn't work, and that's why the author of circe-notifications chose the tools that he did.

More input sources for org-drill

I've been trying to use org-drill regularly for the last few weeks. I don't know how well it's been going but I have been sticking to the routine religiously. I haven't yet really tried out incremental reading, but in an attempt to make it as easy as possible, I wanted to have a pdf-reader integration, and some kind of integration with Kindle highlights. Browser integration is pretty straight-forward, thanks to some simple java-script.

I looked for a pdf-reader with some sort of plugin support, but I found nothing in Evince or Okular. I thought about pdfjs but it seemed slightly clunky to open pdfs in a browser, though I might shift to this if I don't like what I finally ended up with. Good old xpdf seemed to be the only pdf reader that had some support for custom keybindings that allowed users to run external commands. With a little Python, I was able to setup a work-flow to capture snippets from xpdf, to add to org-drill. Custom key-bindings somehow don't seem to work on xpdf bundled on Ubuntu. So, I ended up downloading and using the binary available on the xpdf site.

For Kindle highlights support, with minor updates to Thamer Mahmoud's clip2org, I have a simple way of getting all the "new" clippings/highlights as org-drill headlines. I haven't really started using this, and once I do, I may end-up automating even the merging of these items into the org-drill notes file. I'm looking forward to making better use of my Kindle, with this feature!

I don't know if it would be useful to have more context information like section titles/chapter titles when capturing from html/pdf, but it seems like an interesting problem to try to solve.

Also, it might be easier(?) if I probably tried to have a DE level keybinding, and some code to get selection and file name of the currently active window/application.