Clock in and get-shit-done

I had setup a couple of hooks about an year ago that turn off all notifications while I'm clocking in. But, I find myself switching to the browser and jumping to twitter, out of habit. I've tried get-shit-done in the past to help myself break this habit. But enabling get-shit-done manually is step that quickly became a non-habit.

So, I hooked up get-shit-done into an org-clock-in-hook. The snippet below is what I added into a function that is added to this hook.

(with-temp-buffer
  (cd "/sudo::/")
  (shell-command "HOME=/home/punchagan get-shit-done work"))

get-shit-done needs to be run as root, since it does things like modifying /etc/hosts and restarting networking. Just calling get-shit-done as a shell command fails with the error sudo: no tty present and no askpass program specified. I found a couple of ways to fix this. The snippet above piggy-backs on tramp to allow for a way to enter the password for sudo to use. This also means that I don't need to enter the password, as long as the tramp connection is alive.

For someone worried about having such an easy way of running something as root, using something like gnome-ssh-askpass as the askpass program might work better.

(shell-command "SUDO_ASKPASS=\\"/usr/lib/openssh/gnome-ssh-askpass\\" get-shit-done work")

org-drill for making it stick!

Those who read the last few posts here, would know that I have been experimenting with org-drill (a spaced repetition extension to Org mode). I have been using the system (almost) religiously for the past 2 months, and I do find that it has helped a great deal! (in some respects). I have also spent a considerable amount of time trying to reduce the friction to put new stuff into the system, and am constantly on the look out for further improvements.

Using this system has definitely helped with retention, and I find that I can recall quite a few things I have read a few weeks ago, that I would normally have been unable to. Though, I can recall a lot of information, I have been having a feeling of "fragmentation": the feeling of just retaining individual bits/fragments of information, while losing out on actually internalizing the knowledge; not seeing the big picture, etc.

Wozniak (the author of super-memo) warns against learning without understanding, and memorizing before actually learning stuff. I haven't consciously added stuff into the system that I didn't understand (when I added it), but, later it does feel like I have lost some connections or the understanding, and am only holding onto the fragments of information.

The problems as explained in (read: as interpreted by me from) Make it Stick appear to be:

  1. The understanding (if any) at the time of adding stuff into the spaced-repetition system is untested. It may just be familiarity masquerading as understanding.
  2. The lack of any spaced repetitions for the overall concept/understanding and actual repetitions only for individual bits doesn't help retention of the understanding (even if there was any, in the first place).

To work around this, I'm going to try adding questions that test understanding, to the system. The Super-memo team strongly recommends keeping the drill items small and easy to answer. This may be helpful in keeping each drill session short, but I would really like to add conceptual questions to the system, and see how it goes. I hacked org-drill to allow me to type out answers, before looking at the "correct" ones. This is an adaptation of a system that a fellow Hacker Schooler uses, and shared. Also, hopefully forcing myself to type out the answer will help me get around the problem of sometimes saying "yeah I know that", then looking at the answer only to reaffirm the feeling of familiarity, rather than actually testing myself. I'm still going to continue adding quick and short questions that test "bits of information", though. But, hopefully the additional conceptual questions are going to tie things together and help fill in the gaps. Lets see how this goes!

For those interested, my hacks to org-drill below. The code is really a hack, and welcome any suggestions on cleaning up the code.

(advice-add 'org-drill-presentation-prompt :around 'pc/org-drill-presentation-prompt)

(defun pc/org-drill-presentation-prompt (old-fun &rest fmt-and-args)
  "A presentation prompt that allows capturing answers."

  (let ((cb (current-buffer))
	(heading (nth 4 (org-heading-components)))
	(entry-id (org-entry-get (point) "ID"))
	(input ""))
    (switch-to-buffer-other-window "*org-capture-drill-answer*")
    (org-mode)
    (insert "# Hit C-c C-c once you are done answering!\\n")
    (org-insert-heading-respect-content)
    (insert (format "Answer: %s" heading))
    (org-entry-put (point) "QUESTION_ID" entry-id)
    (goto-char (point-max))
    (insert "  ")
    (org-time-stamp-inactive '(16))
    (insert "\\n\\n  ")
    (while (not (and input (equal input "")))
      (ignore-errors
	(execute-kbd-macro input))
      (setq input (read-key-sequence nil)))
    (switch-to-buffer-other-window cb)
    (apply old-fun fmt-and-args)))

(advice-add 'org-drill-reschedule :around 'pc/org-drill-reschedule)

(defun pc/org-drill-reschedule (old-fun)
  "Calls the original reschedule, but also archives the answer"
  (prog1 (funcall old-fun)
    (let ((cb (current-buffer)))
      (switch-to-buffer-other-window "*org-capture-drill-answer*")
      (pc/org-refile-to-datetree "drill.org_archive")
      (message (buffer-name))
      (switch-to-buffer-other-window cb)
      (kill-buffer "*org-capture-drill-answer*"))))

(require 'org-datetree)
(defun pc/org-refile-to-datetree (journal)
  "Refile an entry to journal file's date-tree"
  (interactive "fRefile to: ")
  (let* ((journal (expand-file-name journal org-directory))
	 (date-string (or (org-entry-get (point) "TIMESTAMP_IA")
			  (org-entry-get (point) "TIMESTAMP")))
	 (dct (decode-time (or (and date-string (org-time-string-to-time date-string))
			       (current-time))))
	 (date (list (nth 4 dct) (nth 3 dct) (nth 5 dct))))
    (org-cut-subtree)
    (with-current-buffer (or (find-buffer-visiting journal)
			     (find-file-noselect journal))
      (org-mode)
      (save-excursion
	(org-datetree-file-entry-under (current-kill 0) date)
	(bookmark-set "org-refile-last-stored")))
    (message "Refiled to %s" journal)))

How I learnt to use Emacs' profiler

I learnt to use Emacs' profiler yesterday, after many hours of yak-shaving, trying to get Memacs working. Memacs is a memory extension system for Emacs written by Karl Voit, that I have been meaning to try out for a long time now. Seeing lots of review posts at the turn of the year and watching Karl's recent Emacs Chat with Sacha Chua pushed me to try and finally set it up.

I started writing a module to create a Memacs file – an org archive file – from my browser history. It was pretty easy to write, and I had it spitting out a huge file with 22k entries after about a couple of hours of work. Then I excitedly pulled up my agenda, and turned on the option to view archived entries, only to be super-disappointed. It turned out to be extremely slow! Actually, the agenda never came up with the 22k entries file that I had. At least not in 5 or so minutes, before I got impatient. The performance was unacceptable even when I reduced it to 5k entries.

I was pretty sure it wasn't that slow for Karl in his demo and tweeted to him, asking for a workaround. Meanwhile, I looked at his dot-emacs, but wasn't able to dig out what was needed to speed up things. He confirmed that his performance was way better than what I was getting.

First, I ruled out the possibility of it being because of the SSD, since clearly my CPU usage was peaking, and the task was CPU bound and not I/O. Next, I tried using the same file on a different machine (with a different version of Emacs and org-mode), and it worked blazingly fast. So, it was either the version of Emacs or org-mode that I was using.

I should have stopped, thought clearly, and started experimenting with org version, but hindsight is 20-20. I tried Ubuntu's pre-built Emacs and agendas were fast! I suspected my Emacs build, since I recently started building Emacs from git. I built two or three other versions of Emacs, and wasted a lot of time, before realizing that I wasn't using the org-mode source bundled inside Emacs for the tests, and there were two "independent" variables.

Finally, I began bisecting org-mode's source and found that all hell broke loose with an inconspicuous change around release 8.2.6. It turns out that org-overview was broken before this, and collapsing all the trees in a newly opened org-buffer (default option) wasn't working. Once this bug was fixed, opening huge org files would slow down by a great deal, in turn causing agenda generation to be unbearably slow.

All I had to do was add a #+STARTUP: showeverything to the top of the file. This speeded up things by about 50 times! It turns out, I later found out, that all of this is documented on Worg. I did try a few search engine queries, but sadly none of them brought this up. Adding the following to my config, speeded up agenda generation by about 150-200 times!

(setq org-agenda-inhibit-startup t) ;; ~50x speedup
(setq org-agenda-use-tag-inheritance nil) ;; 3-4x speedup

In the course of all this debugging, I learnt how to use Emacs' profiler. The profile reports along with git bisect, eventually helped me figure out what the problem was.

To profile the CPU usage, all you have to do is add a call like

(profiler-start 'cpu)  ;; or M-x profiler-start

at the place where you wish to start it. Emacs will then start collecting information about where time is being spent, by sampling every sampling-interval seconds (default 10^6 nanoseconds = 1 milli second).

You can view the information being collected, at any point of time using

(profiler-report) ;; or M-x profiler-report

The report is a nice, interactive tree with the percentage of time spent in each call. You can stop profiling by calling (profiler-stop). If you have more than one report, you can compare them by hitting = in one of the report buffers. I'm definitely going to use this for other things! (like speeding up my startup?)

Now that I have Memacs working with reasonably fast agenda views, I'm looking forward to collecting as much personal information as I can! Thanks Karl for writing Memacs. I am going to be a pretty heavy user, I think! There seem to be a few rough edges, though, and I hope to help smoothen them out a little bit, over the next few weeks.

More input sources for org-drill

I've been trying to use org-drill regularly for the last few weeks. I don't know how well it's been going but I have been sticking to the routine religiously. I haven't yet really tried out incremental reading, but in an attempt to make it as easy as possible, I wanted to have a pdf-reader integration, and some kind of integration with Kindle highlights. Browser integration is pretty straight-forward, thanks to some simple java-script.

I looked for a pdf-reader with some sort of plugin support, but I found nothing in Evince or Okular. I thought about pdfjs but it seemed slightly clunky to open pdfs in a browser, though I might shift to this if I don't like what I finally ended up with. Good old xpdf seemed to be the only pdf reader that had some support for custom keybindings that allowed users to run external commands. With a little Python, I was able to setup a work-flow to capture snippets from xpdf, to add to org-drill. Custom key-bindings somehow don't seem to work on xpdf bundled on Ubuntu. So, I ended up downloading and using the binary available on the xpdf site.

For Kindle highlights support, with minor updates to Thamer Mahmoud's clip2org, I have a simple way of getting all the "new" clippings/highlights as org-drill headlines. I haven't really started using this, and once I do, I may end-up automating even the merging of these items into the org-drill notes file. I'm looking forward to making better use of my Kindle, with this feature!

I don't know if it would be useful to have more context information like section titles/chapter titles when capturing from html/pdf, but it seems like an interesting problem to try to solve.

Also, it might be easier(?) if I probably tried to have a DE level keybinding, and some code to get selection and file name of the currently active window/application.

Simple org-clock and gnome/pidgin integration

See update below

I have been trying to get back to using org-mode and its clocking functionality, more often than not. I used to use it a lot, a few years ago, and haven't been using it, since I had been in my last job.

To help with it, I decided to integrate clocking in and out with changing status on Pidgin, and turning notifications on and off in Gnome. Here's a few lines of code that does this for me.

(defadvice org-clock-in (after pc/org-clock-in (&optional select start-time))
  "Turn gnome notifications off."
  (dbus-send-signal
   :session
   "org.gnome.SessionManager"
   "/org/gnome/SessionManager/Presence"
   "org.gnome.SessionManager.Presence"
   "SetStatus" 2)
  (shell-command "purple-remote setstatus?status=unavailable"))

(defadvice org-clock-out (after pc/org-clock-out (&optional switch-to-state fail-quietly at-time))
  "Turn gnome notifications back on."
  (dbus-send-signal
   :session
   "org.gnome.SessionManager"
   "/org/gnome/SessionManager/Presence"
   "org.gnome.SessionManager.Presence"
   "SetStatus" 0)
  (shell-command "purple-remote setstatus?status=available"))

Update <2014-11-01 Sat>

@baali tried to use my code, and it turns out it didn't work for him, because I forgot to mention that (ad-activate 'org-clock-in) needs to be run, after the defadvice code. I have no idea how it worked for me, without doing that. May be because I have defadvice for other functions?

Also, while debugging this, I found that defadvice is a deprecated way of doing this, and add-function is the way to go now. But, instead of advising the function, I decided to make use of org-clock-in-hook.

Here is the new code.

(defun pc/turn-off-notifications ()
  "Turn gnome notifications off."
  (dbus-send-signal
   :session
   "org.gnome.SessionManager"
   "/org/gnome/SessionManager/Presence"
   "org.gnome.SessionManager.Presence"
   "SetStatus" 2)
  (shell-command "purple-remote setstatus?status=unavailable"))

(defun pc/turn-on-notifications ()
  "Turn gnome notifications back on."
  (dbus-send-signal
   :session
   "org.gnome.SessionManager"
   "/org/gnome/SessionManager/Presence"
   "org.gnome.SessionManager.Presence"
   "SetStatus" 0)
  (shell-command "purple-remote setstatus?status=available"))

(add-hook 'org-clock-in-hook 'pc/turn-off-notifications)
(add-hook 'org-clock-out-hook 'pc/turn-on-notifications)

Learning to use Org-drill

Org-drill is an Org-mode extension that provides spaced-repetition and flash-card functionality. It has a wonderful documentation on Worg, but somehow I couldn't get myself to read the whole document, and setup org-drill, until now.

The setup is quite straight forward, once you have org-mode along with the contrib packages. Just (require 'org-drill), and you are all set! To add a new card, all you need to do is add a :drill: tag to the items you wish to "Org-drill". You can start a review session with simply M-x org-drill. You will be shown flash cards, and you can rate how correct and comfortable you were, in answering the questions. Based on your responses, the cards are scheduled for review. Start another review session, whenever you need one!

What I could only understand once I got myself to read the whole document was that:

  • The default scope of a drill session is the current file. But, you can start sessions with scopes like current tree, current directory, or a specified list of files. This is super-useful!
  • The review sessions are not automatically scheduled, based on when you schedule failed flash-cards. Scheduling the review for a card only sets a due-date for them, and effects, what you are asked in your next session.

Cram mode and incremental reading are also things I want to try, as I go along.

Happy Learning!

Recurse Center, 2014-07-21

Yesterday

  • It was exciting to have so many new Hacker Schoolers in the space. But, the excitement was kinda distracting too.
  • I spent most of the day looking at shairport, and libshairport and trying to wrap it, so I can use it to get data for the spectrum analyzer via the Air-play protocol. I was able to call it using ctypes, but it doesn't seem discoverable.
  • I also briefly looked at Julia and dwm, to try and port dwm to Julia. It doesn't seem like too much work, but is going to take me a while given how comfortable I'm with both Julia and C.
  • Yaron Minsky's talk on distributed systems was a great introduction to Distributed systems.
  • After the talk, I spent a little time updating my notes from an old org-mode workshop to present it at the emacs user group meet up today.

Today

  • I hope to get the spectrum analyzer to use libshairport and be able to play music sent through Air play.
  • Spend some time talking to the emacs user group about org-mode.
  • If I have any time, after getting this to work, I may play around with the dwm port.

Org based posts in Nikola

Chen Bin asked me to share my Nikola+Org work-flow, and share an example post.

The org source for any post can be found by changing the URL of a post from .html to .org.

I don't have much of a work-flow, because I don't post too often, but here is what I typically do, to make a new post.

I start off by creating a new post using

$ nikola new_post

and then give the post a title and start editing the post in Emacs.

I have a simple snippet that lets me insert tags, based on existing tags.

Once I'm happy with the content of a post, I run nikola auto to build the source and serve it locally, and see if the post "looks" reasonable, after being rendered.

Once, I'm happy with it, I commit the post and deploy it using nikola deploy.

GITHUB_SOURCE_BRANCH = 'master'
GITHUB_DEPLOY_BRANCH = 'gh-pages'
GITHUB_REMOTE_NAME = 'origin'

# Commands to execute to deploy. Can be anything, for example,
# you may use rsync:
# "rsync -rav output/* joe@my.site:/srv/www/site"
# And then do a backup, or ping pingomatic.
# To do manual deployment, set it to []

I also have a plugin, that posts captured bookmarks and quotes onto the blog, with a single command. I should probably make the sources of my blog open, and push it onto GitHub.

Update [2015-05-13 Wed]

  • I use my own plugin for Nikola which lets me write posts in org-mode. There is a similar plugin by Chen Bin, that exports posts to intermediate html, that is then used by Nikola.
  • The source for my blog is now on GitHub
  • I also have custom elisp to be able to make a new post, and deploy the site from within Emacs.

Bookmarks and Quotes plugin

I used to have a separate page for bookmarks and quotes on the old blog. They will now be shared as regular posts with 5 or more bookmarks/quotes. I use a bookmarklet in my browser to capture links to an org-file using org-capture protocol and then I have a small plugin to Nikola, that looks at such captured bookmarks and quotes and makes new posts out of them. This workflow is inspired by Brett Trepestra's web excursions plugin.

The plugin is currently a part of my blog source, and not published to Nikola's plugin repository since it is very specific to my setup. But if anybody is interested, I can publish the code somewhere public.

If you are interested to get these links and bookmarks into your feed reader, you can subscribe to their feed: [quotes], bookmarks. These posts will also make it to the main feed, though.

UPDATE [2015-11-15 Sun 20:45]

  • The quotes feed no longer exists!
  • The bookmarks feed doesn't really get too many updates. I use pinboard for my bookmarks, now.

Blogging with Nikola and Org-mode

Sigh! I made yet another change to the way this blog gets published. But, I have a feeling this mechanism, is here to stay!

We've been using Nikola quite regularly for our Ultimate site, and I quite like it. I've also contributed a bunch of features to Nikola to get it to work the way I would like it to. This weekend I ported my blog from o-blog to Nikola. The only thing that was stopping me from doing it, until now is the fact that all my posts are in org markup and porting them over to one of the formats that Nikola supports would be a PITA. So, I wrote a pretty simple plugin to Nikola to support posting from org files. And then with a little Python, I was able to move everything over to use Nikola.

Hopefully, I'll keep my blog more updated, from here on!