Piddling Plugins

Mon, 23 Jan 2012 12:15:37 GMT

The Shuffletron music player, in various branches, has accumulated some neat features (particularly last.fm scrobbling in Brit Butler's branch) that deserve merging, and ought to be cleanly separated from the core of the program. Leslie Polzer sent me a novel implementation early on which used generic functions for extensibility, adding/removing methods via the MOP as plugins load/unload. Clever as that was (and I'm impressed how little code is required, rereading the patch now), I wasn't comfortable with it, and the lack of a pressing need for a plugin interface let me put it off for a good long while.

Building extensibility around generic functions seemed the right thing to do though, and a slightly different idea, of writing plugins in the style of mixins and calling CHANGE-CLASS to enable them at runtime, stuck in the back of my head until (with some prodding) I was motivated to try it out. It's hardly a new idea (both Gsharp and McCLIM contain implementations of similar ideas, as does the AMOP book, just to name a few examples), and a minimal implementation doesn't take much code at all:

(defvar *configurations* (make-hash-table :test 'equal))

(defun configuration (plugins)
  (or (gethash plugins *configurations*)
      (setf (gethash plugins *configurations*)
            (make-instance 'standard-class
                           :name (format nil "MY-APPLICATION~{/~A~}" plugins)
                           :direct-superclasses (cons (find-class 'my-application)
                                                      (mapcar #'find-class plugins))))))

(defun reconfigure (application plugins &rest initargs)
  (apply #'change-class application (configuration plugins) initargs))

(defun active-plugins (instance)
  (mapcar #'class-name (rest (sb-mop:class-direct-superclasses (class-of instance)))))

(defun enable-plugin (application plugin &rest initargs)
  (apply #'reconfigure application (adjoin plugin (active-plugins application)) initargs))

(defun disable-plugin (application plugin)
  (reconfigure application (remove plugin (active-plugins application))))

(defun make-application (&rest initargs)
  (apply 'make-instance (configuration '()) initargs))

This isn't an ideal implementation, and there's a limit to how good it's going to get when CLOS doesn't fully support anonymous classes. However, a more serious attempt should work on arbitrary classes, provide a place to hang init/shutdown code for plugins, and the ability to list which plugins are enabled within an instance.

Piddling Plugins

Piddling-plugins adds these features using only slightly more code than above, along with some superfluous macro magic for writing defun-style definitions that are extensible by plugins. I've made light use of it in a branch of Shuffletron, confirming to myself that it's a good fit.

The code is tiny and self-explanatory, so I'll just post the examples from the README file for fun.

Examples

Imagine we have written a music player, looking something like this deliberately simplified code:

(defclass music-player () ())

(defun run-music-player ()
  ;; You need to set or bind *application* to your application instance
  ;; if you use defun-extensible. It's a good idea even if you don't.
  (let ((*application* (make-instance 'music-player)))
    (init-audio)
    (init-library *application*)
    (loop (execute-command (read-line)))))

Functions extensible by plugins can be defined using DEFUN-EXTENSIBLE. This is just syntactic sugar for defining a generic function specialized on the application object, with a wrapper that passes in the value of *APPLICATION*.

(defun-extensible execute-command (command)
  ...)

(defun-extensible play-song (song)
  ...)

(defun-extensible song-finished (song)
  ...)

Plugins extend the behavior of the application by defining methods on the extensible functions (or rather the generic functions defined behind the scenes, which are prefixed by "EXTENDING-"):

(defclass scrobbler ()
  ((auth-token :accessor auth-token)))

(defmethod plugin-enabled (app (plugin-name (eql 'scrobbler)) &key &allow-other-keys)
  (setf (auth-token app) (get-auth-token))
  (format t "~&Scrobbler enabled.~%"))

(defmethod plugin-disabled (app (plugin-name (eql 'scrobbler)))
  (format t "~&Scrobbler disabled.~%"))

(defmethod extending-song-finished :after ((plugin scrobbler) song)
  (scrobble song (auth-token plugin)))

(defclass status-bar () ())

(defmethod extending-play-song :after ((plugin status-bar) song)
  (redraw-status-bar))

(defmethod extending-execute-command :after ((plugin status-bar) command-line)
  (declare (ignore command-line))
  (redraw-status-bar))

To enable a plugin:

(enable-plugin *application* NAME [INITARGS...])

To disable a plugin:

(disable-plugin *application* NAME)

To set precisely which plugins are enabled:

(reconfigure *application* LIST-OF-PLUGINS [INITARGS...])

References

Strandh R., Hamer J., Baumann G. "Using Stealth Mixins to Achieve Modularity" (2007)
- Implentation in Gsharp: http://common-lisp.net/cgi-bin/gitweb.cgi?p=projects/gsharp/gsharp.git;a=blob;f=utilities.lisp
"Modes" implementation in McCLIM's ESA framework (formerly part of Climacs)
- http://git.boinkor.net/gitweb/mcclim.git/blob/HEAD:/ESA/utils.lisp#l446
Throwaway classes (comp.lang.lisp): http://groups.google.com/group/comp.lang.lisp/browse_thread/thread/78ef4a5fcd6a1661?pli=1
The Art of the Metaobject Protocol, Section 2.4
- http://www.foldr.org/~michaelw/lisp/amop-programmatic-class.lisp
ContextL - http://common-lisp.net/project/closer/contextl.html
Dynamic Classes - http://common-lisp.net/project/dynamic-classes/

Shuffletron bug: dirent.dt_type

Sat, 12 Dec 2009 19:15:29 GMT

Shuffletron, despite its toyish demeanor, makes an effort to do certain things correctly and efficiently which are critical to its being useful as a music player. To that end, it bypasses the hopeless Common Lisp pathname system in favor of using strings and POSIX system calls, and sidesteps the increasingly prevalent and often misguided "UTF-8 or bust!" mentality in favor of a more robust Unix-like approach of being encoding-agnostic by passing 8-bit characters transparently.

In particular, I made a point of ensuring that the library scan (a recursive walk of the directory) completes as quickly as possible, within reason. My benchmark was the combination of find and grep, which not only traverses the entire tree but does some useful processing on the result. It's not possible to reach that speed without writing excessively low-level Lisp code, but I got close enough that I was happy with the result. It may sound surprising for a task like directory traversal, but any cute processing your Lisp's filesystem interface does - consing/converting strings, decoding characters, building pathnames, converting C structures into nicer Lisp structures, etc. - quickly eats away at this peak performance. Those system calls run a lot faster than I first expected, once the relevant kernel caches are warmed up.

I've been playing with sshfs this evening and wanted to try using it to access my music library on the other side of the planet from a local Shuffletron instance (with appalling but sufficient latency and bandwidth, despite ostensibly being multi-megabit connections on both sides). Oddly, Shuffletron couldn't find any files - it said the library was empty. I soon recalled a potential corner I had cut in the filesystem scanning code.

While scanning the directory tree, you have to figure out whether each entry is a file, directory, or something else. Originally I had done this by calling stat and checking the st_mode field (I'd borrowed the functions from some code that had other reasons to stat the file). While converting the code from sb-posix to osicat-posix for portability to CCL (and beyond!), I cleaned it up to use the struct dirent.d_type field instead, obviating the call to stat. man 2 stat says this about the d_type field:

Other than Linux, the d_type field is available mainly only on BSD systems. This field makes it possible to avoid the expense of calling stat(2) if further actions depend on the type of the file. If the _BSD_SOURCE feature test macro is defined, then glibc defines the following macro constants for the value returned in d_type:

It's available on Linux (except for old versions)? Great, my program only works on Linux anyway, I'll use it! So went my reasoning. I had some misgivings on account of the note "If the file type could not be determined, the value DT_UNKNOWN is returned in d_type," but it seemed to work well enough, and figured I'd just move on rather than write the extra eight lines (!) of code to deal with what appeared to be an obscure case.

With sshfs, I finally found that obscure case. The dtype is always set to DT_UNKNOWN, and shuffletron can't get past the root directory of the library. Despite having written a (simple) FUSE filesystem myself, it isn't immediately clear how you can make that work from the filesystem's side, but it is clear that it was time to add a fallback to calling stat. Having done so, Shuffletron now happily scans my music library over sshfs, which would be great, except that it also takes something like 15 minutes walk the directory tree (even using find!), because every directory appears to involve several trips over the internet. Hundreds of directories times a few seconds each.. not good (this same thing annoys me when copying multiple small files via SCP, incidentally). Just goes to show that presenting something (a computer many thousands of miles away) as something it's not (a local filesystem) can only work so well. If only there were an interface (is there an interface?) to indicate the desired access pattern, so that sshfs could pull all that information quickly in a single batch and keep it in its cache.

This little fix will be present in the forthcoming Shuffletron 0.0.5 release, along with other great features that I've been dutifully testing since July or so. :)

Informal Methods

Piddling Plugins

Piddling Plugins

Examples

References

Shuffletron bug: dirent.dt_type