leastfixedpoint

Tangents

The following is a mixed bag of projects that are generally interesting, but that don’t warrant a page of their own. Some of them are more ideas in need of fleshing out than they are finished works.

Macromod, an implementation of syntax-case for R5RS scheme

In 2004, as I was getting more and more into syntax and parsing, I built an implementation of Scheme’s syntax-case macro facility from the original papers. I also developed a test suite for the implementation, cherry-picking the best of the syntax-case brain-teasers from the newsgroups and the web, and started the groundwork for an implementation of Flatt’s module system for Scheme. Sadly, this little project has languished for a few years; I’d love to have finished building it out into a full portable Flatt-style module system!

Spaceviz

A while back, I wondered where on earth all my hard-disk space went, and so I wrote a little tool, spaceviz, to draw me a semi-interactive map of my home directory. As it happens, Mac OS X includes a feature for displaying aggregate directory sizes, but it wasn’t apparent to me at the time! (For the record: in the Finder, under View → Show View Options, there’s a Calculate all sizes check box when you have a Finder window open, which if enabled, lets you get useful information out of sorting the window by size.)

JSON and JSON-RPC

I’ve been enthusiastic about JSON since I first heard about it back in 2005. I’ve written JSON codecs for a few languages:

UBF-A libraries in a few languages

I’ve been interested in Joe Armstrong’s UBF protocols since I first read about them in 2002 or so. I produced an experimental implementation of all three of UBF-A, UBF-B and UBF-C in Scheme— get in touch if you’re interested in the UBF-B or UBF-C code—and took the UBF-A code on, along with Joe’s Erlang implementation and C and Python implementations I built, to a multi-language UBF-A library that you can find here, complete with fledgling test suite.

If you’re looking for UBF implementations in general, or UBF-B implementations in particular, Joseph Norton’s library looks like a good bet, although it does not currently seem to be actively maintained. It seems also only to support Erlang, whereas previously it supported Python (not based on my code) and Java.

Pretty-printing XML from Scheme

So S-expressions—pardon me, XML—have become popular in recent years, with more and more library support on more and more platforms. In 2004, I was using MzScheme’s XML libraries, and filled a missing piece with xxexpr, a library for writing or pretty-printing SXML-encoded XML.

Documentation tools

I’ve built a couple of javadoc-like tools over the years, for use in programs I was writing at the time, either because there weren’t any that suited my requirements, the existing tools weren’t any good, or I simply felt like it. docscm is my 2002 effort for extracting API documentation from Scheme code, and when in 2007 I noticed that, to my surprise, there were no javadoc-like tools for .NET programs, I constructed ndocproc to fill the gap.

Icing request flow

Icing, a fledgling Scheme web framework

A gift-list type web site we built for a customer ended up producing so much useful library code that we took the time to pull out the reusable parts into Icing, a Scheme-language web development framework. Icing is a set of Scheme libraries, built atop SISCweb and making use of SXML and SXpath, that support servlet development within a J2EE environment. The reason it’s called “Icing” is that the project it was extracted from was called “Cake”…

Icing contains database access libraries, continuation-based web control flow, an XSLT-based presentation system, and a few other miscellaneous utilities. Unfortunately, development of the system stopped dead, since RabbitMQ came along just after it was released, and took away all my attention!

A Seaside-like system for server-side Javascript

I built a little system, called Savannah, around Chris Double’s javascript servlet library, which used Rhino’s continuations to build Seaside-like control flow. For more, see the blog posts that were written about the code:

A noddy HTTP server for Scheme

A couple of years back, I needed a flexible, lightweight, simple HTTP daemon library that I could use to invoke Scheme code dynamically. I had been using SISC, and didn’t much fancy pulling in a full J2EE servlet environment, so I wrote httpd.scm, a library that uses Java’s server sockets via SISC’s FFI to serve HTTP and invoke Scheme code in response to requests.

Since then, I’ve been using SISC much less; currently, the library is written to be portable, with specific bindings for working with Scheme48 1.8 and recent versions of Racket.

Screenshot of the console running against google.co.uk

A REPL for in-browser Javascript exploration

Debugging Javascript can be really tedious. Even Firebug can feel a bit heavy and awkward sometimes. Back in 2005 or so, while working on some client-side Javascript code, I built a Greasemonkey script that adds a Smalltalk-workspace-like REPL to any web page, letting you explore the Javascript objects running within it.

Various Diffing and Merging Tools

Besides synchrotron, I’ve built a Hunt-McIlroy diff and an implementation of diff3 on top of it in Squeak Smalltalk.

Just as an aside, the algorithm I use to construct a picture of the commit-history of a synchrotron repository—see the dvcsdemo to see what I mean—seems eerily similar to a register allocation algorithm! It’d be interesting to tease out that connection a little more.

Network protocol debugging aids

It’s always handy to be able to stick a tap or trace in between two endpoints in a network flow. Years ago there wasn’t anything I found suitable (the situation may be different, these days!), so I built my own: passthru is a program for dumping TCP or UDP data flowing through it on its way to somewhere else. I’ve used it extensively when debugging RabbitMQ; I find it much more convenient than tcpdump (but that’s likely just me).

X-Windows in pure scheme

As part of my frustration with the state of user interfaces, and in particular user interface libraries such as GTK+ and friends, in 2004 I quixotically started implementing an X11 core protocol client in pure scheme (MzScheme, to be precise). I got as far as building a noddy binary pattern-matching language out of macros, and using that to build a codec for a fair chunk of the core X protocol. The library was developed as far as being able to establish a connection, including parsing out the display and screen information. At the time I lost momentum, the next steps were to open a window and draw something. It’d be nice to dust this code off some day and see how much further I can get.

(For a very interesting illustration of some of the things that can come out of an approach like this, see Joe Armstrong’s ex11 (code, paper).)

RabbitMQ Streams (neé Feeds Hub)

Experimental WireIt-based user interface for Streams

I was involved1 in the early planning stages of RabbitMQ Streams, an interesting application built around RabbitMQ that makes it into a kind of Yahoo!-pipes-like, grid-like extensible message-processing pipeline construction toolkit. You can get a feel for the kinds of things it can do by checking out

Implementing and thinking about the design of Streams has led to some really interesting ideas2 about the topology inherent in a messaging-middleware system, which we hope will lead to improvements in RabbitMQ and maybe even AMQP itself.

Camstream

Schematic of Camstream's operation

As we were working toward the 1.0 release of RabbitMQ, back in 2007, I built a program called camstream to show off some of the features of the RabbitMQ server. It’s a Java program for Linux, OS X and Windows which uses AMQP and a simple from-first-principles video compression scheme that I cooked up to route live, streaming video from one or more webcams to one or more displays.

It uses a small JNI library on Linux to access the V4L video API, the Quicktime Java APIs for OS X video capture, and the Java Media Framework for everyone else.

Listening to your Webcam

In 2008, I discovered ARSS, “a program that analyses a sound file into a spectrogram and is able to synthesise this spectrogram, or any other user-created image, back into a sound.” Having experimented with webcams a few months before that as part of camstream, it occurred to me to see what kinds of sounds I could get by piping live video into ARSS.

Here’s an example frame, with the raw output from the camera on the left, and on the right, the results after processing with ImageMagick and immediately before conversion to a sound:

Example frames from the webcam-listening program

The resulting frame sounds like this (mp3). For more info, and the source code, see the blog post I made about the setup.

MPSIDL and fastmsg

Between 2000 and 2002, while I was working for eServGlobal, I built a few experimental high-speed message passing and remote-procedure-call systems for C++ and Java. The message passing libraries never saw use—eServGlobal had an extremely fast system already in place—but MPSIDL, a simple system similar to (but not compatible with) CORBA and its IDL, did see use on a few projects. Interfaces were specified in an IDL-like language, and there were bindings for C++ and Java on multiple platforms. You can get a feel for the kinds of things the project was aiming for by taking a look at the simple example in the documentation. The code is BSD-licensed, and still available on Sourceforge. (Incidentally, the project’s sourceforge web pages were generated by the offline-renderer in pyle!)

gen_packet socket option for Erlang/OTP

At the beginning of 2008, I found myself wanting to experiment with using Erlang’s built-in header-parsing abilities to process incoming AMQP 0-8/0-10 frames. The available options didn’t cover the slightly peculiar packet format AMQP specifies, so I wrote a patch (against OTP R12B-0) adding a new TCP socket option

{gen_packet, SizeLength, SizeOffset, SizeDelta}

which decodes packets by

  • requiring SizeLength + SizeOffset bytes to be available before analysing packets any further
  • reading the big-endian SizeLength-byte-long number at SizeOffset
  • adding SizeDelta to that value, and
  • considering the resulting value to be the expected length in bytes of the next packet to be returned, including its header

Some example protocols that gen_packet could be used for:

This protocol ...   ... works with this option
----------------------------------------------
IPv4                {gen_packet, 2, 2, 0}
AMQP 0-8            {gen_packet, 4, 3, 8}
AMQP 0-10           {gen_packet, 2, 2, 0}
PostgreSQL          {gen_packet, 4, 1, 1}

See this erlang-questions post I made for more details; for the patch itself, take the slightly-updated one from my follow-up posting. It hasn’t made its way into an OTP release; perhaps it’ll be picked up sometime in future.

Snarl

Snarl is a Growl-like user interface widget for Squeak Smalltalk. Smalltalk expressions like

Snarl label: 'The time' body: ('The time is ', (Time now) asString)

cause a translucent notification to appear on the right-hand-side of the screen for a few seconds before fading away. If the user moves the mouse over the notification, it stops fading until they move the mouse away again. A callback can be passed in to the notification widget for taking some action if the user clicks on it. The code is only about 205 lines long, including class comments; the speed and concision of development in Smalltalk can be amazing.


Erlang-IRCd

erlang-ircd is a pluggable Erlang IRC daemon application or library. The user supplies a callback module which the daemon uses to create and delete rooms, route messages etc. The code is embryonic at the moment: a stub callback module is implemented that uses a toy process-per-room implementation of shared rooms. It would be interesting to have a callback module for routing messages into and out of RabbitMQ, but I haven’t built one yet.

  1. along with Michael Bridgen, who led the project, developed most of the code, and coordinated its eventual deployment with the sponsoring customer ↩

  2. which I’m writing up at the moment, actually ↩