Large-scale Message-broker Federation

During the second half of 2009, I led a RabbitMQ/LShift project for the Ocean Observatory Initiative (OOI) where we were asked to design and prototype a messaging system based on AMQP and RabbitMQ that could grow smoothly all the way from LAN-sized networks up to large data-centers and the cross-continental networks connecting them.

The first phase of the research was completed toward the end of January, 2010. I released a design document and project report that describe the design approach and the implementation of the prototype system, respectively.¹ The design document briefly describes the “exchange space” and “exchange point” ideas that gave the project its direction, and the project report includes a list of starting points for future research.

I left RabbitMQ shortly thereafter to start preparation for my PhD research, but the project is continuing at the OOI.

About OOI

The Ocean Observatory Initiative (OOI) is a multi-year project run by a number of American and international universities and other organisations to build a globe-spanning sensing, data collection, data transport and data processing system for observing the world’s oceans.

Links

• The design document and project report describing the software that resulted from the project.

• OOI-CI, the Cyberinfrastructure project within OOI. Our research contributed to this area of the overall project.

• I gave a talk at Thoughtworks’s London Geek Night in May 2009 which describes some of this project’s research into message distribution patterns.