Random Hacks

Write a 32-line chat client using Ruby, AMQP & EventMachine (and a GUI using Shoes)

Eric Kidd — Fri, 08 May 2009 18:06:00 -0400

Have you ever considered using instant messages to communicate between programs? You can do this using Jabber’s XMPP protocol, of course. But it’s also worth taking a look at AMQP, a distributed messaging protocol first used at JPMorgan Chase. AMQP is fast, easy to use, and implemented by at least 4 open source servers.

To try it out, install the excellent Ruby AMQP bindings, and set up the RabbitMQ server (which is written in Erlang using Mnesia). On a Mac, you might do something like this:

sudo gem install amqp
sudo port install python25 rabbitmq-server
sudo rabbitmq-server

Once your server is running, save the following code as chat.rb:

require 'rubygems'
gem 'amqp'
require 'mq'

unless ARGV.length == 2
  STDERR.puts "Usage: #{$0} <channel> <nick>"
  exit 1
end
$channel, $nick = ARGV

AMQP.start(:host => 'localhost') do
  $chat = MQ.topic('chat')

  # Print any messages on our channel.
  queue = MQ.queue($nick)
  queue.bind('chat', :key => $channel)
  queue.subscribe do |msg|
    if msg.index("#{$nick}:") != 0
      puts msg
    end
  end

  # Forward console input to our channel.
  module KeyboardInput
    include EM::Protocols::LineText2
    def receive_line data
      $chat.publish("#{$nick}: #{data}",
                    :routing_key => $channel)
    end
  end
  EM.open_keyboard(KeyboardInput)
end

Now, run copies in two different terminals:

ruby chat.rb channel_1 sarah
ruby chat.rb channel_1 joe

Everything you type into one terminal will be relayed to the other.

How it works

The following line creates a topic exchange named “chat”:

$chat = MQ.topic('chat')

A topic exchange allows many-to-many communication. Here, we bind a listener to our exchange, and ask to receive all messages tagged with our channel name:

queue.bind('chat', :key => $channel)

Note that :key may be hierarchical, and it may contain wildcards. To write data to our topic exchange, we use publish:

$chat.publish("#{$nick}: #{data}",
              :routing_key => $channel)

Our keyboard input is processed using EventMachine, a Ruby library for writing high-performance, multi-protocol servers. It’s very similar to Python’s Twisted library, though it has less documentation and support for fewer protocols.

We use EventMachine’s EM.open_keyboard to create a asynchronous keyboard input channel, and we use EM::Protocols::LineText2 to treat the keyboard input as a line-oriented protocol.

Adding a Shoes GUI

Shoes is an eccentric, entertaining, and highly-portable GUI library by _why the lucky stiff. With a certain amount of grotesque kludging (and some pointers from “s1kx” on the #shoes IRC channel), I managed to get the Mac version of Shoes to talk to EventMachine. You may find that this code fails strangely on your computer. Honestly, I don’t know anything about Shoes. And I’m doing some pretty bad things with threads.

First, the pretty pictures:

Next, the code:

Shoes.setup { gem 'amqp' }
require 'mq'

$app = Shoes.app(:width => 256) do
  background(gradient('#CFF', '#FFF'))
  @output = stack(:margin => 10)

  def nick str
    span(str, :stroke => red)
  end

  def display text
    @output.append do
      if text =~ /^([^:]+): (.*)$/
        para nick("#{$1}: "), $2
      else
        para text
      end
    end
  end
end

Thread.new do
  begin
    AMQP.start(:host => 'localhost') do
      MQ.topic('chat')
      queue = MQ.queue('shoes')
      queue.bind('chat')
      queue.subscribe do |msg|
        $app.display(msg)
      end
    end
  rescue => e
    # Try to report at least _some_ errors
    # where we'll be able to see them.
    $app.display(e.to_s)
  end
end

Note that the GUI client listens to all channels simultaneously, because it doesn’t pass a :key to bind. And when writing code to run in a Shoes background thread, don’t expect to see any error messages.

Learning more about AMQP

The Ruby AMQP documentation page has a good list of papers, magazine articles, and other background material on AMQP.

Financial crisis background and Munger on the banks

Eric Kidd — Tue, 05 May 2009 09:36:00 -0400

Charlie Munger is the long-time partner of Warren Buffet. Of the two, he’s the more politically conservative. Their company, Berkshire Hathaway, has recently bought big stakes in several of the better-off investment banks.

Recently, Munger sharply criticized the management of the investment banks, saying they’ve grown too politically powerful. The key quote:

“We need to remove from the investment banking and the commercial banking industries a lot of the practices and prerogatives that they have so lovingly possessed,” Munger said. “If they are too big to fail, they are too big to be allowed to be as gamey and venal as they’ve been – and as stupid as they’ve been.” (Bloomberg.com, via Baseline Scenario.)

What does the bankers’ stupidity have to do with the usual themes of this blog? Well, much of the crisis comes down to bad probability calculations: the big banks have been treating highly correlated events as though they were independent events.

Some good background material on the crisis:

A neat visual metaphor for CDOs: Filling a tower of champagne glasses.
Khan Academy videos on the financial crisis (middle column, halfway down).
A confession by the programmer who wrote the software that encouraged this whole mess.

Designing programs with RSpec and Cucumber (plus a book recomendation)

Eric Kidd — Thu, 30 Apr 2009 15:07:00 -0400

Over the last couple of years, I’ve occasionally written Ruby programs using RSpec and (more recently) Cucumber. These two tools are inspired by Test Driven Development (TDD), a school of thought which says you should write unit tests before implementing a feature.

When doing TDD, you work inwards from the interface to the implementation. You start by writing a test case against the interface you wish you had, and then you make that test case work. This is a subtle shift in how you approach a design problem, but it frequently results in beautiful APIs. (And you also get a fully automated test suite for your software, liberating you to make much larger changes without fear of breaking things.)

The problem with the word “test”

Unfortunately, the name “Test Driven Development” is misleading. Most folks think of “testing” as something you do after development is complete. But TDD is really more of a design activity—you’re specifying how your APIs should work before you actually start coding.

Dan North spent some time struggling to teach developers about TDD. After a while, he decided that the main barrier to understanding was the word “test.” He proposed replacing TDD with Behavior Driven Development (BDD), and he started referring to unit tests as “specifications.”

In the Ruby community, the most popular BDD tool is RSpec. Using RSpec, you might specify an API something like this:

describe "simplify_name" do
  it "should convert all letters to lowercase" do
    simplify_name("AbC").should == "abc"
  end

  it "should remove everything but letters and spaces" do
    simplify_name(" Joe Smith 3 -+\n").should == "joe smith"
  end
end

After writing this specification, you would then go ahead and implement simplify_name. And from then on, whenever you changed your program, you could automatically check it against this specification.

Using specifications to communicate with clients and users

By itself, RSpec is mostly useful for programmers. Sure, a specification looks a lot like English. But would you really want to show it to an end user?

Cucumber goes one step further. Instead of using code to specify how an API should work, it uses plain text to describe how a user interface should work. For example:

Feature: Log in and out
  As an administrator
  I want to restrict access to certain portions of my site
  In order to prevent users from changing the content

  Scenario: Logging in
    Given I am not logged in as an administrator
    When I go to the administrative page
    And I fill in the fields
      | Username | admin  |
      | Password | secret |
    And I press "Log in"
    Then I should be on the administrative page
    And I should see "Log out"

  Scenario: Logging out
    ...

Here’s the neat part: This specification is actually an executable program. Each line of text corresponds to a “step”, which is defined in another file. Here’s an example from the standard webrat_steps.rb file:

Then /^I should see "([^\"]*)"$/ do |text|
  response.should contain(text)
end

Cucumber encourages you to think at a very high level, and to specify how different users will actually use your software. It’s particularly helpful if you need to communicate between programmers and end-users.

My experiences with RSpec and Cucumber

I’ve been using RSpec on and off for a couple of years now, and Cucumber since late last year. Initially, I found both tools fascinating, but also a bit frustrating. Both RSpec and Cucumber have very strong opinions about how you should write software. Now, I found those opinions very interesting, and I was quite happy to be influenced by the assumptions built into the tools. But every now and then, I would need to do something that the authors of RSpec and Cucumber hadn’t anticipated, and I would inevitably wind up struggling to make things work.

But recent versions of RSpec and Cucumber are richer and more flexible. They cover more important cases straight out of the box, and they’re easier to customize. So I can finally recommend both tools for real-world projects: They’ll still guide your thinking, but they should give you enough flexibility to handle the corner-cases.

The RSpec (and Cucumber) book

Unfortunately, the documentation for RSpec and Cucumber is scattered around the web, and there aren’t enough online guides showing the best way to solve common problems.

But the Pragmatic Press is working on The RSpec Book, which contains a large section on Cucumber, and a walkthrough of a typical development session using Cucumber and RSpec.

Currently, the RSpec book is available as a “beta book”. This is a downloadable, DRM-free PDF, with periodic updates throughout the publishing process. Right now, between one-third and one-half of the chapters have been roughed in, and the book is already very useful.

So if you’re curious about RSpec and Cucumber, have a look around the two web sites, and maybe watch some of the screencasts. If you decide to investigate further, pick up the beta book and dive in.

Remote root holes reported as "denial of service"

Eric Kidd — Thu, 30 Apr 2009 12:57:00 -0400

Via LWN.

If you’re a Linux system administrator, you shouldn’t put your faith in security advisories. The kernelbof blog accuses Linux distributors of being too quick to label security bugs as “denial of service” attacks:

I’m wondering why kernel developers (or vendors?) continue to claim that kernel memory corruption are just Denial of Service. Most of the times they _are_ exploitable.

As an example, the author quotes Ubuntu Security Notice 751:

The SCTP stack did not correctly validate FORWARD-TSN packets. A remote attacker could send specially crafted SCTP traffic causing a system crash, leading to a denial of service.

(Emphasis added.)

The author claims, however, to have created an exploit for this bug. He says his exploit allows a remote attacker to gain root access, often on the first attempt. If this is true, it would give him a quick way to gain control over any Linux system which has a process listening to an SCTP socket.

Ubuntu’s security team is not doing system administrators any favors by labeling memory corruption as “denial of service” attacks. If you can corrupt memory, there are some terrifyingly clever ways to run code. And marking memory as non-executable won’t necessarily protect you.

If you administer a Linux system, you should probably aim to patch alleged “denial of service” bugs as quickly as you can.

Installing TortoiseGit

Eric Kidd — Sun, 11 Jan 2009 09:55:00 -0500

On December 12th, Frank Li released TortoiseGit 0.1. When we downloaded this initial release at work, we were underwhelmed:

git log	OK, but not as good as gitk
git commit	Broken

On January 4th, however, Frank Li released TortoiseGit 0.2. He’d been extremely busy:

git log	OK, but not as good as gitk
git commit	OK (except for `add` and `rm`)
git add	Broken (see Bug 6 for workaround)
git rm	Broken
git status	OK
git pull	OK
git push	OK
SSH	OK (tested with PuTTY)
git clone	Always clones to home directory (see Bug 8)
Clean merge	OK
Conflicted merge	Manual, as with command-line tool
Submodules	No support

Basically, TortoiseGit 0.2 is almost usable, and the project is proceeding at a breakneck pace. If you have Windows users that you want to migrate to Git—and who don’t want to use the command-line tool—it’s worth a look.

Installation instructions follow.

Ubiquitous Hoogle

Eric Kidd — Mon, 01 Sep 2008 10:33:00 -0400

Ubiquity is an experimental Firefox plugin. It’s a “graphical command line” similar to QuickSilver on the Macintosh.

You can easily add your own commands to Ubiquity. The following article shows how to create a Hoogle search command that looks up Haskell functions by name or by type signature.

You can press Return or click on one the links in the preview.

Probability monads at Hac 07 II

Eric Kidd — Tue, 02 Oct 2007 07:50:00 -0400

From October 5-7, I’ll be at the Haskell Hackathon in Freiburg.

I’ll be working on probability monads, attempting to turn my various blog articles into a real Haskell library.

Some resources:

Control.Monad.Probability source code: This is the as-yet-unreconstructed code from my blog posts.
Draft paper on probability monads: This paper explains the theory, and it has terser implementations of some ideas.

If you were a peer reviewer, or gave me feedback on the paper, my humble thanks–and apologies. I haven’t had a chance to revise the paper yet, and so your feedback is not yet included.

See you at the Hackathon!

Freiburg in October: Scheme, Dylan, and probability monads

Eric Kidd — Tue, 18 Sep 2007 12:36:00 -0400

Good morning! I’ll be in Freiburg for several events this October, including CUFP and the Haskell Hackathon.

Commercial Users of Functional Programming (CUFP)

On October 4th, I’ll be speaking at CUFP ’07, describing the use of Scheme in a real-world multimedia engine. Some likely topics:

How we switched to Scheme (and why refactoring is your friend).
How our artists learned to program in Scheme (it’s all about the tools).
The tension between functional programming and objects: Can we have both?

Dylan Beer Night

Once upon a time, I dreamt of generic functions and built RPMs for Gwydion Dylan.

Some current and former Dylan hackers are hoping to meet in Freiburg, most likely on October 4th. If you’re at ICFP or one of the workshops, we’d love to hear from you.

Haskell Hackathon

I’ll be at the Haskell Hackathon from Friday to Sunday.

Perhaps it’s time to whip Control.Monad.Probability into shape?

September 8th, 2007

Eric Kidd — Tue, 18 Sep 2007 11:15:00 -0400

Just married! Our thanks and love to everyone,
for everything.

Ruby-style metaprogramming in JavaScript (plus a port of RSpec)

Eric Kidd — Sun, 01 Jul 2007 19:00:00 -0400

Programming in Ruby makes me happy. It’s a lovable language, with a pleasantly quirky syntax and lots of expressive power.

Programming in JavaScript, on the other hand, frustrates me to no end. JavaScript could be a reasonable language, but it has all sorts of ugly corner cases, and it forces me to roll everything from scratch.

I’ve been trying to make JavaScript a bit more like Ruby. In particular, I want to support Ruby-style metaprogramming in JavaScript. This would make it possible to port over many advanced Ruby libraries.

You can check out the interactive specification, or look at some examples below. If the specification gives you any errors, please post them in the comment thread, and let me know what browser you’re running!