Back in August, I wrote two posts about an experimental framework for distributed functional programming called Girder. The idea, in summary, was to make distributed code look as much as possible like ordinary Clojure, as opposed to structuring it explicitly around message passing (as in the actor model) or data flows (as in map/reduce, storm, et al). As I say, it was an experiment, but it was also something a cri de coeur (a mini-manifesto, if you will) against extraneous impingements on my code. Anyway, it sounds interesting, go back and read the posts, but you don't need to …

In part 1 and part2 of this almost unbearably exciting series, I outlined the concept of distributing purely functional programs and went through some implementation details of Girder.

So far, however, I've only asserted that it works, so today I want to start explaining how I used Amazon Web Services to test pretty cheaply on gobs of machines.

The art of AWS wrangling was somewhat new to me, and I went through a few iterations before deciding on the right mix of tools. Needless to say, the right mix ended up being fairly Clojure heavy, which helps to smooth out …

Update 2015-01-12

The algorithm as it exists in HEAD is somewhat different from the below, in ways that I'll describe (eventually) in an another post. In some ways, it's closer to Fork-Join, but with important differences to support reentrancy, share results of duplicate requests and adjust for the costs of distribution.

Recap of recap

In a previous post, I introduced a framework called Girder (the code is on github), which aims to facilitate Plain Old Functional Programming on distributed systems. By POFP, I mean code that, as much as possible, consists of normal looking calls to normal looking functions, the …

Update 2015-01-12

The algorithm as it exists in HEAD is somewhat different from the below, in ways that I'll describe (eventually) in an another post. In some ways, it's closer to Fork-Join, but with important differences to support reentrancy, share results of duplicate requests and adjust for the costs of distribution.

OSS and Commercial Grids

Grid computing has always suffered the reputation of a buzzword that one suspects might not actually mean anything, but it has become especially ill-defined with the rise of open-source distributed computation frameworks like Spark, Storm and Grampa Hadoop. These extensively documented systems don't need much …