Yak herding for misers - wrangling hundreds of AWS Instances with Clojure

yak herding

Back in August, I wrote two posts about an experimental framework for distributed functional programming called Girder. The idea, in summary, was to make distributed code look as much as possible like ordinary Clojure, as opposed to structuring it explicitly around message passing (as in the actor model) or data flows (as in map/reduce, storm, et al). As I say, it was an experiment, but it was also something a cri de coeur (a mini-manifesto, if you will) against extraneous impingements on my code. Anyway, it sounds interesting, go back and read the posts, but you don't need ...

more ...

vanholes - Van Laarhoven Lenses in Clojure

In two previous posts, I went on about lenses in Clojure. Pinholes comprised a small library of higher order functions to formalize and simplify the viewing and manipulation of complex nested structures. Tinholes did essentially the same thing, but with macros instead. In both cases, there's recursion going on, as we burrow through layers of nesting, but macros had the advantage of doing it all before compilation, giving core.typed a chance to check our work.

The macro post was inexplicably popular, catapulting me to levels of fame I never expected to achieve without consciously emulating an early De ...

more ...

tinholes - performant, strongly typed lenses in Clojure

In a previous post, I built up a framework for lens-like constructs in Clojure: essentially some fancified versions of assoc-in and get-in to allow for bidirectional transformations along the nesting path and some utilities to generate special-purpose getter/setter functions. The name, "pinhole," is supposed to suggest a more primitive, utilitarian mechanism for achieving focus.

While still ruing (sort of) other mistakes, I found myself worrying that a triumphal sentence near the end of the piece

What's more, thanks to the expressive power of dynamic Clojure,
and higher order functions, these lenses are not just simple to
use but ...
more ...

Pinholes 2 - Ignorance, misrepresentation and prejudice edition

Sseveral comments regarding the pinholes post, have forced me, against the deepest elements of my nature, to engage in thought. Since that might never happen again, I thought it meet to record the event.

I'm going to say "I" a lot, because this is mostly my opinions.

Bidirectional programming

As pointed out by Christian Schuhegger in a comment on the original post, lenses were originally introduced to computer science in the context of bidirectional programming, rather than as a tool for dealing with deeply nested immutable structures. He points to a good list of papers on the subject on ...

more ...

Pinholes - Idiomatic Clojure Lenses

Lenses are a construct for getting, "setting" or "modifying" values within data structures, especially deeply nested data structures. The quotation marks have the usual meaning when they show up in funktionsprache:1 not mutating anything per se, but instead producing an object, or reference thereto, that is identical except for the requested change.

In Scala, the need for lenses is pretty glaring, as illustrated in Eugene Yokota's great explanation of lenses in Scalaz, because of the centrality of case classes. In his example problem, a turtle is represented with three case classes:

  case class Point(x: Double, y: Double ...
more ...

'Ducers Wild -- a concise guide to the menagerie

TL;DR

It's not too long, but, to summarize the summary, if you read Rich Hickey's 2014 blog post on transducers first, his 2012 post on reducers will be easier to understand.

Brief Definitions

Herewith, all in one place, are Clojuresque definitions of:

  • reducible
  • reducing function
  • transducer
  • reducer
  • folder
  • decomplected

Longer elaborations of these definitions follow in the subsequent section.

reducing function

Anything that can be used as the first argument of the reduce function, e.g. + or conj. Generally, it's a binary function that returns something of the type of its first argument, which is supposed ...

more ...

Testing on Hundreds of AWS Instances - (sort of Distributed functional programming, part 3a)

In part 1 and part2 of this almost unbearably exciting series, I outlined the concept of distributing purely functional programs and went through some implementation details of Girder.

So far, however, I've only asserted that it works, so today I want to start explaining how I used Amazon Web Services to test pretty cheaply on gobs of machines.

The art of AWS wrangling was somewhat new to me, and I went through a few iterations before deciding on the right mix of tools. Needless to say, the right mix ended up being fairly Clojure heavy, which helps to smooth ...

more ...

Comedy - A brief syllabus in honor of Joan Rivers

I have a confession to make. I don't find comedy funny. There are exceptions - Gilbert Gottfried and Louis CK can occasionally reduce me to putty - but, by and large, I appreciate comedy from a technical rather than aesthetic standpoint, combined with an admiration for and empathy with the drive that compels people to perfect their craft.

I have similar feelings about magicians. Having spent a significant portion of my youth reading about and reverse-engineering illusions, before concluding that certain qualities necessary for success in the field are innate rather than learned, and I would always lack them. For example ...

more ...

Distributed purely functional programming, part 2

In a previous post, I introduced a framework called Girder (the code is on github), which aims to facilitate Plain Old Functional Programming on distributed systems. By POFP, I mean code that, as much as possible, consists of normal looking calls to normal looking functions, the only requirement for which is referential transparency: there are no side effects, and calls to the same function with the same arguments should always return the same value.

Functional programming, in my view, is dramatically less functional when you use a framework that requires you to cast your algorithm in terms of message passing ...

more ...

Quick tip on getting lein to recognize your new dependency

Ever realized you needed to modify some dependency (e.g. I needed to add weak cache support to core.cache), go a head with forking on github, lein install your new version, modify your project.clj to pick it up, and then... somehow... you still seem to get the old version? That's usually because some other dependency is explicitly requesting a version of that library. If you do lein classpath, you'll see your library show up, in two different versions, with the one you don't want first.

In my case, the relevant bit of classpath shows

   /Users ...
more ...