Dynamically generating high performance functions in clojure

Question

I'm trying to use Clojure to dynamically generate functions that can be applied to large volumes of data - i.e. a requirement is that the functions be compiled to bytecode in order to execute fast, but their specification is not known until run time.

e.g. suppose I specify functions with a simple DSL like:

(def my-spec [:add [:multiply 2 :param0] 3])

I would like to create a function compile-spec such that:

(compile-spec my-spec)

Would return a compiled function of one parameter x that returns 2x+3.

What is the best way to do this in Clojure?

Answer 1

Even if you don't AOT compile your code, as soon as you define a function it gets compiled to bytecode on the fly.

Answer 2

Hamza Yerlikaya has already made the most important point, which is that Clojure code is always compiled. I'm just adding an illustration and some information on some low-hanging fruit for your optimisation efforts.

Firstly, the above point about Clojure's code always being compiled includes closures returned by higher-order functions and functions created by calling eval on fn / fn* forms and indeed anything else that can act as a Clojure function. Thus you don't need a separate DSL to describe functions, just use higher order functions (and possibly macros):

(defn make-affine-function [a b]
  (fn [x] (+ (* a x) b)))

((make-affine-function 31 47) 5)
; => 202

Things would be more interesting if your specs were to include information about the types of parameters, as then you could be interested in writing a macro to generate code using those type hints. The simplest example I can think of would be a variant of the above:

(defmacro make-primitive-affine-function [t a b]
  (let [cast #(list (symbol (name t)) %)
        x (gensym "x")]
    `(fn [~x] (+ (* ~(cast a) ~(cast x)) ~(cast b)))))

((make-primitive-affine-function :int 31 47) 5)
; => 202

Use :int, :long, :float or :double (or the non-namespace-qualified symbols of corresponding names) as the first argument to take advantage of unboxed primitive arithmetic appropriate for your argument types. Depending on what your function's doing, this may give you a very significant performance boost.

Other types of hints are normally provided with the #^Foo bar syntax (^Foo bar does the same thing in 1.2); if you want to add them to macro-generated code, investigate the with-meta function (you'll need to merge '{:tag Foo} into the metadata of the symbols representing the formal arguments to your functions or let-introduced locals that you wish to put type hints on).

---

Oh, and in case you'd still like to know how to implement your original idea...

You can always construct the Clojure expression to define your function -- (list 'fn ['x] (a-magic-function-to-generate-some-code some-args ...)) -- and call eval on the result. That would enable you to do something like the following (it would be simpler to require that the spec includes the parameter list, but here's a version assuming arguments are to be fished out from the spec, are all called paramFOO and are to be lexicographically sorted):

(require '[clojure.walk :as walk])

(defn compile-spec [spec]
  (let [params (atom #{})]
    (walk/prewalk
     (fn [item]
       (if (and (symbol? item) (.startsWith (name item) "param"))
         (do (swap! params conj item)
             item)
         item))
     spec)
    (eval `(fn [~@(sort @params)] ~@spec))))

(def my-spec '[(+ (* 31 param0) 47)])

((compile-spec my-spec) 5)
; => 202

The vast majority of the time, there is no good reason to do things this way and it should be avoided; use higher-order functions and macros instead. However, if you're doing something like, say, evolutionary programming, then it's there, providing the ultimate flexibility -- and the result is still a compiled function.