Better alternative to pmap in Clojure for parellizing moderately inexpensive functions over big data?

views:

631

answers:

+9 Q:

Better alternative to pmap in Clojure for parellizing moderately inexpensive functions over big data?

You can use some sort of map/reduce implemented by hand. Also take a look at swarmiji framework.

"A distributed computing system that helps writing and running Clojure code in parallel - across cores and processors"

edbond 2010-01-20 19:26:09

swarmiji if a library for distributed computing in Clojure. I got the impression this quiestion was focusing more on single-system-parallel-exacution.

Arthur Ulfeldt 2010-01-20 20:12:23

+7 A:

This question: how-to-efficiently-apply-a-medium-weight-function-in-parallel also addresses this problem in a very similar context.

The current best answer is to use partition to break it into chunks. then pmap a map function onto each chunk. then recombine the results. map-reduce-style.

Arthur Ulfeldt 2010-01-20 20:09:39

Would I really want to use `pmap` on each chunk? I think that will still create a future per seq item. It would make more sense to me to just `map` a `future` onto each chuck.

Alex Stoddard 2010-01-20 21:26:34

the idea is to increase the chunk size so that it beats the coordination overhead while still filling all the cores. Not all data sets have a sweet spot like this.

Arthur Ulfeldt 2010-01-21 00:35:24

Ah-ha. I needed to think at a level of one additional abstraction. I `pmap` a function over the chunks and that function will `map` my processing function over each member of the chunk. Is that what you mean?

Alex Stoddard 2010-01-21 13:42:13

That is the map-reduce style of processing. Split your input to chunks, fire jobs in parallel on each chunk and then join the results. http://en.wikipedia.org/wiki/MapReduce . The only question is the size and the number of chunks.

edbond 2010-01-21 14:53:37

@Alex Stoddard, thats it. Im fighting back the urge to say that all problems can be solved by one more level of abstraction.

Arthur Ulfeldt 2010-01-21 19:39:16

One has to be careful not to (silently!) skip some of the inputs with `partition` due to the fact that it never produces chunks smaller then specified. E.g. `(partition 5 [1 2])` evaluates to en empty lazy seq! `clojure.contrib.seq-utils/partition-all` (soon to be `clojure.contrib.seq/partition-all`) puts together a short final chunk instead (`((1 2))` with arguments as above).

Michał Marczyk 2010-02-01 22:08:56

(partition 5 5 '() [1 2]) will leave the small-sized chunk on the end and not drop anything.

Arthur Ulfeldt 2010-09-14 21:01:45

+4 A:

Sadly not a valid answer yet, but something to watch for in the future is Rich's work with the fork/join library coming in Java 7. If you look at his Par branch on github he's done some work with it, and last I had seen the early returns were amazing.

Example of Rich trying it out.

http://paste.lisp.org/display/84027

Runevault 2010-01-21 04:13:17

Actually I have discovered this can be tried now with Java6, the Clojure "par" branch from github and the jsr166y.jar file that Rich Hickey made available at: http://cloud.github.com/downloads/richhickey/clojure/jsr166y.jar

Alex Stoddard 2010-01-21 19:57:42

Ohhh really? May have to give that a look, as Par looks amazing. Thanks for the tip, as I missed this.

Runevault 2010-01-22 11:59:42

ansaurus

tags:

views:

answers:

Better alternative to pmap in Clojure for parellizing moderately inexpensive functions over big data?

related questions