tags:

views:

35

answers:

2
+2  Q: 

What is the simplest way of parallelization over a cluster with SSH and NFS?

I have a lot of trivially parallelizable computations and a lot (100s) of cores distributed overs SSH + NFS network. What is the simplest way of parallelization. The problem is that I don't know how long each task will take so I need some kind of queue. Is there something that is very easy to use?

+1  A: 

Mincemeat.py is a lightweight Python implementation of MapReduce designed just for this purpose. There's not an easy way to launch all of the processes (unless you have something like Condor on the cluster), but otherwise it should fit your needs.

Michael Fairley  2010-10-25 23:13:04
+1  A: 

You may also have a look at mpssh & autossh.

http://www.mynitor.com/2010/08/16/top-50-ssh-helper-tools-omg/

telld  2010-10-26 11:57:16

related questions

Is there a better Java implementation of SSH2 than JSch?
Check out from a remote SVN repository TO a remote location?
Download files to local drive when sshed
Are there any good free .Net network libraries? (FTP, SFTP, SSH, etc.)
How do you install an ssh server on qnx?
How do I remove the passphrase for the SSH key without having to create a new key?
ssh-agent with passwords without spawning too many processes
Why does cisco IOS require domain-name to be set before SSH keys can be generated?
Setting the default ssh key location
Managing authorized_keys on a large number of hosts.
Inter-convertability of asymmetric key containers (eg: X.509, PGP, OpenSSH)
Which is the best way to bring a file from a remote host to local host over an SSH session?
Is it possible to forward ssh requests that come in over a certain port to another machine?
Can you have virtual users using an SFTP server?
Why does Vista complain about a dead process when I use Cygwin X11 ssh and how do I get it to shut up?
ssh hangs when command invoked directly, but exits cleanly when run interactive
Getting ssh to execute a command in the background on target machine
How do you use ssh in a shell script?
Avoid traffic shaping by using ssh on port 443
Alternative SSH Application to Plink
What are some good SSH Servers for windows?
Keep Remote Directory Up-to-date
Allow user to set up an SSH tunnel, but nothing else
How do I setup Public-Key Authentication?
Gtk SSH client for Linux?