Is there a good tool out there to do this on a Linix machine using the bash shell? All I need is to issue different commands on a set of nodes in a cluster and when one of them is done with the job, I'd like to submit another one. Something very similar to what Hadoop can do. I would be interested in knowing the status of the job as well but even otherwise is fine. Any suggestions?
+1
A:
Do you just need a batch job queuing system? It might be overkill for you needs, but I'm fond of Platform LSF (if you're allergic to commercial solutions try the open source version, LAVA). Keep PBS/OpenPBS in mind, too.
Mark Johnson
2010-09-30 21:10:24
@Mark: Yeap. That is what I am looking for. Thanks for the LAVA link. I will check that out now.
Legend
2010-09-30 21:25:56
@Mark: I was looking into these but am not able to figure out if I can use them without root privileges. Will keep looking though. Thanks
Legend
2010-09-30 22:48:25
+1
A:
Programs like Sun Grid Engine and Mosix will let you submit all of your jobs at once and will automatically load balance them (meaning it will queue them up so that only the right number are running on a given node at a time).
I've had good experiences with these in scientific computing (on Linux with bash) and highly recommend either.
starghost
2010-09-30 21:21:57
@starghost: Oh... might seem a little odd but can I use them without root privileges? I mean install that on a per user basis?
Legend
2010-09-30 21:25:29
That's a good question-- not that I know of. But once you get them on, they're really nice to have.
starghost
2010-09-30 22:36:55
@starghost: I see... +1 for that... Thank You. I will open a support ticket with out admin... In the mean time, I really would love something that is a little portable in the sense that I can install and play around with...
Legend
2010-09-30 22:47:49
A:
You could also use mounted NFS/SMB partition and write some scripts yourself.
anttir
2010-10-07 13:58:49