tags:

views:

91

answers:

1
+5  Q: 

Monitoring Progress/Debugging Parallel R Scripts

Among the choices I have for quickly parallelizing simple code (snowfall, foreach, and so on), what are my options for showing the progress of all the slave processes? Do any of the offerings excel in this regard?

I've seen that snowfall 1.70 has sfCat(), but it doesn't seem to cat output to the master R session.

+3  A: 

That's where it can turn into black art... I notice that you did not list MPI or PVM -- those old workhorses of parallel computing do have monitors. You may find solutions by going outside of R and relying on job schedulers (slurm, torque, ...)

If you can't do that (and hey, there are reasons why we like the simplicity of snow, foreach, ...) then maybe you can alter your jobs to log a 'heartbeat' or progress message every N steps. You can log to text files (if you have a NFS or SMB/CIFS share), log to a database, or heck, send a tweet with R. It will most likely be specific to your app, and yes, it will have some cost.

Dirk Eddelbuettel  2010-01-27 16:24:57

related questions

How do I append a newline character for all lines except the last one?
How do I use a foreach loop in Java to loop through the values in a HashMap?
Foreach Loop Microsoft SSIS - equivalent to break statement
Why can't I .add a Dictionary(Key, Value) in a foreach?
C Foreach or Similar
How do foreach loops work in C#?
Why foreach statements accept objects that implement the 'Collection' pattern instead of accept only objects that implement IEnumerable?
How do you find the last loop in a For Each (VB.NET)?
How to use foreach keyword on custom Objects in C#
When to use FOR-CASE (Foreach/switch in C#)?
C#, For Loops, and speed test... Exact same loop faster second time around?
Changing item in foreach thru method.
Can you enumerate a collection in C# out of order?
How to Convert all strings in List<string> to lower case using LINQ?
Does foreach always create a copy on a none reference in PHP?
Why does my attempt to trim strings in a List<string> not appear to work?
LINQ equivalent of foreach for IEnumerable<T>
Visual c++ "for each" portability
Is the order of objects returned by FOREACH stable?
How to find the foreach index
Does Class need to implement IEnumerable to use Foreach
How can I enumerate all *.exes and the details about each?
What's the fastest way to copy the values and keys from one dictionary into another in C#?
foreach access the index or an associative array
C#: Can I have a method return IEnumerator<T> and use it in a foreach loop?