best (python) setup for cpu / memory intensive task

Question

i'm doing simulation which generates thousands of result objects. Each object size is around 1mb, and all the result objects should be on memory to be queried for various ad hoc reports. And it takes 1~2 secs to make one result object.

So it takes more than 5 minutes to get one simulation done even though i fully use my quad-core cpu with parallel execution. And the task process takes more than 4~5 gb memory for one simulation set. The problem is, I want to run more simulation sets simultaneously and get it done more quickly.

Currently, I'm doing this job using c# and ironpython on windows vista64, quad-core cpu with 8g memory. I gonna order a new computer, 24 gb memory with better cpu and ultimately, i may buy workstation with multi cpus and more memories.

So my question is, what is the best way to utilize new hardware? I consider one of the combinations below.

1. ironpython + c# on windows 64
2. ironpython + c# (mono) on linux 64
3. jython + java on windows 64
4. jython + java on linux 64

Simulation engine is written in c# / java, and i use python to make reports. Which combination do you guys think is the best?

Is there no big difference between .net and java platform to handle memory consuming task? Is there no difference between windows and linux?

I sometimes run my current c# + ironpython code on my ubuntu laptop (32bit, 2g ram) and feel that it seems pretty stable compared to windows .net env on the same spec hardware. But i dont know when the underlying hardware is pretty better.

And i welcome any kind of suggestion regardless of the choices above.

Answer 1

Since you can install all of those for free and it sounds like you already have the code implemented in both .Net and Java then I suggest you benchmark the program on all four platforms (windows/linux * java/.net).

It sounds like all the heavy lifting is done in Java/C#, so I suspect the relative performance of Jython vs. IronPython is largely irrelevant.

Answer 2

@Dave is spot on, if you really care benchmark each combination and see.

Personally I'd suggest you stick with the tool set that you are most comfortable with, be that Windows, Java, Linux, .Net or any random combination there of. Your level of productivity in maintaining and developing your software usually trumps any minor performance gain you might get from switching OS or VM.