views:

181

answers:

4

We have a command line exe that takes input from a text file and produces an output text file. It is used for complex industrial simulations.

The source code for this exe is long gone. Now it was easy enough to create a .NET wrapper which controls the execution of this exe and links in with an external app via a web service.

Unfortunetely a new requirement is to run optimization over this black box model. Now there are various methods to perform black box optimization but they all require calling the executable thousands (millions?) of times. Its obvious that the creation and parsing of disk based text files is the bottleneck of the simulation process.

Is there anyway I can trick this executable into not writing to a physical disk? If we were on Unix I suppose pipes would do the trick, but our deployment server is Windows Server 03.

It just occurred to me that a ramdrive might solve this problem, but I havent played with one of those since MS-DOS 6. Any commercial products worth looking at? Does anyone have any other ideas for emulating a physical drive through code? We are on .NET 3.5.

A: 

Pipeing output is possible in windows as well - if the executable generates output only to standard out, you can use that.

If it actually writes to a file: since windows caches filesystem writes, I could imagine that simply writing, then deleting small files is almost as fast as a ramdisk. Have you actually tried running the program at a realistic rate, deleting all output post-run? If you keep an eye on CPU usage and the disk queue you should get an idea if plain old disk caching suffices.

Eamon Nerbonne
Can you please expand on how this might work?
Alex
The same way it does on unix: try `dir | more` for example. Basically, if your program is coded to write directly to disk, this isn't an option.
Eamon Nerbonne
Unfortunetely this is the case. If we had the source code we wouldnt be writing to disk at all.
Alex
A: 

If you're running on Vista, there's a commercial Ramdisk product that may suit. The x64 version may be needed if your system is already using most of its memory, to make sure you don't end up doing too much page swapping.

Another option is to spend a bit of cash on a 15000 RPM disk or a SSD (solid state disk), although that'll be slower than a RAM disk.

In the long term though it may be cheaper to reverse engineer the processing tool and rewrite it from scratch to avoid the bottlenecks.

devstuff
We are running within a corporate SOE. Accessing or modifying hardware is not an option. Our app also has to be compatible with VMs for testing purposes. Regarding the reverse engineering - blackbox optimization is part of that process.
Alex
A: 

If you run it in a virtual machine, caching for its virtual disks is the responsibility of the host OS... which means, you can run your Windows server inside a VM on more or less anything, and get an extra layer of caching. Does that work in the environment?

Andrew McGregor
Interesting approach, but we only have access to the VM. The host OS is out of bounds.
Alex
I should have originally mentioned that our deployment server is already a VM.
Alex