How can I spawn a process on Windows and see which files it uses?

A second twist: if the process tries to open a file, I would like to be able to make it wait until I can create that file—and only then let it resume and open the file

You just put yourself into Hack City with that requirement - you're right that ETW would've been a far easier solution, but it also has no way to block the file call.

Basically, here's what you're going to have to do:

Create the process suspended
Create two named pipes in opposite directions whose names are well known (perhaps containing the PID of the process)
Hook LoadModule, and the hook will watch for Kernel32 to get loaded
When Kernel32 gets loaded, hook CreateFileW and CreateFileA - also hook CreateProcessEx and ShellExecute
When your CreateFile hook hits, you write the name to one of the named pipes, then execute a ReadFile on the other one until the parent process signals you to continue.
When your CreateProcessEx hook hits, you get to do the same process all over again from inside the current process (remember that you can't have the parent process do the CreateProcess'ing because it'll mess up inherited handles).
Start the child process.

Keep in mind that you'll be injecting code and making fixups to an in-memory image that may be a different bitness than yours (i.e. your app is 64-bit, but it's starting a 32-bit process), so you'll have to have both x86 and amd64 versions of your shim code to inject. I hope by writing this lengthy diatribe you have convinced yourself that this is actually an awful idea that is very difficult to get right and that people who hook Win32 functions make Windows OS developers sad.

LOL --- yes, I probably will make Windows OS developers sad. This looks like a "promising" approach --- thanks very much! I'll wait a bit for other ideas.

Jim Blandy 2010-09-09 16:14:38

I can't emphasize enough - *awful idea*. What's your scenario that you want to do this?

Paul Betts 2010-09-09 19:34:21

Have you seen <a href="http://www.cs.berkeley.edu/~billm/memoize.html">memoize</a>? It's only the core of an idea (which he's mistaken for a full solution), but I think it's an awesome core. With a nicely Pythonic set of classes for doing common build operations, support for parallel builds (which I think I know how to make work), and a persuasive demo, I think it could take off.

Jim Blandy 2010-09-10 15:27:05

If I can't make it support Linux, OSX, and Windows, then the project becomes much less interesting to me.

Jim Blandy 2010-09-10 15:30:11

So why do you need to block the file operation here? The original project doesn't do that, it just uses strace - you could get identical information from ETW

Paul Betts 2010-09-10 16:22:38

Right. The question is, how can one run compilations in parallel in such a build system? One approach is to let a 'spawn' operation in the build script start the subprocess, but return immediately, even as the spawned program begins to run, until we've got the number of processes we wanted running. The catch is, what happens when an earlier program will produce an output file needed by a later program? We can't let the later program start until the earlier program has finished writing the file. But we don't necessarily know that the later program needs the file until we've started running it.

Jim Blandy 2010-09-12 02:41:20

There are messy details here, such as the fact that we can't tell that one program will write a given file until after the other program has tried to open it. I think heuristics may be acceptable here, since we can at least correctly detect post-facto that an anomaly has occurred.

Jim Blandy 2010-09-12 02:44:07

Run the build once in 'instrumentation mode' where everything is run one piece at a time - then you know the dependency order and can run it in 'parallel mode'

Paul Betts 2010-09-12 19:18:09

Well, when a compilation needs to be re-run, you can't tell what files it will read this time --- header files may have changed, say, to now #include a generated file. So data from past runs isn't a reliable predictor of future behavior. Talking more with friends locally, my current favorite solution is to require the script author to tell the build system what commands can run in parallel. The data provided by ETW would be sufficient to detect and warn about cases where the final result might have been dependent on the specific order of execution. So, no hacks necessary.

Jim Blandy 2010-09-14 17:54:29

ansaurus

tags:

views:

answers:

How can I spawn a process on Windows and see which files it uses?

related questions