Is it possible to execute an exe file that is included in the project as a resource? Can I fetch the file as a byte array and execute it in memory?
I don't want to write the file to a temporary location and execute it there. I'm searching for a solution where I can execute it in memory. (It's not a .NET assembly.)
Its not very easy to create a new process from a memory image, all of the kernel functions are geared towards loading an image from disk. See the process section of the Windows NT/2000 native API reference for more information - page 161 has an example of manually forking a process.
If it is ok to run the code from within your own process then you can create a small DLL that will take a pointer to executable data and run it.
A much better way is to create a temporary DLL file with FILE_FLAG_DELETE_ON_CLOSE attribute. This way the file will be deleted automatically when it is no longer used.
I don't think there is a way to load DLL from a memory (rather than a file).
It's quite possible - I've done it myself - but it's fiddly and more so from managed code. There's no .NET API for it, nor is there a native API for it which you can PInvoke. So you'll have to fenagle the load by hand, which will require some knowledge of the PE (Portable Executable) file format used for modules such as DLLs and EXEs - http://msdn.microsoft.com/en-us/magazine/cc301805.aspx. There'll be a lot of pointer manipulation (mandating use of unsafe {} blocks) and PInvoke.
First load the PE file into memory (or use MapViewOfFile). A PE file is internally made up of different sections containing code, data or resources. The offsets of each section in the file don't always match intended in-memory offsets, so some minor adjustments are required.
Every PE file assumes it'll be loaded at a certain base address in virtual memory. Unless you can ensure this you'll need to walk the PE file's relocation table to adjust pointers accordingly.
Each PE file also has an import table listing which other DLLs' functions it wants to call. You'll need to walk this table and call LoadLibrary() / GetProcAddress() to fill in each import.
Next, memory protection needs to be set correctly for each section. Each section's header notes the protection it wants, so it's just a matter of calling VirtualProtect() for each section with the correct flags. At a minimum you'll need to VirtualProtect the loaded module with PAGE_EXECUTE_READWRITE or you're unlikely to be able to execute any code.
Lastly for a DLL you need to call its entry point, whose address can be found in the PE header; you can then freely call exported functions.
Since you want to run an EXE, you've got some additional headaches. You can just spin up a new thread and call the EXE's entry point from it, but many EXE's may get upset since the process is set up for you, not the EXE. It also may well kill your process when it tries to exit. You might want to spawn a new process therefore - perhaps another copy of your main EXE with special arguments to tell it it's going to run some different code - in which case you'd have to fenagle the EXE into its memory space. You'd probably want to do most of the above work in the new process, not the old. You could either create a named pipe and send the data across from one EXE to the other, or allocate a named shared memory area with MapViewOfFile. Of course the EXE may still get upset since the process its running in still isn't its own.
All in all its far easier just to write to a temporary file and then use Process.Start().
If you still want to do it the hard way, take a look at this example in unmanaged code: http://www.joachim-bauch.de/tutorials/load_dll_memory.html. This doesn't cover executables, just DLLs, but if the code therein doesn't scare you you'd be fine extending the process to cover executables.