Is there an x86 or x64 emulator that passes system calls back to the Windows API?

Question

I want to emulate windows programs (not VM, true emulation) under windows. This would require the emulator to make calls back to the system APIs, but the program itself would be emulated. The reason is I want to change the opcode formats for research purposes.

The process should be:

1. Take existing program.

2. Disassemble and then reassemble with my new opcode formats.

3. Put the new format into the PE with a stub calling the emulator and passing the new code.

4. The emulator would have to pass system calls from the emulated side back to windows API calls.

I can do all these steps, except I need an open source emulator with the ability to pass the API calls out. I could try Bochs or QEMU, but I think I'd have to add in the system calls, which I could do if needed. I wonder if there is already something closer to what I need.

I know I would have to change the instruction decoding in the emulator to match my new formats, but that is a given.

Thanks.