So i have a native executable (both for OS X and Windows) tucked inside a .jar, which is then programatically extracted to a temp location and executed from there. Unfortunately this executable will not run properly on my Windows machine (Win 7 64bit), when i manually set the compatibility mode to a Windows XP preset however, it will work.
Is there a way i can do this programatically from Java?
I believe there is a way to do this by changing the registry, so you will have to use Java to programmatically change some registry keys. This is not an ideal solution of course and definitely not a cross-platform one either.
Here's the link on how to achieve that.
Also, you'll want a Java package that utilizes the Windows API, a quick Google search reveals this. But I'm sure there are more ways to access the registry.
I'd think that you'd really want to fix the incompatibility in your native exe. Compatibility mode is really for apps where the user just doesn't have a choice. I can't tell you how many hours I've wasted debugging goofy behavior caused by the user accidentally turning compatibility mode on - best to avoid it if you have any choice in the matter.