Since Maxima.exe is a native executable, the only security policies you can apply are those supported by the base Windows operating system. Unfortunately, that means your options are pretty limited: the only really useful thing you can do is run Maxima.exe in the context of a user account with limited privileges. When it comes to disallowing write access to most of your system, that should be sufficient, though.
The dated-but-still-useful article Safe Impersonation With Whidbey shows how to run a worker function in the context of another user account: you would then start Maxima.exe from that worker function.
Running as a limited user will of course still allow various Maxima.exe instances to interfere with each other to a certain degree (i.e. overwrite files created in the instance working directory). Starting each instance with a unique, randomly-created working directory (e.g. based on a GUID) may offer sufficient security for your purposes: if not, you may actually need to create distinct user accounts for each instance (or at least have a pool of user accounts for that purpose).
In the end, it all comes down to "how would I solve my security problem using Windows base-only features", and unfortunately you won't have access to the much nicer .NET security features or the facilities found on non-Windows platforms (jails, systrace, etc.).
