Android absolutely supports pipes as well as unix domain sockets. Use of exec is somewhat discouraged, but works at the moment.
See the source of any android terminal emulator with a local shell option for an example of how to do it. Essentially your gui just replaces the terminal emulator, and your engine replaces the shell.
If using exec becomes a problem in the future, you will need to compile your engine as a jni library rather than a stand alone executable. That's not necessarily too hard - just graft it onto the ndk hello-jni example, and have a single jni function that calls main(). Call this from a java thread. Communicate with pipes as before, or rig up some other message passing scheme using jni.
Note that the "use a service" answers are going to require a java wrapper in the service too. As of the moment, you can't make a pure-native service using any supported/endorsed mechanism unless you are the platform vendor making and registering a system service.
Also be aware that your engine will need to be able to save any state, and do so at the same time the java side would need to under the android activity lifecycle (essentially, once you get paused you become killable without further notice)