Yes you can do this without a window manager.
first: You need to boot into a non X session, init level 1 or 3.
second: You need to start the X server, in a basic mode by just calling X or xinit.
third: Start your app. You may need to take a little more control over where your app is positioned on the screen and its dimensions in your code, as you will not have a window manager to help with this.
alternatively: you can launch one of the more basic window managers to see how they play with your system. Motif Window Manager (mwm) and Tab Window Manager (twm)
Note: While working without a window manager, you may get into a state where you cannot do some very basic operations (ex: close a window, move a window, resize a window). You may find that you cannot survive without at least some of the more basic window management functions. Until you close the loop on this, remember, Ctrl + Alt + Backspace will kill the XServer.