Technically you can't prevent anything, as WMs can do whatever they want, but most reasonable window managers will let you control this.
The preferred modern way to do it is to set _NET_WM_WINDOW_TYPE semantic type if any of those are applicable. For example, in many WMs a dialog type may imply not-maximizable.
http://standards.freedesktop.org/wm-spec/1.3/
It sounds like none of these apply to your app though probably, so you'll have to set the specific hints.
To avoid maximization you just want to make the window not-resizable. As you've discovered, "fighting" the resize by just resizing back is a Bad Idea. It has infinite loop potential among other things.
XSetWMSizeHints() is the correct way to avoid maximization. Set min size = max size. voila, not resizable.
To avoid minimization, you have to use a bit of old legacy cruft called the Mwm hints. Unfortunately this involves cut-and-pasting a struct definition and then setting a property to the bits of the struct.
I just googled for MWM hints docs, and one of the results is me suggesting documenting them, 9 years ago ;-)
http://mail.gnome.org/archives/wm-spec-list/2001-December/msg00044.html
Unfortunately, none of the results are actual docs.
You can likely figure it out from http://git.gnome.org/browse/gtk+/tree/gdk/x11/MwmUtil.h and gdk_window_set_mwm_hints() http://git.gnome.org/browse/gtk+/tree/gdk/x11/gdkwindow-x11.c#n4389
MwmUtil.h is the struct that's cut-and-pasted everywhere (into most WMs and toolkits).
The _NET_WM_ALLOWED_ACTIONS hint is set on your window by the WM indicating which features the WM has decided to put on the window. The main purpose of this hint is that pagers and task lists and other desktop components can then offer the matching actions for the window.
The specs that cover all this are the ICCCM (old spec, still mostly valid) and EMWH (new extensions and clarifications, since ICCCM left lots of things unaddressed).
For gory details, try the source code... for example recalc_window_features() in metacity's window.c file, currently on line 6185 http://git.gnome.org/browse/metacity/tree/src/core/window.c#n6185
A philosophical adjustment when coding for X: mileage will vary with window manager. The "mainstream" ones lots of people use generally will follow the specs and work about as you'd expect. However, there are all kinds of WMs out there, some broken, others deliberately quirky. The worst thing you can do is try to "fight" or work around the WM, because basically all ways of doing that will end up breaking the app when running with a sane WM. Your best bet is make things follow the specs, work with the normal WMs, and if you get users upset that they can resize your not-resizable window because their WM allows that, you just have to tell them to complain to whoever provides that WM. The whole point of the pluggable WM design is that the WM determines some of this behavior, rather than the app.
Good luck. Modern X is pretty complex and coding Xlib with no toolkit is kind of asking for things to be... not quite right. But you can probably get it going well enough. :-P