Is there a good library to use for gathering user input in Linux from the mouse/keyboard/joystick that doesn't force you to create a visible window to do so? SDL lets you get user input in a reasonable way, but seems to force you to create a window, which is troublesome if you have abstracted control so the control machine doesn't have to be the same as the render machine. However, if the control and render machines are the same, this results in an ugly little SDL window on top of your display.
Edit To Clarify:
The renderer has an output window, in its normal use case, that window is full screen, except when they are both running on the same computer, just so it is possible to give the controller focus. There can actually be multiple renderers displaying a different view of the same data on different computers all controlled by the same controller, hence the total decoupling of the input from the output (Making taking advantage of the built in X11 client/server stuff for display less useable) Also, multiple controller applications for one renderer is also possible. Communication between the controllers and renderers is via sockets.