I have a simple but unanswered question.
Do I have to clean up all DisplayLists, Textures, (Geometry-)Shaders and so on by hand via the glDelete* functions, or does the GPU mem get freed automagically when my Program exits/crashes?
Note: GPU mem refers to dedicated memory on a dedicated Graphics card, not CPU memory.
When your program exits (or crashes) then any memory it currently has allocated should be freed eventually in the same way that main memory is usually freed when a program exits. It may be some time before the GPU "realises" that the memory is available for use again.
However, you shouldn't rely on this behaviour as it may depend on how the graphics card drivers have been implemented. It's far better to make explicit clean up calls when you (as programmer) know that you won't need that memory again.
All your GPU resources will be released when your program exits. An easy way to test is to not delete things, and run your app repeatedly to see if it fails the allocations after a few iterations.
As others mentioned, your OS should release the resources. That's what OSes are for. It's worth noting that this has nothing to do with OpenGL, but is something that is part of the charter of well behaved OSes. The OS is there to handle all the system resources. OpenGL ones are just but a subset of them, and they are no different from, say a file handle. Now to get more concrete, you should specify which OS you care about.
BTW, This is where I take exception with ChrisF's answer. It should not be up to the driver to handle this. OS driver models will have a clear interface between the user-mode OpenGL driver (that shouldn't do actual gfx resource allocation, since it's shared in the machine), the OS (that provides the equivalent of system calls to allocate resources) and the kernel-mode driver (that is merely there to execute the OS orders in a way that is compatible with the gpu). This is at least the case with the WIN2K and WDDM models.
So... if your process crashes or otherwise terminates, in those models, it's the OS responsibility to call the kernel-mode driver to free all the resources that were associated with the process.
Now, whether you should or not is really something that is a little like asking tabs-or-spaces in source code. Different people have different beliefs here. "the OS will do it anyways, quitting immediately is a better end-user experience" vs "I want to know whether I'm leaking memory because if my program is long-running, I really don't want it to hit OOM errors. Best way to do that is to be leak-free throughout" are the 2 main lines of thought I'm aware of.
Free the context, everything else is local to the context (unless you enabled display list sharing) and will go away along with it.