How do I find the name of the running X window window manager from a program?
If I start another window manager when one is already running, then an error occurs, so there must be a way for the other window manager to detect the first one.
How does it work?
You may find an "atom" that has the information, but I don't think there is a 100% guarantee that all window managers use the same atom. Do an "xlsatoms" to list the atoms on your server, or "xprop" (and click) to see the properties (including atoms and their values) of a particular window.
By the EWMH spec, a compliant window manager will set the _NET_SUPPORTING_WM_CHECK property on the root window to a window ID.
If the _NET_SUPPORTING_WM_CHECK property exists and contains the ID of an existing window, then a ICCCM2.0-compliant window manager is running. If the property exists but does not contain the ID of an existing window, then a ICCCM2.0-compliant window manager exited without proper cleanup. If the property does not exist, then no ICCCM2.0-compliant window manager is running.
That window (not the root window, but the one described by a property on the root window) should have a _NET_WM_NAME property on it, which is what you are looking for.