But I can't seem to find much about how the NX protocol actually works. I have heard it does something with sending X11 commands. But does this mean that the listening clients need to have an x server to run the actual commands and display them?
Basically, I am trying to figure out if it is possible to write an NX client for a web browser, because it sounds interesting to me. Thoughts?
Yes. NX is essentially compressed X-Window protocol.
It's not a spec, but here is a general introduction to how it works: http://www.nomachine.com/documents/NX-XProtocolCompression.php
The client doesn't need to be an X-server, but it will probably need to be able to handle at least some subset of the X protocol.
If you are going to create an web based NX client, make sure you look at noVNC which is a web based VNC/RFB client. Better yet, fork noVNC and add NX support. That way you don't have to waste time on input, events positioning, networking, etc.
Disclaimer: I am the creator of noVNC. Implementing other remote desktop protocols (NX, RDP, Spice) is on my long term todo list (part of the reason for the name). If you're serious, contact me via github and I can give you some direction/thoughts and put you in touch with somebody else who has also expressed interest.