Like Firefox, XULRunner only ships with support for ogg (and soon, webm) in the HTML5 video tag. Is there a relatively simple way to add h.264 support to it for all three major platforms? Perhaps a compilation flag, or a plugin I can add to it?
There is no easy way to do this. You could probably do it with an add-on, but it'd be non-trivial.
In short, not really. Firefox, unlike Opera or Epiphany, but like Chrome, builds the codec directly into the browser. The ‘glue’ needed to support the codec is specific to the codec and to the browser. You would need to write your own C++ code to support a new codec in the browser.
If you’re deadset on doing this, then you’re probably going to have to hack on code. And if you’re going to be hacking, the best approach would be to support GStreamer rather than a specific codec. Adding GStreamer support would enable you to agnostically support any codec installed in the system, including H.264.
Bug 422540 (GStreamer backend for HTML5 video element) on bugzilla.mozilla.org already contains some fairly recent patches for Firefox to add GStreamer support. I can’t vouch as to whether they would apply cleanly or not, but I think that is a good starting point for you to work off (and if successful, would lead to you being able to play H.264).
This blog post contains a link to a precompiled Firefox with GStreamer support (albeit heavily outdated — see bug 422540 for more recent patches).
Hope this helps!