Catching parse errors on the client might be possible, but it's really solving the wrong problem.
I know this isn't what you asked for, but unless you're doing something truly XHTML-specific like embedding some other markup language, you should serve your page as text/html instead of application/xhtml+xml. Even if it's XHTML. By serving it as text/html you'll avoid the problem you're running into and allow your page to work in IE as well. Note that it's the MIME type and not the doctype declaration that determines which parser is used -- using a transitional doctype won't do it.
That said, if you're really sure you want your page parsed as XHTML, it's better to handle this kind of error on the server. Generate your page by building up a DOM and then send the result of serializing it. If that's not an option, then start by generating the page as you do now but don't transmit it to the client yet. Take the XHTML that you've generated and parse it server-side with a validating XHTML parser (or at the very least, a generic XML parser). If you get errors, display whatever error page you want. Otherwise, serialize the parsed DOM and send that to the client.
In summary, the basic rules for using application/xhtml+xml are:
- Don't.
- (For advanced users) Don't, unless you've proven that you're doing something that won't work if the page is served as text/html. This applies to a tiny, tiny fraction of a percent of XHTMl documents.
- If you must serve your page as application/xhtml+xml, generate it with some method that guarantees validity.
- Unless you really know what you're doing, never use application/xhtml+xml for a page that includes user input.
Remember that XHTML is just a reformulation of HTML 4 plus the ability to embed other languages. If you don't use the embedding, what you have is HTML 4 with a different but almost completely compatible syntax. The overwhelming majority of XHTML documents out there are served as text/html and thus treated like HTML 4 by browsers.