My idea: allow this exception to be thrown. Use Application_Error handler to write code, that redirects (using Response.Redirect - this is important, since this gives users’ browser ability to go back) user to a custom error page. On this page write some text explaining that users had incorrectly input some text. Something like:
"Dear user, you have entered some invalid text, like “<” or “.”. Please, enter text using only characters and numbers".
Put a link on that page, and this link can contain a javascript "back" command:
href="javascript: history.go(-1)"
Users after clicking suchlink will be redirected by their browsers to the previous page, where they can re-edit their input.