What causes java.io.CharConversionException with EOF or isHexDigit messages in Tomcat?

Question

This exception peppers our production catalina logs on a simple 'getParameter()' call.

WARNING: Parameters: Character decoding failed. Parameter skipped.

java.io.CharConversionException: EOF
    at org.apache.tomcat.util.buf.UDecoder.convert(UDecoder.java:82)
    at org.apache.tomcat.util.buf.UDecoder.convert(UDecoder.java:48)
    at org.apache.tomcat.util.http.Parameters.urlDecode(Parameters.java:411)
    at org.apache.tomcat.util.http.Parameters.processParameters(Parameters.java:393)
    at org.apache.tomcat.util.http.Parameters.processParameters(Parameters.java:509)
    at org.apache.tomcat.util.http.Parameters.handleQueryParameters(Parameters.java:266)
    at org.apache.catalina.connector.Request.parseParameters(Request.java:2361)
    at org.apache.catalina.connector.Request.getParameter(Request.java:1005)
    at org.apache.catalina.connector.RequestFacade.getParameter(RequestFacade.java:353)
    at javax.servlet.ServletRequestWrapper.getParameter(ServletRequestWrapper.java:158)

Or Sometimes:

java.io.CharConversionException: isHexDigit
    at org.apache.tomcat.util.buf.UDecoder.convert(UDecoder.java:87)
    at org.apache.tomcat.util.buf.UDecoder.convert(UDecoder.java:48)
    at org.apache.tomcat.util.http.Parameters.urlDecode(Parameters.java:411)
    at org.apache.tomcat.util.http.Parameters.processParameters(Parameters.java:393)
    at org.apache.tomcat.util.http.Parameters.processParameters(Parameters.java:509)
    at org.apache.tomcat.util.http.Parameters.handleQueryParameters(Parameters.java:266)
    at org.apache.catalina.connector.Request.parseParameters(Request.java:2361)
    at org.apache.catalina.connector.Request.getParameter(Request.java:1005)
    at org.apache.catalina.connector.RequestFacade.getParameter(RequestFacade.java:353)
    at javax.servlet.ServletRequestWrapper.getParameter(ServletRequestWrapper.java:158)

Answer 1

Just hypothesizing here. Seems like the URL-decoding of parameters or their values fails (URL-encoding means encoding some characters using the %XX or %XXXX notation where XX or XXXX is the hexadecimal code of the character in ISO-8859-1 or Unicode). In the first case the error might be happening because there aren't enough hexadecimal characters after the % character. In the second case this might be happening because a character after the % character isn't hexadecimal.

Answer 2

Another thing to investigate is the URIEncoding in your Tomcat "Connector" configuration. If the link is in a UTF-8 encoded page, it will encode the URL to bytes with UTF-8, then URL encode any of the bytes that need it. However, by default, Tomcat thinks that those bytes are ISO-8859-1, which can lead to problems.

The inverse may also be true: if the page is ISO-8859-1, and Tomcat's URIEncoding has been set to UTF-8, a similar error could result.

Here's a useful discussion about the issues in this area: Charset Pitfalls in JSP/Servlet Containers

Answer 3

It could also be this (from Wikipedia):

There exists a non-standard encoding for Unicode characters: %uxxxx, where xxxx is a Unicode value represented as four hexadecimal digits. This behavior is not specified by any RFC and has been rejected by the W3C. The third edition of ECMA-262 still includes an escape(string) function that uses this syntax, but also an encodeURI(uri) function that converts to UTF-8 and percent-encodes each octet.

So you could be using the old escape function in Javascript, but since later versions of Tomcat are stricter about such things (5.5.17 let this encoding slide), only now are you beginning to see exceptions.

Answer 4

I started receiving this error when users were sending '%' over an ajax request. Turns out I wasn't escaping the parameters before making the request. A complete write up of this scenario and fix is covered in this blog post