Special characters incongruence

Question

Hello I'm building a Spring MVC web application that runs on Tomcat 6.0.20 and JDK 1.6.0_19. When I send some special characters through an HTML form some of them are stored as question marks ?

For example these symbols are stored correctly: €, á, é, í, ‰, etc But some symbols are replaced with ? like: ₤, ♪, ☺

MySQL tables charset is utf-8. My jsp also use utf-8

<%@ page language="java" contentType="text/html; charset=UTF-8" pageEncoding="UTF-8" %>

I have included org.springframework.web.filter.CharacterEncodingFilter in web.xml as suggested here

When I debug the POST request when sending 3 characters €a₤ with firebug I get: %E2%82%ACa%E2%82%A4 which is correct since E2 82 AC is the code for € and E2 82 A4 is the code for ₤ but ₤ is stored as ? in the database. When I save ₤ directly into the database it is displayed correctly in the webpage.

How can I fix this?

Answer 1

There are several areas where you need to check that character encoding is being correctly done as some default to UTF-8 and others do not (for example, iso-8859-1). Check:

• JDBC driver
• Java Virtual Machine (can only parse latin-based and unicode - can run native2ascii to convert non-latin characters to unicode)
• Database encoding
• Tomcat/JBoss, etc (server)
• browser
• operating system

It's a pain in the neck to have to check all of this stuff, but someone's gotta do it. I recommend setting everything to UTF-8 as that is generally what people agree on

To me it sounds like the JVM or your JDBC driver could be the source of the problem

Answer 2

First print those characters to an UTF-8 capable stdout. If they look wrong, then you need to set the HTTP request encoding. If they look right, then the problem is indeed in the data access layer. Use an independent DB management tool to manually insert and review those characters. If they look wrong, then you need to set the DB (and table!) encoding. If they look right, then the problem is in the JDBC driver / connectionstring. Also check if the JDBC driver version matches with the DB and is the latest available.

You can find more background information, practical explanation and detailed solutions in this article.