views:

12170

answers:

10

I am having problems submitting forms which contain UTF-8 strings with Ajax. I am developing a Struts web application which runs in a tomcat server. This is the environment i set up to work with UTF-8:

  • I have added to tomcat's conf/server.xml the attributes URIEncoding="UTF-8" useBodyEncodingForURI="true" into the Connector tag.

  • I have a utf-8_general_ci database

  • I am using the next filter to ensure my request and responses are encoded in UTF-8

    package filters;
    
    
    import java.io.IOException;
    import javax.servlet.*;
    
    
    public class UTF8Filter implements Filter {
    public void destroy() {}
    
    
    public void doFilter(ServletRequest request,ServletResponse response, FilterChain chain)
     throws IOException, ServletException {
     request.setCharacterEncoding("UTF-8");
     response.setContentType("text/html;charset=UTF-8");
     chain.doFilter(request, response);
    }
    
    
    public void init(FilterConfig filterConfig) throws ServletException {
    }
    

    }

  • I use this filter in WEB-INF/web.xml

  • I am using the next code for my json responses:

    public static void populateWithJSON(HttpServletResponse response,JSONObject json) { String CONTENT_TYPE="text/x-json;charset=UTF-8"; response.setContentType(CONTENT_TYPE); response.setHeader("Cache-Control", "no-cache"); try { response.getWriter().write(json.toString()); } catch (IOException e) { throw new ApplicationException("Application Exception raised in RetrievedStories", e); } }

Everything seems to work fine (content coming from the database is displayed properly and i am able to submit forms which are stored in UTF-8 in the database). The problem is that i am not able to submit forms with AJAX. I use JQuery and i thought the problem was the lack of contentType field in ajax request. But i was wrong. I have a really simple form to submit comments which contains of an id and a body. The body field can be in different languages such as spanish, german or whatever.

If i submit my form with body textarea containing contraseña, firebug shows me:

Request Headers

  • Host localhost:8080
  • Accept-Charset ISO-8859-1, utf-8;q=0.7;*q=0.7
  • Content-Type application/x-www-form-urlencoded; charset UTF-8

If i excecute in Firebug Copy Location with parameters the econding seems already wrong

http://localhost:8080/Cerepedia/corporate/postStoryComment.do?&body=contrase%C3%B1a&id=88

This is my JQuery code:

function addComment() {
    var comment_body = $("#postCommentForm textarea").val();
    var item_id = $("#postCommentForm input:hidden").val();
    var url = rooturl+"corporate/postStoryComment.do?"; 
    $.post(url, { id:  item_id, body: comment_body } ,
        function(data){
        /* Do stuff with the answer */
    }, "json");  }

A submission of a form with JQuery is causing the next error server side (Note i am using hibernate).

Any help or ideas is really appreciated.

javax.servlet.ServletException: org.hibernate.exception.GenericJDBCException: Could not execute JDBC batch update
    at org.apache.struts.action.RequestProcessor.processException(RequestProcessor.java:520)
    at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:427)
    at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:228)
    at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
    at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:462)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.cerebra.cerepedia.security.AuthorizationFilter.doFilter(AuthorizationFilter.java:78)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.cerebra.cerepedia.hibernate.HibernateSessionRequestFilter.doFilter(HibernateSessionRequestFilter.java:30)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at filters.UTF8Filter.doFilter(UTF8Filter.java:14)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:261)
    at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
    at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:581)
    at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
    at java.lang.Thread.run(Unknown Source)
Caused by: org.hibernate.exception.GenericJDBCException: Could not execute JDBC batch update
    at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:103)
    at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:91)
    at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
    at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:249)
    at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:235)
    at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:139)
    at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:298)
    at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:27)
    at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1000)
    at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:338)
    at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:106)
    at com.cerebra.cerepedia.item.dao.ItemDAOHibernate.addComment(ItemDAOHibernate.java:505)
    at com.cerebra.cerepedia.item.ItemManagerPOJOImpl.addComment(ItemManagerPOJOImpl.java:164)
    at com.cerebra.cerepedia.struts.item.ItemAction.addComment(ItemAction.java:126)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.apache.struts.actions.DispatchAction.dispatchMethod(DispatchAction.java:269)
    at org.apache.struts.actions.DispatchAction.execute(DispatchAction.java:170)
    at org.apache.struts.actions.MappingDispatchAction.execute(MappingDispatchAction.java:166)
    at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425)
    ... 26 more
Caused by: java.sql.BatchUpdateException: Incorrect string value: '\xF1a' for column 'body' at row 1
    at com.mysql.jdbc.ServerPreparedStatement.executeBatch(ServerPreparedStatement.java:657)
    at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeBatch(NewProxyPreparedStatement.java:1723)
    at org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:48)
    at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:242)
    ... 44 more
26-ago-2008 19:54:48 org.apache.catalina.core.StandardWrapperValve invoke
GRAVE: Servlet.service() para servlet action lanzó excepción
java.sql.BatchUpdateException: Incorrect string value: '\xF1a' for column 'body' at row 1
    at com.mysql.jdbc.ServerPreparedStatement.executeBatch(ServerPreparedStatement.java:657)
    at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeBatch(NewProxyPreparedStatement.java:1723)
    at org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:48)
    at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:242)
    at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:235)
    at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:139)
    at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:298)
    at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:27)
    at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1000)
    at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:338)
    at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:106)
    at com.cerebra.cerepedia.item.dao.ItemDAOHibernate.addComment(ItemDAOHibernate.java:505)
    at com.cerebra.cerepedia.item.ItemManagerPOJOImpl.addComment(ItemManagerPOJOImpl.java:164)
    at com.cerebra.cerepedia.struts.item.ItemAction.addComment(ItemAction.java:126)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.apache.struts.actions.DispatchAction.dispatchMethod(DispatchAction.java:269)
    at org.apache.struts.actions.DispatchAction.execute(DispatchAction.java:170)
    at org.apache.struts.actions.MappingDispatchAction.execute(MappingDispatchAction.java:166)
    at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425)
    at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:228)
    at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
    at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:462)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.cerebra.cerepedia.security.AuthorizationFilter.doFilter(AuthorizationFilter.java:78)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.cerebra.cerepedia.hibernate.HibernateSessionRequestFilter.doFilter(HibernateSessionRequestFilter.java:30)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at filters.UTF8Filter.doFilter(UTF8Filter.java:14)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:261)
    at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
    at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:581)
    at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
    at java.lang.Thread.run(Unknown Source)
javax.servlet.ServletException: java.lang.NumberFormatException: null
    at org.apache.struts.action.RequestProcessor.processException(RequestProcessor.java:520)
    at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:427)
    at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:228)
    at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
    at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:449)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.cerebra.cerepedia.security.AuthorizationFilter.doFilter(AuthorizationFilter.java:78)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.cerebra.cerepedia.hibernate.HibernateSessionRequestFilter.doFilter(HibernateSessionRequestFilter.java:30)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at filters.UTF8Filter.doFilter(UTF8Filter.java:14)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:261)
    at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
    at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:581)
    at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
    at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NumberFormatException: null
    at java.lang.Long.parseLong(Unknown Source)
    at java.lang.Long.valueOf(Unknown Source)
    at com.cerebra.cerepedia.struts.item.ItemAction.addComment(ItemAction.java:120)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.apache.struts.actions.DispatchAction.dispatchMethod(DispatchAction.java:269)
    at org.apache.struts.actions.DispatchAction.execute(DispatchAction.java:170)
    at org.apache.struts.actions.MappingDispatchAction.execute(MappingDispatchAction.java:166)
    at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425)
    ... 26 more
26-ago-2008 20:13:25 org.apache.catalina.core.StandardWrapperValve invoke
GRAVE: Servlet.service() para servlet action lanzó excepción
java.lang.NumberFormatException: null
    at java.lang.Long.parseLong(Unknown Source)
    at java.lang.Long.valueOf(Unknown Source)
    at com.cerebra.cerepedia.struts.item.ItemAction.addComment(ItemAction.java:120)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.apache.struts.actions.DispatchAction.dispatchMethod(DispatchAction.java:269)
    at org.apache.struts.actions.DispatchAction.execute(DispatchAction.java:170)
    at org.apache.struts.actions.MappingDispatchAction.execute(MappingDispatchAction.java:166)
    at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425)
    at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:228)
    at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
    at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:449)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.cerebra.cerepedia.security.AuthorizationFilter.doFilter(AuthorizationFilter.java:78)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.cerebra.cerepedia.hibernate.HibernateSessionRequestFilter.doFilter(HibernateSessionRequestFilter.java:30)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at filters.UTF8Filter.doFilter(UTF8Filter.java:14)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:261)
    at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
    at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:581)
    at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
    at java.lang.Thread.run(Unknown Source)
+4  A: 

have you tried adding the following before the call :

$.ajaxSetup({ 
 scriptCharset: "utf-8" , 
 contentType: "application/json; charset=utf-8"
});

The options are explained here.

contentType : When sending data to the server, use this content-type. Default is "application/x-www-form-urlencoded", which is fine for most cases.

scriptCharset : Only for requests with 'jsonp' or 'script' dataType and GET type. Forces the request to be interpreted as a certain charset. Only needed for charset differences between the remote and local content.

Pat
+1  A: 

Thanks for the answer Pat, I think contentType was already set correctly. See the output of firebug, i posted before

Content-Type application/x-www-form-urlencoded; charset UTF-8

Apparently firebug does not display correctly UTF-8 strings but they were received correctly in the server. I did some logging in the server to test that. I was using a MySQL database with collation utf-8_general_ci but Hibernate was complaining as i posted in before. I changed the collation of my database to latin1_general_c1 and now everything seems to work fine.

This solution will only work while i stay withing latin1 languages (italian, german, spanish, english...). But what if i want to use russian. Does anyone why i can not use a database with utf-8 collation?

Sergio del Amo
A: 

I hade the same problem and fixed it with downgrading to mysql-connector-odbc-3.51.16.

Anders B
A: 

I had the same problem also and I fixed it in this way: In php before storing the data in the database I used htmlentities() function. And during showing the data I used the html_entity_decode() function. This worked. I strongly hope this will work for you too.

Thanks Adnan

adnan
+2  A: 

I'm having the same problem. I saw IE 8 sends this header:

content-type = application/x-www-form-urlencoded

while Firefox sends this:

content-type = application/x-www-form-urlencoded; charset=UTF-8

My solution was just forcing in jQuery to use the firefox content-type:

$.ajaxSetup({ scriptCharset: "utf-8" ,contentType: "application/x-www-form-urlencoded; charset=UTF-8" });
Jesús Alonso
A: 

hi guys,

im having a related problem. im loading in a value from a DB through an ajax post call. $.ajax({ type: 'POST', url: "/lc360/template/gettext", data: "template_id=" + $(this).attr('id'), success: function(msg){ $("#mail_content").val(msg + "\n" + $("#sender_email").html());}, contentType: "application/x-www-form-urlencoded; charset=ISO-8859-1", scriptCharset: "ISO-8859-1", dataType: "html" });

my text im reading in is german and im loading it into a textarea. when I check in firebug, firstly, I find that the header Request says "Content-Type application/x-www-form-urlencoded; charset=UTF-8". so somehow it gets overwritten.

Also when i check the response through firebug it says correctly my text has been replied with words correctly as "für". In my text area how ever or when I alert out the response, the "ü" is replaced by a diamond shaped black figure with a question mark in it. What is causing this?

Bart
BalusC
A: 

oh i have this problem a lot.. the meta doesnt work always in your php data operations, so just type this at the beginning..

Michael
A: 

Michael, what should type at the begining?

Muaawiya
A: 

I have the same problem with ajax POST in jQuery. contentType settings dosn't help. On server side i've saved received http request bytes to the file as it is, and then viwed this saved request with HexEdit. utf8 strings are already as question signs like '????, ????' (char code 0x003F). It means that jQuery sends it in wrong way.

Anton
A: 

As the exception is a jdbc error, your best approach is to capture the input before it is sent to the database.

As per the error:

java.sql.BatchUpdateException: Incorrect string value: '\xF1a' for column 'body' at row 1

... a single character is causing the exception.

It may be the case that you will need to override some characters manually. You will find, when working with non-latin-alphabet languages (as I do), that this is a common pain.

Theo

Theofanis Pantelides