java - Why Arabic Letters Are Not Inserting In Database?

Question

I am helping to one of my friend. He is creating web-application using JSF 2.0 & mysql.

While creating database he have used below query.

CREATE DATABASE dbName DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

Till date website is working fine. Today client tried entering Arabic text and they said that the output is coming weird. What my friend do is after entering the data to DB, he also prints the same data on another page saying Congratulations, XYZ ABC is added successfully. However he see output as Congratulations, ù????§ù? ?aù?ù?ù??? ù??aù?ù?ù? ?μ?ˉù?ù? ù?ù??¨ù??|?? is added successfully. I don't understand why he get like that when Database characters are set properly.

web.xml content is as below.

<?xml version="1.0" encoding="UTF-8"?>
<web-app version="3.0" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd">
    <context-param>
        <param-name>javax.faces.PROJECT_STAGE</param-name>
        <param-value>Development</param-value>
    </context-param>
    <servlet>
        <servlet-name>Faces Servlet</servlet-name>
        <servlet-class>javax.faces.webapp.FacesServlet</servlet-class>
        <load-on-startup>1</load-on-startup>
    </servlet>
    <servlet-mapping>
        <servlet-name>Faces Servlet</servlet-name>
        <url-pattern>/faces/*</url-pattern>
    </servlet-mapping>
    <session-config>
        <session-timeout>
            600
        </session-timeout>
    </session-config>
    <welcome-file-list>
        <welcome-file>faces/index.xhtml</welcome-file>
    </welcome-file-list>
    <filter>
        <filter-name>restrict</filter-name>
        <filter-class>com.sac.filter.MyFilter</filter-class>
    </filter>
    <filter-mapping>
        <filter-name>restrict</filter-name>
        <url-pattern>*.xhtml</url-pattern>
    </filter-mapping>
    <filter>
        <filter-name>MyFacesExtensionsFilter</filter-name>
        <filter-class>org.apache.myfaces.webapp.filter.ExtensionsFilter</filter-class>
    </filter>
    <filter-mapping>
        <filter-name>MyFacesExtensionsFilter</filter-name>
        <servlet-name>Faces Servlet</servlet-name>
    </filter-mapping>

    <servlet>
        <servlet-name>DisplayImage</servlet-name>
        <servlet-class>com.sac.databean.DisplayImage</servlet-class>
    </servlet>
    <servlet-mapping>
        <servlet-name>DisplayImage</servlet-name>
        <url-pattern>/DisplayImage</url-pattern>
    </servlet-mapping>
    <servlet>
        <servlet-name>SaveMyImage</servlet-name>
        <servlet-class>com.sac.databean.SaveMyImage</servlet-class>
    </servlet>
    <servlet-mapping>
        <servlet-name>SaveMyImage</servlet-name>
        <url-pattern>/SaveMyImage</url-pattern>
    </servlet-mapping>

<!-- for not using css and js of default richfaces   -->
    <context-param>
        <param-name>org.richfaces.SKIN</param-name>
        <param-value>plain</param-value>
    </context-param>
    <context-param>
        <param-name>org.richfaces.LoadStyleStrategy</param-name>
        <param-value>None</param-value>
    </context-param>
    <context-param>
        <param-name>org.richfaces.enableControlSkinning</param-name>
        <param-value>false</param-value>
    </context-param>
<!-- for not using css and js of default richfaces   -->    

</web-app>

On each .xhtml page, he have <?xml version='1.0' encoding='UTF-8' ?>

Please let me know if you need anything else.

---

Edit 1

In my JSF filter, I also added req.setCharacterEncoding("UTF-8"); in doFilter(). Still in Database I see ???????????

---

Edit 2

In JSF page I have <h:inputText value="#{PersonalInformationDataBean.fullName}"> and when I print the fullName value in Java bean as System.out.println("my name while entering is " + fullName);m I get output as my name while entering is ???????????? ????.

This means there is problem while entering data

Can someone help what is going ODD?

Answer 1

However he see output as Congratulations, ù????§ù? ?aù?ù?ù??? ù??aù?ù?ù? ?μ?ˉù?ù? ù?ù??¨ù??|?? is added successfully. I don't understand why he get like that when Database characters are set properly.

This is known as Mojibake. This is not a DB encoding problem, but a HTTP encoding problem. Setting the POST request character encoding as you did is indeed the proper solution.

---

Edit 1: In my JSF filter, I also added req.setCharacterEncoding("UTF-8"); in doFilter(). Still in Database I see ???????????

Question marks occur when the both sides of the connection are aware of their own encoding. Sent/retrieved characters which are not covered by the encoding of one side will be replaced by question marks. Arabic characters doesn't occur in ISO-8859-1 and hence they're replaced by question marks. That's the difference with Mojibake whereby characters are been sent without checking if the encoding used by the other side really supports the character. You'll end up incorrectly encoded characters which presents itself as an unintelligible sequence of characters.

In this particular case, the JDBC driver is by itself aware that it's using ISO-8859-1 by default to transmit the characters to DB, while the retrieved characters are in UTF-8 (the MySQL JDBC driver doesn't look at the DB table encoding, even though it's properly been set to UTF-8 in your case). You need to explicitly tell the JDBC driver to use UTF-8 to decode characters before transmitting data to DB. This is to be done as JDBC connection properties which are definied as query string parameters in the JDBC URL like so:

jdbc:mysql://localhost:3306/db_name?useUnicode=yes&characterEncoding=UTF-8

If you're using a container-managed datasource, then just specify those properties separately the same way as you did for the username and password

useUnicode=yes
characterEncoding=UTF-8

See also:

• Unicode - How to get the characters right?