Spring MVC 3.0.5: Forward Sets Character Encoding

The Problem

Alright, second try explaining my problem: I was specifying a REST interface returning a JSON object. This was done via Spring's @ResponseBody annotation. The response I got would not display some special characters (like ä,ö,ü) correctly using a browser (e.g. Firefox 14.0.1). Those characters turned into weird signs.

Important: the REST interface was accessed using an URL being forwarded internally.

HTTP Headers

The special characters (like ä,ö,ü) were displayed by two weird signs each indicating that the character encoding was UTF-8 being displayed using ISO-8859-1 (when using CURL on my Ubuntu terminal, which uses UTF-8 by default, those symbols were displayed correctly).

On the HTTP protocol level I was sending a request without any specific character encoding:

GET /servlet-root/some/forwarded/url HTTP/1.1
User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
Host: localhost:8080
Accept: application/json

Note: the URL is being forwarded internally using Spring forwarding functionality. This is important as it is the source of the problem.

@RequestMapping("/some/forwarded/url")
public String handle() {
  return "forward:/the/original/url";
}

The corresponding response header I got was:

HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Set-Cookie: JSESSIONID=57F7A6ECFDDEA925D2565E67A4B718CD; Path=/servlet-root/; HttpOnly
Pragma: no-cache
Cache-Control: no-cache,no-store
Expires: Wed, 31 Dec 1969 23:59:59 GMT
Last-Modified: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: application/json;charset=ISO-8859-1
Content-Language: en-US
Transfer-Encoding: chunked
Date: Wed, 25 Jul 2012 13:58:45 GMT

Note: the Content-Type is defining the charset ISO-8859-1:

Content-Type: application/json;charset=ISO-8859-1

and also not how a Content-Language is defined:

Content-Language: en-US

Now, sending the same header to the original URL (/servlet-root/the/origianl/url) instead of the one being forwarded (/servlet-root/some/forwarded/url) with the same request header, I received a very different response header:

HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Set-Cookie: JSESSIONID=FC01F1AABA8DD80A420BE0C70E2B422C; Path=/ubicon-webapp/; HttpOnly
Pragma: no-cache
Cache-Control: no-cache,no-store
Expires: Wed, 31 Dec 1969 23:59:59 GMT
Last-Modified: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Date: Wed, 25 Jul 2012 14:18:47 GMT

Note: no charset and no Content-Language is defined:

Content-Type: application/json

Client-Side Solutions

Now there are theoretically two ways to tell the server to return UTF-8.

  • One is by setting the charset in the Accept header:
    Accept: application/json;charset=UTF-8
  • Theoretically, the other is by setting the Accept-Charset:
    Accept-Charset: UTF-8

Both telling the server that only those charsets are acceptable. I tried both using curl, but setting the Accept-Charset did not have any effect. The commands I issued were

curl -i --header "Accept: application/json;charset=UTF-8" http://localhost:8080/servlet-root/some/forwarded/url

and

curl -i --header "Accept: application/json" --header "Accept-Charset: UTF-8" http://localhost:8080/servlet-root/some/forwarded/url

I am actually using jQuery and its $.getJSON(...) function to access the REST interface. To set the Accept header the following commands can be used before calling the $.getJSON(...) function:

$.ajaxSetup({
  accepts : {
    json : 'application/json; charset=UTF-8',
  },
});

or

$.ajaxSetup({
  headers: { 
    Accept : "application/json; charset=UTF-8",
  },
});

Respectively setting the Accept-Charset header field would be done calling:

$.ajaxSetup({
  headers: { 
    Accept-Charset: "UTF-8",
  },
});

Yet, as verfied by using curl, this does not work.

I found these solutions at

The jQuery documentation is rather vague on the methods used above, but still: also refer to the following jQuery documention pages for reference:

  • http://api.jquery.com/jQuery.ajax/
  • http://api.jquery.com/jQuery.ajaxSetup/
  • http://api.jquery.com/jQuery.getJSON/

Encoding VS Charset

There is also some kind of discrepancy between "encoding" and "charset". Maybe this plays a role in why using

Accept: application/json;charset=UTF-8

works and

Accept-Charset: UTF-8

does not. There are articles talking about this here

Some claim Accept-Charset "is no more" and others actually try to determine the difference or priority between the two without success:

An offical statement (by W3C) and an example mentioning those can be found here (this may already be outdated though):

I am too busy writing this article now. So understanding this might come later. Somewhere around here the current official definition should be findable, but how it is to be interpreted is another story:

The Cause

The cause lies within Spring forwarding functionality. When forwarding a View is generated. On this View the locale is automatically being inferred and set on the Response object. This, on the other hand, automatically sets the character set to ISO-8859-1. I will augment this section soon going further into detail.

I pretty much narrowed things down, I think. Let me show you a calling stack first:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
Daemon Thread [http-bio-8080-exec-1] (Suspended)
        Response.getContentType() line: 494
        Http11Processor(AbstractHttp11Processor<S>).prepareResponse() line: 1052
        Http11Processor(AbstractHttp11Processor<S>).action(ActionCode, Object) line: 697
        Response.action(ActionCode, Object) line: 170
        Response.sendHeaders() line: 350
        OutputBuffer.doFlush(boolean) line: 317
        OutputBuffer.flush() line: 299
        CoyoteOutputStream.flush() line: 103
        Utf8Generator.flush() line: 1085
        ObjectMapper.writeValue(JsonGenerator, Object) line: 1606
        MappingJacksonHttpMessageConverter.writeInternal(Object, HttpOutputMessage) line: 153
        MappingJacksonHttpMessageConverter(AbstractHttpMessageConverter<T>).write(T, MediaType, HttpOutputMessage) line: 181
        AnnotationMethodHandlerAdapter$ServletHandlerMethodInvoker.writeWithMessageConverters(Object, HttpInputMessage, HttpOutputMessage) line: 975
        AnnotationMethodHandlerAdapter$ServletHandlerMethodInvoker.handleResponseBody(Object, ServletWebRequest) line: 933
        AnnotationMethodHandlerAdapter$ServletHandlerMethodInvoker.getModelAndView(Method, Class, Object, ExtendedModelMap, ServletWebRequest) line: 882
        AnnotationMethodHandlerAdapter.invokeHandlerMethod(HttpServletRequest, HttpServletResponse, Object) line: 428
        AnnotationMethodHandlerAdapter.handle(HttpServletRequest, HttpServletResponse, Object) line: 414
        DispatcherServlet.doDispatch(HttpServletRequest, HttpServletResponse) line: 790
        DispatcherServlet.doService(HttpServletRequest, HttpServletResponse) line: 719
        DispatcherServlet(FrameworkServlet).processRequest(HttpServletRequest, HttpServletResponse) line: 644
        DispatcherServlet(FrameworkServlet).doGet(HttpServletRequest, HttpServletResponse) line: 549
        DispatcherServlet(HttpServlet).service(HttpServletRequest, HttpServletResponse) line: 621
        DispatcherServlet(HttpServlet).service(ServletRequest, ServletResponse) line: 722
        ApplicationFilterChain.internalDoFilter(ServletRequest, ServletResponse) line: 304
        ApplicationFilterChain.doFilter(ServletRequest, ServletResponse) line: 210
        ApplicationDispatcher.invoke(ServletRequest, ServletResponse, ApplicationDispatcher$State) line: 684
        ApplicationDispatcher.processRequest(ServletRequest, ServletResponse, ApplicationDispatcher$State) line: 471
        ApplicationDispatcher.doForward(ServletRequest, ServletResponse) line: 402
        ApplicationDispatcher.forward(ServletRequest, ServletResponse) line: 329
        InternalResourceView.renderMergedOutputModel(Map<String,Object>, HttpServletRequest, HttpServletResponse) line: 238
        InternalResourceView(AbstractView).render(Map<String,?>, HttpServletRequest, HttpServletResponse) line: 250
        DispatcherServlet.render(ModelAndView, HttpServletRequest, HttpServletResponse) line: 1047
        DispatcherServlet.doDispatch(HttpServletRequest, HttpServletResponse) line: 817
        DispatcherServlet.doService(HttpServletRequest, HttpServletResponse) line: 719
        DispatcherServlet(FrameworkServlet).processRequest(HttpServletRequest, HttpServletResponse) line: 644
        DispatcherServlet(FrameworkServlet).doGet(HttpServletRequest, HttpServletResponse) line: 549
        DispatcherServlet(HttpServlet).service(HttpServletRequest, HttpServletResponse) line: 621
        DispatcherServlet(HttpServlet).service(ServletRequest, ServletResponse) line: 722
        ApplicationFilterChain.internalDoFilter(ServletRequest, ServletResponse) line: 304
        ApplicationFilterChain.doFilter(ServletRequest, ServletResponse) line: 210
...

On line 12 the JacksonHttpMessageConverter handles the JSON conversion of the given object. After it is done, the CoyoteOutputStream is flushed. This process also sets the response header fields (line 3) including the Content-Type field. The content of this field is generated by Response.getContentType() on line 2.

Now let's have a look at Response.getContentType():

1
2
3
4
5
6
7
8
9
10
11
public String getContentType() {
 
    String ret = contentType;
 
    if (ret != null 
        && characterEncoding != null
        && charsetSet) {
        ret = ret + ";charset=" + characterEncoding;
    }
    return ret;
}

This is where it get interesting: The ret and characterEncoding are never null. The charsetSet variable varies.
It is

  • true, if the REST interface is accessed via the forwarded URL and
  • false, if the REST interface is accessed directly.

So where is the character encoding set explicitly? Let's start with a calling stack again:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Daemon Thread [http-bio-8080-exec-7] (Suspended (modification of field charsetSet in Response))
        Response.setCharacterEncoding(String) line: 411
        Response.setLocale(Locale) line: 875
        ResponseFacade.setLocale(Locale) line: 351
        HttpSessionSecurityContextRepository$SaveToSessionResponseWrapper(ServletResponseWrapper).setLocale(Locale) line: 197
        UrlRewriteWrappedResponse(ServletResponseWrapper).setLocale(Locale) line: 197
        DispatcherServlet.render(ModelAndView, HttpServletRequest, HttpServletResponse) line: 1022
        DispatcherServlet.doDispatch(HttpServletRequest, HttpServletResponse) line: 817
        DispatcherServlet.doService(HttpServletRequest, HttpServletResponse) line: 719
        DispatcherServlet(FrameworkServlet).processRequest(HttpServletRequest, HttpServletResponse) line: 644
        DispatcherServlet(FrameworkServlet).doGet(HttpServletRequest, HttpServletResponse) line: 549
        DispatcherServlet(HttpServlet).service(HttpServletRequest, HttpServletResponse) line: 621
        DispatcherServlet(HttpServlet).service(ServletRequest, ServletResponse) line: 722
        ApplicationFilterChain.internalDoFilter(ServletRequest, ServletResponse) line: 304
        ApplicationFilterChain.doFilter(ServletRequest, ServletResponse) line: 210
...

The DispatcherServlet.doDispatch(HttpServletRequest, HttpServletResponse) checks at some point if the called handler returns a view to render or not:

...
// Did the handler return a view to render?
if (mv != null && !mv.wasCleared()) {
	render(mv, processedRequest, response);
        //...
}
...

If so, it calls the DispatcherServlet.render(ModelAndView, HttpServletRequest, HttpServletResponse) method. Otherwise it does not.


It turns out that the handler responsible for forwarding DOES return a view to render, while the handler responsible for providing the JSON does NOT.

This is important because DispatcherServlet.render(ModelAndView, HttpServletRequest, HttpServletResponse) sets the locale of the view (if nothing is specified by the request or a cookie, it will default to some locale) and eventually calls Response.setLocale(Locale). The latter function does not only set the locale but also the character encoding:

@Override
public void setLocale(Locale locale) {
    // ...
    coyoteResponse.setLocale(locale);
 
    // ...
    CharsetMapper cm = getContext().getCharsetMapper();
    String charset = cm.getCharset( locale );
    if ( charset != null ){
        coyoteResponse.setCharacterEncoding(charset);
    }
}

Thus we have a character encoding set when using the forwarded URL and no character encoding if not.

Now, the question is: why would a forwarded request get a locale and a character encoding by default? I have no answer to that ... I opened an issue on the Springframwork JIRA. Let's see what they say: https://jira.springsource.org/browse/SPR-9647

Server-Side Solutions

There are several solutions I or others came up with. One of them is by using the CharsetEncoderFilter supplied by Spring itself. It can be used to force the character encoding on both the Request and the Response by adding the following code snippet to the web.xml as the very first filter:

<filter>
	<filter-name>characterEncodingFilter</filter-name>
	<filter-class>org.springframework.web.filter.CharacterEncodingFilter</filter-class>
	<init-param>
		<param-name>encoding</param-name>
		<param-value>UTF-8</param-value>
	</init-param>
	<init-param>
		<param-name>forceEncoding</param-name>
		<param-value>true</param-value>
	</init-param>
</filter>
<filter-mapping>
	<filter-name>characterEncodingFilter</filter-name>
	<url-pattern>/*</url-pattern>
</filter-mapping>

I found this solution here:

This solution has the disadvantage of always setting the request encoding to UTF-8, because only if the character encoding is forced on the request, the character encoding of the reponse will be set as well (see http://fstyle.de/hp_fstyle/wordpress/2012/07/27/spring-mvc-characterencodingfilter/). Yet, at least as far as I can see, this might cause certain non UTF-8 requests to be misinterpreted.

So another filter based solution might be to set the encoding of the response only to UTF-8 by default. I came up with a custom filter which is attached to this post (download here). Only setting the response encoding might cause problems though which is hinted at in https://jira.springsource.org/browse/SPR-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel. Again, maybe this question will be answered in the Stackoverflow discussion referred to by http://fstyle.de/hp_fstyle/wordpress/2012/07/27/spring-mvc-characterencodingfilter/.

Another method setting the character encoding by hand is doing so specifically in the forwarding request mapping:

@RequestMapping("/some/forwarded/url")
public String handle(HttpServletResponse response) {
  response.setCharacterEncoding("UTF-8");
  return "forward:/the/origianl/url";
}

Another way of doing this is not specifying the encoding the forwarding handler method but in the handler method being forwarded to, like this:

@RequestMapping("/the/original/url")
public @ResponseBody Object handle(HttpServletResponse response) {
  response.setCharacterEncoding("UTF-8");
  return new SomeSerializableObject();
}

Also Spring 3.1 offers new funtionality in this direction using annotations:
the procudes annotation of the @RequestMapping annotation. I did not test this yet, but it should work.

@RequestMapping(value = "/the/original/url" produces="application/json;charset=UTF-8")
public @ResponseBody Object handle() {
  return new SomeSerializableObject();
}

Other

At first I was suspecting that as the return type of the forwarding controlleris String (which of course was nonsense as no @ResponseBody annotation was used; furthermore I was able to confirm this error in thought by using the Eclipse Debugger). Thus the StringHttpMessageConverter might play a role. Solving the issue with the StringHttpMessageConverter not returning UTF-8 was addressed quite a few times. The solution using the CharsetEncodingFilter, as mentioned above, is said to not work (http://stackoverflow.com/questions/3616359/who-sets-response-content-type-in-spring-mvc-responsebody). Yet the same source mentions other methods to change the default character encoding of StringHttpMessageConverter the to UTF-8 using new features of Spring 3.1's configuration methods:

<mvc:annotation-driven>
  <mvc:message-converters register-defaults="true">
    <bean class="org.springframework.http.converter.StringHttpMessageConverter">
      <property name="supportedMediaTypes" value = "text/plain;charset=UTF-8" />
    </bean>
  </mvc:message-converters>
</mvc:annotation-driven>

Or code-based configuration:

@Configuration
@EnableWebMvc
public class WebConfig extends WebMvcConfigurerAdapter {
 
  private static final Charset UTF8 = Charset.forName("charset=UTF-8");
 
  @Override
  public void configureMessageConverters(List<HttpMessageConverter<?>> converters) {
    StringHttpMessageConverter stringConverter = new StringHttpMessageConverter();
    stringConverter.setSupportedMediaTypes(Arrays.asList(new MediaType("text", "plain", UTF8)));
    converters.add(stringConverter);
 
    // Add other converters ...
  }
}

https://jira.springsource.org/browse/SPR-9099 mentions that the former can also be emulated using normal bean instantiations like this:

<bean id="stringConverter" class="org.springframework.http.converter.StringHttpMessageConverter">
  <property name="supportedMediaTypes" value="text/plain;charset=UTF-8"/>
</bean>

Another method mentioned by https://jira.springsource.org/browse/SPR-9099 which supposedly works in Spring 3.1 is by using the new procudes annotation of the @RequestMapping annotation.

@RequestMapping(value = "/foo", produces="text/plain;charset=UTF-8")
public @ResponseBody String handle() {
  return "Foo";
}

The End

VN:F [1.9.22_1171]
Rating: 0.0/10 (0 votes cast)
VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)

2 Responses to “Spring MVC 3.0.5: Forward Sets Character Encoding”

  1. Martin Becker says:

    Hi, thanks for the info. I still have not tested it yet. So I can't tell you anything about it. I will as soon as I get time to try it out.

    VN:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
    VN:F [1.9.22_1171]
    Rating: 0 (from 0 votes)

  2. pogacsa says:

    3.1 producse version does not work for me with FireFox.

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)

Leave a Reply