There were basically three aspects, which made internationalization difficult:

1. All levels of a system and application are effected (e.g. operating system, I/O operations, database, web-tier, web-services). If the settings don’t match at some point, it might results in wrong character conversion. Usually this means we have to look at database encoding, file system encoding and http request/response encoding.
2. A lot of software which is being used (e.g. 3rd party components and libraries) uses a default character encoding, which is usually ISO-8859-1 (Latin-1). Is this western ignorance?
3. You don’t have the time to understand the complicated and often unclear issues surrounding character set encoding. If the world could just decide to switch to one global encoding, our live would be much easier.

The goal is to use only one character encoding within the Java application (in our case UTF-8 seems to be fine for the job), so we only have to handle different character sets at entry and exit points (web, files, …).

Let’s look at places which might need some attention:

Application servers

Start these with the correct JVM arguments.

Default file encoding
The default file encoding is being used by the InputStreamReader and OutputStreamReader. If it is not set, the file encoding of the operating system will be used which can lead to unexpected results if you have a team which works on different systems or if the deployment system differs from the development environment. Set it with -Dfile.encoding=UTF-8

Next check and if necessary configure the default character encoding.

Resin
The default value is ISO-8859-1.
see Specify the default character encoding for the environment.

Tomcat
Default encoding of Tomcat 5 is UTF-8. If not, you can specify it in $CATALINA_BASE/conf/web.xml or in your webapp’s own web.xml.

WebSphere
Default character encoding is UTF-8. For more information see: Developing J2EE Global Applications : Character Encoding

Database

Switch the complete database, individual tables or individual columns to UTF-8. How this can be done differs per database system.

For some JDBC database drivers you have to specify the encoding explicitly, others drivers are smart enough to determine the database encoding automatically.

Oracle 10g
In order to review the current settings enter SELECT * FROM V$NLS_PARAMETERS;
NLS_CHARACTERSET and NLS_LENGTH_SEMANTICS are interesting for us. Oracle recommends using Unicode character set AL32UTF8 for all new system deployments.
If you don’t want to change the settings for the database, you can use the NCHAR, NVARCHAR2, and NCLOB datatypes instead. Their default encoding is AL16UTF16.

Additional information:
Changing The Character Set In Oracle Applications
Character Semantics and Globalization

Spring Framework

Since Spring handles you requests, it needs some extra configuration:
Add filter to web.xml and spring configuration
In this case the Spring framework does most of the work for you. Without such framework you might have to do some conversion between different character encoding types yourself.

Java Servlet Pages

Use: <%@ page pageEncoding=”UTF-8″ contentType=“text/html; charset=UTF-8″ %>

To set the default page encoding used for all jsp files, use

<jsp-property-group>
(…)
<page-encoding>utf-8</page-encoding>
</jsp-property-group>

Additional information:
Setting Properties for Groups of JSP Pages

It is a good idea to add
<META http-equiv=”Content-Type” content=”text/html; charset=UTF-8″>
to the html as well.

Templating Engines and layout frameworks

Sitemesh
By default Sitemesh uses ISO-8859-1. All used JSP pages should define UTF-8 as encoding. If you have various decorators and includes, these must all use the same encoding.

Velocity
Also Velocity uses ISO-8859-1 as default. This has been the large pitfall on my first internationalization project. I wasted a lot of time before I knew this.
Velocity allows you to specify the character encoding of your template resources on a template by template basis. The output encoding is an application specific setting and can be set in the runtime configuration with following configuration key: output.encoding (it might be a good idea to set input.encoding as well)
More information: Velocity Developer Guide

Freemarker
You can specify the charset of the template in the template itself and the charset of the output with the
setOutputEncoding(outputCharset) method of a Freemarker processing environment.

Resource bundles

Edit message bundles for non western languages in UTF-8 mode and then convert this file to an ascii format for Java. Call native2ascii encoding, specifying the original file has UTF-8 encoding:
native2ascii -encoding UTF-8 messages_cn.txt messages_cn.properties

Additional information

Java World: Multibyte-character processing in J2EE
An in-depth look at Java’s character type
A tutorial on character code issues

Bookmark It

Next I had another look at Webmin and it turned out to be less difficult than expected. Just download webmin-1.270.tar.gz, copy it to /usr/local, unpack it here and run ./setup.sh. (last 3 steps as root or with the sudo command).

Since I already had my fetchmail configuration files, I just copied those as well and configured the cron jobs in Webmin. You can also use MailServe for the settings, if you don’t want to switch between different configurations.

The migration is now complete and all that is left are extra features like configuring automatic logfile analysis (AWstats or Webalizer), scheduled backups etc.

Update [March 2007]:
The PowerBook is still in use as our home-server and things have been going well. The speed is good and noise is low. Recently I moved it to our media room, where it is also being used as media center, directly connected to the LCD TV and controlled with a bluetooth keyboard/trackpad.

Bookmark It

Update [2007]:
The old internal hard disk of the PowerBook died in December 2006. I noticed it early enough and was able to rescue the data and replace it with a new and larger disk. For a notebook of that age it had to happen sooner or later.
The LaCie Big Disk has been a disappointment. In contrast to my external Firewire disk it does not spin down when it is not being used for a while and since the cooling of the enclosure is not very effective, the disk tends to heat up quiet a bit in the summer.

Bookmark It

Intranet
Today I wanted so setup the intranet server. On our intranet we use a couple of web-applications, which require a MySQL database, Apache and PHP. Sometimes I also use it too develop and test new sites and applications, so it should also support Java-applications and Ruby On Rails, but that can wait.

Previously I have been using the MAMP package to setup a MySQL, Apache, PHP environment on my Mac, so that’s what I used as well for the new server and after I had transferred the SQL data to the new database, everything was working, basically again within minutes. But then I had to realize, that MAMP is not really suited as solution for a server, because you have to start it as user who is logged in. So I had to start all over again but it is not that difficult to install and configure the components by hand either.

Server software
Apache is already installed and configured. You can start it with the sharing control panel. MySQL can be downloaded for Mac OS X from the official site and includes a control panel, where you can specify, that the database server should start automatically. For PHP I chose the PHP 5 installer provided by Marc Liyanage. I adjusted the default Apache settings (as usual in /etc/httpd/httpd.conf) by hand, which is not a problem since I have been working with Apache for many years. Getting one of the web-applications to work was a bit difficult, since it was written for PHP 4 and expected certain settings in php.ini, which were not set in the default installation. Once Apache and PHP were configured I was able to use phpMyAdmin to create the databases again and everything was working again.
For our LinkTheater network media player I installed the official LinkTheater server software as well as the AVeL Link Server, which works as well with the LinkTheater. Both of these were working but disappointing, since the Windows versions have a user interface with settings and some extra functions. But both applications become interesting, when you examine the software package, because they are pure Java applications. Maybe they can be customized and extended.

The only service left on the old server now, is the mail server with postfix, procmail, imap and fetchmail. I have also been looking at Webmin, which I usually used on the Linux server to administer and configure it. Webmin should run on Mac OS X but the installation seems complicated and I haven’t had time to try it yet.

Bookmark It

Server hardware
As new hardware I chose my old 15” PowerBook, which has a 1.25Ghz G4 processor and 1.25GB RAM. I have been considering a new Intel powered Mac Mini, but the expense would have been too high at this point and it is better to use 3 year old hardware anyhow. For storage I will be using some of my external Firewire hard-disks.

Every Mac OS X already ships with most of the software, which is needed for a server. The only thing is, that Apple sort of hides its functionality behind the user-friendly front-end of Mac OS X. One could buy an official Mac OS X Server license which has nice administration interfaces and some extra software, but for a home-server, that would be too much and also too expensive.

Backup and Remote Access
I started with backups of the old files on the PowerBook, a clean Mac OS 10.4.6 installation and formating of the external 150GB FW800 disk. Next I enabled remote desktop and remote login (ssh), so that I can administer the new server remotely from my laptop. Apple’s remote desktop service includes support for VNC viewers, so you don’t need the Remote Desktop administration software. With a VNC client like Chicken of the VNC I can now control the server.

File sharing
Next I configured file sharing. With Apple’s sharing control panel you can turn personal and windows file sharing on, but that doesn’t quiet give you the flexibility you will need for a file server with multiple users and different shares. Luckily Michael Horn wrote the tool Sharepoints, which makes it very easy to define shared folders and manage users and groups. With this useful application, the Mac OS X file server was up and running within minutes. If you are a Unix geek, you can of course also configure configuration files with vi directly.

There is a small problem with external disks, like those I am using on my file server. By default Mac OS X only mounts these automatically when a user logs in. But in the case of the file server the external disks should be mounted when the system boots. Here the shareware TinkeTool System can help. On the Volumes tab of System Setup you can enable automatic mount of removable hard disk volumes.

Restoring files
It took the rest of the day to move files from the old to the new file server. Once the mp3 files were moved, I installed the SlimServer and it was delightful, how easy the installation was, compared to a manual installation on a Linux system (actually installing the RPM package on Mandrake etc. is easy as well).

Next we will look at the intranet web- and database server.

Bookmark It