Unicode with Exponential

PostgreSQL

To use unicode with Exponential you should use PostgreSQL. MySQL has Unicode support in version 4.1.1 and newer, which is not stable at the time of writing this.

Create database

To be able to run Exponential with Unicode (UTF-8) support you need to create a database which has support for this. This is done with the createdb command.

$ createdb -E UNICODE mydatabase

To list the available databases and see what kind of charsets they support you can run psql -l.

$ psql -l

       List of databases

  Name   | Owner  | Encoding

-----------+----------+-----------

 nextgen  | postgres | UNICODE

 template0 | postgres | SQL_ASCII

 template1 | postgres | SQL_ASCII

(3 rows)

MySQL

From version 4.1.1 MySQL has support for unicode. To convert your existing database to utf-8 do the following:

Stop mysql

mysqladmin -uroot -p<password> shutdown

You need to run this to convert your database

myisamchk -r -q --set-character-set=utf-8

 /path/to/mysql_data_files/database/ez*.MYI

Example

myisamchk -r -q --set-character-set=utf-8

 /var/lib/mysql/nextgen/ez*.MYI

If you create a new database you can ( from MySQL version 4.1.1 ) use:

create database nextgen character set utf8;

Setting charsets in Exponential

To make Exponential handle Unicode properly you need to configure which charsets the different layers use. You will need to configure database, internal locale and default template charset. This is configured in the files:

  • site.ini
  • i18n.ini
  • template.ini
site.ini:

[DatabaseSettings]

Charset=utf-8

 

i18n.ini:

[CharacterSettings]

Charset=utf-8

 

template.ini:

[CharsetSettings]

DefaultTemplateCharset=utf-8

That's it. You can now enjoy Unicode characters with Exponential.

Known problems

The search engine does not properly index languages which does not use space as word separator, like chinese and japanese. Update: The current unstable (svn) version has support for this. It will be standard from version 3.2.

Comments

can't override the i18n.ini file

step by step doc: move to utf-8

Hello!

I'm running on Exponential 3.5.1. After adding a lot of content, I found that in some cases german charakters dont work correctly in some cases!

See:

http://ez.no/community/forum/setu...ative_image_text_and_charset_problem

http://ez.no/community/forum/setu...omega_omega_not_working_in_xml_field

The solution seems to be, to move to UTF-8 encoding

I have learnd that I have to do the following
1. Update mySQL from 4.0.x to 4.1
2. Update DB-Tables
3. Change ini-Settings

Does anyone have a step-by-step Doku how this is done?

NEED URGENT HELP!

Thanx,
Peter

Does myisamchk really update all tables?

After running this command, the database tables do not seem to get updated. Judging from the mysql manual and command output, it seems like only the indexes get updated.

Could somebody please clarify this issue? TIA.

Set-collation, not set-character-set

It may be beneficial to include that as of 4.1.11 and later, set-collation is to be used, rather than set-character-set.

From: http://dev.mysql.com/doc/mysql/en/news-4-1-11.html

<quote>The --set-character-set option for myisamchk was changed to --set-collation. The value needed for specifying how to sort indexes is a collation name, not a character set name. (Bug #8349)</quote>

Convert to Unicode

What i did to convert the database instead of the scripts, was to do a dump of the database, then replace the strings <em>latin1</em> with <em>utf8</em>. (Important to use unicode editor when doing the replace) And then reimport the database.

This seems to work. The database was clean, so i don't know how this works for databases with lots of content.

MySQL 4.1.3 update

The myisamchk command no longer works as the characterset names have changed within MySQL.
To convert to utf-8 unicode ci you need to use.



myisamchk -r -q --set-character-set=utf8_general_ci mydb/ez*.MYI


utf-8 convert

Hi,

first of all: If there is really no good reason to convert the charset, just don't do it!!

If you have an existing db which uses iso-8859-1 encoding and you want to switch to utf-8 you have to convert your db table content too.

Be aware: In some mysql versions there are bugs belonging to the utf-8 charset! (just search in the forums of ez.no and mysql.com!)

So, do the described steps above and write an "update.php" script.

Example code:



function &db_utf_query($selectsql ='', $tablesql='', $wheresql='', $limitsql='', $ordersql='')


{


   $db =& eZDB::instance();


   $db->setIsSQLOutputEnabled( false );


        $returnArray = $db->arrayQuery( "SELECT 

                $selectsql


        FROM


         $tablesql


         $wheresql


         $ordersql


         $limitsql


   " );


    

   return $returnArray;


}


 

$codepage = eZCodePage::instance("iso-8859-1", false);


 

$db =& eZDB::instance();


$db->setIsSQLOutputEnabled( false );


 

include_once ('lib/ezi18n/classes/ezutf8codec.php');


include_once ('lib/ezi18n/classes/ezcodepage.php');


 

$codepage = eZCodePage::instance("iso-8859-1", false);


 

$db =& eZDB::instance();


$db->setIsSQLOutputEnabled( false );


 

 

$selectsql      = "id, enumelement, enumvalue";


$tablesql       = "ezenumvalue";


$wheresql       = "";


$limitsql       = "";


$ordersql       = "";


 

 

$sqlarray = db_utf_query($selectsql, $tablesql, $wheresql, $limitsql, $ordersql);


 

$count_elements = count ($sqlarray);


echo "\n\n\n----------------------------------\n";


echo " Updating $tablesql - $count_elements items \n";


echo "----------------------------------\n";


 

 

if (count($sqlarray) > 0)


{


        foreach ($sqlarray as $item)


        {


                


                $update_item = $item;


                


                $update_item['enumelement'] = $codepage->convertString ($item['enumelement']);


                $update_item['enumvalue'] = $codepage->convertString ($item['enumvalue']);


                        


                $sql = "UPDATE 

                                `ezenumvalue` 

                        SET 

                                `enumelement` = '".$update_item['enumelement']."',


                                `enumvalue` = '".$update_item['enumvalue']."'


                        WHERE 

                                `id` = '".$update_item['id']."'";


                        


                $db->query($sql);


        }       


}


 

 

$selectsql      = "*";


$tablesql       = "ezcontentobject_attribute";


$wheresql       = " WHERE `data_text` NOT LIKE \"<?xml%\" AND data_text != \"\" ";


$limitsql       = "";


$ordersql       = "";


 

$sqlarray = db_utf_query($selectsql, $tablesql, $wheresql, $limitsql, $ordersql);


 

$count_elements = count ($sqlarray);


echo "\n\n\n----------------------------------\n";


echo " Updating $tablesql - $count_elements items \n";


echo "----------------------------------\n";


 

 

$i = 1;


if (count($sqlarray) > 0)


{


        foreach ($sqlarray as $item)


        {


                


                $update_item = $item;


                


                // echo "id: " . $item['id'] . " / Inhalt: " . $item['keyword'] . "\n";


                $update_item['data_text'] = $codepage->convertString ($item['data_text']);


                // echo $update_item['keyword'] . "\n";


                        


                $sql = "UPDATE 

                                `ezcontentobject_attribute` 

                        SET 

                                `data_text` = '".$update_item['data_text']."'


                        WHERE 

                                `id` = '".$update_item['id']."' 

                                AND `version` = '".$update_item['version']."' 

                                 ";


                        


                $db->query($sql);


                


                if ($i % 1000 == 0)


                        echo " $i. ";


                $i++;


        }       


}


 

$selectsql      = "*";


$tablesql       = "ezcontentobject_name";


$wheresql       = "";


$limitsql       = "";


$ordersql       = "";


 

$sqlarray = db_utf_query($selectsql, $tablesql, $wheresql, $limitsql, $ordersql);


 

$count_elements = count ($sqlarray);


echo "\n\n\n----------------------------------\n";


echo " Updating $tablesql - $count_elements items \n";


echo "----------------------------------\n";


 

 

$i = 1;


if (count($sqlarray) > 0)


{


        foreach ($sqlarray as $item)


        {


                


                $update_item = $item;


                


                // echo "id: " . $item['id'] . " / Inhalt: " . $item['keyword'] . "\n";


                $update_item['name'] = $codepage->convertString ($item['name']);


                // echo $update_item['keyword'] . "\n";


                        


                $sql = "UPDATE 

                                `ezcontentobject_name` 

                        SET 

                                `name` = '".$update_item['name']."'


                        WHERE 

                                `contentobject_id` = '".$update_item['contentobject_id']."' 

                                AND `content_version` = '".$update_item['content_version']."' 

                                 AND `content_translation` = '".$update_item['content_translation']."' 

                                 ";


                        


                $db->query($sql);


                


                if ($i % 500 == 0)


                        echo " $i. ";


                $i++;


        }       


}




(and so on)

If anyone needs such a convert script I will make a contribution, write to emil.webber@gmx.at.

Kind regards,
Georg.

Log in or create a user account to comment.

Contents

Configuration

Configuration
    WebDAV setup
    Exponential running on a CGI version o...
    Path prefix
    Locale Settings
    Introduction
    Configuration files
    Site access
    Common settings
    Multi Site
    Directory structure
    Language and charset
       Installing a Language pack
       Multilanguage site
       Unicode with Exponential
       Creating a new translation
       How to setup a Multilingual Site
    Cron jobs
    Login handler
    Search engine
    Tips & Tricks
Security
Optimization
Backup & Restore
Troubleshooting


Created

12/06/2003
10:16:50 am
by B�rd Farstad

Last updated

21/07/2004
10:20:26 pm
by Bj�rn Dieding@xrow.de

Authors

B�rd Farstad
Ole Morten Halvorsen
Bj�rn Dieding@xrow.de



This page is part of the Exponential documentation. The documentation is available under the GNU Free Documentation License. All contributions will be released under the terms of this license.