Tutorial Using Unicode in Visual Basic 6.0 Updated 25-August-2008 03:48

 

Copyright © 2003 - 2008 Unisuite/CyberActiveX Unicode and the Unicode Logo are trademarks of Unicode, Inc. Microsoft and Visual Basic are trademarks of Microsoft Corporation This is a work in progress so expect to see updates often. The latest version can always be found at www.unisuite.com Download a self-contained Compiled HTML(CHM) for off-line reference at http://www.unisuite.com/download/UnicodeTutorial.chm Send comments, suggestions, corrections, or contribute Unicode Vb source code to: webmaster@unisuite.com

Although Visual Basic 6.0 stores strings internally as Unicode(UTF-16) it has several limitations:

1. Ships with ANSI only controls (Label, Textbox, etc.).
2. Properties Window in IDE is ANSI only. Unicode strings are displayed as '????'
3. PropertyBag automatically converts Unicode strings to ANSI.
4. Clipboard functions are ANSI only.
5. Menus are ANSI only.

The purpose of this tutorial is to resolve these issues and provide working VB code solutions. The level of difficulty of these solutions vary but in general require intimate knowledge of ActiveX Controls and Classes. Subclassing and API programming are a must to gain functionality that Vb does not directly support.

The amount of information gathered during development of Unicode aware controls was so overwhelming that it made sense to organize it and this Tutorial proved to be an ideal place to bring eveything together under one roof.

Tutorial Development Tools
Microsoft® Frontpage® 2003
Microsoft® Platform SDK Feb 2007 Edition for Vista
Visual Basic 6.0 - Service Pack 6 It is highly recommended that you use the Latest Vb6 runtime libraries. Release Date: 29-Mar-2004 Size: 1,039.8 Kb List of Fixes VB6.0-KB290887-X86.exe

 

Note: Dates in this tutorial are displayed as dd-mmm-yyyy (Example: 11-Mar-2004) to eliminate any ambiguities.

*  These issues are resolved in Vb.Net although you will have to go through a learning curve to get up to speed with the language.
    Review these tables to determine the minimum system requirements:

1. System Requirements Visual Basic .NET 2003 Standard.
2. System Requirements Visual Studio .NET 2003.

This flowchart shows basic program flow of:

1. Send ANSI/DBCS text to ANSI Controls or ANSI API.
2. Send Unicode text to Unicode Controls
3. Send Unicode text to ANSI/Unicode API using MSLU.
4. Convert UTF8 to Unicode.
5. Convert ANSI/DBCS to Unicode.
6. Get Charset/Codepage from LCID.

Note: You should have a utility on your system called CharMap.Exe which will allow you to browse and select Unicode characters.
 

SBCS(Single-Byte) and DBCS(Double-Byte) Character Sets are different character sets from Unicode.
 

Character codes for "A" in ANSI, Unicode, and DBCS

CharSet from Platform SDK wingdi.h

* DBCS - Double-Byte Character Set

DBCS is actually not the correct terminology for what Windows uses. It is actually MBCS where a character can be 1 or 2 bytes. To illustrate this consider the following code which will take a Unicode string of English and Chinese characters, convert to a byte array of MBCS Chinese, dump the byte array to the immediate window, and finally convert it back to a Unicode string to display in a Unicode aware textbox. The byte array when converted using Chinese(PRC) LCID = 2052 contains single bytes for the english characters and double bytes for the Unicode characters. This proves that it is MBCS and not DBCS:

The following demos shows how to display Chinese on an English-U.S. system without changing your Regional settings 

Example of VbTextBox using DBCS Charset 134, CHINESE_GB2312

• Set your TextBox to any Font that supports script CHINESE_GB2312 (Listview1.Font.Charset=134). Example "Arial Unicode MS". Make sure you select the CHINESE_GB2312 script from the Font Dialog box.

• Add this DBCS string:
  TextBox1 = "CHS: ¡¡ÄãÕæµÄÄܹ»ÔÚÎı¾¿òÖÐʹÓñ³¾°Í¼Æ¬£¬"
  

• Warning: If you apply an XP Theme to the TextBox below via an IDE Manifest it will display ANSI instead of DBCS.

Example of Vb ListView using DBCS Charset 134, CHINESE_GB2312
 

• Set your Listiew to any Font that supports script CHINESE_GB2312 (Listview1.Font.Charset=134). Example "Arial Unicode MS". Make sure you select the CHINESE_GB2312 script from the Font Dialog box.

• In report mode add this DBCS string:
  ListItems.Add , , "CHS: ¡¡ÄãÕæµÄÄܹ»ÔÚÎı¾¿òÖÐʹÓñ³¾°Í¼Æ¬£¬"
  

.

See MSDN for more information about DBCS:
Issues Specific to the Double-Byte Character Set (DBCS)
ANSI, DBCS, and Unicode: Definitions
Calling Windows API Functions
DBCS Sort Order and String Comparison
DBCS String Manipulation Functions
DBCS-Enabled KeyPress Event
Designing an International-Aware User Interface
Font, Display and Print Considerations in a DBCS Environment
Identifiers in a DBCS Environment
Processing Files That Use Double-Byte Characters
 

DBCS String Conversion

StrConv Function

The global options of the StrConv function are converting uppercase to lowercase, and vice versa. In addition to those options, the function has several DBCS-specific options. For example, you can convert narrow letters to wide letters by specifying vbWide in the second argument of this function. You can convert one character type to another, such as hiragana to katakana in Japanese. StrConv enables you to specify a LocaleID for the string, if different than the system's LocaleID.

You can also use the StrConv function to convert Unicode characters to ANSI/DBCS characters, and vice versa. Usually, a string in Visual Basic consists of Unicode characters. When you need to handle strings in ANSI/DBCS (for example, to calculate the number of bytes in a string before writing the string into a file), you can use this functionality of the StrConv function.

The easiest way to add Unicode test strings to your project is to make a resource file with a Unicode aware editor and compile it with RC.exe. That way you can test your controls without having to cut/paste the strings when you need them.
Use Notepad if you are on NT or later, WordPad or UltraEdit if using Win9x.
Download the complete resource file with source here.

"Welcome" in several languages

"Hello" in several languages
* Needs Code2000 Font to see this

Other methods for creating a string at Vb Runtime:

Note: Under the hood StrConv inserts a BOM (FEFF) before the CJK Unified Ideographs.

More stuff to play with:

XP now supports a total of 136 locales, which includes the 126 locales supported by Windows 2000 and adds the following:

•  Divehi – Maldives
•  Galician – Spain
•  Gujarati – India
•  Kannada – India
•  Kyrgyz (Cyrillic) – Republic of Kyrgyzstan
•  Mongolian (Cyrillic) – Mongolia
•  Punjabi (Gurmukhi) – India
•  Syriac – Syria
•  Telugu – India
•  Invariant locale

Other international features New to XP:

• New Indic and Arabic scripts(Gujarati, Gurmukhi, Telugu, Kannada, Syriac, Divehi).
• More robust font display for East Asian languages.
• Improved Regional Settings support.
• Largely improved MUI support.
• New Location (GEO).
• Support for GB18030.

New Locales in Windows XP Service Pack 2

Windows XP Service Pack 2 introduces 25 additional locales and another 11 with Service Pack 2 Update:

These locales are automatically installed when you update Windows XP to SP2. You can select new locales in Regional and Language Options. They are not supported in Windows Server 2003.

International Locale Codes
 

Click Icon to your left to Load Table.
 Please Wait...

Related resources

• FAQs about Locales & Languages
• ISO 8859 character sets
• Character sets recognized by Internet Explorer
• NLS Information for Microsoft Windows XP
• Windows codepages
• Locales and Language Groups
• List of Windows XP's Three Letter Acronyms for Languages
• Windows 2000 - List of Locale IDs and Language Groups
• Windows XP - List of Locale IDs, Input Locale, and Language Collection

Also see Table above

Table of Known Code Pages

* DBCS - Double-Byte Character Set
* This is a pseudo codepage. There is no corresponding NLS file. This code page ID can only be used with WideCharToMultiByte() and MultiByteToWideChar() API calls.

Fonts used on Windows XP-SP1:

Win2000/XP/Office2000/Office2003 should already have Arial Unicode MS installed.
Win98 users will need a Unicode font to render Unicode glyphs.

Also see Multilingual Unicode TrueType Fonts on the Internet.

Sample Font Support for several Languages
Test Platform WinXP-SP1

Sample	Arial Unicode MS	Code 2000	* Microsoft Tahoma or Arial	* Microsoft Sans Serif	Bitstream Cyberbit	TITUS Cyberbit Basic
ARA: مـرحبــاً
CHS: 欢迎
CHT: 歡迎
GEO: სასურველი
GRK: Καλώς ήλθατε
HEB: בִּרוּבִים חַבָּאִים
HIN: रवागत
JPN: よろてそ
KOR: 여보세요
PAN: ਜੀ ਆਇਆਂ ਨੂੰ
RUS: Добро пожаловать
TAM: 쏅얧주ø
THA: การต้อนรับ
URD: स्वागत
VIE: tính từ

* Tahoma, Arial do not support all these languages but works due to Uniscribe Font Fallback. Apparently MS San Serif does not use or support Font Fallback. Font Fallback is only available on Platforms Win2000 or later.

Microsoft Layer for Unicode Technology(UNICOWS.DLL)

While you can make a separate programs for specific platforms it is often desirable to make one program that will work on all platforms. By using Microsoft Layer for Unicode Technology (Unicows.DLL, 240kb), a single executable can run on both NT-based and Win9x Platforms. In this case you can use DrawTextW or TextOutW Lib "Unicows" for all platforms.

1. The Unicows.Dll forwards calls to the system API if you are running on NT, 2000, or XP platforms.

2. MSLU does not support the display of characters that the system cannot display. Therefore do not expect to see Chinese under Win98 English even though you have installed a font that supports Chinese(Arial Unicode MS for example). For more info see Newsgroup microsoft.public.platformsdk.mslayerforunicode and http://trigeminal.com/usenet/usenet035.asp.

Use this conditional compilation directive to test your code with and without Unicows:

#If UNICOWS Then Public Declare Function DrawTextW Lib "UNICOWS" (ByVal hdc As Long, ByVal lpStr As Long, ByVal nCount As Long, lpRect As RECT, ByVal wFormat As Long) As Long Public Declare Function MultiByteToWideChar Lib "UNICOWS" (ByVal CodePage As Long, ByVal dwFlags As Long, ByRef lpMultiByteStr As Any, ByVal cchMultiByte As Long, ByVal lpWideCharStr As Long, ByVal cchWideChar As Long) As Long Public Declare Function TextOutW Lib "UNICOWS" (ByVal hdc As Long, ByVal x As Long, ByVal y As Long, ByVal lpString As Long, ByVal nCount As Long) As Long Public Declare Function WideCharToMultiByte Lib "UNICOWS" (ByVal CodePage As Long, ByVal dwFlags As Long, ByVal lpWideCharStr As Long, ByVal cchWideChar As Long, ByRef lpMultiByteStr As Any, ByVal cchMultiByte As Long, ByVal lpDefaultChar As String, ByVal lpUsedDefaultChar As Long) As Long #Else Public Declare Function DrawTextW Lib "user32" (ByVal hdc As Long, ByVal lpStr As Long, ByVal nCount As Long, lpRect As RECT, ByVal wFormat As Long) As Long Public Declare Function MultiByteToWideChar Lib "kernel32" (ByVal CodePage As Long, ByVal dwFlags As Long, ByRef lpMultiByteStr As Any, ByVal cchMultiByte As Long, ByVal lpWideCharStr As Long, ByVal cchWideChar As Long) As Long Public Declare Function TextOutW Lib "gdi32" (ByVal hdc As Long, ByVal x As Long, ByVal y As Long, ByVal lpString As Long, ByVal nCount As Long) As Long Public Declare Function WideCharToMultiByte Lib "kernel32" (ByVal CodePage As Long, ByVal dwFlags As Long, ByVal lpWideCharStr As Long, ByVal cchWideChar As Long, ByRef lpMultiByteStr As Any, ByVal cchMultiByte As Long, ByVal lpDefaultChar As String, ByVal lpUsedDefaultChar As Long) As Long #End If

Under Project/Properties/Make set conditional compilation arguments to UNICOWS = -1 or UNICOWS = 0.

Note:

You may not need MSLU at all if your program uses only DrawText or TextOut. In this case you can simply wrap the ANSI and Wide versions into a Sub. Do not expect to see Unicode on Win9x platforms even if you are using a Unicode font such as Arial Unicode MS:

Uniscribe Architecture

In 1999 Microsoft introduced Uniscribe, a Windows system-level component that could take advantage of OpenType fonts. Microsoft Windows 2000 and applications Internet Explorer 5 and Office 2000 were released with support for Uniscribe built in.

For Windows 2000 and later, supports the processing of complex scripts, that is, those scripts that need special processing to properly render them. It includes a subset of the features found in GDI+ in Windows 2000 and Windows XP. The rules governing the shaping and positioning of glyphs are specified and catalogued in The Unicode Standard: Worldwide Character Encoding, Version 2.0, Addison-Wesley Publishing Company.
http://msdn.microsoft.com/library/en-us/mslu/winprog/other_existing_unicode_support.asp?frame=true
http://www.microsoft.com/msj/1198/multilang/multilang.aspx

A complex script has at least one of the following attributes:

• Allows bidirectional rendering.
• Has contextual shaping.
• Has combining characters.
• Has specialized word-breaking and justification rules.
• Filters out illegal character combinations.

You may wonder how WinXP displays Unicode correctly even when you haven't selected a Font which supports all the required characters.
"Font fallback: this mechanism, made available through Uniscribe (see section on Complex Scripts Support), provides a fallback font (or a default font) when dealing with complex scripts. If the selected font face does not include any glyphs for the complex script that is about to be displayed, Uniscribe selects a default hardcoded font for the given script. For example, if you have Hindi text and the font is Courier, then Uniscribe will use the Mangal font. This technique is internal to Uniscribe and developers can not add additional fonts to the list of fallback fonts."
Note: Set flags to SSA_FALLBACK

Uniscribe is installed with Internet Explorer 5.0 or later, MS Office, Win2000, WinXP. Here are some versions of Uniscribe I found (including one found on Win98SE):

Best results in tests run on Win98SE has been with Uniscribe version 1.0405.2415.1. Microsoft Office 2003 version has not been tested yet.

A Vb wrapper for this library can be found in Internationalization with Visual Basic by Michael S. Kaplan. It comes with a CD containing sample sourcecode. The sample includes a Uniscribe-aware version of ExtTextOutW. More info here.
 

A more complex C++ example can be found at "Supporting Multilanguage Text Layout and Complex Scripts with Windows NT 5.0". Dont be mislead by 'Windows NT 5.0' in the title because this demo also works on Win98. More info here.

This sample has been update and can be found on the Microsoft® Platform SDK(August 2002 Edition, Windows XP SP1) if you have it installed under C:\Program Files\Microsoft SDK\Samples\winui\globaldev\CSSamp. You may encounter problems compiling this due to missing or outdated files.

To build an application that supports Unicode on all Platforms AND uses Uniscribe you could use something similar to this:

Provides services for applications on international issues, including conversion between code pages, font linking, code page "guessing", line breaking, and more. Installed with Internet Explorer 5.5 or later.
http://msdn.microsoft.com/workshop/misc/mlang/mlang.asp
http://msdn.microsoft.com/workshop/misc/mlang/reference/objects/CMultiLanguage.asp
 

The only Vb wrapper for this library I can find is here.

Provides a programming interface(control) for formatting text. This can be used in lieu of Fm20.Dll Unicode TextBox. No distribution issues and comes with source code.

Rich Edit version	Unicode	New	DLL	XP - SP1	XP	Me	2000	NT	98	95
1.0			Riched32.dll	Emulator	Emulator		Emulator
2.0		Supports Unicode	Riched20.dll							May be installed
3.0		Expanded support for complex scripts, partly due to Uniscribe.	Riched20.dll
4.1		Hyphenation, page rotation, and Text Services Framework (TSF) support.	Msftedit.dll

Links:
http://msdn.microsoft.com/library/psdk/winui/richedit_5a7n.htm
http://msdn.microsoft.com/library/en-us/shellcc/platform/commctls/richedit/richeditcontrols.asp
About Rich Edit Controls