Unicode workaround via UTF-16 - problems

[Dave Addey]

Hi all,

I've read on this list about Unicode coming in v2.0, and I look forward to
when it does.

In the meantime, I've been trying a workaround.  I'd like to store my UTF-16
strings in Valentina, without Valentina knowing or caring that they are in
this format.  I use REALbasic 5, and can convert back and forth between text
encodings at will, so I'm able to take a random set of bytes, of known
string encoding, and use this string in REALbasic.

In theory, byte-based sorting on an UTF-16 string (stored and referenced as
bytes) is a valid sorting process.  Because I can pretty much always assume
that each character takes 2 bytes in UTF-16, I'm happy that any sort I do
will come back accurately enough sorted for my liking.  Sorting would only
'break' if my string contained characters which are outside of the UTF-16
set, and this is unlikely in normal usage (in my opinion).

But, my problem is this.  When I try and store a Unicode UTF-16 string in a
Valentina string field, I can't do so for any strings that begin with the
null character (&h00).  Strings that start with other characters are fine
(e.g. Japanese characters which use both bytes in UTF-16).  And since most
strings in Western alphabets contain mostly ASCII characters, most of my
UTF-16 strings begin with &h00 .

So, for example:

    LibraryDB.ImportedSongsTable.Album.Value = mysong.Album

...where LibraryDB.ImportedSongsTable.Album is a VVarChar, and mysong.Album
is a UTF-16 string.

In this example, if the first byte of mysong.Album is &h00, the value of
LibraryDB.ImportedSongsTable.Album.Value is nil, even though there are other
characters after the initial &h00 value in mysong.Album .

All I really want is to transfer some bytes into a Valentina field (I'm
assuming this would be a string) and get them back out again, with byte
sorting on this field.  Is there a way to do this in the current release?

This would allow me to add Unicode support to my app before it is available
'native' in Valentina.

BTW, I'm using Valentina 1.9.7, REALBasic 5.2.1 on Mac OS X 10.2.8.

Thanks in advance,

Dave.