[V4RB] Re: Unicode workaround via UTF-16 - problems

[Dave Addey]

Hi Ruslan,

Thanks for the response!  I'm still having a few problems...

> Then I think you need use VarBinary or FixedBainry strings

This sounds ideal! Thanks.

>> In theory, byte-based sorting on an UTF-16 string (stored and referenced as
>> bytes) is a valid sorting process.
> 
> I think this is not correct, Dave.

You're right.  2 bytes is not enough for *all* characters in the world.  And
characters which require more than 2 bytes would "break" my sorting (as
UTF-16 uses 4 bytes to store them, with the first 2 bytes as an identifier).

But, according to IBM...

"All of the most common characters in use for all modern writing systems
are already represented with 2 bytes. Characters in surrogate space take
4 bytes, but as a proportion of all world text they will always be very
rare."

So I should be pretty safe :-)

This quote is from an excellent article I found at:

http://www-106.ibm.com/developerworks/library/utfencodingforms/

> I still think this will not work for SOME HARD languages.

I agree.  But see the quote above :-)

So, I've switched to using VVarBinary rather than VVarChar for fields where
I want to store my UTF-16 data.  My problem comes when I try and sort on
these fields.  If I do the following:

  Dim DBrec as VCursor
  DBrec = LibraryDB.SQLSelect("select * from ImportedSongs order by Album,
TrackNumber")
  
...where Album is a VVarBinary(512) and TrackNumber is a Vlong, then my RB
application crashes when trying to execute this statement, even with only
one record in the ImportedSongs table.

If I remove the "Album" field from the "order by" clause, then the query
works fine.

I have tried creating the Album field with and without indexing, and it
doesn't seem to make a difference.

I create the Album field in the constructor of my ImportedSongs object:

  Name = "ImportedSongs"
  Album = New VVarBinary("Album", 512)

Any idea what I'm doing wrong?  Is it possible to order by VVarBinary
fields?  The crash is very unfriendly when I try ;-)

FYI, I've updated to REALbasic 5.2.2 and Valentina 1.9.8 (for RB5).

Thanks in advance for any help,

Dave.