Some news on optimizations in indexing for next 5.6, x10 - x15 times speed up for Strings

[Ruslan Zasukhin]

Hi Valentina Developers!

a quick note to share latest results.

BENCH DESCRIPTION:

We have here bench,
    which build index for a STRING[120] column.
    it contains big enough strings  80-90 chars.
    **1 millions** records in table. Only 1 field.
    all records have unique random STR value.

Macbook Pro, i7, 2.2Hhz,  8GB RAM.
HDD 7200 rpm, with speed about 50M/sec.  (yep slow enough)


RESULTS:

** Valentina DB 5.5.8 (latest release) build index in about 120 - 130 sec.

** First optimization, speed up it to 70-80 sec.

** Sqlite, which uses single-byte encoding have show 35-40 sec,
What is okay because x2 smaller files with big strings.

I underline 1 field in table, so we have fair comparison of build index
algs.  If Table will have N fields, then Valentina gets additional benefit
because of columnar format


** ONE MORE optimization enabled/implemented, and now
    Valentina DB 5.6 beta do job in 7-8 seconds.


Pretty cool  :-) 
Best possible time for this task is about 5 seconds.



Same optimization now is enabled for
    float, double, ullong, llong numeric types.

For other numeric types (<=4 bytes) optimization was implemented
far far ago.


-- 
Best regards,

Ruslan Zasukhin
VP Engineering and New Technology
Paradigma Software, Inc

Valentina - Joining Worlds of Information
http://www.paradigmasoft.com

[I feel the need: the need for speed]