can I query raw bits using bit logic in a huge RS database?

[Ruslan Zasukhin]

On 6/10/12 3:12 PM, "Ruslan Zasukhin" <ruslan_zasukhin at valentina-db.com>
wrote:

> On 6/10/12 2:00 PM, "Aaron Andrew Hunt" <aaronandrewhunt at gmail.com> wrote:
> 
> Hi Aaron,
> 
>> It looks like Valentina can do XOR on the bits, but maybe lack of indexing
>> means the query will be slow ...

I have to think here.

you say about 50-100 bytes per bit-string ...
Let 50

And 1 billion such records ...
    50 * 10^9 bytes = 50 GB

3 billions 
    150 GB

And if use var-length then x2 size average
    300 GB 

This is the size of field  bit-string on disk ...

Having 200MB speed disk, we get
    5 seconds per GB to load - best speed

And 
    300GB * 5 = 1500 seconds  to just load field to do scan
                    = 25 minutes.

Of course yet CPU and RAM tasks overhead ...


Note, that even if computer has 8-16GB of RAM, this not help too much,
because info size only for this single column is 300GB ...


I'd say it is very important split data in your case to achieve time you
need.


-- 
Best regards,

Ruslan Zasukhin
VP Engineering and New Technology
Paradigma Software, Inc

Valentina - Joining Worlds of Information
http://www.paradigmasoft.com

[I feel the need: the need for speed]