Bloblets

[Ruslan Zasukhin]

On 11/20/05 3:47 PM, "Ed Kleban" <Ed at Kleban.com> wrote:

>> We did have such feature in 1.x but nobody have use it,
>> So we have not open it for v2 for now.
>> 
> 
> That is most unfortunate, since it means I will have to do so manually
> outside of the database if I want to avoid unneccessary disk access -- which
> is one of the key things I'm looking to the database to manage for me so
> that I don't have to.  Specifically, I need to store a large number of
> strings that will range in size from typically, say... 2 MB to 20 MB.   I
> then need to perform completely random accesses to pull out substrings from
> those strings, the position and length of which are stored in database
> tables.  

Hmm. 3 times Hmm. 

You sure this is best possible solution ?!
I doubt.


> Sometimes some of those 2-20 MB string will be in memory already,
> but other times they will only be on disk and likely not in either
> Valentina's or the OS's cache.
> 
> Since Valentina doesn't support substring access, I'll have to store every
> such string in a separate file outside of the database, store the filenames
> in V2, and use the OS to perform the random accesses -- or read in the whole
> thing and suffer the performance consequences.  Not a difficult or
> unmanageable task, but like I said... unfortunate.
 
>> Also always remember about disk, cache, ...
>> 
>> Here not always faster to read 100 bytes then then e.g. 4Kb
>> 
> 
> I fully expect to have to do a disk access to pull in a cache page or two,
> both into the OS cache as well as Valentina's, if I need a 100 byte
> substring from a 10 MB source.  I don't expect to have to read 10 MB worth
> of pages when I only need 100 bytes.

10MB yes, this is many.

Just I think can be better solution to avoid such huge data of ATOMIC sets.


-- 
Best regards,

Ruslan Zasukhin
VP Engineering and New Technology
Paradigma Software, Inc

Valentina - Joining Worlds of Information
http://www.paradigmasoft.com

[I feel the need: the need for speed]