Bloblets

[Ed Kleban]

On 11/20/05 2:17 AM, "Ruslan Zasukhin" <sunshine at public.kherson.ua> wrote:

> On 11/20/05 10:13 AM, "Ed Kleban" <Ed at Kleban.com> wrote:
> 
>> Is there any way to read just a little bit of data from a Blob or Text
>> without having to read the entire amount of data into memory?
>> 
>> It would be nice to be able to request   100 bytes starting from byte 10293
>> for example.
> 
> We did have such feature in 1.x but nobody have use it,
> So we have not open it for v2 for now.
> 

That is most unfortunate, since it means I will have to do so manually
outside of the database if I want to avoid unneccessary disk access -- which
is one of the key things I'm looking to the database to manage for me so
that I don't have to.  Specifically, I need to store a large number of
strings that will range in size from typically, say... 2 MB to 20 MB.   I
then need to perform completely random accesses to pull out substrings from
those strings, the position and length of which are stored in database
tables.  Sometimes some of those 2-20 MB string will be in memory already,
but other times they will only be on disk and likely not in either
Valentina's or the OS's cache.

Since Valentina doesn't support substring access, I'll have to store every
such string in a separate file outside of the database, store the filenames
in V2, and use the OS to perform the random accesses -- or read in the whole
thing and suffer the performance consequences.  Not a difficult or
unmanageable task, but like I said... unfortunate.

> Also always remember about disk, cache, ...
> 
> Here not always faster to read 100 bytes then then e.g. 4Kb
> 

I fully expect to have to do a disk access to pull in a cache page or two,
both into the OS cache as well as Valentina's, if I need a 100 byte
substring from a 10 MB source.  I don't expect to have to read 10 MB worth
of pages when I only need 100 bytes.