importing from binaryStream into BLOB field

Tue Jul 23 10:36:45 CDT 2013

And a clarification about speed: the two method below both give the correct results for samples(i) when reading the unformatted file:

        //read as string (Fast!)
        m=bis.Read(nDataBytes)
        for j as integer=1 to nData
          samples(j) = m.DoubleValue((j-1)*8)
        next

        //read as real values (slow!)
        for j as integer=1 to nData
          m.DoubleValue((j-1)*8)=bis.ReadDouble
          samples(j) = m.DoubleValue((j-1)*8)
        next

but the first is so much faster! If only I could store the memory block m in the BLOB and retrieve the samples later…

Franco

On Jul 23, 2013, at 5:22 PM, Franco Vaccari <vaccari at units.it> wrote:

>> On Jul 23, 2013, at 4:08 PM, Beatrix Willius <bwillius at gmx.de> wrote:
>> 
>>> The code looks ultra complicated. Is there a reason why you can't use simply read and write for some bytes?
> 
> I may be wrong, but I got the idea that the easiest way to store thousands of doubles in a single record of the table was to keep them in a memoryblock and store this in a BLOB. I'm no expert here, if there are better ways of doing this, I'll be happy to explore…
> 
> 
> I try to clarify what I'm doing:
> 
> I have files written by some external computational code (fortran unformatted output). I need to import those data in a DB. Structure of file is:
> 
> - global file header with number of seismograms (N)
> 
> - header of seismogram n. 1
> - seismogram n.1      <-- vector with thousands of values
> 
> - header of seismogram n. 2
> - seismogram n.2      <-- vector with thousands of values
> 
> …
> …
> 
> - header of seismogram (n. N)
> - seismogram n.N      <-- vector with thousands of values
> 
> 
> Each seismogram becomes a record in my DB table, with fields for the metadata (position, magnitude, component etc) read from the header. The header is read in a loop over the seismograms, and after importing all the seismograms, the queries on the header values get the expected results. That means that I'm correctly pointing to the correct bytes.
> 
> I want to store the vectors of data in a BLOB file. I will never need to search for those values, I only need to access the vector as a whole  when I select a given seismogram, to plot it. So the idea is to prepare a memoryBlock with a clone of the bytes taken from the file, for the part where the vectors are stored. And here is the problem.
> 
> READING FROM FILE INTO THE MEMORYBLOCK: 
> 
>        // samples
>        bis.Position=endPosition+376 // x()
>        redim samples(nData)
>        dim m as MemoryBlock
>        m=new MemoryBlock(nDataBytes) 
>        m.LittleEndian=true
> 
> CHECKING THE VALUES IN THE DEBUGGER (this code is just for debugging):
> 
>        // m.LittleEndian=true
>        //read as real values (slow!)
>        for j as integer=1 to nData
>          m.DoubleValue((j-1)*8)=bis.ReadDouble
>          samples(j) = m.DoubleValue((j-1)*8)
>        next
> 
> I properly read the data. 4096 samples, nDatabytes is 4096*8 (samples are doubles, 8 bytes each), read 4096*8 bytes in a memoryblock, and from the memory block I recover the correct values in samples(i). (e.g. 0.36, 0.45, -0.4 etc). So READING is ok. 
> 
> 
> STORING  THE MEMORYBLOCK INTO BLOB:
> 
> Immediately after checking that I can recover the data values from the memoryblock, I store in the BLOB field the very same memoryblock m:
> 
>        tblSeismograms.BlobField("seisSamples").WriteRawData m
> 
> 
> READING FROM BLOB:
> 
> I get the cursor with query:  select recID,seisAmaxa,seisNsamples,seisSamples from Seismograms where recID = 170
> 
> I try to retrieve the values from the BLOB field:
> 
>    m=new MemoryBlock(nDataBytes)
>    m.LittleEndian=true
>    m=inCursor.BlobField("seisSamples").ReadRawData
>    dim samples(-1) as double
>    dim istop as integer= m.size/8
>    for i as integer=1 to iStop
>      samples.append m.DoubleValue(i-1)*8
>    next
> 
> iStop, taken from the memoryblock size,  get the expected value (4096), so the length of the memoryBlock is still correct. But values of samples are complete garbage. An example being:
> 
> -10,657,301,492,307,700,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000
> 
> (that's ONE value as shown in the debugger, sample(2), with enough thousands separators…)
> 
> So obviously bytes are no longer as they were before storing them in the BLOB. Why?
> 
> I'm banging my head on the wall since two days on this. ReadData and ReadRawData (what's the difference?) return a string, and that string is written to the memoryblock, so I suspect encodings could play a role. I'm reading just numbers, so I expect that ASCII would do it. I tried converting into UTF16, before writing to BLOB, and back to ASCII after reading but nothing changed. Tried several combinations, no success.
> 
> Could the BLOB segment size influence this? I create that with   
> 
> fldSeisSamples = tblSeismograms.CreateBLOBField("seisSamples",1024)
> 
> But I tried 4096 and nothing changes…
> 
> Lost…
> 
> Franco
> 
>> 
>> And no, there shouldn't be any encoding trouble for raw data or memoryblocks. The encoding should be nil. I'm using a Blob for saving text exactly for this reason.
>> 
>> Can you give us more details about what you write and what you get?
>> 
>> On 23.07.2013, at 13:15, Franco Vaccari <vaccari at units.it> wrote:
>> 
>>> I suppose it might be a problem of encodings, as I believe the memoryBlock is read/written as a string, although I guessed that using WriteRawData and ReadRawData should free me from the encodings trouble. I tried playing with convertEncoding when reading/writing the memoryBlock, with no improvements. 
>>> 
>>> What am I doing wrong?
>>> 
>>> What is the difference between ReadData and ReadRawData, (and Write as well)? I can't find ReadRawData in the Wiki, but it autocompletes in Xojo for the BlobField…
>>> 
>>> Is there a better way to clone the sequence of bytes of the file I'm reading, and store it in the BLOB field?
>> 
>> Mit freundlichen Grüßen/Regards
>> 
>> Trixi Willius
>> 
>> http://www.mothsoftware.com
>> Mail Archiver X: The email archiving solution for professionals
>> 
>> _______________________________________________
>> Valentina mailing list
>> Valentina at lists.macserve.net
>> http://lists.macserve.net/mailman/listinfo/valentina
> 
> _______________________________________________
> Valentina mailing list
> Valentina at lists.macserve.net
> http://lists.macserve.net/mailman/listinfo/valentina