importing from binaryStream into BLOB field

Tue Jul 23 10:22:42 CDT 2013

> On Jul 23, 2013, at 4:08 PM, Beatrix Willius <bwillius at gmx.de> wrote:
> 
>> The code looks ultra complicated. Is there a reason why you can't use simply read and write for some bytes?

I may be wrong, but I got the idea that the easiest way to store thousands of doubles in a single record of the table was to keep them in a memoryblock and store this in a BLOB. I'm no expert here, if there are better ways of doing this, I'll be happy to explore…

I try to clarify what I'm doing:

I have files written by some external computational code (fortran unformatted output). I need to import those data in a DB. Structure of file is:

- global file header with number of seismograms (N)

- header of seismogram n. 1
- seismogram n.1      <-- vector with thousands of values

- header of seismogram n. 2
- seismogram n.2      <-- vector with thousands of values

…
…

- header of seismogram (n. N)
- seismogram n.N      <-- vector with thousands of values

Each seismogram becomes a record in my DB table, with fields for the metadata (position, magnitude, component etc) read from the header. The header is read in a loop over the seismograms, and after importing all the seismograms, the queries on the header values get the expected results. That means that I'm correctly pointing to the correct bytes.

I want to store the vectors of data in a BLOB file. I will never need to search for those values, I only need to access the vector as a whole  when I select a given seismogram, to plot it. So the idea is to prepare a memoryBlock with a clone of the bytes taken from the file, for the part where the vectors are stored. And here is the problem.

READING FROM FILE INTO THE MEMORYBLOCK: 

        // samples
        bis.Position=endPosition+376 // x()
        redim samples(nData)
        dim m as MemoryBlock
        m=new MemoryBlock(nDataBytes) 
        m.LittleEndian=true

CHECKING THE VALUES IN THE DEBUGGER (this code is just for debugging):

        // m.LittleEndian=true
        //read as real values (slow!)
        for j as integer=1 to nData
          m.DoubleValue((j-1)*8)=bis.ReadDouble
          samples(j) = m.DoubleValue((j-1)*8)
        next

I properly read the data. 4096 samples, nDatabytes is 4096*8 (samples are doubles, 8 bytes each), read 4096*8 bytes in a memoryblock, and from the memory block I recover the correct values in samples(i). (e.g. 0.36, 0.45, -0.4 etc). So READING is ok. 

STORING  THE MEMORYBLOCK INTO BLOB:

Immediately after checking that I can recover the data values from the memoryblock, I store in the BLOB field the very same memoryblock m:

        tblSeismograms.BlobField("seisSamples").WriteRawData m

READING FROM BLOB:

I get the cursor with query:  select recID,seisAmaxa,seisNsamples,seisSamples from Seismograms where recID = 170

I try to retrieve the values from the BLOB field:

    m=new MemoryBlock(nDataBytes)
    m.LittleEndian=true
    m=inCursor.BlobField("seisSamples").ReadRawData
    dim samples(-1) as double
    dim istop as integer= m.size/8
    for i as integer=1 to iStop
      samples.append m.DoubleValue(i-1)*8
    next

iStop, taken from the memoryblock size,  get the expected value (4096), so the length of the memoryBlock is still correct. But values of samples are complete garbage. An example being:

-10,657,301,492,307,700,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000

(that's ONE value as shown in the debugger, sample(2), with enough thousands separators…)

So obviously bytes are no longer as they were before storing them in the BLOB. Why?

I'm banging my head on the wall since two days on this. ReadData and ReadRawData (what's the difference?) return a string, and that string is written to the memoryblock, so I suspect encodings could play a role. I'm reading just numbers, so I expect that ASCII would do it. I tried converting into UTF16, before writing to BLOB, and back to ASCII after reading but nothing changed. Tried several combinations, no success.

Could the BLOB segment size influence this? I create that with   

fldSeisSamples = tblSeismograms.CreateBLOBField("seisSamples",1024)

But I tried 4096 and nothing changes…

Lost…

Franco

> 
> And no, there shouldn't be any encoding trouble for raw data or memoryblocks. The encoding should be nil. I'm using a Blob for saving text exactly for this reason.
> 
> Can you give us more details about what you write and what you get?
> 
> On 23.07.2013, at 13:15, Franco Vaccari <vaccari at units.it> wrote:
> 
>> I suppose it might be a problem of encodings, as I believe the memoryBlock is read/written as a string, although I guessed that using WriteRawData and ReadRawData should free me from the encodings trouble. I tried playing with convertEncoding when reading/writing the memoryBlock, with no improvements. 
>> 
>> What am I doing wrong?
>> 
>> What is the difference between ReadData and ReadRawData, (and Write as well)? I can't find ReadRawData in the Wiki, but it autocompletes in Xojo for the BlobField…
>> 
>> Is there a better way to clone the sequence of bytes of the file I'm reading, and store it in the BLOB field?
> 
> Mit freundlichen Grüßen/Regards
> 
> Trixi Willius
> 
> http://www.mothsoftware.com
> Mail Archiver X: The email archiving solution for professionals
> 
> _______________________________________________
> Valentina mailing list
> Valentina at lists.macserve.net
> http://lists.macserve.net/mailman/listinfo/valentina