Large database, consistently corrupted header when building index for 2 fields, 100% reproducible

Matthew Jew mjew at icnc.com
Tue Oct 7 09:24:38 CDT 2008


Hello,

I have a very large database, which I am building by importing the  
contents of about 60 smaller databases.
Each of the smaller databases is error-free (can be diagnosed  
successfully with no problems.)

However, when my application copies the contents of those databases  
into one very large database
(about 9 million records) two of the indexes are corrupted. (Both  
fields are unsigned long fields.)

This line is from the very verbose diagnosis output:

PAGE (5446, 5440) Corrupted Header (Values Count). Corrupted Header  
(Values count < RecID count). At least one value counter is wrong.

The diagnosis finds no other problems. (No other indexes have  
problems, no tables have problems.)

During the importing, I turn off indexing (for speed).

At the end, I turn back on indexing. The indexing process takes about  
2 hours and 30 minutes (or so).
Most of the time indexing is spent on a single field that is being  
indexed by word (that takes 2 hours).

The other fields just take a short time to index (a few minutes).

The re-indexing process completes successfully. However, when I do a  
diagnosis of the database,
it consistently has a corrupted header for those two unsigned long  
fields.

I have rebuilt this database about 6 times, using the same source  
data, and also using slightly
different source data.

I have tried rebuilding the index by discarding the .ind file part and  
letting Valentina Studio
rebuild the index. (I use the 4-part file option, with  
separate .vdb, .ind., .dat, and .blb files).
(There is no BLOB data in this database.) This resulted in the same  
problem.

The .dat file for the large database is about 3 GB.
The verbose diagnosis it about 36 MB.

What should I do?

I have added code that flushes the DB after rebuilding each field's  
index, but that didn't seem to fix it.

Help!

- Matthew





More information about the Valentina mailing list