Large database, consistently corrupted header when building index for 2 fields, 100% reproducible

[Ruslan Zasukhin]

On 10/7/08 5:24 PM, "Matthew Jew" <mjew at icnc.com> wrote:

Hi Matthew, 

> I have a very large database, which I am building by importing the
> contents of about 60 smaller databases.
> Each of the smaller databases is error-free (can be diagnosed
> successfully with no problems.)

ok
 
> However, when my application copies the contents of those databases  into one
> very large database (about 9 million records) two of the indexes are
> corrupted. (Both  fields are unsigned long fields.)
> 
> This line is from the very verbose diagnosis output:
> 
> PAGE (5446, 5440) Corrupted Header (Values Count). Corrupted Header
> (Values count < RecID count). At least one value counter is wrong.
 
> The diagnosis finds no other problems. (No other indexes have
> problems, no tables have problems.)

Wow, ULONG. ...

---- 
> During the importing, I turn off indexing (for speed).

ok

> At the end, I turn back on indexing. The indexing process takes about  2 hours
> and 30 minutes (or so). Most of the time indexing is spent on a single field
> that is being  indexed by word (that takes 2 hours).

What cache size you have. Make it as big as possible during indexing.
Helps a lots.

> The other fields just take a short time to index (a few minutes).

ok
 
> The re-indexing process completes successfully. However, when I do a
> diagnosis of the database, it consistently has a corrupted header for those
> two unsigned long  fields.

> What should I do?

* Are this 2 ULONG fields from the same table?

* Does this table have OTHER ULONG fields?

* how many records in THIS table ?


* So I assume that if EXPORT only this one ULONG field into text field,
    then create a new db with one table and one ulong field,
    and import back this data
        then reindex => should reproduce problem.

If this does, then we get 9M * 4 bytes = 36Mb table size,
    compressed about 5Mb.
    

You must be able send this db without index to me.
Then I will be able reproduce it and debug asap.


Also I remember that in Mantis present similar bug report from Ivan.
I need check all this.



-- 
Best regards,

Ruslan Zasukhin
VP Engineering and New Technology
Paradigma Software, Inc

Valentina - Joining Worlds of Information
http://www.paradigmasoft.com

[I feel the need: the need for speed]