[KinoSearch] Merging indexes, etc

[Marvin Humphrey]

On Oct 26, 2006, at 6:36 AM, henka at cityweb.co.za wrote:

> Splitting the process into two steps (delete_docs_by_term() and
> add_invindexes()) unfortunately results in the error:
>
> Can't locate object method "_release_locks" via package  
> "self" (perhaps
> you forgot to load "self"?) at
> /usr/lib/perl5/site_perl/5.8.7/i486-linux/KinoSearch/InvIndexer.pm  
> line
> 273.

Hmm.  What the hell is "self" doing there?  That really suggests  
memory corruption of some kind -- the pointer that used to lead Perl  
to the class name has been reused and is now pointing to something  
completely different and wrong.  That's why I suspected problems with  
DESTROY.

I have one more idea, but after that, I'm going to need a failing  
test case I can go tear into.

> The logic I used:
>
> my $iv;
> foreach (sub-indexes) {
>   $iv= KinoSearch::InvIndexer->new(...);
>   $iv->delete_docs_by_term(...);
>   $iv->finish(optimize=>0);
>   $iv= KinoSearch::InvIndexer->new(...); # same sub-index as previous
>   $iv->add_invindexes(...);
>   $iv->finish(optimize=>0);
> }

Try undef-ing the InvIndexer before assigning another one on top of  
it.  Take a look at the create_test_invindex() function in <http:// 
www.rectangular.com/svn/kinosearch/trunk/buildlib/ 
KinoSearchTestInvIndex.pm> for a template.

I'm not sure that you're seeing the same problem I've been seeing,  
but code like in your example, where a new InvIndexer instance is  
created in the same scalar variable, has caused me problems in the  
past.  The symptoms were slightly different: it was lock contention  
rather than a failed call to _release_locks.

The sequence "$invindexer = new_invindexer(); $invindexer =  
new_invindexer();" is problematic, because the second assignment  
statement triggers this order of operations:

      1) Create new InvIndexer, put it in a mortal SV on
         the Perl stack.
      2) Assign the new InvIndexer object to the scalar
         which previously held another invindexer.
      3) The old scalar's reference count is decremented
         and the old InvIndexer's DESTROY method gets called.

In step 1, the new InvIndexer requests a lock.  But the old  
InvIndexer doesn't release its lock until step 3.

I'd like to solve this problem, but I'm not sure how to do it in the  
general case.  I've thought of doing something like this, using  
closures:

     # in new()
     $self->{release_locks} = sub { $self->do_release_locks };

     # in finish()
     $self->{release_locks}->();
     undef $self->{release_locks};

     # in DESTROY
     $self->{release_locks}->() if defined $self->{release_locks};

But I think that would still fail if finish() never gets called.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/