[KinoSearch] Re: Internal error: field_num 111 > max_field_num 6

[Marvin Humphrey]

On Jun 11, 2007, at 9:32 PM, Matthew Berk wrote:

> Would a simple locking mechanism (I know you have one for .2) work  
> to alleviate the problem under mod_perl?

Serializing access, say by forcing each complete search into a  
subroutine and preventing everybody else from using the shared  
resources until the call returns... I think that would solve the  
concurrency issue.

However, you're going to get memory errors, because when one of  
KinoSearch's C struct objects gets copied to multiple threads, the  
internal refcount doesn't get incremented.  (KS objects keep their  
own refcounts, distinct from Perl's SvREFCNT.)  When the first of  
several Perl wrapper objects referencing the C object gets DESTROYed,  
the KS refcount will fall to 0 and the C object will clean itself  
up.  From now on, any other Perl wrapper objects referencing that C  
struct are pointing at freed memory.  That's Bad.

A script which segfaults to illustrate the problem is below.

Solving this is hard.  Incrementing the KS refcounts of objects  
copied when Perl spawns a thread is a PITA, because CLONE gets  
invoked as a *package* method.  You have to maintain global  
registries keeping track of all live object references, then iterate  
over all the items in the registries when CLONE tips you off that a  
thread is starting.

Maintaining such a scheme is messy, labor-intensive and error-prone  
-- and that's the biggest reason among several why KS doesn't do  
threads.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/

#----------------------------------------------------------

#!/usr/bin/perl
use strict;
use warnings;

use KinoSearch::Store::FSFolder;
use Carp qw( cluck );
use File::Path qw( rmtree );

sub warn_destroy {
     my $self = shift;
     cluck "Destroying";
     KinoSearch::Util::Obj::DESTROY($self);
}

{
     no warnings 'once';
     *KinoSearch::Store::FSFolder::DESTROY = *warn_destroy;
}

use threads;

rmtree 'foofolder';
mkdir 'foofolder' or die $!;

my $folder = KinoSearch::Store::FSFolder->new(
     path => 'foofolder',
);
warn $folder;

for my $num ( 0 .. 10 ) {
     my $thread = threads->create( sub {
         $folder->refcount_inc;
         my $out = $folder->open_outstream("file_$num");
     });
     $thread->join;
}