[KinoSearch] Re: Internal error: field_num 111 > max_field_num 6

[Matthew Berk]

Does this problem also hold under MP, where each use is a separae proc? I take it the answer is yes, because child processes get copies of class data from the parent process. Any other recommendations for safely running this under MP?

Thanks!

Matthew



-----Original Message-----
From: kinosearch-bounces+matthew=openlist.com at rectangular.com on behalf of Marvin Humphrey
Sent: Tue 6/12/2007 7:53 AM
To: KinoSearch discussion forum
Subject: Re: [KinoSearch] Re: Internal error: field_num 111 > max_field_num 6
 

On Jun 11, 2007, at 9:32 PM, Matthew Berk wrote:

> Would a simple locking mechanism (I know you have one for .2) work  
> to alleviate the problem under mod_perl?

Serializing access, say by forcing each complete search into a  
subroutine and preventing everybody else from using the shared  
resources until the call returns... I think that would solve the  
concurrency issue.

However, you're going to get memory errors, because when one of  
KinoSearch's C struct objects gets copied to multiple threads, the  
internal refcount doesn't get incremented.  (KS objects keep their  
own refcounts, distinct from Perl's SvREFCNT.)  When the first of  
several Perl wrapper objects referencing the C object gets DESTROYed,  
the KS refcount will fall to 0 and the C object will clean itself  
up.  From now on, any other Perl wrapper objects referencing that C  
struct are pointing at freed memory.  That's Bad.

A script which segfaults to illustrate the problem is below.

Solving this is hard.  Incrementing the KS refcounts of objects  
copied when Perl spawns a thread is a PITA, because CLONE gets  
invoked as a *package* method.  You have to maintain global  
registries keeping track of all live object references, then iterate  
over all the items in the registries when CLONE tips you off that a  
thread is starting.

Maintaining such a scheme is messy, labor-intensive and error-prone  
-- and that's the biggest reason among several why KS doesn't do  
threads.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/

#----------------------------------------------------------

#!/usr/bin/perl
use strict;
use warnings;

use KinoSearch::Store::FSFolder;
use Carp qw( cluck );
use File::Path qw( rmtree );

sub warn_destroy {
     my $self = shift;
     cluck "Destroying";
     KinoSearch::Util::Obj::DESTROY($self);
}

{
     no warnings 'once';
     *KinoSearch::Store::FSFolder::DESTROY = *warn_destroy;
}

use threads;

rmtree 'foofolder';
mkdir 'foofolder' or die $!;

my $folder = KinoSearch::Store::FSFolder->new(
     path => 'foofolder',
);
warn $folder;

for my $num ( 0 .. 10 ) {
     my $thread = threads->create( sub {
         $folder->refcount_inc;
         my $out = $folder->open_outstream("file_$num");
     });
     $thread->join;
}





_______________________________________________
KinoSearch mailing list
KinoSearch at rectangular.com
http://www.rectangular.com/mailman/listinfo/kinosearch

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/ms-tnef
Size: 4757 bytes
Desc: not available
Url : http://www.rectangular.com/pipermail/kinosearch/attachments/20070612/7244d3de/attachment.bin