[KinoSearch] Memory leak in KinoSearch::Index::PostingsWriter::add_batch

[Marvin Humphrey]

On Jul 30, 2007, at 2:24 PM, Nathan Kurz wrote:

> On 7/30/07, Dan Sully <daniel at electricrain.com> wrote:
>> Unfortunately I don't understand where kino_PostWriter_add_batch()  
>> is defined.
>
> Presumably Marvin is busy with something, so I'll try an interim
> answer.

Thanks, Nathan.  I have a double deadline Wednesday: testing begins  
and the project manager leaves.  :\  It's a fun gig and I'm enjoying  
it, but I'm also looking forward to having more time to work on KS  
again as it finishes up.  Sorry I haven't gotten out a reply to you;  
your emails can be challenging. ;)

> I haven't looked much at the Index creation side of things,
> but that would be defined in c_src/KinoSearch/Index/PostingsWriter.c
> as PostWriter_add_batch().

Yes.  FWIW, this section of code is pretty involved and could  
withstand some simplification. ;)

> (c_src/KinoSearch/Util/MemoryPool.c).  If it's a true leak (as opposed
> to just  large buffer) it might be easiest to track down with
> valgrind.

It's either a true leak, or something else is wrong -- Dan's numbers  
are definitely bigger than they should be.   I think around 45-50  
megs ought to be worst case for an InvIndexer in 0.20_04 (and we can  
bring that down in future releases).

Tracking leaks in a Perl app with valgrind is a little tricky.

   * Linux only.
   * A debugging perl is required.
   * Set the PERL_DESTRUCT_LEVEL environment variable to 2.
   * You probably need a suppressions file like the one
     in $KS_DIST/devel/p588_valgrind.supp.
   * Valgrind won't catch leaks from circular refs at the Perl
     level, as those get cleaned up during global destruct.

perlhack has more details.   Check the bleadperl version, as it has  
some recent doc patches.

   http://search.cpan.org/~rgarcia/perl-5.9.5/pod/perlhack.pod

[... time passes ...]

I just ran the KS test suite under valgrind, using the script  
$KS_DIST/devel/valgrind_test.plx, and it came up clean.  So we'll  
need some way of duplicating the problem.

Thanks for the report, Dan.  I'll attend to this when I can take a  
break from fighting fires at $contract_job.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/