[KinoSearch] Memory leak in KinoSearch::Index::PostingsWriter::add_batch

[Marvin Humphrey]

On Jul 30, 2007, at 4:28 PM, Dan Sully wrote:

> * Marvin Humphrey shaped the electrons to say...
>
>> How about running with a subset of your document collection?  As  
>> soon  as a byte gets lost, Valgrind notices.
>
> I've managed to get it to index 5 documents. :)
>
> valgrind output is attached.

I don't see anything in there that's KS related, so I think we'll  
need to find a threshold big enough to trip the problem, but not so  
big that it dies.  We might start seeing stuff at around 30 megs.    
Once we hit 70 or 80, I'd be surprised if nothing showed.

FWIW, the DynaLoader stuff isn't being caught by the KS suppressions  
file:

==4137== 40 bytes in 1 blocks are still reachable in loss record 3 of 9
==4137==    at 0x4A2080E: malloc (vg_replace_malloc.c:149)
==4137==    by 0x400BBDC: (within /lib/ld-2.4.so)
==4137==    by 0x40111FC: (within /lib/ld-2.4.so)
==4137==    by 0x400D1E5: (within /lib/ld-2.4.so)
==4137==    by 0x4010C0A: (within /lib/ld-2.4.so)
==4137==    by 0x4B251A9: (within /lib/libdl-2.4.so)
==4137==    by 0x400D1E5: (within /lib/ld-2.4.so)
==4137==    by 0x4B2571C: (within /lib/libdl-2.4.so)
==4137==    by 0x4B25120: dlopen (in /lib/libdl-2.4.so)
==4137==    by 0x41DEF8: XS_DynaLoader_dl_load_file (DynaLoader.xs:191)
==4137==    by 0x4C53CD: Perl_pp_entersub (pp_hot.c:2877)
==4137==    by 0x4993BF: Perl_runops_debug (dump.c:1459)

If you want to clean things up, you can generate a suppressions file  
with --gen_suppressions with this 5-document pass.  Info here:

http://valgrind.org/docs/manual/manual-core.html#manual-core.suppress
http://valgrind.org/docs/manual/writing-tools.html#writing- 
tools.suppressions
http://valgrind.org/docs/manual/mc-manual.html#mc-manual.suppfiles

However, we'll probably be able to see what's going on without that.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/