[KinoSearch] Possible memory leak when iterating over the results in KinoSearch

[Marvin Humphrey]

Nils Grunwald:

> When iterating over a hits collected from a large index, the memory is never
> released. In this small test case I get out of memory after a few
> iterations, and I see memory grow with each iteration of the loop. I think
> the problem lies in calling hits->next. Is there something I have not seen
> to do to explicitly destroy the objects, or do you need something else to
> help me find the problem ? 

You should not have to explicity destroy the objects.

Thanks for the test case -- it exposed two rather insidious memory leaks.  KS
trunk has a test_valgrind Build target that runs the test suite under
Valgrind, but it wasn't detecting either of them.

The first one I found originated in the mmap refactoring from last month.
Some mapped regions were being disposed of properly with munmap(), but others
were not.  For whatever reason, Valgrind wasn't showing these as leaks.

> Interestingly, the same thing was also leaking
> memory in the last build I used (rev 3892) but at a much slower pace.

Yes, that's right.  This one is older.  It's been there at least since
DocReader was ported to C in June.  

The problem was a refcounting error on the Perl hash that's used for the Doc
object's fields.  These hashes would stick around after the Doc object was
destroyed.  

When Perl shuts down, ordinarily it just lets the system dispose of everything
that's left over rather than winding up in an orderly manner.  However, to
detect memory leaks with Valgrind, it's necessary to set the environment
variable PERL_DESTRUCT_LEVEL to 2 or above, which engages an "everybody out of
the pool NOW" global destruction mechanism.  The hashes were being cleaned up
then, so Valgrind didn't expose the glitch.

Commits r3971 and r3972 should fix the leakage.

  svn co -r 3972 http://www.rectangular.com/svn/kinosearch/trunk ks

Cheers,

Marvin Humphrey