[KinoSearch] Possible bug in Kinosearch::Invindexer

[Marvin Humphrey]

On Mon, Dec 22, 2008 at 03:42:17PM +0100, Nils Grunwald wrote:
> Thanks a lot for the bugfix about the memory leak, it was indeed 
> insidious, I spent half a day using valgrind and not understanding why i 
> could not find any leak. 

Oof, been there. :\  Leaks are fewer and farther between since the
test_valgrind Build target came into being, but it's imperfect.

To expose the Perl refcounting error, I inserted a hard call to the C exit()
function in the loop and triggered it at different iter counts.  When the
Valgrind numbers changed, I knew we weren't done yet.

> It can be reproduced with almost the same script as last time: if indexing a
> large number of documents without calling finish, Invindexer crashes while
> adding a document or while finishing with this error message:
> 
> Can't write to a closed OutStream at 
> ../c_src/KinoSearch/Store/OutStream.c:80 do_flush

I had limited time to address the problem today.  I improved some error
reporting so that we at least get to see what file we're trying to write to.
It's a ".lextemp" file, which points to PostingsWriter.  By bringing the
mem_threshold at which PostingsWriter flushes a sorted run to disk down to 1k,
I was able to make it crash after on the 23rd iter.  Hopefully I'll be able to
zap the bug tomorrow.

Marvin Humphrey