[KinoSearch] Near real-time indexing support

Marvin Humphrey marvin at rectangular.com
Tue Aug 4 20:01:37 PDT 2009


Greets,

I'm currently putting the finishing touches on KinoSearch 0.30_05, which will
be the first release to include support for near-real-time indexing.

Say that you maintain an index for user comments on a large website, and you'd
like the comments to be searchable almost immediately after the user hits the
submit button.  

Adding to an index is usually fast, but every once in a while, a large
consolidation process kicks off and commit() takes a long time to complete.
Or, to put it another way, average index update performance is very good, but
worst-case performance is substantially worse and has been the limiting factor
constraining real-time indexing designs up till now.

A new cookbook entry, KinoSearch::Docs::Cookbook::FastUpdates, explains how to
use the new classes IndexManager and BackgroundMerger to achieve consistently
quick index update times and make systems like the immediate-access comment
index possible:

    http://xrl.us/fastupdates (Link to www.rectangular.com)

As a postscript, support for near-real-time indexing builds on the work that
was done earlier to integrate mmap into our index file formats, and move
sort-cache generation to index-time so that Searchers always open quickly.
Thanks again to Nate Kurz for getting the ball rolling early with mmap.

Marvin Humphrey





More information about the kinosearch mailing list