[KinoSearch] Near real-time indexing support
Marvin Humphrey
marvin at rectangular.com
Tue Aug 4 20:01:37 PDT 2009
Greets,
I'm currently putting the finishing touches on KinoSearch 0.30_05, which will
be the first release to include support for near-real-time indexing.
Say that you maintain an index for user comments on a large website, and you'd
like the comments to be searchable almost immediately after the user hits the
submit button.
Adding to an index is usually fast, but every once in a while, a large
consolidation process kicks off and commit() takes a long time to complete.
Or, to put it another way, average index update performance is very good, but
worst-case performance is substantially worse and has been the limiting factor
constraining real-time indexing designs up till now.
A new cookbook entry, KinoSearch::Docs::Cookbook::FastUpdates, explains how to
use the new classes IndexManager and BackgroundMerger to achieve consistently
quick index update times and make systems like the immediate-access comment
index possible:
http://xrl.us/fastupdates (Link to www.rectangular.com)
As a postscript, support for near-real-time indexing builds on the work that
was done earlier to integrate mmap into our index file formats, and move
sort-cache generation to index-time so that Searchers always open quickly.
Thanks again to Nate Kurz for getting the ball rolling early with mmap.
Marvin Humphrey
More information about the kinosearch
mailing list