[KinoSearch] serializing safely

[Marvin Humphrey]

On Jun 14, 2007, at 6:07 PM, Hans Dieter Pearcey wrote:

> Can Searchers be treated analogously to file handles, i.e. shared  
> between
> processes (opened in a parent, shared between children) as long as  
> only one
> process uses it at a time, or is there per-process state that will  
> get screwed
> up?

I believe that will work.  In general, I wouldn't recommend doing  
things that way with large indexes because you'll end up wasting a  
lot of RAM... but that may not matter here.

FWIW, the optional read-locking mechanism needed for use with NFS  
will break -- since it uses lock files that remember their pids --  
but it's off by default.

> This doesn't really help with the question at hand, since I don't  
> plan on
> preloading the searchers, but it's an interesting thing to keep in  
> mind.

I don't think you should rule out pre-loading.  KinoSearch is heavily  
optimized for the use case of running many queries against a single  
view of an index.

The costs for warming the a Searcher vary linearly with index size,  
and get significantly higher if you perform sorting or range  
operations.  To put things in perspective, for very large indexes  
(larger than you're likely to see for any one individual's email), it  
can conceivably take several seconds to warm up a Searcher, then a  
fraction of a second to process the query.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/