[KinoSearch] KinoSearch::Docs::Cookbook::ReusingSearchers

[Marvin Humphrey]

On Sep 13, 2007, at 12:12 AM, Henry wrote:

> Dunno if you missed the above post (or you're playing catchup):  any
> comments on whether all search nodes can be "masters" for
> aggregating/sorting results?

Yeah, that will work fine.

You had  also asked about the MultiSearcher sort.  I'd been back- 
burnering that one because I was hoping that a new approach would  
present itself during the course of fixing other things.  Well, I  
believe that one has.

What we need is to do is break up nodes by task.  We'll have a  
dedicated lexicon server, a dedicated document server, one or more  
master search nodes, and multiple machines dedicated to the task of  
crunching through posting lists.  The lexicon server and the document  
server will be abstractions behind which one or more machines will sit.

These tasks are already more or less broken up inside IndexReader, so  
all we'll need to do is build the appropriate subclasses of  
DocReader, LexReader, etc.

If you'll recall, the problem with the MultiSearcher sort has to do  
with the overhead of loading large fields into memory to cut down on  
disk seeks.  This solves that problem by loading the whole lexicon  
into one shared space for the whole search cluster.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/



_______________________________________________
KinoSearch mailing list
KinoSearch at rectangular.com
http://www.rectangular.com/mailman/listinfo/kinosearch