[KinoSearch] Towards a stable C API... via indirect dispatch

[Marvin Humphrey]

On Oct 28, 2007, at 12:52 PM, Nathan Kurz wrote:

> Good summary, but I'm not sure that there are that many cases where we
> would be breaking the binary API.  The link you provide suggests that
> things will break if we add a new method to the end of the vtable, but
> I don't think this applies to the way Boilerplater works.   Am I
> wrong?

It does apply.  Say we add Scorer_Advance as a new virtual method,  
and deprecate Scorer_Next and Scorer_Skip_To. The size of the Scorer  
vtable will increase by 1 function pointer.

Now say that you have a separate distro, with your own custom Scorer  
subclass, MyScorer, which defines a single custom method,  
MyScorer_Foo.  Your externally compiled code expects to find  
MyScorer_Foo in the vtable starting at an offset of sizeof 
(KINO_SCORER_VTABLE) -- but that sizeof() was calculated when  
KINO_SCORER_VTABLE was smaller than it is now. Thus, all the compiled- 
in, fixed offsets for your custom method calls are now wrong.

   /* This now jumps into Scorer_Advance.  Bad! */
   MyScorer_Foo(myscorer);

> My instinct is that this is a fine solution to the problem, but that
> it's not really a problem that we need to solve right now.

In one sense, we don't have to solve it right away, because  
presenting a public C API isn't immediately necessary.  But if we can  
work things out, all kinds of possibilities open up.  For instance,  
subclassing HitCollector from Perl will never scale, because  
HitColl_Collect() gets called way too often when iterating over a  
large result set.  But subclassing HitCollector is powerful stuff,  
and I'd like to allow it.

> I'm not
> seeing how KinoSearch is different than any other shared library ---
> couldn't we just bump up the major version number every time the
> binary API changes?

It's not any different from any other shared library.  But preserving  
a traditional binary API is a FPITA.  I expect continuing flux for  
some time to come.  Freezing the vtable layouts would suck, and hard.

> I may just be being misled by the trivial examples you are using.
> Could you offer a better example of a case where this technique would
> be necessary?

Say that we don't get to the positional scoring improvements soon ;)  
and we revert to the original plan of publishing 0.20 with the  
current scoring behavior.  Yet when the positional scoring engine is  
finished, we'd still like to publish it as  
KSx::Search::PositionAwareScorer, so that those in the know can  
override the default engine and get more relevant search results.

Under this scheme, you can release that as an XS distro, and _all_  
your compiled-in method calls will continue work as advertised no  
matter how much we monkey around with the vtables in the core library.

Essentially, the vtable layout is no longer part of the public binary  
API.  What we promise instead is that 1) a vtable exists and 2) e.g.  
Kino_Scorer_Destroy_OFFSET tells you where to find the right function  
pointer in that vtable.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/



_______________________________________________
KinoSearch mailing list
KinoSearch at rectangular.com
http://www.rectangular.com/mailman/listinfo/kinosearch