[KinoSearch] Towards a stable C API... via indirect dispatch

[Nathan Kurz]

On 10/28/07, Marvin Humphrey <marvin at rectangular.com> wrote:
> > The link you provide suggests that
> > things will break if we add a new method to the end of the vtable, but
> > I don't think this applies to the way Boilerplater works.   Am I
> > wrong?
>
> It does apply.  Say we add Scorer_Advance as a new virtual method,
> and deprecate Scorer_Next and Scorer_Skip_To. The size of the Scorer
> vtable will increase by 1 function pointer.
>
> Now say that you have a separate distro, with your own custom Scorer
> subclass, MyScorer, which defines a single custom method,
> MyScorer_Foo.  Your externally compiled code expects to find
> MyScorer_Foo in the vtable starting at an offset of sizeof
> (KINO_SCORER_VTABLE) -- but that sizeof() was calculated when
> KINO_SCORER_VTABLE was smaller than it is now. Thus, all the compiled-
> in, fixed offsets for your custom method calls are now wrong.
>
>    /* This now jumps into Scorer_Advance.  Bad! */
>    MyScorer_Foo(myscorer);


There must be something that I'm still not getting.  I can now see
that things will break in some cases, but I don't see breakage in your
example.  But maybe I've been away from the code too long.

I'm presuming you are building a KSx extension as a shared library.
Boilerplater runs, and using a local copy of the Kinosearch header
files, creates a vtable struct for MyScorer that takes into account
appropriate inheritance rules.  This vtable is instantiated as a
global variable in MyScorer.so as KINO_MYSCORER in MyScorer.r, with
the external KinoSearch methods represented by symbols identified by
function names.  At run-time, ld.so recognizes that MyScorer.so is
incomplete, finds the symbols in KinoSearch.so, and does whatever it
does to magically patch things up so the function pointers work.

OK, so then the user upgrades KinoSearch.so (or a second user grabs
MyScorer.so and a newer version of KinoSearch.so).  From what I can
tell, the MyScorer vtable is still going to work fine.   Pointers to
functions local to MyScorer.so will work as before, and pointers to
functions in the new KinoSearch.so, since they are found by ld.so by
name, should work fine as well.   Thus, in my view,
MyScorer_Foo(myscorer) should work as expected so long as the symbol
names don't change, even if the format of the parent Scorer vtables
change completely. No?

Where I do see the problem is when the new KinoSearch.so tries to call
Scorer_Advance(myscorer).  The new code goes to the offset in the
vtable of a new Scorer at which it expects to find an entry for the
Advance method, and instead (unknowingly) finds a pointer to
MyScorer_Foo():  boom!

But what I don't understand yet is how your proposed solution is going
to deal with this.  Since the MyScorer vtable has no knowledge of the
Advance method, looking within MyScorer.so for
Kino_MyScorer_Advance_OFFSET is just going to cause ld.so to fail
during run-time linking.  Since adding another level of indirection to
all of our class methods seems like a pretty convoluted way to just
check  for a compatible library version, I presume I'm missing
something.

I think I can see (through a glass, darkly) that this technique could
work for a language where vtables are chained and crawled at runtime,
but even then I'm not sure it is really a good thing.  Suppose that
MyScorer had overriden Scorer_Skip_To with a custom My_Scorer_Skip_To,
and then the core KinoSearch switched to using Scorer_Advance.   While
the offset method you are proposing could allow MyScorer to seemlessly
 abandon the custom My_Scorer_Skip_To for the default Scorer_Advance,
this silent switch to wrong behaviour seems worse than a crash.

I'm not sure I believe that architectural changes like this can or
should be dealt with automatically.   I'm probably missing something,
but instinctively I think that checking for (and crashing on) major
version incompatibility is the best you can do.   While the technique
you propose would essentially automate bumping up this version number,
it seems like there should be a simpler way of accomplishing that if
that is the only goal.  But again, I'm probably missing something
essential here.

Nathan Kurz
nate at verse.com

_______________________________________________
KinoSearch mailing list
KinoSearch at rectangular.com
http://www.rectangular.com/mailman/listinfo/kinosearch