tree 543f2e984b656b1f856e4a4f7c1e7d2e4d3149e7
parent 8bacebb64a582d71eee5005a2de377fb1027da69
author Marc Kupietz <kupietz@ids-mannheim.de> 1788438408 +0200
committer Marc Kupietz <kupietz@ids-mannheim.de> 1788438408 +0200

Guard log-likelihood against windows covering the whole corpus

The contingency table of ll() classifies all N corpus tokens by whether
they lie inside a window around the node, using window_size * O1 as the
row total. That stops being a partition once the windows cover more than
the corpus: N - window_size * O1 turns negative, and the score became
NaN through a log() of a negative number, with only a warning from log()
itself to indicate what had happened.

Such a node has to cover more than 1/w of the corpus, so with the
default window it is out of reach, but in DeReKo "der" (2.90% of all
tokens) gets there from a context of 18 left and 18 right, "die" from
19+19 and "und" from 24+24.

These cases now yield NA with a warning that names the condition and
suggests a smaller window. The check is vectorized, so that only the
affected values become NA.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: I5e11396c67bc9103ec73745726d052857536df7f