Correct the log-likelihood scaling and fix the German announcement
collocatordb 1.8.0 scales the log-likelihood contingency table with the
window size as a whole rather than in its row total alone, so LL values
change here too. MI, MI², MI³ and nPMI do not, as they only need the
expected co-occurrence frequency, which is the same either way. Thanks
to Tim Feldmüller for spotting the inconsistency.
The German announcement still described LDaf as Log-Dice of the auto
focus window, and thus as at most as high as LD. That was true only for
the intermediate state that has since been reverted; the English text
had been corrected, the German one had not, because the replacement that
was supposed to update it silently matched nothing. Both are now
rewritten from the same wording.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: I42174ac2500f8e1bc3e54739310736526a1d5d47
diff --git a/Changelog.md b/Changelog.md
index 75d62ed..28cb306 100644
--- a/Changelog.md
+++ b/Changelog.md
@@ -9,6 +9,13 @@
Sketch Engine. Building against an older collocatordb would show the previous
values under the new explanations, which is why the minimum version is
enforced
+- LL values change as well: its contingency table used to scale only the row
+ total with the window size, leaving cells that do not add up to one sample.
+ Following Evert (2004), the sample consists of co-occurrence tokens, so the
+ sample size and both marginals scale with the window. Thanks to
+ [Tim Feldmüller](https://github.com/feldmueller) for spotting this. MI, MI²,
+ MI³ and nPMI are unaffected, as they only need the expected co-occurrence
+ frequency, which is unchanged
- LDaf and the order of the collocates are unchanged. It is the auto focus
score, not Log-Dice of the auto focus window, and its denominator grows with
the width of the selected window, which is what makes it sensitive to how
diff --git a/templates/de/about.html.ep b/templates/de/about.html.ep
index ce3062e..d74cfd1 100644
--- a/templates/de/about.html.ep
+++ b/templates/de/about.html.ep
@@ -48,10 +48,13 @@
\(14 + \log_2 \frac{2f_{1,2}}{f_1 + f_2}\), also ohne den Fenstergrößenfaktor, der bisher
auf \(f_1\) angewendet wurde. Die Werte sind dadurch höher als zuvor, um bis zu
\(\log_2\) der Fenstergröße, und direkt mit denen anderer Werkzeuge wie Sketch Engine
- vergleichbar. LDaf ist jetzt der Log-Dice-Wert des Auto-Focus-Fensters und damit
- höchstens so hoch wie LD; die Differenz zwischen beiden zeigt, wie stark die
- Kookkurrenzen in diesem Fenster konzentriert sind. Die übrigen Maße ändern sich nicht,
- da die Fenstergröße dort in die erwartete Häufigkeit gehört.
+ vergleichbar. Auch LL ändert sich: Seine Vierfeldertafel wird jetzt als Ganzes mit der
+ Fenstergröße skaliert und nicht mehr nur in der Zeilensumme – Dank an
+ <a href="https://github.com/feldmueller">Tim Feldmüller</a> für den Hinweis.
+ LDaf bleibt unverändert, ebenso die Reihenfolge der Kollokatoren: Es ist der
+ Auto-Focus-Wert, der den Fenstergrößenfaktor bewusst behält, weil er ihn erst empfindlich
+ dafür macht, wie konzentriert ein Paar auftritt. MI, MI², MI³ und nPMI ändern sich nicht,
+ da sie nur die erwartete Häufigkeit benötigen, und die bleibt gleich.
</dd>
</dl>
</html>
diff --git a/templates/en/about.html.ep b/templates/en/about.html.ep
index 379610d..897f059 100644
--- a/templates/en/about.html.ep
+++ b/templates/en/about.html.ep
@@ -46,10 +46,13 @@
\(14 + \log_2 \frac{2f_{1,2}}{f_1 + f_2}\), without the window size factor that used to be
applied to \(f_1\). Values are therefore higher than before, by up to \(\log_2\) of the
window size, and directly comparable to those of other tools such as Sketch Engine.
+ LL changes as well: its contingency table is now scaled with the window size as a whole
+ rather than in its row total alone – thanks to
+ <a href="https://github.com/feldmueller">Tim Feldmüller</a> for spotting this.
LDaf is unchanged, as is the order of the collocates: it is the auto focus score, which
keeps the window size factor deliberately, since that is what makes it sensitive to how
- concentrated a pair is. The other measures are unaffected as well, the window size
- belonging in their expected frequency.
+ concentrated a pair is. MI, MI², MI³ and nPMI are unaffected, as they only need the
+ expected co-occurrence frequency, which is unchanged.
</dd>
</dl>
</html>