Document the corrected Log-Dice, the effective window and the auto focus
Three things were left unexplained, and one of them has changed:
LD is now computed as Rychly (2008) defines it, without the window size
factor that used to be applied to f1, so its values are higher than
before, by up to log2 of the window size, and comparable to those of
other tools. This is announced in the info tab.
LDaf is unchanged, as is the resulting order of the collocates, but it is
easily mistaken for Log-Dice of the auto focus window. It is the auto
focus score itself, whose denominator grows with the width of the
selection, which is what makes it sensitive to how concentrated a pair
is, and thus what makes it order collocates usefully enough to serve as
the default order. It is therefore not on the scale of LD and can be
higher or lower than it.
LL, MI, MI², MI³ and nPMI are computed over the positions in which the
collocate actually occurs, not over the whole context window. Calling
these the attested positions, and their number the effective window size,
keeps them apart from the auto focus, which is Belica's term for the not
necessarily contiguous selection of positions that yields the highest
score. Both are visible in the auto-focus column, which marks the
attested positions with a hollow and the selected ones with a filled
square.
Explained in the info tab in both languages, in the tooltips of the LDaf
and auto-focus columns, and the LDaf tooltip now gives the formula.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: Id6293c71438d4b42d3a461697c181e3b6a072a83
diff --git a/templates/de/about.html.ep b/templates/de/about.html.ep
index ab92c0c..ce3062e 100644
--- a/templates/de/about.html.ep
+++ b/templates/de/about.html.ep
@@ -9,6 +9,30 @@
Die hier verwendeten Modelle beruhen zum einen auf einer Erweiterung von word2vec (Mikolov et al. 2013), wang2vec (Ling et al. 2015)
und zum anderen auf einfachen Kookkurrenzhäufigkeiten und Analysemethoden, die auf diesen operieren.
</p>
+<h3>Effektive Fenstergröße und Auto-Focus</h3>
+<p>
+ Die Assoziationsmaße LL, MI, MI², MI³ und nPMI werden nicht über das gesamte Kontextfenster
+ berechnet, sondern nur über die <em>belegten Positionen</em>, also diejenigen, an denen der
+ Kollokator mindestens einmal vorkommt. Die erwartete Häufigkeit beruht damit auf dieser
+ effektiven Fenstergröße, wodurch ein Kollokator, der nur an wenigen Positionen auftritt,
+ höher bewertet wird als einer, der sich über viele verteilt. Das hilft dabei, echte
+ Kollokationen von Wörtern zu unterscheiden, die lediglich in denselben Kontexten vorkommen.
+ Die Werte sind dadurch allerdings nicht unmittelbar mit denen von Werkzeugen vergleichbar,
+ die stets mit einer festen Fenstergröße rechnen.
+</p>
+<p>
+ LDaf ist demgegenüber der <em>Auto-Focus</em>-Wert (Belica), also der höchste Wert, den
+ irgendeine – nicht notwendig zusammenhängende – Auswahl von Positionen erreicht, wie er auch
+ aus der Kookkurrenzdatenbank <a href="http://corpora.ids-mannheim.de/">CCDB</a> bekannt ist.
+ Da sein Nenner mit der Breite der Auswahl wächst, bevorzugt er Paare, deren Kookkurrenzen
+ konzentriert auftreten – weshalb er die Kollokatoren in aller Regel am nützlichsten ordnet
+ und hier als Standardsortierung dient. Er ist Dice-artig, aber kein Log-Dice, liegt also
+ nicht auf derselben Skala wie LD und kann höher oder niedriger als dieses ausfallen. Beides
+ lässt sich an der Auto-Focus-Spalte der Kollokationstabelle ablesen: Die belegten Positionen
+ sind mit ◽ markiert, die vom Auto-Focus ausgewählten mit ◾. LD selbst hängt von keinem
+ Fenster ab.
+</p>
+
<h3>Disclaimer</h3>
<p>
Bitte beachten Sie, dass wir – anders als z.B. <a href="https://korap.ids-mannheim.de/">KorAP</a> – DeReKoVecs
@@ -18,6 +42,16 @@
</p>
<h3>Geplante Ausfallzeiten und Ankündigungen</h3>
<dl id="downtimes">
- <dt></dt><dd>–</dd>
+ <dt>04.09.2026</dt>
+ <dd>
+ Log-Dice (LD) wird jetzt genau so berechnet, wie es Rychlý (2008) definiert,
+ \(14 + \log_2 \frac{2f_{1,2}}{f_1 + f_2}\), also ohne den Fenstergrößenfaktor, der bisher
+ auf \(f_1\) angewendet wurde. Die Werte sind dadurch höher als zuvor, um bis zu
+ \(\log_2\) der Fenstergröße, und direkt mit denen anderer Werkzeuge wie Sketch Engine
+ vergleichbar. LDaf ist jetzt der Log-Dice-Wert des Auto-Focus-Fensters und damit
+ höchstens so hoch wie LD; die Differenz zwischen beiden zeigt, wie stark die
+ Kookkurrenzen in diesem Fenster konzentriert sind. Die übrigen Maße ändern sich nicht,
+ da die Fenstergröße dort in die erwartete Häufigkeit gehört.
+ </dd>
</dl>
</html>
diff --git a/templates/en/about.html.ep b/templates/en/about.html.ep
index 9842776..379610d 100644
--- a/templates/en/about.html.ep
+++ b/templates/en/about.html.ep
@@ -9,6 +9,28 @@
The models used here are based on an extension of word2vec (Mikolov et al. 2013), wang2vec (Ling et al. 2015)
and on the other hand on simple co-occurence counts and analysis methods that operate on these.
</p>
+<h3>Effective window size and auto focus</h3>
+<p>
+ The association measures LL, MI, MI², MI³ and nPMI are not computed over the whole context
+ window, but only over the <em>attested positions</em>, i.e. those positions in which the
+ collocate occurs at least once. The expected frequency is based on this effective window,
+ which rates a collocate occurring in few positions higher than one spread over many, and thus
+ helps to tell actual collocations from words that merely occur in the same contexts. Note
+ that the values are therefore not directly comparable to those of tools that always compute
+ with a fixed window size.
+</p>
+<p>
+ LDaf, in contrast, is the <em>auto focus</em> score (Belica), i.e. the highest value that any
+ – not necessarily contiguous – selection of positions reaches, an idea also known from the
+ collocation database <a href="http://corpora.ids-mannheim.de/">CCDB</a>. As its denominator
+ grows with the width of the selection, it rewards pairs whose co-occurrences are
+ concentrated, which is why it usually orders collocates most usefully and is used as the
+ default order here. It is Dice-like, but not Log-Dice, so it is not on the same scale as LD
+ and can be higher or lower than it. Both can be read off the auto-focus column of the
+ collocation table, where the attested positions are marked with ◽ and those selected by the
+ auto focus with ◾. LD itself does not depend on any window.
+</p>
+
<h3>Disclaimer</h3>
<p>
Please note that – unlike e.g. <a href="https://korap.ids-mannheim.de/">KorAP</a> – we cannot operate DeReKoVecs
@@ -18,6 +40,16 @@
</p>
<h3>Scheduled Downtimes and Announcements</h3>
<dl id="downtimes">
- <dt></dt><dd>–</dd>
+ <dt>2026-09-04</dt>
+ <dd>
+ Log-Dice (LD) is now computed exactly as defined by Rychlý (2008),
+ \(14 + \log_2 \frac{2f_{1,2}}{f_1 + f_2}\), without the window size factor that used to be
+ applied to \(f_1\). Values are therefore higher than before, by up to \(\log_2\) of the
+ window size, and directly comparable to those of other tools such as Sketch Engine.
+ LDaf is unchanged, as is the order of the collocates: it is the auto focus score, which
+ keeps the window size factor deliberately, since that is what makes it sensitive to how
+ concentrated a pair is. The other measures are unaffected as well, the window size
+ belonging in their expected frequency.
+ </dd>
</dl>
</html>
diff --git a/templates/index.html.ep b/templates/index.html.ep
index 646ddf5..1d16c4c 100644
--- a/templates/index.html.ep
+++ b/templates/index.html.ep
@@ -720,7 +720,7 @@
<div style="display:none;" id="ll_ttt">Log-likelihood: $$2\sum_{ij}O_{ij}\log\frac{O_{ij}}{E_{ij}}$$<p class="citation">Dunning, T. (1993): Accurate methods for the statistics of surprise and coincidence. Comput. Linguist. 19, 1 (March 1993), 61-74.</p>
<p class="citation">Evert, Stefan (2004): <a href="http://purl.org/stefan.evert/PUB/Evert2004phd.pdf">The Statistics of Word Cooccurrences: Word Pairs and Collocations.</a> PhD dissertation, IMS, University of Stuttgart. Published in 2005, URN urn:nbn:de:bsz:93-opus-23714.</p></div>
<div style="display:none;" id="logdice_ttt">Log-Dice: $$14 + \log_2 \frac{2f_{1,2}}{f_1 + f_2}$$<p class="citation">Rychlý, Pavel (2008): <a href="http://www.fi.muni.cz/usr/sojka/download/raslan2008/13.pdf">A lexicographer-friendly association score.</a> In Proceedings of Recent Advances in Slavonic Natural Language Processing, RASLAN, 6–9, 2008</p></div>
- <div style="display:none;" id="logdiceaf_ttt">Log-Dice using "auto-focus", i.e. the window that yields the highest score.</div>
+ <div style="display:none;" id="logdiceaf_ttt">Log-Dice using "auto-focus", i.e. the highest value reached by any – not necessarily contiguous – selection \(S\) of positions: $$\max_S \left(14 + \log_2 \frac{2f_{1,2}(S)}{f_2 + f_1|S|}\right)$$ Dividing by the width of the selection makes the measure sensitive to how concentrated a pair is, which is what distinguishes actual collocations from words that merely occur in the same contexts. Note that LDaf is therefore not on the same scale as LD and can be higher or lower than it.</div>
<div id="ids_logo">
<a href="http://www.ids-mannheim.de/" target="_blank"><img src="/derekovecs/img/IDS-neu_farbig.svg" alt="Leibniz-Institut für Deutsche Sprache"/></a>
</div>
@@ -942,7 +942,7 @@
<th id="logdice_tt">LD</th>
<th id="logdiceaf_tt">LDaf</th>
<th id="delta_tt" title="Delta to log-Dice score in reference corpus. ⚠: If the collocate is not within the top 200 of the reference corpus, a reference value of min(lD)-0.1 is assumed.">Δ</th>
- <th id="af_win" title="Positions around the target word that are selected by the auto-focus function are marked with ◾. Positions at which the collocate appears at least once are marked with ◽."><span class="regular"><%= loc 'af_window' %></span></th>
+ <th id="af_win" title="Positions around the target word that are selected by the auto-focus function are marked with ◾. Positions at which the collocate appears at least once are marked with ◽. LL, MI, MI², MI³ and nPMI are computed over all positions marked here, the attested ones, rather than over the whole context window, so that a collocate occurring in few positions is rated higher than one spread over many. LD does not depend on any window; LDaf is the auto-focus score of the ◾ positions and is not on the same scale as LD."><span class="regular"><%= loc 'af_window' %></span></th>
<th title="PMI³ restricted to left neighbour">l-PMI³</th>
<th title="PMI³ restricted to right neighbour">r-PMI³</th>
<th title="nPMI restricted to left neighbour">l-nPMI</th>