Document the corrected Log-Dice, the effective window and the auto focus
Three things were left unexplained, and one of them has changed:
LD is now computed as Rychly (2008) defines it, without the window size
factor that used to be applied to f1, so its values are higher than
before, by up to log2 of the window size, and comparable to those of
other tools. This is announced in the info tab.
LDaf is unchanged, as is the resulting order of the collocates, but it is
easily mistaken for Log-Dice of the auto focus window. It is the auto
focus score itself, whose denominator grows with the width of the
selection, which is what makes it sensitive to how concentrated a pair
is, and thus what makes it order collocates usefully enough to serve as
the default order. It is therefore not on the scale of LD and can be
higher or lower than it.
LL, MI, MI², MI³ and nPMI are computed over the positions in which the
collocate actually occurs, not over the whole context window. Calling
these the attested positions, and their number the effective window size,
keeps them apart from the auto focus, which is Belica's term for the not
necessarily contiguous selection of positions that yields the highest
score. Both are visible in the auto-focus column, which marks the
attested positions with a hollow and the selected ones with a filled
square.
Explained in the info tab in both languages, in the tooltips of the LDaf
and auto-focus columns, and the LDaf tooltip now gives the formula.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: Id6293c71438d4b42d3a461697c181e3b6a072a83
diff --git a/templates/index.html.ep b/templates/index.html.ep
index 646ddf5..1d16c4c 100644
--- a/templates/index.html.ep
+++ b/templates/index.html.ep
@@ -720,7 +720,7 @@
<div style="display:none;" id="ll_ttt">Log-likelihood: $$2\sum_{ij}O_{ij}\log\frac{O_{ij}}{E_{ij}}$$<p class="citation">Dunning, T. (1993): Accurate methods for the statistics of surprise and coincidence. Comput. Linguist. 19, 1 (March 1993), 61-74.</p>
<p class="citation">Evert, Stefan (2004): <a href="http://purl.org/stefan.evert/PUB/Evert2004phd.pdf">The Statistics of Word Cooccurrences: Word Pairs and Collocations.</a> PhD dissertation, IMS, University of Stuttgart. Published in 2005, URN urn:nbn:de:bsz:93-opus-23714.</p></div>
<div style="display:none;" id="logdice_ttt">Log-Dice: $$14 + \log_2 \frac{2f_{1,2}}{f_1 + f_2}$$<p class="citation">Rychlý, Pavel (2008): <a href="http://www.fi.muni.cz/usr/sojka/download/raslan2008/13.pdf">A lexicographer-friendly association score.</a> In Proceedings of Recent Advances in Slavonic Natural Language Processing, RASLAN, 6–9, 2008</p></div>
- <div style="display:none;" id="logdiceaf_ttt">Log-Dice using "auto-focus", i.e. the window that yields the highest score.</div>
+ <div style="display:none;" id="logdiceaf_ttt">Log-Dice using "auto-focus", i.e. the highest value reached by any – not necessarily contiguous – selection \(S\) of positions: $$\max_S \left(14 + \log_2 \frac{2f_{1,2}(S)}{f_2 + f_1|S|}\right)$$ Dividing by the width of the selection makes the measure sensitive to how concentrated a pair is, which is what distinguishes actual collocations from words that merely occur in the same contexts. Note that LDaf is therefore not on the same scale as LD and can be higher or lower than it.</div>
<div id="ids_logo">
<a href="http://www.ids-mannheim.de/" target="_blank"><img src="/derekovecs/img/IDS-neu_farbig.svg" alt="Leibniz-Institut für Deutsche Sprache"/></a>
</div>
@@ -942,7 +942,7 @@
<th id="logdice_tt">LD</th>
<th id="logdiceaf_tt">LDaf</th>
<th id="delta_tt" title="Delta to log-Dice score in reference corpus. ⚠: If the collocate is not within the top 200 of the reference corpus, a reference value of min(lD)-0.1 is assumed.">Δ</th>
- <th id="af_win" title="Positions around the target word that are selected by the auto-focus function are marked with ◾. Positions at which the collocate appears at least once are marked with ◽."><span class="regular"><%= loc 'af_window' %></span></th>
+ <th id="af_win" title="Positions around the target word that are selected by the auto-focus function are marked with ◾. Positions at which the collocate appears at least once are marked with ◽. LL, MI, MI², MI³ and nPMI are computed over all positions marked here, the attested ones, rather than over the whole context window, so that a collocate occurring in few positions is rated higher than one spread over many. LD does not depend on any window; LDaf is the auto-focus score of the ◾ positions and is not on the same scale as LD."><span class="regular"><%= loc 'af_window' %></span></th>
<th title="PMI³ restricted to left neighbour">l-PMI³</th>
<th title="PMI³ restricted to right neighbour">r-PMI³</th>
<th title="nPMI restricted to left neighbour">l-nPMI</th>