templates/en/about.html.ep - ids-kl/derekovecs - Gitiles

 <html>
 <p>
     DeReKoVecs (Fankhauser & Kupietz 2022, 2017, 2019; Kupietz et al. 2018) is the new open lab of the Corpus Linguistics group at IDS Mannheim. Similar to the
     Collocation Database <a href="http://corpora.ids-mannheim.de/">CCDB</a> (Keibel &amp; Belica 2007, Belica 2011), DeReKoVecs serves for investigating and comparing
     of measurements, dimension reduction procedures, visualizations etc., to track down detailed paradigmatic and syntagmatic relations
     between words based on their use in very large corpora such as the German Reference Corpus DeReKo (Kupietz et al. 2010, 2018) or the Reference Corpus of the Contemporary Romanian Language CoRoLa (Barbu Mititelu et al. 2018, Cristea et al. 2017).
 </p>
 <p>
     The models used here are based on an extension of word2vec (Mikolov et al. 2013), wang2vec (Ling et al. 2015)
     and on the other hand on simple co-occurence counts and analysis methods that operate on these.
 </p>
 <h3>Queries</h3>
 <p>
     Several word forms separated by blanks are added into one query vector, i.e. the search is
     around the centre of the alternatives, while a &ldquo;|&rdquo; separates several independent
     searches. Beyond that, &ldquo;+&rdquo; and &ldquo;-&rdquo; do arithmetic in the vector space:
     <span class="mono">König - Mann + Frau</span> searches the neighbours of the position
     vec(König) - vec(Mann) + vec(Frau) rather than the position of a single word. A sign is an
     operator only at the beginning of a token, so hyphenated words such as
     <span class="mono">Nord-Süd-Dialog</span> remain searchable.
 </p>
 <p>
     The syntagmatic view follows along. The predictive score of a candidate is
     \(\sigma(q \cdot u)\), which is linear in \(q\) before the sigmoid, so an expression
     yields the contexts that the computed position predicts &ndash; for &ldquo;König - Mann +
     Frau&rdquo; the ones König and Frau predict but Mann does not. The positive terms are
     averaged rather than summed, so that the values stay in the range a single word form
     occupies: the sigmoid is informative over a narrow range only, and the strongest collocates
     of a single word already sit at the top of it. A balanced analogy (+1 -1 +1) is one term and
     is therefore on the same scale as a single word form.
 </p>
 <p>
     The count based collocators, in contrast, remain restricted to a single word form: they are
     looked up per node in the co-occurrence database and cannot be computed from the query
     vector. Operands that are not in the vocabulary are named above the result and left out of
     the computation.
 </p>

 <h3>Effective window size and auto focus</h3>
 <p>
     The association measures LL, MI, MI², MI³ and nPMI are not computed over the whole context
     window, but only over the <em>attested positions</em>, i.e. those positions in which the
     collocate occurs at least once. The expected frequency is based on this effective window,
     which rates a collocate occurring in few positions higher than one spread over many, and thus
     helps to tell actual collocations from words that merely occur in the same contexts. Note
     that the values are therefore not directly comparable to those of tools that always compute
     with a fixed window size.
 </p>
 <p>
     LDaf, in contrast, is the <em>auto focus</em> score (Belica), i.e. the highest value that any
     – not necessarily contiguous – selection of positions reaches, an idea also known from the
     collocation database <a href="http://corpora.ids-mannheim.de/">CCDB</a>. As its denominator
     grows with the width of the selection, it rewards pairs whose co-occurrences are
     concentrated, which is why it usually orders collocates most usefully and is used as the
     default order here. It is Dice-like, but not Log-Dice, so it is not on the same scale as LD
     and can be higher or lower than it. Both can be read off the auto-focus column of the
     collocation table, where the attested positions are marked with ◽ and those selected by the
     auto focus with ◾. LD itself does not depend on any window.
 </p>

 <h3>Disclaimer</h3>
 <p>
     Please note that  – unlike e.g. <a href="https://korap.ids-mannheim.de/">KorAP</a> – we cannot operate DeReKoVecs
     as a service with high reliability. DeReKoVecs is only intended as a glass laboratory.
     Things can change unannounced at any time and even a complete breakdown over a longer period of time cannot be excluded with certainty.
     If you would like to use DeReKoVecs for your own research in the longer term, please contact us.
 </p>
 <h3>Changes</h3>
 <p>
     <strong>2026-09-06</strong>

         The search field understands vector arithmetic, see <em>Queries</em> above.
 </p>
 <p>
     <strong>2026-09-04</strong>

         Log-Dice (LD) is now computed exactly as defined by Rychlý (2008),
         \(14 + \log_2 \frac{2f_{1,2}}{f_1 + f_2}\), without the window size factor that used to be
         applied to \(f_1\). Values are therefore higher than before, by up to \(\log_2\) of the
         window size, and directly comparable to those of other tools such as Sketch Engine.
         LL changes as well: its contingency table is now scaled with the window size as a whole
         rather than in its row total alone.
         LDaf is unchanged, as is the order of the collocates: it is the auto focus score, which
         keeps the window size factor deliberately, since that is what makes it sensitive to how
         concentrated a pair is. MI, MI², MI³ and nPMI are unaffected, as they only need the
         expected co-occurrence frequency, which is unchanged.
 </p>

 <h3>Scheduled Downtimes and Announcements</h3>
 <dl id="downtimes">
     <dt></dt><dd>–</dd>
 </dl>
 </html>
	<html>
	<p>
	DeReKoVecs (Fankhauser & Kupietz 2022, 2017, 2019; Kupietz et al. 2018) is the new open lab of the Corpus Linguistics group at IDS Mannheim. Similar to the
	Collocation Database <a href="http://corpora.ids-mannheim.de/">CCDB</a> (Keibel & Belica 2007, Belica 2011), DeReKoVecs serves for investigating and comparing
	of measurements, dimension reduction procedures, visualizations etc., to track down detailed paradigmatic and syntagmatic relations
	between words based on their use in very large corpora such as the German Reference Corpus DeReKo (Kupietz et al. 2010, 2018) or the Reference Corpus of the Contemporary Romanian Language CoRoLa (Barbu Mititelu et al. 2018, Cristea et al. 2017).
	</p>
	<p>
	The models used here are based on an extension of word2vec (Mikolov et al. 2013), wang2vec (Ling et al. 2015)
	and on the other hand on simple co-occurence counts and analysis methods that operate on these.
	</p>
	<h3>Queries</h3>
	<p>
	Several word forms separated by blanks are added into one query vector, i.e. the search is
	around the centre of the alternatives, while a “\|” separates several independent
	searches. Beyond that, “+” and “-” do arithmetic in the vector space:
	<span class="mono">König - Mann + Frau</span> searches the neighbours of the position
	vec(König) - vec(Mann) + vec(Frau) rather than the position of a single word. A sign is an
	operator only at the beginning of a token, so hyphenated words such as
	<span class="mono">Nord-Süd-Dialog</span> remain searchable.
	</p>
	<p>
	The syntagmatic view follows along. The predictive score of a candidate is
	\(\sigma(q \cdot u)\), which is linear in \(q\) before the sigmoid, so an expression
	yields the contexts that the computed position predicts – for “König - Mann +
	Frau” the ones König and Frau predict but Mann does not. The positive terms are
	averaged rather than summed, so that the values stay in the range a single word form
	occupies: the sigmoid is informative over a narrow range only, and the strongest collocates
	of a single word already sit at the top of it. A balanced analogy (+1 -1 +1) is one term and
	is therefore on the same scale as a single word form.
	</p>
	<p>
	The count based collocators, in contrast, remain restricted to a single word form: they are
	looked up per node in the co-occurrence database and cannot be computed from the query
	vector. Operands that are not in the vocabulary are named above the result and left out of
	the computation.
	</p>

	<h3>Effective window size and auto focus</h3>
	<p>
	The association measures LL, MI, MI², MI³ and nPMI are not computed over the whole context
	window, but only over the <em>attested positions</em>, i.e. those positions in which the
	collocate occurs at least once. The expected frequency is based on this effective window,
	which rates a collocate occurring in few positions higher than one spread over many, and thus
	helps to tell actual collocations from words that merely occur in the same contexts. Note
	that the values are therefore not directly comparable to those of tools that always compute
	with a fixed window size.
	</p>
	<p>
	LDaf, in contrast, is the <em>auto focus</em> score (Belica), i.e. the highest value that any
	– not necessarily contiguous – selection of positions reaches, an idea also known from the
	collocation database <a href="http://corpora.ids-mannheim.de/">CCDB</a>. As its denominator
	grows with the width of the selection, it rewards pairs whose co-occurrences are
	concentrated, which is why it usually orders collocates most usefully and is used as the
	default order here. It is Dice-like, but not Log-Dice, so it is not on the same scale as LD
	and can be higher or lower than it. Both can be read off the auto-focus column of the
	collocation table, where the attested positions are marked with ◽ and those selected by the
	auto focus with ◾. LD itself does not depend on any window.
	</p>

	<h3>Disclaimer</h3>
	<p>
	Please note that – unlike e.g. <a href="https://korap.ids-mannheim.de/">KorAP</a> – we cannot operate DeReKoVecs
	as a service with high reliability. DeReKoVecs is only intended as a glass laboratory.
	Things can change unannounced at any time and even a complete breakdown over a longer period of time cannot be excluded with certainty.
	If you would like to use DeReKoVecs for your own research in the longer term, please contact us.
	</p>
	<h3>Changes</h3>
	<p>
	<strong>2026-09-06</strong>

	The search field understands vector arithmetic, see <em>Queries</em> above.
	</p>
	<p>
	<strong>2026-09-04</strong>

	Log-Dice (LD) is now computed exactly as defined by Rychlý (2008),
	\(14 + \log_2 \frac{2f_{1,2}}{f_1 + f_2}\), without the window size factor that used to be
	applied to \(f_1\). Values are therefore higher than before, by up to \(\log_2\) of the
	window size, and directly comparable to those of other tools such as Sketch Engine.
	LL changes as well: its contingency table is now scaled with the window size as a whole
	rather than in its row total alone.
	LDaf is unchanged, as is the order of the collocates: it is the auto focus score, which
	keeps the window size factor deliberately, since that is what makes it sensitive to how
	concentrated a pair is. MI, MI², MI³ and nPMI are unaffected, as they only need the
	expected co-occurrence frequency, which is unchanged.
	</p>

	<h3>Scheduled Downtimes and Announcements</h3>
	<dl id="downtimes">
	<dt></dt><dd>–</dd>
	</dl>
	</html>