| Marc Kupietz | d6dfdf0 | 2019-12-06 09:53:34 +0100 | [diff] [blame] | 1 | <html> |
| 2 | <p> |
| Marc Kupietz | b37bd84 | 2024-04-05 11:22:30 +0200 | [diff] [blame] | 3 | DeReKoVecs (Fankhauser & Kupietz 2022, 2017, 2019; Kupietz et al. 2018) is the new open lab of the Corpus Linguistics group at IDS Mannheim. Similar to the |
| Marc Kupietz | d6dfdf0 | 2019-12-06 09:53:34 +0100 | [diff] [blame] | 4 | Collocation Database <a href="http://corpora.ids-mannheim.de/">CCDB</a> (Keibel & Belica 2007, Belica 2011), DeReKoVecs serves for investigating and comparing |
| 5 | of measurements, dimension reduction procedures, visualizations etc., to track down detailed paradigmatic and syntagmatic relations |
| Marc Kupietz | 014220c | 2024-08-28 15:33:43 +0200 | [diff] [blame] | 6 | between words based on their use in very large corpora such as the German Reference Corpus DeReKo (Kupietz et al. 2010, 2018) or the Reference Corpus of the Contemporary Romanian Language CoRoLa (Barbu Mititelu et al. 2018, Cristea et al. 2017). |
| Marc Kupietz | d6dfdf0 | 2019-12-06 09:53:34 +0100 | [diff] [blame] | 7 | </p> |
| 8 | <p> |
| 9 | The models used here are based on an extension of word2vec (Mikolov et al. 2013), wang2vec (Ling et al. 2015) |
| 10 | and on the other hand on simple co-occurence counts and analysis methods that operate on these. |
| 11 | </p> |
| Marc Kupietz | 1e00eb8 | 2026-09-04 16:38:30 +0200 | [diff] [blame] | 12 | <h3>Effective window size and auto focus</h3> |
| 13 | <p> |
| 14 | The association measures LL, MI, MI², MI³ and nPMI are not computed over the whole context |
| 15 | window, but only over the <em>attested positions</em>, i.e. those positions in which the |
| 16 | collocate occurs at least once. The expected frequency is based on this effective window, |
| 17 | which rates a collocate occurring in few positions higher than one spread over many, and thus |
| 18 | helps to tell actual collocations from words that merely occur in the same contexts. Note |
| 19 | that the values are therefore not directly comparable to those of tools that always compute |
| 20 | with a fixed window size. |
| 21 | </p> |
| 22 | <p> |
| 23 | LDaf, in contrast, is the <em>auto focus</em> score (Belica), i.e. the highest value that any |
| 24 | – not necessarily contiguous – selection of positions reaches, an idea also known from the |
| 25 | collocation database <a href="http://corpora.ids-mannheim.de/">CCDB</a>. As its denominator |
| 26 | grows with the width of the selection, it rewards pairs whose co-occurrences are |
| 27 | concentrated, which is why it usually orders collocates most usefully and is used as the |
| 28 | default order here. It is Dice-like, but not Log-Dice, so it is not on the same scale as LD |
| 29 | and can be higher or lower than it. Both can be read off the auto-focus column of the |
| 30 | collocation table, where the attested positions are marked with ◽ and those selected by the |
| 31 | auto focus with ◾. LD itself does not depend on any window. |
| 32 | </p> |
| 33 | |
| Marc Kupietz | d6dfdf0 | 2019-12-06 09:53:34 +0100 | [diff] [blame] | 34 | <h3>Disclaimer</h3> |
| 35 | <p> |
| Marc Kupietz | d68f5c3 | 2024-08-23 11:50:32 +0200 | [diff] [blame] | 36 | Please note that – unlike e.g. <a href="https://korap.ids-mannheim.de/">KorAP</a> – we cannot operate DeReKoVecs |
| Marc Kupietz | d6dfdf0 | 2019-12-06 09:53:34 +0100 | [diff] [blame] | 37 | as a service with high reliability. DeReKoVecs is only intended as a glass laboratory. |
| 38 | Things can change unannounced at any time and even a complete breakdown over a longer period of time cannot be excluded with certainty. |
| 39 | If you would like to use DeReKoVecs for your own research in the longer term, please contact us. |
| 40 | </p> |
| Marc Kupietz | 4df5533 | 2024-04-05 09:55:56 +0200 | [diff] [blame] | 41 | <h3>Scheduled Downtimes and Announcements</h3> |
| Marc Kupietz | 9ae184c | 2023-11-04 18:21:54 +0100 | [diff] [blame] | 42 | <dl id="downtimes"> |
| Marc Kupietz | 1e00eb8 | 2026-09-04 16:38:30 +0200 | [diff] [blame] | 43 | <dt>2026-09-04</dt> |
| 44 | <dd> |
| 45 | Log-Dice (LD) is now computed exactly as defined by Rychlý (2008), |
| 46 | \(14 + \log_2 \frac{2f_{1,2}}{f_1 + f_2}\), without the window size factor that used to be |
| 47 | applied to \(f_1\). Values are therefore higher than before, by up to \(\log_2\) of the |
| 48 | window size, and directly comparable to those of other tools such as Sketch Engine. |
| 49 | LDaf is unchanged, as is the order of the collocates: it is the auto focus score, which |
| 50 | keeps the window size factor deliberately, since that is what makes it sensitive to how |
| 51 | concentrated a pair is. The other measures are unaffected as well, the window size |
| 52 | belonging in their expected frequency. |
| 53 | </dd> |
| Marc Kupietz | 9ae184c | 2023-11-04 18:21:54 +0100 | [diff] [blame] | 54 | </dl> |
| Marc Kupietz | d6dfdf0 | 2019-12-06 09:53:34 +0100 | [diff] [blame] | 55 | </html> |