tree 4c104f08038e061e3ff876691b6ca536be6d50b2
parent b6416be4b6192a6f498acca1e4fc3b05127c7dd6
author Marc Kupietz <kupietz@ids-mannheim.de> 1788439576 +0200
committer Marc Kupietz <kupietz@ids-mannheim.de> 1788439576 +0200

CA: drop collocates occurring less often than expected by default

collocationAnalysis() ranks and thresholds by logDice, which expresses
how salient a pair is rather than how surprising it is. A frequent word
can therefore appear among the top collocates although the node does not
attract it at all: among the collocates of Grund in a 5+5 window,
"Berlin" reaches a logDice of 3.84, close to "triftiger" at 3.96, while
co-occurring 1.74 bits less often than expected.

Filtering these out was documented as a recipe, dplyr::filter(O > E),
but a default that quietly rewards frequent, unattracted collocates is a
trap for exactly those users who do not know the measure well enough to
apply the recipe. The new minObservedExpectedRatio parameter therefore
defaults to 1, keeping only collocates that occur at least as often as
expected, which corresponds to a non-negative pmi. It can be raised to
demand a stronger contrast, or set to 0 for the unfiltered result of
earlier versions, e.g. in order to study repulsion.

The filter is applied wherever minOccur is, so it also governs which
collocates are recursed into, and it is passed on to the per virtual
corpus analyses. Rows without an expected frequency are kept rather than
silently dropped. collocationScoreQuery() is deliberately left alone:
there the pairs to score are given explicitly, so filtering them would
throw away the answer that was asked for.

Being collected from the call frame rather than from a list, the cacheAs
parameter check picked the new parameter up by itself, so a cache file
written with a different ratio is recomputed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: I98328a8766847b4c95363211a939403b1912e9b7