Score the first collocate of a query with the right window size
true_window_size counts the positions a pair was observed in and is
passed on as the window size of the expected frequency. It started at
one, but the first key of the iteration takes the same branch as any
further key of the same collocate, which increments it. The first
collocate of every query was therefore scored with a window one position
too wide, so that its expected frequency came out too high and its pmi,
npmi, md, lfmd and llr too low.
The test data shows this: for "Aluminium" and "Anwendungstechnologie",
ln_count is 16 and rn_count 0, so all 16 co-occurrences sit in the single
position left of the node, and the aggregate pmi has to be the pmi of
that position. It was 8.4592 while ln_pmi was 9.4592, exactly log2(2)
apart. They now agree, and npmi, md, lfmd and llr change accordingly,
each matching an independent calculation with a window of one.
Counting up from zero fixes it. Note that md_nws is unaffected, as it
deliberately uses the nominal window size, and logDice no longer takes a
window size at all.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: I58c26bcdf273e320442be5abb5728219682ac951
diff --git a/src/collocatordb.cc b/src/collocatordb.cc
index 4da2e06..6eda410 100644
--- a/src/collocatordb.cc
+++ b/src/collocatordb.cc
@@ -859,7 +859,12 @@
/* fixed size, so it lives on the stack and cannot be leaked */
uint64_t sumWindow[2 * WINDOW_SIZE];
memset(sumWindow, 0, sizeof(sumWindow));
- int true_window_size = 1;
+ /* Counted up from zero, since the first key of the iteration takes the same
+ branch as any further key of the same collocate, which increments it. It
+ used to start at one, so that the first collocate of every query was scored
+ with a window one position too wide, which inflated its expected frequency
+ and thus lowered its pmi, npmi, md, lfmd and llr. */
+ int true_window_size = 0;
int usedPositions = 0;
if (w1 > _vocab.size()) {
diff --git a/tests/basic_test.c b/tests/basic_test.c
index 53e2736..d98aac4 100644
--- a/tests/basic_test.c
+++ b/tests/basic_test.c
@@ -28,7 +28,7 @@
void test_collocation_scores() {
COLLOCATORDB* cdb = open_collocatordb(dbpath);
TEST_ASSERT(cdb != NULL);
- char *expected = " { \"f1\": 217,\"w1\":\"Aluminium\", \"N\": 152743, \"collocates\": [{\"word\":\"Anwendungstechnologie\",\"f2\":16,\"f\":16,\"npmi\":0.594849,\"pmi\":8.4592,\"llr\":188.227,\"lfmd\":16.4592,\"md\":12.4592,\"md_nws\":10.1373,\"dice\":0.137339,\"ld\":11.1358,\"ln_count\":16,\"rn_count\":0,\"ln_pmi\":9.4592,\"rn_pmi\":-1,\"ldaf\":11.1358,\"win\":32,\"afwin\":32}]}\n";
+ char *expected = " { \"f1\": 217,\"w1\":\"Aluminium\", \"N\": 152743, \"collocates\": [{\"word\":\"Anwendungstechnologie\",\"f2\":16,\"f\":16,\"npmi\":0.715481,\"pmi\":9.4592,\"llr\":211.02,\"lfmd\":17.4592,\"md\":13.4592,\"md_nws\":10.1373,\"dice\":0.137339,\"ld\":11.1358,\"ln_count\":16,\"rn_count\":0,\"ln_pmi\":9.4592,\"rn_pmi\":-1,\"ldaf\":11.1358,\"win\":32,\"afwin\":32}]}\n";
char *produced = get_collocation_scores_as_json(cdb, 62, 966);
TEST_CHECK(strcmp(produced, expected) == 0);
TEST_MSG("Expected: %s", expected);