Bring the Readme up to date
The dependency was still given as libcollocatordb 1.5.0, while Makefile.PL
has refused to build against anything below 1.8.0 since the Log-Dice
correction. Anybody following the Readme got a build error instead of an
explanation.
The link into the source for the undocumented result components pointed at
templates/index.html.ep#684, which is the t-SNE iteration loop; the table
heads that carry them are around #938.
The image slimming section still described a single slim(toolkit) command
producing an image called ids-kl/derekovecs.slim. The tool is mint now, and
one command cannot do it: mint keeps what the running server touches, so it
needs a model to probe, which is why the example models stay in the first
image and Dockerfile.remove-example-data takes them out again afterwards.
The three steps are the ones the CI runs, with the sizes they produce.
The result of / was undocumented, unlike the one of getClassicCollocators,
so the fields of the syntagmatic entries had nowhere to be explained -
including max(z), which is the reason for looking at them at all. The
example is a real answer for "Grund".
Also: vector expressions in the opening paragraph, cutoff and threshold in
the getPosWiseW2VCollocators parameters, and the copyright year.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: Ide30904629510337bc620a4d3817a72cea0bce41
diff --git a/README.md b/README.md
index 8f983e3..dec08e4 100644
--- a/README.md
+++ b/README.md
@@ -2,6 +2,8 @@
Visualizes paradigmatic and syntagmatic relations between words based on [wang2vec](https://github.com/wlin12/wang2vec) / structured skip-n-gram (Ling et al. 2015) word embeddings (Mikolov et al. 2013) and word embedding networks.
+A query can be a vector expression: `König - Mann + Frau` is answered with the neighbours of the position that arithmetic yields, and with the contexts that position predicts. See [the `word` parameter](#the-word-parameter).
+
DeReKoVecs (Fankhauser / Kupietz 2017, 2019, 2022; Kupietz et al. 2018) serves as part of the new open lab of the Corpus Linguistics group at IDS Mannheim. Similar to the
Collocation Database [CCDB](http://corpora.ids-mannheim.de/) (Keibel / Belica 2007, Belica 2011), DeReKoVecs serves for investigating and comparing
of measurements, dimension reduction procedures, visualizations etc., to track down detailed paradigmatic and syntagmatic relations
@@ -11,7 +13,7 @@
### Dependencies
-* [libcollocatordb](https://korap.ids-mannheim.de/gerrit/plugins/gitiles/ids-kl/collocatordb) >= v1.5.0 (which builds against the rocksdb of the distribution, no rocksdb 5.11 from source needed)
+* [libcollocatordb](https://korap.ids-mannheim.de/gerrit/plugins/gitiles/ids-kl/collocatordb) >= v1.8.0 (which builds against the rocksdb of the distribution, no rocksdb 5.11 from source needed). `Makefile.PL` refuses to build against an older one: Log-Dice follows Rychlý's definition from 1.8.0 on, and the interface would show the previous values under the current explanations.
### Build and install
@@ -142,7 +144,7 @@
## Web Service API
-In addition to the web user interface, derekovecs also provides a web api which is however still very unsystematic and **not stable**. To figure out the meaning of still undocumented result components, have a look at the table head mouse-overs in the GUI or at the source code around [here](https://korap.ids-mannheim.de/gerrit/plugins/gitiles/ids-kl/derekovecs/+/refs/heads/master/templates/index.html.ep#684).
+In addition to the web user interface, derekovecs also provides a web api which is however still very unsystematic and **not stable**. To figure out the meaning of still undocumented result components, have a look at the table head mouse-overs in the GUI or at the source code around [here](https://korap.ids-mannheim.de/gerrit/plugins/gitiles/ids-kl/derekovecs/+/refs/heads/master/templates/index.html.ep#938), where the table heads carry them.
| Command | Parameters | Description |
|---------------------------|---------------------------------|-------------------------------------------------------------------|
@@ -177,6 +179,51 @@
returned in `unknown`; `operands` is the number of operands the query vector
was built from.
+#### Result of `/?word=…&json=1`
+
+```jsonc
+{
+ "word" : "Grund", // the query as it was parsed
+ "operands" : 1, // operands the query vector was built from,
+ // for the first "|" separated part
+ "unknown" : "", // blank separated query tokens that are not
+ // in the vocabulary and were left out
+ "list" : [ // one array per "|" separated part
+ [
+ {
+ "word" : "Grund",
+ "rank" : 492, // frequency rank in the vocabulary
+ "dist" : 0.99999976, // cosine similarity to the query vector
+ "vector" : [] // the length normalized embedding
+ }
+ ]
+ ],
+ "collocators" : [ // predictive collocates of the query vector,
+ // null for a model without a .net file
+ {
+ "word" : "dafür",
+ "rank" : 243, // frequency rank in the vocabulary
+ "dot" : 4.006, // max(z): highest score before the sigmoid,
+ // q . syn1neg[collocate, position]
+ "max" : 0.98129, // max(a): the same maximum after it. Read from
+ // a table quantizing z in steps of 1/83 and
+ // capping it at 6, so collocates can share a
+ // max(a) but not a max(z)
+ "average" : 0.09813, // mean activation over the auto focus positions
+ "prob" : 3.0353e-4, // activation sum over the auto focus positions,
+ // normalized by their total activation
+ "cprob" : 3.0353e-4, // co-norm of the column normalized activations
+ // over the auto focus positions
+ "overall" : 3.0365e-5,// activation sum over the whole window,
+ // normalized by the total window sum
+ "pos" : 16, // binary encoded auto focus positions
+ "heat" : [] // per position maximum activation, left to
+ // right, the node in the middle
+ }
+ ]
+}
+```
+
### Get classical (count-based) collocates
| Command | Parameters | Description |
@@ -226,7 +273,7 @@
| Command | Parameters | Description |
|----------------------------|-------------------------|------------------------------------------------------------|
-| /getPosWiseW2VCollocators | w(,max=200,format=json) | get top `max`predictive collocates position-wise of word w |
+| /getPosWiseW2VCollocators | w(,max=200,cutoff=750000,threshold=0.2,format=json) | get top `max` predictive collocates position-wise of the query `w`, which may be a vector expression |
### Examples
@@ -258,13 +305,32 @@
docker build -t idscorpuslinguistics/derekovecs .
```
-### Optional: Slim down image using [Slim(toolkit)](https://github.com/slimtoolkit/slim)
+### Optional: Slim down the image using [mint](https://github.com/mintoolkit/mint)
+
+The released image is built in three steps, as in the
+[CI pipeline](.gitlab-ci.yml). mint keeps what the server touches while it is
+running, so it needs a model to probe: the example models stay in the first
+image and are removed again in the last step.
```bash
-slim build --include-path /usr/local/share/perl5 --mount ./example-models:/example-models:z --env MOJO_CONFIG=/example-models/example-docker.conf idscorpuslinguistics/derekovecs
+# 1. build with the example models still in place
+sed -e 's/RUN rm -rf example-models//' Dockerfile > Dockerfile.large
+docker build -f Dockerfile.large -t idscorpuslinguistics/derekovecs:large .
+
+# 2. run the server, probe it over HTTP and drop everything it did not touch
+docker run --rm -v /var/run/docker.sock:/var/run/docker.sock mintoolkit/mint \
+ --crt-api-version 1.46 build --http-probe=true --http-probe-start-wait 60 \
+ --include-workdir=true --include-path=/usr/local \
+ --env MOJO_CONFIG=/derekovecs/example-models/example-docker.conf \
+ --tag idscorpuslinguistics/derekovecs:tmp \
+ idscorpuslinguistics/derekovecs:large
+
+# 3. remove the example models and the toolchain mint kept
+docker build -f Dockerfile.remove-example-data -t idscorpuslinguistics/derekovecs .
```
-Will build an image `ids-kl/derekovecs.slim` reduced to ~25% of the original size.
+mint takes the image from about 715 MB to 290 MB, and dropping the example
+models afterwards leaves about 109 MB.
### Run docker image
@@ -286,7 +352,7 @@
**Contributors**: [Peter Fankhauser](https://www.ids-mannheim.de/digspra/personal/fankhauser.html), [Rainer Perkuhn](https://www.ids-mannheim.de/digspra/personal/perkuhn.html), [Tim Feldmüller](https://www.ids-mannheim.de/digspra/personal/feldmueller.html)
-Copyright (c) 2016-2024, [Leibniz Institute for the German Language](http://www.ids-mannheim.de/), Mannheim, Germany
+Copyright (c) 2016-2026, [Leibniz Institute for the German Language](http://www.ids-mannheim.de/), Mannheim, Germany
DeReKoVecs is published under the [Apache 2.0 License](LICENSE).