tree d59698eebe5848b9fd9635ecb01d5354a6786977
parent f47b8cb9dc7009090022e4ba7d4831ce60e8b8fc
author Marc Kupietz <kupietz@ids-mannheim.de> 1785476882 +0900
committer Marc Kupietz <kupietz@ids-mannheim.de> 1785476882 +0900

Write the memory mappable model form after training

derekovecs memory maps <model>.vecs and <model>.words next to the model. When
they are missing it builds them on its first start, which takes a while and
needs write access to the directory the models live in. A server in a
container usually has that directory mounted read only and then does not start
at all:

  Converting /models/dereko-2026-i.vecs to memory mappable structures
  Cannot open /models/dereko-2026-i.vecs.words or .../dereko-2026-i.vecs.vecs

dereko2vec writes both files right after saving the vectors now, and
vecs2mmap does it for models that were trained earlier.

The conversion mirrors init_net() of derekovecs including the order of the
floating point operations. Verified against the model of the derekovecs test
data: the vectors come out byte identical, and derekovecs serves the same
neighbours and collocators from the generated files, with the model directory
mounted read only and without converting anything itself.

Words are cut off at 49 characters plus the terminator instead of filling all
50 bytes, so that every entry is terminated within its own record.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: I297e2db98bd425aa388e441aca6d2b0b14f0c66c