Write the memory mappable model form after training
derekovecs memory maps <model>.vecs and <model>.words next to the model. When
they are missing it builds them on its first start, which takes a while and
needs write access to the directory the models live in. A server in a
container usually has that directory mounted read only and then does not start
at all:
Converting /models/dereko-2026-i.vecs to memory mappable structures
Cannot open /models/dereko-2026-i.vecs.words or .../dereko-2026-i.vecs.vecs
dereko2vec writes both files right after saving the vectors now, and
vecs2mmap does it for models that were trained earlier.
The conversion mirrors init_net() of derekovecs including the order of the
floating point operations. Verified against the model of the derekovecs test
data: the vectors come out byte identical, and derekovecs serves the same
neighbours and collocators from the generated files, with the model directory
mounted read only and without converting anything itself.
Words are cut off at 49 characters plus the terminator instead of filling all
50 bytes, so that every entry is terminated within its own record.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: I297e2db98bd425aa388e441aca6d2b0b14f0c66c
diff --git a/src/vecs2mmap.c b/src/vecs2mmap.c
new file mode 100644
index 0000000..3d72355
--- /dev/null
+++ b/src/vecs2mmap.c
@@ -0,0 +1,29 @@
+/* Writes the memory mappable form of an already trained model, so that
+ * derekovecs does not have to build it itself on its first start. dereko2vec
+ * does this right after training, this tool is for models that were trained
+ * before, or whose converted files were lost.
+ */
+
+#include <stdio.h>
+#include <string.h>
+
+#include "mmap_vecs.h"
+
+int main(int argc, char **argv) {
+ int i;
+
+ if (argc < 2 || strcmp(argv[1], "-h") == 0) {
+ printf("usage: %s <model.vecs> [<model.vecs> ...]\n\n"
+ "Writes <model.vecs>.vecs and <model.vecs>.words, the memory\n"
+ "mappable form derekovecs uses.\n", argv[0]);
+ return argc < 2 ? 1 : 0;
+ }
+
+ for (i = 1; i < argc; i++) {
+ printf("Converting %s to memory mappable structures\n", argv[i]);
+ fflush(stdout);
+ if (convert_vecs_to_mmap(argv[i]) != 0)
+ return 1;
+ }
+ return 0;
+}