Add standalone MetaExporter tool for dumping index metadata
MetaExporter reads the stored per-document metadata of an existing
Lucene index directly from disk and prints selected (or all) fields as
TSV, CSV or line-delimited JSON, without going through the web service
API. This makes it possible to e.g. diff the textSigles of two index
instances to find missing or surplus texts, instead of crawling the
metadata via many slow API requests.
- New main class de.ids_mannheim.korap.index.MetaExporter with options
-i/--index, -f/--fields, --format, -o/--output, --no-header,
--multi-sep and --list-fields. The primary-data fields tokens/base are
always skipped.
- Packaged as a shaded Krill-MetaExporter.jar alongside Krill-Indexer.
- Usage documented in the pom.xml header comment and the class Javadoc.
- TestMetaExporter builds a one-document index from the bzk test
resources and verifies field listing and TSV/CSV/JSON export.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Change-Id: Ib3394e237cc77dcd04b5edd08b4e7934351796bd
diff --git a/pom.xml b/pom.xml
index 36beae8..3d8657c 100644
--- a/pom.xml
+++ b/pom.xml
@@ -29,8 +29,23 @@
for example:
$ java -jar target/Krill-Indexer.jar -c src/test/resources/krill.properties
- -i src/test/resources/bzk -o index/
-
+ -i src/test/resources/bzk -o index/
+
+ ** MetaExporter
+ # after packaging (see above)
+ # dumps the stored per-document metadata of an existing index,
+ # without using the web service API.
+
+ $ java -jar target/Krill-MetaExporter.jar -i [index directory]
+ -f [comma-separated fields] [format tsv|csv|json] [-o output file]
+
+ for example (list fields, then dump text sigles for diffing instances):
+
+ $ java -jar target/Krill-MetaExporter.jar -i index/ (option) list-fields
+ $ java -jar target/Krill-MetaExporter.jar -i index/ -f textSigle (option) no-header
+
+ (full option names use a double-dash prefix; see the MetaExporter Javadoc)
+
-->
<groupId>de.ids-mannheim.korap.krill</groupId>
@@ -278,6 +293,23 @@
</configuration>
</execution>
<execution>
+ <id>metaexporter</id>
+ <phase>package</phase>
+ <goals>
+ <goal>shade</goal>
+ </goals>
+ <configuration>
+ <createDependencyReducedPom>false</createDependencyReducedPom>
+ <transformers>
+ <transformer
+ implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
+ <mainClass>de.ids_mannheim.korap.index.MetaExporter</mainClass>
+ </transformer>
+ </transformers>
+ <finalName>${project.artifactId}-MetaExporter</finalName>
+ </configuration>
+ </execution>
+ <execution>
<id>server</id>
<phase>package</phase>
<goals>