| commit | fd966c59178d7fe53bf517ed086d7c78c1043ea6 | [log] [tgz] |
|---|---|---|
| author | Akron <nils@diewald-online.de> | Mon Sep 03 15:29:37 2018 +0200 |
| committer | Akron <nils@diewald-online.de> | Mon Sep 03 15:29:37 2018 +0200 |
| tree | 0ad990f1ef1b8f271923d0ec8bfb46e3eb727e5b | |
| parent | 2423bba708a1036e6b11e6ce547cf23578f7a12c [diff] |
Test VC with large vector and large index Change-Id: I5d5ae5a31620ae868c222e38301f8248dcbc085d
A Corpusdata Retrieval Index using Lucene for Look-Ups
Description
Krill is a Lucene based search engine for large annotated corpora, used as a backend component of the KorAP Corpus Analysis Platform at the IDS Mannheim.
! This software is in its early stages and not stable yet! Use it on your own risk!
Features
Krill is the reference implementation for KoralQuery, covering most of the protocols features, including ...
Fulltext search
"Find all occurrences of the phrase 'sea monster'!"
"Find all case-insensitive words matching the regular expression /krak./!"*Token-based annotation search
"Find all plural nouns in accusative!"Span-based annotation search
"Find all nominal phrases!"Distance search
"Find a verb that is in a distance of five words to the noun 'Squid'!"Positional search
"Find a noun at the end of a nominal phrase!"Nested queries
"Find a determiner at the beginning of a named entity, that occurs at the end of a sentence!"and many more ...
Multiple annotation resources; Virtual Collections; Partial highlightings; Support for overlapping spans; Relational queries; Hierarchical queries ...
Prerequisites
Java 7 or 8 SDK, Git, Maven 3. Further dependencies are resolved using Maven.
Setup
To fetch the latest version of Krill ...
$ git clone https://github.com/KorAP/Krill $ cd Krill
To run the test suite ...
$ mvn clean test
To build a Krill library and install it in your local Maven repository (needed for Kustvakt) ...
$ mvn install
Caveats
Krill operates on tokens and is limited to a single tokenization stream. Token annotations therefore have to rely on that tokenization, span annotations have to wrap at least one token. Punctuations are currently not supported. The order of results is currently bound to the order of documents in the index, but this is likely to change.
Development and License
Authors: Nils Diewald, Eliza Margaretha
Copyright (c) 2013-2018, IDS Mannheim, Germany
Krill is developed as part of the KorAP Corpus Analysis Platform at the Institute for German Language (IDS), funded by the Leibniz-Gemeinschaft and supported by the KobRA project, funded by the Federal Ministry of Education and Research (BMBF).
For recent changes and compatibility issues, please consult the Changes file.
Krill is published under the BSD-2 License. The Eclipse format style is based on the default style in Eclipse, licensed under the Eclipse Public License. Parts of the test corpus by courtesy of the DeReKo project.
To cite this work, please refer to:
Diewald, Nils and Margaretha, Eliza (2017): Krill: KorAP search and analysis engine. In: Journal for Language Technology and Computational Linguistics (JLCL), 31 (1), pp. 63-80.
Contributions
Contributions to Krill are very welcome! Before contributing, please reformat your code changes according to the KorAP style guideline, provided by means of an Eclipse style sheet. You can either reformat using Eclipse or (recommended) using Maven with the command
$ mvn java-formatter:format
References
Annotation tools and models used in preparation of the test corpora are based on the following work:
Belica, Cyril (1994): A German Lemmatizer. MECOLB Final Report MLAP93-21/WP2. Luxemburg.
Bohnet, Bernd (2010): Top accuracy and fast dependency parsing is not a contradiction. In Proceedings of COLING, pp 89–97, Beijing, China.
Bohnet, Bernd, Joakim Nivre, Igor Boguslavsky, Richard Farkas, Filip Ginter, and Jan Hajic (2013): Joint Morphological and Syntactic Analysis for Richly Inflected Languages. Transactions of the Association for Computational Linguistics, 1, pp. 415-428.
Faruqui, Manaal and Sebastian Padó (2010): Training and Evaluating a German Named Entity Recognizer with Semantic Generalization. In Proceedings of KONVENS 2010, Saarbrücken, Germany.
Finkel, Jenny R. and Christopher D. Manning (2009): Joint parsing and named entity recognition. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, NAACL ’09, pp. 326–334, Stroudsburg, PA, USA.
Hockenmaier, Julia, Gann Bierner, and Jason Baldridge (2000): Providing Robustness for a CCG system. In Proceedings of the ESSLLI Workshop on Linguistic Theory and Grammar Implementation, Birmingham, United Kingdom.
Manning, Christopher D., Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky (2014): The Stanford CoreNLP Natural Language Processing Toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 55-60.
Schmid, Helmut (1995): Improvements in Part-of-Speech Tagging with an Application to German. In Proceedings of the ACL SIGDAT-Workshop. Dublin, Ireland.
Schmid, Helmut (1994): Probabilistic Part-of-Speech Tagging Using Decision Trees. In Proceedings of International Conference on New Methods in Language Processing. Manchester, United Kingdom.