| commit | f24a6187e302cd6ac264dfba5e0ebb1baf2417b9 | [log] [tgz] |
|---|---|---|
| author | Akron <nils@diewald-online.de> | Tue Sep 03 17:20:32 2024 +0200 |
| committer | Marc Kupietz <kupietz@ids-mannheim.de> | Tue Sep 03 18:02:43 2024 +0200 |
| tree | 65d433ab58c47dbce4577f2c2b118c55f8519086 | |
| parent | 98a2da187e117d65436deef57f5d812c55f6470a [diff] |
Please SELinux Change-Id: I008e5912705301dee2e00701d781348759d1c4e6
The KorAP Corpus Analysis Platform consists of several independent components, but they can easily be installed together using Docker. This repository contains a recipe to install all components needed to run KorAP on a local machine with a single command.
In addition, all relevant tools are installed and made available that are necessary for data conversion and indexing of corpora in the widely used TEI-P5 (I5) format for KorAP. For different options of the tools we refer to the respective repositories.
Install docker and docker compose (>= v2; as a CLI plugin).
To get KorAP running, an index is required. For testing, there is a test index available as a docker image. Just run
INDEX='example-index' docker compose -p korap --profile=lite --profile=example up
to start the example image and the service with Linux (See here for more information on Windows).
Otherwise it's possible to download the sample index provided by Kustvakt. To download, intialize and run KorAP pointing to that index folder (in this example stored in the index folder in the local directory), run
INDEX=./index docker compose -p korap --profile=lite up
This will make the frontend be available at localhost:64543.
To use your own index, please follow the instructions on Corpus Conversion first.
To run the service with a user management system, first create a directory data in your working directory and then start it with
INDEX=./index docker compose -p korap --profile=full up
Login with user1 and password1. To change authentication settings, see the /kusvakt/ldap folder inside the docker container and Kustvakt's LDAP Settings Wiki for documentation.
In order to create an index based on existing corpus data, some conversion steps are usually necessary. In the case of a conversion from TEI P5 (I5) format, the tools required for this have already been installed with the command above.
In the following we take the open part of the Dortmunder Chatkorpus 2.2 (Beißwenger & Storrer 2008) as an example to build an index.
The file is located at example/dck-part1.i5.xml.
The command ...
docker run --rm \ -v ${PWD}/example:/data:z korap/kalamar:latest \ tei2korapxml \ --inline-tokens '!cmc#morpho' \ --no-tokenizer \ --input /data/dck-part1.i5.xml > dck.zip
... will convert the i5 file into a KorAP-XML file using tei2korapxml.
This format is designed to add further arbitrary annotations to the primary data. In this example, however, we will stick with the inline annotations that the example corpus already contains and will make available later under the label cmc.
To convert the KorAP-XML archive in a second step into individual Krill compatible JSON files, the following command ...
mkdir json
docker run --rm -u root \ -v ${PWD}:/kalamar/data:z korap/kalamar:latest\ korapxml2krill archive \ -z \ -i /kalamar/data/dck.zip \ --jobs -1 \ --token 'cmc#morpho' \ --base-paragraphs 'DeReKo#Structure' \ --base-sentences 'DeReKo#Structure' \ -o ./data/json/
... will use korapxml2krill.
Depending on how the source data is designed, different parameters must be specified for the conversion.
Here, the inline token annotation is used as the basis for word tokenization, and the included document structure is used for default annotation of sentence and paragraph boundaries.
Krill's indexer tool can now be used to index the JSON files:
mkdir index
docker run -u root --rm -v ${PWD}:/data:z korap/kustvakt \ Krill-Indexer.jar -c /kustvakt/kustvakt-lite.conf \ -i /data/json -o /data/index/
After that, the index can be loaded with the aforementioned call and is searchable via the browser.
Windows with Powershell requires environment variables to pass in a different way. In addition the PWD variable is not set beforehand. To run, e.g., the KorAP one-liner with Windows, you have to start
$env:INDEX='example-index'; $env:PWD='.'; docker compose -p korap --profile=lite --profile=example up
Authors: Nils Diewald, Harald Lüngen, Marc Kupietz
Copyright (c) 2022-2024, IDS Mannheim, Germany
KorAP-Docker is published under the BSD-2 License.
The example corpus corresponds to the release part of the Dortmunder Chatkorpus 2.2 as prepared by DeReKo. The corpus is released under the CC BY 4.0 License. Legal restrictions may arise from data protection legislation.
Beißwenger, Michael / Storrer, Angelika (2008): Corpora of Computer-Mediated Communication. In: Anke Lüdeling & Merja Kytö (Eds): Corpus Linguistics. An International Handbook. Volume 1. Berlin. New York (Handbooks of Linguistics and Communication Science 29.1), pp. 292--308.