| commit | d7e4cd4cd64313733406ddd3b43905d530600c5c | [log] [tgz] |
|---|---|---|
| author | Marc Kupietz <kupietz@ids-mannheim.de> | Thu Feb 22 18:08:11 2024 +0100 |
| committer | Marc Kupietz <kupietz@ids-mannheim.de> | Thu Feb 22 18:08:11 2024 +0100 |
| tree | d4beffdad12c86ddbb68f236f7562c658f09c602 | |
| parent | e782b85a9f044922fd076a5ae9ce033ba5fcff8d [diff] |
CI: Bump actions/cache to v4 Change-Id: I3be9e874235b51a1c83edcde71d3a9d74778b8bb
The KorAP Corpus Analysis Platform consists of several independent components, but they can easily be installed together using Docker. This repository contains a recipe to install all components needed to run KorAP on a local machine with a single command.
In addition, all relevant tools are installed and made available that are necessary for data conversion and indexing of corpora in the widely used TEI-P5 (I5) format for KorAP. For different options of the tools we refer to the respective repositories.
Install docker and docker compose (>= v2).
To get KorAP running, an index is required. For testing, there is a test index available as a docker image. Just run
INDEX='example-index' docker-compose -p korap --profile=lite --profile=example up
to start the example image. Otherwise it's possible to download the sample index provided by Kustvakt. To download, intialize and run KorAP pointing to that index folder (in this example stored in the index folder in the local directory), run
INDEX=./index docker-compose -p korap --profile=lite up
This will make the frontend be available at localhost:64543.
To use your own index, please follow the instructions on Corpus Conversion first.
To run the service with a user management system, create a file $(pwd)/data/kalamar.production.conf containing the following configuration:
{ Kalamar => { plugins => ['Auth'] }, 'Kalamar-Auth' => { client_file => '/kalamar/super_client_info' } }
Then start the service with
INDEX=./index docker-compose -p korap --profile=full up
Login with user1 and password1. To change authentication settings, see the /kusvakt/ldap folder inside the docker container and Kustvakt's LDAP Settings Wiki for documentation.
In order to create an index based on existing corpus data, some conversion steps are usually necessary. In the case of a conversion from TEI P5 (I5) format, the tools required for this have already been installed with the command above.
In the following we take the open part of the Dortmunder Chatkorpus 2.2 (Beißwenger & Storrer 2008) as an example to build an index.
The file is located at example/dck-part1.i5.xml.
The command ...
docker run --rm \ -v ${PWD}/example:/data:z korap/kalamar:latest-conv \ tei2korapxml \ --inline-tokens '!cmc#morpho' \ --input /data/dck-part1.i5.xml > dck.zip
... will convert the i5 file into a KorAP-XML file using tei2korapxml.
This format is designed to add further arbitrary annotations to the primary data. In this example, however, we will stick with the inline annotations that the example corpus already contains and will make available later under the label cmc.
To convert the KorAP-XML archive in a second step into individual Krill compatible JSON files, the following command ...
mkdir json
docker run --rm -u root \ -v ${PWD}:/kalamar/data:z korap/kalamar:latest-conv\ korapxml2krill archive \ -z \ -i /kalamar/data/dck.zip \ --jobs -1 \ --token 'cmc#morpho' \ --base-paragraphs 'DeReKo#Structure' \ --base-sentences 'DeReKo#Structure' \ -o ./data/json/
... will use korapxml2krill.
Depending on how the source data is designed, different parameters must be specified for the conversion.
Here, the inline token annotation is used as the basis for word tokenization, and the included document structure is used for default annotation of sentence and paragraph boundaries.
Krill's indexer tool can now be used to index the JSON files:
mkdir index
docker run -u root --rm -v ${PWD}:/data:z korap/kustvakt \ Krill-Indexer.jar -c /kustvakt/kustvakt-lite.conf \ -i /data/json -o /data/index/
After that, the index can be loaded with the aforementioned call and is searchable via the browser.
Authors: Nils Diewald, Harald Lüngen, Marc Kupietz
Copyright (c) 2022, IDS Mannheim, Germany
KorAP-Docker is published under the BSD-2 License.
The example corpus corresponds to the release part of the Dortmunder Chatkorpus 2.2 as prepared by DeReKo. The corpus is released under the CC BY 4.0 License. Legal restrictions may arise from data protection legislation.
Beißwenger, Michael / Storrer, Angelika (2008): Corpora of Computer-Mediated Communication. In: Anke Lüdeling & Merja Kytö (Eds): Corpus Linguistics. An International Handbook. Volume 1. Berlin. New York (Handbooks of Linguistics and Communication Science 29.1), pp. 292--308.