Gitiles
Code Review Sign In
korap.ids-mannheim.de / lora-sp / bunc2tei / 02a7b2a30b2565ebc2cc5ce931eb734e2bdf668d
commit02a7b2a30b2565ebc2cc5ce931eb734e2bdf668d[log] [tgz]
authorlora-sp <lora.spassova@swhk.ids-mannheim.de>Thu Apr 06 11:30:04 2023 +0200
committerlora-sp <lora.spassova@swhk.ids-mannheim.de>Thu Apr 06 11:30:04 2023 +0200
tree997357c48bc8bd29eb5aaeb8c15df5fa9e27e76b
parent2f3bd9fefde8cd58abc937c3efeaa1c03799ae62 [diff]
deleted folders

Change-Id: Ia58e4e90b2ca7729e6ae9831a8a24a104d73b61f
2 files changed
tree: 997357c48bc8bd29eb5aaeb8c15df5fa9e27e76b
  1. bunc2tei.py
  2. escape_amp.py
  3. ill-formed_docs.txt
  4. readme.md
readme.md

Creating a TEI corpus file out of xml documents

This tool is to be used on data from the Bulgarian National Corpus. It contains two script files,bunc2tei.py and escape_amp.py.

Usage

Converting the xml files and storing them into one corpus file

The script bunc2tei.py can be executed with the command bunc2tei.py *.xml inside a directory containing the data.

  1. A new xml file is created in the main function, with the element teiCorpus as its root.
  2. The corpus file parsed as such and saved into the file tree_structure in a new folder called input. This file will serve as the tree from which the new corpus will be built.
  3. To build the corpus, each xml file is first passed to the function convert. Convert tries parsing the xml file. If the parsing fails, the xml file is faulty and needs to be cleaned first. This is done by the script escape_amp.py and will be explained in the next section. If the parsing succeeds, the file is converted and appended to the corpus:
    1. To convert the file, the data (text metadata, text elements) is first stored in a list.
    2. The tree structure of the xml file is then deleted.
    3. The new tree structure is built and the data is inserted into the correct position.

The resulting corpus file is stored in the file corpus.p5.xml inside the folder output.

Additional preprocessing for unescaped symbols

The script escape_amp.py aims at fixing xml files that cannot be processed due to unescaped symbols. In the Bulgarian National Corpus, this was the case for ampersand symbols inside text elements. The script can be executed using escape_amp.py *.xml inside a directory containing the data.

  1. The script tries to parse the xml files. If the parsing fails, the file is passed to the function escape.
  2. Escape looks for all &-symbols that are not yet escaped and that are not themselves used to escape another symbol using a regular expression.
  3. The unescaped &-symbols are replaced by their escaped variant inside the file.