blob: 19599147bc9bc93b2c0ccf12fc1fb6add3fd5ab2 [file] [log] [blame]
Marc Kupietz23446562025-10-28 14:36:50 +01001 - Added support for spaCy annotations on XPOS (spacy/p),
2 UPOS (spacy/u), morphological features (spacy/m) and
3 dependency relations (spacy/d).
4
Akronec01ff42025-10-17 11:59:33 +020050.63 2025-10-17
6 - Publisher is now indexed as string, when
7 K2K_PUBLISHER_STRING is set as an environment
8 variable.
9
Akroncb12af72025-07-15 14:36:10 +0200100.62 2025-07-15
11 - Remove lock from tar builder.
Akron8b03ba52025-07-15 09:16:18 +020012 - Don't create a list of files (that is passed to forks).
Akrond5d21252025-07-15 09:36:04 +020013 - Add slimmed MANIFEST to repo.
Akroncb12af72025-07-15 14:36:10 +020014
Akron88566d22025-04-30 13:01:22 +0200150.61 2025-04-30
16 - Support certainty in OpenNLP/Morpho.
17
Marc Kupietzaeac7532025-04-14 20:00:33 +0200180.60 2025-04-22
19 - Partially support ripunzip if available.
20
Akron0a0d1f92024-11-14 14:31:42 +0100210.59 2024-11-14
22 - UsePath::Iterator::Rule instead of Mojo::File.
23
Akronce033502024-09-11 10:51:49 +0200240.58 2024-09-11
25 - Remove Directory::Iterator and replace it with Mojo::File.
Akron311e29b2024-09-11 11:46:09 +020026 - Add performance hint.
Akronce033502024-09-11 10:51:49 +020027
Akronaf18edb2024-07-26 12:11:48 +0200280.57 2024-07-26
Akron9d01c1e2024-07-25 18:29:10 +020029 - Support award notes in i5.
Akron1f951222024-07-25 17:45:21 +020030 - Add support for idno (with @rend) in i5.
31 - Add support for ISBN in i5.
Akronaf18edb2024-07-26 12:11:48 +020032 - Translator is now indexed as Text in i5, when
33 K2K_TRANSLATOR_TEXT is set as an environment
34 variable.
Akron9d01c1e2024-07-25 18:29:10 +020035
Akron5530a552022-02-17 17:53:15 +0100360.56 2024-06-05
37 - Add support für corpusexplorer.
38
Akron24ad3c02024-06-03 12:38:20 +0200390.55 2024-06-04
40 - Add support for xenodata to i5.
41
Akron2cfe9bd2024-05-02 16:28:52 +0200420.54 2024-05-02
Akronc0ac4ff2024-04-15 18:03:15 +020043 - Fix 'cache' parameter. (reported by kupietz)
44 - Fix cache deletion for certain scenarios.
Akronebbac2e2024-03-22 10:31:23 +010045 - Improve information on the number of jobs
46 running in parallel.
Akron2cfe9bd2024-05-02 16:28:52 +020047 - Add support for KoKoKom <u> attributes.
Akronc0ac4ff2024-04-15 18:03:15 +020048
Akron2cfe9bd2024-05-02 16:28:52 +0200490.53 2024-03-20
Marc Kupietzb8c53822024-03-16 18:54:08 +010050 - Added Spacy support. (kupietz)
Marc Kupietz7fe9cd92024-03-18 11:50:22 +010051 - Support 'pos' as an alternative to 'ctag'
52 in Treetagger. (kupietz)
53 - Change default certainty value in TreeTagger
54 to 1.
Marc Kupietzb8c53822024-03-16 18:54:08 +010055
Akron2cfe9bd2024-05-02 16:28:52 +0200560.52 2024-01-23
Akrona3518372024-01-22 23:29:00 +010057 - Introduced 'quiet' flag.
58
Akron2daf8fe2023-02-27 12:55:04 +0100590.51 2023-12-23
Akron2532f1b2023-05-15 13:41:24 +020060 - Support ICC meta.
Akron01c6fb52023-08-25 12:22:33 +020061 - Fix date handling for years of length < 2.
Akron2daf8fe2023-02-27 12:55:04 +010062 - Improve emoji detection (rebecca).
Akron18ce3b32023-12-13 15:44:11 +010063 - Upgrade minimum perl version required.
Akron2532f1b2023-05-15 13:41:24 +020064
Akrona472a242023-02-13 13:46:30 +0100650.50 2023-02-13
66 - Fix 'temporary-extract' configuration
67 information.
68
Akron4a7ab012023-02-12 12:59:38 +0100690.49 2023-02-12
Marc Kupietz400590b2022-12-23 16:02:36 +010070 - Support for UDPipe POS, lemma and dependency
71 annotations (kupietz).
Akron4a7ab012023-02-12 12:59:38 +010072 - Remove last bit of Sys::Info dependency.
73 (fixes #9)
Marc Kupietz400590b2022-12-23 16:02:36 +010074
Akron2dd0e5d2022-11-15 09:44:43 +0100750.48 2022-11-15
Akronaa166fa2022-11-10 14:15:14 +010076 - Improve support for text siglen including
77 underscore in corpus parts.
Akron2dd0e5d2022-11-15 09:44:43 +010078 - Split morphological features in NKJP.
Akronaa166fa2022-11-10 14:15:14 +010079
Akronddf33192022-08-08 16:44:39 +0200800.47 2022-08-08
Akron64f7fae2022-07-27 12:45:33 +020081 - Support for preferred language transformation.
Akron1a2535d2022-07-28 16:31:43 +020082 - Support for NKJP taxonomies.
Akronddf33192022-08-08 16:44:39 +020083 - Support for NKJP 'orig' values.
Akron64f7fae2022-07-27 12:45:33 +020084
Akrona65cd682022-07-21 15:40:40 +0200850.46 2022-07-21
86 - Support NKJP Meta, Morpho and NamedEntities.
87
Akron3c9b27c2022-03-04 13:08:13 +0100880.45 2022-03-04
Akroneb370a02022-02-24 13:33:40 +010089 - Due to problems installing Archive::Tar::Builder
90 in certain environments, this is now optional,
91 with a pure perl fallback archiver.
Akron3c9b27c2022-03-04 13:08:13 +010092 - Support externalLink and internalLink universally in
93 i5 meta data.
Akroneb370a02022-02-24 13:33:40 +010094
Akronf6833102022-02-17 18:35:03 +0100950.44 2022-02-17
Akron8c85e9f2022-01-03 16:27:10 +010096 - Improve Gingko Metadata support.
Akrone1cde962022-02-07 20:00:29 +010097 - Fix data-URIs by always refering to UTF-8.
Akronf6833102022-02-17 18:35:03 +010098 - Warn on wrong token order.
Akronc4ad7472022-01-28 19:12:50 +010099 - Improve Gingko Metadata support.
100 - Updated all dependencies.
Akron8c85e9f2022-01-03 16:27:10 +0100101
Akron9a2545e2022-01-16 15:15:50 +01001020.43 2022-01-17
103 - Fix temporary extract handling when defined
104 in a config file.
Akron303c4fd2022-01-16 15:14:46 +0100105 - Improve handling of invalid certainty values
106 in TreeTagger.
Akron84b53ad2022-01-14 12:39:15 +0100107 - Add log slimming function.
Akron9a2545e2022-01-16 15:15:50 +0100108
Akron8c85e9f2022-01-03 16:27:10 +01001090.42 2021-10-11
Akronb9c33812020-10-21 16:19:35 +0200110 - Replaced Log4perl with Log::Any.
Akron0ffbd522021-02-16 12:01:19 +0100111 - Ignore level < 0 structures in DeReKo, but support
112 them for base annotations.
Akron6882d7d2021-02-08 09:43:57 +0100113 - Define resources in Makefile.
Akron56d5f172021-03-16 18:37:39 +0100114 - Add GitHub action for CI.
Akronfca010b2021-10-11 15:52:48 +0200115 - Remove MANIFEST file from repo.
Akronabb36902021-10-11 15:51:06 +0200116 - Introduce Gingko support.
Akron8ad06c42022-01-11 17:07:49 +0100117 - Fix data URIs to always encode percentage-wise.
Akronb9c33812020-10-21 16:19:35 +0200118
1190.41 2020-08-10
Akron07e24772020-04-23 14:00:54 +0200120 - Added support for RWK annotations.
Akron1cdbc9d2020-05-07 15:28:54 +0200121 - Improved DGD support.
Akrone3e05362020-06-16 17:19:09 +0200122 - Fixed bug in RWK support that broke on
123 some KorAP-XML files.
Akron414ec952020-08-03 15:48:43 +0200124 - Separate "real data" test suite from artificial
125 tests to prepare for CPAN release.
Akron39df7ce2020-08-04 15:55:26 +0200126 - Optimizations and cleanup based on profiling.
Akron129e4412020-08-05 15:30:12 +0200127 - Remove MultiTerm->add() in favor of
128 MultiTerm->add_by_term().
Akron47426f02020-08-06 13:28:53 +0200129 - Optimization by reducing calls to _offset().
Akron6a4cb162020-08-06 16:00:33 +0200130 - Introduced add_span() method to MultiTermToken.
Akron11daf962020-08-07 16:29:22 +0200131 - Removed deprecated 'primary' flag.
Akron6aed0562020-08-07 16:46:00 +0200132 - Removed deprecated 'pretty' flag.
Akron56deacb2020-08-10 10:03:55 +0200133 - Fix RWK paragraph handling.
Akrond2cd8e42020-10-30 16:37:19 +0100134 - Updated 'Clone' dependency in Makefile.
Akron0b04b312020-10-30 17:39:18 +0100135 - Make Sys::Info optional.
Akrondcbee642020-10-30 18:01:43 +0100136 - Fixes a bug in XIP::Dependency and added
137 dependency checks for all annotation libraries.
Akron07e24772020-04-23 14:00:54 +0200138
Akrondec43122020-03-03 11:22:25 +01001390.40 2020-03-03
Akrona0d5af32020-03-01 12:46:30 +0100140 - Fixed XIP parser.
Akronb62d92a2020-03-01 16:32:00 +0100141 - Added example corpus of the
142 Redewiedergabe-Korpus.
143 - Fixed span offset bug.
144 - Fixed milestones behind the last
145 token bug.
Akrondec43122020-03-03 11:22:25 +0100146 - Fixed gap behind last token bug.
147 - Fixed <base/s:t> length.
Akrona0d5af32020-03-01 12:46:30 +0100148
Akron6e886f72020-02-19 07:42:32 +01001490.39 2020-02-19
Akron7d5e6382019-08-08 16:36:27 +0200150 - Added Talismane support.
Akron0d68a4b2019-11-13 15:42:11 +0100151 - Added "distributor" field to I5 metadata.
Akron20294552019-11-29 16:15:35 +0100152 - Added DGD link field to I5 metadata.
Akronb05b8422019-12-11 13:47:57 +0100153 - Improve logging.
Akronc29b8e12019-12-16 14:28:09 +0100154 - Added support for DGD pseudo-sentences
155 based on anchor milestones.
Akron8f69d632020-01-15 16:58:11 +0100156 - Added brief explanation of the format.
Akrond4c5c102020-02-11 11:47:59 +0100157 - Fixed parsing of editionStmt.
158 - Added documentation for supported I5 metadata
159 fields.
Akron6e886f72020-02-19 07:42:32 +0100160 - Added integrated benchmark mechanism.
Akron7d5e6382019-08-08 16:36:27 +0200161
Akron57510c12019-01-04 14:58:53 +01001620.38 2019-05-22
Akron9b04f602019-03-08 18:45:35 +0100163 - Stop file processing when base tokenization
164 is wrong.
Akron57510c12019-01-04 14:58:53 +0100165 - Added DGD support.
Akron9b04f602019-03-08 18:45:35 +0100166
Akroneaffe932019-03-07 17:14:42 +01001670.37 2019-03-06
Akron263274c2019-02-07 09:48:30 +0100168 - Support for 'koral:field' array.
169 - Support for Koral versioning.
Akron4e1712c2019-02-04 22:29:37 +0100170 - Added tests for english sources.
Akron6bf3cc92019-02-07 12:11:20 +0100171 - Added support for external links for
172 Wikipedia resources.
Akron63d03ee2019-02-13 18:49:38 +0100173 - Ignore temporary extraction
174 on directory archiving.
Akron955b75b2019-02-21 14:28:41 +0100175 - Remove extract_text and extract_doc in
176 favor of extract_sigle for archives.
Akron263274c2019-02-07 09:48:30 +0100177
Akroned9baf02019-01-22 17:03:25 +01001780.36 2019-01-22
179 - Support for non-word tokens (fixes #5).
180
Akron6eff23b2018-09-24 10:31:20 +02001810.35 2018-09-24
182 - Lift minimum version of Perl to 5.16 as for
183 "fc"-feature.
184
Akrondd1c0f12018-07-19 06:45:28 +02001850.34 2018-07-19
186 - Preliminary support for HNC.
187
Akron28dc17f2018-02-01 15:31:41 +01001880.33 2018-02-01
Akron4c679192018-01-16 17:41:49 +0100189 - Added LWC support.
Akron28dc17f2018-02-01 15:31:41 +0100190 - Fixed TreeTagger certainties.
Akron4c679192018-01-16 17:41:49 +0100191
Akron3c56f502017-10-24 15:37:27 +02001920.32 2017-10-24
Akron9a062ce2017-07-04 19:12:05 +0200193 - Fixed tar building process in script.
Akron3c56f502017-10-24 15:37:27 +0200194 - Support file extensions in base tokenization parameter.
Akron9a062ce2017-07-04 19:12:05 +0200195
Akron0a6cce12017-06-30 23:03:21 +02001960.31 2017-06-30
Akron3abc03e2017-06-29 16:23:35 +0200197 - Fixed exit codes in script.
Akron0a6cce12017-06-30 23:03:21 +0200198 - Use CORE::fc for case folding.
Akron3abc03e2017-06-29 16:23:35 +0200199
Akrond5bb4342017-06-19 11:50:49 +02002000.30 2017-06-19
201 - Fixed permission handling in test suite.
Akronce125b62017-06-19 11:54:36 +0200202 - Added preliminary CMC support.
Akrond5bb4342017-06-19 11:50:49 +0200203
Akronda3097e2017-04-23 19:53:57 +02002040.29 2017-04-23
205 - support --to-tar flag.
206
Akron9ec88872017-04-12 16:29:06 +02002070.28 2017-04-12
Akron86db52e2017-04-11 20:36:43 +0200208 - Improved overwriting behaviour for unzip.
Akron9ec88872017-04-12 16:29:06 +0200209 - Introduced --sequential-extraction flag.
Akron86db52e2017-04-11 20:36:43 +0200210
Akron63f20d42017-04-10 23:40:29 +02002110.27 2017-04-10
Akron636aa112017-04-07 18:48:56 +0200212 - Support configuration files.
Akron81500102017-04-07 20:45:44 +0200213 - Support temporary extraction.
Akron63f20d42017-04-10 23:40:29 +0200214 - Support serial conversion.
215 - Support input-base.
Akron636aa112017-04-07 18:48:56 +0200216
2170.26 2017-04-06
218 - Support wildcards on input.
219
Akron5809fea2017-03-14 20:02:26 +01002200.25 2017-03-14
Akron7e2eb882017-01-18 17:28:07 +0100221 - Updated to Mojolicious 7.20
222 - Fixed meta treatment in case analytic and monogr
223 are available
Akron4fa37c32017-01-20 14:43:10 +0100224 - Added DRuKoLa support to script
Akron38873012017-02-06 20:27:37 +0100225 - Liberated document and text sigle handling to be
226 compliant with CoRoLa.
Akron41ac10b2017-02-08 22:47:25 +0100227 - Added support for pagebreak annotations.
Akron08d54452017-02-16 23:19:49 +0100228 - Renamed "pages" to "srcPages".
Akron60a8caa2017-02-17 21:51:27 +0100229 - Fixed handling of prefixes for text sigles.
Akron3bd942f2017-02-20 20:09:14 +0100230 - Support for MarMoT.
Akron5809fea2017-03-14 20:02:26 +0100231 - Fix case insensitivity.
Akron55778f02017-03-14 20:47:26 +0100232 - Added preliminary support for diacritic insensitivity.
Akron3ec0a1c2017-01-18 14:41:55 +0100233
Akron3741f8b2016-12-21 19:55:21 +01002340.24 2016-12-21
235 - Added --base-sentences and --base-paragraphs options
236
Akron6f9fef52016-11-03 17:06:40 +01002370.23 2016-11-03
Akron2fd402b2016-10-27 21:26:48 +0200238 - Added wildcard support for document extraction
Akron2812ba22016-10-28 21:55:59 +0200239 - Fixed archive iteration to not duplicate the first archive
240 - Added parallel extraction for document sigles
Akron13d56622016-10-31 14:54:49 +0100241 - Improved return value for existing files
Akron3741f8b2016-12-21 19:55:21 +0100242 - Don't warn on recursion in CoreNLP/Constituency
Akron2fd402b2016-10-27 21:26:48 +0200243
Akron20807582016-10-26 17:11:34 +02002440.22 2016-10-26
245 - Added support for document extraction
Akronb4bbec72016-10-26 20:21:02 +0200246 - Fixed archive naming
Akron20807582016-10-26 17:11:34 +0200247
Akronb4bbec72016-10-26 20:21:02 +02002480.21 2016-10-24
Nils Diewaldb3e9ccd2016-10-24 15:16:52 +0200249 - Improved Windows support
250
Akron4c0cf312016-10-15 16:42:09 +02002510.20 2016-10-15
252 - Fixed treatment of temporary folders in script
253
Akronbdb64652016-08-17 23:30:01 +02002540.19 2016-08-17
Akron92ad95b2016-08-15 23:38:56 +0200255 - Added test for direct I5 support.
256 - Fixed support for Mojolicious 7.
257 - Added script test.
Akron5f51d422016-08-16 16:26:43 +0200258 - Fixed setting multiple annotations in
259 script.
Akrone2b902d2016-08-16 16:50:11 +0200260 - Fixed output of version and help messages.
Akron7d4cdd82016-08-17 21:39:45 +0200261 - Added script test for extraction.
Akron651cb8d2016-08-16 21:44:49 +0200262 - Fixed extraction with multiple archives and prefix
263 negation support.
Akron7d4cdd82016-08-17 21:39:45 +0200264 - Added script test for archives.
Akron1924bbe2016-06-22 16:05:41 +0200265
Akronbdb64652016-08-17 23:30:01 +02002660.18 2016-07-08
267 - Added REI test.
268 - Added multiple archive support to korapxml2krill.
269 - Added support for prefix negation in korapxml2krill.
270 - Added support for Malt#Dependency.
271 - Improved test suite for caching and REI.
272 - Added support for MDParser annotation.
273 - Added batch processing class for documents.
274
Akron1cd5b872016-03-22 00:23:46 +01002750.17 2016-03-22
276 - Rewrite siglen to use slashes as separators.
Akron5f51d422016-08-16 16:26:43 +0200277 - Zip listing optimized. Does no longer work with primary data
278 in text.xml files.
Akron1cd5b872016-03-22 00:23:46 +0100279
Akron11c80302016-03-18 19:44:43 +01002800.16 2016-03-18
281 - Added caching mechanism for
Akron5f51d422016-08-16 16:26:43 +0200282 metadata.
Akron11c80302016-03-18 19:44:43 +0100283
Akron35db6e32016-03-17 22:42:22 +01002840.15 2016-03-17
285 - Modularized metadata handling.
286 - Simplified metadata handling.
Akron5f51d422016-08-16 16:26:43 +0200287 - Added --meta option to script.
288 - Removed deprecated --human option from script.
Akron35db6e32016-03-17 22:42:22 +0100289
Akronc13a1702016-03-15 19:33:14 +01002900.14 2016-03-15
Akron151676d2016-03-14 20:12:14 +0100291 - Renamed ::Index to ::Annotate and ::Field to ::Index.
Akron5f51d422016-08-16 16:26:43 +0200292 - Renamed 'allow' to 'anno' as parameters of the script.
293 - Added readme.
Akron151676d2016-03-14 20:12:14 +0100294
Akron5b254312016-03-10 00:29:56 +01002950.13 2016-03-10
Akron44feb4e2016-03-02 12:45:47 +0100296 - Removed korapxml2krill_dir.
Akron5f51d422016-08-16 16:26:43 +0200297 - Renamed dependency nodes.
298 - Made dependency relations more effective (trimmed down TUIs)
299 ! This is currently very slow !
Akron44feb4e2016-03-02 12:45:47 +0100300
Akrondc898d82016-02-28 23:49:19 +01003010.12 2016-02-28
Akrone10ad322016-02-27 10:54:26 +0100302 - Added extract method to korapxml2krill.
Akron5f51d422016-08-16 16:26:43 +0200303 - Fixed Mate/Dependency.
304 - Fixed skip flag in korapxml2krill.
305 - Ignore spans outside the token range
306 (i.e. character offsets end before tokens have started).
Akrone10ad322016-02-27 10:54:26 +0100307
Akron941c1a62016-02-23 17:41:41 +01003080.11 2016-02-23
Akron44feb4e2016-03-02 12:45:47 +0100309 - Merged korapxml2krill and korapxml2krill_dir.
Akron941c1a62016-02-23 17:41:41 +0100310
Akron96165ad2016-02-15 18:09:41 +01003110.10 2016-02-15
312 - Added EXPERIMENTAL support for parallel jobs.
313
Akronc1babed2016-02-15 11:48:18 +01003140.09 2016-02-15
315 - Fixed temporary directory handling in scripts.
Akron5f51d422016-08-16 16:26:43 +0200316 - Improved skipping for archive handling in scripts.
Akronc1babed2016-02-15 11:48:18 +0100317
Akron150b29e2016-02-14 23:06:48 +01003180.08 2016-02-14
319 - Added support for archive streaming.
Akron5f51d422016-08-16 16:26:43 +0200320 - Improved scripts.
Akron150b29e2016-02-14 23:06:48 +0100321
Akron8c84aa52016-02-13 21:26:54 +01003220.07 2016-02-13
323 - Improved support for Schreibgebrauch meta data
Akron5f51d422016-08-16 16:26:43 +0200324 (IDS flavour).
Akron8c84aa52016-02-13 21:26:54 +0100325
3260.06 2016-02-11
Akron49a47652016-02-12 18:17:19 +0100327 - Improved support for Schreibgebrauch meta data
Akron5f51d422016-08-16 16:26:43 +0200328 (Duden flavour).
Akron49a47652016-02-12 18:17:19 +0100329
Akron93d620e2016-02-05 19:40:05 +01003300.05 2016-02-04
Akrone4c2e412016-01-28 15:10:50 +0100331 - Changed KorAP::Document to KorAP::XML::Krill.
Akron5f51d422016-08-16 16:26:43 +0200332 - Renamed "Schreibgebrauch" to "Sgbr".
333 - Preparation for GitHub release.
Akrone4c2e412016-01-28 15:10:50 +0100334
Akron9c0488f2016-01-28 14:17:15 +01003350.04 2016-01-28
Akron69a4a2f2016-01-17 12:55:50 +0100336 - Added PTI to all payloads.
Akron5f51d422016-08-16 16:26:43 +0200337 - Added support for empty elements.
338 - Added support for element attributes in struct.
339 - Added meta data support for Schreibgebrauch.
340 - Fixed test suite for meta data.
Akron69a4a2f2016-01-17 12:55:50 +0100341
3420.03 2014-11-03
Nils Diewald78674672014-11-03 21:43:12 +0000343 - Added new metadata scheme.
Akron5f51d422016-08-16 16:26:43 +0200344 - Fixed a minor bug in the constituency tree building.
345 - Sorted terms in tokens a priori.
Nils Diewald78674672014-11-03 21:43:12 +0000346
Akron69a4a2f2016-01-17 12:55:50 +01003470.02 2014-07-21
Nils Diewaldf03c6802014-07-21 16:39:44 +0000348 - Sentence annotations for all providing foundries
Akron5f51d422016-08-16 16:26:43 +0200349 - Starting subtokenization
Nils Diewaldf03c6802014-07-21 16:39:44 +0000350
Akron69a4a2f2016-01-17 12:55:50 +01003510.01 2014-04-15
Akron5f51d422016-08-16 16:26:43 +0200352 - [bugfix] for first token annotations
Nils Diewald7b847222014-04-23 11:14:00 +0000353 - Sentences are now available from all foundries that have it
354 - <>:p is now <>:base/para
Akron5f51d422016-08-16 16:26:43 +0200355 - Added <>:base/text