blessCacheAs() vouches for a cache file that an older version wrote, so that it is used again as it is. Files from the 1.3.0.9000 development cycle already hold correctly computed scores, the corrections having landed there, but say nothing about the version that wrote them and would otherwise be recomputed. What actually produced a file is left as it stands and the blessing recorded beside it, so cacheAsInfo() keeps saying where the numbers come from. A file recording no parameters has nothing left to compare once blessed, and is then reused for any call naming it
cacheAsInfo() reads back what a cacheAs file was produced by: the parameters of the call, the KorAP instance, the index revision its corpus had at the time, and the version of RKorAPClient that wrote it. For a result kept next to a document, that is what says which numbers it rests on
cacheAs is now offered by frequencyQuery(), corpusStats(), collocationScoreQuery() and textMetadata() as well, not only by collocationAnalysis(). It is a different thing from the connection's cache, which is a transparent speed-up: a cacheAs file belongs to the caller and is what keeps an analysis reproducible, since KorAP corpora grow and the same query returns different numbers next year. That is worth having for the quick functions too, where caching for speed would be pointless
cacheAs files record the version that wrote them and are refused, with a warning, when that is older than 1.4.0. Their contents are finished results including the association scores, which this version computes differently, so an old file would silently hand back numbers that would not be arrived at again - something no comparison of parameters can notice. The file is then recomputed and overwritten; pass a different name to keep it. How loud a query is no longer counts as a parameter either: verbose does not change what is returned
the names of a named vc vector are now used as labels by frequencyQuery(), corpusStats() and collocationScoreQuery(), as collocationAnalysis() already did. frequencyQuery() ignored them, corpusStats() put them into row names, which the first bind_rows() drops, and collocationScoreQuery() derived a label from the corpus definitions instead, so that c(before = ..., since = ...) came out as "1990 & pubDat…". Where a vector carries no names, nothing changes: no label column appears that was not there before
collocationAnalysis() now discards collocates that occur less often than expected by chance. logDice, by which it ranks and thresholds, expresses how salient a pair is rather than how surprising, so a frequent word could appear among the top collocates although the node does not attract it at all: for Grund in a 5+5 window, Berlin reaches a logDice of 3.84, close to triftiger at 3.96, while co-occurring 1.74 bits less often than expected. The new minObservedExpectedRatio parameter defaults to 1 and keeps such pairs out. Raise it to demand a stronger contrast, e.g. 2 for collocates occurring at least twice as often as expected, or set it to 0 for the unfiltered result of earlier versions, e.g. to study repulsion. collocationScoreQuery() is unaffected, as there the pairs to score are given explicitly
changed logDice values: logDice() is now computed as defined by Rychlý (2008), 14 + log2(2 * O / (O1 + O2)), so that its values are comparable to those of Sketch Engine and other tools. It previously multiplied the node frequency by the window size, 14 + log2(2 * O / (w * O1 + O2)), which added a count of window positions to a count of word tokens and made the coefficient asymmetric, so that swapping node and collocate changed the score. Because w * O1 dominated the denominator for a frequent node, rare collocates were penalised: for triftiger as a collocate of Grund in a 5+5 window, logDice was 0.64, below Berlin at 2.03, although Berlin co-occurs with Grund less often than chance predicts. The values are now 3.96 and 3.84. Scores rise by up to log2(w), that is by up to 3.32 for the default context of 5 left and 5 right, so collocationAnalysis() with the default thresholdScore = "logDice" and threshold = 2 is now somewhat more permissive when recursing. Results computed with a total window size of 1, as in the light verb construction example of the Readme, are unaffected. The other association scores are unchanged: they take the window size into account through the expected frequency E, which is correct
changed ll() values: the contingency table of ll() scaled only its row total by the window size, an inconsistency spotted by Tim Feldmüller, leaving cells that do not add up to one sample. Following Evert (2004), the sample consists of co-occurrence tokens, so the sample size and both marginals scale with the window: an occurrence of either word takes part in window_size pairs. The expected co-occurrence frequency is unchanged at window_size * O1 * O2 / N, which is why pmi, mi2 and mi3 are unaffected, but log-likelihood values differ, by 0.1% to well over 100% depending on the frequencies and the window. This also removes the case where N - window_size * O1 turned negative and the score became NaN, since window_size * (N - O1) cannot: the warning added earlier in this development version is therefore gone again, having treated a symptom of this
collocationAnalysis() now stores the analysis parameters in its cacheAs file and compares them on the next call. If they differ, the cached result is not the one that was asked for, so it is recomputed and the file overwritten, with a warning naming the parameters that differ. This catches the case of a parameter being changed while an old cache file is still lying around. Cache files written by 1.3.0 do not contain the parameters yet and keep being used as they are
dropped the PTXQC dependency, which was imported for two small string functions (lcpCount() and lcsCount(), used by queryStringToLabel()) but pulled in rmzqc, jsonvalidate and V8, and with them the only dependency requiring a libv8 installation. The two functions are now implemented in the package itself, 10 to 65 times faster than the originals, and with unchanged results
cacheAs parameter to collocationAnalysis() for transparent result caching: if the specified RDS file exists, the cached result is returned immediately; otherwise the analysis runs and the result is saved to the file (.rds extension is added automatically if omitted)max_delta_<score>, winner<score>, loser_score<score> columns etc.), including explicit winner/loser webUIRequestUrl columns for association scores, ranks, and percentile ranks. Missing per-label concordance URLs are now derived by replacing the cq parameter of an available row URL with the target label's vc, and unsuffixed consensus winner_webUIRequestUrl / loser_webUIRequestUrl columns are populated when score-based URL choices agree. The names and semantics of these columns may still change without a deprecation cycleimputed, n_imputed and imputed_<label> columns, and reported in verbose mode, so that presence/absence artifacts can be told apart from measured contrasts (filter(!imputed)). See the new "Interpreting multi-VC comparisons" section in ?collocationAnalysis for how to read the comparison columnscollocationAnalysis() now warns when repeated node/collocate/label rows are reduced to their first occurrence for the comparison columns, instead of dropping them silentlycollocationScoreQuery() now accepts a vector of collocates and queries every combination of collocate and virtual corpusKorAPConnection() is now a regular constructor function instead of the S4 class generator, so that its manual page documents how it is actually called. new("KorAPConnection", ...) and all existing calls keep working unchangedfetchAnnotations() morphology so MarMoT and other foundries keep all features from nested spans (#30)xml2, retaining multiple lemma/POS values and improving robustness of token alignmentfetchAnnotations() method added to KorAPQuery class, to fetch annotations for all collected matchesKorAPConnection(verbose=) default via env var KORAP_VERBOSE or R option rkorap.verbose (explicit argument still wins)ci functionauth() function (see ?auth)httr to httr2for all API requestsggplot2 3.6.0 (thanks @teunbrand)corpusQuery resultsmergeDuplicateCollocates functioncitation("RKorAPClient")matchStart and matchEnd columns to corpusQuery results, that contain the start and end position of the match in the textmergeDuplicateCollocates to merge collocation analysis results for different context positionsquery to collocation analysis results\link{} targets missing packagecorpusStats methodspan parameter in collocation analysis functionstextMetadata method, if available, to retrieve also custom metadata for a text based on its siglecorpusQuery method, if availabletextMetadata KorAPConnection method to retrieve all metadata for a text based on its siglewebUiRequestUrl column also to corpusStats results, so that also virtual corpus definitions can be linked to / tested directly in the KorAP UImetadataOnly is set, also tokenized snippets are now retrieved in corpus queries (stored in collectedMatches$tokens)%>% (pipe function)hc_add_onclick_korap_search() now also works with bar, column and pie highchartscollocationAnalysis function:maxRecurse - apply collocation analysis recursively maxRecurse timesaddExamples - If TRUE, examples for instances of collocations will be added in a column example. This makes a difference in particular if 'node' is given as a lemma query.thresholdScore - association score function to use for computing the threshold that is applied for recursive collocation analysis callsthreshold- minimum value of thresholdScore function call to apply collocation analysis recursively## Not run ... ## End(Not run)This was necessary to meet the time limits of automatic CRAN checks under all circumstances and has otherwise no significance,
The change of the minor version from 6 to 7 automatically invalidates your cached query results. However, since the API has not changed, you can continue using the old cached results by copying or renaming your cache folder.
On linux, for example, this can look like this:
mv ~/.cache/R/R.cache/RKorAPClient_0.6 ~/.cache/R/R.cache/RKorAPClient_0.7
or like this:
mv ~/.Rcache/RKorAPClient_0.6 ~/.Rcache/RKorAPClient_0.7
To find the RKorAPClient cache directory for your environment, you can use:
R.cache::getCacheRootPath()
collocationAnalysis method (client-sided)randomizePageOrder to fetch result pages in randomized orderignoreCollocateCase in collocationScoreQuerywithinSpan (default: base/s=s) in collocationScoreQueryverbose=TRUEsmooth added to hc_freq_by_year_ciUnknown or uninitialised column: 'condition'.as consequences.
hc_freq_by_year_ci.demo("writtenVsSpoken")