Search the embedding space with vector expressions
"König - Mann + Frau" now looks for the neighbours of
vec(König) - vec(Mann) + vec(Frau) instead of the neighbours of a single
word. A '+' or '-' is an operator only at the beginning of a token, so
hyphenated words like "Nord-Süd-Dialog" stay searchable, and blank
separated words keep entering the query with a '+', as before.
_get_neighbours() already had the branch for it, but nothing ever filled
wl->sep: the wordlist came from malloc() and the signs were read out of
uninitialised memory, so a multi word query subtracted operands at
random. It is filled now, and its indexing corrected - sep[b] is the sign
of operand b, not sep[b-1].
The predictive collocators answer for the whole query as well.
getCollocators() read the input weights of wl->wordi[0], the first
operand, and nothing said so: "Haus Auto" answered with the collocators
of Haus, "Auto Haus" with those of Auto. The score of a candidate is
sigmoid(q . syn1neg[collocate, position]), linear in q before the
sigmoid, so q can be the signed combination the paradigmatic side
searches around rather than the vector of one word.
The terms are averaged rather than summed. The sigmoid is informative
over a narrow range only - the strongest collocates of a single word
reach 0.994 to 0.997 on dereko-2026-ii, against the 0.9975 that MAX_EXP
allows - so a plain sum would push them into saturation. Dividing by the
number of terms keeps every query in the range MIN_RESP and the auto
focus are calibrated for, and leaves one word exactly as it was, divisor
one. A balanced analogy is one term as well, +1 -1 +1.
Those tails now saturate instead of being dropped. Activations outside
±MAX_EXP used to skip the collocate entirely, removing the strongest ones
from the list and from the position and target sums.
The count based collocators cannot follow: they are looked up per node in
the co-occurrence database. They used to be fetched for the first
paradigmatic neighbour, i.e. for the word nearest to the query vector,
which for an expression is not what was asked for. The interface says so
in place of that table now, and get_neighbours() reports the number of
operands so it can tell the two cases apart.
Operands outside the vocabulary are named above the result instead of
being dropped in silence. The vocabulary lookup only recognised a miss at
the very end of the vocabulary, so in a merged model a missing word
resolved to whatever sits at the boundary between the two vocabularies;
it returns -1 in both halves now.
Two more things the tokeniser uncovered. The best array was sized for
10 * max(N, 200) entries but sorted over N * para_threads, which fits
only as long as the syntagmatic threads take half of them - it is sized
for num_threads slices now. And its unfilled slots kept wordi == 0 from
the memset, which the result loop emitted as vocab[-1 * max_w] once a
thread found fewer candidates than were asked for; slots start at -1 and
the loop stops there.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: Ia91727eb669d73fc055adf652500464b8a743795
diff --git a/templates/index.html.ep b/templates/index.html.ep
index 9322028..ce38536 100644
--- a/templates/index.html.ep
+++ b/templates/index.html.ep
@@ -2,6 +2,13 @@
<html>
<head>
<% my $plain_title = $title; $plain_title=~s/<[^>]+>//g; %>
+%# Whether the query was answered at all: not found, and an expression whose
+%# operands cancel each other out, both leave the first list empty.
+% my $have_results = ($lists && @$lists > 0 && $lists->[0] && @{$lists->[0]} > 0);
+%# The count based collocators come out of the co-occurrence database, which is
+%# keyed by one node word, so they have no answer for a query of several
+%# operands. The predictive ones do, see getCollocators().
+% my $single_word = ($operands == 1);
<title><%= $plain_title %>:<%= $word %> · IDS word vector analysis</title>
%# Result pages are computed for one word each and there is one for every
%# word of the vocabulary, so they are not something to index or to follow.
@@ -44,8 +51,18 @@
});
var urlParams = new URLSearchParams(window.location.search);
var currentWords = urlParams.get("word");
- var CIIsearchWords = (currentWords && (currentWords.includes(" ") || currentWords.includes("|")) ? '('+currentWords.replace(/[ |]+/g, " oder ")+')' : currentWords);
+ % use Mojo::ByteStream 'b';
+ // The vocabulary entries the query vector was built from, as the server
+ // resolved them. A query can be a vector expression, "König - Mann +
+ // Frau", and the subtracted operands are deliberately not in here: they
+ // are not marked in the maps, and not part of the KorAP queries built
+ // from the result.
+ var searchedWords = <%= b(Mojo::JSON::to_json($added // '')) %>;
+ var targetWords = " " + searchedWords + " ";
+ var CIIsearchWords = (searchedWords.includes(" ") ? '('+searchedWords.replace(/\s+/g, " oder ")+')' : searchedWords);
var collocatorTable = null;
+ var classicCollocatorTable = null;
+ var singleWord = <%= $single_word ? 'true' : 'false' %>;
var plainTitle ="<%= $plain_title %>"
var korapPath="/";
if (plainTitle.match(/-en/)) {
@@ -105,8 +122,8 @@
var collocatorTable_activated = false;
$( "#tabs" ).on( "tabsactivate", function( event, ui ) {
if (localStorage) localStorage['tab'] = ui.newTab.index();
- if(ui.newTab.index() == 3 && !collocatorTable_activated) {
- classicCollocatorTable.columns.adjust();
+ if(ui.newTab.index() == 3 && !collocatorTable_activated && collocatorTable) {
+ if (classicCollocatorTable) classicCollocatorTable.columns.adjust();
collocatorTable.columns.adjust();
collocatorTable_activated = true;
}
@@ -143,10 +160,9 @@
return changeCharColor(str, heat, word);
}
- % use Mojo::ByteStream 'b';
var paraResults = <%= b(Mojo::JSON::to_json($lists)) %>;
var urlprefix = new URLSearchParams(window.location.search);
- if (paraResults.length > 0 && paraResults[0] != null) {
+ if (paraResults.length > 0 && paraResults[0] != null && paraResults[0].length > 0) {
var nvecs = [],
nwords = [],
nranks = [],
@@ -157,7 +173,7 @@
nranks = nranks.concat(paraResults[i].map(function(a){return a.rank;}));
nmarked = nmarked.concat(paraResults[i].map(function(a){return a.marked;}));
}
- showMap({target: " "+urlParams.get('word')+" ", mergedEnd: <%= $mergedEnd %>, words: nwords, vecs: nvecs, ranks: nranks, marked: nmarked} );
+ showMap({target: targetWords, mergedEnd: <%= $mergedEnd %>, words: nwords, vecs: nvecs, ranks: nranks, marked: nmarked} );
var t = $('#firsttable').DataTable({
data: [].concat.apply([], paraResults),
"sScrollY": "780px",
@@ -311,19 +327,24 @@
var filterQuot = /^quot/;
var ccResult;
var baseURL = window.location.pathname.replace(/[/]$/, '')
- classicCollocatorTable = makeClassicCollocatorTable('#classicoloctable', baseURL, paraResults[0][0].rank)
+ // Only for a one word query: the node of a count based profile is a
+ // word, and paraResults[0][0] is the nearest word to the query
+ // vector, which is not what was asked for.
+ if (singleWord) {
+ classicCollocatorTable = makeClassicCollocatorTable('#classicoloctable', baseURL, paraResults[0][0].rank)
- $('#show-details').change(function (e) {
- var columns = classicCollocatorTable.columns(".detail");
- if(this.checked) {
- columns.visible(true);
- $("#ccd").css('width', 'auto');
- } else {
- columns.visible(false);
- $("#ccd").css('width', '680px');
- }
- classicCollocatorTable.columns.adjust().draw();
- } );
+ $('#show-details').change(function (e) {
+ var columns = classicCollocatorTable.columns(".detail");
+ if(this.checked) {
+ columns.visible(true);
+ $("#ccd").css('width', 'auto');
+ } else {
+ columns.visible(false);
+ $("#ccd").css('width', '680px');
+ }
+ classicCollocatorTable.columns.adjust().draw();
+ } );
+ }
$("td.collocator").click(function(){
queryKorAPCII(this.textContent + " /w1:5,s0 " + CIIsearchWords);
@@ -694,7 +715,7 @@
$.post(baseURL+'/getVecsByRanks',
JSON.stringify(nranks),
function(data, status){
- showMap({target: " "+urlParams.get('word')+" ", mergedEnd: <%= $mergedEnd %>, words: nwords, vecs: data, ranks: nranks, marked: Array(100).fill(false)} );
+ showMap({target: targetWords, mergedEnd: <%= $mergedEnd %>, words: nwords, vecs: data, ranks: nranks, marked: Array(100).fill(false)} );
}, 'json');
}
}
@@ -771,6 +792,12 @@
</div>
</div>
</div>
+ %# Operands that are not in the vocabulary are dropped from the query
+ %# vector, which for an expression like "König - Mannn + Frau" would
+ %# otherwise silently answer a different question.
+ % if($unknown ne '' && $have_results) {
+ <div id="unknownwords"><%= loc 'not_in_vocabulary' %> <span class="mono"><%= $unknown %></span></div>
+ % }
<div id="topwrapper">
<div style="visibility: hidden;" id="tabs">
<ul>
@@ -780,7 +807,11 @@
% }
<li><a href="#tabs-1"><%= loc 'paradigmatic_tsne' %></a></li>
<li><a href="#tabs-2"><%= loc 'paradigmatic_som' %></a></li>
- <li><a href="#tabs-3"><%= loc 'syntagmatic' %></a></li>
+ %# No syntagmatic tab for a model without a .net file, i.e. without
+ %# the output weights the predictive collocators are read from.
+ % if($collocators) {
+ <li><a href="#tabs-3"><%= loc 'syntagmatic' %></a></li>
+ % }
% }
<li><a href="#tabs-4">Info</a></li>
</ul>
@@ -807,7 +838,7 @@
</div>
% }
<div id="tabs-1" style="display: flex; padding: 5px; flex-flow: row wrap;">
- % if($lists && (@$lists) > 0 && (@$lists)[0]) {
+ % if($have_results) {
<div id="wrapper">
<div id="first" style="width: 230px; margin-bottom: 15px;">
<table class="display compact nowrap" id="firsttable">
@@ -893,6 +924,7 @@
</div>
% }
</div>
+ % if($collocators) {
<div id="tabs-3" style="display: flex; padding:5px; flex-flow: row wrap;">
<div style="margin-right: 20px; margin-bottom: 10px;" id="secondt">
<table class="display compact nowrap" id="secondtable">
@@ -928,6 +960,9 @@
</tbody>
</table>
</div>
+ % if(!$single_word) {
+ <div id="ccd" class="notice"><%= loc 'ca_single_word_only' %></div>
+ % } else {
<div id="ccd" style="">
<table class="display compact nowrap" id="classicoloctable">
<thead>
@@ -973,11 +1008,13 @@
</tbody>
</table>
</div>
+ % }
<!--
<div style="clear:both" ></div>
<div style="float: right; overflow: hidden" id="extra"><button onClick="showCollocatorSOM()"> </button></div>
-->
</div>
+ % }
<div id="tabs-4" style="display: flex; padding:5px; flex-flow: row wrap;">
<div id="info">
<h3><%== loc 'about' %></h3>