Document selection produces a set of documents you can use at later stages of your analyses, for example for clustering or 2d mapping.
Most of your analysis requests will operate on some subset of documents stored in the Lingo4G index. The
category of analysis stages
groups various ways of specifying which documents to select for processing.
This article is an overview of the available document selector stages and their typical use cases. For in-depth descriptions of specific selectors and their properties, see the document selector API reference.
This article assumes you are familiar with the structure and concepts behind Lingo4G analysis request JSONs.
Common document selectors
The following stages should cover the document selection needs of most typical analysis requests.
stage selects documents that match the
query you provide. Coupled with the
component, which parses
Lucene-like query syntax,
documents:byQuery is the most likely source of documents in your analyses.
Let's use the
stage to select the top 100 arXiv abstracts containing the
dark energy phrase and created in 2016 or later.
If you execute the above request in the
JSON Sandbox app, you should see a result similar to the
following JSON. Note that we present only the top 5 results for brevity. In the real response, the number of
elements in the
documents array is not greater than the
value you set in the request.
Document selection JSON output
documents array, which holds the internal identifier and weight (importance) of each
selected document. The semantics of the document's weight property depends on the specific document selection
stage. In case of the
documents:byQuery stage, each document's weight is the search score returned
by Apache Lucene, which Lingo4G uses to perform query-based searches.
Some document selection stages may add extra information on top of the list of selected documents. In our case,
section, which shows the total number of documents matching the query. The number of matches may be larger than
the document selection limit you provide in the request. See the
reference documentation for a detailed description of its output JSON.
stage selects the documents that are most semantically-similar to the multidimensional
embedding vector you provide. In contrast to
documents:byQuery, which requires certain words to be present in the selected documents, the embedding-based document selection
performs a more "fuzzy", semantics-based matching. You may use the embedding-based document selection to
discover documents that are hard to find using keyword-based methods.
To use embedding-based selectors, your index must contain label and document embedding vectors. See the learning embeddings article for detailed instructions.
Let's select documents that are semantically similar to one specific seed document. We'll break the request down into three stages:
Selecting the seed document using the
Retrieving the embedding vector of the seed document using the
Retrieving the semantically similar documents using the
The request contains three named stages corresponding to the above list:
similarDocument. Note how the request uses
stage references to pass results from one stage to another. We
$.output.stages property to output only the results of document stages, the output of the
vector stage is not relevant.
If you run the above request in the JSON sandbox app, you should see a response similar to the following JSON.
$.result.seedDocuments section contains the identifier of the seed document, while the
$.result.similarDocuments section contains the documents whose embedding vectors lie close to the
seed document's vector. The
stage computes document weights as the dot product between the search vector and the result document's vector,
normalized to the 0...1 range. In our case, the first document on the list is the same as the seed document,
hence the weight of 1.0. Also note, that the embedding-based documents selector does not output the
Looking at the response to the original request, it is not possible to tell if the selected documents are
indeed semantically-similar to the seed document. You can use the
stage to retrieve the contents, such as title or abstract, of the documents returned by the document selection
stages. See the document content retrieval
tutorial for detailed explanations and example requests.
Lingo4G can also learn multidimensional embedding vectors for labels. Therefore, instead of the seed document
vector, you can pass a label vector to the
stage. In this arrangement, Lingo4G selects documents that are semantically similar to one or more labels you
The following request returns 20 documents whose embedding vectors lie closest to the embedding vector of the LIGO label (which stands for Laser Interferometer Gravitational-Wave Observatory).
This request in-lines all the necessary dependencies into the
documents:embeddingNearestNeighbors stage. The
property contains the
stage, which in turn uses the
stage to provide a literal label.
If you run the above request in the JSON Sandbox app, you should see a list of matching document identifiers,
along with the 0...1 similarity scores. Again, to verify that the resulting documents relate to the seed
label, you can a
stage to retrieve the titles and abstracts of the papers. See the
Document content retrieval article for a complete tutorial.
We can extend the above request to return the embedding-wise similar documents, but only those that do not contain the LIGO keyword. These would be the related documents that are impossible to find using the traditional keyword-based method.
similarDocuments stage is the same as in the previous request, with retrieval limit
increased to 100.
similarDocumentsWithoutKeywords stage uses the
stage to remove from
similarDocuments those documents that contain the LIGO word. To
this end, the request uses the
components to intersect the list of all similar documents with those documents that do not contain the
stage retrieves the titles and abstracts of the selected documents to confirm that they are related to the
seed label but do not contain it.
Let's examine the top results Lingo4G returns for this request.
Document 29454 talks about "Next Generation Gravitational Wave Detectors", so it's very much related – LIGO, is also a gravitational wave detector. Document 40400 does not contain the LIGO word, but does contain the acronym spelled out. Further documents talk about various aspecs of gravitation wave detection, which is again close related to what LIGO does.
documents:sample stage takes a
random sample of the documents matching the query
you provide. In many cases you can save time and resources by processing a random subset of a large document set
instead of the whole set.
One natural use case for
documents:sample is computing the occurrence statistics for a list of
labels. The following request computes the numbers of occurrences of the photon, electron and
proton labels across papers published between 2006 and 2008.
scope component is the query defining the subset of
documents for which to compute the occurrence frequencies. The
stage uses the
stage to provide the list of labels for which to compute frequencies. Finally, the
tfSample stage computes the estimated occurrence counts. Notice how we use the
stage in the
scope property to take a 10% sample of all the documents matched by the
scope query. For comparison, the request also computes the same statistics using all documents in
If you run the above request in JSON Sandbox, you should see a result similar to the following JSON.
tfSample section contains estimated numbers of occurrences (the
tf section shows the accurate values computed using all documents in scope. Notice that
the estimates can be either larger or smaller than the actual value, sometimes by a noticeable margin as it is
the case with the proton label. Also, in most cases the estimates will contain fractional parts due to
the scaling Lingo4G applies as part of the sampling process.
The response also contains the
status section, which describes the specific tasks Lingo4G performed
to process the request. The
elapsedMs property shows the time Lingo4G took to complete the specific
task. Notice that computing estimated frequencies was 8 times faster than computing the accurate result. For
large scopes this may be a reduction of minutes to seconds.
Common use cases
The output of a document selection stage contains very limited information on its own: just a list of internal document identifiers and their weights. Practical requests will usually combine document selection with other stages to obtain a results meaningful to end users.
Source of documents for other stages
stages provide input for other types of stages, for example:
Document similarities for clustering and 2d mapping. For a list of documents, you can compute a matrix of similarities between pairs of documents on the list. Matrices are not very useful on their own, but you can pass them as input for clustering and 2d embedding.
The following request computes the numbers of documents containing the deep learning phrase in arXiv articles published in 2012, 2014, 2016 and 2018.
The request defines the search phrase part of the query, which is common to all counting periods, in the
components section, so that all stages can reuse it. In the
stages section, the
request defines four stages corresponding to the annual periods in which we want to count documents. Each such
stage uses the
query:filter to intersect the
phrase part of the query with the counting period. Each stage sets the
property to zero, so that Lingo4G only counts the matches, which is usually faster than selecting the
identifiers of the matching documents.
If you run the request in the JSON Sandbox app, you should get a response similar to the following JSON.
As expected, the number of papers containing the deep learning phrase grows exponentially after 2012.