/

vectors

The vectors:​* stages return sets of multidimensional embedding vectors which you can then use to compute vector-based similarity matrices.

The matrix:​knn​Vectors​Similarity stage is the most likely consumer of the results computed by the vectors:​* stages. To compute vector-based similarities between, for example, a set of documents matching a query, your request first needs to use vectors:​precomputed​Document​Embeddings to prepare a subset of embedding vectors corresponding to your documents and then submit the subset of vectors to the matrix computation stage.

You can use the following vectors source stages in your analysis requests:

vectors:​for​Document​Field​Values

✏️ Write overview for vectors:​for​Document​Field​Values.

vectors:​from​Embedding​Service

Returns embedding vectors for one or more fields, computed using an external service providing text embeddings.

vectors:​from​Vector​Field

Returns vectors from an explicit field (requires that an external vector field is added to documents during indexing.

vectors:​precomputed​Document​Embeddings

Returns the precomputed document embedding vectors, narrowed down to the list of documents you provide.

vectors:​precomputed​Label​Embeddings

Returns the precomputed label embedding vectors, narrowed down to the list of labels you provide.

vectors:​reference

References the results of another vectors:​* stage defined in the request.

To avoid large responses, the JSON output of the vectors stage does not include the actual vectors, but only the total number of vectors and a list of undefined vectors.

vectors:​for​Document​Field​Values

Computes vectors corresponding to a set of values.

{
  "type": "vectors:forDocumentFieldValues",
  "sourceValues": {
    "type": "values:reference",
    "auto": true
  },
  "sourceVectors": {
    "type": "vectors:reference",
    "auto": true
  },
  "values": {
    "type": "values:reference",
    "auto": true
  }
}

The typical use case of this stage is computing vector representations for values of one indexed field, such as arXiv paper category or author, so that you can then cluster and 2d-map those field values.

To compute the vectors, this stage requires three pieces of data:

Based on the three inputs, Lingo4G averages all source​Vectors corresponding to the same field value and outputs the averaged vector as the representation of that field value.

The following request demonstrates how you can use this stage to cluster and 2d map the values of the category field of arXiv papers. 

{
  "name": "Clustering and 2d mapping category field values",
  "variables": {
    "field": {
      "name": "Field name",
      "value": "category"
    },
    "scope": {
      "name": "Scope query",
      "comment": "Selects papers from which to collect field values and field similarities.",
      "value": "created:2015*"
    },
    "link_density": {
      "name": "Link density threshold",
      "comment": "The larger the threshold, the larger the number of clusters.",
      "value": 0.2
    },
    "min_similarity": {
      "name": "Minimum similarity",
      "comment": "The minimum required similarity between field values. The larger the minimum required similarity, the more unclustered values.",
      "value": 0.7
    },
    "max_neighbors": {
      "name": "Max neighbors",
      "comment": "The maximum number of similar field values to use in the similarity matrix. The larger the number of neighbors, the more 'fuzzy' the clusters.",
      "value": 2
    }
  },
  "stages": {
    "documents": {
      "type": "documents:byQuery",
      "query": {
        "type": "query:string",
        "query": {
          "@var": "scope"
        }
      },
      "limit": "unlimited"
    },
    "directorates": {
      "type": "values:unique",
      "values": {
        "type": "values:fromDocuments",
        "valueCollector": {
          "type": "valueCollector:fromContentField",
          "fieldName": {
            "@var": "field"
          }
        },
        "multipleValues": "COLLECT_ALL"
      }
    },
    "vectors": {
      "type": "vectors:forDocumentFieldValues",
      "sourceValues": {
        "type": "values:fromDocuments",
        "valueCollector": {
          "type": "valueCollector:fromContentField",
          "fieldName": {
            "@var": "field"
          }
        },
        "multipleValues": "COLLECT_ALL"
      },
      "sourceVectors": {
        "type": "vectors:precomputedDocumentEmbeddings"
      },
      "values": {
        "type": "values:reference",
        "use": "directorates"
      }
    },
    "matrix": {
      "type": "matrix:knnVectorsSimilarity",
      "minSimilarity": {
        "@var": "min_similarity"
      },
      "maxNeighbors": {
        "@var": "max_neighbors"
      }
    },
    "clusters": {
      "type": "clusters:cd",
      "linkDensityThreshold": {
        "@var": "link_density"
      }
    },
    "2dMap": {
      "type": "embedding2d:lv"
    }
  },
  "output": {
    "stages": [
      "directorates",
      "clusters",
      "2dMap",
      "matrix"
    ]
  },
  "tags": [
    "Clustering"
  ]
}

Clustering and 2d mapping the values of the category field of arXiv papers.

source​Values

Type
values
Default
{
  "type": "values:reference",
  "auto": true
}
Required
no

Values corresponding to source vectors.

This list must be aligned with the source​Vectors.

source​Vectors

Type
vectors
Default
{
  "type": "vectors:reference",
  "auto": true
}
Required
no

Vectors corresponding to source values.

This list must be aligned with the source​Values.

values

Type
values
Default
{
  "type": "values:reference",
  "auto": true
}
Required
no

The values for which to compute vectors.

The list must contain unique values, use values:​unique to ensure the uniqueness.

vectors:​from​Embedding​Service

Returns embedding vectors for all text fields provided by the documentContent property. All fields are concatenated into a single text block, which is then sent to an external embedding​Service:​* component. This stage is useful to retrieve dynamically computed embeddings for a relatively small number of documents (because computing embeddings using large language models is typically costly). 

{
  "type": "vectors:fromEmbeddingService",
  "documentContent": null,
  "embeddingService": {
    "type": "embeddingService:reference",
    "auto": true
  },
  "valueSeparator": "\n"
}

document​Content

Type
documentContent
Default
null
Required
yes

In this property, you should provide documents and text fields to compute embeddings for. The document​Content property is an instance of the documentContent stage: it can include one or more field specifications, including maximum length limits.

embedding​Service

Type
embeddingService
Default
{
  "type": "embeddingService:reference",
  "auto": true
}
Required
no

A reference to an embedding service component. Embedding service components may only be declared in the project descriptor's shared components section.

value​Separator

Type
string
Default
"\n"
Required
no

A text separator inserted between multiple field values, if more than one field (or value) is present in a document.

vectors:​from​Vector​Field

Retrieves vector data from an explicit document field. The data must be provided during document indexing, most likely computed from an external source (like an external vector model). 

{
  "type": "vectors:fromVectorField",
  "documents": {
    "type": "documents:reference",
    "auto": true
  },
  "fieldName": null,
  "maxInMemoryKnnSubIndexSize": 50000000
}

documents

Type
documents
Default
{
  "type": "documents:reference",
  "auto": true
}
Required
no

The set of documents for which vector field data should be returned.

field​Name

Type
project:vectorFields
Default
null
Required
yes

Document field containing external vector data added during indexing.

max​In​Memory​Knn​Sub​Index​Size

Type
number
Default
50000000
Required
no

Maximum size, in bytes, of the in-memory temporary kNN index for vectors produced by this stage.

Lingo4G can significantly speed up the computation of vector-based similarities for a small subset of documents by creating and querying a temporary kNN index containing just the vectors corresponding to those documents. Lingo4G creates the temporary index only when its estimated size in bytes is smaller than the max​In​Memory​Knn​Sub​Index​Size threshold.

With the default value of the threshold, Lingo4G will create temporary kNN indices for up to ~400k document vectors.

To disable the creation of the temporary kNN indices, set max​In​Memory​Knn​Sub​Index​Size to 0. We don't recommend doing this in production settings as it can slow down the computation of vector-based similarities by an order of magnitude.

vectors:​precomputed​Document​Embeddings

Returns the precomputed document embedding vectors, narrowed down to the list of documents you provide.

{
  "type": "vectors:precomputedDocumentEmbeddings",
  "documents": {
    "type": "documents:reference",
    "auto": true
  },
  "maxInMemoryKnnSubIndexSize": 50000000
}

You can use this stage to compute vector-based similarities between a set of documents:

{
  "name": "Computing vector-based similarities between documents matching a query",
  "stages": {
    "similarities": {
      "type": "matrix:knnVectorsSimilarity",
      "vectors": {
        "type": "vectors:precomputedDocumentEmbeddings",
        "documents": {
          "type": "documents:byQuery",
          "query": {
            "type": "query:string",
            "query": "clustering"
          }
        }
      }
    }
  }
}

Using vectors:​precomputed​Document​Embeddings to build a similarity matrix between document matching the clustering query.

The request combines the vectors:​precomputed​Document​Embeddings stage with documents:​by​Query to build a subset of document embeddings. Then, the matrix:​knn​Vectors​Similarity stage uses the subset of vectors to compute the similarity matrix.

documents

Type
documents
Default
{
  "type": "documents:reference",
  "auto": true
}
Required
no

The set of documents to which to narrow down the set of document embedding vectors.

max​In​Memory​Knn​Sub​Index​Size

Type
number
Default
50000000
Required
no

Maximum size, in bytes, of the in-memory temporary kNN index for vectors produced by this stage.

Lingo4G can significantly speed up the computation of vector-based similarities for a small subset of documents by creating and querying a temporary kNN index containing just the vectors corresponding to those documents. Lingo4G creates the temporary index only when its estimated size in bytes is smaller than the max​In​Memory​Knn​Sub​Index​Size threshold.

With the default value of the threshold, Lingo4G will create temporary kNN indices for up to 400k document vectors.

To disable the creation of the temporary kNN indices, set max​In​Memory​Knn​Sub​Index​Size to 0. We don't recommend doing this in production settings as it can slow down the computation of vector-based similarities by an order of magnitude.

vectors:​precomputed​Label​Embeddings

Returns the precomputed label embedding vectors, narrowed down to the list of labels you provide.

{
  "type": "vectors:precomputedLabelEmbeddings",
  "labels": {
    "type": "labels:reference",
    "auto": true
  },
  "maxLabelsForSubIndex": 0.05
}

You can use this stage to compute vector-based similarities between a set of documents:

{
  "name": "Computing vector-based similarities between labels",
  "stages": {
    "similarities": {
      "type": "matrix:knnVectorsSimilarity",
      "vectors": {
        "type": "vectors:precomputedLabelEmbeddings",
        "labels": {
          "type": "labels:fromDocuments",
          "documents": {
            "type": "documents:byQuery",
            "query": {
              "type": "query:string",
              "query": "clustering"
            }
          },
          "maxLabels": {
            "type": "labelCount:fixed",
            "value": 200
          }
        }
      }
    }
  }
}

Using vectors:​precomputed​Label​Embeddings to build a similarity matrix between labels related to clustering.

The request combines the vectors:​precomputed​Label​Embeddings stage with labels:​from​Documents to build a subset of label embeddings for labels related to the query clustering. Then, the matrix:​knn​Vectors​Similarity stage uses the subset of vectors to compute the similarity matrix.

labels

Type
labels
Default
{
  "type": "labels:reference",
  "auto": true
}
Required
no

The list of labels to which to narrow down the set of embedding vectors.

max​Labels​For​Sub​Index

Type
number
Default
0.05
Constraints
value >= 0 and value <= 1
Required
no

Determines the threshold for creating a temporary kNN index.

Lingo4G can significantly speed up the computation of vector-based similarities for a small subset of labels by creating and querying a temporary kNN index containing just the vectors corresponding to the input labels. Lingo4G creates the temporary index only when the number of input labels divided by the total number of labels in the index is smaller or equal to the value of this property.

For example, if max​Labels​For​Sub​Index is 0.3, if the input labels list contains fewer than 30% of all labels in the index, Lingo4G creates a temporary index to speed up the computation of similarities. We don't recommend setting this property to 0.0 or 1.0 in production.

Consumers of vectors:​*

The following stages and components take vectors:​* as input:

Stage or component Property
document​Scorer:​by​Vector​Similarity
  • vectors
    • label​Scorer:​by​Vectors​Similarity
  • vectors
    • matrix:​knn​Vectors​Similarity
  • vectors
    • matrix​Rows:​knn​Vectors​Similarity
  • rows
  • columns
    • vectors:​for​Document​Field​Values
  • source​Vectors
    •