matrix​Rows:​by​Query

For each row, builds a query specific to the row's document and takes query results as the row's column values.

{
  "type": "matrixRows:byQuery",
  "columns": {
    "type": "documents:reference",
    "auto": true
  },
  "maxNeighbors": 10,
  "normalized": true,
  "queryBuilder": {
    "type": "queryBuilder:reference",
    "auto": true
  },
  "rows": {
    "type": "documents:reference",
    "auto": true
  },
  "threads": "auto"
}

You can use this component to build similarity matrices based on the values of content fields of documents.

Note that depending on the number of rows on input and the complexity of the query, computing all rows of the matrix may take significant time.

The following request clusters and 2d maps arXiv papers based on their category (the set field).

{
  "name": "Clusters and 2d maps papers based on their category",
  "stages": {
    "documents": {
      "type": "documents:byQuery",
      "query": {
        "type": "query:string",
        "query": "clustering"
      }
    },
    "similarityBySet": {
      "type": "matrix:fromMatrixRows",
        "type": "matrixRows:byQuery",
        "maxNeighbors": 2,
        "queryBuilder": {
          "type": "queryBuilder:string",
          "variables": [
            {
              "input": "set",
              "variable": "SET",
              "maxValues": 1,
              "quote": true
            }
          ],
          "query": "set:(<SET>)",
          "maxQueriesForDebugLogging": 2
        }
      }
    },
    "2dMapBySet": {
      "type": "embedding2d:lv"
    },
    "clustersBySet": {
      "type": "clusters:byValues",
      "values": {
        "type": "values:fromDocumentField",
        "fieldName": "set",
        "multipleValues": "COLLECT_FIRST"
      }
    }
  },
  "output": {
    "stages": [
      "documents",
      "2dMapBySet",
      "clustersBySet"
    ]
  }
}

The request combines matrix​Rows:​by​Query with the query​Builder:​string component, which from each document extracts its set field value and runs the set:​<​S​E​T> query to get other papers belonging to the same category.

In practice, you may want to use matrix​Rows:​composite to combine matrix​Rows:​by​Query with and keyword- or vector-based similarities to perform clustering or 2d mapping based the composite similarity function.

matrix​Rows:​composite

Aggregates multiple matrix row sources into one matrix row source.

{
  "type": "matrixRows:composite",
  "matrixRows": [],
  "normalized": true,
  "weightAggregation": "SUM"
}

All input matrices must have the same dimensions - they must have the same number of rows and columns.

The output matrix will have the same number of rows as the input matrices. Columns of the output matrix will be a union of the values provided by the input matrix rows, aggregated using the weight​Aggregation function.

matrix​Rows:​from​Matrix

Converts a full matrix into a source of rows. Mostly useful for debugging and testing.

{
  "type": "matrixRows:fromMatrix",
  "matrix": {
    "type": "matrix:reference",
    "auto": true
  }
}

matrix​Rows:​keyword​Document​Similarity

Computes keyword-based (More-Like-This) similarities between documents.

{
  "type": "matrixRows:keywordDocumentSimilarity",
  "index": {
    "columns": {
      "documents": null
    },
    "fields": {
      "type": "featureFields:reference",
      "auto": true
    },
    "maxColumnDocumentsForSubIndex": 0.3,
    "maxInMemorySubIndexSize": 8000000,
    "rows": {
      "documents": null,
      "labelCollector": {
        "type": "labelCollector:topFromFeatureFields",
        "failIfEmbeddingsNotAvailable": true,
        "fields": {
          "type": "featureFields:reference",
          "auto": true
        },
        "labelFilter": {
          "type": "labelFilter:reference",
          "auto": true
        },
        "labelListFilter": {
          "type": "labelListFilter:truncatedPhrases"
        },
        "labelWeighting": "EMBEDDING",
        "minTf": 0,
        "minWeight": 0,
        "minWeightMass": 1,
        "tieResolution": "AUTO"
      },
      "maxQueryLabelsPerRowDocument": 10,
      "minQueryLabelsPerRowDocument": 0,
      "threads": "auto"
    },
    "threads": "auto"
  },
  "maxNeighbors": 10,
  "minQueryLabelsRequiredInColumnDocument": 1,
  "normalized": false,
  "threads": "auto"
}

To compute the keyword-based document similarity matrix rows, Lingo4G performs the following steps:

1. For each row document, Lingo4G uses the label​Collector you provide to extract up to max​Query​Labels​Per​Row​Document labels that characterize the document.

   If the number of labels extracted from the row document is smaller than min​Query​Labels​Per​Row​Document, Lingo4G excludes the document from processing. The corresponding row in the similarity matrix will be empty.

2. If the number of column documents in relation to the total number of documents in the index is larger than max​Column​Documents​For​Sub​Index, Lingo4G creates a temporary inverted index for the column documents to improve the performance of the similarity matrix building.

3. For each row document, Lingo4G builds a search query consisting of labels extracted in step 1. Lingo4G restricts the query to find matches only among the column documents. Additionally, the query matches only documents that contain at least min​Query​Labels​Required​In​Column​Document of the document's labels obtained in step 1.

4. For each row document, Lingo4G runs the corresponding query it built in step 3 to retrieve up to max​Neighbors matching column documents. Lingo4G uses up to threads to execute the queries in parallel.

5. For each row document, Lingo4G builds the corresponding row of the similarity matrix using the matching documents retrieved in step 4.

   For example, assuming that the query corresponding to document at index 2 in the row documents array matched column documents at index 0 and 3, Lingo4G puts the following values into the similarity matrix:

   $$M=\left[\begin{array}{cccc}\mathrm{·}& \mathrm{·}& \mathrm{·}& \mathrm{·}\\ \mathrm{·}& \mathrm{·}& \mathrm{·}& \mathrm{·}\\ \mathrm{<msub>\; <mi>s</mi>\; <mi>0</mi>\; </msub>}& \mathrm{·}& \mathrm{·}& \mathrm{<msub>\; <mi>s</mi>\; <mi>3</mi>\; </msub>}\\ \mathrm{·}& \mathrm{·}& \mathrm{·}& \mathrm{·}\end{array}\right]$$

   where $s_0$ and $s_3$ are the search scores obtained in step 4 for document at index 2.

   Note that output matrix is rectangular: the number of is equal to the number of row documents and the number of columns is equal to the number of column documents on input.

6. If normalized is true, Lingo4G normalizes values each row of matrix $M$ to fall in the 0...1 range.

matrix​Rows:​knn​Vectors​Similarity

Computes a label or document similarity matrix based on multidimensional vector distance.

{
  "type": "matrixRows:knnVectorsSimilarity",
  "maxNeighbors": 10,
  "minSimilarity": 0,
  "threads": "auto",
  "vectors": {
    "columns": {
      "type": "vectors:reference",
      "auto": true
    },
    "rows": {
      "type": "vectors:reference",
      "auto": true
    }
  }
}

For each vector in the rows vector set, Lingo4G finds up to max​Neighbors closest vectors in the columns vector set and transfers the cosine similarities between the nearest vectors to the output matrix. Therefore, the output matrix is rectangular: the number of rows is equal to the number of vectors in the rows set, the number of columns is equal to the number of vectors in the columns vector set.

If you use vectors:​precomputed​Document​Embeddings as the input vector set, this stage computes similarities between documents. Similarly, if you provide vectors:​precomputed​Label​Embeddings on input, this stage computes similarities between labels. You can also mix label and document embeddings in computation of the same matrix to compute label-to-documents or documents-to-labels similarities.

In many cases, embedding-based similarities offer better results compared to their and keyword-based counterparts.

matrix​Rows:​weighted

Applies additional value weighting to the provided matrix rows.

{
  "type": "matrixRows:weighted",
  "matrixRows": {
    "type": "matrixRows:reference",
    "auto": true
  },
  "offset": 0,
  "weight": 1
}

Lingo4G computes each value of the output matrix using the following formula:

${\mathrm{value}}_{\mathrm{out}}={\mathrm{value}}_{\mathrm{in}}*\mathrm{weight}+\mathrm{offset}$

where:

$\mathrm{weight}$ is value of the weight property of this stage,
$\mathrm{offset}$ is value of the offset property of this stage.

This stage is most useful in combination with matrix​Rows:​composite, where you can use it to balance the weights of the matrices being composed.