matrix​Rows:​from​Matrix

Converts a full matrix into a source of rows. Mostly useful for debugging and testing.

{
  "type": "matrixRows:fromMatrix",
  "matrix": {
    "type": "matrix:reference",
    "auto": true
  }
}

matrix​Rows:​keyword​Document​Similarity

Computes keyword-based (More-Like-This) similarities between documents.

{
  "type": "matrixRows:keywordDocumentSimilarity",
  "index": {
    "columns": {
      "documents": null
    },
    "fields": {
      "type": "featureFields:reference",
      "auto": true
    },
    "maxColumnDocumentsForSubIndex": 0.3,
    "maxInMemorySubIndexSize": 8000000,
    "rows": {
      "documents": null,
      "labelCollector": {
        "type": "labelCollector:topFromFeatureFields",
        "fields": {
          "type": "featureFields:reference",
          "auto": true
        },
        "labelFilter": {
          "type": "labelFilter:reference",
          "auto": true
        },
        "labelListFilter": {
          "type": "labelListFilter:truncatedPhrases"
        },
        "minTf": 0,
        "minTfMass": 1,
        "tieResolution": "AUTO"
      },
      "maxQueryLabelsPerRowDocument": 10,
      "minQueryLabelsPerRowDocument": 0,
      "threads": "auto"
    },
    "threads": "auto"
  },
  "maxNeighbors": 10,
  "minQueryLabelsRequiredInColumnDocument": 1,
  "normalized": false,
  "threads": "auto"
}

To compute the keyword-based document similarity matrix rows, Lingo4G performs the following steps:

1. For each row document, Lingo4G uses the label​Collector you provide to extract up to max​Query​Labels​Per​Row​Document labels that characterize the document.

   If the number of labels extracted from the row document is smaller than min​Query​Labels​Per​Row​Document, Lingo4G excludes the document from processing. The corresponding row in the similarity matrix will be empty.

2. If the number of column documents in relation to the total number of documents in the index is larger than max​Column​Documents​For​Sub​Index, Lingo4G creates a temporary inverted index for the column documents to improve the performance of the similarity matrix building.

3. For each row document, Lingo4G builds a search query consisting of labels extracted in step 1. Lingo4G restricts the query to find matches only among the column documents. Additionally, the query matches only documents that contain at least min​Query​Labels​Required​In​Column​Document of the document's labels obtained in step 1.

4. For each row document, Lingo4G runs the corresponding query it built in step 3 to retrieve up to max​Neighbors matching column documents. Lingo4G uses up to threads to execute the queries in parallel.

5. For each row document, Lingo4G builds the corresponding row of the similarity matrix using the matching documents retrieved in step 4.

   For example, assuming that the query corresponding to document at index 2 in the row documents array matched column documents at index 0 and 3, Lingo4G puts the following values into the similarity matrix:

   $$M=\left[\begin{array}{cccc}\mathrm{·}& \mathrm{·}& \mathrm{·}& \mathrm{·}\\ \mathrm{·}& \mathrm{·}& \mathrm{·}& \mathrm{·}\\ \mathrm{<msub>\; <mi>s</mi>\; <mi>0</mi>\; </msub>}& \mathrm{·}& \mathrm{·}& \mathrm{<msub>\; <mi>s</mi>\; <mi>3</mi>\; </msub>}\\ \mathrm{·}& \mathrm{·}& \mathrm{·}& \mathrm{·}\end{array}\right]$$

   where $s_0$ and $s_3$ are the search scores obtained in step 4 for document at index 2.

   Note that output matrix is rectangular: the number of is equal to the number of row documents and the number of columns is equal to the number of column documents on input.

6. If normalized is true, Lingo4G normalizes values each row of matrix $M$ to fall in the 0...1 range.

matrix​Rows:​knn​Vectors​Similarity

Computes a label or document similarity matrix based on multidimensional vector distance.

{
  "type": "matrixRows:knnVectorsSimilarity",
  "maxNeighbors": 10,
  "threads": "auto",
  "vectors": {
    "columns": {
      "type": "vectors:reference",
      "auto": true
    },
    "rows": {
      "type": "vectors:reference",
      "auto": true
    }
  }
}

For each vector in the rows vector set, Lingo4G finds up to max​Neighbors closest vectors in the columns vector set and transfers the cosine similarities between the nearest vectors to the output matrix. Therefore, the output matrix is rectangular: the number of rows is equal to the number of vectors in the rows set, the number of columns is equal to the number of vectors in the columns vector set.

If you use vectors:​precomputed​Document​Embeddings as the input vector set, this stage computes similarities between documents. Similarly, if you provide vectors:​precomputed​Label​Embeddings on input, this stage computes similarities between labels. You can also mix label and document embeddings in computation of the same matrix to compute label-to-documents or documents-to-labels similarities.

In many cases, embedding-based similarities offer better results compared to their and keyword-based counterparts.