matrix:​cooccurrence​Label​Similarity

Computes a label similarity matrix based on how the labels co-occur in the documents you provide.

{
  "type": "matrix:cooccurrenceLabelSimilarity",
  "cooccurrenceWindowSize": 32,
  "documents": {
    "type": "documents:reference",
    "auto": true
  },
  "fields": {
    "type": "featureFields:reference",
    "auto": true
  },
  "labels": {
    "type": "labels:reference",
    "auto": true
  },
  "normalized": true,
  "similarityWeighting": "INCLUSION",
  "threads": "auto"
}

You can use the cooccurrence-based similarity matrix to to cluster or 2d-map a set of labels.

Lingo4G computes the co-occurrence-based label similarity matrix in the following way:

1. For each pair (labelA, labelB) of input labels, Lingo4G scans the input documents to compute:

   • $f_{\mathrm{A}}$ : how many times labelA occurred in the input documents,
   • $f_{\mathrm{B}}$ : how many times labelB occurred in the input documents,
   • $f_{\mathrm{AB}}$ how many times both labelA and labelB occurred in the document, at most cooccurrence​Window​Size words apart.

2. Based on the $f_{\mathrm{A}}$ , $f_{\mathrm{B}}$ and $f_{\mathrm{AB}}$ frequencies and the similarity​Weighting method you choose, Lingo4G computes the $s_{\mathrm{AB}}$ and $s_{\mathrm{BA}}$ similarity values and puts them in the output matrix at rows and columns corresponding to the indices of labelA and labelB in the input labels list.

   For example, if on the labels list labelA is at index 0, and labelB is at index 2, Lingo4G puts the corresponding similarities at the following locations in the output matrix:

   $$M=\left[\begin{array}{cccc}\mathrm{·}& \mathrm{·}& \mathrm{<msub>\; <mi>s</mi>\; <mi>AB</mi>\; </msub>}& \mathrm{·}\\ \mathrm{·}& \mathrm{·}& \mathrm{·}& \mathrm{·}\\ \mathrm{<msub>\; <mi>s</mi>\; <mi>BA</mi>\; </msub>}& \mathrm{·}& \mathrm{·}& \mathrm{·}\\ \mathrm{·}& \mathrm{·}& \mathrm{·}& \mathrm{·}\end{array}\right]$$

3. If normalized is true, Lingo4G globally normalizes all values in the similarity matrix to the 0...1 range.

Note that depending on the similarity​Weighting you choose, the matrix may or may not be symmetrical.

The following request computes co-occurrence counts for a list of labels across all documents containing at least one of the labels.

{
  "name": "Computing co-occurrence counts for a list of labels.",
  "stages": {
    "labels": {
      "type": "labels:direct",
      "labels": [
        {
          "label": "cluster"
        },
        {
          "label": "galaxies"
        },
        {
          "label": "factorization"
        },
        {
          "label": "photon"
        },
        {
          "label": "algebra"
        }
      ]
    },
    "matrix": {
      "type": "matrix:cooccurrenceLabelSimilarity",
      "labels": {
        "type": "labels:reference",
        "use": "labels"
      },
      "documents": {
        "type": "documents:byQuery",
        "query": {
          "type": "query:forLabels",
          "labels": {
            "type": "labels:reference",
            "use": "labels"
          }
        },
        "limit": "unlimited"
      },
      "similarityWeighting": "COOCCURRENCES",
      "normalized": false
    }
  }
}

Note how the requests uses the query:​for​Labels query component to select documents containing at least one of the input labels. We also use the C​O​O​C​C​U​R​R​E​N​C​E​S similarity weighting and disable matrix normalization to get the actual number of label co-occurrences.

A typical use-case for matrix:​cooccurrence​Label​Similarity is clustering or 2d mapping of labels. The following request briefly demonstrates this use case.

{
  "name": "Clustering and 2d-mapping labels based on co-occurrence similarity.",
  "stages": {
    "documents": {
      "type": "documents:byQuery",
      "query": {
        "type": "query:string",
        "query": "clustering"
      },
      "limit": "unlimited"
    },
    "labels": {
      "type": "labels:fromDocuments",
      "maxLabels": {
        "type": "labelCount:fixed",
        "value": 1000
      }
    },
    "similarities": {
      "type": "matrix:cooccurrenceLabelSimilarity"
    },
    "clusters": {
      "type": "clusters:ap"
    },
    "map2d": {
      "type": "embedding2d:lv"
    }
  },
  "output": {
    "stages": [
      "documents",
      "labels",
      "clusters",
      "map2d"
    ]
  }
}

The above request collects 1000 labels from documents containing the word clustering, computes a co-occurrence similarity matrix for those labels and then arranges the labels into clusters and a 2d-map. The request uses the explicit output.stages array to prevent the output of the similarity matrix. Also note that the request uses auto-references to pass results between all the stages.

If you run the request in the JSON Sandbox app, you should see an interactive visualization of the clusters and the 2d map. Also have a look at the diagram tab, for a graphical representation of the connections between various stages.

Finally, if your index contains label embeddings, the embedding-based matrix:​knn​Vectors​Similarity stage may provide better clusters and 2d-maps.

matrix:​direct

A matrix where you directly provide all values in rows and columns.

{
  "type": "matrix:direct",
  "matrix": {
    "columns": 0,
    "indices": [],
    "values": []
  }
}

Direct matrices are useful mostly for debugging purposes or when you want to cluster or 2d-map a set of similarities coming from an external source.

matrix:​element​Wise​Product

Computes the element-by-element product of two matrices.

{
  "type": "matrix:elementWiseProduct",
  "factorA": null,
  "factorB": null
}

matrix:​from​Matrix​Rows

Materializes matrix rows into an in-memory matrix.

{
  "type": "matrix:fromMatrixRows",
  "matrixRows": {
    "type": "matrixRows:reference",
    "auto": true
  }
}

This stage is useful mostly for debugging requests involving matrix rows.

matrix:​keyword​Document​Similarity

Computes a document similarity matrix based on the labels the documents share.

{
  "type": "matrix:keywordDocumentSimilarity",
  "documents": {
    "type": "documents:reference",
    "auto": true
  },
  "fields": {
    "type": "featureFields:reference",
    "auto": true
  },
  "labelCollector": {
    "type": "labelCollector:topFromFeatureFields",
    "fields": {
      "type": "featureFields:reference",
      "auto": true
    },
    "labelFilter": {
      "type": "labelFilter:reference",
      "auto": true
    },
    "labelListFilter": {
      "type": "labelListFilter:truncatedPhrases"
    },
    "minTf": 0,
    "minTfMass": 1,
    "tieResolution": "AUTO"
  },
  "maxDocumentsForSubIndex": 0.3,
  "maxInMemorySubIndexSize": 8000000,
  "maxNeighbors": 8,
  "maxQueryLabelsPerDocument": 4,
  "minQueryLabelsPerDocument": 1,
  "minQueryLabelsRequiredInSimilarDocument": 1,
  "normalized": true,
  "threads": "auto"
}

You can use the keyword-based document similarity matrix to cluster or 2d-map a set of documents.

To compute the keyword-based document similarity, Lingo4G performs the following steps:

1. For each document in the input documents, Lingo4G uses the label​Collector you provide to extract up to max​Query​Labels​Per​Document labels that characterize the document.

   If the number of labels extracted from the document is smaller than min​Query​Labels​Per​Document, Lingo4G excludes the document from processing. The corresponding row in the similarity matrix will be empty.

2. If the number of input documents in relation to the total number of documents in the index is larger than max​Documents​For​Sub​Index, Lingo4G creates a temporary inverted index that improves the performance of the similarity matrix building.

3. For each input document, Lingo4G builds a search query consisting of labels extracted in step 1. Lingo4G restricts the query to find matches only among the input documents. Additionally, the query matches only documents that contain at least min​Query​Labels​Required​In​Similar​Document of the document's labels obtained in step 1.

4. For each input document, Lingo4G runs the corresponding query it built in step 3 to retrieve up to max​Neighbors matching documents. Lingo4G uses up to threads to execute the queries in parallel.

5. For each input document, Lingo4G builds the corresponding row of the similarity matrix using the matching documents retrieved in step 4.

   For example, assuming that the query corresponding to document at index 2 in the input documents array matched documents at index 0, 2, and 3, Lingo4G puts the following values into the similarity matrix:

   $$M=\left[\begin{array}{cccc}\mathrm{·}& \mathrm{·}& \mathrm{·}& \mathrm{·}\\ \mathrm{·}& \mathrm{·}& \mathrm{·}& \mathrm{·}\\ \mathrm{<msub>\; <mi>s</mi>\; <mi>0</mi>\; </msub>}& \mathrm{·}& \mathrm{<msub>\; <mi>s</mi>\; <mi>2</mi>\; </msub>}& \mathrm{<msub>\; <mi>s</mi>\; <mi>3</mi>\; </msub>}\\ \mathrm{·}& \mathrm{·}& \mathrm{·}& \mathrm{·}\end{array}\right]$$

   where $s_0$ , $s_2$ and $s_3$ are the search scores obtained in step 4 for document at index 2.

   Note that matrix $M$ is square and asymmetrical – the search query for document at index 0 or 3 may not return document 2.

6. If normalized is true, Lingo4G normalizes values each row of matrix $M$ to fall in the 0...1 range.

In the Apache Lucene, Solr and Elasticsearch world, this kind of similarity is also called More-Like-This similarity.

The following request uses keyword documents similarity to cluster and 2d map documents.

{
  "name": "Clustering and 2d-mapping documents using keyword document similarity.",
  "stages": {
    "documents": {
      "type": "documents:byQuery",
      "query": {
        "type": "query:string",
        "query": "clustering"
      },
      "limit": 10000
    },
    "matrix": {
      "type": "matrix:keywordDocumentSimilarity",
      "documents": {
        "type": "documents:reference",
        "use": "documents"
      }
    },
    "documents2dEmbedding": {
      "type": "embedding2d:lv",
      "matrix": {
        "type": "matrix:reference",
        "use": "matrix"
      }
    },
    "cluster": {
      "type": "clusters:ap",
      "matrix": {
        "type": "matrix:reference",
        "use": "matrix"
      },
      "inputPreference": -10000,
      "softening": 0.2
    }
  },
  "output": {
    "stages": [
      "documents",
      "documents2dEmbedding",
      "cluster"
    ]
  }
}

If you run the above request in the JSON Sandbox app, you should see the documents represented as a 2d map with point color corresponding to the top-level cluster the document belongs to.

matrix:​keyword​Label​Document​Similarity

Computes a rectangular similarity matrix a list of labels and a set of documents you provide.

{
  "type": "matrix:keywordLabelDocumentSimilarity",
  "documents": {
    "type": "documents:reference",
    "auto": true
  },
  "fields": {
    "type": "featureFields:reference",
    "auto": true
  },
  "labels": {
    "type": "labels:reference",
    "auto": true
  },
  "maxSimilarDocumentsPerLabel": 5,
  "threads": "auto"
}

You can use this stage to overlay labels on a pre-existing 2d map of documents in such a way that labels summarize the documents lying in various areas of the map.

Lingo4G computes the label-document similarity matrix in the following way:

1. For each label from the input labels list, Lingo4G builds a search query consisting of that label, covering the fields you provide and limited to the set of target documents.

2. For each input label, Lingo4G executes the search query it built in step 1 and collects up to max​Similar​Documents​Per​Label documents. Then, Lingo4G sets the value in the output similarity matrix at the row corresponding to label index and column corresponding to the index of the matching document to equal the search score of the matching document.

The following request uses the matrix:​keyword​Label​Document​Similarity stage to describe a 2d map of documents by placing labels near groups of documents related to those labels.

{
  "name": "Labeling a 2d map of documents using label-document similarity matrix.",
  "stages": {
    "documents": {
      "type": "documents:byQuery",
      "query": {
        "type": "query:string",
        "query": "clustering"
      },
      "limit": 10000
    },
    "labels": {
      "type": "labels:fromDocuments",
      "maxLabels": {
        "type": "labelCount:fixed",
        "value": 500
      }
    },
    "documents2dEmbedding": {
      "type": "embedding2d:lv",
      "matrix": {
        "type": "matrix:keywordDocumentSimilarity"
      }
    },
    "label2dOverlay": {
      "type": "embedding2d:lvOverlay",
      "matrix": {
        "type": "matrix:keywordLabelDocumentSimilarity",
        "labels": {
          "type": "labels:reference",
          "use": "labels"
        },
        "documents": {
          "type": "documents:reference",
          "use": "documents"
        }
      },
      "embedding2d": {
        "type": "embedding2d:reference",
        "use": "documents2dEmbedding"
      }
    }
  }
}

See the reference documentation for the embedding2d:​lv​Overlay stage for in-depth explanation of this kind of requests.

matrix:​knn2d​Distance​Similarity

Computes a matrix of similarities based on Euclidean distance between 2d points.

{
  "type": "matrix:knn2dDistanceSimilarity",
  "embedding2d": {
    "type": "embedding2d:reference",
    "auto": true
  },
  "maxNearestPoints": 8
}

You can use this stage to identify clusters of nearby points in a 2d map.

Lingo4G computes the matrix in the following way:

1. For each 2d point $p$ in the input embedding2d, find up to max​Nearest​Points with respect to the Euclidean distance.

2. For each nearest point $p_n$ found in step 1, compute the similarity using the following formula:

   $$s=\frac{1}{1+e^d}$$

   where $d$ is the Euclidean distance between points $p$ and $p_n$ .

   The above similarity formula converts 2d distances to the 0...1 in such a way that the similarity for points with zero distance is 1 and similarity for points that are infinitely apart is 0.

3. Put similarity $s$ computed in step 2 into the similarity matrix at the row and column location corresponding to points $p$ and $p_n$ .

The following request uses the matrix:​knn2d​Distance​Similarity stage to identify clusters of nearby points in a 2d map:

{
  "name": "Clustering of points on a 2d map of labels",
  "stages": {
    "documents": {
      "type": "documents:byQuery",
      "query": {
        "type": "query:string",
        "query": "clustering"
      },
      "limit": "unlimited"
    },
    "labels": {
      "type": "labels:fromDocuments",
      "maxLabels": {
        "type": "labelCount:fixed",
        "value": 2000
      }
    },
    "map2d": {
      "type": "embedding2d:lv",
      "matrix": {
        "type": "matrix:knnVectorsSimilarity",
        "vectors": {
          "type": "vectors:precomputedLabelEmbeddings"
        }
      }
    },
    "clustersOnMap2d": {
      "type": "clusters:ap",
      "matrix": {
        "type": "matrix:knn2dDistanceSimilarity",
        "embedding2d": {
          "type": "embedding2d:reference",
          "use": "map2d"
        },
        "maxNearestPoints": 16
      },
      "inputPreference": -1,
      "softening": 0.5
    }
  }
}

The above request collects 2000 labels from documents containing the word clustering and arranges them on a 2d map using the matrix:​knn​Vectors​Similarity similarity. Then, it clusters the label points on the 2d map using the clusters:​ap clustering algorithm and the matrix:​knn2d​Distance​Similarity similarity.

If you run the above request in the JSON Sandbox app, you should see the map of labels with proximity-based clusters represented as different colors. Switch to the diagram for a visualization of the data flow in the request.

matrix:​knn​Vectors​Similarity

Computes a label or document similarity matrix based on multidimensional vector distance.

{
  "type": "matrix:knnVectorsSimilarity",
  "maxNeighbors": 10,
  "threads": "auto",
  "vectors": {
    "type": "vectors:reference",
    "auto": true
  }
}

You can use this stage to cluster or 2d-map a set of documents or labels based on the similarity of their corresponding embedding vectors.

For each vector in the input set of multidimensional vectors, Lingo4G finds up to max​Neighbors closest vectors in the same vector set and transfers the cosine similarities between the nearest vectors to the output matrix. Therefore, the output matrix is square, its size is equal to the number of vectors in the input vectors set and values fall in the 0...1 range.

If you use vectors:​precomputed​Document​Embeddings as the input vectors set, this stage computes similarities between documents. Similarly, if you provide vectors:​precomputed​Label​Embeddings on input, this stage computes similarities between labels.

In many cases, embedding-based similarities offer better clustering and 2d mapping results compared to their co-occurrence and keyword-based counterparts. The following request arranges 2000 labels into a 2d map where similar labels are close to each other.

{
  "name": "Computing a 2d map of labels based on multidimensional embedding similarity",
  "stages": {
    "documents": {
      "type": "documents:byQuery",
      "query": {
        "type": "query:string",
        "query": "clustering"
      },
      "limit": "unlimited"
    },
    "labels": {
      "type": "labels:fromDocuments",
      "maxLabels": {
        "type": "labelCount:fixed",
        "value": 2000
      },
      "documents": {
        "type": "documents:reference",
        "use": "documents"
      }
    },
    "map2d": {
      "type": "embedding2d:lv",
      "matrix": {
        "type": "matrix:knnVectorsSimilarity",
        "vectors": {
          "type": "vectors:precomputedLabelEmbeddings",
          "labels": {
            "type": "labels:reference",
            "use": "labels"
          }
        }
      }
    }
  }
}