Keyword similarity

Document similarity based on shared keywords and phrases is the most straightforward and easy to understand.

Let's see how it works by building a request that performs the following:

1. select documents containing the clustering word,
2. compute the similarity matrix based on common phrases,
3. create a 2d map of the documents from the similarity matrix.

Let's start with a document selector that selects documents matching the clustering query:

{
  "stages": {
    "documents": {
      "type": "documents:byQuery",
      "query": {
        "type": "query:string",
        "query": "clustering"
      }
    }
  }
}

These results alone are not very useful, so let's add the similarity stage to compute the similarities among the selected documents:

{
  "stages": {
    "documents": {
      "type": "documents:byQuery",
      "query": {
        "type": "query:string",
        "query": "clustering"
      }
    },
    "similarities": {
      "type": "matrix:keywordDocumentSimilarity"
    }
  },
  "output": {
    "stages": [
      "documents"
    ]
  }
}

This request uses the matrix:​keyword​Document​Similarity stage to compute similarity based on shared labels. It produces a square similarity matrix with rows and columns corresponding to the document set you provide in the documents property. Our request relies on Lingo4G's auto reference resolution to resolve this property automatically.

The request also introduces an output section to prevent outputting the raw similarity matrix, which is rarely useful alone and increases response size.

Finally, let's add the 2d​Map stage to create a 2d map from the similarity matrix:

{
  "stages": {
    "documents": {
      "type": "documents:byQuery",
      "query": {
        "type": "query:string",
        "query": "clustering"
      }
    },
    "similarities": {
      "type": "matrix:keywordDocumentSimilarity"
    },
    "2dMap": {
      "type": "embedding2d:lv",
      "matrix": {
        "type": "matrix:reference",
        "use": "similarities"
      }
    }
  },
  "output": {
    "stages": [
      "documents",
      "2dMap"
    ]
  }
}

The 2d embedding is computed by the embedding2d:​lv stage. Note that this stage accepts a similarity matrix rather than documents. This means that embedding2d:​lv is agnostic to the specific type of entities it processes. It is the input similarity matrix that defines the semantics and interpretation of the 2d embeddings. Let state this once again for clarity:

Let us illustrate this further by looking at the results of the request we built. Copy and paste the request into the JSON Sandbox app and press the Execute button.

In the JSON results tab, you should see output similar to the following (only the first 5 elements of each array shown for brevity):

{
  "result" : {
    "documents" : {
      "matches" : {
        "value" : 20057,
        "relation" : "GREATER_OR_EQUAL"
      },
      "documents" : [
        {
          "id" : 14539,
          "weight" : 5.828844
        },
        {
          "id" : 277469,
          "weight" : 5.8251066
        },
        {
          "id" : 218609,
          "weight" : 5.8239636
        },
        {
          "id" : 216337,
          "weight" : 5.8107133
        },
        {
          "id" : 192190,
          "weight" : 5.785284
        }
      ]
    },
    "2dMap" : {
      "points" : [
        {
          "x" : -7.236797,
          "y" : -5.8753614
        },
        {
          "x" : 1.736527,
          "y" : -7.3485107
        },
        {
          "x" : -6.2543592,
          "y" : -4.847388
        },
        {
          "x" : -1.7186593,
          "y" : -11.761698
        },
        {
          "x" : -5.1737633,
          "y" : -6.1466236
        }
      ]
    }
  }
}

Because of index alignment, the 2d coordinates of document 14539, located at index 0 of the documents array, are available at index 0 of the points array.

If you switch to the documents map tab of the results area, you should see all the document points visualized:

Although the 2d map is currently unlabeled, you can clearly see areas where documents cluster globally and locally.

Refer to the documentation for the matrix:​keyword​Document​Similarity stage for a detailed description of its similarity algorithm. Feel free to experiment with the stage properties, such as max​Neighbors or min​Query​Labels​Required​In​Similar​Document to see their impact on the 2d embedding.

Embedding similarity

Let's now modify the previous request by swapping the matrix:​keyword​Document​Similarity stage for matrix:​knn​Vectors​Similarity stage, which uses embedding vectors to compute similarities.

{
  "stages": {
    "documents": {
      "type": "documents:byQuery",
      "query": {
        "type": "query:string",
        "query": "clustering"
      }
    },
    "similarities": {
      "type": "matrix:knnVectorsSimilarity",
      "vectors": {
        "type": "vectors:precomputedDocumentEmbeddings"
      }
    },
    "2dMap": {
      "type": "embedding2d:lv",
      "matrix": {
        "type": "matrix:reference",
        "use": "similarities"
      }
    }
  },
  "output": {
    "stages": [
      "documents",
      "2dMap"
    ]
  }
}

The matrix:​knn​Vectors​Similarity computes a square similarity matrix between the list of vectors you provide. In our example, we use the vectors:​precomputed​Document​Embeddings stage, which returns document embedding vectors corresponding to the list of documents you provide in its documents. We rely on Lingo4G's auto reference resolution mechanism to resolve that property automatically.

If you run the modified request in the JSON Sandbox app, you'll notice that the 2d maps resulting from embedding-based similarities are more tightly clustered compared to the keyword similarity based maps.

{
  "similarities": {
    "type": "matrix:knnVectorsSimilarity",
    "vectors": {
      "type": "vectors:fromVectorField",
      "fieldName": "embedding"
    }
  }
}

Content field similarity

Let's explore another similarity type — this time based on arbitrary per-document search queries. This type of similarity connects documents based on the equal values of one or more content fields.

The arXiv example project, on which we run our example requests, contains the set field. For each document, this field defines the top-level part of the paper's arXiv category, such as cs for Computer Science or astro-ph for Astrophysics.

Let's analyze a request that 2d maps documents by the search query-based similarity:

{
  "stages": {
    "documents": {
      "type": "documents:byQuery",
      "query": {
        "type": "query:string",
        "query": "clustering"
      }
    },
    "similarities": {
      "type": "matrix:fromMatrixRows",
      "matrixRows": {
        "type": "matrixRows:byQuery",
        "rows": {
          "type": "documents:reference",
          "use": "documents"
        },
        "queryBuilder": {
          "type": "queryBuilder:string",
          "variables": [
            {
              "values": {
                "type": "valueCollector:fromContentField",
                "fieldName": "set"
              },
              "variable": "SET",
              "quote": true
            }
          ],
          "query": "set:(<SET>)"
        }
      }
    },
    "2dMap": {
      "type": "embedding2d:lv",
      "matrix": {
        "type": "matrix:reference",
        "use": "similarities"
      }
    }
  },
  "output": {
    "stages": [
      "documents",
      "2dMap"
    ]
  }
}

Let's break down the highlighted fragment into pieces:

• The matrix​Rows:​by​Query component computes the rows of the query-based similarity matrix. For each row, corresponding to one document from the documents list, the component builds and executes a document-specific search query (see below). The results of the query are the row document's neighbors — they give rise to the values of that matrix row.

• The query​Builder:​string component builds the document-specific search query.

  The query property defines the query template. The template can contain variable references (denoted by angle brackets) that the query builder fills in with values of the indexed fields of the document. For multivalued fields, Lingo4G joins the values with the operator you define, O​R by default.

  In our example, the query builder declares one variable, S​E​T, and binds that variable to the values of the document's set field. As a result, for each document, the query returns other documents that share the largest number of set field values with the document being processed.

• The matrix:​from​Matrix​Rows stage materializes the rows from the matrix​Rows:​by​Query stage into a complete matrix required by the 2d mapping stage.

  For the curious: matrix:​* vs matrix​Rows:​*

  Until now, all stages generating matrices were of type matrix:​*. This request, however, uses a component of the matrix​Rows:​* type. The distinction comes from the fact that certain algorithms, such as clustering or 2d mapping, require a complete matrix on input, while others, like contrast score computation, can process matrix rows one-by-one. To cater to the latter group of algorithms, Lingo4G implements the same similarity computation methods both as matrix:​* stages and as matrix​Rows:​* components. They compute the same results, but the latter do not materialize the whole similarity matrix, significantly reducing memory usage in algorithms that can consume similarities row-by-row.

  Since the by-query similarity is currently only available in the matrix​Rows:​* form, and we're performing 2d embedding, which requires a materialized matrix, we wrap our matrix​Rows:​by​Query component with the matrix:​from​Matrix​Rows stage, which materializes the rows into the full matrix suitable to pass to the embedding2d:​lv stage for 2d embedding.

If you run the request in the JSON Sandbox app, you should see a result similar to the following:

As expected, since the similarity function is restricted to a nominal field with a limited number of distinct values, the 2d map contains groups corresponding to the individual values of the set field.

The content field-based similarity has limited value on its own but can be part of a composite similarity function together with some content-based similarity method, such as keyword or embedding similarity.

Composite similarity

Let's try the matrix​Rows:​composite component to fuse different similarity matrices, allowing us to 2d map or cluster documents based on multiple criteria.

This request computes a composite similarity matrix that fuses two similarity functions: embedding and keyword-based similarity.

{
  "stages": {
    "documents": {
      "type": "documents:byQuery",
      "query": {
        "type": "query:string",
        "query": "clustering"
      }
    },
    "similarities": {
      "type": "matrix:fromMatrixRows",
      "matrixRows": {
        "type": "matrixRows:composite",
        "matrixRows": [
          {
            "type": "matrixRows:weighted",
            "weight": 0.1,
            "matrixRows": {
              "type": "matrixRows:byQuery",
              "rows": {
                "type": "documents:reference",
                "use": "documents"
              },
              "queryBuilder": {
                "type": "queryBuilder:string",
                "variables": [
                  {
                    "values": {
                      "type": "valueCollector:fromContentField",
                      "fieldName": "set"
                    },
                    "variable": "SET",
                    "quote": true
                  }
                ],
                "query": "set:(<SET>)"
              }
            }
          },
          {
            "type": "matrixRows:knnVectorsSimilarity",
            "vectors": {
              "rows": {
                "type": "vectors:precomputedDocumentEmbeddings"
              },
              "columns": {
                "type": "vectors:precomputedDocumentEmbeddings"
              }
            }
          }
        ]
      }
    },
    "2dMap": {
      "type": "embedding2d:lv",
      "matrix": {
        "type": "matrix:reference",
        "use": "similarities"
      }
    },
    "clustersBySet": {
      "type": "clusters:byValues",
      "values": {
        "type": "values:fromDocumentField",
        "fieldName": "set",
        "multipleValues": "COLLECT_FIRST"
      }
    }
  },
  "output": {
    "stages": [
      "documents",
      "2dMap",
      "clustersBySet"
    ]
  }
}

Let's break down the similarity computation part:

• The matrix​Rows:​composite component fuses two or more similarity matrix row sources. In our example, we use two similarity matrix components:

  • matrix​Rows:​by​Query produces similarities based on the number of shared content field values.

    We wrap the matrix​Rows:​by​Query component in the matrix​Rows:​weighted component to apply some weighting to the content field-based similarity. This allows us to increase or decrease the impact of this specific similarity on the final similarity matrix. In the example, we weigh the content field similarity by 0.1, leaving the embedding similarity unweighted (which is equivalent to a weight of 1.0).

  • matrix​Rows:​knn​Vectors​Similarity produces similarities based on the document's multidimensional vectors.

• Since the 2d mapping stage requires a materialized similarity matrix, we wrap the matrix​Rows:​composite in the matrix:​from​Matrix​Rows stage to collect the matrix rows produced by the composite component into a full matrix.

Additionally, the request adds the clusters​By​Set stage, which groups the documents by the first value of their set field. This helps us track the impact of the content field similarity on the final result.

If you run the request in the JSON Sandbox app, you should see a result like this:

To see the impact of the set field similarity on the result, temporarily lower the weight property from 0.1 to 0.0 and re-run the analysis.

By comparing the screenshots, you can see that the similarity based on the shared set field values helps bring the separate brown areas together while still maintaining the local groupings.