Lingo4G plugin API

This reference describes the API of the Lingo4G plugin.

The Lingo4G plugin makes it easy to display 2d embeddings produced by Lingo4G version 2.x. The Lingo4GDocumentMap plugin creates the layers required to display a 2d document embedding produced by Lingo4G analysis API v2.

import { DotAtlas, Lingo4GDocumentMap } from "@carrotsearch/dotatlas";

// Fetch results from Lingo4G (pseudo code below)
const lingo4GApiUrl = "https://yourserver.com/lingo4g/api/v2";
const result = await fetch(lingo4GApiUrl + "/analysis");

const dotAtlas = DotAtlas.with(Lingo4GDocumentMap).embed({
  "element": document.querySelector("#visualization"),
  "lingo4g:result": result
});

When you provide a new lingo4g:result, the Lingo4GDocumentMap plugin creates three layers:

elevation layer

The plugin derives point coordinates on the elevations layer will from the 2d document coordinates returned by Lingo4G.

The plugin derives elevation values from the document's similarity to the cluster exemplar, which should highlight the "importance" of each document and create a naturally-looking map.

markers layer

Point coordinates on the markers layer represent the 2d document coordinates returned by Lingo4G, marker size depends on the similarity to the cluster exemplar. Marker color will represent the cluster the document belongs to.

The plugin uses a heuristic to compute a marker size scaling factor that makes the visualization legible regardless of the number of documents being visualized. You can use the lingo4g:baseMarkerSize property to fine-tune the size of markers this plugin produces.

labels layer

Coordinates and text of labels represent the labels embedding part of the Lingo4G response.

See the lingo4g:result documentation for the list of Lingo4G analysis stage results the JSON response must contain to build the layers. To customize the layers this plugin crates, use the lingo4g:customize:layers callback.

Note that in order to get a naturally-looking map, you will usually need an embedding with at least a few thousands of documents.

Properties

lingo4g:result

The result JSON returned by the Lingo4G analysis API v2.

For the plugin to work, the response must contain results of the following analysis stages:

Stage type Stage name Purpose
labels:* labels The text of labels to display on the map.
embedding2d:* labelEmbeddings2d Coordinates of the labels.
embedding2d:* documentEmbeddings2d Coordinates of the documents to display on the map.
clusters:* documentClusters Clusters of documents to show on the map. The plugin uses the clusters to determine the colors and elevation values of documents' markers.
labelClusters:* documentClusterLabels Document cluster labels. The plugin uses document cluster labels to determine the display priority for the labels on the map.

By default, the plugin expects the stages to be named as listed in the Stage name column of the table. You can customize the stage names using the lingo4g:result:stageName:* properties of this plugin.

The following figure shows an example Lingo4G analysis request JSON that generates all the data required for the plugin to display a 2d map of documents. See Lingo4G documentation for more information about building Lingo4G analysis request JSONs.

{
  "name": "Document 2d embedding with labels and clusters by embedding similarity",
  "comment": "Puts documents on a 2d map based on the multidimensional document embedding similarity. Additionally, computes 2d coordinates of labels to describe specific areas of the map and clusters documents based on their 2d embedding distance.",
  "variables": {
    "query": {
      "name": "Query",
      "comment": "Selects documents to arrange into a 2d map.",
      "value": "clustering"
    },
    "maxDocuments": {
      "name": "Max documents",
      "comment": "Maximum number of documents to include in the 2d map.",
      "value": 5000
    },
    "maxLabels": {
      "name": "Max labels",
      "comment": "Maximum number of labels to put on the 2d map.",
      "value": 500
    }
  },
  "stages": {
    "documents": {
      "type": "documents:byQuery",
      "query": {
        "type": "query:string",
        "query": {
          "@var": "query"
        }
      },
      "limit": {
        "@var": "maxDocuments"
      }
    },
    "labels": {
      "type": "labels:fromDocuments",
      "maxLabels": {
        "type": "labelCount:fixed",
        "value": {
          "@var": "maxLabels"
        }
      }
    },
    "documentEmbeddings2d": {
      "type": "embedding2d:lv",
      "matrix": {
        "type": "matrix:knnVectorsSimilarity",
        "vectors": {
          "type": "vectors:precomputedDocumentEmbeddings"
        }
      }
    },
    "labelEmbeddings2d": {
      "type": "embedding2d:lvOverlay",
      "matrix": {
        "type": "matrix:keywordLabelDocumentSimilarity",
        "labels": {
          "type": "labels:reference",
          "use": "labels"
        }
      },
      "embedding2d": {
        "type": "embedding2d:reference",
        "use": "documentEmbeddings2d"
      }
    },
    "documentClusters": {
      "type": "clusters:ap",
      "matrix": {
        "type": "matrix:knn2dDistanceSimilarity",
        "embedding2d": {
          "type": "embedding2d:reference",
          "use": "documentEmbeddings2d"
        },
        "maxNearestPoints": 96
      },
      "softening": 0.2,
      "inputPreference": -10000
    },
    "documentClusterLabels": {
      "type": "labelClusters:documentClusterLabels"
    }
  },
  "output": {
    "stages": [
      "documents",
      "documentEmbeddings2d",
      "documentClusters",
      "documentClusterLabels",
      "labels",
      "labelEmbeddings2d"
    ]
  },
  "tags": [
    "2D Embeddings"
  ]
}

lingo4g:result:stageName:labels

Name of the stage that provides the labels to display, labels by default.

lingo4g:result:stageName:labelEmbeddings2d

Name of the stage that provides the 2d embeddings of the labels, labelEmbeddings2d by default.

lingo4g:result:stageName:documentEmbeddings2d

Name of the stage that provides the 2d embeddings of the documents to display, documentEmbeddings2d by default.

lingo4g:result:stageName:documentClusters

Name of the stage that provides the clusters of documents to display, documentClusters by default.

lingo4g:result:stageName:documentClusterLabels

Name of the stage that provides the labels of the document clusters, documentClusterLabels by default.

lingo4g:baseMarkerSize

A base value used to compute the size of document markers, 5.0 by default.

The size of markers created by this plugin is proportional to lingo4g:baseMarkerSize. To make the markers smaller or larger, decrease or increase this property.

lingo4g:customize:layers

A function called before the layers created by this plugin are displayed in dotAtlas. You can use this callback to customize the properties or data points of the automatically created layers.

The provided value must be a function with the following signature:

const customizeLayers = function (layersMap, context) { };

The function will be called with the following parameters:

layersMap

A map of layers ready to be displayed.

The map will contain the following keys: elevations, markers, labels. Under each key there is the layer object representing the specific layer. Use the Elevation layer API, Marker layer API and Labels layer API, respectively, to tune the layers as required.

result

The lingo4g:result for which the layers were created.

Below is an example customization function that displays the labels in italic, derives marker color from point elevation and "sharpens" elevations.

import { DotAtlas, Lingo4GDocumentMap } from "@carrotsearch/dotatlas";


const dotAtlas = DotAtlas.with(Lingo4GDocumentMap).embed({
  "element": document.querySelector("#visualization"),
  "lingo4g:result": { /* Lingo4G result here */ },
  "lingo4g:customize:layers": ({ markers, elevations, labels }, result) => {
    labels.set("labelFontStyle", "italic");
    elevations.get("points").forEach(p => p.elevation = Math.pow(p.elevation, 2))
    markers.get("points").forEach(p => {
      p.color = `rgb(${(255 * p.elevation).toFixed(0)}, 0, 0)`;
    });
  }
});

lingo4g:layer:elevations

The elevations layer object created by this plugin.

You can use this property to change the visual properties of the layer, register event listeners or call layer-specific methods.

const elevations = dotAtlas.get("lingo4g:layer:elevations");
elevations.set("lightAzimuth", Math.PI);

lingo4g:layer:markers

The markers layer object created by this plugin. Use this property to get and set properties of the markers layer.

lingo4g:layer:labels

The labels layer object created by this plugin. Use this property to get and set properties of the labels layer.