Learning embeddings

To build label and document embedding vectors, perform the following steps:

1. Index your documents if you haven't done so.

2. Tune embedding parameters to match the size of your project.

   If you are working with one of the example data sets, the embedding parameters already match the expected size of the index.

   If you are learning embeddings for your own data, perform the following steps.

   1. Find out what size is your index:

      l4g stats -p <project-descriptor-path>

      You should see output similar to:

      DOCUMENTS INDEX (commit 'data/commits/_2')
      
      Live documents     2.02M
      Deleted documents  39.82k
      Size on disk       2.14GB
      Segments           14
      Directory type     MMapDirectory

   2. Based on the Size on disk value, edit your project descriptor to apply the following embedding parameter changes.

      Size on disk	Embedding learning parameters
      < 5GB	No parameter changes needed
      5GB — 50GB	Use the following indexer.embedding.labels section in your project descriptor. { "model": { "vectorSize": 128 } }
      > 50GB	Use the following indexer.embedding.labels section in your project descriptor. { "model": { "vectorSize": 160 } }

3. Run embedding learning command:

   l4g learn-embeddings -p <project-descriptor-path>

   Leave the command running until you see a stable completion time estimate of the label embedding learning task:

   1/1 Embeddings > Learning label embeddings   [    :    :6k docs/s]   4% ~18m 57s

   If the estimate is unreasonably high (multiple hours or days), edit the project descriptor and add a hard timeout on the label embedding learning time:

   {
     "indexer": {
       "embedding": {
         "labels": {
           "model": {
             "timeout": "2h"
           }
         }
       }
     }
   }

   As a rule of thumb, a timeout equal to 1x–2x indexing time should yield embeddings of sufficient quality. See the FAQ section for more ways to lower the label embedding learning time.

   Once the learn-embeddings command completes successfully, you can use label and document embeddings during analysis, for example to compute embedding-based similarity matrices.

Updating embeddings

When you add documents to your Lingo4G index (using incremental indexing) and if this index contains document embeddings, Lingo4G will automatically add embedding vectors for any new documents.

To bring the embedding vectors into synchronization with the index, perform one of the following steps:

• If you performed feature reindexing to discover new labels, rebuild both label and document embeddings:

  l4g learn-embeddings -p <project-descriptor-path> --recompute-label-embeddings --recompute-document-embeddings

• If you added documents to the index using incremental indexing without the follow-up feature reindexing step, document embeddings are updated automatically.

  There is no point in rebuilding label embeddings in this case because the labels did not change.

Also note that updating embeddings after adding documents or reindexing labels is not mandatory. All analysis operations will still work, omitting documents and labels with empty embedding vectors. This situation is perfectly normal – the least frequent labels will never have their corresponding embedding vectors due to the insufficient data.

Limitations and caveats

The current Lingo4G's implementation of label and document embeddings has the following limitations you should be aware of.

FAQ