labelFilter

labelFilter:* accepts or rejects an individual label based on the label's text. You can use label filters to customize the processing inside label collectors as well as label fetching stages, such as labels:byPrefix or labels:embeddingNearestNeighbors.

You can use the following label filters in your analysis requests:

labelFilter:acceptAll: Accepts all labels.
labelFilter:acceptLabels: Accepts labels appearing in the set of labels provided by the referenced labels:* source.
labelFilter:autoStopLabels: Rejects any meaningless labels Lingo4G automatically identified during indexing.
labelFilter:characterCount: Accepts or rejects labels based on the number of characters they have.
labelFilter:complement: Rejects labels that are accepted by another filter (inverts the result of another label filter).
labelFilter:composite: Accepts labels if they are accepted by all or any of the label filters you provide.
labelFilter:dictionary: Accepts labels matched by the label dictionary you provide.
labelFilter:hasEmbedding: Accepts labels that have a multidimensional embedding vector available.
labelFilter:rejectLabels: Rejects label appearing on the closed list you provide.
labelFilter:rejectTokens: Rejects labels that contain words and prases present in the labels you provide.
labelFilter:surface: Accepts labels based on their exact appearance (case-sensitive characters). You can configure this filter to remove all-uppercase, capitalized or acronym-like labels.
labelFilter:switch: Enables or disables the label filter you provide.
labelFilter:tokenCount: Accepts labels based on the number of words in the label.

labelFilter:reference: References a labelFilter:* component defined in the request or in the project's default components.

labelFilter:acceptAll

Accepts all labels.

{
  "type": "labelFilter:acceptAll"
}

labelFilter:acceptLabels

Accepts labels appearing in the set of labels provided by the referenced labels:* source.

{
  "type": "labelFilter:acceptLabels",
  "labels": {
    "type": "labels:reference",
    "auto": true
  }
}

An example request below retrieves a list of top labels that appear in documents matching the query magnetic field, but limited to only those labels that also appear in documents that match the query pulsar. In effect, the result of this request is an intersection of label sets between these two document queries.

{
  "name": "Retrieve document labels from documents matching 'magnetic field' but only appearing in documents matching 'pulsar'.",
  "stages": {
    "labels": {
      "type": "labels:fromDocuments",
      "maxLabels": {
        "type": "labelCount:fixed",
        "value": 10
      },
      "documents": {
        "type": "documents:byQuery",
        "query": {
          "type": "query:string",
          "query": "\"magnetic field\""
        },
        "limit": "unlimited"
      },
      "labelAggregator": {
        "type": "labelAggregator:topWeight",
        "labelCollector": {
          "type": "labelCollector:topFromFeatureFields",
          "labelFilter": {
            "type": "labelFilter:acceptLabels",
            "labels": {
              "type": "labels:fromDocuments",
              "documents": {
                "type": "documents:byQuery",
                "query": {
                  "type": "query:string",
                  "query": "pulsar"
                },
                "limit": "unlimited"
              }
            }
          }
        }
      }
    }
  }
}

The result of the above request on the reference Arxiv project is shown below. Compare it to a similar request using the rejectLabels component.

{
  "result" : {
    "labels" : {
      "labels" : [
        {
          "label" : "quantum",
          "weight" : 6804.0
        },
        {
          "label" : "electron",
          "weight" : 4625.0
        },
        {
          "label" : "star",
          "weight" : 4096.0
        },
        {
          "label" : "solar",
          "weight" : 3070.0
        },
        {
          "label" : "plasma",
          "weight" : 2651.0
        },
        {
          "label" : "Hall",
          "weight" : 2350.0
        },
        {
          "label" : "jet",
          "weight" : 2252.0
        },
        {
          "label" : "equation",
          "weight" : 2044.0
        },
        {
          "label" : "magnetization",
          "weight" : 1889.0
        },
        {
          "label" : "superconducting",
          "weight" : 1889.0
        }
      ]
    }
  }
}

labels

Type

labels

Default

{
  "type": "labels:reference",
  "auto": true
}

Required

A reference source of labels to accept.

labelFilter:autoStopLabels

Rejects any meaningless labels Lingo4G automatically identified during indexing.

{
  "type": "labelFilter:autoStopLabels",
  "minCoverage": 0.4,
  "removalStrength": 0.35
}

minCoverage

Type

number

Default

0.4

Constraints

value >= 0 and value <= 1

Required

Reject labels that appear in the stop labels list and have the coverage lower than minCoverage.

removalStrength

Type

number

Default

0.35

Constraints

value >= 0 and value <= 1

Required

Reject labels that appear in the stop labels list and have the score lower than removalStrength.

labelFilter:characterCount

Accepts or rejects labels based on the number of characters they have.

{
  "type": "labelFilter:characterCount",
  "minCharacters": 4,
  "minCharactersAveragePerToken": 2.9
}

minCharacters

Type

number

Default

Constraints

value >= 0

Required

Rejects labels that have fewer than minCharacters Java characters (Unicode surrogate pairs count as two characters).

minCharactersAveragePerToken

Type

number

Default

2.9

Constraints

value >= 0

Required

Rejects labels where: (label characters) / (word count) is smaller than minCharactersAveragePerToken. This option can be used to prune automatically discovered labels with very short tokens (for example repeated MathML expressions).

labelFilter:complement

Rejects labels that are accepted by another filter (inverts the result of another label filter).

{
  "type": "labelFilter:complement",
  "labelFilter": null
}

labelFilter

Type

labelFilter

Default

null

Required

yes

The label filter whose results should be negated.

labelFilter:composite

Accepts labels if they are accepted by all or any of the label filters you provide.

{
  "type": "labelFilter:composite",
  "labelFilters": {},
  "operator": "AND"
}

The keys of nested composite label filters are used for information purposes only. For example, this request retrieves labels from documents matching the query magnetic field, but limits labels to multi-term phrases avoiding the use of any query terms.

{
  "name": "Retrieve document labels from documents matching 'magnetic field' with composite filtering criteria.",
  "stages": {
    "labels": {
      "type": "labels:fromDocuments",
      "maxLabels": {
        "type": "labelCount:fixed",
        "value": 10
      },
      "documents": {
        "type": "documents:byQuery",
        "query": {
          "type": "query:string",
          "query": "\"magnetic field\""
        },
        "limit": "unlimited"
      },
      "labelAggregator": {
        "type": "labelAggregator:topWeight",
        "labelCollector": {
          "type": "labelCollector:topFromFeatureFields",
          "labelFilter": {
            "type": "labelFilter:composite",
            "labelFilters": {
              "phrases-only": {
                "type": "labelFilter:tokenCount",
                "minTokens": 2,
                "maxTokens": 5
              },
              "acronym-filters":  {
                "type": "labelFilter:surface",
                "removeCapitalized": true,
                "removeUppercase": true,
                "removeAcronyms": true
              },
              "avoid-query-terms": {
                "type": "labelFilter:dictionary",
                "exclude": [
                  {
                    "type": "dictionary:glob",
                    "entries": [
                      "* magnetic *",
                      "* field *"
                    ]
                  }
                ]
              }
            }
          }
        }
      }
    }
  }
}

The result of the above request on the reference Arxiv project is shown below.

{
  "result" : {
    "labels" : {
      "labels" : [
        {
          "label" : "neutron star",
          "weight" : 1259.0
        },
        {
          "label" : "black hole",
          "weight" : 1045.0
        },
        {
          "label" : "quantum dot",
          "weight" : 934.0
        },
        {
          "label" : "cosmic rays",
          "weight" : 839.0
        },
        {
          "label" : "phase transition",
          "weight" : 709.0
        },
        {
          "label" : "ground state",
          "weight" : 679.0
        },
        {
          "label" : "solar wind",
          "weight" : 582.0
        },
        {
          "label" : "phase diagram",
          "weight" : 532.0
        },
        {
          "label" : "active region",
          "weight" : 489.0
        },
        {
          "label" : "angular momentum",
          "weight" : 420.0
        }
      ]
    }
  }
}

labelFilters

Type

object of labelFilter

Default

{}

Required

A set of other named labelFilter:* components.

operator

Type

string

Default

"AND"

Constraints

one of [OR, AND]

Required

Declares the way label filters from filters are combined. The operator property supports the following values:

OR: Creates a disjunction composite filter. A label is filtered if it occurs in any nested filters.
AND: Creates a conjunction composite filter. A label is filtered if it occurs in all nested filters.

labelFilter:dictionary

Rejects labels that appear in any of the referenced dictionaries.

{
  "type": "labelFilter:dictionary",
  "exclude": []
}

exclude

Type: array of dictionary
Default: []
Required: no

An array of one or more dictionary:* components.

labelFilter:hasEmbedding

Accepts only those labels that have a multidimensional label embedding vector available.

{
  "type": "labelFilter:hasEmbedding"
}

labelFilter:rejectLabels

Rejects labels appearing in the set of labels provided by the referenced labels:* source.

{
  "type": "labelFilter:rejectLabels",
  "labels": {
    "type": "labels:reference",
    "auto": true
  }
}

The example request below retrieves a list of top labels that appear in documents matching the query magnetic field, removing any labels that also appear in documents that match the query pulsar. In effect, the result of this request is an intersection of label sets between these two document queries.

{
  "name": "Retrieve document labels from documents matching 'magnetic field' but only appearing in documents matching 'pulsar'.",
  "stages": {
    "labels": {
      "type": "labels:fromDocuments",
      "maxLabels": {
        "type": "labelCount:fixed",
        "value": 10
      },
      "documents": {
        "type": "documents:byQuery",
        "query": {
          "type": "query:string",
          "query": "\"magnetic field\""
        },
        "limit": "unlimited"
      },
      "labelAggregator": {
        "type": "labelAggregator:topWeight",
        "labelCollector": {
          "type": "labelCollector:topFromFeatureFields",
          "labelFilter": {
            "type": "labelFilter:rejectLabels",
            "labels": {
              "type": "labels:fromDocuments",
              "documents": {
                "type": "documents:byQuery",
                "query": {
                  "type": "query:string",
                  "query": "pulsar"
                },
                "limit": "unlimited"
              }
            }
          }
        }
      }
    }
  }
}

The result of the above request on the reference Arxiv project is shown below. Compare it to a similar request using the acceptLabels component.

{
  "result" : {
    "labels" : {
      "labels" : [
        {
          "label" : "magnetic",
          "weight" : 22570.0
        },
        {
          "label" : "field",
          "weight" : 21930.0
        },
        {
          "label" : "magnetic field",
          "weight" : 15999.0
        },
        {
          "label" : "model",
          "weight" : 5156.0
        },
        {
          "label" : "spin",
          "weight" : 5057.0
        },
        {
          "label" : "state",
          "weight" : 4698.0
        },
        {
          "label" : "energy",
          "weight" : 3598.0
        },
        {
          "label" : "effects",
          "weight" : 3571.0
        },
        {
          "label" : "phase",
          "weight" : 2832.0
        },
        {
          "label" : "temperature",
          "weight" : 2815.0
        }
      ]
    }
  }
}

labels

Type

labels

Default

{
  "type": "labels:reference",
  "auto": true
}

Required

A reference source of labels to reject.

labelFilter:rejectTokens

Rejects labels that contain words and prases present in the labels you provide.

{
  "type": "labelFilter:rejectTokens",
  "analyzer": "english",
  "featureExtractor": "",
  "labels": {
    "type": "labels:reference",
    "auto": true
  },
  "rejectContaining": "ANY_TOKEN",
  "splitCompounds": false
}

Compared to dictionary-based filtering, this filter makes it easier to filter out labels containing different surface forms of the same label. For example, when looking for labels semantically-similar to the AI label, you may want to exclude results that contain the AI token. Dictionary-based filtering may only be partially effective here because certain labels, such as Artificial Intelligence (AI) contain a different surface form ((AI)) of the token. The labelFilter:rejectTokens performs surface-form-insensitive filtering, so providing in the labels list the AI label will filter out labels containing any surface form, such as (AI) or ai.

analyzer

Type

string

Default

"english"

Required

The analyzer to use to tokenize the labels.

featureExtractor

Type: string
Default: <empty string>
Required: no

Feature extractor to use to process the labels.

labels

Type

labels

Default

{
  "type": "labels:reference",
  "auto": true
}

Required

The list of labels whose parts to filter out.

rejectContaining

Type

string

Default

"ANY_TOKEN"

Constraints

one of [ANY_TOKEN, ALL_TOKENS, EXACT_LABEL]

Required

Configures the filtering mode.

The rejectContaining property supports the following values:

ANY_TOKEN: Removes labels that contain any token (word) present in any of the input labels. This mode results in the strongest filtering.
ALL_TOKENS: Removes labels that contain all tokens (words) present in one (or more) of the input labels, in any order. This mode filters out fewer labels than the ANY_TOKEN mode.
EXACT_LABEL: Removes labels that contain a subsequence equal to any of the input labels. This mode filters out the fewest labels.

splitCompounds

Type

boolean

Default

false

Required

If true, Lingo4G splits dash- and slash-based compounds, such as AI-based or AI/ML, into separate tokens.

With compound splitting enabled, if the input labels contains the AI-based label, the ANY_TOKEN mode will reject any label containing the AI or based word, regardless of their surface form.

labelFilter:surface

Accepts labels based on conditions applying to their exact surface appearance (case-sensitive characters). You can configure this filter to remove all-uppercase, capitalized or acronym-like labels.

{
  "type": "labelFilter:surface",
  "removeAcronyms": false,
  "removeCapitalized": false,
  "removeUppercase": false
}

removeAcronyms

Type

boolean

Default

false

Required

If code, reject labels that appear to be acronyms (more than one letter, capitalized to non-capitalized letter count ratio >= 0.5).

removeCapitalized

Type

boolean

Default

false

Required

If code, reject labels that are capitalized (first letter in uppercase, remaining letters in lowercase).

removeUppercase

Type

boolean

Default

false

Required

If code, reject labels with all-uppercase letters.

labelFilter:switch

Enables or disables the label filter you provide.

{
  "type": "labelFilter:switch",
  "enabled": true,
  "labelFilter": null
}

You can use this filter to dynamically activate or deactivate any label filter based on the value of a boolean variable passed to the enabled property. Without the labelFilter:switch component, the only way to deactivate a filter would be to remove the filter from the request.

The following request shows how to control any label filter using a variable value. This requires three elements:

Defining a boolean variable, called enableLabelFiltering in our request.
Wrapping the filter to control with a labelFilter:switch filter.
Referencing the boolean variable in the enabled property of the labelFilter:switch filter.

{
  "name": "Enabling / disabling a label filter using a variable.",
  "variables": {
    "enableLabelFiltering": {
      "name": "Enable label filtering",
      "value": false
    }
  },
  "stages": {
    "documents": {
      "type": "documents:byQuery",
      "query": {
        "type": "query:string",
        "query": "clustering"
      },
      "limit": 1000
    },
    "labels": {
      "type": "labels:fromDocuments",
      "labelAggregator": {
        "type": "labelAggregator:topWeight",
        "labelCollector": {
          "type": "labelCollector:topFromFeatureFields",
          "labelFilter": {
            "type": "labelFilter:switch",
            "enabled": {
              "@var": "enableLabelFiltering"
            },
            "labelFilter": {
              "type": "labelFilter:tokenCount",
              "minTokens": 2,
              "maxTokens": 5
            }
          }
        }
      }
    }
  }
}

With the labelFilter:switch you can dynamically control label filters without changing the structure of the request. Additionally, boolean variables get represented as check boxes in the request variable editor.

enabled

Type

boolean

Default

true

Required

Enables or disables the label filter you provide.

If true, applies the labelFilter to all input labels. If false, does not apply any filtering and accepts all input labels.

labelFilter

Type

labelFilter

Default

null

Required

yes

The label filter to control.

labelFilter:tokenCount

Accepts labels based on the number of tokens (words) in the label.

{
  "type": "labelFilter:tokenCount",
  "maxTokens": 8,
  "minTokens": 3
}

maxTokens

Type

integer

Default

Constraints

value >= 0

Required

Reject labels that have more than maxTokens words.

minTokens

Type

integer

Default

Constraints

value >= 0

Required

Reject labels that have fewer than minTokens words.

Consumers of `labelFilter:*`

The following stages and components take labelFilter:* as input:

Stage or component	Property
`documents:rwmd`	`labelFilter`
`labelCollector:allFromContentFields`	`labelFilter`
`labelCollector:allFromFeatureFields`	`labelFilter`
`labelCollector:topEmbeddingNearestNeighbors`	`labelFilter`
`labelCollector:topFromFeatureFields`	`labelFilter`
`labelFilter:complement`	`labelFilter`
`labelFilter:composite`	`labelFilters`
`labelFilter:switch`	`labelFilter`
`labels:byPrefix`	`labelFilter`
`labels:embeddingNearestNeighbors`	`labelFilter`
`labels:filtered`	`labelFilter`
`labels:fromText`	`labelFilter`
`valueCollector:fromContentField`	`labelFilter`
`values:unique`	`labelFilter`
`vector:fromText`	`labelFilter`

API

Pages and sections

Content

labelFilter

labelFilter:acceptAll

labelFilter:acceptLabels

labels

labelFilter:autoStopLabels

minCoverage

removalStrength

labelFilter:characterCount

minCharacters

minCharactersAveragePerToken

labelFilter:complement

labelFilter

labelFilter:composite

labelFilters

operator

labelFilter:dictionary

exclude

labelFilter:hasEmbedding

labelFilter:rejectLabels

labels

labelFilter:rejectTokens

analyzer

featureExtractor

labels

rejectContaining

splitCompounds

labelFilter:surface

removeAcronyms

removeCapitalized

removeUppercase

labelFilter:switch

enabled

labelFilter

labelFilter:tokenCount

maxTokens

minTokens

Consumers of `labelFilter:*`

labelFilter

label​Filter:​accept​All

label​Filter:​accept​Labels

labels

label​Filter:​auto​Stop​Labels

min​Coverage

removal​Strength

label​Filter:​character​Count

min​Characters

min​Characters​Average​Per​Token

label​Filter:​complement

label​Filter

label​Filter:​composite

label​Filters

operator

label​Filter:​dictionary

exclude

label​Filter:​has​Embedding

label​Filter:​reject​Labels

labels

label​Filter:​reject​Tokens

analyzer

feature​Extractor

labels

reject​Containing

split​Compounds

label​Filter:​surface

remove​Acronyms

remove​Capitalized

remove​Uppercase

label​Filter:​switch

enabled

label​Filter

label​Filter:​token​Count

max​Tokens

min​Tokens

Consumers of label​Filter:​*

labelFilter:acceptAll

labelFilter:acceptLabels

labelFilter:autoStopLabels

minCoverage

removalStrength

labelFilter:characterCount

minCharacters

minCharactersAveragePerToken

labelFilter:complement

labelFilter

labelFilter:composite

labelFilters

labelFilter:dictionary

labelFilter:hasEmbedding

labelFilter:rejectLabels

labelFilter:rejectTokens

featureExtractor

rejectContaining

splitCompounds

labelFilter:surface

removeAcronyms

removeCapitalized

removeUppercase

labelFilter:switch

labelFilter

labelFilter:tokenCount

maxTokens

minTokens

Consumers of `labelFilter:*`