{"componentChunkName":"component---node-modules-carrotsearch-gatsby-theme-apidocs-templates-documentation-page-js","path":"/dictionaries/","result":{"data":{"html":{"frontmatter":{"id":"dictionaries","title":"Dictionaries"},"html":"
\n

Dictionaries

\n\n

\n Lingo3G uses dictionaries to improve the quality of clustering for a\n specific language. This section introduces the available dictionaries and\n the common syntax of dictionary entries.\n

\n\n
\n

\n Types of dictionaries\n

\n\n

Lingo3G supports the following dictionaries:

\n\n
\n
Label dictionary
\n
\n

\n Controls the labels Lingo3G chooses to describe clusters. You can use\n the label dictionary to remove certain labels, such as domain-common\n phrases or abusive language. You can also use the label dictionary to\n promote other labels, such as product or brand names.\n

\n
\n\n
Synonym dictionary
\n
\n

\n Defines words and phrases that represent the same concept, such as\n photo and picture to let Lingo3G know to put\n documents containing synonymous words or phrases in the same cluster.\n

\n
\n\n
Tag dictionary
\n
\n

\n Categorizes individual words based on their grammatical or semantic\n function, such as preposition, verb or proper name. You can reference\n the categories in the label dictionary to, for example, remove labels\n ending in a preposition, such as information about.\n

\n
\n
\n
\n\n
\n

\n Scopes of dictionaries\n

\n\n

\n Lingo3G applies dictionaries of all types in two complementary scopes:\n

\n\n
\n
Global
\n
\n

\n Global dictionaries apply to all clustering requests. Lingo3G ships\n with basic global dictionaries suitable for most types of documents.\n

\n\n

\n In typical use cases, the global dictionaries are static –\n Lingo3G loads them once on\n startup and reuses for all clustering requests.\n

\n
\n\n
Per-request
\n
\n

\n Per-request dictionaries apply for a specific clustering request in\n addition to the global dictionaries. You can use per-request\n dictionaries to extend global dictionaries with temporary entries\n provided by end-end users at runtime.\n

\n\n

\n Lingo3G supports per-request dictionaries both in the\n REST API\n and in the\n Java API.\n

\n
\n
\n
\n\n
\n

\n Location of dictionary files\n

\n\n
\n

\n Document Clustering Server\n

\n\n

\n The Document Clustering Server (REST API) reads dictionary files from\n the web/service/resources directory in the server's\n distribution directory. After you edit some of the files, restart the\n Document Clustering Server for the changes to take effect.\n

\n
\n\n
\n

\n Java API\n

\n\n

\n By default, Lingo3G Java API tries to read dictionaries from the JAR\n file from which the supplier of its corresponding\n LanguageComponents implementation comes from. The following\n table lists the locations of dictionary files for specific languages,\n including the name of the JAR file and path within the JAR.\n

\n\n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
LanguageJAR fileJAR path
English\n lingo3g-2.0.0.jar\n /
Dutch\n lingo3g-lang-dutch-2.0.0.jar\n /
Polish\n lingo3g-lang-polish-2.0.0.jar\n /
\n Danish, Finnish, French, German, Hungarian, Italian, Norwegian,\n Portuguese, Romanian, Russian, Spanish, Swedish, Turkish\n \n lingo3g-lang-carrot2-2.0.0.jar\n /
\n Arabic, Armenian, Bulgarian, Croatian, Czech, Estonian,\n Galician, Greek, Hindi, Indonesian, Irish, Latvian, Lithuanian,\n Thai\n \n lingo3g-lang-lucene-2.0.0.jar\n /
Chinese simplified, Chinese traditional, Japanese, Korean\n lingo3g-lang-lucene-cjk-2.0.0.jar\n /
\n
\n

\n Location of dictionary files in the standard Lingo3G distribution.\n

\n
\n
\n\n
\n Heads up, don't edit dictionaries inside JARs!\n\n

\n If you use Lingo3G Java API, don't modify the default dictionaries\n in-place. Instead, copy the relevant dictionaries to your\n application-specific location and\n load them from that location.\n

\n
\n
\n
\n\n
\n

\n Label matchers\n

\n\n

\n Label and synonym dictionaries use a common syntax based on\n label matcher entries. A matcher entry describes how\n Lingo3G decides whether to apply the dictionary-specific action to the\n specific word or phrase. For label dictionaries the action is removing or\n promoting the matching label. For synonym dictionaries, the action is\n including the matching label in the synonym set.\n

\n\n

The following excerpt shows the available label matchers.

\n\n
\n
\n 
{\n  \"exact\": [\n    \"Literal case-sensitive match\"\n  ],\n  \"glob\": [\n    \"starts with phrase *\",\n    \"* contains words *\"\n  ],\n  \"regexp\": [\n    \"(?).+BadLabel.+\",\n    \"(?)^[0-9]\\\\s*.*\"\n  ]\n}
\n
\n
\n

An example of three different label matcher types.

\n
\n
\n\n

\n\n

\n The exact, glob and\n regexp properties are optional and can contain an array of\n string entries for the specific matcher types, described in the following\n sections. If a word or a label matches any matcher of any type,\n Lingo3G applies the dictionary-specific action to the matching label.\n

\n\n
\n

\n Exact matcher\n

\n\n

\n Exact matchers require exact, case-sensitive equality\n between the word or phrase and the dictionary entry. Exact matcher\n entries are fast to parse and very fast to apply during clustering.\n

\n\n
\n
\n 
{\n  \"exact\": [\n    \"DevOps\",\n    \"Windows 2000\"\n  ]\n}
\n
\n
\n

\n An example label dictionary with two\n exact matcher entries.\n

\n
\n
\n\n

\n The above label dictionary definitions match labels\n DevOps and Windows 2000, but does not match\n Devops or Windows 2000 machine.\n

\n\n

\n For case-insensitive matching, use\n glob matchers (preferably) or\n case-insensitive\n regular expression matchers.\n

\n
\n\n
\n

\n Glob matcher\n

\n\n

\n Glob matcher allows simple word-based wildcard matching. The primary use\n case of the glob matcher is case-insensitive matching of literal\n phrases, as well as \"begins with…\", \"ends with…\" or\n \"contains…\" types of expressions. Glob matcher entries are fast\n to parse and very fast to apply.\n

\n\n
\n
\n 
{\n  \"glob\": [\n    \"more information\",\n    \"more information *\",\n    \"* about *\",\n    \"big ?\",\n    \"+ apple\"\n  ]\n}
\n
\n
\n

\n An example label dictionary with glob matcher entries.\n

\n
\n
\n\n
\n

\n Matching rules\n

\n\n
    \n
  • \n

    \n Each entry must consist of one or more space-separated tokens.\n

    \n
    • \n\n
  • \n

    \n A token is a sequence of arbitrary characters, such as words,\n numbers, identifiers.\n

    \n
    • \n\n
  • \n

    \n Matching of unquoted tokens is case-, accent- and\n grammatical-form-insensitive.\n

    \n
    • \n\n
  • \n

    \n Matching of quoted tokens is literal: case- and\n grammatical-form-sensitive.\n

    \n\n

    \n For example, \"Rating***\" matches only the\n Rating*** string, exactly comparing the case, grammatical\n form and occurrences of special characters. Glob matcher allows\n the * character in quoted tokens and matches them\n literally, not as a wildcard.\n

    \n
    • \n\n
  • \n

    \n To include quote characters in the token, escape them with the\n \\ character, for example:\n \\\"information\\\".\n

    \n
    • \n\n
  • \n

    \n Glob matcher recognizes the following wildcard-matching tokens:\n

    \n\n
      \n
    • \n

      ? matches exactly one (any) word.

      \n
      • \n
    • \n

      * matches zero or more words.

      \n
      • \n
    • \n

      \n + matches one or more words. This token is\n functionally equivalent to: ? *.\n

      \n
      • \n
    \n\n

    \n The * and + wildcards are\n possessive in the regular expression matching sense: they\n match the maximum sequence of tokens until the next token\n in the pattern. These wildcards will be suitable in most label\n matching scenarios.\n In rare cases, you may need to use the reluctant wildcards.\n

    \n
    • \n\n
  • \n

    \n Glob matcher recognizes the following reluctant wildcard-matching\n tokens:\n

    \n\n
      \n
    • \n

      *? matches zero or more words (reluctant).

      \n
      • \n\n
    • \n

      \n +? matches one or more words (reluctant). This\n token is functionally equivalent to: ? *?.\n

      \n
      • \n
    \n\n

    \n The reluctant wildcards match the minimal sequence of\n tokens until the next token in the pattern.\n

    \n
    • \n\n
  • \n

    Glob matcher recognizes the following word category tokens:

    \n\n
      \n
    • \n {fnc}: matches a function word (about,\n have)\n
      • \n
    • \n {verb}: matches a verb (have,\n allows)\n
      • \n
    • \n {noun}: matches a noun (website,\n test)\n
      • \n
    • {adj}: matches an adjective (cool)
      • \n
    • {adv}: matches an adverb (fully)
      • \n
    • \n {geo}: matches a geographical reference\n (London)\n
      • \n
    • \n {name}: matches a proper noun (John)\n
      • \n
    • \n numeric: matches numeric values (10.4,\n 2021, 14:30, 3/4)\n
      • \n
    \n\n

    \n The specific set of words each category token will match depends\n on the language in which you perform clustering and on the\n contents of the\n tag dictionaries. Note that\n category-based matching may be\n incorrect for some inputs\n due to the simplicity of Lingo3G's word tagger.\n

    \n
    • \n\n
  • \n

    \n Glob matcher imposes the following restrictions on wildcard\n operators:\n

    \n\n
      \n
    • \n

      \n You cannot use wildcards (*, +) to\n express string prefixes or suffixes. For example,\n programm*, is not supported.\n

      \n
      • \n\n
    • \n

      Glob matcher does not support greedy operators.

      \n
      • \n
    \n
    • \n
\n\n
\n Heads up, normalization of unquoted tokens!\n\n

\n Glob matcher normalizes the case, accents and grammatical form of\n unquoted tokens before trying to match them against labels. For\n example, under default settings and the English language, the token\n naïve matches, among others, the following labels:\n naïve, naive, naïvely or\n NAIVELY.\n

\n\n

\n Glob matcher normalizes character case based on the\n ROOT Java locale, which performs language-neutral case\n conflation according to Unicode rules.\n

\n\n

\n Grammatical form and accent normalization depends on the related\n Lingo3G parameters:\n useHeuristicStemming,\n useBuiltInWordDatabaseForStemming\n and\n accentFolding. By default, both grammatical form and accent normalization is\n enabled.\n

\n
\n
\n\n
\n

\n Example entries\n

\n\n

\n The following table shows a number of example glob entries. The\n \"Non-matching strings\" column also has an explanation why there is no\n match.\n

\n\n

\n All examples assume that the processing language is English and that\n normalization of grammatical forms and accents is enabled, which is\n the Lingo3G default.\n

\n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n\n \n \n \n \n \n\n \n \n \n \n \n\n \n \n \n \n \n\n \n \n \n \n \n\n \n \n \n \n \n\n \n \n \n \n \n\n \n \n \n \n \n\n \n \n \n \n \n\n \n \n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n\n \n \n \n \n \n
EntryMatching stringsNon-matching strings
more information\n

More information

\n

MORE INFORMATION

\n

\n more informations\n \n informations is a different grammatical form of\n information.\n \n

\n 		\n

\n more information about\n \n Pattern does not contain wildcards, only 2-word strings can\n match.\n \n

\n

\n some more information\n \n Pattern does not contain wildcards, only 2-word strings can\n match.\n \n

\n
more information *\n

more information

\n

More information about

\n

More information about a

\n

more informations

\n

more informations about

\n 		\n

\n some more information\n \n Pattern does not have wildcards at the beginning, matching\n strings must start with more information.\n \n

\n
* information *\n

information

\n

more information

\n

information about

\n

a lot more information on

\n

informations

\n

more informations about

\n

some more informations

\n 		\n

\n more info\n \n info is not a different grammatical of\n information, so it does not match.\n \n

\n
naïve *\n

\n naive\n \n naive is an accent-normalized variant of\n naïve\n \n

\n

\n naïvely\n \n naïvely is a different grammatical form of\n naïve\n \n

\n

naive approach

\n

Naively

\n 		\n

\n naiive\n \n Accent normalization does not correct spelling errors.\n \n

\n
{fnc}\n

\n of\n

\n

\n for\n

\n 		\n

\n clustering\n \n clustering is not a function word\n \n

\n
{adj} {noun}\n

\n fast algorithm\n

\n 		\n

\n factorization algorithm\n \n factorization is not an adjective\n \n

\n
to {verb} a {noun}\n

\n to visit a hotel\n

\n 		\n

\n to plan a vacation\n \n This string should also match the rule. It doesn't due to\n the limitation of Lingo3G word categorization algorithm.\n Lingo3G categorizes all occurrences of plan as a\n noun, regardless of the context of the word. In this case,\n the context implies plan is a verb, but Lingo3G\n does not take this into account.\n \n

\n
+ information\n

too much information

\n

more information

\n 		\n

\n information\n \n + requires at least one word before\n information.\n \n

\n

\n more information about\n \n about is an extra word not covered by the pattern.\n \n

\n
\"Information\" *\n

Information

\n

Information about

\n

Information ABOUT

\n 		\n

\n information\n \n The Information token is case-sensitive, it does\n not match information.\n \n

\n

\n information about\n \n The Information token is case-sensitive, it does\n not match information.\n \n

\n

\n Informations about\n \n Normalization of grammatical forms does not apply to quoted\n tokens, so Informations does not match pattern\n token \"Information\".\n \n

\n
data ?\n

data mining

\n 		\n

\n data\n \n ? requires a word after data.\n \n

\n

\n data mining research\n \n ? matches one word, so research\n does not match\n \n

\n
\"Programm*\"\n

Programm*

\n 		\n

\n Programmer\n \n Programm* token is taken literally, it matches only\n Programm*.\n \n

\n

\n Programming\n \n Programm* token is taken literally, it matches only\n Programm*.\n \n

\n
\\\"information\\\"\n

\"information\"

\n

\n \"INFOrmation\"\n \n Escaped quotes are taken literally, so match is\n case-insensitive\n \n

\n 		\n

\n information\n \n Escaped quotes not found in the string being matched.\n \n

\n

\n \"information\n \n Escaped quotes not found in the string being matched.\n \n

\n
* protein protein *\n

This pattern will never match any input.

\n\n

\n The reason for this is that * makes a possessive\n match, that is matches the maximum number of words until the\n next token in the pattern. Therefore, the first occurrence of\n the protein token in the pattern will correspond\n to the last occurrence of that word in the input\n label, leaving no content to match the second occurrence of\n protein in the pattern. As a result, there is no\n such sequence that can ever match this a pattern.\n

\n\n

\n To match labels with a doubled occurrence of some word, use\n the\n reluctant variant\n of the wildcard.\n

\n
*? protein protein *\n

protein protein

\n

selective protein protein interaction

\n

protein protein protein

\n 		\n

\n protein\n \n Two occurrences of protein on input are required.\n \n

\n

\n selective protein-protein interaction\n \n The \"protein-protein\" string counts as one token and it\n therefore does not match the two-token\n protein protein part of the pattern.\n \n

\n
programm*\n Illegal pattern, combinations of the * wildcard and\n other characters are not supported.\n
\"informationIllegal pattern, unbalanced double quotes.
*\n Illegal pattern, there must be at least one non-wildcard token.\n
\n
\n
\n\n
\n

\n Regular expression matcher\n

\n\n

\n The regular expression matcher checks words or labels against a list of\n regular expressions you provide.\n

\n\n
\n
\n 
{\n  \"regexp\": [\n    \"Windows 9[58]\",\n    \"(?)^[0-9]\\\\s*.*\"\n  ]\n}
\n
\n
\n

\n An example label dictionary with two\n regexp matcher entries.\n

\n
\n
\n\n

\n The regular expressions must follow\n Java syntax. If any fragment of a label matches any regular expression in the\n dictionary, Lingo3G will apply the dictionary-specific action to the\n label.\n

\n\n
\n Heads up, performance impact!\n\n

\n Regular expression-based matching is a powerful mechanism, but it can\n also result in a dramatic decrease of clustering performance.\n Therefore, use it only when a similar effect cannot be achieved by a\n reasonable number of\n exact\n and glob matching\n entries.\n

\n
\n
\n
\n
","tableOfContents":[{"heading":"Types of dictionaries","anchor":"types-of-dictionaries"},{"heading":"Scopes of dictionaries","anchor":"scopes-of-dictionaries"},{"heading":"Location of dictionary files","anchor":"location-of-dictionary-files","sections":[{"heading":"Document Clustering Server","anchor":"location-of-dictionary-files:document-clustering-server"},{"heading":"Java API","anchor":"location-of-dictionary-files:java-api"}]},{"heading":"Label matchers","anchor":"label-matchers","sections":[{"heading":"Exact matcher","anchor":"label-matchers:exact"},{"heading":"Glob matcher","anchor":"label-matchers:glob","sections":[{"heading":"Matching rules","anchor":"label-matchers:glob:rules"},{"heading":"Example entries","anchor":"label-matchers:glob:examples"}]},{"heading":"Regular expression matcher","anchor":"label-matchers:regexp"}]}]}},"pageContext":{"slug":"/dictionaries/"}},"staticQueryHashes":["1164750013"]}