{"componentChunkName":"component---node-modules-carrotsearch-gatsby-theme-apidocs-templates-documentation-page-js","path":"/tag-dictionary/","result":{"data":{"html":{"frontmatter":{"id":"tag-dictionary","title":"Tag dictionaries"},"html":"
\n

Tag dictionaries

\n\n

\n Tag dictionaries categorize individual words based on their grammatical or\n semantic function, such as preposition, verb or proper name. You can\n reference the categories in the\n label dictionary to, for example, remove\n labels ending in a preposition, such as information about.\n

\n\n
\n

\n Built-in tag dictionaries\n

\n\n

\n Lingo3G comes with built-in tag dictionaries for several languages. These\n dictionaries are precompiled into space- and lookup-efficient data\n structures, so you cannot modify them. However, you can use the\n custom tag dictionary to override the built-in\n category for specific words.\n

\n\n

\n To turn off the built-in tag dictionaries, set the\n \n useBuiltInWordDatabaseForLabelFiltering\n parameter to false.\n

\n
\n\n
\n

\n Custom tag dictionaries\n

\n\n

\n Lingo3G\n reads the tag\n dictionary from a JSON file named\n language.tags.json, where language is the\n language name. The tag dictionary must be an array of objects with the\n following structure:\n

\n\n
\n
\n 
[\n  // zero or more includes.\n  {\n    \"include\": \"relative-path-to-include.json\",\n    \"required\": true\n  },\n\n  // one or more tags, with word entries.\n  {\n    \"comment\": \"Optional comment.\",\n    \"tag\": \"[tag]\",\n    \"words\": [\n      \"word1\",\n      \"word2\"\n    ]\n  }\n]
\n
\n\n
\n

The structure of a Lingo3G tag dictionary.

\n
\n
\n\n

Each object in the array must be of one of the following types:

\n\n
\n
include
\n
\n

\n Includes all entries from another tag dictionary into this dictionary.\n You can use includes to split a large tag dictionary into smaller ones\n or extract a common part of a set of tag dictionaries.\n

\n\n
\n 
{ \"include\": \"name.json\", \"required\": true }
\n
\n\n
\n
include
\n
\n

\n (required) Path to the tag dictionary to include, relative to the\n location of this dictionary.\n

\n
\n\n
required
\n
\n

\n If true and the file to include does not exist,\n Lingo3G will trigger an error.\n

\n\n

\n If false, Lingo3G will ignore a non-existing include\n file.\n

\n
\n
\n
\n\n
word category set
\n
\n

Assigns a category tag to a set of words:

\n\n
\n 
{ \"tag\": \"...\", \"words\": [ \"a\", \"b\", ... ] }
\n
\n\n
\n
tag
\n
\n

\n The category tag or tags to assign for the words listed in the\n words array.\n

\n\n

\n Use one or more comma-separated categories from the following\n list:\n

\n\n
    \n
  • \n fnc: function word (about, have)\n
    • \n
  • verb: verb (have, allows)
    • \n
  • noun: noun (website, test)
    • \n
  • adj: adjective (cool)
    • \n
  • adv: adverb (fully)
    • \n
  • \n geo: geographical reference (London)\n
    • \n
  • name: proper noun (John)
    • \n
  • \n numeric: numeric values (10.4,\n 2021, 14:30, 3/4)\n
    • \n
  • \n sep: phrase separator, such as e.g. or\n ie. Lingo3G removes phrase separators from processing\n and therefore will not allow them to appear in cluster labels at\n all.\n
    • \n
\n
\n\n
words
\n
\n

The list of words to which to apply the tag.

\n\n

\n Lingo3G matches the words against the input text in a\n case-insensitive way and does not apply any further processing,\n such as grammatical form normalization. Therefore, include all\n alternative grammatical and spelling variants of a word if\n necessary, for example: naïve, naive,\n naïvely, naively.\n

\n
\n
\n
\n
\n\n

A real-world dictionary can look like this:

\n\n
\n
\n 
[\n  { \"include\": \"legal-words.json\", \"required\": true },\n  {\n    \"tag\": \"fnc\",\n    \"words\": [ \"a\", \"about\" ]\n  },\n  {\n    \"tag\": \"fnc,verb\",\n    \"words\": [ \"have\" ]\n  },\n  {\n    \"tag\": \"verb\",\n    \"words\": [ \"go\", \"allow\" ]\n  },\n  {\n    \"tag\": \"numeric\",\n    \"words\": [ \"thousand\" ]\n  }\n]
\n
\n\n
\n

Tag dictionary example.

\n
\n
\n\n
\n Tip: how Lingo3G uses word categories by default.\n\n

\n The default label dictionary uses the part of speech information in the\n following way:\n

\n\n
    \n
  • \n removes labels being, starting or ending in a function word or verb\n
    • \n
  • removes labels being or ending in an adjective or adverb
    • \n
  • removes labels being or beginning in a number
    • \n
  • \n slightly boosts labels containing proper nouns or geographic terms\n
    • \n
\n\n

\n You can customize this default behavior by editing the\n common-rules.labels.json dictionary.\n

\n
\n
\n
","tableOfContents":[{"heading":"Built-in tag dictionaries","anchor":"built-in"},{"heading":"Custom tag dictionaries","anchor":"custom"}]}},"pageContext":{"slug":"/tag-dictionary/"}},"staticQueryHashes":["1164750013"]}