{"componentChunkName":"component---node-modules-carrotsearch-gatsby-theme-apidocs-templates-documentation-page-js","path":"/java-dictionaries/","result":{"data":{"html":{"frontmatter":{"id":"java-dictionaries","title":"Dictionaries"},"html":"
\n

Dictionaries

\n\n

\n Lingo3G uses dictionaries to improve the quality of clustering for a\n specific language. This article shows how to customize dictionaries in the\n Java API.\n

\n\n

\n Lingo3G delegates the management of dictionaries and other language\n resources to the LanguageComponent instance. You can use its\n methods to list the available languages, customize the location of\n dictionaries or even replace the Lingo3G's built-in language resources, such\n as stemmers. This chapter shows the basic use cases related to dictionary\n customizations.\n

\n\n
\n

\n Customizing dictionaries\n

\n\n

\n The simplest way to customize Lingo3G\n dictionaries is to copy the\n default dictionaries\n to your application-specific location, make the necessary changes and\n provide a custom ResourceLookup implementation when loading\n language resources. The following example loads English resources from a\n class-relative classpath location.\n

\n\n
\n
\n 
ResourceLookup resourceLookup = new ClassRelativeResourceLookup(getClass());\nLanguageComponents english =\n    LanguageComponents.loader()\n        // We restrict the loaded resources to just English.\n        .limitToLanguages(\"English\")\n        // Note how we chain the overridden resource lookup location with the\n        // provider's default resources. Otherwise we'd have to copy over\n        // all resources from their default locations.\n        .withResourceLookup(\n            prov ->\n                new ChainedResourceLookup(\n                    List.of(resourceLookup, prov.defaultResourceLookup())))\n        .load()\n        .language(\"English\");
\n
\n
\n Loading English language components from a class-relative classpath\n location.\n
\n
\n
\n\n
\n

\n Using ephemeral dictionaries\n

\n\n

\n You can provide extra ephemeral dictionary entries for a specific\n clustering request. Lingo3G applies these extra entries as an addition to\n the default dictionaries. For example, if the end-user wants to remove\n specific labels from the clustering result they are currently viewing,\n your software can add such labels to the ephemeral label dictionary and\n rerun the clustering.\n

\n\n

\n The dictionaries field of\n Lingo3GClusteringAlgorithm\n groups all ephemeral dictionaries. You can add entries to the label,\n synonym and word dictionaries.\n

\n\n
\n

\n Label dictionaries\n

\n\n

\n To add an ephemeral label dictionary, create a new instance of the\n LabelMatcher class and add entries to its\n glob, regex or exact fields. Then\n set the matcher instance on your\n Lingo3GClusteringAlgorithm instance.\n

\n\n

\n The following example adds two entries to the glob matcher of\n the label dictionary.\n

\n\n
\n
\n 
Lingo3GClusteringAlgorithm algorithm = new Lingo3GClusteringAlgorithm();\n\n// Create an ephemeral label filter that excludes two glob patterns.\nLabelMatcher dictionary = new LabelMatcher();\ndictionary.glob.set(\"* mining *\", \"* data-mining *\");\n\n// Configure Lingo3G to use the ephemeral filters.\nalgorithm.dictionaries.labels.set(List.of(dictionary));
\n
\n\n
\n

Adding entries to the ephemeral label dictionary in Java API.

\n
\n
\n\n

\n By default, LabelMatcher instances assign a zero weight to\n the labels they match, removing them from the result. You can set the\n weight field of the matcher to a value larger than 1 to\n promote the labels.\n

\n
\n\n
\n

\n Synonym dictionaries\n

\n\n

To add an ephemeral synonym dictionary, do the following:

\n\n
    \n
  1. \n

    \n For each set of synonymous labels, create a SynonymSet\n instance, providing the list of labels Lingo3G should treat as\n synonymous. Note that Lingo3G allows only glob-style label matchers\n in synonym definitions.\n

    \n\n

    \n Additionally, you can provide one label to represent all the\n synonymous labels in cluster labels.\n

    \n
    2. \n\n
  2. \n

    \n Set a list of synonym sets on the\n dictionaries.synonyms field of yourLingo3GClusteringAlgorithm\n instance.\n

    \n
    3. \n
\n\n

\n The following example adds one entry making the software and\n tools label synonymous and represented by\n Tools & Software for cluster labeling purposes.\n

\n\n
\n
\n 
Lingo3GClusteringAlgorithm algorithm = new Lingo3GClusteringAlgorithm();\n\n// Create an ephemeral synonym dictionary.\nList<String> synonyms = List.of(\"software\", \"tools\");\nSynonymSet synonymSet = new SynonymSet(\"Tools & Software\", synonyms);\n\n// Configure Lingo3G to use the extra synonyms.\nalgorithm.dictionaries.synonyms.set(List.of(synonymSet));
\n
\n\n
\n

Adding entries to the ephemeral synonym dictionary in Java API.

\n
\n
\n
\n\n
\n

\n Tag dictionaries\n

\n\n

\n To add an ephemeral tag dictionary, create one or more instances of the\n Tag class defining the list of words and the part-of-speech\n tag for those words. Then set the list of tags on instance on your\n Lingo3GClusteringAlgorithm instance.\n

\n\n

\n The following example adds the word whereas with the\n fnc tag.\n

\n\n
\n
\n 
Lingo3GClusteringAlgorithm algorithm = new Lingo3GClusteringAlgorithm();\n\n// Create an ephemeral tag dictionary.\nTag fncTag = new Tag(\"fnc\", List.of(\"whereas\"));\n\n// Configure Lingo3G to use the extra tags.\nalgorithm.dictionaries.tags.set(List.of(fncTag));
\n
\n\n
\n

\n Adding entries to the ephemeral word tag dictionary in Java API.\n

\n
\n
\n
\n
\n\n
\n

\n Listing supported languages\n

\n\n

The following code lists all languages supported by Lingo3G:

\n\n
\n
\n 
Lingo3GClusteringAlgorithm algorithm = new Lingo3GClusteringAlgorithm();\nSet<String> supported =\n    LanguageComponents.loader()\n        // We can load language resources only for the given algorithm(s)\n        // which also limits the returned set of supported languages.\n        .limitToAlgorithms(algorithm)\n        .load()\n        .languages();\nSystem.out.println(\"Lingo3G supports:\\n  \" + String.join(\", \", supported));
\n
\n\n
\n

Listing all supported languages in the Java API.

\n
\n
\n\n

It prints:

\n\n
\n 
Lingo3G supports:\n  Arabic, Armenian, Bulgarian, Chinese-Simplified, Chinese-Traditional, Croatian, Czech, Danish, Dutch, English, Estonian, Finnish, French, Galician, German, Greek, Hindi, Hungarian, Indonesian, Irish, Italian, Japanese, Korean, Latvian, Lithuanian, Norwegian, Polish, Portuguese, Romanian, Russian, Spanish, Swedish, Thai, Turkish
\n
\n\n

\n Note that the code uses the limitToAlgorithms method to limit\n the list to the languages Lingo3G supports. The unfiltered list contains\n all languages defined in the\n Carrot2\n framework; Lingo3G does not support some of those languages.\n

\n
\n
","tableOfContents":[{"heading":"Customizing dictionaries","anchor":"customizing-dictionaries"},{"heading":"Using ephemeral dictionaries","anchor":"ephemeral-dictionaries","sections":[{"heading":"Label dictionaries","anchor":"ephemeral-dictionaries:label-dictionaries"},{"heading":"Synonym dictionaries","anchor":"ephemeral-dictionaries:synonym-dictionaries"},{"heading":"Tag dictionaries","anchor":"ephemeral-dictionaries:tag-dictionaries"}]},{"heading":"Listing supported languages","anchor":"listing-supported-languages"}]}},"pageContext":{"slug":"/java-dictionaries/"}},"staticQueryHashes":["1164750013"]}