{"componentChunkName":"component---node-modules-carrotsearch-gatsby-theme-apidocs-templates-documentation-page-js","path":"/release-notes-2.0.x-migration-guide/","result":{"data":{"html":{"frontmatter":{"id":"release-notes-2.0.x-migration-guide","title":"2.x migration guide"},"html":"
\n

2.x migration guide

\n\n

\n Version 2.0.0 modernizes the whole Lingo3G ecosystem, including end-user\n applications and the programming APIs. This article discusses the rationale\n behind the changes and provides migration procedures for the affected\n elements of Lingo3G.\n

\n\n
\n

\n Overview of changes\n

\n\n

\n Lingo3G 2.0.0 comes with the first major API refactoring in several years.\n At first, we tried to avoid incompatible API changes to eliminate the need\n for manual code updates and data format conversions. Eventually, it became\n obvious that the accumulated technical debt and technological progress\n made sustaining the old architecture impractical and it was time to change\n it.\n

\n\n

\n The primary motivation for changes was to simplify the programming\n interfaces to make your Lingo3G integration code cleaner and easier to\n maintain. We also wanted to modernize the underlying technology stack, so\n that Lingo3G's technical debt would not carry over to your systems.\n

\n\n

The following list summarizes the changes and their driving factors.

\n\n \n
\n\n
\n

\n Migration guide\n

\n\n
\n

\n REST API (DCS)\n

\n\n

\n Lingo3G 2.x REST API is not backward compatible with the 1.x API –\n you will need to update your Lingo3G integration code to make it work\n with version 2.x. This section lists the key elements to look for in\n your existing code.\n

\n\n
    \n
  1. \n

    \n First, have a look at the new REST API.\n

    \n\n

    \n Before you start converting any existing code, have a look at the\n basics of the new API.\n

    \n
    2. \n\n
  2. \n

    \n Identify suite-dcs.xml modifications.\n

    \n\n

    \n Previously, Lingo3G used the suite-dcs.xml component\n suite definition to declare document sources, algorithms and\n parameter changes. If you edited that file to change parameters or\n add custom document sources, identify these changes first.\n

    \n\n

    \n The new REST API does not support document sources, so if your code\n used built-in or custom document sources, you need to modify the\n code to fetch documents directly from the source and pass them to\n Lingo3G REST API for clustering.\n

    \n\n

    \n If you modified suite-dcs.xml to change Lingo3G\n parameter values, pass the modified parameters directly in the\n clustering request\n or in a\n request template.\n

    \n
    3. \n\n
  3. \n

    Switch from XML to JSON.

    \n\n

    \n The new REST API accepts and returns\n data in JSON format, so\n you will need to modify your code to use the new format. If you\n write your client code in Java, you can use\n data model classes\n to produce the required JSON structure.\n

    \n
    4. \n\n
  4. \n

    Convert dictionaries to JSON.

    \n\n

    \n If you edited any dictionary files in your existing DCS setup,\n covert the dictionaries\n from XML to JSON and copy them to the\n DCS dictionary directory.\n

    \n
    5. \n
\n
\n\n
\n

\n Java API\n

\n\n

\n Lingo3G 2.x Java API is not backward compatible with the 1.x API –\n you will need to update your Lingo3G integration code to make it work\n with version 2.x. This section lists the key elements to look for in\n your existing code.\n

\n\n
    \n
  1. \n

    \n First, have a look at the new API.\n

    \n\n

    \n Before you start converting any existing code, have a look at the\n basics of the new API, code examples included in the distribution package and\n JavaDoc. This will give you an idea of\n what the updated code should look like.\n

    \n
    2. \n\n
  2. \n

    Identify the controller.

    \n\n

    \n Look for references to the Controller class in your\n existing code. You should see a fragment similar to:\n

    \n\n
    \n 
    Controller controller = ControllerFactory.createCachingPooling(...)\n
    \n
    \n\n

    \n Lingo3G 1.x used the controller as a cache and threading\n coordinator. Controller does not have a direct\n replacement in the 2.x API, but the place where you initialized it\n is very likely the right place to\n load and initialize the language components. Your code should load language components once and reuse them for\n all subsequent clustering calls, also in\n concurrent threads.\n

    \n
    3. \n\n
  3. \n

    \n Identify document source(s) and the process the\n method.\n

    \n\n

    \n Lingo3G 1.x API collected the input for clustering using classes\n implementing the IDocumentSource interface. Look for an\n implementation of this class that is part of the processing chain\n invoked on the controller, or search for an explicit list of\n documents set on the parameters passed to the\n process method. For example:\n

    \n\n
    \n 
    CommonAttributesDescriptor.attributeBuilder(processingAttributes)\n  .documents(...)\n...\nProcessingResult result = controller.process(processingAttributes,\n  Lingo3GClusteringAlgorithm.class);\n
    \n
    \n\n

    \n Replace the controller.process(...) call with the 2.x\n Java API calls:\n Lingo3G algorithm instantiation,\n document stream preparation\n and\n clustering call.\n

    \n\n

    \n Note that with the new API, your code needs to provide an explicit\n stream of documents for clustering. For each document, provide only\n the fields that Lingo3G should use for clustering.\n

    \n
    4. \n\n
  4. \n

    Identify parameter customizations.

    \n\n

    \n With the 1.x API, your code could set custom parameter values during\n controller initialization or at processing-time. Identify any\n parameters that your code customizes. Their names in the old and new\n API are identical or very similar.\n

    \n\n

    \n With the 2.x API, your code should\n set parameters\n as part of Lingo3G algorithm instantiation. To decouple parameter\n customization from the clustering call, consider\n cloning a \"blueprint\" instance.\n

    \n\n

    \n If your code uses any of the following parameters, you need to make\n manual adjustments:\n

    \n\n
      \n
    • \n

      \n reloadResources\n is not available in the new API. A functional equivalent is\n loading a new language components instance\n explicitly. However, we don't recommend reloading language\n components in production code due to the computational overhead.\n To modify dictionaries at runtime, use\n ephemeral dictionaries\n instead.\n

      \n
      • \n\n
    • \n

      \n languageRecognition,\n minLanguageRecognitionConfidence\n have been removed from the new API as part of language detection\n and multilingual clustering removal.\n

      \n\n

      \n To perform multilingual clustering, use an external language\n detection library, such as\n Lingua, to split documents into separate language and cluster each\n language separately.\n

      \n
      • \n\n
    • \n

      \n titleFields is now called\n boostFields\n and the corresponding\n titleWordLabelScorerWeight is now called\n boostedFieldScorerWeight.\n

      \n\n

      \n Furthermore, Lingo3G 2.x does not boost the\n title field by default any more. If you'd like to\n boost the title field,\n set the\n boostFields\n parameter to title.\n

      \n
      • \n\n
    • \n

      \n generateLabelHighlights is removed without\n replacement. If your application relies heavily on highlighting\n occurrences of cluster labels in the text, consider switching to\n Lingo4G, which\n offers comprehensive support for label highlighting.\n

      \n
      • \n
    \n
    5. \n\n
  5. \n

    Identify resource location customizations.

    \n\n

    \n If your existing code customizes resource locations, you may see a\n code fragment like this:\n

    \n\n
    \n 
    ResourceLookup resourceLookup = new ResourceLookup(new DirLocator(resourcesDir));\n...\nLexicalDataLoaderDescriptor.attributeBuilder(controllerInitAttrs)\n  .resourceLookup(resourceLookup);\n
    \n
    \n\n

    \n In Lingo3G 2.x API, you can\n customize resource locations\n \n when loading language components.\n

    \n\n

    \n Lingo3G 2.x also changes the dictionary file naming conventions. The\n following table illustrates the changes for English, other languages\n follow the same convention.\n

    \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n\n \n \n \n \n \n \n
    Dictionary typeOld nameNew name
    Label dictionarylabel-dictionary.en.xmlenglish.labels.json
    Word dictionaryword-dictionary.en.xmlenglish.tags.json
    Synonymssynonyms.en.xmlenglish.synonyms.json
    \n\n

    \n Finally, Lingo3G 2.x switches the format of dictionary files from\n XML to JSON. You can use a command line tool to\n convert\n your custom dictionaries.\n

    \n
    6. \n\n
  6. \n

    Convert dictionaries to JSON.

    \n\n

    \n If your existing code loads custom XML dictionary files, you need to\n rename\n and convert them to the JSON format. Version 2.x replaces the\n XML-based\n token-matching rules\n with simpler and more intuitive\n glob expressions.\n

    \n\n

    \n Lingo3G 2.x comes with a migration command line tool you can use to\n covert your custom dictionaries from XML to JSON. The following\n examples assume you convert English dictionaries, change the input\n file name if needed.\n

    \n\n

    To perform the conversion, perform these steps:

    \n\n
      \n
    1. \n

      \n Open a command prompt and change current directory to the\n top-level Lingo3G distribution folder.\n

      \n
      2. \n\n
    2. \n

      To convert a label dictionary, run:

      \n\n
      \n 
      java -jar lib\\lingo3g-migration-2.0.0.jar convert-labels label-dictionary.en.xml
      \n
      \n\n

      This command produces english.labels.json.

      \n
      3. \n\n
    3. \n

      To convert a synonym dictionary, run:

      \n\n
      \n 
      java -jar lib\\lingo3g-migration-2.0.0.jar convert-synonyms synonyms.en.xml
      \n
      \n\n

      This command produces english.synonyms.json.

      \n
      4. \n\n
    4. \n

      To convert a word dictionary to a tag dictionary, run:

      \n\n
      \n 
      java -jar lib\\lingo3g-migration-2.0.0.jar java -jar convert-words word-dictionary.en.xml
      \n
      \n\n

      The command produces english.tags.json.

      \n
      5. \n
    \n\n

    \n Note that conversion tools \"flatten\" all internal includes present\n in XML files. If to keep the dictionaries separate, remove include\n statements before conversion and run the converter on each file\n separately.\n

    \n
    7. \n\n
  7. \n

    Update license file lookup locations.

    \n\n

    \n Lingo3G 2.x Java API slightly changes the\n license file lookup locations. Most notably, Lingo3G now uses the class loader of Lingo3G API\n JAR to look up the license resource instead of the thread's context\n class loader.\n

    \n
    8. \n
\n
\n\n
\n

\n C# API\n

\n\n

\n Lingo3G 2.x discontinued the native C# API due to the declining\n developer interest. If your application relies on the C# API, consider\n converting your C# code to call\n Lingo3G REST API instead.\n

\n
\n\n
\n

\n Clustering Workbench\n

\n\n

\n Lingo3G 2.x discontinued the desktop Clustering Workbench application in\n favor of the browser-based\n Clustering Workbench app.\n

\n\n

\n The following elements of the old desktop-based Workbench are not\n available in the new browser-based version:\n

\n\n
\n
Multiple results panels
\n
\n

\n The old Workbench could open multiple clustering results tabs for\n simultaneous viewing. To achieve the same with the new Workbench,\n use browser tabs to work on multiple clustering results at the same\n time.\n

\n
\n\n
Aduna Cluster Map
\n
\n

\n The old Workbench had an option to present document clusters as an\n interactive graph of nodes. This kind of visualization is not\n currently available in the new Workbench, but we're hoping to add an\n equivalent counterpart at some point.\n

\n
\n\n
Benchmark view
\n
\n

\n You could use the old Workbench to run performance benchmarks of\n Lingo3G clustering. This function is not possible to reliably\n implement in the new Workbench and is not available. The best way to\n asses Lingo3G performance is to benchmark it as part of the\n application you are developing. If you need any performance or\n benchmarking advice, please contact us at\n info@carrotsearch.com.\n

\n
\n
\n\n

\n The new Workbench comes with features the old version did not have, such\n as\n clustering data from Excel spreadsheets\n or\n configurable document view.\n

\n
\n
\n
","tableOfContents":[{"heading":"Overview of changes","anchor":"changes"},{"heading":"Migration guide","anchor":"migration-guide","sections":[{"heading":"REST API (DCS)","anchor":"migration-guide:rest-api"},{"heading":"Java API","anchor":"migration-guide:java-api"},{"heading":"C# API","anchor":"migration-guide:csharp-api"},{"heading":"Clustering Workbench","anchor":"migration-guide:workbench"}]}]}},"pageContext":{"slug":"/release-notes-2.0.x-migration-guide/"}},"staticQueryHashes":["1164750013"]}