REST API (DCS)

Lingo3G 2.x REST API is not backward compatible with the 1.x API – you will need to update your Lingo3G integration code to make it work with version 2.x. This section lists the key elements to look for in your existing code.

1. First, have a look at the new REST API.

   Before you start converting any existing code, have a look at the basics of the new API.

2. Identify suite-dcs.xml modifications.

   Previously, Lingo3G used the suite-dcs.xml component suite definition to declare document sources, algorithms and parameter changes. If you edited that file to change parameters or add custom document sources, identify these changes first.

   The new REST API does not support document sources, so if your code used built-in or custom document sources, you need to modify the code to fetch documents directly from the source and pass them to Lingo3G REST API for clustering.

   If you modified suite-dcs.xml to change Lingo3G parameter values, pass the modified parameters directly in the clustering request or in a request template.

3. Switch from XML to JSON.

   The new REST API accepts and returns data in JSON format, so you will need to modify your code to use the new format. If you write your client code in Java, you can use data model classes to produce the required JSON structure.

4. Convert dictionaries to JSON.

   If you edited any dictionary files in your existing DCS setup, covert the dictionaries from XML to JSON and copy them to the DCS dictionary directory.

Java API

Lingo3G 2.x Java API is not backward compatible with the 1.x API – you will need to update your Lingo3G integration code to make it work with version 2.x. This section lists the key elements to look for in your existing code.

1. First, have a look at the new API.

   Before you start converting any existing code, have a look at the basics of the new API, code examples included in the distribution package and JavaDoc. This will give you an idea of what the updated code should look like.

2. Identify the controller.

   Look for references to the Controller class in your existing code. You should see a fragment similar to:

   Controller controller = ControllerFactory.createCachingPooling(...)

   Lingo3G 1.x used the controller as a cache and threading coordinator. Controller does not have a direct replacement in the 2.x API, but the place where you initialized it is very likely the right place to load and initialize the language components. Your code should load language components once and reuse them for all subsequent clustering calls, also in concurrent threads.

3. Identify document source(s) and the process the method.

   Lingo3G 1.x API collected the input for clustering using classes implementing the IDocumentSource interface. Look for an implementation of this class that is part of the processing chain invoked on the controller, or search for an explicit list of documents set on the parameters passed to the process method. For example:

   CommonAttributesDescriptor.attributeBuilder(processingAttributes)
     .documents(...)
   ...
   ProcessingResult result = controller.process(processingAttributes,
     Lingo3GClusteringAlgorithm.class);

   Replace the controller.process(...) call with the 2.x Java API calls: Lingo3G algorithm instantiation, document stream preparation and clustering call.

   Note that with the new API, your code needs to provide an explicit stream of documents for clustering. For each document, provide only the fields that Lingo3G should use for clustering.

4. Identify parameter customizations.

   With the 1.x API, your code could set custom parameter values during controller initialization or at processing-time. Identify any parameters that your code customizes. Their names in the old and new API are identical or very similar.

   With the 2.x API, your code should set parameters as part of Lingo3G algorithm instantiation. To decouple parameter customization from the clustering call, consider cloning a "blueprint" instance.

   If your code uses any of the following parameters, you need to make manual adjustments:

   • reloadResources is not available in the new API. A functional equivalent is loading a new language components instance explicitly. However, we don't recommend reloading language components in production code due to the computational overhead. To modify dictionaries at runtime, use ephemeral dictionaries instead.

   • languageRecognition, minLanguageRecognitionConfidence have been removed from the new API as part of language detection and multilingual clustering removal.

     To perform multilingual clustering, use an external language detection library, such as Lingua, to split documents into separate language and cluster each language separately.

   • titleFields is now called boostFields and the corresponding titleWordLabelScorerWeight is now called boostedFieldScorerWeight.

     Furthermore, Lingo3G 2.x does not boost the title field by default any more. If you'd like to boost the title field, set the boostFields parameter to title.

   • generateLabelHighlights is removed without replacement. If your application relies heavily on highlighting occurrences of cluster labels in the text, consider switching to Lingo4G, which offers comprehensive support for label highlighting.

5. Identify resource location customizations.

   If your existing code customizes resource locations, you may see a code fragment like this:

   ResourceLookup resourceLookup = new ResourceLookup(new DirLocator(resourcesDir));
   ...
   LexicalDataLoaderDescriptor.attributeBuilder(controllerInitAttrs)
     .resourceLookup(resourceLookup);

   In Lingo3G 2.x API, you can customize resource locations when loading language components.

   Lingo3G 2.x also changes the dictionary file naming conventions. The following table illustrates the changes for English, other languages follow the same convention.

   Dictionary type	Old name	New name
   Label dictionary	label-dictionary.en.xml	english.labels.json
   Word dictionary	word-dictionary.en.xml	english.tags.json
   Synonyms	synonyms.en.xml	english.synonyms.json

   Finally, Lingo3G 2.x switches the format of dictionary files from XML to JSON. You can use a command line tool to convert your custom dictionaries.

6. Convert dictionaries to JSON.

   If your existing code loads custom XML dictionary files, you need to rename and convert them to the JSON format. Version 2.x replaces the XML-based token-matching rules with simpler and more intuitive glob expressions.

   Lingo3G 2.x comes with a migration command line tool you can use to covert your custom dictionaries from XML to JSON. The following examples assume you convert English dictionaries, change the input file name if needed.

   To perform the conversion, perform these steps:

   1. Open a command prompt and change current directory to the top-level Lingo3G distribution folder.

   2. To convert a label dictionary, run:

      java -jar lib\lingo3g-migration-2.1.3.jar convert-labels label-dictionary.en.xml

      This command produces english.labels.json.

   3. To convert a synonym dictionary, run:

      java -jar lib\lingo3g-migration-2.1.3.jar convert-synonyms synonyms.en.xml

      This command produces english.synonyms.json.

   4. To convert a word dictionary to a tag dictionary, run:

      java -jar lib\lingo3g-migration-2.1.3.jar java -jar convert-words word-dictionary.en.xml

      The command produces english.tags.json.

   Note that conversion tools "flatten" all internal includes present in XML files. If to keep the dictionaries separate, remove include statements before conversion and run the converter on each file separately.

7. Update license file lookup locations.

   Lingo3G 2.x Java API slightly changes the license file lookup locations. Most notably, Lingo3G now uses the class loader of Lingo3G API JAR to look up the license resource instead of the thread's context class loader.

C# API

Lingo3G 2.x discontinued the native C# API due to the declining developer interest. If your application relies on the C# API, consider converting your C# code to call Lingo3G REST API instead.

Clustering Workbench

Lingo3G 2.x discontinued the desktop Clustering Workbench application in favor of the browser-based Clustering Workbench app.

The following elements of the old desktop-based Workbench are not available in the new browser-based version:

Multiple results panels

The old Workbench could open multiple clustering results tabs for simultaneous viewing. To achieve the same with the new Workbench, use browser tabs to work on multiple clustering results at the same time.

Aduna Cluster Map

The old Workbench had an option to present document clusters as an interactive graph of nodes. This kind of visualization is not currently available in the new Workbench, but we're hoping to add an equivalent counterpart at some point.

Benchmark view

You could use the old Workbench to run performance benchmarks of Lingo3G clustering. This function is not possible to reliably implement in the new Workbench and is not available. The best way to asses Lingo3G performance is to benchmark it as part of the application you are developing. If you need any performance or benchmarking advice, please contact us at info@carrotsearch.com.

The new Workbench comes with features the old version did not have, such as clustering data from Excel spreadsheets or configurable document view.