aligned​Fragments view

The aligned​Fragments view is configured with this block in our request:

"alignedFragments": {
  "contextChars": 80,
  "maxFragments": 10,
  "fields": {
    "type": "contentFields:grouped",
    "groups": [
      {
        "fields": {
          "@var": "fieldsToCompare"
        },
        "config": {
          "maxValueLength": 3000
        }
      }
    ]
  }
},
  

The most important properties are the max​Fragments, which declares the maximum number of the longest duplicated fragments returned, and context​Chars which tells how many characters to the left and right of the duplicated passage should be returned. An additional fields component is used to configure constraints on returned value. In case of very large duplicated fragments (or very large document values) it may be useful to truncate the returned fragments (this is the role of max​Value​Length above).

Here is an excerpt from the JSON output for our original request:

"overlaps": [
  {
    "pair": [
      870338,
      1172681
    ],
    "stats": {
      "a": {
        "abstract": {
          "fragments": 2,
          "fieldLength": 1105,
          "overlapLength": 773,
          "ratio": 0.6995475113122172
        }
      },
      "b": {
        "abstract": {
          "fragments": 2,
          "fieldLength": 1235,
          "overlapLength": 772,
          "ratio": 0.6251012145748988
        }
      }
    },
    ...
    

The output for the stage (overlaps) contains an array of objects corresponding to document pairs from similarity analysis. The pair property contains internal identifiers of the documents for which duplicated text fragments follow. The stats property contains the per-field information about the number of fragments, total field value length and the overlap length (in characters).

What follows in the output for the aligned fragments view:

"alignedFragments": [
  {
    "length": 584,
    "context": {
      "a": "  ⁌overlap⁍Penalized regression methods, such as undefined regularization, are routinely\nused in high-dimensional applications, and there is a rich literature on\noptimality properties under sparsity assumptions. In the Bayesian paradigm,\nsparsity is routinely induced through two-component mixture priors having a\nprobability mass at zero, but such priors encounter daunting computational\nproblems in high dimensions. This has motivated an amazing variety of\ncontinuous shrinkage priors, which can be expressed as global-local scale\nmixtures of Gaussians, facilitating computation. In sharp contrast⁌\\overlap⁍ to the\nfrequentist literature, little is known about the properties of such priors and\nthe convergence and concentration of the corresponding posterior…",
      "b": "  ⁌overlap⁍Penalized regression methods, such as undefined regularization, are routinely\nused in high-dimensional applications, and there is a rich literature on\noptimality properties under sparsity assumptions. In the Bayesian paradigm,\nsparsity is routinely induced through two-component mixture priors having a\nprobability mass at zero, but such priors encounter daunting computational\nproblems in high dimensions. This has motivated an amazing variety of\ncontinuous shrinkage priors, which can be expressed as global-local scale\nmixtures of Gaussians, facilitating computation. In sharp contrast⁌\\overlap⁍ to the\ncorresponding frequentist literature, very little is known about the properties\nof such priors. Focusing on a broad class of shrinkage priors, we…"
    }
  },
  {
    "length": 189,
    "context": {
      "a": "…this article, we propose a new class of Dirichlet--Laplace (DL) priors,\nwhich possess optimal posterior concentration and ⁌overlap⁍lead to efficient posterior\ncomputation exploiting results from normalized random measure theory. Finite\nsample performance of Dirichlet--Laplace priors relative to alternatives is\nassessed⁌\\overlap⁍ in simulated and real data examples.\n",
      "b": "…Bayesian\nLasso, are suboptimal in high-dimensional settings. A new class of Dirichlet\nLaplace (DL) priors are proposed, which are optimal and ⁌overlap⁍lead to efficient\nposterior computation exploiting results from normalized random measure theory.\nFinite sample performance of Dirichlet Laplace priors relative to alternatives\nis assessed⁌\\overlap⁍ in simulations.\n"
    }
  }
],

Each aligned fragment is an object with the length of the duplicated fragment (in characters) and a text value for documents ("a" indicates the first document and "b" the second identifier from the pair element). The next value contains an ellipsis character where the value was truncated (…) and a pair of special markers to indicate the start and end of the fragment: ⁌overlap⁍, ⁌\overlap⁍. These markers contain fairly unique Unicode characters, so they're not likely to appear anywhere in the text. They can be used directly for highlighting; here is how Lingo4G Explorer shows aligned fragments:

fragments​In​Fields view

The fragments-in-fields view is configured similarly to the aligned fragments, although the interpretation of highlighting directives is a bit different.

"fragmentsInFields": {
  "contextChars": 600,
  "fields": {
    "type": "contentFields:grouped",
    "groups": [
      {
        "fields": {
          "@var": "fieldsToCompare"
        },
        "config": {
          "maxValues": 10,
          "maxValueLength": 3000
        }
      }
    ]
  }
}
  

The algorithm takes each document's field value and all the overlap regions, then tries to find a display window that maximizes the number of overlaps present inside. The intuition here is to show the context of each overlap within the value — much like snippets from the matching Web page in a search engine. The max​Values defines the maximum number of value ranges to display, the max​Value​Length is the maximum length (in characters) of each value. If max​Value​Length is larger than the actual field length, the entire field value is shown, with all overlapping fragments highlighted. An additional context​Chars property controls the leading and trailing context of each value range selected for display.

Here is the relevant JSON output in the response. Note the ⁌overlap⁍ and ⁌\\overlap⁍ highlight markers. We used a very large max​Value​Length of 3000 so the entire field value is returned as one passage, with no truncations.

"fragmentsInFields": {
  "a": {
    "abstract": [
      "  ⁌overlap⁍Penalized regression methods, such as undefined regularization, are routinely\nused in high-dimensional applications, and there is a rich literature on\noptimality properties under sparsity assumptions. In the Bayesian paradigm,\nsparsity is routinely induced through two-component mixture priors having a\nprobability mass at zero, but such priors encounter daunting computational\nproblems in high dimensions. This has motivated an amazing variety of\ncontinuous shrinkage priors, which can be expressed as global-local scale\nmixtures of Gaussians, facilitating computation. In sharp contrast⁌\\overlap⁍ to the\nfrequentist literature, little is known about the properties of such priors and\nthe convergence and concentration of the corresponding posterior distribution.\nIn this article, we propose a new class of Dirichlet--Laplace (DL) priors,\nwhich possess optimal posterior concentration and ⁌overlap⁍lead to efficient posterior\ncomputation exploiting results from normalized random measure theory. Finite\nsample performance of Dirichlet--Laplace priors relative to alternatives is\nassessed⁌\\overlap⁍ in simulated and real data examples.\n"
    ]
  },
  "b": {
    "abstract": [
      "  ⁌overlap⁍Penalized regression methods, such as undefined regularization, are routinely\nused in high-dimensional applications, and there is a rich literature on\noptimality properties under sparsity assumptions. In the Bayesian paradigm,\nsparsity is routinely induced through two-component mixture priors having a\nprobability mass at zero, but such priors encounter daunting computational\nproblems in high dimensions. This has motivated an amazing variety of\ncontinuous shrinkage priors, which can be expressed as global-local scale\nmixtures of Gaussians, facilitating computation. In sharp contrast⁌\\overlap⁍ to the\ncorresponding frequentist literature, very little is known about the properties\nof such priors. Focusing on a broad class of shrinkage priors, we provide\nprecise results on prior and posterior concentration. Interestingly, we\ndemonstrate that most commonly used shrinkage priors, including the Bayesian\nLasso, are suboptimal in high-dimensional settings. A new class of Dirichlet\nLaplace (DL) priors are proposed, which are optimal and ⁌overlap⁍lead to efficient\nposterior computation exploiting results from normalized random measure theory.\nFinite sample performance of Dirichlet Laplace priors relative to alternatives\nis assessed⁌\\overlap⁍ in simulations.\n"
    ]
  }
}

Here is how this information (combined with the extra fields output) is displayed in the Explorer:

If we wanted to display shorter snippets with overlap highlights, instead of full values, we could set the max​Value​Length to a smaller value (300 characters) and the context characters to 100 characters, like this:

"fragmentsInFields": {
  "contextChars": 100,
  "fields": {
    "type": "contentFields:grouped",
    "groups": [
      {
        "fields": {
          "@var": "fieldsToCompare"
        },
        "config": {
          "maxValues": 10,
          "maxValueLength": 300
        }
      }
    ]
  }
}
  

The output would then look as shown below (note ellipsis truncation markers):

Finally, note that the algorithm for choosing which snippets to choose is independent for each document in a pair, so it is possible that different regions of each document (with different duplicated fragments) are selected and returned as output. For side-by-side view, it is recommended to set the max​Value​Length to a number exceeding maximum field length of any document.