1. Authors
2. Abstract
3. Test data
4. Related Work

Melanie Weis, Felix Naumann

The problem of detecting duplicate entities that describe the same real-world object is an important data cleansing task, necessary to improve data quality. For data stored in a flat relation, numerous solutions to this problem exist. As XML becomes increasingly popular for data representation, algorithms to detect duplicates in nested XML documents are required.

In this paper, we present a domain-independent algorithm that effectively identifies duplicates in a XML document. The solution adopts a top-down traversal of the XML tree structure to identify duplicate elements on each level. Pairs of duplicate elements are detected using a thresholded similarity function, and are then clustered by computing the transitive closure. To minimize the number of pairwise element comparisions, an appropriate filter function is used. The similarity measure involves string similarity for pairs of strings, which is measured using their edit distance. To  increase efficiency, we avoid the computation of edit distance for pairs of strings using three filtering methods subsequently. First experiments show that our approach detects XML duplicates accurately and efficiently. [more]

• Country Dataset

Here you find several scientific work which also deal with Duplicate Detection in XML data:

• A Duplicate Detection Benchmark for XML (and Relational) Data

• DogmatiX Tracks down Duplicates in XML
• Fuzzy Duplicate Detection on XML Data
• XML Duplicate Detection Using Sorted Neighborhoods