• Tobias Vogel

DAQS (DAta Quality as a WebService) is a comprehensive data clensing project. Its goal is to provide the full duplicate detection workflow via webservice with as less manual interaction as possible. The challenge is to enable the computer taking decisions that usually a human expert takes.

The workflow consists mainly of three steps.

1. In the problem classification phase, the available dataset is analyzed and the degree of missing information is estimated. (see below)
2. If the semantics (a.k.a fine-grained data types) of the dataset is unclear, classes have to be assigned to the attributes.
3. The actual duplicate detection is performed with the similarity measures derived from the classes.

 There are four types of datasets, that can be present.

1. Datasets can have semantic annotations for the attributes (and a mapping/a separator). Consequently, the duplicate detection task can be performed nearly automatically.
2. The datasets only have a mapping (and a separator). Thus, it is clear, which attributes to compare, but not, how.
3. In datasets without a mapping, only the tuples and attributes are distinguishable, but it is not clear, which attributes to compare with which other attributes.
4. In case of unstructured documents, not even tuples/attributes can be recognized. They have to be retrieved, first.

In case that there are no semantics assigned to the dataset's attributes, they have to be assigned by the service. In DAQS, we use an instance-based as well as an machine learning classification approach to do that.

The figure shows the instance-based and machine learning classification classes.

Often, the original datasets contained many null values. As we only took 500 tuples for our experiments, we selected 500 tuples from the best-filled tuples of each dataset randomly.

There is a little demo created for the wheelmap.org project.

Wheelmap-Client

Wheelmap-Service-Endpoint (RESTful)