D3.3 Enabling tools for data interoperability, distributed data storage and training distributed AI models. First version

SEDIMARK D3.3 “Enabling tools for data interoperability, distributed data storage and training distributed AI models” is a report produced by the SEDIMARK project. It aims at providing insights in relation to progress made for:

• Management of interoperability in data flows by proper definition and usage of metadata and semantic technologies
• Distributed storage considering interoperability and scalability issues of such storages.
• Finally, on the use of distributed AI and analytics, considering in particular federated learning.

Data quality is considered to be of the highest importance for companies to improve their decision-making systems and the efficiency of their products. In this current data-driven era, it is important to understand the effect that “dirty” or low-quality data (i.e., data that is inaccurate, incomplete, inconsistent, or contains errors) can have on a business. Manual data cleaning is the common way to process data, accounting for more than 50% of the time of knowledge workers. SEDIMARK acknowledges the importance of data quality for both sharing and using/analysing data to extract knowledge and information for decision-making processes.

Common types of dirty or low-quality data include:

1. Missing Data: Incomplete datasets where certain values are not recorded.
2. Inaccurate Data: Data that contains errors, inaccuracies, or typos. This can happen due to manual data entry errors or system malfunctions.
3. Inconsistent Data: Data that is inconsistent across different sources or within the same dataset. For example, the same entity may be represented in different ways (e.g., "Mr. Smith" vs. "Smith, John").
4. Duplicate Data: Repetition of the same data in a dataset, which can distort analyses and lead to incorrect results.
5. Outliers: Data points that deviate significantly from the majority of the dataset, potentially skewing the analysis.
6. Bias: Data that reflects systematic errors due to a particular group's overrepresentation or underrepresentation in the dataset.
7. Unstructured Data: Data that lacks a predefined data model or organization, making it difficult to analyse.

Thus, one of the main work items of SEDIMARK is to develop a usable data processing pipeline that assesses and improves the quality of data generated and shared by the SEDIMARK data providers.

This document, along with all the public deliverables and documents produced by SEDIMARK, can be found in the Publications & Resources section.