SEDIMARK Logo

D3.3 Enabling tools for data interoperability, distributed data storage and training distributed AI models. First version

SEDIMARK · January 18, 2024
D3.3

SEDIMARK D3.3 “Enabling tools for data interoperability, distributed data storage and training distributed AI models” is a report produced by the SEDIMARK project. It aims at providing insights in relation to progress made for:

  • Management of interoperability in data flows by proper definition and usage of metadata and semantic technologies
  • Distributed storage considering interoperability and scalability issues of such storages.
  • Finally, on the use of distributed AI and analytics, considering in particular federated learning.

Data quality is considered to be of the highest importance for companies to improve their decision-making systems and the efficiency of their products. In this current data-driven era, it is important to understand the effect that “dirty” or low-quality data (i.e., data that is inaccurate, incomplete, inconsistent, or contains errors) can have on a business. Manual data cleaning is the common way to process data, accounting for more than 50% of the time of knowledge workers. SEDIMARK acknowledges the importance of data quality for both sharing and using/analysing data to extract knowledge and information for decision-making processes.

Common types of dirty or low-quality data include:

  1. Missing Data: Incomplete datasets where certain values are not recorded.
  2. Inaccurate Data: Data that contains errors, inaccuracies, or typos. This can happen due to manual data entry errors or system malfunctions.
  3. Inconsistent Data: Data that is inconsistent across different sources or within the same dataset. For example, the same entity may be represented in different ways (e.g., "Mr. Smith" vs. "Smith, John").
  4. Duplicate Data: Repetition of the same data in a dataset, which can distort analyses and lead to incorrect results.
  5. Outliers: Data points that deviate significantly from the majority of the dataset, potentially skewing the analysis.
  6. Bias: Data that reflects systematic errors due to a particular group's overrepresentation or underrepresentation in the dataset.
  7. Unstructured Data: Data that lacks a predefined data model or organization, making it difficult to analyse.

Thus, one of the main work items of SEDIMARK is to develop a usable data processing pipeline that assesses and improves the quality of data generated and shared by the SEDIMARK data providers.

This document, along with all the public deliverables and documents produced by SEDIMARK, can be found in the Publications & Resources section.

Subscribe to SEDIMARK!

* required

🎉 Excited to share that Tarek Elsaleh will represent @sedimark at OpenSource Community Day 2025 in Madrid (23–24 Sept)! He’ll speak on how EU projects like SEDIMARK are advancing open data & innovation. 🇪🇺

📍 Don’t miss this key #opensource event!

🚀 Just Published: A Practical Guide to Multivariate Time Series Forecasting with Crossformer Package.
This tool is useful for forecasting tasks in domains like energy or sensor networks—where handling multiple correlated signals is essential.

Load More
crossmenu
SEDIMARK
Privacy Overview

This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.