At the heart of #SEDIMARK lies a powerful AI architecture, featuring MageAI and MLFlow, designed to revolutionize energy optimization. MageAI orchestrates seamless AI workflows, while MLFlow automates robust machine learning pipelines, enabling accurate and localized energy consumption predictions. Our integrated system supports dynamic customer management strategies, ensuring a transparent, scalable, and efficient solution for energy forecasting.

We leverage capabilities like federated learning, real-time inference, and secure data exchange to enhance the reliability and usability of our system. This AI-driven approach empowers users to make data-driven decisions, fostering a sustainable energy future. Through advanced analytics and actionable insights, we are transforming energy distribution and enhancing customer experiences.

We are poised to optimize energy management for the future, making waves in energy efficiency. Learn how #SEDIMARK is leading the charge in #AIforEnergy, #DataDriven, and #EnergyInnovation. #METLEN #AI #Sustainability #Innovation #EnergyEfficiency

On March 12^th, Luis Sánchez, Technical Coordinator of the SEDIMARK project, had the privilege of presenting at the Water Projects Europe Workshop on From Inland Water to Digital Connectivity: Shaping the Future of Water Management. The presentation was part of the session on Understanding Digital Water Management and Interoperability, focused on the SEDIMARK Project and its innovative approach that could fit into the needs and challenges of transforming water management through digital technologies.

The SEDIMARK Project is developing a Distributed, Intelligent and Trustworthy Data Marketplace based on the Data Spaces paradigm. This platform enables seamless data sharing between stakeholders, such as governments, utilities, and researchers, by focusing on data interoperability, quality, and trustworthiness. By ensuring that data is accurate, transparent, and secure, SEDIMARK empowers informed decision-making, improving water resource management and fostering collaboration across sectors.

The SEDIMARK Project is pioneering the development of a Distributed and Intelligent Data Marketplace, a cutting-edge platform aimed at facilitating the seamless exchange of water-related data across various stakeholders, including governments, water utilities, researchers, and private companies. At the heart of SEDIMARK lies the Data Spaces paradigm—a new approach that enables data to be shared and exchanged across a range of sectors and industries while ensuring data privacy and security.

A primary goal of the project is to enhance data interoperability—ensuring that data from various sources can work together seamlessly to support decision-making. By leveraging data interoperability, SEDIMARK enables the integration of disparate datasets such as environmental data, water consumption data, and climate-related data, empowering stakeholders to make informed decisions based on a complete picture of water systems.

But, as we all know, data is only valuable if it’s trustworthy. That’s why the SEDIMARK Project places a significant focus on data quality and trustworthiness. Trust is a key enabler for the success of digital water management systems, and ensuring data quality, provenance, and transparency is essential to building that trust. The project employs mechanisms that ensure that data shared within the marketplace is accurate, verifiable, and useful for the end-users. One of the most exciting aspects of the SEDIMARK platform is its ability to support data exchange scenarios. Whether it's about sharing water quality data from rivers, monitoring urban water usage, or assessing the impact of climate change on freshwater availability, SEDIMARK offers a mechanism for exchanging this critical data across borders and sectors, thereby enabling better decision-making and more efficient water management practices.

Overcoming Barriers to Digital Water Management

Following the presentation, a dynamic closing round-table with other experts in the field of digital water management was organized. The significant barriers and challenges that organizations and stakeholders face when trying to implement digital and smart water management solutions were explored.

One of the major challenges that were discussed was the lack of standardized data formats and protocols, which often hinders data interoperability. Despite advancements in technology, different water management systems often use incompatible data formats, making it difficult for stakeholders to share and use the data effectively. Standardizing data formats and establishing common protocols for data exchange could go a long way in overcoming this issue.

Another challenge raised was the resistance to adopting new technologies. Many organizations, especially in more traditional sectors like water management, are hesitant to embrace digital transformation due to concerns about cost, complexity, and the learning curve associated with new systems. Overcoming this resistance requires demonstrating the tangible benefits of digital water management solutions, such as improved resource efficiency, cost savings, and better environmental outcomes.

Lastly, the issue of data privacy and security was discussed. As water management systems become more interconnected, the amount of data being exchanged increases exponentially. This raises concerns about the protection of sensitive data, such as personal information or proprietary business data. Ensuring robust security protocols and establishing clear guidelines around data ownership and access are critical for fostering trust among stakeholders.

Moving forward

In conclusion, the workshop provided a valuable opportunity to discuss how digital connectivity is shaping the future of water management. The SEDIMARK Project and similar initiatives are laying the groundwork for a more interconnected, efficient, and sustainable approach to managing water resources. However, the road to fully digitalized water management systems is not without its challenges. Overcoming barriers like data interoperability, technology adoption, and data security will require collaborative efforts from governments, industry leaders, and technology developers.

Quoting our Technical Coordinator: “As I left the workshop, I felt optimistic about the progress being made in digital water management and excited about the role that the technological enablers that projects like SEDIMARK are developing will play in the future of sustainable water use and management.”

The future of urban mobility is increasingly shaped by data, driving smarter, more sustainable, and highly efficient cities. As urban centers expand, they face the pressing challenge of managing traffic congestion while ensuring accessible and environmentally friendly transportation solutions for all citizens. By integrating cutting-edge technologies, cities can now monitor traffic in real time and implement strategic interventions that improve safety, reduce emissions, and enhance overall mobility.

One of the primary catalysts behind this transformation is the ability to collect and analyze vast amounts of mobility data. This data-driven approach influences travel behaviors, optimizes transportation networks, and ensures equitable mobility benefits. A prime example of this innovation is the ACUMEN pilot project in Helsinki, where real-time data sharing is facilitated through advanced multi-layered digital twins. These digital representations of the city’s transportation ecosystem allow various systems and service providers to collaborate seamlessly, fostering a dynamic and responsive urban mobility environment.

A significant trend within this data-driven revolution is the emergence of secure, intelligent data-sharing platforms. One such example is the SEDIMARK pilot project, which has introduced a secure mobility data marketplace. The pilot is implemented as a minimum viable system and utilizes Helsinki’s mobility digital twins to create a highly adaptive model.

This marketplace is designed to provide city developers with real-time, actionable data, enabling data-driven decision-making for urban development, leading to more efficient and sustainable urban solutions—not only for mobility but also for infrastructure and condition management.

* image from Vesa Laitinen (c) Forum Virium Helsinki

As technology continues to advance, we are witnessing a transformation in how we approach problem-solving. In particular, artificial intelligence (AI) is reshaping the landscape of urban innovation, empowering individuals and communities to collaborate in creating smarter, more inclusive cities.

The Evolution of Problem-Solving Tools

In the past, challenges were tackled by specialists working in silos, relying on closed processes and limited datasets. However, with the advent of AI-powered tools, the decision-making process is becoming increasingly democratized. From machine learning algorithms and predictive analytics to intelligent automation, AI provides real-time insights that are actively shaping urban development.

One of the most notable shifts is how AI is enabling urban planners to analyze complex datasets, such as traffic patterns, infrastructure needs, and public service optimization. AI-assisted platforms are also fostering direct community engagement, with interactive dashboards and smart simulations encouraging citizens to actively participate in shaping their urban environment. Through methods like SEDIMARK data spaces and AI, we can ensure the reliability and integrity of data, which is crucial in building trust and making informed decisions. The ACUMEN pilot project uses an AI tool to promote seamless mobility, contributing significantly to the advancement of urban development.

Empowering Everyday Problem-Solvers

AI is no longer limited to data scientists or engineers. With the rise of user-friendly AI-powered tools, individuals from all walks of life are now empowered to participate in innovation. Whether it’s through AI-driven tourist assistance via chatbots, citizen-driven urban design platforms, or AI-powered tools that help communities anticipate environmental challenges, the accessibility of these tools has opened the world of innovation to a broader audience.

By breaking down barriers to information and expertise, AI fosters a collaborative environment where diverse perspectives can come together to address the challenges of urban life. This inclusivity is vital in building cities that serve the needs of all residents—not just those with specialized knowledge.

The Synergy Between AI and Human Ingenuity

While AI undoubtedly enhances problem-solving capabilities, human ingenuity remains central to the process. AI augments our ability to analyze complex data and scenarios, but creativity, ethical reasoning, and emotional intelligence continue to drive the best solutions. The future of urban innovation lies in finding harmony between AI’s computational power and human intuition, ensuring that technology becomes a tool that amplifies human agency rather than replacing it.

* image from Outi Neuvonen (c) Helsinki Partners

SEDIMARK participated in one of the meetings organized by the ETSI cross cutting Context Information Management (CIM), providing two contributions on relevant topics regarding Context Information standardization. The following presentations were carried out by the Universidad de Cantabria team:

• Modelling and handling of Data Quality for trustworthy data sharing: The SEDIMARK's approach - Laura Martin - Universidad de Cantabria
• STF676 Results: Handling of data catalogues and data services with NGSI-LD - Luis Sanchez - Universidad de Cantabria, ETSI STF Expert

The meeting took place last October 9th, and all the contents and outcomes can be found in the following link.

After the thrilling session that was held at the EBDVF 2024 in Budapest on October 4th, titled "Leveraging Technologies for Data Management to Implement Data Spaces," where around 30 attendees gathered to hear representatives from six key projects—SEDIMARK, Waterverse, EnRichMyData, GREEN.DAT.AI, DATABRI-X, and STELAR—present their latest developments, we come to summarize the main aspects covered in the session and the outcomes coming out from it.

Dr. Luis Sánchez, Technical Coordinator of the SEDIMARK project, was there to present SEDIMARK’s latest developments on its presentation: “SEDIMARK: A Decentralised Marketplace for Trustworthy and Enriched Data and Services Exchange”.

Discussion topics and insights

The aforementioned projects are working to foster greater interoperability, standardization, and the seamless sharing of high-quality data across various sectors and domains. During the session, the presenters provided insights into how their respective projects are supporting the creation of Data Spaces. They highlighted a mapping between their advancements and the different Building Blocks outlined in the DSSC (Data Spaces Support Centre) Blueprint, showing how these technologies fit into the larger framework of interconnected Data Spaces. By aligning their work with the DSSC Blueprint, the projects are contributing to a more cohesive and interoperable ecosystem for data sharing.

The discussion that followed between the project experts and the audience underscored the critical need for interoperable data, which is vital for facilitating collaboration and innovation across industries. Participants emphasized the importance of establishing clear mechanisms that enable data providers and consumers to know precisely what data and services they are accessing, while also ensuring pathways for improving the quality of this data. The session also highlighted the importance of ensuring that data can flow smoothly between different systems and platforms, fostering trust and transparency in the process.

Moreover, the complementarity of the different projects' developments emerged as a key point, opening up opportunities for future collaboration. By leveraging the unique strengths of each project, the panellists and attendees recognized that a unified approach to implementing Data Spaces could accelerate progress toward a more interoperable and standardized data-sharing infrastructure across sectors.

Take-aways In conclusion, the session served as a pivotal forum for discussing the next steps toward realizing fully operational Data Spaces, emphasizing both the technological advancements being made and the ongoing need for collaboration to ensure the success of this ambitious vision.

In today’s fast-paced digital landscape, effective data processing is a critical component for any organization looking to derive insights and drive innovation. However, setting up data pipelines - from extraction to transformation and loading (ETL) - has traditionally required a high level of expertise. We’re happy to announce a solution that could democratize data orchestration for users of all experience levels: the SEDIMARK Data Orchestrator powered by Mage.ai and enhanced with Generative AI.

Simplified Data Processing with AI Assistance

Our platform integrates the power of Large Language Models (LLMs) to automatically generate Mage.ai pipeline blocks, helping even those with minimal technical background create robust data workflows. Instead of spending hours - or even days - writing code and configuring pipelines, users now simply need to upload their dataset into the Orchestrator GUI.

Once the dataset is in place, the system, with a bit of guidance through a helpful prompt, takes over the heavy lifting. Using generative AI, the platform produces custom Mage.ai templates and workflows specifically tailored to your data. This eliminates the need for users to dive deep into code or ETL specifics.

How It Works

Whether you’re dealing with traffic provided data, weather records, or IoT data streams, the process starts with uploading your dataset into the Orchestrator GUI.

With the help of generative AI and LLMs, the platform processes the data structure and requirements, and instantly generates Mage.ai pipeline blocks.

These blocks are based on pre-defined templates for tasks like data cleaning, transformation, anomaly detection, and prediction, all while allowing the flexibility to adapt to any dataset. You no longer must start from scratch.

As a less experienced user, all you need is a brief guiding prompt. The system understands the context of the data and the desired outcome, and it provides a workflow that’s ready to run.

Democratizing Data Engineering

Data engineering has often been a domain reserved for those with extensive technical know-how. With the introduction of our Generative AI-powered Data Orchestrator, this is no longer the case. By reducing the complexity and time involved in configuring ETL pipelines, we’re empowering organizations to:

1. Accelerate time-to-value. With AI doing most of the setup work, teams can focus on what truly matters—extracting insights from their data, not configuring workflows.
2. Reduce the learning curve. No more spending weeks learning the intricacies of ETL processes. With our platform, even unexperienced users can be up and running in no time.
3. Produces customizable workflows. While the platform provides default templates, advanced users still have the flexibility to customize their pipelines to meet more complex or specific requirements.

What’s Next?

With the launch of this new capability, we’re excited to see how businesses will leverage it to innovate. Whether you’re building predictive models, automating anomaly detection, or simply making data-driven decisions faster, the Sedimark Data Orchestrator simplifies every step of the process.

Last week, we had the privilege of meeting with our partners at LINKS Foundation in Torino for a productive General Assembly meeting.

Over the course of two days, we engaged in in-depth discussions and collaborative efforts to define the next steps as we approach the final year of SEDIMARK project.

Thanks to the hard work and dedication of the entire team, we’ve made significant progress and are excited about what’s to come! 💡 Stay tuned for more updates as we continue working towards a decentralized AI-enabled marketplace!

In today’s data-driven world, finding relevant datasets is crucial for researchers, data scientists and businesses. This has led to the development of dataset recommendation systems. Similarly, as the movie recommendation system used by Netflix guiding users to discover the most relevant movies, the dataset recommender system aims to guide users to navigate this complex landscape of dataset discovery efficiently. 

Recommender systems learn to analyse user behaviour to make intelligent suggestions. This can be achieved with a variety of techniques, ranging from content-based filtering approaches that focus on the item descriptions and recommend similar items to those that the user has previously interacted with; through collaborative filtering approaches that recommend items based on interactions of similar users; to hybrid approaches combining the content and collaborative filtering. 

High-quality recommender systems provide many benefits to users such as efficiency by automating the search for relevant items, personalisation improving user satisfaction and enhanced discovery exposing users to items they may not be aware of.

It is clear that an efficient recommender system has many advantages in various domains and dataset recommendation is no different. However, with the exponential growth of data, often residing at various locations, finding the right dataset for a specific task or project has become increasingly challenging [1]. Moreover, numerous datasets lack high-quality descriptions making the discovery even harder [2]. This is particularly important for content-based recommender systems as they rely on high-quality metadata. Therefore insufficient dataset metadata information brings challenges associated with effective dataset recommendations, as high-quality recommendations rely on high-quality metadata information. 

In SEDIMARK, we aim to address the challenge of poor quality metadata in dataset recommendation with the development of novel techniques for dataset metadata enrichment. With automatic and efficient metadata enrichment, SEDIMARK can improve the overall user experience and dataset discoverability and drive better decision-making for the future.

[1] Chapman, Adriane, et al. "Dataset search: a survey." The VLDB Journal 29.1 (2020): 251-272.

[2] Reis, Juan Ribeiro, Flavia Bernadini, and Jose Viterbo. "A new approach for assessing metadata completeness in open data portals." International Journal of Electronic Government Research (IJEGR) 18.1 (2022): 1-20.