At the heart of #SEDIMARK lies a powerful AI architecture, featuring MageAI and MLFlow, designed to revolutionize energy optimization. MageAI orchestrates seamless AI workflows, while MLFlow automates robust machine learning pipelines, enabling accurate and localized energy consumption predictions. Our integrated system supports dynamic customer management strategies, ensuring a transparent, scalable, and efficient solution for energy forecasting.

We leverage capabilities like federated learning, real-time inference, and secure data exchange to enhance the reliability and usability of our system. This AI-driven approach empowers users to make data-driven decisions, fostering a sustainable energy future. Through advanced analytics and actionable insights, we are transforming energy distribution and enhancing customer experiences.

We are poised to optimize energy management for the future, making waves in energy efficiency. Learn how #SEDIMARK is leading the charge in #AIforEnergy, #DataDriven, and #EnergyInnovation. #METLEN #AI #Sustainability #Innovation #EnergyEfficiency

The future of urban mobility is increasingly shaped by data, driving smarter, more sustainable, and highly efficient cities. As urban centers expand, they face the pressing challenge of managing traffic congestion while ensuring accessible and environmentally friendly transportation solutions for all citizens. By integrating cutting-edge technologies, cities can now monitor traffic in real time and implement strategic interventions that improve safety, reduce emissions, and enhance overall mobility.

One of the primary catalysts behind this transformation is the ability to collect and analyze vast amounts of mobility data. This data-driven approach influences travel behaviors, optimizes transportation networks, and ensures equitable mobility benefits. A prime example of this innovation is the ACUMEN pilot project in Helsinki, where real-time data sharing is facilitated through advanced multi-layered digital twins. These digital representations of the city’s transportation ecosystem allow various systems and service providers to collaborate seamlessly, fostering a dynamic and responsive urban mobility environment.

A significant trend within this data-driven revolution is the emergence of secure, intelligent data-sharing platforms. One such example is the SEDIMARK pilot project, which has introduced a secure mobility data marketplace. The pilot is implemented as a minimum viable system and utilizes Helsinki’s mobility digital twins to create a highly adaptive model.

This marketplace is designed to provide city developers with real-time, actionable data, enabling data-driven decision-making for urban development, leading to more efficient and sustainable urban solutions—not only for mobility but also for infrastructure and condition management.

* image from Vesa Laitinen (c) Forum Virium Helsinki

As technology continues to advance, we are witnessing a transformation in how we approach problem-solving. In particular, artificial intelligence (AI) is reshaping the landscape of urban innovation, empowering individuals and communities to collaborate in creating smarter, more inclusive cities.

The Evolution of Problem-Solving Tools

In the past, challenges were tackled by specialists working in silos, relying on closed processes and limited datasets. However, with the advent of AI-powered tools, the decision-making process is becoming increasingly democratized. From machine learning algorithms and predictive analytics to intelligent automation, AI provides real-time insights that are actively shaping urban development.

One of the most notable shifts is how AI is enabling urban planners to analyze complex datasets, such as traffic patterns, infrastructure needs, and public service optimization. AI-assisted platforms are also fostering direct community engagement, with interactive dashboards and smart simulations encouraging citizens to actively participate in shaping their urban environment. Through methods like SEDIMARK data spaces and AI, we can ensure the reliability and integrity of data, which is crucial in building trust and making informed decisions. The ACUMEN pilot project uses an AI tool to promote seamless mobility, contributing significantly to the advancement of urban development.

Empowering Everyday Problem-Solvers

AI is no longer limited to data scientists or engineers. With the rise of user-friendly AI-powered tools, individuals from all walks of life are now empowered to participate in innovation. Whether it’s through AI-driven tourist assistance via chatbots, citizen-driven urban design platforms, or AI-powered tools that help communities anticipate environmental challenges, the accessibility of these tools has opened the world of innovation to a broader audience.

By breaking down barriers to information and expertise, AI fosters a collaborative environment where diverse perspectives can come together to address the challenges of urban life. This inclusivity is vital in building cities that serve the needs of all residents—not just those with specialized knowledge.

The Synergy Between AI and Human Ingenuity

While AI undoubtedly enhances problem-solving capabilities, human ingenuity remains central to the process. AI augments our ability to analyze complex data and scenarios, but creativity, ethical reasoning, and emotional intelligence continue to drive the best solutions. The future of urban innovation lies in finding harmony between AI’s computational power and human intuition, ensuring that technology becomes a tool that amplifies human agency rather than replacing it.

* image from Outi Neuvonen (c) Helsinki Partners

In the fast-evolving world of data science and AI, ensuring that workflows are reproducible, portable, and scalable is essential for success. However, many modern tools prioritize ease of use over standardization, making it difficult to share and execute workflows across different environments.

The SEDIMARK team tackled this challenge by creating a transformation methodology to convert Mage.ai workflows into Common Workflow Language (CWL) and Python-based workflows. This work aims to ensure compatibility with industry standards, improving the portability, reproducibility, and scalability of data pipelines. By integrating standardized workflows, SEDIMARK enables organizations to confidently manage their AI pipelines in secure, decentralized environments.

In the fast-evolving world of data science and AI, ensuring that workflows are reproducible, portable, and scalable is essential for success. However, many modern tools prioritize ease of use over standardization, making it difficult to share and execute workflows across different environments.

Why transform Mage.ai Workflows to CWL?

Mage.ai is a powerful tool for building data pipelines quickly and easily. However, its native workflows are not compatible with industry-standard formats like CWL, which are essential for reproducibility and portability.

CWL (Common Workflow Language) is an open standard that ensures workflows can be shared, reproduced, and executed across different platforms. It’s widely used in fields like bioinformatics and data science to standardize workflows for deployment in cloud environments, HPC clusters, and edge computing platforms.

By converting Mage.ai pipelines to CWL, organizations participating in SEDIMARK can achieve:

• Reproducibility by ensuring that workflows produce consistent results across different systems.
• Portability by allowing workflows to be moved between local machines, cloud platforms, and decentralized environments.
• Interoperability through seamlessly integration with other tools and platforms, such as Apache Airflow or Nextflow.

How SEDIMARK achieved the transformation

The SEDIMARK team developed a two-step methodology to transform Mage.ai pipelines into standardized workflows:

1. Mage to Python Transformation:
   The first step is converting Mage blocks into standalone Python scripts, ensuring compatibility with CWL and other execution environments. This involves:
   • Removing Mage-specific dependencies.
   • Formatting the code for Python standards.
   • Ensuring the Python scripts can be executed independently.
2. Python to CWL Transformation:
   Once the Mage blocks are converted to Python, they are then wrapped in CWL workflows. This process involves:
   • Creating CWL tool definitions for each Python script.
   • Assembling a complete CWL workflow that links the individual steps.
   • Generating a ZIP package containing all the necessary files for workflow execution.

This transformation enables organizations to leverage SEDIMARK’s decentralized marketplace while ensuring that their data pipelines remain compatible with industry standards.

The SEDIMARK team developed a two-step methodology to transform Mage.ai pipelines into standardized workflows:

1. Mage to Python Transformation:
   The first step is converting Mage blocks into standalone Python scripts, ensuring compatibility with CWL and other execution environments. This involves:
   • Removing Mage-specific dependencies.
   • Formatting the code for Python standards.
   • Ensuring the Python scripts can be executed independently.
2. Python to CWL Transformation:
   Once the Mage blocks are converted to Python, they are then wrapped in CWL workflows. This process involves:
   • Creating CWL tool definitions for each Python script.
   • Assembling a complete CWL workflow that links the individual steps.
   • Generating a ZIP package containing all the necessary files for workflow execution.

This transformation enables organizations to leverage SEDIMARK’s decentralized marketplace while ensuring that their data pipelines remain compatible with industry standards.

Why this matters for SEDIMARK and beyond

The SEDIMARK project aims to establish a distributed data marketplace, where organizations can securely exchange data pipelines, AI models, and other digital assets. The transformation from Mage.ai to CWL ensures that these assets are portable, reproducible, and compatible with existing standards.

For data scientists, engineers, and organizations participating in SEDIMARK, this transformation bridges the gap between intuitive pipeline design and standardized execution, enabling scalable and secure data processing.

1. Introduction

As part of the SEDIMARK toolbox that Users will use for configuring AI and Data Processing pipelines for their use case, AI tasks such as forecasting are made readily available for inferencing on Data Assets. Time series forecasting has a wide range of applications across various fields, including financial market prediction, weather forecasting, and traffic flow prediction.

In this tutorial, we will use Python to demonstrate the basic AI workflow for time series forecasting, specifically focusing on temperature forecasting for agriculture use cases. Accurate temperature forecasting is crucial for agriculture as it helps farmers plan their activities, manage crops, and optimize yields.

The Jupyter notebook that contains the content of this tutorial can be downloaded from Github.

2. Environment Setup

We need to install some toolboxes and libraries for this experiment. Therefore, please copy and use the below command in your python terminal:

pip install numpy pandas matplotlib scikit-learn torch

3. Data Preprocessing

In this section, we generate the simulation data and apply the preprocessing.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.preprocessing import MinMaxScaler

# Generate sample data
date_rng = pd.date_range(start='2023-01-01', end='2023-06-30', freq='D')
df = pd.DataFrame(date_rng, columns=['date'])
df['temperature'] = np.random.randint(20, 35, size=(len(date_rng)))

# Set date as index
df.set_index('date', inplace=True)

# Visualize data
df['temperature'].plot(figsize=(12, 6), title='Temperature Time Series')
plt.show()

# Normalize data
scaler = MinMaxScaler(feature_range=(0, 1))
df['temperature_scaled'] = scaler.fit_transform(df['temperature'].values.reshape(-1, 1))

# Split into training and testing sets
train_size = int(len(df) * 0.8)
train, test = df[:train_size], df[train_size:]

# Create dataset for Transformer
def create_dataset(data, time_step=1):
    X, Y = [], []
    for i in range(len(data) - time_step - 1):
        X.append(data[i:(i + time_step), 0])
        Y.append(data[i + time_step, 0])
    return np.array(X), np.array(Y)

time_step = 10
X_train, y_train = create_dataset(train['temperature_scaled'].values, time_step)
X_test, y_test = create_dataset(test['temperature_scaled'].values, time_step)

# Convert to PyTorch tensors
import torch
X_train = torch.tensor(X_train.reshape(X_train.shape[0], time_step, 1), dtype=torch.float32)
y_train = torch.tensor(y_train, dtype=torch.float32)
X_test = torch.tensor(X_test.reshape(X_test.shape[0], time_step, 1), dtype=torch.float32)
y_test = torch.tensor(y_test, dtype=torch.float32)

4. Build the Simple Transformer Model

We use the toolbox and librires support provided by Pytorch to create a simple and basic Transformer model (Encoder-Decoder).

import torch.nn as nn
import torch.optim as optim

class TransformerModel(nn.Module):
    def __init__(self, num_heads, d_model, num_encoder_layers, num_decoder_layers, dff):
        super(TransformerModel, self).__init__()
        self.encoder_layer = nn.TransformerEncoderLayer(d_model=d_model, nhead=num_heads, dim_feedforward=dff)
        self.transformer_encoder = nn.TransformerEncoder(self.encoder_layer, num_layers=num_encoder_layers)
        self.decoder_layer = nn.TransformerDecoderLayer(d_model=d_model, nhead=num_heads, dim_feedforward=dff)
        self.transformer_decoder = nn.TransformerDecoder(self.decoder_layer, num_layers=num_decoder_layers)
        self.flatten = nn.Flatten()
        self.dense1 = nn.Linear(d_model * time_step, dff)
        self.dense2 = nn.Linear(dff, 1)

    def forward(self, src):
        encoder_output = self.transformer_encoder(src)
        decoder_output = self.transformer_decoder(encoder_output, encoder_output)
        flatten_output = self.flatten(decoder_output)
        dense_output = self.dense1(flatten_output)
        output = self.dense2(dense_output)
        return output

# Hyperparameters
num_heads = 2
d_model = 64
num_encoder_layers = 2
num_decoder_layers = 2
dff = 128

# Create model
model = TransformerModel(num_heads, d_model, num_encoder_layers, num_decoder_layers, dff)
criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

# Train model
num_epochs = 50
batch_size = 64
train_loader = torch.utils.data.DataLoader(dataset=list(zip(X_train, y_train)), batch_size=batch_size, shuffle=True)

for epoch in range(num_epochs):
    model.train()
    for batch_X, batch_y in train_loader:
        optimizer.zero_grad()
        outputs = model(batch_X)
        loss = criterion(outputs.squeeze(), batch_y)
        loss.backward()
        optimizer.step()
    print(f'Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item():.4f}')

5. Model Evaluation

We evaluate our trained model on the created data.

import math
from sklearn.metrics import mean_squared_error

model.eval()
with torch.no_grad():
    train_predict = model(X_train).squeeze().numpy()
    test_predict = model(X_test).squeeze().numpy()

# Inverse transform the predictions
train_predict = scaler.inverse_transform(train_predict.reshape(-1, 1))
test_predict = scaler.inverse_transform(test_predict.reshape(-1, 1))
y_train = scaler.inverse_transform(y_train.reshape(-1, 1))
y_test = scaler.inverse_transform(y_test.reshape(-1, 1))

# Calculate RMSE
train_score = math.sqrt(mean_squared_error(y_train, train_predict))
test_score = math.sqrt(mean_squared_error(y_test, test_predict))
print(f'Train Score: {train_score} RMSE')
print(f'Test Score: {test_score} RMSE')

# Visualize predictions
plt.figure(figsize=(12, 6))
plt.plot(df['temperature'], label='Actual Data')
plt.plot(df.index[time_step:train_size], train_predict, label='Train Predict')
plt.plot(df.index[train_size+time_step+1:], test_predict, label='Test Predict')
plt.legend()
plt.show()

6. Conclusion

This tutorial demonstrates how to use a basic Transformer model for time series forecasting, specifically for temperature prediction in agriculture. Accurate temperature forecasting is essential for agricultural planning and decision-making, helping farmers optimize crop management and improve yields. Through this example, readers can gain a fundamental understanding of applying Transformers to time series forecasting and further research and optimize the model for better prediction performance.

In today’s fast-paced digital landscape, effective data processing is a critical component for any organization looking to derive insights and drive innovation. However, setting up data pipelines - from extraction to transformation and loading (ETL) - has traditionally required a high level of expertise. We’re happy to announce a solution that could democratize data orchestration for users of all experience levels: the SEDIMARK Data Orchestrator powered by Mage.ai and enhanced with Generative AI.

Simplified Data Processing with AI Assistance

Our platform integrates the power of Large Language Models (LLMs) to automatically generate Mage.ai pipeline blocks, helping even those with minimal technical background create robust data workflows. Instead of spending hours - or even days - writing code and configuring pipelines, users now simply need to upload their dataset into the Orchestrator GUI.

Once the dataset is in place, the system, with a bit of guidance through a helpful prompt, takes over the heavy lifting. Using generative AI, the platform produces custom Mage.ai templates and workflows specifically tailored to your data. This eliminates the need for users to dive deep into code or ETL specifics.

How It Works

Whether you’re dealing with traffic provided data, weather records, or IoT data streams, the process starts with uploading your dataset into the Orchestrator GUI.

With the help of generative AI and LLMs, the platform processes the data structure and requirements, and instantly generates Mage.ai pipeline blocks.

These blocks are based on pre-defined templates for tasks like data cleaning, transformation, anomaly detection, and prediction, all while allowing the flexibility to adapt to any dataset. You no longer must start from scratch.

As a less experienced user, all you need is a brief guiding prompt. The system understands the context of the data and the desired outcome, and it provides a workflow that’s ready to run.

Democratizing Data Engineering

Data engineering has often been a domain reserved for those with extensive technical know-how. With the introduction of our Generative AI-powered Data Orchestrator, this is no longer the case. By reducing the complexity and time involved in configuring ETL pipelines, we’re empowering organizations to:

1. Accelerate time-to-value. With AI doing most of the setup work, teams can focus on what truly matters—extracting insights from their data, not configuring workflows.
2. Reduce the learning curve. No more spending weeks learning the intricacies of ETL processes. With our platform, even unexperienced users can be up and running in no time.
3. Produces customizable workflows. While the platform provides default templates, advanced users still have the flexibility to customize their pipelines to meet more complex or specific requirements.

What’s Next?

With the launch of this new capability, we’re excited to see how businesses will leverage it to innovate. Whether you’re building predictive models, automating anomaly detection, or simply making data-driven decisions faster, the Sedimark Data Orchestrator simplifies every step of the process.

In today’s data-driven world, finding relevant datasets is crucial for researchers, data scientists and businesses. This has led to the development of dataset recommendation systems. Similarly, as the movie recommendation system used by Netflix guiding users to discover the most relevant movies, the dataset recommender system aims to guide users to navigate this complex landscape of dataset discovery efficiently. 

Recommender systems learn to analyse user behaviour to make intelligent suggestions. This can be achieved with a variety of techniques, ranging from content-based filtering approaches that focus on the item descriptions and recommend similar items to those that the user has previously interacted with; through collaborative filtering approaches that recommend items based on interactions of similar users; to hybrid approaches combining the content and collaborative filtering. 

High-quality recommender systems provide many benefits to users such as efficiency by automating the search for relevant items, personalisation improving user satisfaction and enhanced discovery exposing users to items they may not be aware of.

It is clear that an efficient recommender system has many advantages in various domains and dataset recommendation is no different. However, with the exponential growth of data, often residing at various locations, finding the right dataset for a specific task or project has become increasingly challenging [1]. Moreover, numerous datasets lack high-quality descriptions making the discovery even harder [2]. This is particularly important for content-based recommender systems as they rely on high-quality metadata. Therefore insufficient dataset metadata information brings challenges associated with effective dataset recommendations, as high-quality recommendations rely on high-quality metadata information. 

In SEDIMARK, we aim to address the challenge of poor quality metadata in dataset recommendation with the development of novel techniques for dataset metadata enrichment. With automatic and efficient metadata enrichment, SEDIMARK can improve the overall user experience and dataset discoverability and drive better decision-making for the future.

[1] Chapman, Adriane, et al. "Dataset search: a survey." The VLDB Journal 29.1 (2020): 251-272.

[2] Reis, Juan Ribeiro, Flavia Bernadini, and Jose Viterbo. "A new approach for assessing metadata completeness in open data portals." International Journal of Electronic Government Research (IJEGR) 18.1 (2022): 1-20.

Have you ever wondered how a smart city manages to keep everything from urban planning to environmental monitoring running smoothly? The answer lies in something called Spatial Data Infrastructure (SDI). While it might sound technical, SDI framework plays a crucial role in making geographic information accessible and integrated, benefiting everyone.

Imagine a world where data about locations – from urban planning maps to environmental monitoring systems – is at your fingertips. SDI turns this vision into reality. By connecting data, technology, and people, SDI helps improve decision-making and efficiency in numerous areas of our lives.

Smart City: SEDIMARK Helsinki Pilot and Spatial Data

The SEDIMARK Helsinki pilot aims to demonstrate how Digital Twin technology can revolutionize urban mobility with spatial data as the backbone. SEDIMARK's context broker (NGSI-LD) handles linked data, property graphs, and semantics using three main constructs: Entities, Properties, and Relationships. This integration opens up opportunities for new services and the development of a functional city, aiming to enhance geospatial data integration within urban digital twins. In Helsinki, the approach focuses on transitioning from a monolithic architecture to a modular, API-driven approach, developing Digital Twin viewers and tools, and collaborating on a city-wide Geospatial Data.

Join us on this journey as we dive into the world of Spatial Data Infrastructure and see how it's making our city smarter, more efficient, and better prepared for the future.

Photo credit. https://materialbank.myhelsinki.fi/images/attraction?sort=popularity&openMediaId=6614

When we think of data, especially from diverse traffic sources, beauty isn't typically the first thing that comes to mind. Instead, we imagine numbers, graphs, and charts, all designed to convey information quickly and efficiently. However, what if we could see data not just as a tool for analysis, but as a source of inspiration, capable of producing visuals as captivating as a masterpiece by Vincent van Gogh? Just like van Gogh's "Starry Night" finds beauty in complexity and chaos, we can render data into beautiful, meaningful visualizations.

The Complexity of Traffic Data

Traffic data is inherently complex. It comes from a variety of sources of interoperable systems and devices. Each source provides a different perspective, capturing the flow of vehicles, the density of traffic, and the speed of travel at any given time. When combined, these data points create a comprehensive picture of urban movement.

From Chaos to Clarity

Much like the seemingly chaotic yet harmonious art, raw traffic data can appear overwhelming. However, through careful visualization and simulation, patterns and insights emerge. Advanced algorithms process the data, identifying trends and correlations that aren't immediately apparent. For instance, heat maps can show areas of high congestion, while flow diagrams can illustrate the movement of vehicles through a city over time.

The beauty of data

Data visualization is an art form in its own right. The choice of colors, shapes, and lines can transform a simple graph into a work of art. For example, a time-lapse visualization of traffic flow can resemble the dynamic motion in an urban city with streams of vehicles.

Helsinki mobility digital twin

Helsinki mobility digital twin paves the way for a future where cities leverage data. This data-driven revolution, fueled by powerful data visualization, holds immense potential for creating a more efficient, sustainable, and safer urban transportation landscape.

So, can traffic data be beautiful? Absolutely. All it takes is the right perspective and a touch of creativity to turn numbers into a work of art.

Photo credit: Kuva.