The role of SKIM in DaFab system architecture

Introduction: Turning a Copernicus data scale archive into something you can search

Copernicus has grown into an archive where the limiting factor is rarely the availability of pixels, but the ability to find the right ones. To do so, users still have to start from copernicus product descriptors (e.g. Sentinel-2 tile, date, processing level, etc…) and only later test whether the data contains the signal of interest. DaFab system addresses this gap by generating secondary, AI‑derived metadata at scale and exposing it as a discovery surface, so that users can begin with a thematic question instead of beginning with file selection (e.g. “How many agriculture parcels are there ?” in smart-agriculture thematic and “Where can I find water anomalies ?” for water-analysis one).

Usability, Impact, and a New Horizon for DaFab and the Global Rucio Community

Over the past year, the DaFab project has become a catalyst for the evolution of the Rucio data management system. While initially designed to support the ATLAS experiment at Cern, today Rucio serves a far wider community of scientific collaborations with complex data needs. The DaFab initiative, centered on extracting value from massive Copernicus Earth Observation archives, has pushed Rucio into new territory, beyond file cataloguing and distributed data placement, and into the realm of rich semantic metadata and powerful filtering.

Introduction

In the realm of both scientific research and production environments, efficiently managing and utilizing metadata is crucial. Metadata serves as the backbone for data discovery, organization, and retrieval, enabling effective data usage across various fields. This is particularly important in areas like Earth Observation (EO), where vast amounts of satellite data need to be processed and analysed to monitor and understand our planet.

The DaFab project, an ambitious initiative, aims to enhance the exploitation of Copernicus data through advanced AI and High-Performance Computing (HPC) technologies. By integrating these technologies, DaFab seeks to improve the timeliness, accuracy, and accessibility of EO data. At the heart of this endeavour lies Rucio, a robust data management system developed by CERN. Rucio’s role is pivotal in achieving key objectives of the project such as creating a unified, searchable catalogue of interlinked EO metadata, improving metadata ingestion and retrieval speeds, and facilitating seamless integration with AI-driven workflows and HPC systems.

In our previous DaFab post, we introduced the overall multi-site orchestration vision. This entry focuses on a specific architectural building block: integrating Karmada with Argo Workflows to enable multi-cluster and multi-site workflow execution driven by rule-based placement. The key outcome is that workflow steps can be dispatched to different Kubernetes clusters (sites) based on explicit rules that reflect data locality and computing resource availability goals, without changing how users author workflows.

Navigating Strategic Pivots and Technical Milestones

DaFab has now reached half of its length, an ideal time to look back in the rear-view mirror and to reassess our roadmap. After 18 month of work we can claim substantial advancements across DaFab work packages, demonstrating agility and a solid technical foundation, despite facing initial challenges such as delayed deliverables and staffing issues. The project has successfully established a robust management structure, implemented a significant technological pivot, and achieved initial technical milestones, laying the groundwork for its full deployment.

Data economy is an ever-increasing market. Thanks to the Copernicus constellation Europe has amassed a considerable amount of Earth Observation data already used by technology-intensive SMEs and large industry in several sectors of the economy. However, the management of data at scale remains challenging, and lowering the entry cost to exploitation of Copernicus data is a key element to foster innovation and fuel this data economy. #DaFab is a new project, launched with the support of the European Space Agency: https://www.euspa.europa.eu/dafab-ai-factory-copernicus-data-scale

Usability, Impact, and a New Horizon for DaFab and the Global Rucio Community

Over the past year, the DaFab project has become a catalyst for the evolution of the Rucio data management system. While initially designed to support the ATLAS experiment at Cern, today Rucio serves a far wider community of scientific collaborations with complex data needs. The DaFab initiative, centered on extracting value from massive Copernicus Earth Observation archives, has pushed Rucio into new territory, beyond file cataloguing and distributed data placement, and into the realm of rich semantic metadata and powerful filtering.

Introduction

In the realm of both scientific research and production environments, efficiently managing and utilizing metadata is crucial. Metadata serves as the backbone for data discovery, organization, and retrieval, enabling effective data usage across various fields. This is particularly important in areas like Earth Observation (EO), where vast amounts of satellite data need to be processed and analysed to monitor and understand our planet.

The DaFab project, an ambitious initiative, aims to enhance the exploitation of Copernicus data through advanced AI and High-Performance Computing (HPC) technologies. By integrating these technologies, DaFab seeks to improve the timeliness, accuracy, and accessibility of EO data. At the heart of this endeavour lies Rucio, a robust data management system developed by CERN. Rucio’s role is pivotal in achieving key objectives of the project such as creating a unified, searchable catalogue of interlinked EO metadata, improving metadata ingestion and retrieval speeds, and facilitating seamless integration with AI-driven workflows and HPC systems.

Workflows processing Earth Observation (EO) data have a problem – the body of available EO data is vast. And growing rapidly. Within the DaFab EU project, AI-driven workflows must process massive quantities of EO data, made available by the Copernicus project, in an efficient and reliable manner. This presents a range of problems, including locating the relevant data, decoupling relatively fast and scalable compute tasks from slower data transfers, storing the data in a way that the workflows can use it, and managing the lifetime of any temporary copies required. This is where DASI (the Data Access and Storage Interface) plays a critical role. It provides the smart bridge between storage systems and compute environments. DASI’s semantically driven data management design helps build intelligent, scalable, and optimized AI workflows in the DaFab project.

Workflows processing Earth Observation (EO) data have a problem – the body of available EO data is vast. And growing rapidly. Within the DaFab EU project, AI-driven workflows must process massive quantities of EO data, made available by the Copernicus project, in an efficient and reliable manner. This presents a range of problems, including locating the relevant data, decoupling relatively fast and scalable compute tasks from slower data transfers, storing the data in a way that the workflows can use it, and managing the lifetime of any temporary copies required. This is where DASI (the Data Access and Storage Interface) plays a critical role. It provides the smart bridge between storage systems and compute environments. DASI’s semantically driven data management design helps build intelligent, scalable, and optimized AI workflows in the DaFab project.

Bridging Cloud-Native Workflows and HPC for DaFab

One of the main goals of the DaFab project is to enable seamless, multisite scientific workflows—without the need to physically move data between sites. This is crucial in environments where data transfer is restricted by bandwidth, administrative domains, or security policies. One of the main DaFab’s goals is to orchestrate distributed workflows across multiple HPC and cloud sites, letting data stay in place while computation moves to where it’s needed.

Introduction

The delineation of agricultural field boundaries from satellite imagery is crucial for precision agriculture, land management, policymaking and crop monitoring.

It can be a stand-alone solution or as the first stage for crop classification, advanced soil moisture analysis, yield forecast, the estimation of damages to crop yield due to natural (e.g. drought, floods) and more.

In the DaFab project, field delineation, which involves outlining agricultural parcels with polygons, offers a solution to several difficulties encountered in yield forecasting and crop classification tasks. This approach is more effective than traditional pixel-by-pixel classification maps because of the reasons:

Multi-Site Workflow Orchestration in the DAFAB Project

DAFAB will design and implement a workflow orchestration system that enhances multi-site application deployment and data discovery. The workflow system will enable applications to express their computations as a graph and declare the data needed at each step in a high-level query language. Workflows will then execute across multiple sites, whether cloud-based (Kubernetes) or high-performance computing (Slurm) environments. By enabling transparent data access and seamless execution of workflow stages, this system shifts the burden of application deployment and data discovery to the platform itself, significantly accelerating development timelines.