The role of SKIM in DaFab system architecture

Introduction: Turning a Copernicus data scale archive into something you can search

Copernicus has grown into an archive where the limiting factor is rarely the availability of pixels, but the ability to find the right ones. To do so, users still have to start from copernicus product descriptors (e.g. Sentinel-2 tile, date, processing level, etc…) and only later test whether the data contains the signal of interest. DaFab system addresses this gap by generating secondary, AI‑derived metadata at scale and exposing it as a discovery surface, so that users can begin with a thematic question instead of beginning with file selection (e.g. “How many agriculture parcels are there ?” in smart-agriculture thematic and “Where can I find water anomalies ?” for water-analysis one).

Usability, Impact, and a New Horizon for DaFab and the Global Rucio Community

Over the past year, the DaFab project has become a catalyst for the evolution of the Rucio data management system. While initially designed to support the ATLAS experiment at Cern, today Rucio serves a far wider community of scientific collaborations with complex data needs. The DaFab initiative, centered on extracting value from massive Copernicus Earth Observation archives, has pushed Rucio into new territory, beyond file cataloguing and distributed data placement, and into the realm of rich semantic metadata and powerful filtering.

Introduction

In the realm of both scientific research and production environments, efficiently managing and utilizing metadata is crucial. Metadata serves as the backbone for data discovery, organization, and retrieval, enabling effective data usage across various fields. This is particularly important in areas like Earth Observation (EO), where vast amounts of satellite data need to be processed and analysed to monitor and understand our planet.

The DaFab project, an ambitious initiative, aims to enhance the exploitation of Copernicus data through advanced AI and High-Performance Computing (HPC) technologies. By integrating these technologies, DaFab seeks to improve the timeliness, accuracy, and accessibility of EO data. At the heart of this endeavour lies Rucio, a robust data management system developed by CERN. Rucio’s role is pivotal in achieving key objectives of the project such as creating a unified, searchable catalogue of interlinked EO metadata, improving metadata ingestion and retrieval speeds, and facilitating seamless integration with AI-driven workflows and HPC systems.