Usability, Impact, and a New Horizon for DaFab and the Global Rucio Community

Over the past year, the DaFab project has become a catalyst for the evolution of the Rucio data management system. While initially designed to support the ATLAS experiment at Cern, today Rucio serves a far wider community of scientific collaborations with complex data needs. The DaFab initiative, centered on extracting value from massive Copernicus Earth Observation archives, has pushed Rucio into new territory, beyond file cataloguing and distributed data placement, and into the realm of rich semantic metadata and powerful filtering.

Workflows processing Earth Observation (EO) data have a problem – the body of available EO data is vast. And growing rapidly. Within the DaFab EU project, AI-driven workflows must process massive quantities of EO data, made available by the Copernicus project, in an efficient and reliable manner. This presents a range of problems, including locating the relevant data, decoupling relatively fast and scalable compute tasks from slower data transfers, storing the data in a way that the workflows can use it, and managing the lifetime of any temporary copies required. This is where DASI (the Data Access and Storage Interface) plays a critical role. It provides the smart bridge between storage systems and compute environments. DASI’s semantically driven data management design helps build intelligent, scalable, and optimized AI workflows in the DaFab project.

Bridging Cloud-Native Workflows and HPC for DaFab

One of the main goals of the DaFab project is to enable seamless, multisite scientific workflows—without the need to physically move data between sites. This is crucial in environments where data transfer is restricted by bandwidth, administrative domains, or security policies. One of the main DaFab’s goals is to orchestrate distributed workflows across multiple HPC and cloud sites, letting data stay in place while computation moves to where it’s needed.