Navigating Strategic Pivots and Technical Milestones

DaFab has now reached half of its length, an ideal time to look back in the rear-view mirror and to reassess our roadmap. After 18 month of work we can claim substantial advancements across DaFab work packages, demonstrating agility and a solid technical foundation, despite facing initial challenges such as delayed deliverables and staffing issues. The project has successfully established a robust management structure, implemented a significant technological pivot, and achieved initial technical milestones, laying the groundwork for its full deployment.

Among the Key Strategic and Management Achievements:

• Detailed governance framework: that includes defined processes, regular monthly Executive Board (EB) and General Assembly (GA) meetings, and a structured peer review system for all deliverables.

• Strategic Pivot from Neuromorphic to GPU: A major strategic achievement was the decision to shift the infrastructure effort from initial targets, like neuromorphic architectures, to GPUs (Graphical Processing Units) for AI workloads. This move, documented in Deliverable D2.1, was a proactive response to the market shift where GPUs had become the “clear winners of the architectural battle for AI workloads”.

• Scientific Advisory Board (SAB): A high-profile Scientific Advisory Board has been implemented, with members from the AI, Earth Observation (EO), and High Performance Computing (HPC) communities, to ensure the project maintains the correct balance between these three core aspects.

• Incremental Engineering Methodology: To mitigate integration risks and complexity, the project moved from a granular Agile approach to an incremental engineering methodology with three planned increments, one per year. The first increment focuses on two pilot use cases: smart agriculture (spearheading implementation) and water analysis (at the design stage).

• Project Documentation and Ethics: All project documentation is maintained in a centralized Google Drive repository, and software is stored on a public Github repository. The AI and Data Ethics deliverable (D7.1) was successfully delivered and accepted, undertaken within the WP1 framework.

Regarding the technical progress implemented across the other core Work Packages:

• WP2: AI and Application Co-design
  • This work package focuses on the infrastructure to tackle AI objectives for Earth Observation data.
  • Task 2.1 has been completed and delivered a performance evaluation methodology for AI hardware, focusing on Power, time-to-results, and data movement analysis.
  • The methodology was validated using the open-source NIVA model as a benchmark for the field delineation use case.
  • Task 2.2, the Neurocomputing framework, is ongoing and involves developing and porting performance monitoring, including an extended roofline model (available on CPU, porting to GPU is in progress).
• WP3: Unified Catalog of Interlinked Metadata
  • This work package successfully implemented a unified metadata catalog.
  • The catalog is built upon the CERN-developed RUCIO data management system and supports the STAC (SpatioTemporal Asset Catalog) open specification for describing geospatial and temporal metadata.
• WP4: Data Processing and Orchestration
  • WP4 developed a data-aware workflow orchestration system.
  • It leverages open-source tools like Argo and Kubernetes and has already demonstrated a successful end-to-end workflow execution.
• WP6: Communication, Dissemination and Exploitation
  • Communication efforts have been active, including launching the project website, holding workshops, and conducting a summer school.
  • A significant milestone was the first end-to-end demo of the DaFab Minimum Viable Product, showcased at a workshop held during the HiPEAC conference.

Overall, the project has met its initial milestones, with the first reporting period successfully delivering 8 deliverables . The implemented incremental approach and the strategic hardware shift position DaFab for robust progress in the second half of its cycle.

A key lesson learned during our first reporting period in DaFab is that agility is paramount. We are operating in an extremely fast-paced AI technology landscape, and the data space market is inherently competitive and innovative. Therefore, we cannot emphasize enough that agility is critical, and the quality of the team remains the key to success.

• News