Optimising the data path for Earth-observation inference at scale

Whether it’s for monitoring floods, assessing agricultural conditions, or detecting wildfires, every Earth-Observation AI system based on satellite remote-sensing data shares a common task: given the area of interest and a time window, find the products that cover it and turn them into model inputs.

Intuitively, a live discovery approach appears attractive: products are discovered, retrieved, and prepared as part of each model run, avoiding the complexity of maintaining a separate data layer.

In reality, a range of problem appear. The original data are packaged in products that do not map to the regions we want to process on demand, and contain many unneeded image. Transfers are unreliable, and slow, which provides a bottleneck to the overall process. And the system interacts with permissions, quotas and preprocessing in complex ways. Beyond this, during development or for operational reasons, parts of the analysis may need to be repeated. As a result, treating a public catalogue and data store as the live data path is a poor choice.

DaFab is no exception. To understand how these challenges are addressed in the project, it helps to first look at what the Copernicus satellite imagery required by the project workflows actually looks like.

From satellite products to model inputs

A Sentinel-2 product is not one model-ready input but a package of assets — spectral bands, scene-classification layers, metadata — bundled into a tile of roughly 100 by 110 kilometres, more than twenty assets, about a gigabyte. A workflow needs only a few: DaFab’s field-boundary workflow uses five layers (red, green, blue, near-infrared, and the Scene Classification Layer); a water-analysis workflow uses a different set. Neither needs “a Sentinel-2 product” in the abstract — each needs selected assets, prepared for the model.

Fetching the whole product to use five layers is wasted movement, and a live lookup pays that waste again on every run: locate, transfer, retry, extract, prepare. The data path then scales with the number of runs, not the number of distinct inputs. A run over 100 tiles moves about 100 GB; repeating the same area ten times while tuning a model moves roughly a terabyte, before another application even asks for the same imagery — and the public archives are not built for this, with their rate limits, failures, and changing endpoints. What is needed is a system that fetches only what is missing, and does not treat every run as if it exists in isolation.

Solving the data path with shared asset preparation

DaFab addresses this by changing the level at which workflows ask for data. Instead of treating the full Sentinel-2 product as the unit to fetch and prepare, a workflow names the product and only the assets it needs from it — moving the system from product-level, per-run fetching to asset-level preparation.

DASI, DaFab’s Data Access and Storage Interface, holds those prepared assets and serves them by semantic metadata rather than by file path. But Copernicus does not speak that language: it exposes whole products through public STAC catalogues and S3 endpoints. Bridging the two is the job of data-tools.

data-tools sits between the public Copernicus archives and the internal DASI infrastructure (Figure 2). It does two things. First, the selective extraction that makes asset-level access possible — pulling only the named bands and turning a roughly one-gigabyte bulk product into targeted per-asset packages. Second, it translates the request from the names a workflow uses into the names DASI stores by: a workflow asks using public Copernicus identifiers — a product ID and a list of asset names — and data-tools turns these into the internal metadata keys DASI is addressed by. A request that arrives as S2C_MSIL2A_…_T01UBS_… with asset_names=B04_10m leaves as a key DASI can look up, store under, and reuse.

The important point is that the unit reused is the asset, not the product. Prepared bands sit in the DASI store, where each run draws the subset it needs and ingests only what the store does not already hold. Field delineation, water analysis, and downstream steps all read the same prepared bands instead of keeping private stores or pulling the same Sentinel-2 assets again. As an implementation detail, DASI can also stream stored assets directly into an application without requiring temporary files on disk, further reducing the overhead between retrieval and inference.

The effect is to decouple the cost of inference from the cost of moving data. DaFab still fetches what it has not seen before, but only once: adding a model, re-running a region, or bringing another kind of inference onto the same imagery reuses assets that are already prepared. The same applies to timing. Because the workflows meet at the store rather than at the archive, the transfer and the processing run as two loosely coupled workflows, rather than one spinning while it waits on the other. The slow step, almost always the data transfer, runs maximally, and processing can be dispatched as data becomes available. Data movement then scales with the number of distinct inputs rather than the number of runs, so the inference DaFab runs over Copernicus data can grow without the data movement growing with it.

• Tech
• Earth
• Workflows
• DASI
• Data Flow
• Data Management