Ai2 Unveils OlmoEarth Platform for Planetary-Scale Geospatial Inference

Ai2 Unveils OlmoEarth Platform for Planetary-Scale Geospatial Inference The Allen Institute for AI (Ai2) has developed the OlmoEarth Platform, an infrastructure system designed to manage large-scale data ingestion, fine-tuning, and inference for Earth observation foundation models. The platform enables organizations to process terabytes of satellite imagery cost-effectively across continent-scale areas. OlmoEarth models are foundation models pretrained on roughly 10 terabytes of multimodal satellite data for applications like deforestation tracking and wildfire risk monitoring. The infrastructure isolates jobs into three distinct stages—data preprocessing (CPU), inference (GPU), and postprocessing (CPU)—to maximize resource efficiency. In a North America wildfire risk run, the platform deployed roughly 19,600 CPUs and 994 GPUs in parallel, achieving a 155× speedup by completing a 4,737-hour serial task in about 30.5 hours. To prevent overwhelming external public STAC APIs, the platform maintains its own continuously updated metadata index using SNS notifications and polling to perform windowed reads on cloud-optimized formats. The execution architecture runs tasks as idempotent, reentrant Docker jobs that dynamically handle provider delays, missing data, and runner crashes. Overcoming the Bottlenecks of Satellite Inference Unlike standard machine learning models that process small inputs in sub-second timelines, geospatial foundation model workloads involve moving terabytes of imagery spanning varied sensor types, spectral bands, time steps, projections, and resolutions. Because jobs frequently spend more time fetching and preparing data than executing model passes, assigning ingestion to GPUs wastes expensive compute. The OlmoEarth Platform addresses this by segregating workflows into three hardware-matched stages: CPU-driven data acquisition and normalization, GPU-focused model inference streaming outputs straight to blob storage, and CPU-driven postprocessing that stitches windows into seamless raster formats like Zarr, GeoTIFF, or GeoJSON. Massive Parallelism via OlmoEarth Run The platform's execution engine, OlmoEarth Run, partitions geographic boundaries into independent compute blocks and subdivides those blocks into windows that can be processed in parallel passes. During a continent-wide wildfire risk analysis across North America, the system fanned out to peak utilization of approximately 19,600 CPUs and 994 GPUs with network throughput surpassing 168 GB/s. This distributed setup reduced roughly 4,737 hours of serial processing to 30.5 hours of wall-clock time. Parallelism, output resolution, model scale, and imagery caching operate as tunable levers balanced against cloud quotas and partner budgets. Optimized Data Retrieval and Fault-Tolerant Execution Because thousands of simultaneous STAC API metadata requests would overwhelm external catalog providers, OlmoEarth maintains an internal metadata index that absorbs upstream data steadily via AWS Open Data SNS alerts or regular polling. During inference, workers execute windowed, range-based reads against cloud-optimized formats such as COG and Zarr to pull only required pixels. The system manages distributed compute failures by running reentrant, idempotent tasks inside containerized virtual machines. If tasks experience API timeouts, missing imagery bands, or outright crashes, automated monitors track failures, invoke alternate providers, and trigger retries. Future Architecture and Roadmap Ai2's planned enhancements include scheduled and event-triggered automated runs, continuous landscape change alerts for events like flooding, and agentic interfaces to help non-specialists curate data and adjust fine-tuned models. The team is also working on faster model architectures, integrating weather inputs like ERA-5, and building precomputed global embeddings to bypass raw imagery forward passes for select applications. While currently operated on Google Cloud, OlmoEarth Run is designed to deploy across multiple clouds or directly inside partner computing environments.

艾伦人工智能研究所推出OlmoEarth平台 实现行星级地理空间推理 艾伦人工智能研究所（Ai2）推出了OlmoEarth平台。该基础设施旨在帮助缺乏工程团队的环保机构弥合技术鸿沟，支持对数十太字节的卫星影像进行低成本的大规模地理空间模型微调、评估与推理。 OlmoEarth基础模型基于约10 TB多模态卫星数据预训练，用于森林砍伐监测、粮食安全和野火风险评估等领域。 平台通过解耦数据获取与预处理（CPU）、推理（GPU）及后处理（CPU）三个阶段，实现硬件配置与工作负载的最优匹配。 在北美野火风险制图任务中，平台峰值并发使用约19,600个CPU和994个GPU，将计算耗时从4,737小时缩短至约30.5小时。 系统通过自建元数据索引缓解公共API查询过载，并利用可重入且幂等的任务机制实现故障自动恢复。 地理空间推理的工程挑战与硬件解耦 与处理兆字节级文本或单张照片的模型不同，地球观测推理涉及跨多谱段、传感器和时间步的多源数据，且下载和对齐耗时往往超过模型运算本身。为避免昂贵的GPU资源浪费在I/O操作上，平台将作业切分为CPU负责的数据获取与预处理、GPU负责的推理，以及CPU负责的拼接与格式导出阶段，保持GPU持续高负荷运转。 大规模分区并行与元数据索引 执行层OlmoEarth Run将大区域分解为独立分区和窗口。针对跨提供商的数据检索痛点，平台维护独立的元数据索引，通过云通知或低频轮询追踪新场景，避免并发突发流量压垮公共STAC服务，并在运行时仅对云优化格式（COG、Zarr）执行局部窗口读取以按需提取像素。 容错机制与未来技术演进 面对网络不可用、波段缺失或云层遮挡等频繁故障，系统为每个任务动态启动Docker容器，通过重试机制、备用数据源切换及健康监控保障可重入和幂等执行。未来，团队计划推进自动化调度、变化检测告警、智能体交互界面、全球预计算嵌入向量以及多云部署支持。

🌍 Learn more about OlmoEarth Platform: https://allenai.org/olmoearth The OlmoEarth models are our family of Earth observation foundation models, pretrained on roughly 10 terabytes of multimodal satellite data. Governments, NGOs, and other mission-driven organizations are already adapting OlmoEarth for applications including deforestation monitoring, food security, and wildfire risk. At Ai2, we know how to train and release powerful open models, and for organizations with strong engineering teams, an open model is all they need to run with. But most organizations in the environmental space – the ones best placed to apply these models – don't have the infrastructure or engineering teams that can manage the full lifecycle: labeling data, fine-tuning models, and running large-scale inference. We’ve spent more than a decade operating platforms like Skylight and EarthRanger , software that users around the world rely on every day, so it has to work every day. That experience taught us what delivering impact takes: running models cost-effectively at the right time and place, monitoring performance, turning raw outputs into actionable insights, and verifying those outputs drive the outcomes partners want. That’s why we built the OlmoEarth Platform : infrastructure for taking geospatial models from fine-tuning and evaluation to large-scale inference. Inference at this scale presents its own set of challenges. Satellite imagery must be found and accessed across multiple providers, aligned across projections and resolutions, and processed efficiently. Results then have to be stitched into geographically consistent maps while the infrastructure recovers from the routine failures of distributed computing. Today, the platform can run inference across continent-scale areas in roughly a day, processing dozens of terabytes of imagery at a cost of fractions of a penny per square kilometer. Developing it meant confronting a series of engineering challenges that others working on large-scale geospatial systems are likely to encounter as well. This post walks through those challenges and the solutions we arrived at. A recent wildfire risk map generated on the OlmoEarth Platform, with statistics. Why satellite inference is challenging Most ML models take in a few megabytes of data and produce a result in under a second—think LLMs processing a paragraph of text or computer vision models analyzing a photo from a smartphone. Earth observation inference operates at a very different scale—a single job fine-tuning a foundation model for maximum performance can move terabytes of data and run for hours. The inputs may span multiple spectral bands, sensor types, and time steps across a large geographic area. They can come from several providers, each using different projections and resolutions, and may include observations that are missing or obscured by clouds. The output is itself a map, so every prediction must remain precisely aligned with the same projection and coordinate grid as the areas around it. Even acquiring the data can be a major challenge. Prediction jobs often spend more time downloading and preparing imagery than running the model itself, making efficient data pipelines critical. Those pipelines must handle high-volume I/O while providing the compute needed to reproject and resample imagery. The right hardware for the right task Because data acquisition and preparation often dominate an inference job’s runtime, assigning that work to GPUs would leave the system’s most expensive hardware doing tasks better suited to CPUs. We therefore divide each job into three stages, each matched to a distinct hardware profile: Data acquisition and preprocessing (CPU, high I/O): Fetch, reproject, align, and normalize imagery, then write it in a format optimized for fast loading during inference. Inference (GPU): Run the model’s forward pass and write minimally processed outputs directly to storage. Postprocessing (CPU): Stitch the per-window outputs together, apply masks or rescaling, and export them in user-friendly formats such as Zarr, GeoTIFF, or GeoJSON. The OlmoEarth Platform distributes these stages across many machines while keeping GPUs fully utilized. Multiprocess data loaders continuously feed each GPU, while completed outputs stream directly to blob storage. One request, hundreds of workers, and thousands of processes OlmoEarth Run, the platform’s execution layer for large-scale inference jobs, divides the geographic region covered by each job into partitions sized for individual compute instances (workers), then subdivides those partitions into smaller windows that the OlmoEarth models process. Because each window can be handled independently in a separate forward pass, work on one part of the map does not need to wait for another. In practice, a state-sized area might become a hundred or so partitions, while a continent-scale run can become thousands. Adjacent partitions overlap slightly, and we reconcile that overlap when the outputs are assembled so no seam appears in the final raster. Because the partitions are independent, the same stage can run across thousands of compute instances at once. We recently used this approach to generate a wildfire-risk map covering all of North America. At peak, the run used roughly 19,600 CPUs and 994 GPUs in parallel, with network throughput exceeding 168 GB/s. That level of parallelism reduced an estimated 4,737 hours of serial compute to about 30.5 hours of wall-clock time—a 155× speedup. Fan-out is not unbounded, though. More workers push against cloud quotas, so parallelism is a per-run knob, one of several that we can adjust on individual jobs. Output resolution trades data volume and compute for detail; model size trades GPU time for accuracy; caching raw imagery trades storage for speed across repeated runs over the same area. The right setting depends on the task – and budget – at hand. Finding and fetching the right pixels Given a geographic region and time range, the platform first has to determine which satellite scenes should feed the model. That means identifying what was captured, where, and when across providers with different catalogs, formats, and publication delays. The selection criteria also depend on the source: for optical imagery such as Sentinel-2, we generally want the least-cloudy scenes available, while for synthetic aperture radar, the available polarization channels may matter more. We rely on public STAC catalogs and open standards wherever possible. But a large inference job can generate thousands of metadata queries at once—far more than external services such as ESA’s or Microsoft Planetary Computer’s STAC APIs are designed to handle concurrently. To avoid overwhelming those services, the OlmoEarth Platform maintains its own metadata index, updated as new imagery is published. For datasets hosted through AWS Open Data, we receive an SNS notification for each new scene. When a provider does not offer a change stream, we poll its upstream index every few minutes. As a result, our requests to external services follow the steady pace of new publications rather than the sudden burst generated by a large inference job. Each index entry stores scene metadata along with pointers to every location where the underlying pixels are available. At runtime, the platform selects the best source and performs windowed reads against cloud-optimized formats such as COG or Zarr, retrieving only the bytes needed for a given partition rather than downloading entire scenes. The index also supports our annotation tools. Because it maintains pointers to Sentinel-1, Sentinel-2, Landsat, and NISAR imagery in cloud-optimized formats, we can serve tiles from any indexed scene through the same windowed-read system, without building a separate ingestion pipeline. The providers that made this workflow easiest shared three characteristics that we would recommend as best practices for publishing Earth observation data: queue-based notifications when new imagery becomes available, storage on major cloud platforms without bespoke rate limits or availability bottlenecks, and cloud-optimized formats that support ranged reads. A sample query against the OlmoEarth satellite imagery index, searching for the least-cloudy Sentinel-2 image over San Francisco during the first week of June. The service returns the best available image along with a pointer to the GeoTIFF file in an Amazon S3 bucket. Handling failure at scale The OlmoEarth Platform is designed to recover from failures automatically. For each task within a stage and geographic partition, it dynamically provisions a virtual machine running our runner Docker container. The runner retrieves the task parameters, executes the work, returns the result, and shuts down. Because every task is reentrant and idempotent, intermittent failures can be handled safely by rerunning it. At this scale, failures are expected: a provider may be slow or briefly unavailable; metadata may indicate that imagery exists even when a required band or window is missing; cloud cover may leave too few usable observations; or a task may crash outright. The platform responds with task tracking, automatic retries, fallback to alternate providers when available, and clear distinctions between retryable and fatal errors. A separate monitoring process detects runners that have stalled or stopped and restarts their tasks. Where we're headed Our roadmap is shaped by the gaps our partners have identified and the capabilities they need most. Among the areas we are working on: Automated model runs. Schedule inference jobs in advance or trigger them whenever the imagery index registers a new scene over an area of interest. Change detection and alerts. Notify users when the landscapes they monitor change, so events such as deforestation or flooding surface as alerts rather than rasters someone has to find and inspect manually. Agentic tools and interfaces. Agents can lower the barrier to using geospatial models, from data curation and feature engineering to identifying ways to improve a fine-tuned model. We want users at any technical level to do work that previously required an experienced ML researcher. Faster models. Work with our research team on more efficient architectures that reduce GPU time per window. More modalities. Add new satellite sensors and data sources to both the OlmoEarth models and the imagery index that supports them. We’re currently focused on incorporating weather data (ERA-5) and satellites that provide more detail on environmental factors. Embeddings. Develop a dedicated embedding model and precompute embeddings at global scale. For many tasks, running inference against those embeddings could replace a full forward pass over raw imagery, making workloads substantially faster and less expensive. Fine-tuning and direct inference will remain important for maximum performance on challenging tasks, but embeddings could open up a much broader range of efficient applications. Run anywhere. OlmoEarth Run requires only virtual machines capable of running a Docker image and access to blob storage. We currently operate it on Google Cloud, but the architecture is designed to support multiple clouds and deployments within a partner’s own account and compute environment. This is only the beginning. Geospatial foundation models, and especially operationalizing them, are still an emerging technology, and many of the organizations best positioned to use them – those working in conservation, food security, disaster response, and climate – have never had access to infrastructure like this. The gap between what these teams need to understand about the planet and what their budgets and technical resources allow them to do remains wide. We are building OlmoEarth to help close it.