Backblaze Pushes Fast Object Storage to Solve AI Data Bottlenecks

The AI Pipeline Starts with Storage: Architecting Scalable Data Foundations With Backblaze — Ai4

Ai4

Because data preparation consumes 80% of engineering effort across the AI lifecycle, maintaining high-throughput active storage directly dictates model iteration speed and prevents GPU starvation.

AI-assisted summary of The AI Pipeline Starts with Storage: Architecting Scalable Data Foundations With Backblaze — original source. Check the original for full context.

Source published: 2025-08-24

Detailed notes

## Background Published on YouTube by the Ai4 channel on August 24, 2025, this presentation features Jeronimo De Leon, Senior Product Manager for AI at Backblaze. According to the video description, the talk explores how object storage supports every phase of the AI lifecycle—from data ingestion and labeling to model deployment and monitoring—to build scalable, cost-efficient data pipelines.

## TL;DR De Leon frames data as the foundational multiplier in AI development ($ ext{AI} = ext{Models} imes ext{Compute} imes ext{Data}$), noting that roughly 80% of project effort is consumed by data discovery, cleaning, and preparation rather than model training. He outlines a five-stage AI lifecycle: ingestion and active archiving, data processing and feature engineering, model training, inference deployment, and continuous monitoring. The presentation emphasizes how immutable object storage, unified metadata indexing, and high-throughput tiers like Backblaze B2 Overdrive eliminate GPU I/O bottlenecks and prevent vendor lock-in. Ultimately, building a resilient, cost-effective storage layer is presented as the essential prerequisite for managing expanding multimodal datasets and accelerating model iteration.

## Atmosphere & style The presentation carries a pragmatic, instructional, and enterprise-focused tone, anchored by De Leon's extensive operational background in MLOps and data architecture. The delivery is brisk and methodical, pairing structured architectural flowcharts with cautionary industry anecdotes about data ownership and lock-in. It feels like an authoritative conference masterclass designed to guide practitioners through the often-overlooked plumbing and infrastructure decisions that determine AI project success.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=4s) The foundational AI equation is Models multiplied by Compute multiplied by Data, where data serves as the critical multiplier that dictates how effectively compute and model architectures can be utilized. - [01:41](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=101s) The AI ecosystem has shifted from model scarcity to model abundance, making the orchestration of specialized, open-source models within a unified pipeline more practical than training monolithic models from scratch. - [01:41](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=101s) As frontier AI labs shift focus from text-based Large Language Models toward multimodal 'world models', organizations must proactively capture and archive all institutional data formats. - [02:47](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=167s) Approximately 80% of engineering effort across the 5-stage AI lifecycle is spent on data discovery, cleaning, and preparation, meaning an organization's storage strategy directly dictates its AI iteration speed. - [04:26](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=266s) Modern AI archives must function as searchable, immediately accessible active archives rather than static cold storage, using rich metadata to unify multimodal inputs. - [04:26](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=266s) Organizations must maintain direct ownership of their raw data assets in independent object storage to prevent vendor lock-in and catastrophic data loss when switching SaaS providers. - [04:26](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=266s) High-quality, curated domain datasets yield significantly greater model performance improvements than raw data volume or clever algorithmic tuning alone. - [08:29](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=509s) Enforcing immutability via Object Lock and standardized versioning during data processing and feature engineering is critical for pipeline auditability and reproducible rollbacks. - [11:31](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=691s) Training efficiency relies on high-throughput data streaming to avoid starving GPU clusters, paired with frequent, immutable checkpointing to minimize compute loss from node failures. - [13:11](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=791s) Scalable inference architectures rely on centralized, immutable model registries in object storage to hydrate weights across hybrid and multi-cloud environments. - [14:43](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=883s) Post-deployment monitoring requires logging production inferences and telemetry back into object storage to detect model drift and trigger continuous retraining loops. - [14:43](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=883s) The exponential growth of video datasets in enterprise AI requires ultra-high-throughput, predictable storage tiers with zero egress fees to sustainably feed GPU workloads.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=4s) Introduction and the Role of Storage in AI Innovation

The AI development pipeline begins and ends with data storage, where architectural decisions dictate how quickly teams can experiment, iterate, and bring AI models into production. Jeronimo De Leon, Senior Product Manager for AI at Backblaze, introduces the session by outlining his decade-long background across the AI ecosystem. His career spans developer advocacy for IBM Watson over ten years ago, building MLOps tooling at Spell (an MLOps platform later acquired by Reddit), architecting data lake infrastructure and conversational agents for Bloomberg's marketing division, and leading data aggregation and LLM insight generation at real estate analytics startup Intellis.

Backblaze operates with the mission to make storing, protecting, and using data "astonishingly easy" through simple onboarding and transparent, predictable pricing. Founded in 2007 and publicly traded since 2021, the company manages over 4,000 petabytes (4 exabytes) of data across more than 300,000 spinning hard drives for over 500,000 customers worldwide. AI organizations represent the platform's fastest-growing customer segment. De Leon frames the foundational equation of modern artificial intelligence: $\text{AI} = \text{Models} \times \text{Compute} \times \text{Data}$. While all three components are essential, data serves as the critical multiplier—compute resources and model architectures cannot be effectively optimized until the underlying data foundation is structured and accessible.

```mermaid flowchart LR accTitle: The AI Multiplier Equation subgraph Multiplier["Core Components"] M["Models"] C["Compute"] D["Data (Multiplier)"] end M --> AI["AI Value & Iteration Speed"] C --> AI D --> AI ```

[01:41](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=101s) The Shift from Model Scarcity to Data Abundance

The AI landscape has rapidly shifted from model scarcity to model abundance. On platforms like Hugging Face, there are currently more than 1.9 million pre-trained models readily available for experimentation and specialized tasks. In today's landscape—particularly with the rise of autonomous AI agents—success is rarely about training a single monolithic model from scratch. Instead, it revolves around orchestrating combinations of specialized models within a single pipeline, much like classical natural language processing (NLP) architectures paired intent classification models with entity extraction before generating a final output.

Quoting Nvidia CEO Jensen Huang—who advised that organizations should "collect as much world data as you can to advance AI development"—De Leon emphasizes that while text-based Large Language Models (LLMs) represent the current state of the art, text alone has fundamental limits. Frontier AI labs like Meta and Google are advancing toward multimodal "world models." Consequently, organizations must actively capture, structure, and archive their institutional data today in preparation for the next generation of foundational models.

[02:47](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=167s) The 5-Stage AI Lifecycle and the 80% Data Bottleneck

Whether an engineering team builds bespoke foundation models or fine-tunes open-source checkpoints, the operational workflow spans five discrete stages: data ingestion and active archiving, data processing, model experimentation and training, model deployment and inference, and continuous monitoring. Managing the end-to-end data lifecycle across these stages is just as intensive as training the models themselves, making scalable storage the common operational substrate across the entire workflow.

```mermaid flowchart TD accTitle: The 5-Stage AI Lifecycle subgraph Pipeline["The End-to-End AI Lifecycle"] S1["1. Ingestion & Active Archive"] S2["2. Data Processing & Preparation"] S3["3. Experimentation & Training"] S4["4. Deployment & Inference"] S5["5. Monitoring & Feedback Loops"] end S1 --> S2 --> S3 --> S4 --> S5 S5 -.->|"Continuous Retraining"| S1 ```

Teams are often surprised to discover where engineering time is actually spent across the AI lifecycle. Roughly 80% of project time is consumed by data-centric tasks: data identification, aggregation, cleaning, labeling, and augmentation. Because model weights and compute capacity are widely accessible, modeling itself is rarely the primary bottleneck. A substandard storage strategy directly impairs time-to-value, whereas an optimized storage layer directly accelerates that 80% preparation phase.

- **80% Effort Allocation**: Data discovery, cleansing, annotation, feature engineering, and ingestion. - **20% Effort Allocation**: Model architecture selection, hyperparameter tuning, training runs, and deployment. - **Core Principle**: Storage strategy directly dictates organizational AI iteration speed.

[04:26](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=266s) Stage 1: Ingestion and Active Archiving

In modern AI architectures, an archive can no longer function as a detached "cold storage" dumping ground; it must remain durable, searchable, and immediately accessible for downstream training pipelines. The primary friction point enterprise teams face is fragmentation—raw data is scattered across SaaS tools, shared file drives, object stores, and disparate cloud providers. Teams frequently define an urgent business problem and select an off-the-shelf model, only to hit a wall when attempting to locate and consolidate the required data.

Organizations generate valuable, AI-ready data every day without realizing it—including Zoom recordings, customer support audio, session replays, and UI screen captures. De Leon shares a cautionary anecdote about a client who relied entirely on a third-party AI SaaS platform for customer support operations. When the vendor contract terminated, the client had no independent data archive and permanently lost their entire multi-year repository of historical customer chat interactions. To prevent vendor lock-in and data loss, organizations must maintain direct ownership of their raw data assets.

```mermaid flowchart LR accTitle: Smart Ingestion Architecture subgraph Sources["Fragmented Sources"] Z["Zoom Recordings"] C["Customer Support Calls"] S["SaaS & Web Logs"] end subgraph Ingestion["Ingestion Layer"] ETL["Airflow / Fivetran"] Meta["Metadata Extraction"] end subgraph Storage["Central Active Archive"] B2["Backblaze B2 (Immutable & Object Locked)"] end Sources --> ETL ETL --> Meta Meta --> B2 ```

Smart ingestion relies on metadata as the universal unifier across multimodal assets such as CSVs, images, PDFs, and video files. Extracting and indexing rich metadata during ingestion transforms passive archives into instantly searchable datasets. Key practices and tooling for this stage include:

- **Predictable Active Archiving**: Storing raw archives on Backblaze B2 for durable, cost-effective, and instantly accessible retrieval. - **Orchestration Tooling**: Leveraging ETL frameworks like Apache Airflow and Fivetran. Because Airflow is open source, modern LLM code generation tools generate clean, accurate Airflow Directed Acyclic Graphs (DAGs) out of the box. - **Data Governance**: Enforcing Object Lock to guarantee immutability against accidental deletion or tampering, alongside lifecycle rules to automatically transition aging data. - **Quality Over Volume**: Echoing Google Research Director Peter Norvig's maxim: "More data beats clever algorithms, but better data beats more data." Cleaning and curating web-scale datasets—similar to how OpenAI invested heavily in human domain experts to curate training corpora compared to earlier iterations like Google Bard—is what drives superior model performance.

[08:29](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=509s) Stage 2: Data Processing, Feature Engineering, and Governance

The data processing phase defines an engineering team's iteration speed and directly establishes model output quality. Raw inputs must be cleansed, annotated, and transformed into structured training sets. Storing these prepared datasets in immutable object storage with Object Lock guarantees auditability and reproducible rollbacks. Features act as the bridge between raw inputs and model learning, requiring standardized storage formats (such as Parquet or Lance) and strict versioning to maintain dataset lineage.

De Leon illustrates this stage using a computer vision video processing pipeline. An unstructured raw video file is ingested, segmented into individual frames or short clips, and annotated via automated ML tagging or manual labeling tools. The pipeline extracts specialized feature representations—such as Histogram of Oriented Gradients (HOG) for edges and contours, Histogram of Optical Flow (HOF) for frame-to-frame movement vectors, and higher-level motion analysis for behavioral patterns. Each transformation step yields intermediate, version-controlled data artifacts that must be stored in object storage alongside evaluation metrics.

```mermaid flowchart TD accTitle: Video Processing and Feature Engineering Flow subgraph Preparation["Ingestion & Segmentation"] V["Raw Video Dataset"] --> F["Frame & Fragment Splitting"] end subgraph Annotation["Labeling"] F --> L["Manual & Automated Tagging (Label Studio/Snorkel)"] end subgraph Extraction["Feature Extraction"] L --> HOG["Histogram of Oriented Gradients (Edges)"] L --> HOF["Histogram of Optical Flow (Motion)"] L --> MA["Motion Analysis (Behavior)"] end subgraph StorageLayer["Versioned Storage"] HOG --> B2["Backblaze B2 (Snapshots & Feature Store)"] HOF --> B2 MA --> B2 end ```

- **Data Partitioning**: Partitioning datasets cleanly into training, validation, and test splits to enforce pipeline control and prevent evaluation leakage. - **Lineage Logging**: Logging every transformation script, schema migration, and upstream source snapshot to ensure end-to-end auditability. - **Processing Frameworks**: Utilizing distributed compute frameworks like Apache Spark, Dask, and Apache Beam. - **Labeling Platforms**: Implementing tools like Label Studio (open-source) and Snorkel AI for weak supervision and programmatic labeling. - **Storage Triggers**: Leveraging bucket event notifications to automatically trigger downstream preprocessing steps as soon as new data lands.

[11:31](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=691s) Stage 3: Model Experimentation and Training

During experimentation and model training, passive data assets are actively transformed into intellectual property. Training efficiency depends on feeding curated datasets into high-performance GPU compute clusters without starving the processors. Implementing data streaming, intelligent pre-fetching, and dataset sharding maximizes I/O throughput, allowing distributed training jobs to run in parallel without storage bottlenecks.

Because training runs are vulnerable to node failures, hardware interruptions, and loss spikes, frequent checkpointing is mandatory. A checkpoint captures the exact state of model weights, optimizer parameters, and runtime environments at a specific step. Frequent checkpointing reduces the compute cost of resuming interrupted runs and provides stable baseline snapshots for targeted fine-tuning experiments.

- **Frameworks & Trackers**: Standardizing on PyTorch and TensorFlow, integrated with experiment tracking systems to monitor loss trajectories responsibly. - **High-Throughput Storage**: Utilizing Backblaze B2 Overdrive tier storage to saturate GPU memory bandwidth and eliminate I/O wait times. - **Checkpoint Versioning**: Storing model checkpoint versions and configurations with default immutability, enabling seamless rollbacks and distributed weight replication.

[13:11](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=791s) Stage 4: Model Deployment and Inference

Delivering models reliably into production requires the same rigor as training them. Fine-tuned model artifacts must be packaged, containerized, and version-controlled to ensure consistent deployment across hybrid and multi-cloud environments. The inference architecture encompasses several distinct storage interactions: receiving application requests, performing real-time preprocessing and vector embeddings, retrieving model weights and configuration parameters from object storage, and logging output predictions.

```mermaid flowchart LR accTitle: Model Inference Architecture subgraph Client["Application"] App["App Request Data"] end subgraph Serving["Inference Engine"] Vec["Preprocessing & Vectorization"] Engine["Model Execution (KServe/TorchServe)"] end subgraph Storage["Storage & Registries"] Reg["Model Weights & Configs (B2 Object Lock)"] Logs["Prediction Logs & Outputs"] end App --> Vec Vec --> Engine Reg --> Engine Engine --> Logs ```

- **Model Serving Infrastructure**: Deploying scalable serving frameworks such as KServe and TorchServe. - **Model Registries**: Maintaining centralized registries to track artifact versions, metadata lineage, and formal deployment approval workflows. - **CI/CD Automation**: Driving automated rollouts and testing pipelines backed by immutable model buckets protected with Object Lock.

[14:43](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=883s) Stage 5: Continuous Monitoring, Feedback Loops, and the Video Explosion

"You can't monitor what you don't manage," De Leon notes. Post-deployment success depends on closing the feedback loop between production inferences and continuous model retraining. In production, every model prediction must be recorded alongside user interactions and downstream ground-truth outcomes to detect statistical model drift and data drift over time. Enterprise production pipelines routinely ingest and analyze terabytes of operational telemetry and structured inference logs.

Quoting Backblaze CEO Gleb Budman—"Video is dramatically more data than an image, which is dramatically more than a text message. That level of data is just going to continue to explode"—De Leon points out that rich video datasets are rapidly becoming the new baseline across enterprise AI workloads. Managing this exponential data growth demands an open, cost-effective storage foundation.

Backblaze positions B2 as an S3-compatible, reliable, and cost-effective enabling layer across all five lifecycle stages. To support demanding AI workloads, Backblaze offers B2 Overdrive, an ultra-high-throughput tier delivering up to terabit-per-second speeds with zero egress fees. This performance tier is engineered specifically for feeding GPU clusters, accelerating training and fine-tuning runs, supporting high-performance simulations, and serving low-latency origins for media and content delivery networks. De Leon concludes by inviting attendees to visit Backblaze at booth 226.

## Notable quotes > Behind every successful AI initiative is a robust, scalable data layer. — Jeronimo De Leon

> [00:04](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=4s) make storing, protecting, and using data 'astonishingly easy' through simple onboarding and transparent, predictable pricing. — Jeronimo De Leon

> [01:41](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=101s) collect as much world data as you can to advance AI development — Jensen Huang

> [02:47](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=167s) Roughly 80% of project time is consumed by data-centric tasks: data identification, aggregation, cleaning, labeling, and augmentation. — Jeronimo De Leon

> [02:47](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=167s) Storage strategy directly dictates organizational AI iteration speed. — Jeronimo De Leon

> [04:26](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=266s) In modern AI architectures, an archive can no longer function as a detached 'cold storage' dumping ground; it must remain durable, searchable, and immediately accessible for downstream training pipelines. — Jeronimo De Leon

> [04:26](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=266s) More data beats clever algorithms, but better data beats more data. — Peter Norvig

> [14:43](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=883s) You can't monitor what you don't manage — Jeronimo De Leon

> [14:43](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=883s) Video is dramatically more data than an image, which is dramatically more than a text message. That level of data is just going to continue to explode — Gleb Budman

## Mentioned Jeronimo De Leon, Backblaze, Backblaze B2, Backblaze B2 Overdrive, IBM Watson, Spell, Reddit, Bloomberg, Intellis, Hugging Face, Jensen Huang, Nvidia, Meta, Google, Google Bard, OpenAI, Peter Norvig, Gleb Budman, Apache Airflow, Fivetran, Parquet, Lance, Apache Spark, Dask, Apache Beam, Label Studio, Snorkel AI, PyTorch, TensorFlow, KServe, TorchServe, Prometheus, whylogs, Object Lock, Histogram of Oriented Gradients (HOG), Histogram of Optical Flow (HOF)

## Video description Jeronimo De Leon, Senior Product Manager, AI, Backblaze

Abstracts

Behind every successful AI initiative is a robust, scalable data layer. This talk explores how object storage supports every phase of the AI lifecycle, from data ingestion and labeling to model deployment and monitoring. We’ll explore how strategic use of object storage across the AI lifecycle enables scalable, efficient pipelines by aligning performance with workload needs. Whether you’re launching your first AI use case or scaling enterprise models, your data architecture decisions today will shape your success tomorrow.

[Source](https://www.youtube.com/watch?v=Z6fd5-eCaO4)

中文简报

Backblaze推高速对象存储以解决AI数据瓶颈由于数据准备在整个AI生命周期中消耗了80%的工程精力，保持高吞吐量的活跃存储直接决定了模型迭代速度并能防止GPU算力闲置。 De Leon将数据定位为AI开发的基础乘数（$\text{AI} = \text{Models} \times \text{Compute} \times \text{Data}$），并指出大约80%的项目精力都消耗在数据发现、清洗与准备上，而非模型训练。他概述了AI生命周期的五个阶段：摄取与活跃归档、数据处理与特征工程、模型训练、推理部署以及持续监控。演讲强调了不可变对象存储、统一元数据索引以及Backblaze B2 Overdrive等高吞吐层如何消除GPU I/O瓶颈并防止供应商锁定。最终，构建一个具有弹性且兼顾成本效益的存储层，被阐述为管理不断增长的多模态数据集以及加速模型迭代的必要前提。现代 AI 的核心等式为“模型 × 算力 × 数据”，其中数据是决定算力与模型架构能否充分发挥效能的关键倍增器；整个 5 阶段 AI 生命周期中约 80% 的工程精力都消耗在数据发现、清洗与准备上，存储架构直接决定了企业的迭代速度。 (2:47) AI 生态已由模型匮乏转向模型充盈，基于统一流水线编排各类专业开源模型已成主流趋势；随着前沿实验室加速迈向多模态“世界模型”，企业必须未雨绸缪地主动采集并归档各类组织内部数据资产。 (1:41) 现代数据归档必须作为可随时检索的“活跃归档”而非静态冷存储，利用丰富元数据打通多模态资产；企业需在独立对象存储中保持对原始数据的绝对控制权，并通过 Object Lock 确立不可变性，以规避 SaaS 平台锁死及数据永久丢失的风险。 (4:26) 高质精炼的数据对模型效果的提升远胜于单纯的数据堆砌或算法微调；在特征工程与 GPU 训练阶段，通过 Parquet/Lance 等标准化格式、细粒度版本控制及高吞吐数据流，能够消除 I/O 瓶颈并确保流水线的可审计与可重现。 (8:29)

Reading summary

Jeronimo De Leon, Senior Product Manager for AI at Backblaze, argues that as pre-trained models and compute become widely accessible, scalable data storage serves as the primary multiplier dictating how quickly organizations can iterate and deploy artificial intelligence. From Model Scarcity to the Data Bottleneck (2:47) With over 1.9 million pre-trained models available on platforms like Hugging Face, De Leon notes that AI engineering has shifted from training single monolithic models from scratch to orchestrating pipelines of specialized models. Framing modern development through the equation AI = Models × Compute × Data, he positions data as the critical multiplier. Because model weights and compute capacity are readily available, modeling itself is rarely the primary bottleneck; instead, roughly 80% of AI project time is consumed by data discovery, cleansing, labeling, and ingestion, leaving only 20% for model selection, training runs, and deployment. This data preparation bottleneck is intensifying as frontier labs such as Meta and Google move beyond text-based Large Language Models toward multimodal "world models." Citing Nvidia CEO Jensen Huang's advice that organizations collect as much world data as possible, De Leon contends that enterprises must actively capture and structure institutional assets today—including Zoom recordings, customer support audio, session replays, and UI captures—to prepare for upcoming model architectures. Data Ownership, Active Ingestion, and Curation (4:26) Enterprise AI initiatives frequently stall at the ingestion phase because raw data is fragmented across SaaS tools, shared drives, and disparate cloud providers. De Leon warns that relying entirely on third-party AI vendors introduces severe operational risk, sharing an anecdote of a client that permanently lost a multi-year repository of customer chat interactions when its third-party SaaS contract ended without an independent archive. To prevent vendor lock-in and data loss, he advises organizations to maintain direct ownership of raw assets in durable, immediately accessible active archives rather than detached cold storage. Transforming fragmented multimodal files (CSVs, PDFs, images, and video) into searchable datasets requires extracting and indexing metadata during ingestion. De Leon highlights orchestration frameworks like Fivetran and open-source Apache Airflow, noting that modern LLMs can generate clean Airflow Directed Acyclic Graphs (DAGs) out of the box. However, he stresses that data quality matters more than raw volume. Quoting Google Research Director Peter Norvig's maxim that "better data beats more data," De Leon attributes OpenAI's performance advantage over early iterations of Google Bard to its investment in human domain experts to clean and curate training corpora. Feature Lineage and High-Throughput GPU Training (8:29) Processing raw inputs into structured training sets requires strict governance, dataset partitioning, and lineage logging. De Leon illustrates this using a computer vision pipeline in which raw video is split into frames, annotated via automated tagging or tools like Label Studio and Snorkel AI, and processed using distributed frameworks such as Apache Spark, Dask, or Apache Beam. The pipeline extracts specialized representations—Histogram of Oriented Gradients (HOG) for edges, Histogram of Optical Flow (HOF) for movement vectors, and motion analysis for behavior—saving each intermediate artifact in standardized formats like Parquet or Lance within immutable, Object Lock-protected storage to ensure reproducible rollbacks. Once curated datasets enter experimentation and training in PyTorch or TensorFlow, the storage layer must feed GPU clusters without causing I/O stalls. De Leon explains that data streaming, intelligent pre-fetching, and dataset sharding maximize throughput across parallel training jobs. Because training runs are vulnerable to hardware interruptions, node failures, and loss spikes, engineering teams must also write frequent, versioned checkpoints capturing model weights and optimizer parameters to resume interrupted runs cost-effectively. Production Inference and Continuous Drift Monitoring (14:43) Delivering fine-tuned models into production requires packaging and containerizing artifacts for consistent serving across hybrid and multi-cloud environments. Using frameworks like KServe and TorchServe backed by CI/CD automation, inference engines preprocess application requests, hydrate model weights from centralized, immutable registries, and log output predictions. De Leon emphasizes that post-deployment success relies on closing the feedback loop between production inference and continuous retraining. Enterprise pipelines routinely collect terabytes of operational telemetry and structured logs—using monitoring tools such as Prometheus and whylogs—to compare predictions against ground-truth outcomes. Detecting statistical model drift or data drift in these logs triggers automated bucket notifications that initiate new data ingestion and retraining cycles. De Leon concludes that the transition toward multimodal AI—underscored by Backblaze CEO Gleb Budman's forecast that video will drive an exponential explosion in data volume compared to text and images—demands an open, cost-effective storage foundation. Across all five lifecycle stages, Backblaze positions its S3-compatible B2 platform (which manages over 4 exabytes of data across more than 300,000 hard drives for 500,000 customers) and its B2 Overdrive tier, which claims up to terabit-per-second throughput with zero egress fees, as purpose-built infrastructure to keep GPU clusters saturated and accelerate organizational iteration speed.

精炼阅读摘要

Backblaze 人工智能高级产品经理 Jeronimo De Leon 指出，人工智能的价值取决于模型、算力与数据的协同，其中数据是核心乘数；鉴于工程团队大约 80% 的时间耗费在数据清洗与准备等环节，具备高吞吐与版本控制能力的存储架构已成为贯穿 AI 全生命周期的关键基础设施。数据乘数效应与研发阶段的80%瓶颈 (1:41) De Leon 提出现代人工智能的核心公式为模型、算力与数据的乘积。当前行业已经从模型稀缺转变为模型过剩，例如 Hugging Face 上已公开超过 190 万个预训练模型，现代 AI 开发往往不再依赖从零训练单一单体模型，而是转向编排多个专用模型协同工作。在算力与模型权重相对容易获取的背景下，数据成为决定系统性能的核心乘数。然而，工程团队的实际精力分配存在明显失衡。约 80% 的项目时间被数据发现、归集、清洗、标注和特征工程等数据密集型任务占用，仅有约 20% 的时间用于模型架构设计、超参数调优及训练部署。由于建模本身已不再是主要瓶颈，低效的存储层设计会直接拖累前期的准备效率，因此优化存储底座是加速整个 AI 迭代周期的根本手段。动态活跃归档与企业数据自主权 (4:26) 在现代 AI 架构中，数据归档不能再被视为离线搁置的“冷数据存储”，而必须保持高持久性、可检索性并能即时供下游管道调用。目前企业面临的主要阻碍是数据碎片化，原始数据广泛分散在各类 SaaS 工具、共享网盘和异构云端中。De Leon 引用案例说明，曾有客户将全部客服业务托付给第三方 AI SaaS 厂商，在合同终止后因缺乏独立归档机制，彻底丧失了沉淀多年的历史对话数据，企业必须将原始数据的所有权保留在自身可控的存储层中。智能摄取流程应通过统一元数据来关联会议录音、客服音频、操作日志等非结构化多模态资源。利用 Apache Airflow 或 Fivetran 等数据编排工具，结合存储层的对象锁定功能确保数据防篡改与审计合规。De Leon 援引 Peter Norvig 的观点强调“优质数据胜过海量数据”，通过专业清洗与高质量标注精选的数据集，比盲目堆叠低质网络抓取数据更能提升模型表现。特征工程版本化与消除训练IO瓶颈 (8:29) 数据处理阶段直接决定了模型的输出质量与迭代速度。以计算机视觉流水线为例，原始视频被切分为帧或短视频片段后，经由 Label Studio 或 Snorkel AI 等工具进行标注，并进一步提取方向梯度直方图（HOG）、光流直方图（HOF）及高阶行为运动分析等特征。此类特征衍生品需要采用 Parquet 或 Lance 等标准化格式持久化存储，并借助 Apache Spark、Dask 等框架记录数据血缘与转换脚本，确保流水线具有审计追溯能力与可复现性。进入模型实验与训练阶段后，海量数据需要无延迟地注入 GPU 计算集群。通过数据流式传输、智能预取和数据集分片，能够防止处理器因 I/O 等待而闲置。同时，分布式训练极易遭遇节点硬件故障或损失率异常，实施高频次检查点机制能够保存模型权重与优化器状态快照，在发生中断时大幅缩减重复计算的开销并便于精准微调。多模态视频激增与生产闭环监控 (14:43) 模型投入生产部署并不意味着数据生命周期的结束。在推理阶段，部署在 KServe 或 TorchServe 上的模型需要配合不可变注册表拉取权重文件，并实时记录每一次推理请求与用户交互日志。生产团队必须持续监控遥测指标以捕捉数据漂移和模型漂移，从而将线上反馈重新注回摄取管道，形成持续再训练的闭环系统。多模态数据正推动存储需求呈指数级爆发。De Leon 引用英伟达首席执行官黄仁勋关于尽可能收集物理世界数据以迈向“世界模型”的论述，并结合 Backblaze 首席执行官 Gleb Budman 的判断指出，视频蕴含的信息量远超图像与文本，正迅速成为企业 AI 负载的新基线。应对以太比特级速率吞吐的视频流与推理日志，迫使组织必须采纳兼具低延迟、大带宽及无出向流量费优势的高性能存储方案。 De Leon 论证了开放且经济的对象存储是维系现代 AI 管道运转的生命线，但随着多模态“世界模型”引发数据体量急剧膨胀，企业如何在控制长期存储成本与维持 GPU 集群超高 I/O 吞吐之间保持架构平衡，仍是工程实践中必须面对的现实课题。

Detailed notes — zh

## Background 本视频由 YouTube 频道 Ai4 于 2025 年 8 月 24 日发布，主讲人为 Backblaze 的 AI 高级产品经理 Jeronimo De Leon。根据视频简介，演讲深入探讨了对象存储如何在从数据摄取、标注到模型部署与监控的整个 AI 生命周期中发挥支撑作用，以帮助企业通过合理匹配工作负载与存储性能来构建可扩展且高效的 AI 数据底座。 ## TL;DR 演讲指出在整个 AI 生命周期中，高达 80% 的工程时间消耗在数据准备与治理上，底层存储架构直接决定了业务的落地周期与创新速度。演讲者将 AI 管道系统拆解为智能摄取与活跃归档、特征工程处理、模型训练加速、推理部署服务以及监控闭环五个核心阶段，强调了元数据治理、对象锁定与血缘追溯的不可或缺性。针对多模态与视频数据爆发带来的算力供给瓶颈，Backblaze 提出了兼具高持久性与经济性的对象存储架构，并推出支持 Tbps 级高吞吐且免公网流出费的 Backblaze B2 Overdrive，全面释放 GPU 集群的计算潜能。 ## Atmosphere & style 演讲整体呈现出务实、严谨且富有工业实践底蕴的技术分享风格，演讲者依托其在 IBM Watson、Spell、彭博社等机构积累的深厚从业背景，讲解条理分明且直击工程痛点。叙事节奏紧凑而流畅，避开了浮夸的概念炒作，通过直观的生命周期图解与切中要害的实操案例（如 SaaS 数据丢失教训与视频特征提取流水线）层层递进。整场分享散发着系统架构师的笃定与理性，为观众带来了从底层数据治理切入、最终闭环至前沿模型迭代的沉浸式技术洞察。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=4s) 底层数据存储与架构设计是决定 AI 项目迭代速度与价值转化时间（Time to Value）的核心基石，而非仅取决于算法本身。 - [01:24](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=84s) 在 AI 成功公式（模型 × 算力 × 数据）中，数据是核心乘数，且整个 AI 生命周期的 80% 时间消耗在数据准备与工程环节，仅 20% 用于模型训练与调优。 - [01:24](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=84s) 开源模型生态（如 Hugging Face 拥有的 190 万+ 模型）使模型不再稀缺，AI 发展重心正向将专业模型流水线组合为智能体（Agent）及构建物理世界模型转移。 - [04:26](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=266s) AI 语境下的数据归档必须是兼具高持久性与极速读写的“活跃归档（Active Archive）”，而不是将数据放入冷存储。 - [04:26](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=266s) 企业必须在自有存储中建立统一的数据湖与活跃归档，避免将高价值交互数据锁定在第三方 SaaS 服务商中导致供应商解约时数据资产丢失。 - [04:26](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=266s) 元数据（Metadata）是异构非结构化数据的通用粘合剂，利用对象锁定（Object Lock）与生命周期规则可兼顾数据防篡改合规性与存储成本优化。 - [08:29](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=509s) 在数据处理与特征工程阶段，将中间结果（从抽帧、标注到提取的特征）在对象存储中进行版本化归档与血缘追踪，是确保实验可复现与合规治理的关键。 - [11:31](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=691s) 模型训练阶段依赖流式传输（Streaming）与数据分片以消除 GPU 计算的 I/O 瓶颈，并需频繁自动保存检查点（Checkpoints）以支持断点续训和微调实验。 - [14:43](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=883s) 通过对象存储沉淀推理日志与用户反馈，并结合监控指标分析模型漂移（Model Drift），能通过存储事件通知自动触发再训练，形成闭环自愈系统。 - [14:43](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=883s) 随着多模态与视频数据的指数级增长，Backblaze 推出 Tbps 级吞吐且免流出费的 B2 Overdrive 存储层，专门满足 GPU 训练和高并发推理的高性能需求。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=4s) 演讲者背景与 Backblaze 的存储基石在 AI 领域，团队迭代与创新的速度往往不由算法单独决定，而是深植于底层的存储架构之中。演讲者 Jeronimo De Leon（Backblaze AI 高级产品经理）拥有超过十年的 AI 从业经验：早年在 IBM Watson 负责开发者推广工具，随后加入 MLOps 初创公司 Spell（后被 Reddit 收购），在彭博社（Bloomberg）主导过市场部门的数据湖基础设施与对话式代理开发，近期又在房地产数据洞察初创公司 Intellis 利用大语言模型聚合房产数据并生成洞察，如今则在 Backblaze 负责 AI 存储产品。 Backblaze 的核心使命是让数据的存储、保护和使用变得“出奇地简单”（astonishingly easy），强调极简上手体验与透明可预测的定价模式。公司成立于 2007 年，于 2021 年正式上市，目前在全球拥有超过 50 万家客户。其中，AI 客户已成为其增长最迅速的群体。在底层硬件规模上，Backblaze 依靠部署在 30 万块机械硬盘上的存储集群，承载着超过 4,000 PB（即 4 EB）的海量数据。 - 公司规模：成立于 2007 年，2021 年上市，全球 500,000+ 客户。 - 存储体量：300,000 块运转中的机械硬盘，总计管理超过 4,000 PB 数据。 - 业务重心：AI 客户已成为当前增速最快的核心业务板块。 --- ### [01:24](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=84s) AI 成功方程式与被低估的“80% 数据瓶颈” 业界广泛流传着一个经典公式：$\text{AI} = \text{模型（Models）} \times \text{算力（Compute）} \times \text{数据（Data）}$。在这三要素中，数据是真正的核心乘数因子（multiplier）。算力与模型固然重要，但如果数据没有存放在合适的位置、以高效的方式流动，对算力与模型的优化都无从谈起。如今 Hugging Face 上已有超过 190 万个开源模型，模型本身早已不是稀缺资源。随着 AI Agent（智能体）的兴起，工程重点已转向如何将多个专用模型进行流水线组合——正如早期自然语言处理（NLP）工作流中，系统会在预测前串联“意图分类”与“实体抽取”模型一样，现成的模型随时可供测试与组合。引用英伟达 CEO 黄仁勋（Jensen Huang）的观点：“你应该尽可能多地收集物理世界的数据，以推动 AI 的演进。”目前大语言模型虽然处于聚光灯下，但仅靠文本数据很快会触及天花板。Meta 和 Google 等巨头正在全力研发下一代“世界模型”（World Models），这要求企业必须提前规划数据归档，为未来多模态模型的演进做好储备。大多数人未曾意识到的是，在 AI 管道的整个生命周期中，**高达 80% 的时间被消耗在数据环节**——包括数据识别、聚合、清洗、标注和增强，而模型训练与调优本身仅占 20%。如果想要加速 AI 创新，就必须优先优化这占据八成时间的底层数据基础设施。 ```mermaid flowchart LR accTitle: AI Lifecycle Overview subgraph S1["数据准备 (80% 耗时)"] direction TB A["1. 摄取与活跃归档"] --> B["2. 数据处理与特征工程"] end subgraph S2["建模与生产 (20% 耗时)"] direction TB C["3. 实验与训练"] --> D["4. 部署与推理"] D --> E["5. 监控与反馈"] end B --> C E -. 反馈数据流.-> A ``` - 开源模型生态：Hugging Face 托管模型数量已突破 190 万个，模型选型不再是主要障碍。 - 时间分配现实：80% 耗费在数据工程与准备阶段，仅 20% 用于建模与训练。 - 核心结论：存储战略直接决定了业务的价值转化时间（Time to Value）。 --- ### [04:26](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=266s) 阶段一：智能数据摄取与活跃归档（Ingest & Active Archive） AI 管道的第一步是数据摄取与归档，但“归档”在 AI 语境下绝不等于放入冷存储（Cold Storage），而必须是兼具高持久性与极速读写能力的“活跃归档”（Active Archive）。当前最大的工程瓶颈在于数据的高度碎片化——原始数据形态各异，零散孤立在各类 SaaS 平台、内部文件共享系统以及不同的公有云环境中。很多团队最常犯的错误，是在明确了业务目标并选定模型后，才惊觉数据分散各处，不得不停下手头工作临时搭建数据流水线。企业实际上每天都在无意识地生成高价值的 AI 就绪数据，例如 Zoom 视频会议录音、客户支持通话录音、用户行为日志以及屏幕录制文件。演讲者分享了一个真实案例：某企业曾采购第三方 AI 客服系统处理用户交互，所有的客户聊天记录均存放在该服务商的系统中；当合同期满终止服务时，由于该企业此前未在自有存储中同步留存数据，导致宝贵的历史客服交互数据全部丢失。实现智能摄取的核心在于元数据（Metadata）治理。元数据是跨越 CSV、图片、视频、PDF 等异构格式的通用粘合剂。在摄取阶段提取丰富的元数据构建可检索的归档库，能大幅缩减后续进入建模阶段所需的时间。在工具链选择上，Backblaze B2 提供了高可预测成本与不可变特性；配合 Apache Airflow 或 Fivetran 等工具构建 ETL 管道。值得一提的是，现代 AI 代码生成工具能够极其精准地生成 Airflow DAG 脚本，极大加速了摄取管道的构建。同时，必须开启对象锁定（Object Lock）防篡改，并利用生命周期规则自动沉淀冷数据以优化开销。 > “更多数据胜过聪明算法，但更好的数据胜过更多数据（More data beats clever algorithms, but better data beats more data）。” —— Peter Norvig（Google 知名研究员） - 孤岛风险：业务数据被锁定在第三方 SaaS 平台可能导致供应商解约时的数据资产永久灭失。 - 管道自动化：利用 AI 自动生成 Apache Airflow DAG 脚本，可大幅提升摄取流水线的构建效率。 - 治理基线：采用 S3 兼容的 Object Lock（对象锁定）保障不可篡改，通过生命周期策略自动优化长期存储成本。 --- ### [08:29](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=509s) 阶段二：数据处理、特征工程与可追溯性治理模型输出质量完全取决于数据处理阶段的工程纪律，处理阶段的吞吐效率直接决定了算法团队的迭代步调。首先，清洗与标注是把原始碎片转化为模型训练材料的关键，清洗后的数据集应存放在具备对象锁定的存储桶中，确保原始输入在合规审计与版本回滚时有据可查。其次，特征工程是从原始数据中提取趋势、模式等高价值信号的桥梁；选用高效的数据存储格式不仅能优化计算性能，还能大幅削减存储成本。以视频数据的处理流水线为例：海量视频首先被拆分为连续的视频片段或单帧图像；随后通过人工标注或机器学习自动化标签进行元数据打标；接着进入特征提取流水线——利用方向梯度直方图（HOG）检测边缘与几何轮廓，利用光流直方图（Optical Flow）捕捉帧间动态位移，结合运动分析提取高维行为模式。在这一过程中，从原始视频碎片、标注集、提取特征到评估指标，每一个中间阶段都会生成高价值的数据资产，并全部在对象存储中进行版本化归档。 ```mermaid flowchart TD accTitle: Video Data Processing Pipeline subgraph Raw["原始输入"] A["原始视频数据集"] --> B["视频分片与抽帧"] end subgraph Feature["标注与特征工程"] B --> C["人工 / ML 自动化标注"] C --> D["特征提取: HOG 边缘检测"] C --> E["特征提取: 光流动态捕捉"] C --> F["行为与运动模式分析"] end subgraph Storage["资产归档"] D & E & F --> G["版本化特征库 (Backblaze B2)"] end ``` 数据处理还必须包含科学的分区（Partitioning）与血缘（Lineage）治理。将数据集严格划分为训练集、验证集与推理集，能显著提升实验的可复现性。合规治理的核心是记录每一次转换操作、Schema 变更以及所引用的数据源，并在对象存储中沉淀版本快照。 - 处理工具：Apache Spark、Dask、Apache Beam 等分布式计算引擎。 - 标注生态：Label Studio（开源）、Snorkel 等工具。 - 自动化联动：利用对象存储的事件通知（Event Notifications）机制，自动触发下游特征提取与训练任务。 --- ### [11:31](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=691s) 阶段三与阶段四：模型训练加速、检查点与高并发推理阶段三（模型实验与训练）是将静态数据转化为动态产品的核心环节。在这一阶段，高吞吐的数据流转至关重要。工程团队通常采用数据流式传输（Streaming）与预取（Prefetching）技术，将清洗后的数据集高效喂入 GPU 高性能计算集群；配合数据分片（Data Sharding）提升 I/O 并发度，消除数据读取瓶颈。训练并非总是一帆风顺，频繁且自动地保存模型检查点（Checkpoints）、配置文件与运行时环境，不仅能大幅降低硬件故障导致的数据丢失风险、便于断点续训，还能为针对特定下游任务的微调实验提供版本基线。阶段四（模型部署与推理）同样高度依赖底层存储。模型交付效率与模型本身质量同等重要。在生产环境中，模型服务需要跨多云环境分发微调权重，实时接收端侧应用请求并快速返回预测。 ```mermaid flowchart LR accTitle: Model Inference Flow subgraph Input["输入阶段"] A["应用请求数据"] --> B["预处理 / 向量化"] end subgraph Inference["推理引擎"] B --> C["推理服务 (KServe / TorchServe)"] D["存储库: 模型权重 & 运行配置"] --> C end subgraph Output["输出阶段"] C --> E["预测输出结果"] E --> F["日志与指标沉淀 (B2 存储)"] end ``` - 训练框架与追踪：以 PyTorch、TensorFlow 为核心训练框架，搭配完善的实验追踪（Experiment Tracking）工具。 - 检查点机制：将 Checkpoints 存入对象存储，确保在节点崩溃时能即时回滚并在分布式训练节点间无缝同步。 - 推理服务基础设施：借助 KServe、TorchServe 等服务化工具，结合模型注册表（Model Registry）严格管理模型发布审批流与 CI/CD 自动化部署。 --- ### [14:43](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=883s) 阶段五：监控反馈闭环与 Backblaze B2 Overdrive 极速存储 “无法管理你未曾监控的事物。”在阶段五中，系统必须为每一次推理调用生成可追溯的日志，并与后续的用户反馈数据相关联。随着线上真实真值（Ground Truth）数据的持续沉淀，团队需要长期监测模型准确率的变化，及早捕获不可避免的模型漂移（Model Drift）。保持结构化日志记录是实现大规模监控的前提，所有日志、指标及预测输出均应归档至版本化存储桶中，并通过存储事件通知自动触发再训练流水线，形成完整的自愈与迭代闭环。引用 Backblaze 首席执行官 Gleb Budman 的洞察：“视频的数据体量远远超出图片，而图片又远远超出文本。数据规模的爆炸式增长才刚刚开始。”视频正在成为多模态 AI 的全新基准线，这种指数级增长对存储层的吞吐能力和经济性提出了严苛挑战。为此，Backblaze 推出了专为 AI 和高性能计算负载打造的全新存储层——**Backblaze B2 Overdrive**。该产品提供高达 Tbps（太比特/秒）级别的超高吞吐带宽，专用于加速模型训练、微调与高并发推理阶段对 GPU 集群的数据供给。其具备免公网流出费（No Egress Fees）、完全兼容 AWS S3 API 的特性，并广泛适用于 AI/ML 训练集群直连、高性能计算（HPC）模拟分析以及低延迟媒体流媒体分发等高吞吐场景。 - 监控工具栈：Prometheus、whylogs 用于指标监控与模型漂移分析。 - 闭环机制：结构化日志实时归档至 B2 存储桶，触发自动化模型再训练任务。 - B2 Overdrive 核心亮点：Tbps 级极速吞吐、零流出费用、专为 GPU 计算集群设计、即插即用的 S3 兼容架构。 ## Notable quotes > [01:24](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=84s) AI = 模型（Models） × 算力（Compute） × 数据（Data） — Jeronimo De Leon > [01:24](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=84s) 你应该尽可能多地收集物理世界的数据，以推动 AI 的演进。 — Jensen Huang > [01:24](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=84s) 高达 80% 的时间被消耗在数据环节——包括数据识别、聚合、清洗、标注和增强，而模型训练与调优本身仅占 20%。 — Jeronimo De Leon > [04:26](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=266s) 更多数据胜过聪明算法，但更好的数据胜过更多数据（More data beats clever algorithms, but better data beats more data）。 — Peter Norvig > [08:29](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=509s) 模型输出质量完全取决于数据处理阶段的工程纪律，处理阶段的吞吐效率直接决定了算法团队的迭代步调。 — Jeronimo De Leon > [14:43](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=883s) 无法管理你未曾监控的事物。 — Jeronimo De Leon > [14:43](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=883s) 视频的数据体量远远超出图片，而图片又远远超出文本。数据规模的爆炸式增长才刚刚开始。 — Gleb Budman > [01:24](https://www.youtube.com/watch?v=Z6fd5-eCaO4&t=84s) 存储战略直接决定了业务的价值转化时间（Time to Value）。 — Jeronimo De Leon ## Mentioned Jeronimo De Leon, Backblaze, Backblaze B2, Backblaze B2 Overdrive, Gleb Budman, Jensen Huang, Peter Norvig, IBM Watson, Spell, Reddit, 彭博社 (Bloomberg), Intellis, Hugging Face, Meta, Google, Zoom, Apache Airflow, Fivetran, Apache Spark, Dask, Apache Beam, Label Studio, Snorkel, PyTorch, TensorFlow, KServe, TorchServe, Prometheus, whylogs, AWS S3 ## Video description Jeronimo De Leon, Senior Product Manager, AI, Backblaze Abstracts Behind every successful AI initiative is a robust, scalable data layer. This talk explores how object storage supports every phase of the AI lifecycle, from data ingestion and labeling to model deployment and monitoring. We’ll explore how strategic use of object storage across the AI lifecycle enables scalable, efficient pipelines by aligning performance with workload needs. Whether you’re launching your first AI use case or scaling enterprise models, your data architecture decisions today will shape your success tomorrow. [Source](https://www.youtube.com/watch?v=Z6fd5-eCaO4)

Source quotes

[1:24] Jeronimo De Leon: AI = 模型（Models） × 算力（Compute） × 数据（Data） [1:24] Jensen Huang: 你应该尽可能多地收集物理世界的数据，以推动 AI 的演进。 [1:24] Jeronimo De Leon: 高达 80% 的时间被消耗在数据环节——包括数据识别、聚合、清洗、标注和增强，而模型训练与调优本身仅占 20%。 [4:26] Peter Norvig: 更多数据胜过聪明算法，但更好的数据胜过更多数据（More data beats clever algorithms, but better data beats more data）。 [8:29] Jeronimo De Leon: 模型输出质量完全取决于数据处理阶段的工程纪律，处理阶段的吞吐效率直接决定了算法团队的迭代步调。 [14:43] Jeronimo De Leon: 无法管理你未曾监控的事物。 [14:43] Gleb Budman: 视频的数据体量远远超出图片，而图片又远远超出文本。数据规模的爆炸式增长才刚刚开始。 [1:24] Jeronimo De Leon: 存储战略直接决定了业务的价值转化时间（Time to Value）。

Source quotes

Jeronimo De Leon: Behind every successful AI initiative is a robust, scalable data layer. [0:04] Jeronimo De Leon: make storing, protecting, and using data 'astonishingly easy' through simple onboarding and transparent, predictable pricing. [1:41] Jensen Huang: collect as much world data as you can to advance AI development [2:47] Jeronimo De Leon: Roughly 80% of project time is consumed by data-centric tasks: data identification, aggregation, cleaning, labeling, and augmentation. [2:47] Jeronimo De Leon: Storage strategy directly dictates organizational AI iteration speed. [4:26] Jeronimo De Leon: In modern AI architectures, an archive can no longer function as a detached 'cold storage' dumping ground; it must remain durable, searchable, and immediately accessible for downstream training pipelines. [4:26] Peter Norvig: More data beats clever algorithms, but better data beats more data. [14:43] Jeronimo De Leon: You can't monitor what you don't manage [14:43] Gleb Budman: Video is dramatically more data than an image, which is dramatically more than a text message. That level of data is just going to continue to explode

Original transcript

[0:04] Hello everyone and we're here to talk [0:05] about the AI pipeline starts with [0:07] storage architecting scalable data [0:09] foundations because your storage [0:11] architecture can determine how quickly [0:14] you can iterate and innovate with AI. [0:18] I'm Jono Dillion. A little bit about me. [0:20] I started my career in AI over 10 years [0:22] ago developer advocacy tools for IBM [0:25] Watson. worked at MLOps startup called [0:27] spell.mml that was acquired by Reddit. [0:30] Uh did data lake infrastructure uh for [0:32] Bloomberg for their marketing division [0:34] and some conversational agents and more [0:36] recently worked at a uh real estate data [0:39] insight startup called Intellis where we [0:40] aggregated uh property real estate data [0:43] and generated insights with LLMs and now [0:46] I'm at Back Blaze. [0:49] So Back Blaze is on a mission to make [0:51] storing, protecting and using data [0:53] astonishingly easy. We pride ourselves [0:55] on being simple, simple to use, simple [0:58] to get started, simple pricing. [1:02] We were founded in 2007 and we've been [1:05] public since 2021 and we have over [1:07] 500,000 customers worldwide and the most [1:11] uh fastest growing segment is AI [1:13] customers like just like you guys. We [1:16] have 4,000 pabytes of of data on 300,000 [1:20] spinning drives. [1:24] So everyone's kind of seen this formula [1:26] AI equals models time comput time data [1:29] and all three factors ma matter but data [1:31] is the multiplier here right you can't [1:33] optimize your models or your compute [1:36] until your data is all in the right [1:37] place [1:41] and this is what I what I mean by like [1:43] hugging face is one of the biggest repos [1:45] um on the web with over 1.9 million [1:48] models so the models are out there [1:50] there's there's plenty of models for us [1:52] to use that can handle a lot of the use [1:54] cases that we're trying to do right now. [1:56] Um, and then with especially with agents [1:58] now, it's probably going to be more [2:00] around what are the combination of [2:02] models that we want to be using in that [2:04] AI workflow. Um, previously uh with NLP [2:09] workflows and in conversational agents, [2:11] you used to have an intent [2:12] classification with entity extraction [2:14] before you did prediction. So, you're [2:16] going to see that combination of [2:18] different models and the the models are [2:19] there for us to test and trial. [2:23] And this is a quote from Jensen Huang. [2:26] No, you should collect as much world [2:27] data as you can to advance AI [2:30] development. Now, LLMs are the top model [2:32] right now, but we can only go so far [2:34] with text. And companies like Meta and [2:36] Google, they're developing the next type [2:38] of models they're calling world models. [2:40] So, you have to think about archiving [2:41] your data in preparation for the next [2:43] model to come. [2:47] So the AI data pipeline primarily [2:49] follows these five stages. You got data [2:52] ingest and active archive, data [2:53] processing, model experimentation and [2:56] training, model deployment and inference [2:58] and monitoring. And we'll go through [2:59] deeper in each of these stages. But as [3:01] you can see the pipeline is just about [3:03] managing data, the managing data, the [3:05] data life cycle as it is about modeling. [3:07] So storage underpins the entire [3:10] pipeline. [3:13] And whether you're uh creating a model [3:16] from scratch or primarily fine-tuning [3:18] because you're you're taking advantage [3:19] of all those pre-trained models that you [3:21] can see there's over 1.9 million of [3:22] them. U the the stages are primarily the [3:25] same. So AI depends on that data [3:27] orchestration and storage that's [3:29] structured accessible at every stage. [3:31] You know efficient storage enables [3:33] faster access, reuse and processing of [3:36] data across all those stages. [3:41] Now this is the kind uh so people get [3:44] most surprised of what is the time spent [3:46] in all those three stage or all those [3:47] five stages. So as you can see 80% of [3:50] the time is uh data related data [3:53] identification aggregation cleansing [3:56] labeling and augmentation consume the [3:58] majority of the time. So modeling is not [4:00] the bottleneck. We have access to a lot [4:02] of models now. So storage strategy [4:05] directly impacts your time to value [4:07] because as you can see if you're wanting [4:09] to uh speed up innovation, if you want [4:11] to optimize your pipeline, you want to [4:14] be optimizing a that 80% time right [4:16] there. Poor poor data infrastructure is [4:19] where your uh where you you slow your AI [4:22] progress. [4:26] So stage one, ingest an archive. An [4:29] archive doesn't always equal cold [4:31] storage. you know AI work AI workflows [4:34] archives need to be both durable and [4:35] accessible uh for processing [4:39] well what the biggest bottleneck and the [4:41] biggest problem that data is everywhere [4:44] you know raw data comes in many shapes [4:46] data scattered and siloed fragmented [4:49] across SAS platforms uh fileshares or [4:52] different cloud environments like the [4:55] what you don't want to get into which is [4:57] very common is you've defined your [4:58] business problem you defined okay here [5:01] is here is the the problem that that we [5:02] have. Um you you've identified here are [5:05] the models that can help us with that [5:07] problem. Then you get realize all the [5:09] data is everywhere. So you have to start [5:11] a data pipeline process before you can [5:13] even get to modeling. [5:21] And most of the time it's not about [5:22] creating more data, it's about keeping [5:24] the right data. You know, you're already [5:26] generating AI ready data often without [5:28] realizing it. uh Zoom recordings, [5:31] support calls, user sessions, screen [5:32] recordings. It's all rich data and these [5:35] data sources could power tomorrow's new [5:37] models. Now, uh I know a client uh who [5:42] was using a third party AI service to [5:43] use to do customer service and uh that [5:46] primarily had all the information of all [5:48] their customer chats. That contract [5:50] ended and they they weren't storing all [5:52] that information on on their own. So, uh [5:55] once they they closed the contract, they [5:56] lost all that customer service data. So [5:58] that's something to think about that [5:59] where does your data actually live? Is [6:01] it living on a third party's tool that [6:03] once you end that contract you lose all [6:05] that data? [6:10] So first thing to think about is smart [6:12] ingestion. So centralizing like [6:14] centralizing data isn't a nice to have. [6:16] It's a strategic imperative. Now things [6:19] to think about is metadata is one of the [6:21] common unifiers across all your data [6:22] formats. As mentioned previously uh in a [6:25] previous role uh we were aggregating uh [6:28] CSV files, images, videos, PDFs and we [6:31] looked into the metadata of all those [6:33] files that allowed us to be able to [6:34] search across all of them. And that's [6:36] something to think about when you're [6:37] ingesting all the data from the [6:38] different sources because searchable [6:40] archives gets you to modeling faster. [6:45] Now some tools we'll go through every [6:47] stage and we'll talk about some tools [6:49] that will help in in each stage. Now, [6:51] backlaze B2 is one of that is is the [6:54] storage kind of built for this type of [6:55] stuff. Backlaze B2 offers predictable [6:57] immutable storage for your raw data [6:59] archives. There's also uh ETL tools, [7:02] ingestion tools to power the pipeline, [7:04] Apache Airflow, Fiverr, who are also [7:07] here. And the cool thing with Apache [7:08] Airflow, it's open source. So, codegen [7:11] works really well with with Apache [7:13] Airflow. uh code generation tools [7:16] primarily uh make perfect Apache Airflow [7:18] uh DAGs which are the scripts that you [7:20] use within Apache Airflow. Um so that's [7:24] so it primarily helps you create those [7:26] pipelines a lot more faster these days. [7:28] You have to think about metadata as we [7:30] mentioned and you have to think about [7:31] governance being built in you know [7:33] enabling object lock to prevent [7:34] tampering and use life cycle rules to [7:36] manage costs by order archiving style [7:38] data. [7:41] One of my favorite quotes [7:43] uh from Peter Novik, Google's research [7:46] director, is more data beats clever [7:48] data. Uh more data beats clever [7:50] algorithms, but better data beats more [7:52] data. Like you think about it, all the [7:55] LLMs, all the top LLMs have access to [7:57] the same data, all the all the text on [7:59] the web, but they all have different [8:01] performances. A big key part of that is [8:03] how they clean the data or them cleaning [8:06] the data. I remember a time where open [8:08] AAI primarily were hiring subject m [8:10] matter experts in every field across [8:12] different languages and they spent the [8:14] time to clean the data that they were [8:16] they pretty much collected across the [8:18] web. Um and that's why you see them kind [8:20] of having more of that uh better better [8:24] results than say goo Google's original [8:26] uh B. [8:29] So that's when we get into data [8:30] processing. The important part um you [8:33] know because the quality of your outputs [8:34] depends on the process on your [8:36] processing discipline. Your processing [8:38] stage defines how fast your team can [8:40] iterate. [8:42] Like first thing you're kind of doing as [8:44] you're bringing your data in, you're [8:46] thinking about cleaning and labeling [8:47] your data. You know data quality is uh [8:49] your AI foundation here. Like labeling [8:52] turns data into training material. You [8:55] want to be storing cleaned and labelled [8:57] data sets on immutable B2 buckets. And [8:59] with object locks, you can retain [9:00] original inputs for auditing and roll [9:02] back [9:05] feature engineering. So features are the [9:08] bridge between raw data and model [9:10] insights. This is where we extract [9:12] meaningful ins uh meaningful signals [9:14] like trends or patterns uh that models [9:16] can learn from. Efficiency. Efficiency [9:19] starts with the right format. This [9:21] helped optimize performance and reduce [9:23] storage costs. and versioning f you got [9:26] to think about versioning feature sets [9:27] to enroll raw back reproducibility and [9:30] compliance. [9:32] So this is just an example of data [9:34] processing of a video file. You start [9:37] you start with uh the video data set. [9:40] You get into data cleaning. You you [9:42] split up the video file into fragments [9:45] or or frames. You're then you're [9:47] starting to label it with annotation [9:48] sides with you're doing that manually or [9:51] automated with ML tagging. And then [9:52] you've got all these processes of [9:54] feature extraction, feature processing, [9:57] uh you know, histogram of war gradients [10:01] for detects edges and shapes. Histogram [10:03] of optic flow captures movement patterns [10:05] between frames. Uh motion analysis [10:07] reveals behavioral insights and so every [10:10] stage generates value data assets from [10:12] raw video fragments to process features [10:14] to evaluation metrics. So all and then [10:17] you have all this stored in version on [10:20] back blaze object storage. [10:27] As part of data processing, you you got [10:29] to think about partitioning. [10:31] Partitioning enables pipeline controls. [10:33] Splitting data sets by training, [10:35] validation, and inference. Uh improves [10:37] reproducibility. Versioning supports [10:39] lineage uh by recording changes to data [10:42] set procedures and labels. Then [10:44] governance starts with lineage. Log [10:46] every transformation, schema change, and [10:48] data source used. So store version [10:50] snapshots on your storage [10:56] and just some tooling examples there. Um [10:59] processing tools like Spark Desk and [11:01] Beam. Uh you've got labeling tools uh [11:04] like label studio and label studio is [11:07] open source. You have snorkel. They're [11:09] also here. You've got to think about the [11:11] table formats and feature stores. Um and [11:14] most of these are open source tools so [11:16] they're available to kind of everyone to [11:17] to use. Um and then you store your [11:20] engineered features enabling object [11:22] versioning for traceability and use [11:25] event notifications to trigger next [11:26] steps automation. [11:31] So stage three model experimentation and [11:33] training and this is the stage that [11:34] turns passive data into your active [11:36] product. You know, experimentation [11:38] drives iteration. [11:41] Like key things within this stage um is [11:44] using streaming or pre-etching to feed [11:46] curated data sets into high performance [11:49] computer environments efficiently. Data [11:51] sharding boosts through uh boosts [11:53] throughput and allows you to paralyze [11:55] training jobs and reduce IIO [11:57] bottlenecks. You want to be saving your [11:59] model checkpoints, configs, and runtime [12:01] environments to enable rollback [12:02] comparison and governed reuse. [12:06] And throughut is really important in [12:07] this stage. [12:12] But training isn't always smooth. You [12:14] know, a checkpoint is a safe state uh [12:18] state of model during training. More [12:21] frequent checkpoints reduce risk of data [12:23] loss and make it easier to resume [12:24] training. So checkpoints enable and [12:26] checkpoints enable targeted fine-tuning [12:28] experiments as well. [12:34] So some tooling examples of model and [12:35] training you've got your core uh [12:36] frameworks that drive training pipelines [12:39] industry standards like PyTorch and [12:40] TensorFlow u you got experiment tracking [12:43] uh which is critical to move fast [12:45] responsibly [12:46] um and data access can match compute [12:49] speed as well. [12:52] So [12:54] this is where uh you've got to think [12:56] about high throughput storage uh B2 [12:59] overdrive tier to reduce training [13:00] latency uh store your model checkpoints [13:03] and convenient default versions and [13:04] replicate data for distributed training [13:11] and deployment and inference. You know [13:12] model delivery is just as important as [13:15] model quality. [13:17] The things that are happening in this [13:18] stage, you you've got your package and [13:20] versioned your fine-tuned models so they [13:22] can be deployed consistently across [13:23] environments. You're serving models [13:25] across across cloud environments. You're [13:27] processing app data points and reducing [13:30] uh and returning predictions. And you're [13:32] storing everything uh for visibility and [13:35] compliance. [13:38] And this is high level uh overview of [13:41] inference. As you can see, you've got [13:44] key areas of data of data storage here. [13:47] Um, you've got your application data [13:49] with with your your inference data with [13:52] your application data. You got your [13:53] pre-processing and this is where you're [13:55] vectorizing that data. Um, it's going [13:58] through your model. Your model is [13:59] pulling in uh weights and configs from [14:01] your data store. And then you have your [14:03] prediction data, output data as well. [14:05] So, you can see how important storage is [14:07] and throughput is in your inference when [14:09] serving your model. [14:15] So it's just some toolings there. So [14:17] model serving infrastructure must be [14:19] reliable and scalable with tools like K [14:21] serve torch serve. Uh model registries [14:24] ensures safe and structured deployment. [14:27] This helps tracks versions metadata and [14:29] approval workflows. U CI/CD drives [14:32] automation and agility and some and you [14:35] you've got to think about storing your [14:36] models in buckets with object lock [14:38] integrity uh with object block for [14:40] integrity and roll back. [14:43] Stage five monitoring and feedback [14:46] because you know you can't monitor what [14:48] you don't manage. So feedback and [14:50] feedback l drives continuous [14:51] improvements. So we have one client [14:54] that's collecting and monitoring uh [14:56] terabytes of log data. [15:00] So things that are happening at this [15:01] stage [15:04] because every prediction should be uh [15:06] traceable. So you're logging predictions [15:08] late to user feedback. Ground truth [15:11] enables real uh evaluation. So you [15:13] measure you you measure accuracy over [15:15] time. Model drift is inevitable. So you [15:17] want to be detecting model drift early [15:19] and keeping log structured because [15:22] structured logging equals scalable [15:23] monitoring. [15:27] So just so the tooling examples there [15:30] you have monitoring tools like [15:31] Prometheus and my logs. Uh you want to [15:33] be storing logs in version buckets using [15:35] event notifications trigger next steps [15:37] automation and retain logs and metrics [15:39] and outcomes in B2. [15:43] A quote from our CEO GL Budman. Now [15:45] video is dramatically more data than an [15:47] image which is dramatically more than a [15:49] text message. That level of data is just [15:51] going to continue to explode. [15:55] We went through how rich video data can [15:57] be. So video is becoming the new [15:59] baseline. U so you really have to think [16:02] about where you're capturing video and [16:04] where you're storing that video because [16:06] there's this exponential growth requires [16:09] storage. [16:13] So optimizing data storage with backways [16:16] because you have to think about storage [16:18] as the leverage as it powers every stage [16:20] of the AI pipeline. [16:23] You can see here storage is the common [16:25] thread. Backlays B2 supports the the [16:28] full ML life cycle with B2 as the enable [16:31] layer. [16:38] So doesn't match backs B2 easy to start [16:40] S3 compatible plug and play reliable by [16:43] design and affordable at scale. [16:49] And we our latest offering is called [16:51] backways B2 overdrive uh up to terabit [16:54] second throughput acceler so to [16:56] accelerate training fine tuning and [16:57] inference no eress freeze and pretty [17:01] much storage built for AI workloads. [17:06] The main use cases for it are AI and ML, [17:09] you know, fast secure delivery to GPU [17:12] clusters, high performance compute, [17:13] scalable storage for simulations and [17:15] analytics, uh, CDN streaming, low [17:18] latency origin for media and content [17:20] delivery. [17:23] So, if you want to learn more, you can [17:25] scan this QR code. I'll leave it up or [17:27] visit us at booth uh 226. [17:30] Um, [17:32] thank you. [17:34] [Applause]