Meta Cuts Edits 4K Video Scrub Lag Tenfold, Deploys AI Coding Agents

@Scale

Eliminating redundant keyframe decoding cuts mobile 4K timeline lag tenfold, while autonomous AI agents take over low-level debugging to shift software engineers into supervisory roles.

AI-assisted summary of Chasing a Moving Target: Going AI-Native on Edits' Video Seek Engine | Boris Baracaldo from Meta — original source. Check the original for full context.

Source published: 2026-10-30

Key ideas

Meta's Edits playback engine achieved a 10x improvement in seek latency, frame rendering rate, and time-to-first-frame across multi-track 4K timelines by decoupling UI touches from decoder execution and eliminating redundant intra-GOP decoding. (12:54)
Seeking backwards requires specialized convergence handling because decoding advances forward from keyframes while the finger moves backward toward them, risking visual flickering if frames crossing the finger boundary are rendered prematurely. (12:54)
Meta addresses the combinatorial explosion of timeline editing permutations by using AI agents to generate deterministic test media, automated ADB gesture suites, and pixel-level validators rather than relying on LLMs for subjective code-level decisions. (23:12)
The long-term development lifecycle transitions human engineers from being 'in the loop' to 'on the loop,' shifting developers into supervisory roles while autonomous agents handle telemetry bisection, patch generation, and experiment rollout cleanup. (27:22)

Detailed notes

## Background Published on YouTube by @Scale on October 30, 2026, this presentation features Meta Android engineer Boris Baracaldo breaking down the video seek engine powering Meta's mobile editing app, Edits. No additional show notes or description were provided on the original video page.

## TL;DR Boris Baracaldo explains the technical challenges of delivering fluid, real-time scrubbing across multi-track 4K timelines on resource-constrained mobile hardware for Meta's Edits app. He walks through architectural solutions to bypass redundant decoding cycles within Groups of Pictures (GOPs) and eliminate visual flickering during backward seeks, unlocking a 10x performance leap. To manage the combinatorial explosion of timeline variables and continuous feature growth, he details Meta's shift toward an AI-native engineering loop. In this setup, autonomous agents leverage deterministic test harnesses and sliced telemetry analysis to build and diagnose features, shifting software engineers from operating directly in the loop to governing supervisory gates on the loop.

## Atmosphere & style The presentation carries the measured, pragmatic cadence of an experienced systems engineer methodically unpacking low-level hardware constraints and decoding mechanics. Baracaldo grounds the discussion in concrete engineering realities rather than speculative AI hype, conveying genuine technical satisfaction when explaining counterintuitive solutions like opposing-direction seek convergence. The pacing feels rigorous and insider-focused, creating an intellectual arc that transitions smoothly from deep codec-level troubleshooting to an architectural vision for autonomous developer tooling.

## Bulletin - [12:54](https://www.youtube.com/watch?v=hAurRPc3A10&t=774s) Meta's Edits playback engine achieved a 10x improvement in seek latency, frame rendering rate, and time-to-first-frame across multi-track 4K timelines by decoupling UI touches from decoder execution and eliminating redundant intra-GOP decoding. - [12:54](https://www.youtube.com/watch?v=hAurRPc3A10&t=774s) Seeking backwards requires specialized convergence handling because decoding advances forward from keyframes while the finger moves backward toward them, risking visual flickering if frames crossing the finger boundary are rendered prematurely. - [23:12](https://www.youtube.com/watch?v=hAurRPc3A10&t=1392s) Meta addresses the combinatorial explosion of timeline editing permutations by using AI agents to generate deterministic test media, automated ADB gesture suites, and pixel-level validators rather than relying on LLMs for subjective code-level decisions. - [27:22](https://www.youtube.com/watch?v=hAurRPc3A10&t=1642s) The long-term development lifecycle transitions human engineers from being 'in the loop' to 'on the loop,' shifting developers into supervisory roles while autonomous agents handle telemetry bisection, patch generation, and experiment rollout cleanup.

## Key takeaways - [00:01](https://www.youtube.com/watch?v=hAurRPc3A10&t=1s) Mobile editing expectations evolved from basic Instagram Stories filters in 2016 to multi-track 4K video manipulation on smartphones with the launch of Meta Edits in 2025. - [01:22](https://www.youtube.com/watch?v=hAurRPc3A10&t=82s) Video scrubbing performance cannot be solved merely by speeding up existing algorithms; the engine must fundamentally eliminate redundant decoding work under mobile hardware constraints. - [03:13](https://www.youtube.com/watch?v=hAurRPc3A10&t=193s) Because delta frames require sequential decoding starting from an I-frame, arbitrary seeking computational cost scales directly with the distance between keyframes (GOP length). - [06:14](https://www.youtube.com/watch?v=hAurRPc3A10&t=374s) Pre-indexing keyframe presentation timestamps during asset import enables the engine to bypass full keyframe resets when a new seek request falls within the active GOP. - [09:25](https://www.youtube.com/watch?v=hAurRPc3A10&t=565s) Decoupling the UI touch event listener from the blocking worker decode thread prevents stale frames from queueing up and degrading perceived responsiveness. - [11:18](https://www.youtube.com/watch?v=hAurRPc3A10&t=678s) Forward seeking maximizes perceived fluidity by interrupting stale decode passes and rendering newly encountered keyframes immediately as intermediate visual progress. - [12:54](https://www.youtube.com/watch?v=hAurRPc3A10&t=774s) Backward scrubbing forces the decoder to fight standard container formats, requiring the engine to wait until forward decoding crosses the finger's backward position to avoid visual flickering. - [12:54](https://www.youtube.com/watch?v=hAurRPc3A10&t=774s) Testing with three overlapping 4K video tracks validated a 10x gain in average seek time, frame rendering rate, and initial frame display latency. - [16:33](https://www.youtube.com/watch?v=hAurRPc3A10&t=993s) The combinatorial explosion of track counts, resolutions, effect stacks, and scrub velocities makes it impossible for human engineers to manually explore the entire optimization problem space. - [18:22](https://www.youtube.com/watch?v=hAurRPc3A10&t=1102s) Global metric averages mask critical performance regressions; observation agents equipped with domain knowledge are required to isolate issues in hyper-specific slices. - [22:29](https://www.youtube.com/watch?v=hAurRPc3A10&t=1349s) Effective agentic workflows require human architectural intent to be translated into rigid, deterministic artifacts and executable tests that return binary pass/fail statuses. - [23:12](https://www.youtube.com/watch?v=hAurRPc3A10&t=1392s) Because LLMs are inherently stochastic, engineering teams should use them to generate deterministic tooling, test assets via FFmpeg, and ADB gesture harnesses rather than making dynamic operational choices. - [26:33](https://www.youtube.com/watch?v=hAurRPc3A10&t=1593s) Removing human-style judgment from deterministic tasks allows LLMs to focus their reasoning power on exploring code diffs to turn red test suites green. - [27:22](https://www.youtube.com/watch?v=hAurRPc3A10&t=1642s) Autonomous engineering workflows shift human developers from being 'in the loop' to 'on the loop,' retaining governance over prioritization while agents manage bisections, PR diffs, and cleanup. - [29:02](https://www.youtube.com/watch?v=hAurRPc3A10&t=1742s) The ultimate engineering benchmark for a mobile video seek engine is complete invisibility to the creator, ensuring the timeline simply reacts instantly to touch.

## Detailed walkthrough ### [00:01](https://www.youtube.com/watch?v=hAurRPc3A10&t=1s) The Evolution of In-App Editing and the Launch of Edits

Back in 2016, Instagram had just released Stories. At that point, editing a video directly inside the app was basic: a user might stitch together a few clips, slap on some text or a simple filter, and finalize everything without ever leaving Instagram. By 2020, short-form video exploded globally with the rollout of Reels, turning daily video recording and editing into a mainstream habit. By 2022, serious creators had outgrown these lightweight, built-in editors and transitioned to specialized third-party desktop and mobile software to produce polished, high-production content.

To meet this creator demand, Meta entered the dedicated mobile editing space in 2025 by announcing Edits, an app tailored for professional-grade video creation directly on mobile hardware. Four months after its announcement, Edits officially launched. The app arrived equipped with support for 4K video playback, overlapping video and audio tracks, splitting, trimming, and an extensive catalog of creative effects. The overarching objective was to package professional video editing capabilities into an interface simple enough to manipulate with just a thumb on a smartphone screen, helping creators translate their artistic vision into content capable of reaching billions.

- In 2016, Instagram Stories offered basic in-app video composition (clips, text, filters). - The 2020 boom driven by Reels made daily short-form video creation ubiquitous. - By 2022, high-end creators migrated toward dedicated external editing suites. - In 2025, Meta announced and subsequently shipped Edits, supporting 4K resolution, multi-track audio/video layering, and non-linear editing tools.

---

[01:22](https://www.youtube.com/watch?v=hAurRPc3A10&t=82s) The Performance Challenge of Video Scrubbing

Boris Baracaldo, an Android engineer working on the shared playback engine behind Instagram Reels, Stories, and the Edits app, spent the prior year laser-focused on optimizing a single core interaction: timeline scrubbing. The goal is keeping frame rendering fast, continuous, and completely synchronized with the user's touch as their finger glides across the composition. While dragging a finger along a timeline feels intuitive and seamless to an end user, the underlying media engine must perform an immense amount of work. Under the hood, scrubbing translates into a rapid cascade of seek requests, each requiring the demuxing, decoding, and pixel rendering of individual video frames onto the mobile display.

The fundamental engineering obstacle is mobile hardware constraints colliding with heavy computational workloads. Decoding costs scale aggressively with resolution: decoding a single 4K frame demands roughly four times the computational power required for a 1080p frame. When a timeline incorporates multiple overlapping tracks, these costs compound multiplicatively, generating an order-of-magnitude increase in required device workloads. When comparing a single 1080p video track side-by-side against three overlapping 4K video tracks, the single track keeps pace smoothly, while the multi-track 4K composition exhibits severe, visible lag as the player struggles to follow the finger. Confronting this challenge requires a fundamental architectural shift. As Baracaldo points out: “Closing a gap of more than 10 times the work. It's not a matter of tuning the algorithm. The algorithm has to do less work, not the same work faster.”

```mermaid flowchart TD accTitle: Video Demuxing and Decoding Pipeline accDescr: Pipeline showing how container metadata and frame data pass through the demuxer and decoder to render pixels.

subgraph Container["MP4 Container"] Meta["Metadata (Resolution, Duration, Keyframe Map)"] RawData["Frame Bitstream Data"] end

subgraph Engine["Playback Engine"] Demux["Demuxer (Extracts Frame Bytes)"] Decode["Decoder (Turns Bytes into Pixels)"] end

Display["Rendered Screen Pixels"]

Meta -.->|"Offsets & GOP structure"| Demux RawData --> Demux Demux --> Decode Decode --> Display ```

---

[03:13](https://www.youtube.com/watch?v=hAurRPc3A10&t=193s) Video Container Architecture and GOP Constraints

To find opportunities to eliminate redundant work, the engine must exploit how video is encoded and stored inside media containers like MP4 files. An MP4 file consists of two primary elements: 1. **Metadata**: Contains fundamental stream properties including display resolution, duration, framerate, and a structural index mapping every keyframe's location throughout the timeline. 2. **Bitstream Data**: Contains the compressed byte payload for every individual frame.

Processing this payload relies on two software components: the demuxer, which unpacks the container to extract raw frame bytes, and the decoder, which converts those compressed bytes into raw displayable pixels.

Visualizing a video track as a traditional physical film strip reveals why seeking is computationally expensive. Video compression algorithms do not store every frame as a standalone image. Instead, they rely on keyframes (I-frames), which store a complete visual representation of an image. The vast majority of remaining frames hold only partial, delta-encoded visual data relative to neighboring frames, a compression strategy designed to drastically lower file sizes. Because delta frames cannot be decoded in isolation, a video decoder can only initiate decoding from a keyframe. Decoding any arbitrary target frame requires sequentially decoding all intermediate delta frames starting from the preceding keyframe, meaning that the computational cost of a single seek operation scales directly with the length of the Group of Pictures (GOP)—the distance between two successive keyframes.

The presentation structure outlines four core stages for addressing this seek engine performance bottleneck: 1. Optimizing an individual seek request by eliminating redundant decoding work within a GOP. 2. Handling a dynamic stream of seek requests while scrubbing to sustain perceived fluidity. 3. Managing the multi-dimensional complexity explosion using integrated AI technology. 4. Leveraging AI to reduce the human maintenance cost of complex video infrastructure.

---

[06:14](https://www.youtube.com/watch?v=hAurRPc3A10&t=374s) Optimizing Single Seek Requests via In-GOP Forward Decoding

In a naive video player architecture, receiving a seek request triggers an automated fallback: the engine inspects the container metadata, jumps back to the nearest keyframe preceding the target timestamp, and sequentially decodes every intermediate frame until reaching the destination. While this guarantees accurate reconstruction, it wastes significant processing cycles. Because intermediate frames decoded along the way are discarded rather than rendered, jumping backward to a keyframe when the playhead is already ahead of it forces the engine to decode the exact same frames twice.

```mermaid flowchart TD accTitle: Single Seek Decision Flow accDescr: Logic to determine whether to skip keyframe jumping if the target lies within the active GOP.

Req["Incoming Seek Request"] --> Check{"Is target in current GOP?"} Check -- "No (Different GOP)" --> Jump["Jump to Preceding Keyframe"] Jump --> DecodeSeq["Decode Linearly Forward to Target"] Check -- "Yes (Same GOP)" --> SkipJump["Skip Keyframe Jump"] SkipJump --> DecodeFromCurrent["Decode Forward from Current Position"] DecodeSeq --> Render["Render Target Frame Pixels"] DecodeFromCurrent --> Render ```

To eliminate this waste, the player must evaluate a specific question on every incoming seek request: *Is the requested timestamp within the same Group of Pictures (GOP) as the player's current position?* If the target timestamp is within the active GOP and lies ahead of the current position, the engine can completely bypass the keyframe jump, continuing forward decoding directly from its current position to the target.

Executing this logic on Android is difficult due to the limitations of platform primitives. The native Android OS `MediaExtractor` API is designed around simple linear media playback: it exposes methods to jump directly to a keyframe, step forward by one frame, and query metadata for that active frame, but it hides the container's underlying keyframe distribution table. As Baracaldo explains: “And this sounds obvious enough but with the primitives exposed by the Android OS and the media extractor APIs this is not so simple because we cannot observe the full structure of the key frames without red coding those that information from the file itself.”

Rather than rewriting a low-level demuxer from scratch, the team built a lightweight indexing mechanism using the existing Android extraction primitives: - At asset import time, the engine traverses the media file once by executing successive keyframe-only jumps. - During this single traversal, the engine logs the exact presentation timestamp (PTS) of every keyframe into an in-memory index table. - When live seek requests arrive during user scrubbing, the engine queries this timestamp map to determine whether the target frame shares the current GOP, bypassing keyframe resets whenever safe.

---

[09:25](https://www.youtube.com/watch?v=hAurRPc3A10&t=565s) Handling Moving Targets and Decoupling UI from Decoder Execution

Optimizing a single isolated seek request is insufficient for dynamic scrubbing because user interaction turns seek targets into moving targets. While the player is busy decoding toward a specific frame, the user's thumb is actively moving across the screen, potentially crossing multiple GOP boundaries. By the time the decoder completes work for a requested frame, that frame may already be completely obsolete.

Android captures timeline touches as a continuous sequence of discrete seek events dispatched along the main UI thread. For instance, an aggressive fling across a timeline from frame 0 to frame 1,000 generates intermediate requests dispatched at intervals such as 100, 300, 500, and beyond. The player architecture decouples this interaction across two distinct threads: 1. **The UI Thread / Request Listener**: Monitors touch input and captures incoming seek targets from UI interactions. 2. **The Worker / Decoder Thread**: Performs blocking, computationally heavy I/O operations—specifically demuxing container data, parsing bitstreams, and running hardware decoder pipelines.

```mermaid flowchart LR accTitle: Dual-Thread Seek Request Architecture accDescr: Architecture showing asynchronous decoupling between UI touch input requests and blocking decode operations.

subgraph UI_Thread["UI Thread"] Touch["User Scrub Gesture"] --> Queue["Dispatched Seek Requests (100, 300, 500...)"] end

subgraph Worker_Thread["Worker Thread (Blocking)"] Worker["Seek Engine Controller"] Pipeline["Demuxing & Hardware Decoding"] Worker <--> Pipeline end

Queue -->|"Async Events"| Worker Worker -.->|"Interrupt & Drop Stale Frames"| Pipeline ```

Because the worker thread performs blocking operations that take tangible time, intermediate UI requests rapidly pile up. If the decoder blindly fulfills every request in sequential order, it wastes CPU and GPU cycles decoding stale frames that the user has already scrubbed past. Furthermore, user scrubbing behavior is unpredictable: a user may rapidly drag forward, pause, fling toward the end of the video, or immediately scrub backward. To maintain low latency and responsive visual feedback, the engine needs active interruption levers, halting decoding operations the moment a newer seek request renders the ongoing decode pass stale.

[11:18](https://www.youtube.com/watch?v=hAurRPc3A10&t=678s) Forward Seeking Logic and the Challenge of Backward Seeking

To optimize seeking, the engine relies on two primary levers: aborting stale work when the seek direction changes so it avoids decoding forward unnecessarily, and rendering as often as possible to refresh pixels on screen whenever progress can be displayed. Outlining the step-by-step logic, Boris Baracaldo explains that when a new seek request arrives, the engine interrupts ongoing work and renders immediately to show visual progress. It then checks whether the new request falls within the same group of pictures (GOP) as the frame just decoded. If it does, the decoder does not jump to a keyframe; if it does not, jumping is unavoidable. However, whenever a keyframe jump occurs, the engine renders that keyframe immediately because it represents the most visual progress achievable per unit of time. From there, it decodes linearly until reaching the target, only rendering at the exact final position if no intervening requests arrive.

```mermaid flowchart TD accTitle: Seek Request Processing Flow accDescr: Flowchart showing decision logic when a seek request arrives, GOP matching, keyframe jumps, and rendering logic. A["Seek request arrives"] --> B["Interrupt & render progress"] B --> C{"Same GOP as last decoded frame?"} C -->|"Yes"| D["Decode linearly forward"] C -->|"No"| E["Jump to GOP keyframe"] E --> F["Render keyframe immediately"] F --> D D --> G{"New request interrupted?"} G -->|"No"| H["Render final target frame"] G -->|"Yes"| A ```

While linear decoding and keyframe jumping work smoothly for forward seeking, backward scrubbing directly clashes with video compression standards. As Baracaldo points out: “Now, this is great for forward seeking. That's what the format is built for. However, if you want to seek backwards and you want to start scrubbing towards the start of the video, then we start fighting the format because we need to jump to the key frame.” Even if the user drags their finger to a timestamp just slightly behind the current frame, the engine must jump all the way back to the keyframe at the start of that GOP and sequentially decode every intervening frame. In media where GOP intervals span up to five seconds, that forces the decoder to process dozens or hundreds of frames just to step backward a tiny fraction of a second.

[12:54](https://www.youtube.com/watch?v=hAurRPc3A10&t=774s) Resolving Backward Seeking Flickering and Results

Handling backward gestures requires treating the scrub as a continuous physical operation rather than isolated seek requests. As the user's finger drags backward, it is actually moving closer to the keyframe that the engine has jumped to. When a second seek request interrupts the first, the target shifts to the finger's new position. This sets up an opposing race: the finger moves backward toward the keyframe while the decoder moves forward from the keyframe toward the finger. Baracaldo cautions that if the decoder has not yet crossed the finger's current position at the time of interruption, rendering that decoded frame would overshoot user intent and introduce flickering—flashing a backward frame followed by a forward jump once the finger's position is reached. The decoder must continue advancing forward until it crosses the boundary of the finger's position. Only then can it safely render and restart the cycle, ensuring monotonic visual progression without visual stutter.

```mermaid flowchart LR accTitle: Opposing Direction Seeking Convergence accDescr: Illustration of keyframe jump decoding moving forward while the finger scrubs backward toward the keyframe. subgraph Stream["GOP Timeline"] KF["Keyframe (Start)"] -->|"Decoding forward"| DF["Decoded Frame"] DF -.->|"Must cross boundary"| FP["Finger Position"] FP -->|"Scrubbing backward"| KF end ```

Combining these optimizations yielded a dramatic performance leap when tested locally on heavy compositions featuring three overlapping 4K video tracks: - **10x improvement** in average seek time. - **10x improvement** in frame rendering rate. - **10x improvement** in time to render the first frame.

Reflecting on these gains, Baracaldo highlights two core takeaways. First, scrubbing is not about hitting an exact seek timestamp instantly; “It's about the perceived performance doing a continuous gesture. So it's physically impossible for us to show every single frame when the user is scrubbing. So the target for us is choosing exactly which frames are actually worth the most to render into the screen.” Second, standard video container formats are engineered for continuous linear playback, not arbitrary scrubbing. To solve this at the root, the team has begun generating proxy media featuring optimized GOP sizes and downscaled resolutions matched to phone screens to streamline previewing and seeking.

[16:33](https://www.youtube.com/watch?v=hAurRPc3A10&t=993s) The Combinatorial Complexity of Video Scrubbing

Despite the performance wins, Baracaldo stresses that video editing engines demand constant upkeep as product teams add layered compositions, multiple video and audio tracks, and real-time AI effects. Shipping an optimization from initial observation to release is costly, prompting the team to investigate agentic workflows to scale development. The difficulty lies in the sheer combinatorial explosion of scrubbing variables:

- **Track resolution & overlap:** Scrubbing over multiple simultaneous 4K or mixed-resolution tracks with varying GOP sizes. - **Effect stacks:** Chains of filters and real-time image processing applied simultaneously. - **Directional asymmetry:** Forward versus backward seeking incurring radically different decoding costs. - **Gesture velocity:** Slow scrubbing signals that the user cares about precise positioning, whereas rapid scrubbing means they only care about rough trajectory rather than individual frames. - **Timeline scale:** Meta Edits allows full frame-level zooming where showing every frame matters, contrasting with zoomed-out views showing 10 minutes of footage where users do not inspect exact landing frames.

Baracaldo explains: “So from this complexity, we cannot realistically explore every combination. So what does it take for us to go AI native on the software building process?”

[18:22](https://www.youtube.com/watch?v=hAurRPc3A10&t=1102s) Going AI-Native: The Software Building Workflow and Observation Agents

To transition to an AI-native engineering model, Meta abstracts the software engineering lifecycle into five discrete stages: 1. **Observing:** Monitoring production telemetry across multidimensional combinations. 2. **Diagnosing:** Isolating bottlenecks, regressions, or decoder warm-up overhead. 3. **Building:** Implementing performance optimizations or bug fixes. 4. **Experimenting:** A/B testing changes so improvements for one cohort do not silently degrade playback for other users across billions of heterogeneous devices. 5. **Cleaning up:** Pruning dead experiment code to prevent crushing technical debt.

To empower agents across these stages, engineers are building a shared domain context layer that encapsulates domain-specific quirks—such as why backward seeks cost more than forward seeks or how GOP boundaries behave. Baracaldo zooms in on the observation stage to illustrate why agents are essential. When production metrics are averaged globally across all users and devices, the resulting numbers look benign and mask deep regressions. The real performance pain clusters in hyper-specific slices, such as a specific scrub velocity over a certain number of overlapping tracks or the latency penalty of warming up a new hardware decoder when switching between clips.

Baracaldo notes: “So it's very hard for one human to actually observe and explore the whole problem space.” By equipping agents with domain knowledge about gesture dynamics, composition shapes, and interaction phases, the agents can automatically slice metric spaces, spin up focused diagnostic dashboards, and formulate technical hypotheses just like a systems engineer. When automated anomaly detectors flag a regression in a specific slice, the observation agent bundles the diagnostics into a structured package that serves as the direct input for the downstream code-building phase.

[21:54](https://www.youtube.com/watch?v=hAurRPc3A10&t=1314s) Building Features with Agents: Human Intent vs. Agent Execution

Baracaldo transitions to the building phase of the workflow, emphasizing the unique challenges of writing and modifying video engine code targeted at resource-constrained mobile hardware like Android devices. Operating an agentic development loop requires a sharp division of labor between engineers and AI models. From humans, the process demands core intent, algorithmic design, architectural foresight, and system design thinking. The agent, in turn, must consume that architectural framing alongside the shared domain knowledge to translate high-level strategies into robust, device-level video processing code.

[22:29](https://www.youtube.com/watch?v=hAurRPc3A10&t=1349s) Defining Deterministic Artifacts and Executable Code

To harness AI agents effectively during development, engineers cannot rely on ambiguous instructions. Instead, they must translate human intent into rigid, deterministic artifacts that an agent can execute and verify autonomously. When an agent is tasked with building or refactoring video playback features, it must know precisely which test cases to run to confirm that the video renders properly without artifacts like flickering or dropped black frames. Achieving this requires formal specifications outlining the expected behavior, structured implementation plans with granular execution steps, and executable code commands that return an unambiguous pass or fail status.

As Boris Baracaldo emphasizes, moving from ambiguous prompts to executable code is critical: “And from this intent we need to create artifacts that are deterministic for an agent to execute.” By giving an agent automated commands to run against the build, the system eliminates subjective guesswork and establishes an objective standard for when the task is genuinely finished.

- **Feature specification**: Explicit document defining the exact functional behavior and performance boundaries expected. - **Implementation plan**: Step-by-step roadmap tailored for the agent to follow sequentially during code generation. - **Executable validation code**: Direct CLI commands or automated test suites that run programmatically to confirm completion.

---

[23:12](https://www.youtube.com/watch?v=hAurRPc3A10&t=1392s) Grounding Stochastic Agents with Deterministic Video Tooling

A major pitfall in building agentic software workflows is treating large language models as general-purpose decision engines for every step of an engineering pipeline. Because LLMs are inherently stochastic, running the same prompt repeatedly yields divergent decisions and inconsistent code paths. To counter this variability, the engineering strategy shifts toward having agents write deterministic tools, harness scripts, and validation harnesses. What once required prohibitive engineering hours for a single human developer to hand-craft—bespoke testing infrastructure, media synthesizers, and platform hooks—is now extremely cheap to generate using AI.

In the context of the mobile video editing engine, this strategy breaks down into generating varied test media, compositing complex edit timelines, driving the UI, and verifying pixel-perfect outputs. Using tools like FFmpeg, agents create expansive test suites with varying resolutions, frame rates, and GOP sizes. They then generate scriptable compositions featuring overlapping video tracks, transitions, text overlays, and animated stickers or GIFs. To drive the application, scripts interface directly with Android testing utilities such as `dumpapp` to inspect the view hierarchy and issue ADB commands to simulate finger gestures. Throughout execution, the screen and system trace logs are captured. Deterministic validator scripts parse these traces and inspect decoded frame pixels to guarantee that playback tracks the user's finger with low latency, with zero flickering or unexpected black frames.

```mermaid flowchart TD accTitle: Deterministic Agent-Driven Video Validation Pipeline accDescr: An agent-driven testing workflow where inputs are generated, executed on a device, and validated at the pixel level.

subgraph Inputs ["1. Deterministic Input Generation"] A["Media Library (FFmpeg: GOP, FPS, Res)"] B["Composition Scripts (Tracks, Effects, Overlays)"] end

subgraph DeviceRun ["2. Device Automation & Capture"] C["UI Automation (dumpapp + ADB Gestures)"] D["Device Under Test (Record Screen & Traces)"] end

subgraph Evaluation ["3. Verification Loop"] E["Deterministic Pixel & Trace Validators"] F{"Pass or Fail?"} G["Agent Loops on Fixes (Explores Alternative Diffs)"] H["Validated Green Build"] end

Inputs --> DeviceRun C --> D D --> E E --> F F -->|"Red (Failure)"| G G --> D F -->|"Green (Pass)"| H ```

Boris Baracaldo explains the strategic design principle: “One of the best uses of agents is not necessarily throwing at every single problem you have and then hoping that it knows how to traverse every single decision that it needs to make because LLMs are stochastic and it will provide different versions of the same answer.” When the agent encounters a failing red test case, its strength lies in looping across dozens or hundreds of alternative implementations until the deterministic test harness turns green.

- **Input media generation**: Synthetic asset creation via FFmpeg across various resolutions, frame rates, and GOP structures. - **Complex composition generation**: Automated generation of test projects containing multi-track video, overlay effects, text, and GIFs. - **Device manipulation**: Android UI state inspection via `dumpapp` combined with ADB commands to execute repeatable scrubbing and tap gestures. - **Trace and pixel inspection**: Programmatic validation of trace logs and screen recordings to verify latency, gesture tracking, and frame consistency.

---

[26:33](https://www.youtube.com/watch?v=hAurRPc3A10&t=1593s) Core Principles for AI-Native Engineering Workflows

Reflecting on the evolution toward an AI-native development model, Baracaldo outlines two primary takeaways for engineering teams. The first principle is establishing an executable definition of "done." Providing an AI agent with qualitative guidelines or text-only descriptions invites ambiguity; success requires a concrete command that executes against the codebase and returns a binary determination of pass or fail.

The second principle is using LLMs to build deterministic tools rather than relying on them to make judgment calls during execution. By leveraging models to write verification scripts, test generators, and run-time harnesses, teams eliminate stochastic judgment from tasks that can be mathematically or procedurally verified. This leaves the model's creative reasoning focused exclusively on resolving edge cases and bridging the remaining engineering gaps.

As Boris Baracaldo emphasizes: “By removing the judgment from things that can be done deterministically, you're actually using LLMs what they're best and that is covering those gaps of judgment that are still remaining.”

---

[27:22](https://www.youtube.com/watch?v=hAurRPc3A10&t=1642s) Moving from Humans in the Loop to Humans on the Loop

The broader future of AI-native engineering extends far beyond isolated code-generation scripts. It connects production observability, bug reproduction, code generation, and live experimentation into a continuous automated lifecycle. In this operational model, telemetry-monitoring agents continuously inspect production performance, slicing across hardware tiers and operating system versions to pinpoint bottlenecks where user friction concentrates. Once an issue is identified, analysis agents inspect the repository, bisect git revisions across app releases, review code diffs, and recreate specific user compositions on physical or emulated devices.

Rather than forcing humans to handle every step of this triage, agents produce comprehensive review packages outlining the problem and proposed remedies. Human engineers retain high-level governance, deciding which technical problems are worth solving. Once approved, coding agents build the fix using the automated test-and-validation loop, package review diffs, and configure rollout experiments. The agents monitor metric telemetry over several days of deployment, draft release summaries for human sign-off, and subsequently perform dead-code cleanup and technical debt reduction.

```mermaid flowchart LR accTitle: Humans On the Loop Development Lifecycle accDescr: End-to-end autonomous engineering loop showing the shift of humans to supervisory review gates.

subgraph Triage ["Automated Triage"] A["Production Telemetry Analysis"] --> B["Bisect Diffs & Reproduce Issues"] end

subgraph BuildAndTest ["Implementation Loop"] C["Agent Code Generation"] --> D["Deterministic Validation Harness"] end

subgraph Operations ["Deployment & Telemetry"] E["Experiment Setup & Ramp"] --> F["Telemetry Readout & Cleanup"] end

subgraph Governance ["Human Supervisory Gates"] H1{{"Decision: Prioritize & Direct"}} H2{{"Decision: Approve Diff"}} H3{{"Decision: Approve Ship"}} end

B --> H1 H1 --> C D --> H2 H2 --> E E --> F F --> H3 ```

This model fundamentally alters the role of the software developer. Boris Baracaldo captures this organizational evolution directly: “Essentially, we're taking humans from being in the loop to on the loop.” Instead of manually shepherding a pull request through compilation, local verification, and canary monitoring, engineers act as supervisors guiding autonomous systems.

- **Production monitoring agents**: Autonomous background workers slicing operational telemetry across dimensions to locate performance bottlenecks. - **Automated bisection & reproduction**: Systems that isolate regression-inducing commits and recreate timeline states on target devices. - **Experiment orchestration**: Automated rollout management, metric tracking over release cycles, and synthesis of decision readouts. - **Codebase hygiene**: Autonomous cleanup of experiment flags, deprecated code paths, and architectural complexity after rollout.

---

[29:02](https://www.youtube.com/watch?v=hAurRPc3A10&t=1742s) Mobile Video Evolution and the Seamless User Experience

The video editing landscape has undergone a dramatic transformation over the past decade. Desktop environments with dedicated expansion cards and specialized hardware acceleration once represented the only viable way to edit complex video compositions. Today, mobile creators expect to manipulate multi-track timelines effortlessly on handheld smartphones using only a thumb. Maintaining high performance in a video seek engine remains a moving target because hardware profiles, operating system updates, and editing features are continuously shifting.

While the engineering and agentic automation behind video playback grow increasingly sophisticated, the ultimate measure of success is complete invisibility to the end user. Creators should never need to understand video codecs, decoder thread allocation, or the autonomous agent loops ensuring frame accuracy. The software must simply react instantly to human touch.

Boris Baracaldo concludes with this core product philosophy: “Their experience should be as simple as putting their finger down and observing how the product just works.”

## Notable quotes > [01:22](https://www.youtube.com/watch?v=hAurRPc3A10&t=82s) It's not a matter of tuning the algorithm. The algorithm has to do less work, not the same work faster. — Boris Baracaldo

> [27:22](https://www.youtube.com/watch?v=hAurRPc3A10&t=1642s) Essentially, we're taking humans from being in the loop to on the loop. — Boris Baracaldo

## Mentioned Boris Baracaldo, Meta, Edits, Instagram, Instagram Stories, Instagram Reels, Android, MediaExtractor, FFmpeg, ADB, dumpapp

[Source](https://www.youtube.com/watch?v=hAurRPc3A10)

中文简报

Meta将Edits的4K视频拖动延迟降低十倍，并部署AI编程智能体通过消除冗余的关键帧解码，移动端4K时间轴延迟降低了十倍，同时自主AI智能体接管了底层调试工作，使软件工程师转向监管角色。 Boris Baracaldo阐述了在资源受限的移动硬件上，为Meta的Edits应用实现多轨道4K时间轴流畅实时拖动所面临的技术挑战。他详细介绍了如何通过架构解决方案跳过图片组（GOP）内的冗余解码周期，并消除后退拖动过程中的画面闪烁，从而实现10倍的性能飞跃。为了应对时间轴变量的组合爆炸和持续的功能增长，他详细说明了Meta向AI原生工程闭环的转变。在这一体系中，自主智能体利用确定性测试工具和切片遥测分析来构建与诊断功能，使软件工程师从直接“参与闭环”（in the loop）转变为负责“监管闭环”（on the loop）。 Meta的Edits播放引擎通过将UI触摸操作与解码器执行解耦，并消除GOP内部的冗余解码，在多轨道4K时间轴上将拖动延迟、帧渲染率和首帧渲染时间提升了10倍。 (12:54) 向后拖动需要专门的收敛处理，因为解码是从关键帧向前推进的，而手指是向后朝向关键帧移动，如果越过手指边界的帧被过早渲染，就会面临画面闪烁的风险。 (12:54) Meta利用AI智能体生成确定性测试媒体、自动化ADB手势测试套件以及像素级验证器，来应对时间轴编辑排列组合的爆炸性增长，而非依赖大语言模型做出主观的代码级决策。 (23:12) 长期的开发生命周期正在推动人类工程师从“参与闭环”转向“监管闭环”，在自主智能体负责遥测二分分析、补丁生成和实验发布清理的同时，开发者转变为监督角色。 (27:22)

Reading summary

Meta engineer Boris Baracaldo explains that achieving real-time scrubbing in the mobile video editing app Edits required eliminating redundant frame decoding at the container level and shifting to an AI-native engineering workflow. To handle the computational load of multi-track 4K timelines on mobile devices, Meta redesigned its seek architecture and paired stochastic language models with deterministic validation harnesses. Hardware Constraints and Container GOP Bottlenecks (1:22) Mobile video scrubbing translates user gestures into rapid cascades of seek requests that demand heavy demuxing, decoding, and rendering. Decoding costs scale aggressively with resolution—a 4K frame requires roughly four times the computational power of a 1080p frame—and compound multiplicatively when multiple 4K tracks overlap. Baracaldo emphasizes that bridging this tenfold computational gap required an algorithm that performs less overall work rather than executing the same tasks faster. This bottleneck stems from standard video container formats like MP4, where compression relies on Groups of Pictures (GOPs). Video files store isolated keyframes (I-frames) alongside intermediate delta frames that cannot be decoded independently. Decoders must sequentially process all preceding delta frames starting from the nearest keyframe to render a target frame. Consequently, seek latency directly scales with GOP length, placing severe processing burdens on resource-constrained mobile hardware during continuous user interaction. In-GOP Decoding and Directional Asymmetry Solutions (6:14) To eliminate redundant decoding, Meta's engine evaluates whether an incoming seek request falls within the active GOP. Because Android's native MediaExtractor APIs do not expose full keyframe tables directly, the team engineered a single-pass indexer during file import that maps keyframe presentation timestamps in memory. If a new forward target shares the active GOP, the engine bypasses the preceding keyframe jump and continues linear decoding forward from the current playhead. The engine also decouples the UI thread from the blocking decoder thread, using interruption levers to abort stale decoding jobs whenever the user's touch moves on. Scrubbing backward introduces directional asymmetry because decoders must jump back to the GOP's keyframe and decode forward while the user's finger moves backward toward that same keyframe. To eliminate visual flickering caused by premature renders, the engine prevents the decoder from rendering intermediate frames until its forward decode path crosses the backward-moving finger position. Together with proxy media generation, these optimizations yielded a tenfold local performance improvement in average seek time, frame rendering rate, and time-to-first-frame across compositions with three overlapping 4K tracks. Managing Multidimensional Complexity with Telemetry Agents (16:33) Maintaining high scrubbing performance is challenged by a combinatorial explosion of production variables, including overlapping track resolutions, varying GOP lengths, real-time effect stacks, scrubbing velocities, and timeline zoom levels. Baracaldo notes that global metric averages tend to mask severe regressions that only appear in specific operational slices, such as high scrub velocities over stacked clips or decoder warm-up latency during asset transitions. To diagnose these edge cases, Meta employs observation agents embedded with video domain knowledge. These agents continuously slice telemetry across hardware tiers, identify localized performance regressions, spin up focused diagnostic dashboards, and formulate technical hypotheses. When an anomaly is detected, the agent packages trace data and repository bisections directly for the downstream code-generation phase. Grounding Stochastic LLMs with Deterministic Tooling (23:12) Because large language models are stochastic and prone to inconsistent implementations, Meta avoids using them for subjective, open-ended decision-making in the playback engine. Instead, engineers provide formal specifications and executable definitions of completion, using agents primarily to generate deterministic testing infrastructure that was previously too expensive to hand-craft. Agents write automated suites that construct synthetic media via FFmpeg across various resolutions and GOP sizes, build complex multi-track timelines, simulate touch gestures using Android's dumpapp and ADB commands, and parse pixel traces to verify zero flickering. When a test fails, the agent autonomously iterates on code patches until the deterministic harness returns an unambiguous pass. Baracaldo describes this as moving engineers from 'in the loop' to 'on the loop,' where humans act as supervisors directing problem selection, reviewing packaged diffs, and approving live rollouts. Baracaldo concludes that while playback engines and autonomous development loops are growing increasingly complex, the engineering objective is total user invisibility: creators should experience instantaneous, fluid scrubbing on mobile hardware without ever having to manage the underlying decoding mechanics.

精炼阅读摘要

Meta工程师Boris Baracaldo指出，为在移动端实现流畅的多轨道4K视频时间线拖拽，核心在于从底层算法消除多余的解码工作，而非单纯提升运算速度；同时，面对极端复杂的工况组合，团队正转向AI原生架构，利用大模型构建确定性验证工具链，推动工程流程迈向“人在环上”的自动化闭环。移动端4K多轨拖拽性能挑战与GOP瓶颈 (1:22) 2025年Meta正式推出面向移动端专业创作的视频剪辑应用Edits，支持4K视频播放、多轨道音频与视频重叠、剪辑分割及特效。然而，在算力受限的智能手机上实现平滑的时间线拖拽（scrubbing）面临严峻的硬件制约。解码单帧4K画面的计算开销约为1080p的四倍，而多条4K轨道叠加时算力需求呈乘数级倍增，导致播放器在跟随手指滑动时产生明显的画面卡顿。Baracaldo强调，要弥合十倍以上的工作量鸿沟，绝不能仅靠微调既有算法，而是必须让底层引擎减少无谓的工作。这一性能瓶颈直接根植于现代视频容器的压缩结构。以MP4为代表的媒体文件依赖图像组（GOP）机制：仅关键帧（I帧）保存完整的静态图像，绝大部分差异帧（Delta帧）必须依赖前序关键帧按序解码。这意味着任意单次寻道（seek）操作的计算开销直接与GOP长度挂钩。当用户手指在屏幕上连续滑动时，系统会密集派发大量寻道请求；如果播放器机械地在每次请求时均跳转回前序关键帧重新串行解码，就会造成严重的重复计算与性能浪费。寻道引擎架构重构：GOP内解码与防闪烁 (6:14) 为消除冗余解码，团队重新梳理了单次寻道判断逻辑。引擎在接收到请求时会首先评估：目标时间戳是否与当前播放头处于同一GOP内。如果处于同一GOP且在当前位置之后，解码器即可完全绕过关键帧跳转，直接自当前帧向前连续解码至目标位置。针对Android原生MediaExtractor API隐藏底层关键帧分布表的技术限制，团队在媒体导入阶段通过单次跳帧扫描，在内存中构建了精确的关键帧时间戳（PTS）索引表，使实时寻道能够精准识别GOP归属。面对用户滑动的动态性，引擎将UI手势监听线程与负责密集计算的后台解码线程解耦，遇到新请求时果断中断旧任务并丢弃过时帧。针对与视频压缩格式逆向冲突的倒放拖拽（backward seek），团队将其建模为物理收敛过程：解码器必须自GOP关键帧向前推进并跨越手指当前位置后方可执行渲染，以避免因目标回跳导致的画面前后闪烁。在三条4K轨道叠加的严苛测试中，该优化使平均寻道时间、帧渲染速率及首帧呈现延迟均实现了10倍的性能提升，同时团队结合手机分辨率生成特定GOP的代理媒体（Proxy Media）以彻底优化预览体验。组合维度的复杂性爆炸与AI观测代理 (16:33) 尽管核心渲染管线大幅提速，多轨道剪辑引擎的持续维护依然面临组合爆炸的挑战。在实际剪辑中，不同分辨率轨道的层叠、实时滤镜特效链、拖拽手势的速度变化（缓拖重在对齐、快划重在轨迹），以及时间线的缩放级别（单帧精修对比十分钟全局预览），构成了极其庞大的多维变量空间。Baracaldo指出，单个工程团队已无法依靠人工测试穷尽所有组合工况，这推动了Meta将AI原生（AI-Native）理念深度引入软件生命周期。 Meta将工程研发抽象为观测、诊断、构建、实验与清理五个阶段。在观测阶段，全局汇总的均值遥测数据往往呈现虚假健康，从而掩盖特定切片下的严重衰退（例如特定拖拽速度叠加多层轨道时硬件解码器的冷启动延迟）。为此，团队为AI观测代理注入视频领域知识，使其能够自动对高维指标进行精细切片，自主生成诊断仪表盘并提出排查假设，将结构化诊断包直接传递给下游的自动化代码构建环节。确定性验证工具与“人在环上”协同范式 (23:12) 在利用AI编写移动端视频引擎代码时，团队确立了“意图与执行分离”的原则。鉴于大语言模型本身具有随机性，若直接依赖模型进行工程判断，容易产生不一致的实现路径。Meta的应对策略是让大模型专注于生成确定性的工具链与测试夹具：利用FFmpeg批量合成多分辨率与多GOP测试视频，使用脚本生成复杂的多轨合成工程，并通过Android的dumpapp与ADB驱动真实的手势交互。随后，由确定性脚本分析屏幕录制与系统Trace日志，严格验证是否存在闪烁或黑帧。这种机制为AI代理确立了以命令返回结果为准的二进制“完成标准”（Pass/Fail），模型通过自动化测试驱动持续修复代码直至变绿。这使得工程师的角色发生根本转变：从自动监控遥测、Git二分定位缺陷，到自动编写代码、部署A/B实验与清理过期特性分支，AI代理承担了完整的闭环执行。人类工程师从步步干预的“人在环中”（Human-in-the-loop）上升为负责架构决策与最终审批的“人在环上”（Human-on-the-loop）监督者。 Baracaldo总结认为，视频编辑引擎的终极目标是将复杂的解码优化与AI自动化体系完全隐蔽于幕后，为创作者提供无感而流畅的单指交互。不过该方案目前主要呈现了本地实验室及特定测试集的显著提效，对于Android低端设备长尾碎片化兼容、持续高负载下的整机发热降频，以及自主代码生成在生产大规模合流中的长期可维护性，仍需依赖全球海量设备数据的长期验证。

Detailed notes — zh

## Background 本内容由 Scale 频道于 2026 年 10 月 30 日发布在 YouTube 平台上，视频全长约 30 分钟。演讲由来自 Meta 的 Boris Baracaldo 主讲，聚焦探讨在专业移动视频剪辑应用 Edits 中打造高性能视频拖拽（Seek）引擎及向 AI-Native 研发范式演进的实践历程。 ## TL;DR Meta 工程师 Boris Baracaldo 深入剖析了移动端多轨 4K 视频编辑面临的算力瓶颈，重点介绍了针对时间轴滑动（Scrubbing）所做的 Seek 引擎底层优化。团队通过构建全局关键帧索引、及早中断陈旧请求以及向后滑动单调渲染等机制，克服了视频编码天然不利于逆向跳转的物理限制，实现了平均 Seek 耗时与渲染帧率整整 10 倍的性能突破。面对多轨道与复杂特效带来的高维组合爆炸，团队进一步引入 AI-Native 研发流程，利用 Agent 自动生成确定性工件、执行多模态测试用例与像素级验证闭环，推动人类工程师从‘人在环路中’全面跃迁至‘人在环上’把控全局架构。 ## Atmosphere & style 整场演讲呈现出典型硅谷顶尖工程团队的硬核、严谨与务实底色。Boris 的讲述节奏沉稳且富有逻辑，从底层物理容器与 GOP 原理切入，层层剥离技术痛点，生动还原了与视频编码格式‘正面硬刚’的调试过程。当话题从底层架构调优过渡到前沿的 AI Agent 研发闭环时，语调中洋溢着对工程范式革新的兴奋感，让观众切实感受到技术极客攻克复杂系统瓶颈时的智性愉悦与前瞻视野。 ## Bulletin - [00:01](https://www.youtube.com/watch?v=hAurRPc3A10&t=1s) Meta 推出专业移动视频编辑应用 Edits，面对多轨 4K 叠加下解码计算量膨胀超过 10 倍的极端性能挑战，确立了“减少不必要计算而非强行加速”的核心优化原则。 - [14:39](https://www.youtube.com/watch?v=hAurRPc3A10&t=879s) 团队通过在导入期预建 GOP 索引表、引入及早中断与向后滑动单调渲染控制，将拖拽平均耗时、渲染帧率与首帧延迟三项核心指标提升了整整 10 倍。 - [22:29](https://www.youtube.com/watch?v=hAurRPc3A10&t=1349s) 针对视频参数与交互场景的组合爆炸，团队构建了以可执行工件与像素级断言为核心的 AI Agent 工具链，有效规避了大语言模型的输出随机性问题。 - [27:39](https://www.youtube.com/watch?v=hAurRPc3A10&t=1659s) 工程团队正从“人在环中（in the loop）”演进为“人在环上（on the loop）”，由 Agent 自主承揽监控切片、代码二分排查、像素回归测试与灰度实验，人类仅负责战略设计与最终发布决策。 ## Key takeaways - [00:01](https://www.youtube.com/watch?v=hAurRPc3A10&t=1s) 移动内容创作从早期单片段编辑演进至 4K 多轨叠加的专业化阶段，计算量呈指数级暴增，促使工程重心转向削减冗余计算。 - [03:13](https://www.youtube.com/watch?v=hAurRPc3A10&t=193s) 单次 Seek 请求的计算成本与关键帧间距直接相关，朴素播放器盲目跳转关键帧会导致大量非渲染帧被重复线性解码。 - [06:14](https://www.youtube.com/watch?v=hAurRPc3A10&t=374s) 由于 Android 原生 MediaExtractor 缺乏全局拓扑接口，团队在视频导入期遍历关键帧预建 GOP 索引表以实现零成本跨组判断。 - [09:25](https://www.youtube.com/watch?v=hAurRPc3A10&t=565s) 连续轻扫手势会引发离散时间戳队列积压，双线程架构必须具备“及早中断”机制以叫停陈旧计算并动态重划解码路径。 - [11:18](https://www.youtube.com/watch?v=hAurRPc3A10&t=678s) 向后快进违背了现代视频格式前向预测的底层逻辑，系统需在解码跨越手指物理位移前抑制渲染以杜绝画面来回闪烁。 - [14:39](https://www.youtube.com/watch?v=hAurRPc3A10&t=879s) 拖拽交互的核心是连续手势下的感知流畅度，通过多项优化实现 10 倍性能跃升并引入代理媒体（Proxy Media）方案规避格式局限。 - [16:33](https://www.youtube.com/watch?v=hAurRPc3A10&t=993s) 多轨重叠、不同 GOP 大小及变速手势造成了维度的组合爆炸，传统人工排查无法穷举，催生了 AI 原生的全流程研发体系。 - [20:07](https://www.youtube.com/watch?v=hAurRPc3A10&t=1207s) 粗粒度全局平均数会掩盖真实卡顿，注入播放领域知识的 Agent 能够自动对生产数据进行多维切片并定位微观瓶颈。 - [22:29](https://www.youtube.com/watch?v=hAurRPc3A10&t=1349s) 应对大语言模型随机性的关键是让其生成确定性工件与边缘测试脚本，利用可执行指令而非主观判断来定义任务完成度。 - [24:40](https://www.youtube.com/watch?v=hAurRPc3A10&t=1480s) 结合录屏与 Trace 追踪的像素级自动化验证闭环，赋予了 Agent 自主生成改动并反复迭代修复缺陷的自愈能力。 - [26:33](https://www.youtube.com/watch?v=hAurRPc3A10&t=1593s) AI-Native 范式的核心经验在于：可执行地定义“完成”，并剥离确定性任务中的主观判断以最大化发挥大模型的逻辑优势。 - [27:39](https://www.youtube.com/watch?v=hAurRPc3A10&t=1659s) 研发协作范式正在转向“人在环上”，Agent 负责监控切片、定位排查、代码生成与灰度实验托管，人类仅做架构把关与决策。 ## Detailed walkthrough ### [00:01](https://www.youtube.com/watch?v=hAurRPc3A10&t=1s) 移动端视频编辑的演进与 Edits 引擎的性能挑战演讲者回顾了短视频与移动端内容创作的发展历程。2016 年 Instagram 推出 Stories（快拍）时，应用内的视频编辑功能相对简单，通常仅限于拼接几个片段、添加少量文字或套用滤镜，整个工作流局限在单一应用内部。到了 2020 年，短视频格式迎来爆炸式增长，Instagram 顺势推出 Reels，几乎所有用户每天都在分享和编辑视频。到 2022 年，主流创作者的需求已经超越了应用内置的简易编辑器，全面转向专业的独立编辑工具以产出更高质感的内容。为了应对这一趋势，Meta 于 2025 年正式宣布推出专业移动视频创作应用 Edits，并在四个月后上线，支持 4K 视频、音视频多轨道重叠叠加、拆分、裁剪以及丰富的特效库。 ```mermaid flowchart LR accTitle: 移动端视频创作工具的演进路线 accDescr: 展示从2016年Instagram Stories到2025年Meta Edits的发展历程及工作量瓶颈。 subgraph Timeline["演进历程"] A["2016: IG Stories (简单片段/文字/滤镜)"] --> B["2020: Reels 爆发 (短视频大众化普及)"] B --> C["2022: 创作专业化 (创作者转向专用工具)"] C --> D["2025: Meta Edits (4K多轨/专业级特效)"] end subgraph Bottleneck["核心性能瓶颈"] D --> E["4K 渲染 = 4倍于 1080p 工作量"] E --> F["多轨道重叠叠加 = 工作量指数倍增 (>10倍)"] F --> G["核心原则：减少不必要计算，而非强行加速"] end ``` 演讲者的核心使命是让创作者仅凭一根大拇指就能在手机上完成专业级视频剪辑，将艺术构想转化为能触达数十亿人的作品。作为支撑 Reels、Stories 和 Edits 视频播放器的 Android 工程师，演讲者过去一年专注于解决该播放器最关键的一项性能体验：时间轴拖拽（Scrubbing）时的即时画面渲染，即当用户手指滑过时间轴时，播放器必须高速且连续地呈现对应帧。在底层机制上，手指滑动触发的操作称为拖拽定位（Seeking），本质是解码并渲染对应帧的像素。用户手指划过的每个位置都会向播放器发起一次 Seek 请求，而移动设备的硬件算力极其受限。视频分辨率会成倍放大解码成本，单张 4K 帧的解码计算量大约是 1080p 帧的 4 倍；一旦叠加多个轨道，计算成本再次翻倍，设备需要承担的工作量会出现数量级的暴增。在单轨 1080p 和三轨 4K 重叠的对比测试中，单轨 1080p 能平滑跟手，而三轨 4K 则出现肉眼可见的明显卡顿脱节。面对超过 10 倍的计算差距，单纯调优算法是不够的，正如演讲者指出的核心原则：“The algorithm has to do less work, not the same work faster.”（算法必须做更少的工作，而不是更快地做同样的工作）。 - 2016 年 Instagram Stories 上线，仅支持基础剪辑、文字与简单滤镜。 - 2020 年 Reels 推出，短视频创作爆发。 - 2022 年专业创作者放弃简易内置编辑器，转向专用创作工具。 - 2025 年 Meta 推出专业移动视频编辑应用 Edits，4 个月后正式发布，支持 4K、多轨道重叠、拆分裁剪及全套特效。 - 4K 单帧解码计算量约为 1080p 的 4 倍；3 轨 4K 叠加会导致计算量膨胀超过 10 倍。 --- ### [03:13](https://www.youtube.com/watch?v=hAurRPc3A10&t=193s) 视频容器原理、GOP 结构与演讲架构规划为了找到削减计算量的方法，需要深入理解视频文件（如 MP4 容器）的物理存储与解码机制。MP4 容器主要由两大部分构成：第一部分是元数据（Metadata），记录视频的分辨率、时长、帧率以及关键帧分布映射表（Keyframe Map）；第二部分则是承载视频全部图像帧的原始数据。将这些数据还原为屏幕像素依赖于两个核心软件组件：解复用器（Demuxer）负责从文件中提取每帧的字节流信息，解码器（Decoder）负责将字节转换成可在屏幕上显示的像素。如果将视频抽象为胶卷长条，上面的帧主要分为包含完整图像信息的关键帧（Keyframe，通常为 I 帧），以及仅存储与相邻帧差异的预测帧。视频编码采用这种方式剥离冗余信息，目的是压缩文件体积；但其副作用是解码器无法从任意预测帧直接启动解码，必须从最近的前置关键帧开始，依赖前后帧信息逐步推导恢复出目标图像。因此，单次 Seek 请求的计算成本与关键帧之间的距离直接成正比，这段包含起始关键帧及后续预测帧的序列即为图像组（GOP，Group of Pictures）。演讲者概括了整场分享的四大核心板块： 1. **单次 Seek 请求的优化算法**：如何在单次定位中规避不必要的解码操作。 2. **连续流式 Seek 请求处理**：在用户持续滑动且只在乎即时感知、不在乎中间某一具体帧时，如何维持连续画面呈现与高感知性能。 3. **多维状态空间与 AI 技术引入**：解释该问题如何演化为多维复杂空间，以及团队如何利用 AI 技术驱动深度优化。 4. **视频基础设施的长期愿景**：探讨 AI 如何降低维护和演进复杂视频底层架构的人力成本，彻底重构视频编辑的 Seek 体验。 --- ### [06:14](https://www.youtube.com/watch?v=hAurRPc3A10&t=374s) 针对单次 Seek 的 GOP 感知与元数据预建优化针对单次 Seek 请求，传统简易播放器的实现极其低效。当播放器处于视频起始位置并收到一个 Seek 目标请求时，播放器必须先跳转到目标位置前最近的关键帧，然后从该关键帧开始按顺序线性解码每一帧，直到抵达目标帧并显示。在此期间解码的大量中间过渡帧并不会被渲染到屏幕上，却白白消耗了解码器算力。更严重的是，当播放器已经在某一位置解码，用户又发出一个新的 Seek 请求时，朴素播放器依然会盲目跳转回最近的关键帧重新开始线性解码，从而造成严重的重复解码与算力浪费。 ```mermaid flowchart TD accTitle: 单次 Seek 请求的 GOP 决策分支流程 accDescr: 展示播放器收到 Seek 请求时，如何利用预建列表判断是否在同一 GOP，从而避免不必要的跳转。 A["收到新的 Seek 请求 (目标时间戳)"] --> B{"当前播放位置与目标 是否在同一 GOP 内？"} B -- "否 (跨 GOP)" --> C["从关键帧列表定位 目标帧前最近的关键帧"] C --> D["跳转到目标关键帧"] D --> E["线性解码至目标帧并渲染"] B -- "是 (同一 GOP 且向前)" --> F["避免关键帧跳转！ 复用当前解码上下文"] F --> E ``` 解决这一问题的理论方案十分明确：在处理每个 Seek 请求时，播放器应先自问：“Am I in the same group of pictures as the new seek request?”（我当前是否与新的 Seek 请求处于同一个图像组内？）。如果处于同一 GOP 且目标在后方，播放器就可以完全省去关键帧跳转，直接从当前已解码位置继续向前线性解码。然而，在 Android 平台落实这一逻辑面临系统 API 的限制。Android 系统的 `MediaExtractor` 接口主要设计用于单向播放场景，其原生行为只支持定位到关键帧、单帧前移以及读取当前帧元数据，无法在不重新解码文件的情况下一次性读取全局关键帧拓扑结构。团队没有选择从零重写底层的 MP4 文件解析逻辑，而是巧妙利用系统的基础 API 进行构建。在视频首次导入（Import）阶段，播放器利用 `MediaExtractor` 的跳转能力在关键帧之间遍历一次，记录每个关键帧的时间戳，在内存中构建出该视频的完整关键帧分布列表。有了这份预构建列表，当任意 Seek 请求到达时，播放器就能以几乎为零的计算成本查询两个时间戳是否属于同一 GOP，从而精准决策是否跳过关键帧回溯。 - 朴素播放器在 Seek 时盲目回退到关键帧，导致大量未渲染帧被重复解码。 - 核心判定逻辑：检查当前帧与目标帧是否处于同一 GOP。 - 挑战：Android `MediaExtractor` API 缺乏直接导出全局关键帧列表的高效接口。 - 解决方案：在视频导入阶段进行一次单向关键帧遍历，采集时间戳并预先构建 GOP 索引表。 --- ### [09:25](https://www.youtube.com/watch?v=hAurRPc3A10&t=565s) 连续滑动流与双线程架构的调度瓶颈解决单次 Seek 优化仅仅是基础，因为在用户实际滑动时间轴时，系统面对的是动态变化的连续目标。正如演讲者所描述：“However, as the user is scrubbing, we're actually chasing a moving target.”（然而，当用户在滑动时间轴时，我们实际上是在追逐一个不断移动的目标）。在播放器全力解码某一个 Seek 请求时，用户手指可能已经划过了数个 GOP 区域，此时继续解码先前的请求毫无意义，甚至存在到达新目标更短、更快的解码路径。 ```mermaid flowchart TD accTitle: 播放器双线程架构与请求堆积瓶颈 accDescr: 展示 UI 线程接收离散手势请求与后台 Worker 线程阻塞式解码之间的异步堆积问题。 subgraph UI_Thread["UI 线程 (高频异步)"] A["检测到用户滑动/快速轻扫手势"] --> B["生成离散 Seek 流 (例如: 100ms -> 300ms -> 500ms)"] B --> C["请求快速压入队列"] end subgraph Worker_Thread["后台 Worker 线程 (耗时阻塞)"] D["解复用 (Demuxing) 与解码 (Decoding)"] E["正在全力处理旧的陈旧请求 (如 100ms)"] end C -. "请求积压 (Piling up)".-> E E --> F["产生严重画面滞后，无法即时跟手"] ``` 在 Android 底层，一个快速轻扫（Fling）手势会被系统分解为一系列离散的请求流（例如从 0 快速划到 1000ms 时，会连续发出到达 100ms、300ms、500ms 等一系列请求）。播放器底层由两个核心工作单元组成：一个是在 UI 线程上监听用户手势请求的观察者，另一个是在后台执行解复用与解码阻塞操作的工作线程（Worker）。由于解码高分辨率视频需要消耗大量时间，UI 线程源源不断送入新请求时，后台 Worker 往往仍停留在早先发出的陈旧请求（Stale Request）上，导致请求队列严重积压。用户在拖拽过程中的操作具有不可预测性，可能会中途加减速、突然反向滑回视频开头或快速甩到结尾。为了解决画面滞后问题，播放器必须具备高频中断（Interrupt often）的能力，迅速调优并响应用户当前的真实位置。为此，系统需要引入两大优化杠杆：其一是“及早中断”（Interrupting early），一旦检测到队列中出现新的 Seek 请求，后台必须立即中断当前的无效解码，重新评估新请求的位置与距离，进而规划最快速的响应路径。 - 快速轻扫手势会被操作系统转化为密集的离散时间戳请求流。 - 播放器采用双线程模型：UI 线程负责收集请求，后台 Worker 线程负责阻塞式的解复用与解码。 - 核心瓶颈：解码耗时远超手势产生速度，导致后台 Worker 陷入陈旧请求的无效运算中。 - 调优抓手：引入及早中断机制，一旦接收到新请求即刻叫停陈旧计算，重新动态规划解码路径。 ### [11:18](https://www.youtube.com/watch?v=hAurRPc3A10&t=678s) 持续渲染与向后快进（Seek）的格式对抗快进引擎的优化围绕两个核心机制展开：中断无效操作与高频渲染反馈。当手势方向改变时，系统无需继续向原方向线性解码；而只要获得展示进度的机会，解码器就会立即刷新屏幕像素。当新的快进请求到达时，系统首先中断当前工作并渲染现有进度，随后判断新请求是否与上一解码帧处于同一个图像组（Group of Pictures, 简称 GOP）内。如果在同一 GOP 内，解码器无需跳回关键帧；若不在，则必须跳回关键帧，并优先渲染该关键帧以在单位时间内提供最大幅度的视觉进度反馈，随后再线性顺序解码直至目标位置。 ```mermaid flowchart TD accTitle: 快进请求处理与向后滑动单调渲染流程 accDescr: 展示从接收快进请求、GOP判断、向后滑动防闪烁边界检测到最终帧渲染的决策逻辑 subgraph SeekHandling["快进请求决策"] A["收到快进请求"] --> B["中断当前解码并渲染当前进度"] B --> C{"是否处于同一 GOP？"} C -- "是" --> D["线性解码至目标帧"] C -- "否" --> E["跳至关键帧并优先渲染关键帧"] E --> D end subgraph BackwardSeek["向后快进防闪烁逻辑"] F["手指持续向左滑动"] --> G["解码方向向右推进"] G --> H{"解码帧是否与手指位置相交？"} H -- "未相交（仍在关键帧侧）" --> I["禁止渲染，继续向前解码"] H -- "相交并越过边界" --> J["单调渲染该帧，重置解码循环"] end D -. "若为向后手势".-> F ``` 向前快进完全契合现代视频格式的原生设计，但向后快进（即在时间线上向视频开头滑动）则是在与格式底层逻辑“对抗”。由于 H.264/H.265 等压缩格式依赖单向帧间预测，即便目标帧离当前帧极近，一旦逆向跨越，解码器也必须跳回该 GOP 的起始关键帧，逐帧向前线性解码。部分视频的 GOP 间隔长达 5 秒，需要解码的帧数极其庞大。解决该问题的关键在于将用户手势视为一个连续操作：当手指持续向左拖动时，手指的物理落点实际上在不断靠近该 GOP 的起始关键帧。解码器在关键帧处向前线性解码，而手指则逆向向它迎面滑动。如果解码中断时两者的位置尚未交汇（即解码出的帧比手指当前所在位置更接近关键帧），此时绝对不能触发屏幕渲染；否则会导致画面先退回后跳进，产生严重的画面闪烁（Flickering）。只有当解码进度真正穿过手指所在的时间边界后，系统才会渲染该帧，从而保证画面在单向滑动过程中保持单调连续呈现。 --- ### [14:39](https://www.youtube.com/watch?v=hAurRPc3A10&t=879s) 10 倍性能跃升与代理媒体（Proxy Media）方案在整合所有针对性优化后，多轨道高负载播放的体验迎来了质的飞跃。在包含 3 条 4K 分辨率重叠视频轨道的高复杂度合成场景下，播放器已经能够无缝紧贴用户手指的滑动轨迹，无论向前还是向后拖曳均极为流畅。本地基准测试结果显示，该套方案带来了整整一个数量级的性能突破。针对此阶段的工程成果，Meta 团队总结了核心指标与技术认知： - **核心指标提升**：在平均快进耗时（Average Seek Time）、帧渲染帧率（Frame Rendering Rate）以及首帧渲染延迟（Time to Render the First Frame）三项关键指标上，均实现了整整 10 倍（10x）的性能提升。 - **感知性能优先**：拖曳过程的重点不是孤立、精准且快速地呈现某一个单帧，而是保障连续手势交互下的感知流畅度。手指高速滑动时，在物理上不可能解码并显示沿途的每一帧，技术目标必须转变为动态筛选“单位时间内最值得渲染上屏的帧”。 - **视频格式局限性**：主流视频封装与编码专为“连续向前播放”设计，完全不适合高频、任意位置的跳转编辑。 - **代理媒体机制**：为了从根本上摆脱格式束缚，团队开始引入专门用于预览和快进的代理媒体（Proxy Media）技术。通过预先生成 GOP 尺寸极短、分辨率完全契合移动端屏幕物理像素的轻量级代理流，能够从底层大幅优化非线性编辑的预览性能。关于滑动体验与格式限制的核心矛盾，演讲者总结道： > “The first one, it's not about the exact seek request and showing one frame fast precisely. It's about the perceived performance doing a continuous gesture.” > > “The second takeaway is that the format is not built for this operation. It's built for continuous forward playback, not for arbitrary moving positions in the video.” --- ### [16:33](https://www.youtube.com/watch?v=hAurRPc3A10&t=993s) 应对组合爆炸：迈向 AI 原生（AI-Native）的研发流程随着 Meta Edits 引入多轨道堆叠、AI 特效等复杂创作能力，播放引擎每前进一步所面临的排列组合都会成倍膨胀。影响滑动性能的维度包括各轨道的视频分辨率、多条音视频轨道在时间轴上的重叠形态、不同媒体资产各异的 GOP 尺寸、层层叠加的实时视频滤镜与特效，以及手势滑动的方向（向前与向后成本高度不对称）、滑动速度（慢速微调要求单帧精度，快速扫读则追求大步长流畅反馈），甚至时间线尺度的缩放级别（在帧级放大下用户需要看清每一帧，而在 10 分钟缩略视图下用户对精准落点并不敏感）。面对如此庞大的参数空间，人类工程师不可能穷举并排查每一个维度的组合瓶颈。为了打破从问题观测到上线修复周期长、工程成本高昂的瓶颈，团队开始探索利用 AI 智能体工作流（Agentic Workflows）重塑软件工程全流程。 ```mermaid flowchart LR accTitle: AI 原生软件工程闭环工作流 accDescr: 展示从生产环境观测、瓶颈诊断、代码构建、实验防退化到技术债清理的闭环结构 subgraph EngineWorkflow["AI-Native 研发全流程"] O["观测 (Observing)"] --> D["诊断 (Diagnosing)"] D --> B["构建 (Building)"] B --> E["实验 (Experimenting)"] E --> C["代码清理 (Cleaning)"] end C -. "闭环迭代".-> O ``` AI 原生开发体系覆盖了五个高层步骤： 1. **观测（Observing）**：在生产环境中持续监控海量多维组合下的实际运行状态。 2. **诊断（Diagnosing）**：准确定位特定指标在细分场景下的性能瓶颈或退化原因。 3. **构建（Building）**：自动或辅助编写底层优化逻辑与缺陷修复代码。 4. **实验（Experimenting）**：面向全球数十亿使用不同硬件规格设备的用户进行严格 A/B 测试，防止某一设备群体的优化以牺牲另一群体体验为代价。 5. **代码清理（Cleaning）**：在密集实验后及时剔除分支废弃代码，防止技术债务无限堆积。所有智能体协同的基础，在于构建一套专注于自身播放引擎特性的“共享上下文理解”（Shared Context Understanding）。该上下文深入底层领域细节，明确记录了正向线性解码开销、逆向跳转对齐逻辑等架构边界，让不同的智能体能够在诊断、观测与构建环节无缝协作。 --- ### [20:07](https://www.youtube.com/watch?v=hAurRPc3A10&t=1207s) 智能体驱动的精细化指标观测与特性构建在多维场景的性能观测中，粗粒度的全局统计往往会掩盖真实缺陷。用户在时间线上拖曳时，每秒钟都会产生形态各异的交互手势。如果仅仅把所有手势、所有设备型号以及所有用户的数据汇总取平均值，得出的平均耗时曲线平缓，根本无法暴露性能瓶颈。真正的性能剧痛点，高度集中在特定参数组合的夹缝中——例如某种特定滑动速度恰好叠加了多轨道交叠切换，或者切换视频轨瞬间硬件解码器冷启动（Warming up a new decoder）带来的瞬时停顿。单一工程师的精力无法巡检整个高维数据空间，而注入了视频播放领域知识的智能体则能够规模化地破解这一难题。智能体学习了交互手势的几何形态、复杂视频轨道的排列组合规律以及不同交互阶段的时序特征后，可以自动对生产数据进行多维切片，动态生成聚焦单点瓶颈的下钻看板（Dashboards），其提问与排查思路完全对齐资深性能工程师的分析模式。一旦切片监控探测器捕捉到指标异常波动或瓶颈浮现，智能体便会自动将其打包为结构化上下文，直接作为后续构建与修复环节的输入材料。在具体特性的代码构建环节（特别是在 Android 移动端设备上运行的高性能视频引擎），AI 原生模式确立了明确的人机分工： - **人类工程师的角色**：负责提供核心意图（Intent）、核心算法设计、顶层架构规划以及系统设计思维。 - **智能体的角色**：负责在给定的系统设计约束与领域上下文下，理解视频处理机制，编写具体模块实现并支撑后续的测试闭环。 ### [22:29](https://www.youtube.com/watch?v=hAurRPc3A10&t=1349s) 构建确定性工件与工具链生态将高层意图转化为 AI Agent 可落地的任务时，工程师必须站在测试的角度逆向推导：为了确保视频播放完全符合预期、杜绝闪烁和黑帧，Agent 究竟需要执行哪些测试用例？要让 Agent 具备确信度，系统必须从模糊的意图生成确定性的交付工件。这涵盖三层结构：定义功能外貌的产品规格说明（Spec）、指导 Agent 逐步完成构建的精确实现计划（Implementation Plan），以及至关重要的可执行代码与命令行指令。Agent 能够直接运行这些指令，从而在每次尝试后获得极为清晰、无歧义的判定，明确知晓任务是已完成还是未完成。 ```mermaid flowchart TD accTitle: Agent 确定性软件构建工作流 accDescr: 展示从人类意图到确定性工件生成，再到驱动工具与像素验证的闭环流程。 subgraph Artifacts["1. 确定性工件"] Spec["功能规格说明 (Spec)"] Plan["实现步骤规划 (Plan)"] Commands["可执行验证指令 (Commands)"] end subgraph Tooling["2. 自动化生成与驱动"] MediaGen["多模态媒体生成 (FFmpeg)"] Composition["复杂轨道组合脚本"] Driver["设备驱动 (ADB / dumpapp)"] end subgraph Verification["3. 闭环验证与迭代"] TracePixel["像素级黑帧/延迟检测"] AgentLoop{"验证通过 (Green)?"} Fix["生成新代码并重试"] end Artifacts --> Tooling MediaGen --> Composition --> Driver Driver --> TracePixel --> AgentLoop AgentLoop --"否 (Red)"--> Fix --> Driver AgentLoop --"是"--> Done["合并就绪 (Done)"] ``` 在软件开发中引入 AI 时，最关键的陷阱在于过度依赖大语言模型（LLM）的自主决策。Boris Baracaldo 强调指出：“One of the best uses of agents is not necessarily throwing at every single problem you have and then hoping that it knows how to traverse every single decision that it needs to make because LLMs are stoastic and it will provide different versions of the same answer.”（Agent 最佳的使用方式，并不是一股脑把遇到的所有问题全甩给它，然后指望它能精准跨越必须要做出的每一个决定，因为大语言模型本身具有随机性，面对同一个问题会给出不同版本的答案。）因此，工程团队的策略是借助 Agent 去低成本编写和生成各类确定性的脚本与工具，把以往单凭人类工程师耗费大量时间才能编写的边缘工具彻底自动化。具体到视频seek引擎与渲染管线，Agent 能够快速构建出极度丰富的测试用例库： - **基准测试素材生成**：利用 FFmpeg 等命令行工具，批量生成涵盖不同分辨率、帧率、GOP 大小（GOP sizes）的视频库，确保覆盖各种编码边缘情况。 - **复合剪辑场景合成**：针对重叠轨道合成、复杂转场特效叠加、动态 GIF 贴图与文本混合等多样化排版，自动生成针对性的工程脚本。 - **界面行为驱动**：针对 Android 设备端集成 dumpapp 插件检查屏幕控件树，并通过 ADB 脚本精准回放滑动、点击等组合手势。 ### [24:40](https://www.youtube.com/watch?v=hAurRPc3A10&t=1480s) 端到端自动化驱动与像素级验证闭环为了真正把“判断视频是否播放正确”从主观视觉感受变成确定性事实，自动化套件在 Agent 驱动设备执行测试时，会同步录制屏幕内容并捕获应用内部生成的 Trace 日志。此时，验证器本身也被封装为确定性的脚本工具。这些验证工具直接读取录屏文件中的每一帧像素与时间戳追踪，将实际呈现结果与手势动作做帧级别的严格对照，从而精准回答一系列关键性能指标：滑动过程中是否出现了偶发黑帧？是否存在掉帧卡顿与画面闪烁？画面内容是否严格贴合手指位移，还是从手势开始到第一帧呈现之间产生了过长的延迟？ Boris Baracaldo 在现场展示了一段演示视频：屏幕上一个完全自主的 Agent 正在自动化驱动剪辑应用，它快速执行点击、拖拽和轨道缩放，同时毫秒级观察屏幕像素变化以判定内容正确性与响应速度，并在控制台中输出醒目的“绿标（Pass）”或“红标（Fail）”。一旦断言失败，Agent 绝不会停滞不前，而是迅速开启自主循环；它极度擅长一口气生成成百上千种不同的代码改动方案，反复在端侧运行可执行指令，直到攻克这些严苛的测试用例并满足像素断言。 ### [26:33](https://www.youtube.com/watch?v=hAurRPc3A10&t=1593s) AI-Native 研发的两大核心启示基于在视频编辑引擎构建中的实战经验，Boris Baracaldo 提炼出了向 AI 原生开发范式演进的两条核心方法论：首先，必须把“完成”以可执行的方式明确定义出来。Boris Baracaldo 明确总结道：“First, it makes a whole lot of difference to define executably what done means.” 团队需要的不是文档或自然语言中模棱两可的需求描述，而是一个 Agent 可以直接在终端执行并返回确定布尔值（Yes 或 No）的指令。只有建立这种确定性的反馈桩，Agent 才能具备自主修复和迭代的抓手。其次，要充分发挥 Agent 编写确定性工具的长处，收窄模型的判断范围。Boris Baracaldo 强调：“By removing the judgment from things that can be done deterministically, you're actually using LLMs what they're best and that is covering those gaps of judgment that are still remaining.”（通过把所有能够确定性完成的事情剥离掉主观判断，你才能让大模型各尽所能，真正去弥补那些依然需要灵活判断的空白。）当繁重复杂的断言、像素对比、数据驱动工作被工具固化下来后，AI Agent 的推理能力才不会被消耗在不可控的随机漂移中。 ### [27:39](https://www.youtube.com/watch?v=hAurRPc3A10&t=1659s) 从“人在环路中”迈向“人在环上”的未来愿景随着各模块工具链的完善，研发闭环的连接方式正迎来根本性变革。在下一代研发生态中，Agent 不仅能写代码，还能长期驻留在生产监控链条中：自主对生产环境的海量监控数据进行多维度切片分析，锁定性能劣化与卡顿集中的瓶颈；另一组 Agent 随即接收这些数据输入，自主针对代码库执行 git bisect（二分排查版本提交）、对比 Diff，甚至复现某一真实用户在特定机型上的复杂剪辑合成场景。 ```mermaid flowchart LR accTitle: 从人在环路中到人在环上的宏观闭环 accDescr: 展示 Agent 自主处理监控、诊断、编码、灰度监控，人类仅负责战略决策的协作模式。 subgraph AgentLoop["Agent 自主执行闭环"] Monitor["生产监控与多维切片"] Bisect["代码二分排查与现场复现"] Build["多方案代码生成与像素测试"] Ops["A/B 实验配置与指标监控"] end subgraph HumanSupervision["人类工程师角色 (人在环上)"] DesignDecision{"评审问题价值与方案设计"} ShipDecision{"基于全量指标决策是否发布"} end Monitor --> Bisect Bisect --> DesignDecision DesignDecision --> Build Build --> Ops Ops --> ShipDecision ``` 在整个软件生命周期中，人类工程师的角色正在发生质的跃迁。Boris Baracaldo 指出：“Essentially, we're taking humans from being in the loop to on the loop.”（从本质上讲，我们正在把人类从‘在环路中工作’转变为‘在环路之上监督’。） - **问题甄别与方案设计**：Agent 负责收集现场证据并打包排查报告，人类工程师只需在顶层判断哪些性能问题值得被解决，并确认架构设计方向。 - **实验与上线全托管**：构建完毕后，Agent 会准备好代码审查包供人类批准，随后接管全套运维流水线——创建并配置 A/B 测试实验、阶梯放量、自动跟踪观察数天的大盘指标，并最终生成完整的读数报告，由人类做出最终的“全量发布（Ship）”决定。 - **系统复杂度清理**：发布之后，代码清理、死代码下线以及日常的复杂度维护，再度交回给 Agent 全自动收尾。回顾视频编辑技术的发展，剪辑工作曾严重依赖昂贵的工作站和桌面专业硬件，如今创作者仅凭一部智能手机、动动大拇指就能随手完成。与此类似，视频播放与 seek 引擎底座始终是一个不断变化的“移动目标”，以往需要人类工程师一步一脚印地将每个优化点从观察带向部署。而在不久的将来，Agent 将贯穿整个流程。在演讲的最后，Boris Baracaldo 总结了工程团队的终极追求：无论底层架构和研发方式多么复杂，用户根本不需要在乎背后的技术，“And in the end, our users don't care about how complex is the technology that they're using or even how we're building it.”——用户需要的体验极其纯粹，只需将手指按在屏幕上，一切就能如丝般顺滑、自然呈现。 ## Notable quotes > [00:01](https://www.youtube.com/watch?v=hAurRPc3A10&t=1s) The algorithm has to do less work, not the same work faster. — Boris Baracaldo > [27:39](https://www.youtube.com/watch?v=hAurRPc3A10&t=1659s) Essentially, we're taking humans from being in the loop to on the loop. — Boris Baracaldo > [26:33](https://www.youtube.com/watch?v=hAurRPc3A10&t=1593s) By removing the judgment from things that can be done deterministically, you're actually using LLMs what they're best and that is covering those gaps of judgment that are still remaining. — Boris Baracaldo > [14:39](https://www.youtube.com/watch?v=hAurRPc3A10&t=879s) It's not about the exact seek request and showing one frame fast precisely. It's about the perceived performance doing a continuous gesture. — Boris Baracaldo > [26:33](https://www.youtube.com/watch?v=hAurRPc3A10&t=1593s) First, it makes a whole lot of difference to define executably what done means. — Boris Baracaldo > [27:39](https://www.youtube.com/watch?v=hAurRPc3A10&t=1659s) And in the end, our users don't care about how complex is the technology that they're using or even how we're building it. — Boris Baracaldo ## Mentioned Boris Baracaldo, Meta, Edits, Instagram Stories, Instagram Reels, Android, MediaExtractor, FFmpeg, dumpapp, ADB [Source](https://www.youtube.com/watch?v=hAurRPc3A10)

Source quotes

[0:01] Boris Baracaldo: The algorithm has to do less work, not the same work faster. [27:39] Boris Baracaldo: Essentially, we're taking humans from being in the loop to on the loop. [26:33] Boris Baracaldo: By removing the judgment from things that can be done deterministically, you're actually using LLMs what they're best and that is covering those gaps of judgment that are still remaining. [14:39] Boris Baracaldo: It's not about the exact seek request and showing one frame fast precisely. It's about the perceived performance doing a continuous gesture. [26:33] Boris Baracaldo: First, it makes a whole lot of difference to define executably what done means. [27:39] Boris Baracaldo: And in the end, our users don't care about how complex is the technology that they're using or even how we're building it.

Source quotes

[1:22] Boris Baracaldo: It's not a matter of tuning the algorithm. The algorithm has to do less work, not the same work faster. [27:22] Boris Baracaldo: Essentially, we're taking humans from being in the loop to on the loop.

Original transcript

[0:01] Hey everyone, [0:03] let's go back to 2016. Stories had just [0:06] released in Instagram and editing a [0:09] video in the app meant maybe adding a [0:11] few clips, some text, maybe a filter and [0:14] the whole flow happened within the same [0:16] app. Then 2020 came along and it was an [0:19] interesting year. But for short video [0:22] format, it was huge. The format [0:24] exploded, reals launched, and nearly [0:27] everyone was sharing and editing video [0:30] almost every single day. By 2022, most [0:34] serious creators had outgrown the inner [0:37] editors and have really transitioned to [0:40] dedicated tools for creating their [0:42] polished content. [0:44] In 2025, Meta entered the space and [0:47] announced Edits, an app for professional [0:49] video creation in your mobile hardware. [0:53] Four months later, edits released with [0:55] support for 4K video, overlapping video [0:58] and audio tracks, splitting, trimming, [1:00] and a whole inventory of effects. [1:04] Making professional video editing simple [1:06] enough for a creator to do with a thumb [1:08] on their mobile phones is exactly what [1:11] Edits is for. And our mission is to [1:14] create tools that allow creators to [1:17] transform their artistic vision into [1:19] polished content that can reach billions [1:21] of people. [1:22] And I've seen this firsthand as an [1:24] Android engineer behind the player that [1:26] supports reals stories and edits. I've [1:30] been over the last year focusing on the [1:32] performance of one action in this [1:35] player, which is scrubbing over a [1:38] timeline and being able to show frames [1:41] fast and continuously as the user scrubs [1:44] over the timeline. [1:46] Let me show you what that looks like. [1:48] Dragging your finger along a timeline is [1:50] something everyone has done and it's an [1:53] action that simply works. But a lot of [1:56] work sits behind getting the player to [1:58] follow your finger across the [2:00] composition precisely as you move the [2:02] finger. As you scrubble your finger, [2:04] what the player does under the hood is [2:06] called seeking. It means decoding and [2:09] rendering the frame so that the pixels [2:11] have be can be rendered on the mobile [2:13] screen. So every position that your [2:15] finger crosses over becomes a seek [2:18] request that the player executes. [2:21] So each of those seek request has a cost [2:24] and model hardware is limited. [2:27] Resolution multiplies the cost of [2:28] decoding every single frame. So a 4K [2:31] frame is about four times the work of [2:33] decoding a 1080p one. And if you add [2:36] overlapping tracks that cost multiplies [2:38] again and you can get a whole magnitude [2:40] of work more that needs to happen on the [2:43] device. So let's observe these side by [2:47] side. On the left you have a single [2:48] 1080p track that keeps up as you scrap [2:51] and on the right you have three [2:53] overlapping 4K tracks that you can start [2:56] seeing how the player visibly lacks [2:58] trying to follow the finger. Closing a [3:00] gap of more than 10 times the work. It's [3:03] not a matter of tuning the algorithm. [3:06] The algorithm has to do less work, not [3:10] the same work faster. [3:13] So to see where we can do less work, [3:16] let's look at how a video is represented [3:18] in a file in a container such as a MP4. [3:22] There's actually two parts to it. One is [3:24] the metadata that tells you things about [3:26] the video like resolution, duration, [3:29] frame rate, and more importantly for us, [3:32] a map of where every key frame sits in [3:35] the video. And we'll get back to this [3:36] map later. The second part of the [3:38] container holds the data for every [3:40] single frame that will be shown in the [3:42] video. And there's two software [3:44] components that turn that data into [3:46] pixels on your screen. First the demoxer [3:50] looks into the file and extracts the [3:52] bite information of every frame and then [3:55] the decoder turns those bytes into [3:57] pixels that can be rendered on a script. [4:01] Now let's imagine the video as a film [4:03] strip where you have a bunch of [4:05] different frames that represent each [4:06] picture that the video displays. [4:09] We have different types of frames that [4:11] can be displayed and the more important [4:13] one of these ones is the key frame which [4:15] has the complete representation of a [4:17] picture that can be rendered. Most every [4:20] other frame doesn't have that. It has a [4:23] partial representation of the picture [4:26] and this is done so the MP4 file has [4:31] less storage size on your computer. So [4:34] by stripping away content from certain [4:36] frames, the video file is smaller. But [4:39] that means that the decoder can only [4:41] start from a key frame where it has the [4:43] full information that it needs. Every [4:45] other frame will need information from [4:48] neighboring locations in order to decode [4:50] the same data. [4:52] What that means for us is that the cost [4:54] of one S request scales with the size of [5:00] the distance between key frames. [5:03] This can also be called as a group of [5:04] pictures which is essentially the [5:07] collection of frames between key frames. [5:11] So during this talk I'm going to cover [5:14] four parts. First we're going to focus [5:17] on the algorithm that we've implemented [5:19] to improve the performance of siki. [5:22] We'll look at one single request first [5:24] and how we do avoid less work on each [5:28] one of those. Then we move on to a [5:31] stream of requests and how we continue [5:33] showing frames and improving the [5:35] perceived performance when the user [5:38] doesn't care about the particular frame [5:39] and just continues moving his finger. [5:43] And then I'm going to explain why this [5:45] problem space can explode into a [5:47] multiple dimensions and how we're [5:49] integrating AI technology to help us [5:51] drive further optimizations. [5:53] I'll finish the talk by providing my [5:55] view into how AI will continue driving [5:58] down the human cost of maintaining and [6:01] improving complex video software and [6:04] hopefully by the end you'll gain a [6:05] better view at how we are transforming [6:08] the sik experience for video editing. [6:11] Let's start with number one. [6:14] So let's start with a single s request [6:17] and let's go back to our film strip [6:20] representation of a video. In this [6:22] diagram, the player is positioned at a [6:24] frame towards the beginning of the track [6:26] and we have just received a secret [6:28] request. Remember I said that we must [6:31] always start from a key frame so we can [6:33] start decoding the VV and the MP4 file [6:37] knows where those key frames are [6:39] positioned. So the very first thing we [6:41] need to do is jump to the key frame that [6:44] is closest to the target right before [6:46] it. Once we get to a key frame, we can [6:49] start decoding linearly every single [6:51] frame till we reach the target and show [6:54] that target completely in the screen of [6:57] your phone. [6:59] Now we've been decoding a lot of frames [7:00] between the key frame and the target and [7:03] most of them will not be rendered but we [7:06] need to decoding nonetheless in order to [7:08] get the full frame representation. [7:12] Now because the format is built for [7:15] decoding forward we started looking at [7:17] this problem. If we start from a certain [7:20] position and we get a new SE request in [7:22] the naive player implementation, you [7:24] would just blindly jump to key frame and [7:26] then go to the target linearly. And you [7:30] can see how this can waste a lot of work [7:32] because we are really coding a lot of [7:34] frames that we don't need to. [7:37] So what's the solution? We need to ask [7:39] ourselves on every S request, am I in [7:42] the same group of pictures as the new S [7:44] request? If we are then we can avoid the [7:47] key frame jump and then simply go [7:49] straight to linearly decoding the [7:51] frames. And this sounds obvious enough [7:54] but with the primitives exposed by the [7:57] Android OS and the media extractor APIs [8:01] this is not so simple because we cannot [8:03] observe the full structure of the key [8:05] frames without red coding those that [8:09] information from the file itself. [8:13] If we look at the API from Android's [8:15] media extraction capabilities, it's [8:18] mainly built so you can stick to a key [8:20] frame, advance one frame, and then get [8:23] metadata from whatever frame you are in. [8:26] And this is the right decision for a [8:29] simple player that is reproducing or [8:31] playing forward. But in our case, when [8:34] we're trying to really drive the player [8:36] towards its limits, it's not good enough [8:39] because we need to observe the structure [8:41] of the full file ahead of time. So [8:44] instead of reimplementing how do we get [8:46] that information from the file itself, [8:48] we're using these primitives as building [8:50] blocks to construct that key frame list [8:53] ourselves. So once per video the moment [8:57] of import we traverse the video one key [9:00] frame at a time because the extractor [9:02] does provide that possibility and every [9:04] time we jump we capture the time of that [9:06] key frame and we build our list once we [9:09] have that list whenever a s request [9:11] comes in we can very cheaply ask are [9:15] these two time stamps in the same group [9:16] of pictures or not and then we can make [9:18] that decision to avoid jumping to a key [9:21] frame or not. [9:25] So far, everything we've been looking at [9:28] is about making a single SIG request [9:30] faster. [9:31] However, as the user is scrubbing, we're [9:34] actually chasing a moving target. And [9:37] maybe while the player is working [9:38] towards a particular SI request, the [9:41] finger of the user is already past maybe [9:44] several group of pictures and there [9:46] might be a better path to reaching that [9:48] point even faster. [9:51] So let's observe at how the player today [9:53] handles s requests as the user swipes. [9:58] When a gesture is detected in the [10:00] Android software, what we get is a [10:02] series of discrete requests. [10:06] If the fling was from 0 to a,000, then [10:09] we'll get several requests such as go to [10:11] 100, then 300, then 500, and so on. And [10:14] the player is built in a way that it has [10:17] two threads, two workers. one of them [10:19] observing the request from the UI and [10:21] the other one doing blocking work to [10:23] actually move the player towards the [10:25] position. So that demaxing and decoding [10:28] that we've been talking about. However, [10:31] this is the operation that takes time [10:33] and many requests are coming in the UI [10:36] thread and we might be working towards a [10:38] stale request while those requests are [10:41] piling up and we need to start thinking [10:43] about how can we interrupt often this [10:47] process so we can start adjusting and [10:49] react faster to the needs of the user [10:53] because as they're scrubbing maybe they [10:55] change speed or they go back to the [10:58] beginning of the video where they fling [10:59] to the [11:00] And that is not something we can [11:01] predict. [11:03] So there are two levers that we can [11:06] really use for optimizing the speed [11:09] here. One of them is interrupting early. [11:11] So as soon as a new request comes in, we [11:14] stop and we take stop of where that new [11:16] request is at. So we can adjust our [11:18] work. If the seat direction change, then [11:21] we don't need to keep the coding forward [11:23] and so on. The second lever is rendering [11:26] often. So every time we get a chance to [11:28] show progress, we do so by rendering and [11:32] refreshing the pixels in the screen. So [11:34] just to go step by step, we start at [11:37] top. When a secret request arrives, we [11:39] interrupt, we render to show the [11:41] progress. And then we ask, is the new [11:44] request in the same group of pictures as [11:46] the last frame we decoded? If it is, [11:49] then we don't jump to the key frame. If [11:51] not, then there's no avoiding that. But [11:53] if we do jump, we do want to show [11:56] progress. So we render the key frame [11:58] because it's the most progress that we [12:00] can that we can show per unit of time. [12:02] After that, we keep decoding linearly [12:04] until we reach the target. And if [12:06] there's no new requests, then we only [12:09] render the moment we reach the final [12:11] position precisely. [12:16] Now, this is great for forward seeking. [12:20] That's what the format is built for. [12:22] However, if you want to seek backwards [12:24] and you want to start scrubbing towards [12:26] the start of the video, then we start [12:29] fighting the format because we need to [12:32] jump to the key frame. There's no [12:33] avoiding that. Even if the position that [12:36] we're going towards is very close to the [12:38] position we're in, we go back and then [12:41] we decode every single frame from the [12:43] beginning of that group of pictures, [12:44] which can be a lot. Some videos have [12:46] group of pictures every five seconds. [12:49] That's a lot of frames. [12:51] So what can we do? [12:54] You need to think about the gesture as a [12:57] continuing operation. So as the finger [13:00] move backwards, they're actually [13:02] starting to move closer to that key [13:04] frame that we have to jump towards. So [13:07] when the first request comes, we jump to [13:09] the key frame. There's no avoiding that. [13:12] And the finger will keep moving. So at [13:14] some point we will get interrupted with [13:16] a new seek request. And the first thing [13:19] to think about is that our target is no [13:21] longer the first request. The mover the [13:24] finger has moved on for that. What we [13:26] need to move towards is that second [13:29] finger that second S request and where [13:31] the finger is now which is actually [13:34] closer to the key frame that it was [13:35] before. So you can think that the finger [13:38] is moving in one direction and then the [13:41] decoding is moving in opposite direction [13:43] and at some point they will meet. If at [13:46] interruption time they have not met that [13:48] means the decoded frame is still closer [13:51] to the key frame than to where the [13:53] finger is. We cannot render because at [13:56] that point we will go beyond the user [13:58] intent and we will incur in flickery [14:01] which is showing one frame back and then [14:04] showing one frame forward when we [14:06] actually reach the finger position and [14:09] that is a bad experience. So we ask [14:12] ourselves if we haven't met then we [14:15] continue sitting forward or decoding [14:17] forward and at some point that boundary [14:21] is going to cross and from that point [14:23] onward we can render that frame and [14:24] start over with the key frame jump and [14:27] linear decoding. This way we ensure that [14:30] we are rendering monotonically the [14:32] frames in one direction without [14:34] incurring any flickering. [14:39] Now we put all of these optimizations [14:41] together and this is how it looks like [14:44] today. Same composition, three [14:47] overlapping tracks, 4K resolution and [14:49] you can see how the player starts [14:52] following your finger precisely be it [14:55] forward or backward. [14:58] We measure this locally with that type [15:00] of compositions as you observed and we [15:04] managed to achieve this order of [15:05] magnitude of improvement. So 10 times [15:08] improvements across average seek time, [15:11] frame rendering rate and time to render [15:13] the first frame. And there's two [15:16] takeaways that I want to give you from [15:18] our work doing these optimizations. The [15:22] first one, it's not about the exact SK [15:25] request and showing one frame fast [15:28] precisely. It's about the perceived [15:30] performance doing a continuous gesture. [15:32] So it's physically impossible for us to [15:34] show every single frame when the user is [15:37] scrubbing. So the target for us is [15:41] choosing exactly which frames are [15:44] actually worth the most to render into [15:46] the screen. [15:49] The second takeaway is that the format [15:50] is not built for this operation. It's [15:53] built for continuous forward playback, [15:58] not for arbitrary moving positions in [16:00] the video. So we've been working around [16:03] the format but this opens up the [16:05] question of what if we create our own [16:07] format for previewing and seeking [16:09] specifically this is a technique that [16:12] we're implementing and I'm not going to [16:14] go into a lot of details about this but [16:17] by creating a proxy media that can have [16:20] a better go size and maybe better [16:24] resolution for what the phone screen can [16:27] show. we can actually optimize the [16:29] preview experience better. [16:33] So, we shipped all of this and it works. [16:37] Your player is much smoother. But we're [16:40] not done. A player like this [16:43] continuously needs improvements, [16:44] especially if you keep shipping more [16:47] complex features. Do you want to support [16:49] more complex media and compositions, [16:52] AI effects, and so on? So each [16:55] optimization or improvement costs a lot [16:58] to ship from observation to release. So [17:00] what we're asking ourselves today is how [17:03] can we leverage agentic workflows to [17:05] actually scale the development process [17:08] of every new optimization that we want [17:10] to deliver. [17:12] So to understand the complexity, [17:15] let's look at all the combinations that [17:17] come into play when you're scrubbing [17:19] over a video. You have the resolution of [17:22] every video track that you are scrubbing [17:24] over. You might have several tracks [17:26] overlapping both video and audio with [17:29] different g size. You can have a lot of [17:31] stacked effects one after the other. The [17:33] direction plays a factor because forward [17:36] and backward are asymmetric in cost. you [17:38] have the speed of the gesture which [17:40] changes the expectation of the user [17:41] because if you're scrubbing slowly you [17:44] may care more about position but if [17:46] you're scrubbing fast then you might not [17:48] care about what exact frames you're [17:50] showing and finally the scale of the [17:52] timeline which we allow inside of edits [17:55] also changes what you expect out of the [17:57] experience because if you seem if you [17:58] zoom in all the way in and edits [18:01] actually give support to frame level [18:04] zoom in the timeline then you actually [18:06] want to see every frame but But if you [18:08] are fully zoomed out and you can see [18:10] maybe 10 minutes of content within the [18:13] width of your screen, then you might not [18:16] care about what exact frame lands on and [18:18] what is displayed on the on the screen. [18:22] So from this complexity, we cannot [18:26] realistically explore every combination. [18:29] So what does it take for us to go AI [18:31] native on the software building process? [18:34] There's a few boxes that we can [18:36] generalize the process to in high level [18:39] terms which I'm showing up here that are [18:42] observing actually monitoring what's [18:44] going on in production across these [18:47] dimensions and combinations diagnosing a [18:50] particular bottleneck or regression in [18:52] one metric building an optimization or [18:55] fixing something experimenting so you [18:57] don't regress the experience for one [19:00] group of user but as you're trying to [19:03] improve the experience for another [19:04] group. And this is important when you [19:07] are trying to serve billions of users [19:09] with all the different devices that they [19:11] might use. And finally, cleaning up the [19:14] code so you don't inherit like a huge [19:16] tech depth as you are experimenting all [19:19] these different variations. One layer [19:21] down, we're building one shared context [19:24] understanding so agents can start [19:26] working on each of these boxes [19:28] independently, diagnosing, observing, [19:30] building, experimenting, and then [19:32] managing the co the code complexity. [19:36] This shared context understanding [19:39] needs to focus on what makes our [19:41] software unique. So gachas about what [19:44] does it mean to decode forward, why [19:48] seeking backward is more complex and [19:50] things like that. In this process, I'm [19:52] going to drill down into two boxes [19:54] specifically because we don't have time [19:56] to go over every single one that I'm [19:58] going to focus on observing all these [20:00] dimensions I touched on and building one [20:04] particular feature using agentic flows. [20:07] Let's start with observation. So [20:09] remember all those combinations, this is [20:11] why they matter. When a user is [20:13] scrubbing on a composition, there's a [20:15] lot of gestures that happen maybe every [20:18] second and every gesture has a unique [20:20] shape and we just if we just average [20:23] metrics across every single gesture, [20:25] across every device and across every [20:27] user, then you'll get an average number [20:29] that may not tell you much about what [20:31] needs improved. When you slice those [20:34] combination of dimensions, then you [20:36] realize that the pain concentrates in [20:39] very specific combinations like maybe [20:41] one speed over one particular number of [20:45] overlapping tracks or if you change from [20:48] one track to the next and there is a [20:50] certain cost to warming up a new [20:52] decoder. All those type of things can [20:54] only surface if you drill down into [20:57] specific views of all these data that [21:00] gets produced in production. [21:02] So it's very hard for one human to [21:04] actually observe and explore the whole [21:07] problem space. [21:09] So agents can help you scale. If we [21:13] build agents and we teach them about [21:15] specific domain knowledge that tells [21:18] them how gestures look like different [21:20] composition shapes, the faces of an [21:23] interaction and you start creating [21:24] agents that can slice views and create [21:28] dashboards that highlight bottlenecks in [21:31] one specific view of all this data. Then [21:34] you actually start making questions the [21:36] same way that an engineer would phrase [21:38] them. And if you have detectors that [21:40] watch those slices at scale when [21:43] something moves or if a bottleneck [21:45] becomes evident, you can create a [21:48] package that becomes an input for the [21:50] next step in the software building [21:52] process. [21:54] We're going to jump to building one [21:58] particular feature and this is [22:00] interesting for a video software [22:02] especially if you are running it in a [22:04] device in a Android phone for example. [22:07] And there's two parts to this. One is [22:09] what the human needs to provide and the [22:12] second one is that what the agent [22:14] understands and what it can do. So from [22:16] humans, we want the intent. We want [22:18] algorithms. We want architecture and [22:21] system design thinking. So very similar [22:24] to the language that we've been using [22:26] throughout this talk. But also we need [22:29] to think about what test cases the agent [22:32] needs to execute in order to be [22:36] confident that it did its job right that [22:38] the video plays the right way that [22:40] there's no flickering black frames etc. [22:44] And from this intent we need to create [22:46] artifacts that are deterministic for an [22:49] agent to execute. That means creating a [22:51] spec of what the feature looks like, an [22:53] implementation plan with very specific [22:55] steps that the agent needs to follow in [22:57] order to finish the implementation and [23:00] executable codes which is commands that [23:03] the agent can execute, can run and get a [23:07] very clear determination of whether its [23:10] job is done or not. [23:12] So let's look at how this second part [23:15] looks like because it's what makes or [23:18] break aic flow that builds software. One [23:20] of the best uses of agents is not [23:22] necessarily throwing at every single [23:24] problem you have and then hoping that it [23:26] knows how to traverse every single [23:28] decision that it needs to make because [23:30] LLMs are stoastic and it will provide [23:33] different versions of the same answer. [23:35] So what we try to do is use the agents [23:37] to code or generate different tools and [23:41] scripts that can deterministically drive [23:44] as much of these actions and as much of [23:47] these decisions as possible. So tooling [23:50] that before was very hard for one human [23:52] engineer to just code on its own. Now [23:55] it's becoming cheaper and cheaper. Let's [23:57] look at this specifically for video. [23:59] First, as we are creating the test [24:01] cases, you need to think what sort of [24:03] media do we want to test. So you can [24:05] have tools that create videos using a [24:08] Vmpeek or other similar tools and these [24:11] input videos can have different [24:13] resolutions, different frame rates, cop [24:15] sizes, etc. So you can have like a very [24:18] nice library of input media. So you can [24:20] validate different cases. [24:23] Then you can script generation for [24:25] different composition types like [24:27] composition of overlapping tracks, [24:30] different effects applied, GIS on top of [24:32] text, whatever you can think of, you can [24:34] create a script for it or rather you can [24:37] tell an agent to create a script for it. [24:40] Then you need to create tools that the [24:42] agent can execute in order to drive [24:44] actions on the app. So there's plugins [24:48] that you can hook into in Android like [24:50] dump app to observe what's on the screen [24:53] and actually click throughout using ADB [24:55] commands. So you can create other [24:56] scripts that can actually execute [24:58] scripted gestures that drive actions [25:02] over those complex compositions. [25:05] As the agent is executing these [25:07] commands, you can record the screen. You [25:09] can observe the traces that are being [25:11] generated by the app. And then you can [25:13] have validators that are also created [25:16] deterministically as scripted tools that [25:18] the agent can execute fit maybe a video [25:22] recording some traces and they can [25:25] observe the pixels in those recordings [25:28] to make assertions about whether the [25:31] video played correctly and you can know [25:34] what is expected at every time step [25:37] because you have scripted every gesture [25:39] and you have generated every test [25:41] fitting. So those checks can [25:43] deterministically deterministically tell [25:45] you is there a black frame, is there a [25:47] flickering, is the frame shown following [25:51] the finger closely, or did it take too [25:53] long from the moment that the gesture [25:55] started? And I can show you how this [25:57] looks like in practice. In this video, [26:00] we have an agent driving the action. [26:02] It's clicking, it's swiping, and as it [26:05] does, it's also observing the pixels in [26:08] the screen to make the determination of [26:10] is the content correct? and was it fast [26:12] enough and can tell you green or red [26:14] depending on the work it did. Once it [26:17] can tell that it's not done, the agent [26:20] can loop and it's very good at [26:22] generating a 100 different [26:23] implementations that try different [26:25] things and try to really drill down into [26:27] different edge cases that solve these [26:30] test cases. [26:33] So there's two takeaways that I want to [26:35] leave with you with this journey that's [26:37] taking us towards an AI native world. [26:40] First, it makes a whole lot of [26:42] difference to define executably what [26:45] done means. And by this, I mean not just [26:49] text, but an actual command that the [26:52] agent can run and that would tell you [26:54] yes or no based on what they're [26:55] building. [26:57] And second, agents can build and excel [27:00] at building scripts and a lot of [27:02] different tools that can [27:03] deterministically validate or run [27:06] different actions. By removing the [27:09] judgment from things that can be done [27:11] deterministically, you're actually using [27:13] LMS what they're best and that is [27:15] covering those gaps of judgment that are [27:18] still remaining. [27:22] So now that we've seen the type of work [27:25] that goes into optimizing a moving [27:27] target like this and making a video [27:29] player faster and how this is going [27:33] towards the AI native world, let's look [27:35] at where this is heading towards. [27:39] This is a slightly bigger loop of the [27:42] same boxes that I showed earlier, but [27:44] what's really different here is how [27:46] we're connecting the different boxes. So [27:47] we can have agents that observe [27:49] production on their own that slice and [27:51] dice all these different dimensions to [27:53] find bottlenecks where the paint [27:55] concentrates. You can have different [27:57] agents that take this as input and can [27:59] actually analyze the code. They can [28:01] biseect app versions, look at divs, look [28:04] at one particular composition that one [28:06] user did on their device and reproduce [28:09] because they have the tools to drive a [28:11] device and try to observe what's going [28:12] on. They can produce packages for humans [28:15] to decide and design how to improve the [28:18] systems and more importantly which [28:20] problems are actually worth pursuing. [28:22] Agents can move forward with the actual [28:24] building with the look we just looked [28:26] at. Produce review packages for humans [28:29] to decide whether we should try this out [28:31] with actual users. And then agents can [28:33] take forward with the operational parts [28:35] of configuring experiments, creating [28:38] them, ramping them up, actually [28:41] observing metrics for a few days and [28:43] creating metric readouts for humans to [28:46] decide whether something ships or not. [28:49] And then agents can take forward the [28:51] cleanup and every other type of code [28:54] complexity management. Essentially, [28:56] we're taking humans from being in the [28:59] loop to on the loop. [29:02] So, let's look back at how the industry [29:04] has been evolving. Video editing used to [29:07] require specialized hardware and desktop [29:10] machines, but now creators can do it in [29:12] their mobile phones with their thumb. [29:14] Also, the video player software itself [29:16] is a moving target, too. And it used to [29:20] require humans to carry every change [29:22] from observation to deployment. And soon [29:25] we're going to observe how increasingly [29:27] agents are going to carry that change [29:29] from observing and trying out different [29:31] combinations and also observing how the [29:34] user reacts to them and completing that [29:37] software development process. [29:39] And in the end, our users don't care [29:42] about how complex is the technology that [29:44] they're using or even how we're building [29:46] it. Their experience should be as simple [29:49] as putting their finger down and [29:52] observing how the product just works. [29:56] Thank you.