SlidesLive Deploys Multimodal Camera Rig to Cut Live Event Video Costs

How a Conference Room Broke Our Computer Vision Assumptions With SlidesLive — Ai4

Ai4

Autonomous conference recording requires fusing audio diarization with lip reading and skeletal tracking because venue acoustics and speaker movements break single-sensor computer vision models.

AI-assisted summary of How a Conference Room Broke Our Computer Vision Assumptions With SlidesLive — original source. Check the original for full context.

Source published:

Detailed notes

## Background Published on YouTube by Ai4 on August 17, 2026, this presentation features Vojta Ciml, Founder and CEO of SlidesLive, exploring the hidden complexities of automated event capture. According to the description, Ciml walks through the evolution of SlidesLive's autonomous recording system, NATE, from simple face detection to a multimodal audio-visual architecture refined across more than 100,000 recorded presentations.

## TL;DR Vojta Ciml explains how unpredictable real-world conference spaces systematically broke early computer vision and acoustic localization assumptions when developing SlidesLive's autonomous recording rigs. To replace cumbersome production crews with a portable single-box solution, the team engineered a multimodal fusion architecture combining audio diarization, lip-motion tracking, dual-camera homography, and full-body skeletal tracking. The platform models spatial intention via stage heatmaps to eliminate visual jitter, while direct screen grabber OCR automatically indexes slide text alongside transcripts. Concluding with practical lessons, Ciml emphasizes the necessity of human-in-the-loop oversight and multimodal cross-validation to permanently archive spoken human knowledge.

## Atmosphere & style The presentation opens with an interactive listening experiment that immediately engages the audience and sets a pragmatic, engineering-focused tone. Ciml's narrative style is candid and self-effacing, balancing technical ambition with honest retrospectives on real-world failures such as pitch-black power outages and seasick camera hunting. The talk feels like an insightful, behind-the-scenes masterclass in applied robotics, combining rigorous computer vision concepts with vivid battle-tested stories from live conference floors.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=atS6iuvPB44&t=4s) Distributed venue PA systems disperse sound throughout the auditorium, making it impossible to deduce a speaker's physical stage location using audio cues alone. - [01:25](https://www.youtube.com/watch?v=atS6iuvPB44&t=85s) Traditional event video production is burdened by excessive cabling, converters, and labor costs, prompting the vision for an autonomous, single-case recording rig. - [02:44](https://www.youtube.com/watch?v=atS6iuvPB44&t=164s) Multimodal fusion pairing acoustic speaker diarization with computer vision lip tracking resolves who is speaking and where they are located on noisy stages and multi-person panels. - [04:38](https://www.youtube.com/watch?v=atS6iuvPB44&t=278s) A single PTZ camera loses room context when zooming, requiring a secondary wide-angle camera and planar homography alignment to prevent tracking blind spots and snap errors. - [06:27](https://www.youtube.com/watch?v=atS6iuvPB44&t=387s) Algorithmic face tracking induces viewer motion sickness through jittery micro-adjustments; spatial heatmaps and a strict two-frame latency budget are needed to emulate smooth human cinematography. - [09:07](https://www.youtube.com/watch?v=atS6iuvPB44&t=547s) Standard face-tracking pipelines fail when presenters turn toward their slides, requiring a transition to full-body skeletal pose estimation to maintain tracking continuity. - [09:58](https://www.youtube.com/watch?v=atS6iuvPB44&t=598s) Unpredictable venue anomalies like sudden blackouts demonstrate that automation should not eliminate humans, but reposition them as high-level supervisors monitoring multiple rooms. - [12:14](https://www.youtube.com/watch?v=atS6iuvPB44&t=734s) Robotics and computer vision models must be stress-tested in messy, high-stakes real-world environments rather than relying on sanitized laboratory benchmarks. - [13:46](https://www.youtube.com/watch?v=atS6iuvPB44&t=826s) Deploying experimental robotic rigs side-by-side with professional human videographers during live events creates a reliable ground-truth baseline for offline simulation and tuning. - [15:02](https://www.youtube.com/watch?v=atS6iuvPB44&t=902s) Direct hardware video capture combined with real-time screen OCR bypasses the inaccuracies of pre-submitted slide decks to deliver synchronized captions and AI summaries. - [16:36](https://www.youtube.com/watch?v=atS6iuvPB44&t=996s) Designing for standard venue drops (XLR, HDMI/SDI, Ethernet) compresses half-day production setups into a 30-minute deployment while balancing edge compute limits through temporal stream consistency.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=atS6iuvPB44&t=4s) The Stage Localization Illusion

Vojta Ciml opens with an interactive thought experiment, asking the audience to close their eyes and determine purely by listening whether he is standing on the left, right, or center of the stage. While humans instinctively rely on binaural auditory cues to pinpoint sound sources, the exercise exposes an immediate flaw when applied to live event spaces. The moment attendees open their eyes, the illusion collapses: the voice does not emanate directly from the presenter’s mouth into their ears, but radiates throughout the auditorium through distributed loudspeakers. This acoustic dispersion makes it virtually impossible to infer a speaker's physical stage position from audio alone.

Ciml introduces himself as the co-founder of SlidesLive, framing this audio misdirection as the first of five fundamental assumptions that fell apart during the development of their autonomous robotic camera systems. SlidesLive has recorded more than 100,000 presentations and thousands of events worldwide—including the Ai4 conference itself, where their robotic systems monitor every track. Driven by the mission to capture and archive presentations for future generations, the team set out to disrupt traditional event video production, which has long been notoriously expensive, labor-intensive, and weighed down by heavy equipment, bizarre converters, and sprawling cable runs.

- SlidesLive has recorded over 100,000 presentations globally across thousands of events. - Distributed venue PA systems break natural auditory spatial awareness by separating the acoustic source from the speaker's physical position.

---

[01:25](https://www.youtube.com/watch?v=atS6iuvPB44&t=85s) The Dream of Single-Box Event Recording

Confronted with the logistical friction of legacy videography, the SlidesLive engineering team posed what seemed to be a straightforward question: "What if everything you need to record a session like this would fit in a single case?" The vision was to compress all the cabling, splitters, converters, and processing hardware into an autonomous robotic camera rig deployed directly in the center of the conference room.

With the customary optimism of early-stage hardware and software design, the team initially wondered, "How hard can that be?" Reality quickly pushed back with an array of edge cases and harsh operational constraints unique to live convention spaces, leading to a series of hard-won lessons in sensor fusion and robotics.

---

[02:44](https://www.youtube.com/watch?v=atS6iuvPB44&t=164s) Multimodal Fusion: Solving the Audio Localization Problem

The initial design assumed that audio tracking alone could locate a speaker. This breakdown is especially acute during panel discussions, where multiple individuals sit beside each other and face tracking cannot determine who currently holds the floor. Without knowing who is speaking, the camera cannot intelligently pan or zoom into an active participant. Compounding the PA speaker issue, conference stages are far removed from acoustically isolated Hollywood studios. Instead, they are exposed to loud ambient noise from adjacent expo floors, bustling corridors, or live musical entertainment. In one real recording sample played by Ciml, ambient noise overwhelmed the camera's built-in microphone entirely.

To solve this, SlidesLive abandoned single-modality tracking in favor of a multimodal fusion architecture. The system decouples voice audio and visual signals: the incoming audio feed is processed for speaker diarization to determine *who* is vocalizing, while computer vision actively tracks lip movement to verify *where* that sound originates on stage. By synthesizing these data streams, the system builds an ephemeral memory profile for every speaker on stage, storing visual facial data, camera coordinates, and a vocal fingerprint.

```mermaid flowchart TD accTitle: Multimodal Speaker Memory and Tracking Flow subgraph SENSORS["Input Streams"] A["Room Audio Feed<br/>(XLR / Console)"] B["PTZ & Room Cameras<br/>(Video Feeds)"] end

subgraph PROCESSING["Feature Extraction"] C["Speaker Diarization<br/>(Voice Fingerprint)"] D["Computer Vision<br/>(Lip Motion Detection)"] E["Spatial Coordinates<br/>(Stage Heatmap)"] end

subgraph MEMORY["Speaker State Profile"] F["Speaker Memory Registry<br/>• Facial Geometry<br/>• Voice Fingerprint<br/>• Camera Target Coordinates"] end

subgraph OUTPUT["Camera Control"] G["PTZ Dynamic Framing & Zoom"] end

A --> C B --> D B --> E C --> F D --> F E --> F F --> G ```

- Audio provides speaker diarization (identity), while computer vision monitors mouth and lip movement to confirm physical activity. - The system registers an active dynamic memory profile per speaker containing face embeddings, stage coordinates, and a voice fingerprint.

---

[04:38](https://www.youtube.com/watch?v=atS6iuvPB44&t=278s) Dual-Camera Homography and Room Context

Cost-efficiency initially suggested that a single pan-tilt-zoom (PTZ) camera per room would suffice to frame the stage. This premise collapsed under the behavioral unpredictability of live events. Masters of ceremonies unexpectedly cross into frame, surprise co-presenters take the podium, and erratic motivational speakers wander completely off the stage toward the back of the hall. When a single camera zooms in and follows an unexpected movement, it sacrifices all surrounding spatial context, leaving the system blind to the rest of the room.

To overcome this spatial blindness, SlidesLive introduced an auxiliary static wide-angle camera that continuously monitors the entire stage and room geometry. Operating as the "eyes" of the primary robotic tracking camera, this wide sensor tracks scene dynamics and predicts presenter movement. To map the two disparate perspectives together, the system relies on planar homography, matching keypoints across different optical fields, resolutions, and camera angles. Because live production affords zero second takes, the alignment algorithm incorporates cross-checking routines to prevent catastrophic tracking jumps—such as erroneously snapping and zooming the primary lens directly into empty stage drapes during a live broadcast.

- A secondary wide-angle camera maintains continuous context over the full room when the primary camera is tight on a subject. - Homography transformations align coordinate systems between disparate resolutions and angles. - Continuous cross-checking prevents sudden tracking snap failures toward background drapes or empty walls.

---

[06:27](https://www.youtube.com/watch?v=atS6iuvPB44&t=387s) Emulating Human Cinematography Without the Seasickness

Automating camera movement revealed subtle cinematic nuances that simple face-following algorithms fail to capture. Human camera operators suffer significant fatigue over an eight-hour conference day, frequently experiencing lapses in concentration that result in delayed panning, framing drift, or lost focus. However, purely algorithmic face tracking introduced its own failure mode: rigid, jittery micro-adjustments. When a presenter fidgets, sways, or shifts weight behind a lectern, a naive face-tracking camera continuously hunts to keep the head centered, inducing visual seasickness in the viewer.

To mimic the relaxed, fluid motion of an experienced human videographer, SlidesLive developed spatial intention modeling using stage heatmaps. The tracking engine monitors movement patterns to determine whether a speaker is merely shifting in place or making a deliberate pacing stride across the stage. The camera filters out minor sway while anticipating walking velocity and directional changes. Crucially, the motion pipeline maintains a strict latency threshold: any tracking delay exceeding two video frames becomes visibly unnatural to the audience, breaking the illusion of an expert operator behind the lens.

- Unfiltered head tracking causes viewers to become seasick due to micro-adjustments from natural speaker sway. - Spatial heatmaps classify speaker behavior between static lectern fidgeting and intentional stage pacing. - Camera panning algorithms must execute within a strict two-frame latency budget to avoid visible robotic lag.

---

[09:07](https://www.youtube.com/watch?v=atS6iuvPB44&t=547s) Beyond Face Tracking: Full-Body Pose and Occlusion

Another faulty assumption was that standard off-the-shelf facial recognition and tracking models would operate reliably across all presenters. In practice, models exhibited inconsistent performance across diverse demographic groups and under adverse conference lighting conditions or varied hairstyles.

Furthermore, real-world presenters frequently violate tracking assumptions by turning their backs completely to the audience to gesture toward their slides. The moment a presenter turns around, face detection models lose all tracking features. SlidesLive consequently migrated from face-only detection to full-scale skeletal pose estimation. By tracking the entire human body frame, the system retains robust tracking continuity even when the face is fully occluded, in deep shadow, or facing the projection screen.

- Pure face-detection pipelines fail whenever a presenter turns around to read or gesture at projected slides. - Moving to full-body skeletal tracking guarantees continuous tracking irrespective of orientation, lighting anomalies, or physical occlusions.

---

[09:58](https://www.youtube.com/watch?v=atS6iuvPB44&t=598s) The Reality of Real-World Chaos and Human-in-the-Loop

The final assumption was that solving the known computer vision edge cases would allow SlidesLive to deploy fully autonomous recording boxes everywhere without human oversight. Real-world event production swiftly proved that unexpected failures are inevitable. Ciml recounts an incident where the main venue suffered a sudden electrical blackout midway through a presentation. While SlidesLive’s rigs stayed operational thanks to integrated battery backups, the camera was left tracking in total, pitch-black darkness.

Hardware faults, severed cables, network drops, and unexpected physical obstructions mean that human operators remain an essential fail-safe. Rather than eliminating humans entirely, SlidesLive’s philosophy repositions the human operator from an exhausted, eight-hour camera-stander to a multi-room supervisor. Autonomous software handles the mundane mechanical framing, freeing technical personnel to manage complex recovery procedures, troubleshoot anomalies, and intervene when system confidence drops.

- Unexpected catastrophic failures, such as complete room power blackouts, break purely automated computer vision assumptions. - Automation relieves humans of grueling, static camera-operating shifts, shifting the human role to high-level technical oversight and multi-room exception handling.

---

[12:14](https://www.youtube.com/watch?v=atS6iuvPB44&t=734s) Multimodal Validation and Preserving Human Knowledge

Ciml concludes the talk by urging computer vision and robotic engineers to aggressively stress-test their models in real-world, high-stakes environments rather than relying on sanitized lab benchmarks. Just as human perception relies on both ears and eyes to resolve ambiguity, robust robotic automation must synthesize multiple sensory inputs, continuously cross-validating each decision before executing physical commands.

He closes with a call to prioritize conference recording, observing that for thousands of years, spoken human wisdom and insights shared during live gatherings have simply evaporated the moment sessions ended. By deploying scalable, autonomous capture technology, SlidesLive aims to permanently preserve this collective knowledge base for future generations.

- Real-world validation must seek out worst-case operational scenarios rather than ideal laboratory conditions. - Preserving spoken presentations prevents valuable domain insights from being lost forever once an event concludes.

---

[13:46](https://www.youtube.com/watch?v=atS6iuvPB44&t=826s) Q&A: Iterative Engineering and Live Simulation

An audience member asks how SlidesLive prioritized and systematically engineered solutions for this vast array of overlapping problems. Ciml explains that their roadmap relies heavily on embedded engagement with the broader computer vision academic community at conferences like ICML and ICLR.

Operationally, their engineering philosophy prioritizes side-by-side field testing. SlidesLive continuously deployed their experimental robotic rigs directly alongside veteran human camera operators during live commercial events. This setup served as a live benchmarking platform: the team captured real human tracking decisions as baseline ground truth, ran offline spatial simulations, and iteratively addressed CV failure modes one variable at a time without endangering the client's final recording delivery.

- System tuning relies on dual-deployment benchmarking, placing autonomous rigs side-by-side with professional human videographers. - Continuous development utilizes conference datasets from computer vision gatherings like ICML and ICLR alongside rigorous offline simulation.

---

[15:02](https://www.youtube.com/watch?v=atS6iuvPB44&t=902s) Q&A: Ingesting Screen Feeds, Slide OCR, and Watermarks

Addressing a question regarding slide capture, Ciml explains why downloading presentation slide decks post-event is notoriously unreliable. Presenters frequently use last-minute modified slide decks, run live software demonstrations, or present decks that differ from the copies submitted to event organizers. To circumvent this, SlidesLive captures direct screen video feeds via hardware grabbers, running optical character recognition (OCR) directly on the rendered display. The extracted visual text is programmatically aligned with the audio transcript to generate high-accuracy captions, search indexes, and automated AI summaries.

Responding to a follow-up inquiry about watermarking, Ciml clarifies that SlidesLive functions strictly as a technology infrastructure and service provider rather than a content rights holder. Watermarks are applied dynamically based entirely on client and event sponsor requirements, leaving full ownership and presentation rights with the original organizers and speakers.

- Direct hardware capture of presentation screens avoids discrepancies caused by outdated, pre-submitted slide files. - Real-time OCR extracted from the presentation display feeds into multimodal AI pipelines for search indexing and structured summarization. - Watermarking remains completely customizable based on organizer branding and sponsor agreements, with SlidesLive maintaining no content ownership.

---

[16:36](https://www.youtube.com/watch?v=atS6iuvPB44&t=996s) Q&A: Simplified AV Tech Riders and Edge Compute Limits

When asked about specialized venue tech riders, Ciml highlights that SlidesLive engineered their system to integrate seamlessly with existing audio-visual infrastructure. Instead of requiring exotic hardware, the rig accepts standard inputs that any venue AV technician already provides: an XLR drop for house audio, an HDMI or SDI line for screen output, an Ethernet connection, and basic AC wall power. This eliminates the standard production footprint of a dedicated six-foot AV table cluttered with conversion hardware that takes half a day to set up and strike, reducing the entire staging process to a 30-minute unpack from a single travel case.

Answering a final question on processing capacity, Ciml acknowledges the strict computational boundaries of running edge hardware inside a self-contained capture box. Because the localized hardware cannot host unlimited computing resources, the tracking algorithms are engineered to tolerate occasional dropped analysis frames. Rather than demanding exhaustive, frame-by-frame inference that would stall edge processors, the pipeline relies on continuous temporal stream consistency to maintain fluid, accurate robotic tracking within an affordable compute budget.

- System integration relies on industry-standard AV drops: balanced XLR audio, SDI/HDMI video feeds, standard AC power, and wired Ethernet. - Equipment footprint replaces bulky production tables and half-day setups with a 30-minute deployment from a single portable case. - Edge compute constraints are addressed by strategically dropping non-critical analytical frames, relying on temporal continuity to maintain camera stability.

## Notable quotes > Event recording sounds like a solved problem. It isn't. — Vojta Ciml

> [00:04](https://www.youtube.com/watch?v=atS6iuvPB44&t=4s) The moment attendees open their eyes, the illusion collapses: the voice does not emanate directly from the presenter’s mouth into their ears, but radiates throughout the auditorium through distributed loudspeakers. — Vojta Ciml

> [01:25](https://www.youtube.com/watch?v=atS6iuvPB44&t=85s) What if everything you need to record a session like this would fit in a single case? — Vojta Ciml

> [01:25](https://www.youtube.com/watch?v=atS6iuvPB44&t=85s) How hard can that be? — Vojta Ciml

> [02:44](https://www.youtube.com/watch?v=atS6iuvPB44&t=164s) The incoming audio feed is processed for speaker diarization to determine who is vocalizing, while computer vision actively tracks lip movement to verify where that sound originates on stage. — Vojta Ciml

> [04:38](https://www.youtube.com/watch?v=atS6iuvPB44&t=278s) When a single camera zooms in and follows an unexpected movement, it sacrifices all surrounding spatial context, leaving the system blind to the rest of the room. — Vojta Ciml

> [06:27](https://www.youtube.com/watch?v=atS6iuvPB44&t=387s) When a presenter fidgets, sways, or shifts weight behind a lectern, a naive face-tracking camera continuously hunts to keep the head centered, inducing visual seasickness in the viewer. — Vojta Ciml

> [06:27](https://www.youtube.com/watch?v=atS6iuvPB44&t=387s) Any tracking delay exceeding two video frames becomes visibly unnatural to the audience, breaking the illusion of an expert operator behind the lens. — Vojta Ciml

> [09:07](https://www.youtube.com/watch?v=atS6iuvPB44&t=547s) The moment a presenter turns around, face detection models lose all tracking features. SlidesLive consequently migrated from face-only detection to full-scale skeletal pose estimation. — Vojta Ciml

> [09:58](https://www.youtube.com/watch?v=atS6iuvPB44&t=598s) Rather than eliminating humans entirely, SlidesLive’s philosophy repositions the human operator from an exhausted, eight-hour camera-stander to a multi-room supervisor. — Vojta Ciml

> [12:14](https://www.youtube.com/watch?v=atS6iuvPB44&t=734s) For thousands of years, spoken human wisdom and insights shared during live gatherings have simply evaporated the moment sessions ended. — Vojta Ciml

> [13:46](https://www.youtube.com/watch?v=atS6iuvPB44&t=826s) System tuning relies on dual-deployment benchmarking, placing autonomous rigs side-by-side with professional human videographers. — Vojta Ciml

## Mentioned Vojta Ciml, SlidesLive, NATE, Ai4, ICML, ICLR, PTZ camera, Optical Character Recognition (OCR), XLR, HDMI, SDI, Ethernet

## Video description Vojta Ciml - Founder & CEO, SlidesLive

Event recording sounds like a solved problem. It isn't. I'll walk through how we evolved NATE, SlidesLive's autonomous recording system, from basic face detection to a multimodal system that fuses video and audio to do what a full production crew would: track speakers, switch cameras, monitor quality, and secure the recording. I'll cover the architecture, failed experiments, tuning process, and what 100,000 recorded talks taught us about building systems that work when there's no second take.

[Source](https://www.youtube.com/watch?v=atS6iuvPB44)

中文简报

SlidesLive部署多模态摄像设备以降低现场活动视频制作成本 由于场馆声学环境和演讲者移动会使单一传感器的计算机视觉模型失效，自主会议录制需要将音频说话人日志与唇读和骨骼追踪相结合。 Vojta Ciml解释了在开发SlidesLive自主录制设备时，不可预测的真实会议空间如何系统性地打破了早期的计算机视觉与声学定位假设。为了用便携式单箱解决方案取代繁重的制作团队，团队设计了一种多模态融合架构，结合了音频说话人日志、唇部动作追踪、双摄像机单应性对齐以及全身骨骼追踪。该平台通过舞台热力图对空间意图进行建模以消除画面抖动，同时通过屏幕直接采集OCR将幻灯片文本与转录文本一同自动建立索引。在总结实践经验时，Ciml强调了人在回路监督与多模态交叉验证的必要性，以永久存档人类口头知识。 分布式场馆扩声（PA）系统会将声音分散到整个礼堂中，导致无法单凭音频线索推断演讲者在舞台上的物理位置。 (0:04) 传统的活动视频制作受累于繁杂的线缆、转换器以及高昂的人工成本，促成了开发便携箱式自主录制设备的构想。 (1:25) 将声学说话人日志与计算机视觉唇部追踪相配对的多模态融合方案，解决了在嘈杂舞台和多人讨论场景下确认谁在发言以及其具体位置的问题。 (2:44) 单个PTZ摄像机在变焦时会丢失全场环境信息，因此需要配备辅助广角摄像机并进行平面单应性对齐，以防止追踪盲区和画面突跳错误。 (4:38) 算法人脸追踪因微小抖动的调整容易引起观众眩晕；必须依靠空间热力图和严格限制在两帧以内的延迟预算，来模拟流畅的人工作运镜效果。 (6:27)

Reading summary

SlidesLive co-founder Vojta Ciml explains how live conference environments routinely break baseline computer vision and audio assumptions, forcing autonomous camera systems to rely on multimodal fusion, dual-camera homography, and full-body pose estimation. Multimodal Fusion Solves Room Acoustics (2:44) Traditional event videography relies on extensive manual setups that SlidesLive sought to condense into a single autonomous case. However, initial assumptions regarding audio-based speaker tracking quickly broke down. Distributed venue public-address systems disassociate sound from physical presenter positions, and adjacent expo noise frequently overwhelms onboard directional microphones. Simple face detection also fails to identify active speakers during multi-person panels. To overcome acoustic dispersion and visual ambiguity, the engineering team built a multimodal fusion architecture. The system decouples incoming audio and visual streams: room audio feeds run through speaker diarization to identify who is speaking, while visual models detect lip motion to locate the active presenter on stage. The platform then constructs a dynamic memory profile for each participant, combining facial geometry, stage coordinates, and vocal fingerprints to steer robotic pan-tilt-zoom (PTZ) cameras. Wide-Angle Homography and Pose Tracking (4:38) Relying on a single PTZ camera introduces spatial blindness whenever unexpected events occur, such as surprise co-presenters entering the frame or speakers wandering into the audience. To preserve broader room context while maintaining tight framing, SlidesLive added an auxiliary static wide-angle camera. Planar homography maps keypoints between the two different optical viewpoints and resolutions, supported by continuous cross-checking to prevent the robotic camera from snapping toward empty walls or background drapes. Off-the-shelf facial tracking models also proved unreliable under irregular stage lighting, diverse demographics, and common presenter behaviors like turning around to point at slides. To eliminate tracking dropouts caused by back-facing speakers or occluded faces, the tracking architecture shifted from facial recognition models to full-body skeletal pose estimation, maintaining continuous positional tracking regardless of orientation. Cinematic Motion and Edge Constraints (6:27) Algorithmic face-centering initially created visual fatigue for viewers by hunting continuously in response to small physical movements, such as a presenter shifting weight behind a podium. To emulate the fluid framing of experienced human videographers, SlidesLive introduced stage heatmaps and intention modeling. The system differentiates between minor lectern fidgets and deliberate stage pacing, smoothing out micro-movements while observing a strict two-frame latency budget to avoid visible tracking lag. Because all processing occurs locally inside a self-contained case, edge compute limits require engineering compromises. Rather than executing exhaustive inference on every frame, the pipeline allows non-critical analytical frames to drop, relying on temporal stream continuity to stay within compute budgets while interfacing with standard venue connections: an XLR audio line, HDMI or SDI video feeds, Ethernet, and basic AC power. Human-in-the-Loop Supervision and Preservation (9:58) Despite extensive automated safeguards, unexpected physical incidents—such as full venue power blackouts—demonstrated that full autonomy without oversight is impractical in high-stakes live environments. Instead of eliminating technical personnel, SlidesLive transitioned human operators from manual single-camera operators into multi-room supervisors who manage edge-case failures and interventions. The autonomous models were developed through side-by-side benchmarking against veteran human operators during live events, using human camera decisions as baseline ground truth alongside academic datasets from conferences like ICML and ICLR. Ciml concludes that robust robotic vision requires rigorous stress-testing against worst-case physical environments rather than clean laboratory benchmarks, arguing that scalable autonomous capture is vital to prevent spoken domain knowledge from being lost once live conferences conclude.

精炼阅读摘要

SlidesLive 联合创始人 Vojta Ciml 指出，传统的单模态计算机视觉与纯音频追踪技术在真实的会议室环境中极易失效；团队为此摒弃了单一镜头和完全无人值守的预设，研发出融合多模态感知、双相机单应性映射、全身姿态估计以及人机协同的自主会议录制系统。 多模态融合破除声学定位假象 (2:44) 在现场会议中，分布式扩音系统会将声音散播至全场，破坏了人类天然的双耳听觉空间定位线索，导致算法无法仅凭音频辨别演讲者在舞台上的实际物理坐标。在多位嘉宾并排就座的小组研讨中，这一缺陷尤为致命，加之临近展区和走廊的环境噪音干扰，单靠麦克风与常规面部追踪根本无法确定发言权的实时归属。 为突破单模态局限，SlidesLive 采用多模态融合架构解耦音频与视觉信号：系统一方面对现场音频进行说话人日志分离以提取声纹特征，识别“谁在发声”；另一方面通过计算机视觉实时捕捉唇部动作，以确认声音在物理舞台上的具体出处。通过整合声纹、面部几何数据与舞台热力图坐标，系统为每位演讲者建立起动态记忆档案，从而驱动云台摄像机实现精准的特写与变焦。 双相机协同与单应性消除盲区 (4:38) 单纯依赖单台平移-俯仰-变焦摄像机的方案在真实的现场动态面前迅速瓦解。现场常有主持人临时走入镜头、演讲嘉宾突然走下舞台等意外情况，单相机一旦拉近特写跟拍移动目标，就会彻底丧失对全场环境的感知，形成严重的视野盲区。 SlidesLive 引入了一台固定广角辅助相机作为主控云台的“眼睛”，负责全天候捕捉整个舞台和房间的几何动态并预判人员走位。系统采用平面单应性算法对齐不同分辨率、视场角和光学角度的坐标系，并引入交叉验证机制以杜绝追踪跳跃，防止摄像机在实况转播中因误判而骤然对准空无一人的舞台幕布或背景白墙。 运镜意图建模与全身骨骼追踪 (6:27) 自动化运镜面临着复杂的视觉审美挑战。人类摄像师在连续工作8小时后容易疲劳走神导致运镜滞后，但机械的人脸追踪算法同样会产生弊端：演讲者在讲台后无意识的轻微晃动会触发镜头不断进行微小调整，给受众造成剧烈的晃动眩晕感。为此，团队基于舞台热力图开发了空间意图模型，通过分析运动模式区分原地微晃与大步走动，过滤细碎抖动并预判行进速度，且运镜执行严格受控在两帧画面延迟以内，以复现专业摄像师的平滑感。 此外，常规面部识别模型在逆光、复杂发型以及演讲者转身面向投影幕布时极易丢失目标。SlidesLive 因而将视觉算法从人脸识别升级为全身骨骼姿态估计，确保在演讲者背对观众指点幻灯片或面部处于阴影中时，镜头仍能保持连续稳定的追踪锁定。 边缘算力约束与人机协同韧性 (9:58) 为了取代重达数百公斤、需耗时半天搭建的传统转接设备与控制台，SlidesLive 将全套系统压缩进单个航空行李箱内，依托标准卡侬口音频、HDMI/SDI视频、以太网与普通市电接口，在30分钟内即可完成部署。面对集成硬件严格的边缘算力限制，算法主动策略性地丢弃部分非关键分析帧，依托时序数据流的连续性来维持运镜稳定性，从而避免算力过载。 面对会场突发全场断电等不可抗力事故，Ciml 强调彻底脱离人工监管是不切实际的。系统的核心价值不是完全取代人类，而是将人力从繁重的单机位机械操作中解脱出来，转变为同时监管多个会场的高级技术巡检人员，在算法置信度下降或硬件异常时介入处理，构建起坚固的人机协同容错屏障。 Ciml 呼吁计算机视觉与机器人研发人员必须在严苛的真实场景中反复压力测试，而非依赖理想化的实验室指标。SlidesLive 通过在真实会议中将自主设备与专业摄像师并排部署获取基准数据，并采用硬件直录屏幕加光学字符识别的方式生成结构化索引，致力于以可扩展的自动化录制方案，永久留存过去容易在会后随风消逝的现场演讲知识。

Detailed notes — zh

## Background 本内容由 Ai4 频道于 2026 年 8 月 17 日发布在 YouTube 平台上。视频记录了 SlidesLive 创始人兼 CEO Vojta Ciml 的主题演讲，介绍其自研自主录制系统 NATE 如何从基础的人脸检测演化为融合音视频的多模态系统，分享从 10 万场录制实战中总结的技术架构、失败教训与边缘计算权衡。 ## TL;DR Vojta Ciml 详细拆解了会议自主录制场景中被现实打破的五大经典假设，展示了从单一声音定位、单目追踪、人脸识别转向多模态音视频融合（声纹与唇动检测）及全尺寸人体姿态估计的技术演进过程。针对无重拍机会的单次直播严苛工况，团队设计了广角环境机几何对齐熔断与抗抖动意图推断算法，并将整套系统极简封装进单一手提箱实现 30 分钟即插即用。通过现场同台人机对照与跳帧容错策略，SlidesLive 在有限边缘算力下平衡了追踪精度与工程鲁棒性，同时保留“人类在环”以从容应对断电断线等极端突发异常。 ## Atmosphere & style 整个演讲务实接地气且极富工程极客色彩，开场通过一个巧妙的“闭眼听声辨位”互动迅速颠覆观众直觉，将现场带入充满启发性的探究氛围。Vojta 的讲述坦诚而幽默，毫无保留地复盘系统在真实极端工况下的各种翻车与硬件尴尬，形成了从天真设想到残酷现实再到硬核工程破局的精彩叙事弧线。整场分享节奏紧凑、干货满满，让听众如同亲历一场在真实世界边缘计算泥淖中不断碰壁、调试并最终实现产品化落地的技术攻坚战。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=atS6iuvPB44&t=4s) 扩音音箱带来的声相漂移与嘈杂环境打破了依靠声音定位的直觉，传统人工录制成本高昂，SlidesLive 致力于将整套会议录制系统浓缩进便携式手提箱。 - [02:44](https://www.youtube.com/watch?v=atS6iuvPB44&t=164s) 团队采用多模态融合方案破局圆桌论坛识别难题，结合音频说话人日志（Diarization）、视觉唇动检测与动态讲者记忆库精准锁定当前发言者。 - [04:38](https://www.youtube.com/watch?v=atS6iuvPB44&t=278s) 单台摄像机极易在讲者离台或多人穿插时丢失上下文，系统引入第二台广角环境相机，并通过单应性矩阵对齐与熔断校验机制避免镜头漂移。 - [06:27](https://www.youtube.com/watch?v=atS6iuvPB44&t=387s) 机械锁定面部会导致画面因讲者无意识晃动而剧烈抖动，算法需利用热力图与步态预测推断运动意图，实现 2 帧以内超低延迟的顺滑拟人运镜。 - [08:00](https://www.youtube.com/watch?v=atS6iuvPB44&t=480s) 面对讲者背对镜头、未成年人识别率低及弱光等缺陷，系统全面放弃纯人脸追踪，升级为全尺寸人体骨骼姿态识别（Pose Estimation）。 - [09:58](https://www.youtube.com/watch?v=atS6iuvPB44&t=598s) 断电、线缆损坏等现实意外表明系统不能脱离“人类在环”（Human-in-the-loop），由机器人承担繁重枯燥的常规运镜，人类操作员专注于极端排障。 - [13:55](https://www.youtube.com/watch?v=atS6iuvPB44&t=835s) 团队将学术前沿（ICLR、ICML 等峰会）与实战工程紧密结合，通过让机器人与人类摄像师同台并跑的双轨模式持续检验并迭代算法极限。 - [15:02](https://www.youtube.com/watch?v=atS6iuvPB44&t=902s) 放弃依赖讲者提前提交的 PPT，转为直接抓取现场投影屏幕视频流并结合 OCR 与语音转录，最大程度还原临时改动与 Demo 演示。 - [16:36](https://www.youtube.com/watch?v=atS6iuvPB44&t=996s) 硬件全面贴合 XLR、HDMI/SDI 等标准视听接口，淘汰笨重控台与长时间布线，将现场部署时间从半天大幅压缩至 30 分钟内。 - [18:08](https://www.youtube.com/watch?v=atS6iuvPB44&t=1088s) 针对边缘计算设备的算力瓶颈，系统利用会议长序列的时序冗余采取跳帧容错策略，在有限资源下优先保障跟踪连续性与运行稳定性。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=atS6iuvPB44&t=4s) 闭上双眼：音响系统如何打破我们对声音定位的直觉 Vojta Ciml 走上讲台，并没有用常规的开场白问好，而是向现场观众提出了一个互动请求：闭上眼睛几秒钟，仔细倾听他的声音，然后试着判断他此时此刻究竟站在舞台的哪个位置——是左边、右边，还是正中央？ 人类天生具备听声辨位的能力，能本能地感知声源的方向与讲话者的站位。然而当观众睁开双眼验证自己的猜测时，却发现直觉完全失灵了。原因在于，会场中安装了大型扩音扬声器（loudspeakers），观众耳朵捕捉到的声音并非直接来自讲演者的嘴唇，而是由分散在房间各处的音箱发出。这种声学环境使得单纯依靠听觉去精准定位讲台上的目标位置变得几乎不可能。这一看似简单的现场演示，正是 Vojta 及其团队在设计自主录制系统时推翻的第一个核心假设。 作为 SlidesLive 的联合创始人兼 CEO，Vojta 解释道，传统视频制作一直以来都极度昂贵、耗费大量人力，且伴随着繁重杂乱的器材堆叠、奇形怪状的线缆与转接盒。SlidesLive 确立了一个雄心勃勃的愿景：将录制整场会议所需的一切线缆、设备与转换器浓缩进一个手提箱中，彻底消除传统人工摄制组的负担，将每一场演讲留存给后代。截至本次演讲，SlidesLive 已在世界各地录制了超过 100,000 场视频，覆盖数千场大型活动（包括本次 Ai4 大会各会场里正在运行的机器人摄像机）。他们当初曾天真地认为“做个机器人摄像机能有多难？”，然而现实环境中的物理与感知陷阱，很快给了他们一记响亮的警钟。 --- ### [02:44](https://www.youtube.com/watch?v=atS6iuvPB44&t=164s) 假设一：通过声音就能锁定讲者位置（多模态融合破局） 团队最初的设想非常直截了当：在圆桌论坛（panel discussion）中，台上并排坐着五位嘉宾，摄像机不可能只追踪静态人脸，必须精准判断是谁在发言，进而驱动变焦镜头拉近特写。他们原本认为“声音会告诉你发言者在哪里”，但现实狠狠击碎了这一构想。除了前面提到的扩音器声相漂移外，会议现场根本不是完全隔音的好莱坞专业摄影棚。真实会场环境往往极度嘈杂：许多舞台紧邻嘈杂的展览大厅（expo hall），旁边甚至可能有现场乐队（live music band）在演奏，摄像机自带麦克风拾取到的环境音几乎是一团乱麻，根本无法直接通过空间声学特征定位目标。 为了克服单一声音定位的崩溃，SlidesLive 转向了多模态数据融合技术（Multimodal Fusion）： - **说话人日志（Speaker Diarization）**：音频信号仅用于声学特征提取与声纹匹配，判断“谁是谁”以及区分不同发言者。 - **计算机视觉唇动检测（Lip Movement Detection）**：利用计算机视觉算法实时检测画面中哪位嘉宾的嘴唇正在开合运动。 - **说话人综合记忆库（Speaker Memory）**：系统在后台为台上每一位讲者建立专属档案，实时绑定视觉人脸特征、摄像机绝对云台位置与声纹指纹（Voice Fingerprint）。通过音视频交叉校验，系统能够在嘈杂的圆桌论坛中准确无误地识别当前正在发言的人，并执行平滑缩放与运镜。 ```mermaid flowchart TD accTitle: 多模态讲者识别架构 subgraph Inputs["多源输入信号"] Audio["音频麦克风流"] Video["视觉摄像机流"] end subgraph Processing["特征提取与检测"] Diar["音频说话人日志<br/>(Speaker Diarization)"] Lip["视觉唇动检测<br/>(Lip Movement Tracking)"] end subgraph Memory["讲者动态记忆库 (Speaker Memory)"] Profile["多维特征绑定<br/>• 面部图像 (Face ID)<br/>• 云台预置位 (PTZ Position)<br/>• 声纹指纹 (Voice Fingerprint)"] end Target["执行精准特写变焦与追踪"] Audio --> Diar Video --> Lip Diar --> Profile Lip --> Profile Profile --> Target ``` --- ### [04:38](https://www.youtube.com/watch?v=atS6iuvPB44&t=278s) 假设二：单台摄像机足以掌控全局（单目局限与双目对齐） 出于成本控制与便携性的考虑，最初的假设是“一个房间只要架设一台摄像机就足够追踪讲者”。然而现场各种无法预测的人类行为迅速打破了单机设想：主持人（MC）可能中途毫无预兆地登台切入，或者第二位讲者突然登场。更极端的是某些充满激情的励志演讲者，会毫无征兆地走下讲台，一路溜达到观众席最后方继续大谈特谈。在这种瞬间，单台长焦追踪摄像机一旦丢失目标，就会彻底丧失整个房间的环境上下文，变成一台无头苍蝇般的盲拍机器。 为此，SlidesLive 引入了第二台广角环境摄像机（Ambient / Wide Camera），用于俯瞰全场并预测运动轨迹。但双相机方案随之带来了严苛的跨设备对齐（Alignment）挑战：两台摄像机的硬件规格完全不同、传感器分辨率各异、视角与视差也有显著差距。工程上必须使用单应性矩阵（Homography）检测两路画面中的特征关键点并进行几何映射对齐。 然而，现场会议录制是不可逆的单次通关过程（Live show, one shot, one take），完全没有“重拍第二条”的机会。一旦几何特征对齐因反光、动态遮挡等偶然因素崩溃，追踪相机可能会瞬间失控，镜头猛然漂移并放大对准背景幕布。因此，算法流水线中必须嵌入多层交叉验证逻辑，任何置信度不足的对齐动作都会被主动熔断，避免直播中出现灾难性的畸形运镜。 --- ### [06:27](https://www.youtube.com/watch?v=atS6iuvPB44&t=387s) 假设三与四：追踪讲者等于跟随人脸（拟人运镜与全尺寸骨骼建模） 在解决了“看得到”之后，真正的艺术难题在于“如何像人类摄影师那样自然地移动相机”。最初的简单直觉是：追踪讲者就是让镜头锁定面部。但在实际观察了人类摄影师的表现后，Vojta 发现人类在长达 8 小时的枯燥会议中同样频频失误——摄影师站立一整天极易走神，经常出现讲者突然走出画面边缘（out of frame）或者镜头失焦的情况；而只要镜头失误脱焦半秒钟，在现场大屏幕上就是一次无法挽回的直播事故。 然而，如果让机器人死板地实时锁定人脸，情况反而会更糟。许多讲者在演讲台（lectern）旁习惯性地轻微摇晃身体（wobbling），若云台机械盲目跟从面部微小位移，画面就会剧烈抖动，让观众产生如同晕船一般的恶心感（seasick）。为此，算法必须具备推断人类意图的能力： - **热力图意图推断**：构建讲者运动趋势热力图，分析讲者是在原地做无意识的身体晃动，还是具有明确意图的跨步走动。 - **步态与速度预测**：预测讲者是准备在台上大范围快步踱步，还是做出突发性的折返转向，从而规划出如同经验丰富、精力充沛的人类运镜师般顺滑的水平平移（Pan）和垂直俯仰（Tilt）。 - **超低延迟要求**：系统闭环控制延迟必须严苛压缩在 2 帧以内；哪怕稍有滞后，人类肉眼就会立刻察觉出机械式的生硬反应。 与此同时，“人脸识别能均等地识别人脸”这一假设也彻底破灭。早期的面部模型在检测未成年人（minors）面孔时准确率骤降；更致命的是，讲者在演示过程中常常会完全转过身背对镜头去看大屏幕上的幻灯片，此时面部完全不可见。 SlidesLive 随后放弃了纯人脸追踪方案，全面升级为全尺寸人体骨骼姿态识别（Full Scale / Pose Estimation），并专门针对弱光环境、各种复杂发型及遮挡状况进行了深度鲁棒性优化。 --- ### [09:58](https://www.youtube.com/watch?v=atS6iuvPB44&t=598s) 假设五：只要搞定算法细节，自动化工作就万事大吉 工程师们常有一种乐观信念：只要逐一攻克上述技术难点，这套自动化录制系统就大功告成了，随后便能批量复制成千上万台设备投入全球各大盛会。但现实世界永远充满了预料之外的极端偶发事件。Vojta 分享了一个令人哭笑不得的现场突发状况：在一场重要会议进行到一半时，整个场馆遭遇了彻底断电。虽然系统配备了应急电池，使得摄像机与控制主机仍在顽强运转，但在断电造成的全场漆黑（pitch black）中，没有任何计算机视觉算法能够凭空找到讲者。 这一残酷现实让团队认识到：在可预见的未来，真实世界场景中依然不可或缺“人类在环”（Human-in-the-loop）。硬件转接盒可能会烧毁，连接线缆会被现场人员意外踩断或扯断。即便系统设计了完备的冗余灾备，现场突发状况依然需要专业人员介入排障。不过，这种分工的底层逻辑已经发生了革命性转变： - 机器人系统承担了连续 8 小时死盯取景器、枯燥繁重的机械性运镜操作，保证绝大部分时间的平滑输出； - 人类操作员从重复劳动中解放出来，专注于处理模型丢失目标、机械断电断线等极端突发异常，成为高阶故障排除的技术指挥官。 在演讲尾声，Vojta 告诫在场的所有 AI 从业者与系统架构师：在构建软硬件系统时，不要躲在理想化的实验室象牙塔里，务必主动把系统推向复杂的现实世界极端工况（worst case scenarios）中经受捶打。正如演讲开头的声音错觉实验一样，单一感官极易受骗，只有融合多模态感知、多重信源交叉验证，才能打造出坚不可摧的工程系统。千百年来无数精彩的现场洞见在演讲结束后随风飘散，他呼吁所有大会主办方认真对待现场录制，借助稳健的技术将人类智慧妥善留存给未来世代。随后，Vojta 宣布进入现场提问环节（Q&A）。 ### [13:55](https://www.youtube.com/watch?v=atS6iuvPB44&t=835s) 解决现实边缘场景难题：双轨测试与学术前沿的持续迭代 面对现场观众关于“多重技术挑战错综交织，SlidesLive 究竟采取何种策略逐步突破”的提问，Vojta Ciml 阐释了团队将行业前沿研究与现场极端实操紧密结合的工程方法论。他表示，团队深度融入了计算机视觉（Computer Vision）学术共同体，通过频繁参与像 Ai4、ICLR 以及 ICML 这类顶级国际人工智能大会，不仅能第一时间接触到最新的视觉算法特性，还能直接与顶尖学者及同行交流探讨，避免在象牙塔外闭门造车。 更为关键的突破策略在于高强度的“实战并行验证”。由于 SlidesLive 承担着海量现场活动的录制任务，团队直接把自研的机器人硬件设备与传统人类摄像师放在同一个会场中协同运行，形成“机器与人类操作员同台竞技”的实测环境。在这种双轨模式下，团队不仅能无休止地在真实高压场景中检验算法极限，还能把录制数据带回实验室进行详尽的场景模拟与回放。面对错综复杂的光照、走位与遮挡问题，团队并没有妄想一蹴而就，而是由不同工程小组逐一攻破单点痛点，以小步快跑、持续迭代的方式逐步将系统打磨成熟。 - **学术生态对接**：持续跟进 ICLR、ICML 等机器学习顶级峰会的前沿 CV 模型成果，快速吸收并转化为工程实践。 - **人机协同测试**：在真实会议中同时部署“人类摄像师”与“自动化录制系统（NATE）”，在不增加交付风险的前提下完成不间断的高密度实地对照。 - **模拟与单点突破**：建立录制数据仿真环境，针对不同会场变量采取逐项排查、渐进优化的策略。 --- ### [15:02](https://www.youtube.com/watch?v=atS6iuvPB44&t=902s) 屏幕画面的实时解析：OCR、文本转录与内容版权归属 随后，Vojta Ciml 分享了他们在幻灯片采集上的技术转变。传统会议录制往往依赖讲者提前提交的 PowerPoint 演示文稿，但在实际演讲中，讲者展示的内容常常与预交文件脱节——他们可能临时修改了幻灯片，或者在台上切出 PPT 进行实时软件 Demo。如果线上观众事后仅仅下载原版课件，体验将严重割裂。为此，SlidesLive 改变了策略：“我们不再依赖讲者预先上传的文件，而是直接抓取投影屏幕的实时视频流。”系统通过计算机视觉对抓取的屏幕画面进行特征识别与 OCR（光学字符识别），并将提取出的文本内容与音频转录文本（Transcript）深度融合，从而生成高精度的 AI 摘要与闭环字幕（Closed Captioning），最大程度保留了现场演示的原貌。 ```mermaid flowchart LR accTitle: 幻灯片与音频的多模态分析处理流程 accDes: 展示直接捕获屏幕视频流并融合音频转录的流程图 subgraph Ingest["现场信号源捕获"] A["现场投影视频流<br/>(HDMI/SDI)"] B["麦克风音频流<br/>(XLR)"] end subgraph Process["多模态特征分析"] C["屏幕特征识别与 OCR"] D["语音转录识别 (ASR)"] end subgraph Output["智能内容生成"] E["多模态信息融合"] F["精准 AI 摘要与带字幕视频"] end A --> C B --> D C --> E D --> E E --> F ``` 针对现场提问中关于“视频水印（Watermarks）处理”的话题，现场引发了一阵笑声。Vojta Ciml 借此机会明确了 SlidesLive 的服务边界与数据所有权原则。他澄清道，添加水印完全取决于主办方或赞助商的定制化诉求，因为 SlidesLive 纯粹是一家技术服务商，平台本身并不拥有演讲内容的所有权。团队的核心职责在于负责任地采集音视频、存储数据并运行计算机视觉管线以确保画面质量与交付稳定性，至于画面上是否加盖水印、由谁加盖，完全尊重客户与版权所有者的决定。 - **直接捕获屏幕而非课件**：通过 HDMI 或 SDI 硬件接口直接捕获演讲者电脑向投影仪输出的实时信号，攻克版本不一致与 Demo 无法归档的痛点。 - **多模态语义对齐**：结合画面 OCR 提取的关键词与演讲原声转录，大幅提升下游大模型生成摘要与索引的准确率。 - **严格的内容权属分界**：坚持“技术承载但不占有内容”的原则，水印设置由客户或赞助商完全自主决定。 --- ### [16:36](https://www.youtube.com/watch?v=atS6iuvPB44&t=996s) 极简现场部署：从半天布设到30分钟“开箱即用” 谈及承接录制任务时是否会在技术要求协议（Tech Rider）中增加特殊门槛，Vojta Ciml 给出了令人意外的答案：他们不仅没有增加奇特要求，反而竭尽全力去贴合传统视听（AV）团队最熟悉的现成工作流。无论是音频接口还是视频接口，系统均设计为标准规范接入：音频走最普及的标准 XLR 卡农线，幻灯片与屏幕信号走通用的 HDMI 或 SDI，再外接一根标准电源线与一根连接外网的以太网网线即可完成物理对接。 这一设计彻底改变了传统影视制作团队冗长的进退场逻辑。在传统工作模式中，活动方通常需要为视频团队预留一张长达 6 英尺（约 1.8 米）的控制台，上面堆满杂乱奇特的设备，往往需要整整半天时间搭建布线，会议结束后还要花大半天拆卸撤场。而 SlidesLive 将整套机器人系统与计算中枢高密度封装在一个精简的便携小箱子中。现场工作人员只需开箱、放置好硬件并插上线缆，整个设置过程通常仅需 30 分钟即可准备就绪，不仅省去了占地庞大的控台和守候一整天的摄像师，也极大减轻了会议场地的后勤压力。 - **通用接口全面兼容**：采用标准 XLR 拾取混音台音频，采用 HDMI/SDI 抓取视频源，避免引入专有适配器。 - **硬件集成微型化**：淘汰了需要笨重三脚架和摄影师全天候值守的模式，转为高度集成的便携式机箱形态。 - **场地筹备成本骤降**：现场部署时间从传统的半天压缩至 30 分钟以内，无需占用 6 英尺长的专用 AV 操作台。 --- ### [18:08](https://www.youtube.com/watch?v=atS6iuvPB44&t=1088s) 边缘计算算力权衡与演讲尾声 在回答全场最后一个关于边缘设备算力瓶颈与实时性权衡的问题时，Vojta Ciml 坦承，在真实硬件盒子上运行复杂的视觉管线确实面临算力上限。由于现场仅依赖一台本地处理设备，“你不可能拥有全天下所有的算力资源”。但他指出，针对会议录制这一特定场景，算法有着独特的容错空间：现场视频是一个持续流动的长数据序列，系统根本不需要在每一帧画面上都维持百分之百无死角的精细分析。 在实际工程实现中，系统策略性地引入了跳帧机制（Frame Dropping）和低功耗计算模型。当算力吃紧或遭遇瞬时计算波峰时，系统允许丢弃部分中间帧，或者在特定帧暂时维持上一状态，只要能保证跟踪主体的连续性和核心决策正确即可。因此，在算法设计上，高运算速度、对低计算资源的强适应性以及成本效益比，是系统能否稳定落地的第一前提。 随着时间耗尽，Vojta Ciml 幽默地向大家致谢，打趣称自己成了“挡在大家与午餐之间的最后障碍”，并热情邀请参会者在随后的午餐与会展期间继续交流探讨。 - **边缘算力现实约束**：单一边缘设备算力有限，杜绝盲目堆叠高参数大模型，速度与低功耗优先。 - **跳帧容错策略**：利用会议画面的时序冗余特性，有选择地丢弃非关键帧，在有限算力下平衡实时跟踪与系统稳定性。 - **演讲圆满收尾**：Vojta Ciml 轻松结束全场问答，强调技术必须回归实用与成本可控的工程本质。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=atS6iuvPB44&t=4s) 闭上眼睛几秒钟，仔细倾听他的声音，然后试着判断他此时此刻究竟站在舞台的哪个位置——是左边、右边，还是正中央？ — Vojta Ciml > [00:04](https://www.youtube.com/watch?v=atS6iuvPB44&t=4s) 做个机器人摄像机能有多难？ — Vojta Ciml > [02:44](https://www.youtube.com/watch?v=atS6iuvPB44&t=164s) 声音会告诉你发言者在哪里 — Vojta Ciml > [04:38](https://www.youtube.com/watch?v=atS6iuvPB44&t=278s) 一个房间只要架设一台摄像机就足够追踪讲者 — Vojta Ciml > [04:38](https://www.youtube.com/watch?v=atS6iuvPB44&t=278s) Live show, one shot, one take — Vojta Ciml > [06:27](https://www.youtube.com/watch?v=atS6iuvPB44&t=387s) 追踪讲者就是让镜头锁定面部 > [06:27](https://www.youtube.com/watch?v=atS6iuvPB44&t=387s) 人脸识别能均等地识别人脸 > [09:58](https://www.youtube.com/watch?v=atS6iuvPB44&t=598s) 在构建软硬件系统时，不要躲在理想化的实验室象牙塔里，务必主动把系统推向复杂的现实世界极端工况（worst case scenarios）中经受捶打。 — Vojta Ciml > [15:02](https://www.youtube.com/watch?v=atS6iuvPB44&t=902s) 我们不再依赖讲者预先上传的文件，而是直接抓取投影屏幕的实时视频流。 — Vojta Ciml > [18:08](https://www.youtube.com/watch?v=atS6iuvPB44&t=1088s) 你不可能拥有全天下所有的算力资源 — Vojta Ciml > [18:08](https://www.youtube.com/watch?v=atS6iuvPB44&t=1088s) 挡在大家与午餐之间的最后障碍 — Vojta Ciml ## Mentioned Vojta Ciml, SlidesLive, NATE, Ai4, ICLR, ICML, 多模态融合 (Multimodal Fusion), 说话人日志 (Speaker Diarization), 唇动检测 (Lip Movement Detection), 单应性矩阵 (Homography), 姿态识别 (Pose Estimation), OCR, ASR, HDMI, SDI, XLR ## Video description Vojta Ciml - Founder & CEO, SlidesLive Event recording sounds like a solved problem. It isn't. I'll walk through how we evolved NATE, SlidesLive's autonomous recording system, from basic face detection to a multimodal system that fuses video and audio to do what a full production crew would: track speakers, switch cameras, monitor quality, and secure the recording. I'll cover the architecture, failed experiments, tuning process, and what 100,000 recorded talks taught us about building systems that work when there's no second take. [Source](https://www.youtube.com/watch?v=atS6iuvPB44)

Source quotes

[0:04] Vojta Ciml: 闭上眼睛几秒钟，仔细倾听他的声音，然后试着判断他此时此刻究竟站在舞台的哪个位置——是左边、右边，还是正中央？ [0:04] Vojta Ciml: 做个机器人摄像机能有多难？ [2:44] Vojta Ciml: 声音会告诉你发言者在哪里 [4:38] Vojta Ciml: 一个房间只要架设一台摄像机就足够追踪讲者 [4:38] Vojta Ciml: Live show, one shot, one take [6:27] 追踪讲者就是让镜头锁定面部 [6:27] 人脸识别能均等地识别人脸 [9:58] Vojta Ciml: 在构建软硬件系统时，不要躲在理想化的实验室象牙塔里，务必主动把系统推向复杂的现实世界极端工况（worst case scenarios）中经受捶打。 [15:02] Vojta Ciml: 我们不再依赖讲者预先上传的文件，而是直接抓取投影屏幕的实时视频流。 [18:08] Vojta Ciml: 你不可能拥有全天下所有的算力资源 [18:08] Vojta Ciml: 挡在大家与午餐之间的最后障碍

Source quotes

Vojta Ciml: Event recording sounds like a solved problem. It isn't. [0:04] Vojta Ciml: The moment attendees open their eyes, the illusion collapses: the voice does not emanate directly from the presenter’s mouth into their ears, but radiates throughout the auditorium through distributed loudspeakers. [1:25] Vojta Ciml: What if everything you need to record a session like this would fit in a single case? [1:25] Vojta Ciml: How hard can that be? [2:44] Vojta Ciml: The incoming audio feed is processed for speaker diarization to determine who is vocalizing, while computer vision actively tracks lip movement to verify where that sound originates on stage. [4:38] Vojta Ciml: When a single camera zooms in and follows an unexpected movement, it sacrifices all surrounding spatial context, leaving the system blind to the rest of the room. [6:27] Vojta Ciml: When a presenter fidgets, sways, or shifts weight behind a lectern, a naive face-tracking camera continuously hunts to keep the head centered, inducing visual seasickness in the viewer. [6:27] Vojta Ciml: Any tracking delay exceeding two video frames becomes visibly unnatural to the audience, breaking the illusion of an expert operator behind the lens. [9:07] Vojta Ciml: The moment a presenter turns around, face detection models lose all tracking features. SlidesLive consequently migrated from face-only detection to full-scale skeletal pose estimation. [9:58] Vojta Ciml: Rather than eliminating humans entirely, SlidesLive’s philosophy repositions the human operator from an exhausted, eight-hour camera-stander to a multi-room supervisor. [12:14] Vojta Ciml: For thousands of years, spoken human wisdom and insights shared during live gatherings have simply evaporated the moment sessions ended. [13:46] Vojta Ciml: System tuning relies on dual-deployment benchmarking, placing autonomous rigs side-by-side with professional human videographers.

Original transcript

[0:04] Hello everyone. I'll start with a question.  Please close your eyes for a second [0:13] and listen to me carefully and try to tell me  where do I stand on the stage? Is it left, right, [0:25] center? Can you tell just by listening to me? [0:30] So, you know, we all have this natural  instinct, this ability to recognize where [0:36] the sound is coming from, where the speaker  is standing, right? Can you open up your eyes? [0:42] Where I was standing, did you guess correctly?  What's what's wrong with this assumption? [0:53] Yes, exactly right. This was the key  problem because we have loud speakers, [1:01] so the sound is not coming just from my mouth,  but from the loud speakers, which is making it [1:05] almost impossible to accurately tell just  by listening to me where I am on the stage. [1:13] So, I'm Vojta, co-founder of SlidesLive,  and today I would like to share with you [1:18] five assumptions we have and how they fell  apart when we were designing robotic cameras. [1:25] To give you just um little bit  of info on what we are doing is [1:33] is we're recording lectures all around the globe.  We have recorded over 100,000 videos worldwide, [1:41] thousands of events, including this one. So,  you can see our robotic cameras in every track [1:47] on AI 4. And our vision is to record every single  presentation and store it for future generations. [1:57] And as you might know, video production has  been historically always very expensive, [2:04] labor-intensive, requires a lot of gear, strange  cables, and boxes. And that made us think, what if [2:14] everything you need to record a session  like this would fit in a single case? [2:21] All the cables, all the boxes, converters.  So, that's why we started working in designing [2:27] what we call the robotic cameras. You  can see one in the middle of this room. [2:31] And we said, you know, how hard can that  be? And of course, we've experienced [2:36] uh number of obstacles along the way. And today  I will share some of those lessons learned. [2:44] So, first uh what we already mentioned is  understanding the sound, my voice. And the [2:50] assumption was pretty easy. Sound tells you where  the speaker is. When you have a panel discussion, [2:56] when you have five people sitting, uh  you really cannot just track their faces. [3:01] You really need to know who is actually  talking, so you can zoom the camera onto them. [3:07] And uh the loudspeakers is one problem, but  there are myriads of other problems um because [3:13] stage like this, surprisingly, is not a sealed  Hollywood studio where you get a perfect sound. [3:20] Unfortunately, it can often it can often  um be more like a very noisy environment. [3:28] This was literally taken you can hear probably  nothing here, which is correct. This is um [3:33] a real footage, a real microphone from the camera,  and some stages are right next to the expo hall, [3:40] so you get a lot of noise in there. You have a  live music band playing right next to it. So, this [3:46] is not really usable for our purpose. That's why  we turned to combine multiple sources together. [3:54] So, we use the voice just for speaker  diarization, just to know who is who, [4:00] but we use the the visual, the computer vision  to actively detect your mouth lips moving, [4:08] which we see in this video. So, and we combine  all these sources together and we create a memory [4:17] for each speaker on stage. So, we store his  visuals, his face. We store the camera position [4:24] and we store his voice fingerprint. And by  combining all these sources together, we are [4:30] able to very accurately tell who is speaking  on stage when you have a panel discussion. [4:38] When I'm speaking about cameras, um of course,  the first assumption we had was if you place [4:44] just one camera in the room, you can record the  stage. That's all you need, you know. Of course, [4:49] you want to save some cost, so one camera should  be technically enough to track the speaker. [4:55] Unfortunately, multiple uh things can  happen such as an MC will come into play [5:01] or a second speaker. Or we even had cases  where speaker just went off stage, just uh [5:10] you know, in the back of the room and started  talking, you know, these motivational speakers [5:15] are sometimes very difficult to predict. And at  that moment, camera loses the track and uh you [5:21] have no context. You have no idea what's happening  in the room. That's why we added a second camera [5:27] to have a vision of the entire room and to be  better at predicting what's going to happen. [5:34] Of course, then you need to do um alignment  of these. We have seen that just at the last [5:40] session, too. So, you need to align two frames  together and you have two different devices, [5:47] different resolutions, and different angles.  Oh, you need to use, of course, the homography, [5:54] and you have the same thing, so you detect some  key points, and then you align them. But again, [6:00] it's a live video, you know, you have just one  shot, you have one take. Uh so, you need to again [6:07] cross-check everything to be to be sure. So,  sometimes the alignment works perfectly, sometimes [6:13] it will fail. So, you want to avoid that failure,  otherwise the camera all of a sudden would jump [6:19] and zoom maybe to to the drapes here, which  would be, of course, terrible during a live show. [6:27] Another thing is, uh and that's my favorite, is  how to move the camera. How to actually follow [6:34] the speaker when I'm walking on a stage.  And again, the assumption was pretty simple. [6:41] Track speaker means following my face, uh which  is pretty straightforward. And let's first take [6:49] a look at uh humans. Honestly, humans are also  not uh the best in this job, because being at a [6:56] conference means you're standing behind the camera  for 8 hours straight, you have just few breaks, [7:03] and humans are not really good at paying  attention, you know, for 8 hours straight. [7:09] So, let's take a look on some of the common  mistakes that we see with the human operators. [7:18] So, we have speakers sometimes walking  out of the frame, which is, of course, [7:23] something we don't want. We have sometimes issues  with the focus. So, these are common things, and [7:31] even the best videographers, they sometimes  just, you know, it's it's one thought, [7:36] and you lose them. So, we want to avoid  this when we have a robotic camera. [7:41] We want to have a a perfect  smooth shot. loving what's going [7:48] And out of frame, it's just a split of a second  and you're there. You have a mistake in a live [7:54] show. So again, that got us thinking.  Um if you are a perfect videographer, [8:01] well rested, you pay close attention, and you  have that natural nice smooth panning and tilt, [8:09] how we can mimic that with a robot?  There are some issues such as um [8:15] you might be just wobbling around the lectern,  and if you would be just tracking the face, [8:21] the camera would cause you to be seasick  because it would non-stop change the position. [8:26] So we are creating these heat maps and deciding  whether the speaker has an intention to stay on [8:32] the ground and it's just wobbling around, or where  he plans to pace. And do I plan to pace quickly, [8:40] or will I do some unexpected move, will I go left  or right? So all these are things that you have to [8:46] keep in mind when you're trying to have a smooth  uh nice panning. Of course, uh a latency has to be [8:54] very fast. Uh you know, if it's just a more  than two frames uh of the of the video, [9:00] then people will notice that uh there's a delay in  in your reaction and will will will know. When I'm [9:07] saying face tracking, um another assumption  was that if we use models to to track faces, [9:16] the model will recognize every face equally well.  Unfortunately, that turned out uh to not be true, [9:24] and we have seen some minors  not being detected correctly. [9:29] And then people sometimes also do things like they  turn their back to you, you know, they they want [9:35] to show some slides, and all of a sudden you don't  see even their face, you see just their back. So, [9:41] we switched from just using purely face to using  a full scale. And also, we wanted to be sure that [9:49] just this works in any circumstance, low light,  you know, different types of hair and so on. [9:58] Which brings me uh to my last assumption  that I would like to present here today, [10:04] which is if you if you take everything  I just said and you just make it right, [10:10] then the job is done, you know. We just going  to figure out these few little details and then [10:15] we can roll out thousands of these cameras and  finally fulfill our mission. Unfortunately, as you [10:22] might know and as we have seen here in previous  sessions, real world is full of surprises. So, [10:29] no matter how well you design your systems,  no matter what edge cases you do anticipate, [10:36] there's always going to be something that uh you  do not anticipate. Like in this example where [10:44] um we had we had a session going on and in  the middle of the session, power cuts out. [10:51] Ooh, fortunately we had batteries, so cameras  were still running, but how do you do the [10:57] tracking and everything when the power is  out and you have a pitch dark pitch black? [11:02] So, you never really know what to expect uh on  when you're uh on a convention or on a conference. [11:10] Which brings me to my thought that um no matter  how well designed these systems will be, for [11:18] foreseeable future we'll still need humans in the  loop, especially in these real world scenarios. [11:26] Because again, there are hosts of issues that  can happen, you know. Um these electronic boxes, [11:33] they do fail, you know, cables snap. So, even if  you have a lot of backups, and you know, we are [11:40] all built about around backups, there's always  something that needs to be troubleshoot it on [11:46] site. Um, so that's But we we have a strong belief  that the mundane work, you know, those 8 hours [11:53] standing behind the camera is something that has  to be replaced. And humans then can focus on those [12:00] technical challenges, those complicated problems  where the the model or the system is lost and [12:07] is not sure how to recover from those, and  human can can can step in and troubleshoot. [12:14] Which brings me to words my final thoughts of this  talk. Um, I would like to encourage everyone to [12:22] to test their systems in the real world scenarios,  to really seek those difficult situations, [12:31] to actively go out there and and ask for the worst  case scenarios when you're building uh systems. [12:39] And the reason is you also, um, initially were  confused when you were using just your ears, [12:46] you know, where I am in the stage, that was the  start of my talk. You needed to plug in your eyes. [12:52] So, as you see, we need to connect the dots.  We need to be able to use multiple approaches, [13:00] cross-check everything, and validate  these approaches all together, [13:05] so we have we have a final system that is  that is reliable. And uh one final thought, [13:12] you guys, I'm sure uh some of you are  hosting events of your own or meetups, [13:18] and for thousands of years, once the talk is  over, that knowledge is lost. And we are on a a to [13:26] to preserve this knowledge for our lens for future  generations. So, we could use your help. Please [13:32] do record your sessions and talks and conferences.  So, thank you very much for attention and [13:39] I'm opening up Q&A. I think we can take few  questions before you go on lunch. Yes? Yes. [13:47] There are a lot of problems that on the these kind  of applications. Okay. But then, how do you design [13:55] to solve all of them? And what steps did you take  because you cannot solve them all together as you [14:01] or else you strategize it. What are the strategies  or what are you working with? So, how did you [14:06] tackle these things? If these are not problems  only of one company, but could be a find of all [14:12] the six that you're like really at. Mhm. Yes.  Um so, of course we're plugged into, you know, [14:20] computer vision community. Um the advantage is  we're attending a lot of these events. So, we get [14:26] access to, you know, all the latest features and  we can talk to people on conferences like these or [14:31] ICLR or ICML. And also we we were testing a lot.  So, we have events running all the time and we're [14:41] bringing the cameras along the human operators.  So, we had a robot and a human at the same time [14:46] and we just tested non-stop tested and we have  simulations. So, we have people working on all [14:53] these problems one step at a time and we just  continuously improve. That's That's our approach. [15:02] We had a pretty simple use case, which is [15:07] slides on the screen. So, historically people  usually show something else on the screen and they [15:13] then send They send you a PowerPoint and  they either show a demo or the PowerPoint [15:17] they're showing is a different one. So, when you  have these abilities to download a slide deck, [15:23] it's usually a different one. So, we decided,  "Okay, if we're not going to use the slides [15:28] that you can download, but we'll capture the  screen, we capture the feed from the screen, [15:32] and we will recognize the features. We also do the  OCR, and we combine this into with the transcript, [15:40] and then we have a better, you know, AI summaries  and close captioning. So, you can plug the slides. [15:46] The slides are valuable source of information, and  we we preserve them as they are, but we do that by [15:51] recognizing the the the screen itself. Watermarks.  Oh, watermarks. Um We You can put watermarks in [16:00] the video. [laughter] So, some customers they ask  us to place the watermarks. It's It's up to them, [16:07] because we are just doing this as a service.  Um and some events they decide, "Yeah, we'll [16:12] we'll put a watermark there." Some some sponsors  they want to have watermark in the video, so it's [16:18] it's always up to the the content owner. We do not  own the content. Just that's a full disclosure. [16:24] We just uh record the content, we store it,  we do the the computer vision stuff, we make [16:29] sure it's their living, but we don't decide  who is having a watermark or not. Thank you. [16:36] Yes. Uh when you guys are taking on jobs,  uh are there any specific requirements that [16:44] you're putting in a tech rider uh that  would differ from a normal tech rider? [16:50] Yeah, that's a good question. Uh so, we were  trying to make this really simple and plug into [16:56] existing workflows. Something that every AV  person is used to. So, you know, just we take [17:02] a standard uh XLR feed to get the audio, we  take HDMI or SDI feed to grab the slides, [17:10] and we have a power cord and Ethernet drop to  grab internet. But, otherwise, it it's basically [17:15] seamlessly integrates into existing workflow. The  only difference is instead of a videographer with [17:21] a tripod and a camera comes a little box where you  unpack our robotic stuff and you're ready to go. [17:28] Yeah. And and an advantage is that you  don't need an extra 6-ft table full of [17:34] a strange gear that you're setting up for half  a day and striking for another half a day. [17:39] Instead, it it might take 30 minutes to set things  up with the box itself. So, really making it easy. [17:46] All right. One last question I get? [18:08] And that's a good question. It For  our case, it works because we have um [18:16] we have a lot of data, continuous data. So,  as I said, we sometimes we just drop a frame [18:21] and sometimes we we just leave it there. And we  don't need like a continuous stream of correct [18:29] analysis. So, for for our purposes, it works. But  yes, and also the computing power is a concern. [18:36] We have just one box. You don't have all  the computing power in the world. So, [18:39] speed and relatively affordable low  computing resources is a necessity. [18:47] Well, thank you. Thank All right. Well,  we are out of time. Thank you for standing [18:51] between me and lunch. Enjoy your lunch  and hopefully I'll see you at some point.