CreativAI Cuts Video Tokens to 7,000 as 7B Model Outperforms GPT-4V

Towards Scalable and Structured Understanding in Video LLMs With KAUST & CreativAI — Ai4

Ai4

Pruning redundant spatial and temporal frames enables compact vision models to outperform GPT-4V on long-form video while avoiding the token explosion that cripples multimodal reasoning.

AI-assisted summary of Towards Scalable and Structured Understanding in Video LLMs With KAUST & CreativAI — original source. Check the original for full context.

Source published: 2025-08-26

Detailed notes

## Background Published on YouTube by the Ai4 channel on August 26, 2025, this presentation features Mohamed Elhoseiny, Associate Professor at KAUST and CEO of CreativAI. The video abstract outlines recent breakthroughs in scalable and structured Vision-Language Models (Video LLMs), spanning instruction-tuned models like MiniGPT-4, long-video frameworks like Goldfish and LongVU, and structured evaluation benchmarks.

## TL;DR Mohamed Elhoseiny traces the trajectory of vision-language systems from his 2013 foundational zero-shot learning formulation to modern instruction-tuned models like MiniGPT-4 and MiniGPT-Video. To solve the token explosion problem in long-form visual narratives, he details the Goldfish retrieval-augmented framework and the LongVU spatiotemporal compression architecture, which retains spatial and semantic fidelity within a compact token budget. The talk concludes with CreativAI, an enterprise video search platform that leverages X-Rows for temporal event localization and X-Columns for multi-dimensional attribute extraction, enabling programmatic querying across vast video archives.

## Atmosphere & style The presentation carries the polished, authoritative cadence of a high-level academic keynote blending foundational AI history with cutting-edge engineering. Elhoseiny speaks with clear, methodical enthusiasm, systematically connecting past theoretical contributions to practical open-source milestones and commercial deployments. The talk feels intellectually rigorous yet accessible, emphasizing pragmatic architectural efficiency, benchmark victories, and real-world developer applications.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=lP-Cpq16W28&t=4s) Foundational 2013 research on zero-shot species classification mapped text descriptions to visual classifiers via joint embeddings, serving as an early linear precursor to CLIP. - [03:44](https://www.youtube.com/watch?v=lP-Cpq16W28&t=224s) Cross-modal retrieval alone cannot perform fine-grained visual reasoning across thousands of species because it measures surface embedding distances rather than analyzing subtle morphological attributes. - [04:50](https://www.youtube.com/watch?v=lP-Cpq16W28&t=290s) MiniGPT-4 demonstrated that connecting a pre-trained visual encoder to an LLM via a single linear projection enables zero-shot conversational reasoning and creative instruction-following. - [06:46](https://www.youtube.com/watch?v=lP-Cpq16W28&t=406s) A small, curated Stage-2 alignment dataset of just 5,000 high-quality conversational pairs effectively eliminates hallucinations and linguistic repetition stemming from noisy pre-training. - [08:00](https://www.youtube.com/watch?v=lP-Cpq16W28&t=480s) The MiniGPT paradigm scaled across modalities to support object grounding (MiniGPT-v2), 3D spatial understanding, radiology, and temporal video analysis with LoRA fine-tuning. - [09:53](https://www.youtube.com/watch?v=lP-Cpq16W28&t=593s) Goldfish enables hour-long video comprehension by chunking long video streams, generating segment descriptions, and retrieving top-matching segments for LLM context without accuracy degradation. - [11:38](https://www.youtube.com/watch?v=lP-Cpq16W28&t=698s) Naive frame-by-frame video processing creates an unsustainable token explosion (over 170k tokens for 20 minutes), requiring spatiotemporal compression to scale effectively. - [12:40](https://www.youtube.com/watch?v=lP-Cpq16W28&t=760s) LongVU combines SigLIP semantics, DINO spatial features, query-aware resolution uplifting, and static token suppression to achieve state-of-the-art long-form reasoning within a 7,000-token budget. - [15:02](https://www.youtube.com/watch?v=lP-Cpq16W28&t=902s) CreativAI applies long-video research to enterprise video intelligence, indexing 24-hour video backlogs in under 4 hours via automated knowledge graph construction. - [16:20](https://www.youtube.com/watch?v=lP-Cpq16W28&t=980s) Structuring unstructured video is achieved using X-Rows for temporal scene localization and X-Columns for multi-dimensional attribute extraction into queryable tabular data. - [17:49](https://www.youtube.com/watch?v=lP-Cpq16W28&t=1069s) Developer APIs allow complex semantic queries over video archives to extract entity presence, multi-character relationships, and high-level narrative outcomes automatically.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=lP-Cpq16W28&t=4s) Zero-Shot Learning and the Origins of Vision-Language Modeling

Reflecting on a 15-year career at the intersection of computer vision and natural language processing, Mohamed Elhoseiny traces his trajectory from academic research across Silicon Valley—collaborating with Yann LeCun at Facebook AI Research (FAIR), Stanford University, and Adobe Research—to his current faculty role at KAUST and his entrepreneurial venture, CreativAI. The long arc of this research has centered on building systems capable of scalable, structured comprehension of visual media.

His passion for vision-language integration began with a 2013 paper focused on zero-shot species discovery. To illustrate the core intuition, he notes that if someone is asked to spot an unfamiliar bird like a parakeet or brambling, they might struggle without prior visual exposure. However, if given a concise natural language description—explaining that it is a small bird with a short orange beak, dark plumage on top, and white plumage underneath—the human brain immediately constructs a mental prototype, making visual identification straightforward among distractors.

``` +-----------------------------------+ | Text Description of Unseen Class | +-----------------+-----------------+ | v [ Text Feature Vector: T ] | v [ Linear Transform / NN: W ] | v +--------------------------------+ | Predicted Classifier: W * T | +----------------+---------------+ | +----------------------------+----------------------------+ | | v v [ Matching Image Feature ] [ Non-Matching Image Feature ] (Pulled closer in space) (Pushed farther apart) ```

To enable machines to mimic this capability, Elhoseiny collected detailed linguistic descriptions for over 200 bird species and formulated an algorithm to project text and visual data into a shared joint embedding space. The model pulled matching image-text pairs belonging to the same class closer together while pushing non-matching pairs apart. By multiplying the text representation of an unseen class $T$ by a learned transformation matrix $W$, the system could directly generate a visual classifier for novel classes without training images. OpenAI’s CLIP paper later cited this 2013 study as the foundational precursor to modern zero-shot cross-modal learning, effectively scaling up this exact formulation with massive web datasets and deep neural encoders.

- Developed an early zero-shot learning formulation in 2013 enabling classification of novel visual categories purely from text descriptions. - Built a joint embedding space using text-image pairing across 200+ bird species, cited by OpenAI as the conceptual predecessor to CLIP. - Characterizes early work as a "linear version of CLIP" created when multimodal modeling received little mainstream attention.

---

[03:44](https://www.youtube.com/watch?v=lP-Cpq16W28&t=224s) From Cross-Modal Retrieval to Instructable Vision Models: VisualGPT and MiniGPT-4

While CLIP revolutionized cross-modal retrieval by matching text prompts to relevant images, Elhoseiny points out its fundamental limitation: retrieval is not reasoning. In fine-grained benchmark evaluations spanning more than 10,000 species, retrieval-based models frequently stumble. Differentiating closely related species often hinges on subtle morphological features, such as a slightly curved crest feather or a specific beak tint, which demand an explicit visual reasoning engine rather than surface-level embedding similarity.

``` Cross-Modal Retrieval (CLIP) Instructable Vision-LLMs (MiniGPT-4) +-----------------------------+ +-----------------------------------+ | Text Query <-> Visual Match | | Visual Token Input -> Reasoning | | (Embedding Distance Only) | | Engine -> Multi-turn Instruction | | Fails on subtle attributes | | Following, Contextual Responses | +-----------------------------+ +-----------------------------------+ ```

This motivated the transition toward instructable Large Vision-Language Models (Video/Image LLMs). In 2021, Elhoseiny’s group introduced VisualGPT, an early framework that interfaced a language model with visual inputs for image captioning, though it was largely constrained to static prompts like "provide a short description." By April 2023, his lab open-sourced MiniGPT-4, emerging alongside LLaVA ("Visual Instruction Tuning") and anticipating commercial multimodal releases from OpenAI.

MiniGPT-4 quickly went viral, accumulating over 25,000 GitHub stars and becoming one of the most widely referenced open-source multimodal repositories. The model demonstrated conversational reasoning over visual inputs: explaining the anomaly of a cactus growing in the middle of a frozen lake, generating marketing ad copy when shown an image of a coffee mug, or analyzing a photo of a restaurant seafood dish to produce a step-by-step recipe listing exact ingredients ("four lobsters, two tablespoons of olive oil").

- Fine-grained recognition across 10,000+ species requires structured visual reasoning beyond standard cross-modal metric retrieval. - VisualGPT (2021) paired language decoders with visual features for captioning prior to general instruction-tuning paradigms. - MiniGPT-4 (April 2023) gained more than 25,000 GitHub stars by demonstrating zero-shot visual reasoning, creative composition, and instruction following.

---

[06:46](https://www.youtube.com/watch?v=lP-Cpq16W28&t=406s) MiniGPT Architecture, Alignment, and the Expansion to Video

The technical architecture of MiniGPT-4 was engineered for parameter efficiency. The framework couples a pre-trained visual encoder aligned with language representations to an LLM backbone using a single trainable linear projection layer. This design rested on the hypothesis that aligning high-quality visual features with linguistic token spaces requires minimal "learning energy."

``` +------------------+ +--------------------+ +------------------+ | Input Image | ---> | Visual Encoder | ---> | Linear Project | +------------------+ +--------------------+ +--------+---------+ | v +------------------+ +--------------------+ +------------------+ | User Instruction | ---> | Large Language | <--- | Visual Tokens | | (e.g., Recipe?) | | Model (LLM) | | (Low-Energy Map) | +------------------+ +---------+----------+ +------------------+ | v [ Coherent Response ] ```

Initial pre-training on large, noisy web-scraped datasets produced an alignment where roughly 30% of model generations suffered from linguistic glitches, hallucinations, or repetitive loops. To resolve this, the team curated a clean instruction-tuning alignment set comprising just 5,000 high-quality image-text conversational pairs. This Stage-2 post-training resolved the output glitches, elevating conversational fluency and grounding.

Following MiniGPT-4, the lab expanded the architecture into an entire model ecosystem: - **MiniGPT-v2**: Integrated fine-grained visual grounding, allowing the model to generate descriptive text while predicting bounding box coordinates for identified objects (e.g., localizing a man in a suit and the plastic cups he is holding). - **MiniGPT-3D & MiniGPT for Radiology**: Tailored the architecture for volumetric, spatial, and clinical imaging data. - **MiniGPT-Video**: Extended the framework from static frames to temporal sequences. The system splits video into distinct frames, integrates subtitle transcriptions, and replaces frozen LLM weights with LoRA (Low-Rank Adaptation) fine-tuning to perform complex tasks such as composing a poem about a blooming flower sequence or summarizing thematic messaging in video essays.

---

[09:53](https://www.youtube.com/watch?v=lP-Cpq16W28&t=593s) Scaling Video Understanding to Hour-Long Content with Goldfish

While initial video-language models operated effectively on short clips of a few minutes, real-world visual narratives require parsing long-form media, such as full-length television episodes. Elhoseiny highlights a scenario: analyzing an episode of the sitcom *Friends* to answer fine-grained situational questions, such as "What does Phoebe show Monica while they are talking?"

``` +------------------------------------------------------------------------+ | Full-Length Video | +---+--------------------+--------------------+--------------------+-----+ | Chunk 1 | Chunk 2 | Chunk 3 | v v v v +------------------------------------------------------------------------+ | Short Video Descriptor Module | +-----------------------------------+------------------------------------+ | [ Segment Text Summaries ] | v +------------------+ +-------------------+ +-------------------+ | User Question | ---> | Retrieval Module | ---> | Top Matching | | | | (Relevance Match) | | Video Segments | +------------------+ +-------------------+ +---------+---------+ | v +-------------------+ | LLM Context & | | Final Answer | +-------------------+ ```

To resolve this challenge without exceeding context limits, the team introduced **Goldfish**, a retrieval-augmented 7-billion parameter video LLM capable of processing up to an hour of video content. The architecture utilizes a segment-chunking pipeline: 1. Long video streams are partitioned into uniform temporal chunks. 2. Each chunk is processed through a short-video descriptor to generate localized semantic descriptions. 3. When a user submits a query, a dedicated cross-modal retrieval module calculates relevance scores between the prompt and the video segments. 4. The top-ranked segments are aggregated and injected directly into the LLM context window to generate the final response.

Goldfish achieved state-of-the-art benchmark results across both short and long video evaluation suites. Unlike standard architectures whose accuracy degrades as video duration lengthens, Goldfish demonstrated robust, consistent reasoning accuracy across variable narrative lengths.

---

[11:38](https://www.youtube.com/watch?v=lP-Cpq16W28&t=698s) LongVU: Overcoming Token Explosion via Spatiotemporal Compression

In a paper presented at ICML, Elhoseiny’s lab introduced **LongVU**, addressing a core challenge of video processing: temporal and spatial redundancy. Treating a video as a naive stack of individual image frames causes an unsustainable "token explosion": - **1 individual frame**: ~144 visual tokens. - **20-minute video**: ~172,000 visual tokens. - **1 day of video**: ~12.3 million tokens (comparable to scaling across streaming platforms like Netflix, which host over 100,000 hours of content).

``` +-------------------------------------------------------------------+ | Raw Input Video | +---------------------------------+---------------------------------+ | v +-------------------------------------------------------------------+ | 1. Diversity-Aware Keyframe Selection (Reduces Frame Redundancy) | +---------------------------------+---------------------------------+ | v +-------------------------------------------------------------------+ | 2. Dual Feature Encoding: SigLIP (Semantics) + DINO (Spatial) | +---------------------------------+---------------------------------+ | v +-------------------------------------------------------------------+ | 3. Query-Aware Uplifting: High-Res (H_W, W_W) vs Low-Res (H_L, W_L)| +---------------------------------+---------------------------------+ | v +-------------------------------------------------------------------+ | 4. Spatiotemporal Token Compression (Zero-out static (x,y) tokens)| +---------------------------------+---------------------------------+ | v +-------------------------------------------------------------------+ | Compact ~7,000 Token Representation -> 7B LLM (Beats GPT-4V) | +-------------------------------------------------------------------+ ```

To compress visual tokens without discarding critical narrative details, LongVU implements four technical stages: 1. **Diversity-Aware Keyframe Selection**: Selects high-information keyframes while minimizing visual similarity between neighboring samples. 2. **Dual Semantic and Spatial Feature Extraction**: Combines SigLIP features (for broad semantic representations) with DINO representations (to preserve fine spatial structures). 3. **Query-Aware Resolution Uplifting**: Allocates visual fidelity based on user queries, applying a high-resolution representation ($H_W \times W_W$) to query-relevant segments and a low-resolution representation ($H_L \times W_L$) to background segments. 4. **Spatiotemporal Token Compression**: Monitors spatial coordinate tokens $(x, y)$ over time; if a coordinate region remains static across frames, redundant background tokens are zeroed out, retaining only dynamic, moving visual elements.

Operating on just ~7,000 tokens, the 7B-parameter LongVU model outperformed GPT-4V and larger proprietary vision models across standard multimodal video benchmarks.

- Naive frame sampling scales unsustainably, reaching 172k tokens for 20 minutes and 12.3M tokens for 24 hours of video. - Combines SigLIP and DINO encoders to capture both semantic context and fine-grained spatial structure. - Spatiotemporal token reduction discards static background coordinates while preserving moving objects. - Achieved state-of-the-art long-form video performance with a 7B parameter model operating within a compact 7,000-token budget.

---

[15:02](https://www.youtube.com/watch?v=lP-Cpq16W28&t=902s) CreativAI: Enterprise Video Query Engine, X-Rows, and X-Columns

In the final portion of the presentation, Elhoseiny details how these long-video research breakthroughs formed the foundation of **CreativAI**, an enterprise query engine designed as a "Perplexity-like experience for video data." The platform enables users to query thousands of hours of unstructured video using natural language, ingesting and indexing massive video backlogs in real time (processing a 24-hour video block in under 4 hours) through automated knowledge graph construction, no-code workflows, and multimodal agents.

``` +-------------------------------------------------------------------+ | CreativAI Query Engine | +-------------------------------------------------------------------+ | Natural Language Query: "Analyze food consumption in Friends" | +-------------------------------------------------------------------+ | +------------------------+------------------------+ | | v v +---------------------------------+ +-----------------------------+ | X-ROWS (Temporal Localization) | | X-COLUMNS (Feature Analytics)| +---------------------------------+ +-----------------------------+ | Locate exact video scenes: | | Extract structured fields: | | - Season 1, Ep 4 eating scene | | - Cuisine Type (Italian) | | - Season 3, Ep 12 dinner scene | | - Healthy vs. Unhealthy | | - Timestamped segment clips | | - Present Characters | +---------------------------------+ +-----------------------------+ | | +------------------------+------------------------+ | v +-------------------------------------------------------------------+ | Structured Analytical Dashboard / Cross-Character Cuisine Plots | +-------------------------------------------------------------------+ ```

The system organizes unstructured visual data into a structured schema using two core concepts: - **X-Rows (Temporal Localization)**: Identifies and isolates discrete video segments matching a semantic query (e.g., retrieving all scenes of characters eating across Seasons 1 and 3 of *Friends*, or all pitch segments featuring Mark Cuban on *Shark Tank*). - **X-Columns (Structured Attribute Extraction)**: Extracts analytical dimensions across those retrieved segments (e.g., classifying cuisine types, assessing whether a meal is healthy, identifying co-occurring characters, determining founder gender, or logging investor deal outcomes).

These dimensions allow the platform to automatically compile structured tables and generate comparative analytics—such as plotting dietary habits across different characters or tracking investment rates across business categories—all accessible via a conversational interface and programmatic APIs.

- Processes thousands of hours of enterprise video, completing a 24-hour video backlog in under 4 hours. - Uses X-Rows for temporal event localization and X-Columns for multi-dimensional attribute extraction and tabular data generation. - Supports knowledge graph indexing, no-code analytical agents, and API integrations for automated video intelligence.

[17:49](https://www.youtube.com/watch?v=lP-Cpq16W28&t=1069s) Practical Applications: Structured Video Querying via API and Conclusion

Mohamed Elhoseiny concludes the presentation by demonstrating the practical accessibility and real-world utility of their video understanding frameworks. The system offers API support designed specifically for developers seeking to parse and extract structured metadata and semantic segments from large-scale video libraries. To illustrate its capability, he presents a concrete use case centered on analyzing a collection of *Shark Tank* episodes.

Using this system, a developer can automatically query and extract precise video segments based on complex, multi-attribute conditions. For example, a user can instruct the model to locate every pitch where Mark Cuban is present, identify the gender of the presenting founder or CEO, determine whether the startup successfully secured an investment deal, and list any other "Sharks" who partnered in the agreement. By turning raw, unstructured visual footage into queryable, structured knowledge, the platform enables downstream analytics and intelligent video search before Elhoseiny wraps up his talk and thanks the audience.

- **Developer API Support**: Provides programmatic access for extracting fine-grained visual events, entity interactions, and semantic segments across video datasets. - **Example Query Capabilities (*Shark Tank* use case)**: - Identify presence of specific individuals (e.g., Mark Cuban). - Extract entity attributes (e.g., gender of the pitching CEO). - Classify high-level narrative outcomes (e.g., whether a deal was closed). - Detect multi-entity relationships (e.g., identifying co-investing Sharks on the panel).

## Notable quotes > [00:04](https://www.youtube.com/watch?v=lP-Cpq16W28&t=4s) linear version of CLIP — Mohamed Elhoseiny

> [03:44](https://www.youtube.com/watch?v=lP-Cpq16W28&t=224s) provide a short description — Mohamed Elhoseiny

> [03:44](https://www.youtube.com/watch?v=lP-Cpq16W28&t=224s) Visual Instruction Tuning — Mohamed Elhoseiny

> [03:44](https://www.youtube.com/watch?v=lP-Cpq16W28&t=224s) four lobsters, two tablespoons of olive oil — Mohamed Elhoseiny

> [06:46](https://www.youtube.com/watch?v=lP-Cpq16W28&t=406s) learning energy — Mohamed Elhoseiny

> [09:53](https://www.youtube.com/watch?v=lP-Cpq16W28&t=593s) What does Phoebe show Monica while they are talking? — Mohamed Elhoseiny

> [15:02](https://www.youtube.com/watch?v=lP-Cpq16W28&t=902s) Perplexity-like experience for video data — Mohamed Elhoseiny

> [15:02](https://www.youtube.com/watch?v=lP-Cpq16W28&t=902s) Analyze food consumption in Friends — Mohamed Elhoseiny

## Mentioned Mohamed Elhoseiny, Yann LeCun, Mark Cuban, Facebook AI Research (FAIR), Stanford University, Adobe Research, KAUST, CreativAI, OpenAI, CLIP, VisualGPT, MiniGPT-4, LLaVA, MiniGPT-v2, MiniGPT-3D, MiniGPT-Video, Goldfish, LongVU, SigLIP, DINO, GPT-4V, Netflix, StoryGPT-V, Vgent, InfiniBench, SpookyBench, Friends, Shark Tank, ICML, Ai4

## Video description Mohamed Elhoseiny, Associate Professor & CEO, KAUST & CreativAI

Abstracts

The rise of Vision-Language Models (VLMs) has opened new frontiers in video understanding, yet scaling these systems to handle long, complex, and structured visual narratives remains a fundamental challenge. In this talk, we explore a suite of recent advances aimed at achieving scalable and structured video comprehension through Large Vision-Language Models (Video LLMs). We begin by examining instructable models like MiniGPT-4 (image-based), MiniGPT-4-v2(image-based), MiniGPT-3D, and its video extension, which leverage multimodal instruction tuning for diverse video tasks. We then introduce Long Video LLMs—including the Goldfish and LongVU models—that tackle the token explosion problem with retrieval-augmented generation and spatiotemporal compression. Further, we address structured understanding via StoryGPT-V and Vgent, which model character consistency and entity-based graph reasoning. To rigorously evaluate progress, we present InfiniBench and SpookyBench, two novel benchmarks designed to probe long-form comprehension and temporal perception in state-of-the-art models. Finally, we extend the discussion to include multimodal capabilities in multilingual, emotional, and action-driven contexts, as well as exploratory work on bridging vision and brain signals.

[Source](https://www.youtube.com/watch?v=lP-Cpq16W28)

中文简报

CreativAI将视频Token削减至7000，7B模型表现超越GPT-4V 剪枝时空冗余帧使紧凑型视觉模型在长视频任务中超越GPT-4V，同时避免了削弱多模态推理能力的Token爆炸问题。 Mohamed Elhoseiny梳理了视觉-语言系统的发展轨迹，从其2013年奠基性的零样本学习构想，一路延伸至MiniGPT-4和MiniGPT-Video等现代指令微调模型。为解决长篇视觉叙事中的Token爆炸难题，他详细介绍了Goldfish检索增强框架以及LongVU时空压缩架构，后者在紧凑的Token预算内保留了空间与语义保真度。演讲最后介绍了企业级视频搜索平台CreativAI，该平台利用X-Rows进行时序事件定位，并利用X-Columns进行多维属性提取，从而支持对庞大视频档案的编程式查询。 Mohamed Elhoseiny 回顾了其 2013 年提出的零样本物种分类研究，该工作通过联合嵌入将文本描述映射到视觉分类器，成为 OpenAI 研发 CLIP 的早期线性先驱架构。 (0:04) MiniGPT-4 证实仅需一个线性投影层连接预训练视觉编码器与大语言模型即可实现指令遵循，并通过包含 5,000 条高质量对话的第二阶段对齐数据集彻底解决了幻觉与重复问题。 (6:46) 针对长视频理解中的上下文与算力瓶颈，Goldfish 采用分块检索增强生成技术支持长达 1 小时的视频推理，而 LongVU 则结合 SigLIP、DINO 和时空 token 压缩技术，在 7,000 个 token 预算内超越了 GPT-4V。 (9:53) 企业级视频查询引擎 CreativAI 能够提供类似 Perplexity 的视频检索体验，利用 X-Rows 识别特定时间段场景，并通过 X-Columns 提取结构化属性以生成多维分析报表。 (15:02)

Reading summary

Multimodal AI must transition from simple cross-modal metric retrieval to structured visual reasoning to make long-form video computationally tractable and queryable, according to computer vision researcher Mohamed Elhoseiny. From Metric Retrieval to Conversational Visual Reasoning (3:44) Multimodal research originated in zero-shot classification, exemplified by Elhoseiny’s 2013 paper where text descriptions of unseen bird species were projected into a shared embedding space using a linear transform matrix—a formulation later cited by OpenAI as a precursor to CLIP. However, Elhoseiny argues that cross-modal metric retrieval is fundamentally distinct from visual reasoning. When differentiating among more than 10,000 species, retrieval models frequently fail on fine-grained morphological details, such as beak tints or crest curvatures, requiring an explicit reasoning engine. To move beyond static captioning models like VisualGPT (2021), Elhoseiny’s lab released MiniGPT-4 in April 2023. The architecture connects a pre-trained visual encoder to a large language model backbone through a single trainable linear projection layer. While initial training on web datasets yielded linguistic hallucinations in roughly 30% of outputs, introducing a clean alignment dataset of just 5,000 instruction-tuning pairs resolved generation glitches. The framework has since expanded into grounded spatial detection (MiniGPT-v2), domain-specific variants for 3D and radiology, and temporal video comprehension using Low-Rank Adaptation (MiniGPT-Video). Scaling Video Contexts via Spatiotemporal Token Compression (11:38) Processing narrative video introduces severe context bottlenecks. A naive frame-stacking approach triggers an unsustainable token explosion: an individual frame requires roughly 144 tokens, a 20-minute video consumes 172,000 tokens, and a 24-hour sequence escalates to 12.3 million tokens. Standard video architectures suffer performance degradation as runtime increases. Elhoseiny’s group addressed this through two distinct architectures. For hour-long narratives, such as full television episodes of Friends, the 7-billion parameter model Goldfish partitions video streams into uniform chunks, generates semantic segment descriptors, and retrieves the most relevant segments into the language model's context. To handle redundant footage more efficiently, the lab developed LongVU. Presented at ICML, LongVU combines diversity-aware keyframe selection, dual semantic and spatial encoding via SigLIP and DINO, query-aware resolution allocation, and spatiotemporal compression that zeroes out static background coordinate tokens across frames. Operating within a compact budget of approximately 7,000 tokens, the 7B LongVU model outperformed GPT-4V and larger proprietary models on standard long-video benchmarks. Structuring Unstructured Video with Temporal and Attribute Schemas (15:02) To operationalize these academic models for enterprise search, Elhoseiny co-founded CreativAI, a query platform designed to index extensive video backlogs at a rate of 24 hours of video in under 4 hours. The platform maps unstructured visual sequences into a two-dimensional tabular data format through natural language queries and automated knowledge graphs. The framework organizes video intelligence into X-Rows and X-Columns. X-Rows execute temporal event localization, retrieving specific scenes matching a query (such as all dining scenes in a television series or specific investor pitches). X-Columns extract discrete analytical dimensions across those retrieved segments, classifying attributes like meal healthiness, cuisine category, participant identity, or deal terms. Delivered via conversational interfaces and programmatic developer APIs, this schema allows users to execute complex multi-variable searches—such as parsing Shark Tank episodes to filter for Mark Cuban's presence, the founder's gender, co-investors, and deal outcomes. Elhoseiny's work emphasizes that scaling video understanding requires moving beyond brute-force token ingestion. By coupling spatiotemporal token pruning with structured retrieval schemas like X-Rows and X-Columns, video LLMs can convert unstructured, long-form footage into computationally efficient, queryable relational knowledge.

精炼阅读摘要

阿卜杜拉国王科技大学（KAUST）教职成员兼CreativAI创始人Mohamed Elhoseiny回顾了多模态AI的研究演进，指出视觉语言理解已从早期零样本图文特征检索演变为具备显式推理能力的视觉大语言模型，并展示了通过时空压缩算法与结构化表格机制化解长视频理解中“Token爆炸”难题的技术路径。从零样本检索迈向指令化视觉推理 (0:04) Elhoseiny的跨模态研究始于2013年针对鸟类物种的零样本分类探索。该研究通过收集200多种鸟类的文本描述，利用线性变换矩阵将文本特征投影至视觉分类器空间，实现无需图像样本即可识别新物种。OpenAI随后在CLIP论文中将这一成果列为跨模态零样本学习的基础先导。然而Elhoseiny指出，单纯的跨模态特征检索并不等同于真正的视觉推理。在涵盖1万多个物种的细粒度评测中，检索模型往往无法分辨羽冠轻微卷曲或喙部特定色泽等细微形态差异，这促使团队转向研发具备显式推理能力的指令化模型。为突破静态检索的局限，其实验室于2021年推出了结合语言解码器与图像特征的VisualGPT，并于2023年4月开源了MiniGPT-4。该开源项目在GitHub积累了超过25,000颗星，展现了对复杂视觉情境的零样本对话与推理能力，例如解释冰冻湖面生长仙人掌的荒谬性、针对咖啡杯图片撰写营销文案，以及根据餐厅海鲜菜肴照片推导出包含具体用量（如四只龙虾、两汤匙橄榄油）的详细烹饪食谱。轻量跨模态对齐与MiniGPT模型生态 (6:46) MiniGPT-4在架构设计上追求极高的参数效率，仅通过单个可训练的线性投影层将预训练视觉编码器与大语言模型底座相连。该方案基于一项技术假设：将优质视觉表征映射到语言Token空间仅需要极低的学习能量。在初期利用网络爬取的大规模噪声数据集预训练时，模型约有30%的生成结果存在语言故障、幻觉或重复循环。研究团队随后构建了仅包含5,000组高质量图文对话样本的指令微调集，通过第二阶段对齐训练消除了生成异常，大幅提升了模型的对话流畅度与视觉对齐效果。在此架构基础上，该团队构建了覆盖多领域的模型生态：MiniGPT-v2引入细粒度视觉定位功能，支持在生成文字描述的同时输出目标边界框坐标；MiniGPT-3D与面向放射学的MiniGPT for Radiology将该机制分别拓展至三维空间数据与临床影像；MiniGPT-Video则将处理维度由静态帧延伸至动态时序序列，通过视频分帧、融合字幕文本并引入LoRA微调，支持对花朵绽放序列创作诗歌或提炼视频短文主旨等时序生成任务。时空Token压缩与长视频架构优化 (9:53) 面对长视频分析任务，直接堆叠独立视频帧将引发不可承受的“Token爆炸”：单帧图像约需144个视觉Token，20分钟视频的Token量将激增至约172,000个，而24小时视频则需占用约1230万个Token。为在有限的上下文窗口内处理长达数十分钟的内容，团队首先推出了70亿参数的检索增强视频模型Goldfish。该模型将长视频切分为均匀片段并生成局部文本摘要，依据用户提问检索相关片段并输入大模型上下文，在分析长达1小时的《老友记》剧集时展现出稳定的推理精度，避免了传统模型性能随视频时长增加而劣化的现象。随后在ICML发表的LongVU模型进一步从底层算法解决时空冗余问题。该架构包含四项核心处理步骤：一是利用多样性感知关键帧选择算法过滤相邻相似帧；二是结合SigLIP宏观语义特征与DINO细粒度空间特征进行双重编码；三是依据用户提问为关键区域分配高分辨率、为背景分配低分辨率的查询感知升采样；四是监控空间坐标并剔除在时间轴上保持静止的冗余背景Token。凭借这套压缩方案，70亿参数的LongVU仅依赖约7,000个Token的紧凑预算即可运行，并在多项长视频评测基准中超越了GPT-4V等规模更大的专有模型。结构化视频分析与CreativAI落地 (15:02) 长视频理解算法的突破促成了企业级查询引擎CreativAI的开发。Elhoseiny将其构想为针对海量视频数据的智能问答分析引擎，支持以自然语言检索上千小时非结构化视频，并能在4小时内完成24小时长视频的解析与索引。平台融合了自动知识图谱构建、无代码智能体工作流以及面向开发者的API接口。该系统的核心组织逻辑建立在两套抽象维度之上：X-Rows实现时序定位，负责定位并切分匹配特定语义的视频片段（例如检索《老友记》中所有角色进食的镜头，或筛选《创智赢家》中投资人Mark Cuban出席的所有陈述环节）；X-Columns负责结构化属性提取，用于从定位片段中提取分析字段（如识别菜系类型、评估饮食健康度、记录出场角色、标记创业者性别或判断最终投资协议是否达成）。这两项能力的结合使企业能够将无序视频画面直接转换为结构化报表并生成交叉统计图表，为大规模商业视频分析提供了落地途径。 Elhoseiny总结指出，多模态视频理解的演进关键在于从粗粒度图文匹配转向细粒度时空推理，并依靠算法级特征压缩破解Token计算开销瓶颈。随着模型在长视频结构化解析上取得进展，如何在更大规模工业级应用中维持超低延迟索引与跨多镜头复杂因果关联，仍是长视频理解领域持续演进的核心方向。

Detailed notes — zh

## Background 本内容由频道 Ai4 于 2025 年 8 月 26 日在 YouTube 平台发布。视频记录了阿卜杜拉国王科技大学（KAUST）副教授兼 CreativAI CEO 穆罕默德·埃尔霍塞尼（Mohamed Elhoseiny）的主题演讲，围绕视频大语言模型（Video LLMs）在长篇幅、复杂结构化视觉理解中的前沿进展、技术架构（如 MiniGPT 系列、Goldfish、LongVU）及评测基准展开介绍。 ## TL;DR 演讲系统回顾了演讲者从 2013 年零样本物种分类（CLIP 线性雏形）到研发开源轰动项目 MiniGPT-4 的学术历程，揭示了视觉大语言模型从简单跨模态检索向细粒度推理的演进。随后，重点解析了攻克长视频 Token 爆炸的两大核心架构——基于分块检索问答的 Goldfish 与时空冗余压缩流水线 LongVU，后者仅用约 7,000 个 Token 即可在长视频理解任务中超越 GPT-4V。最后，介绍了其创立的 CreativAI 工业级落地实践，通过类比表格行列的 X-Rows 与 X-Columns 机制和标准 API，将海量非结构化视频高效转化为可即时查询的结构化知识图谱与商业智能分析数据。 ## Atmosphere & style 整场演讲呈现出严谨、沉稳且极具洞见力的硬核学术工业技术分享氛围，演讲者以第一视角的亲历者姿态娓娓道来。叙事脉络从十余年前的基础理论探索层层推进至前沿开源突破，再自然过渡到高难度的工程架构设计与商业落地，条理极为清晰。聆听过程既能感受到演讲者对视觉几何与物理本质深刻理解所带来的智力愉悦，又能切实领略到学术前沿技术无缝落地为工业生产力工具的笃定与从容。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=lP-Cpq16W28&t=4s) Mohamed Elhoseiny 从 2013 年基于语言描述识别未见鸟类的研究出发，提出了跨模态联合嵌入算法，为后续 OpenAI 的 CLIP 奠定了基础理论框架，被视为其线性雏形。 - [03:44](https://www.youtube.com/watch?v=lP-Cpq16W28&t=224s) 随着识别物种扩展到万种级别，传统基于跨模态检索的 CLIP 机制性能显著下滑，表明细粒度视觉理解必须转向具备深层推理能力的视觉大语言模型（Visual LLM）。 - [03:44](https://www.youtube.com/watch?v=lP-Cpq16W28&t=224s) 团队在 2021 年推出 VisualGPT 后，于 2023 年 4 月早于 OpenAI 官方发布多模态能力之前开源了 MiniGPT-4，迅速成为最具影响力的开源视觉指令微调项目之一。 - [06:54](https://www.youtube.com/watch?v=lP-Cpq16W28&t=414s) MiniGPT-4 凭借高效的两阶段对齐架构，仅依靠预训练视觉编码器加线性映射层，并通过 5,000 对高质量指令数据微调，就彻底消除了复读与幻觉缺陷。 - [06:54](https://www.youtube.com/watch?v=lP-Cpq16W28&t=414s) 从 MiniGPT-4 演进出的多模态家族涵盖了 MiniGPT-v2（细粒度目标定位）、MiniGPT-3D（空间点云理解）与 MiniGPT-4-video（视频帧与字幕多模态联合理解）。 - [09:52](https://www.youtube.com/watch?v=lP-Cpq16W28&t=592s) 面对整集电视剧等长视频的计算复杂度瓶颈，Goldfish 模型通过视频时序分块、特征索引与动态检索 Top-K 片段的机制，实现了约 1 小时超长视频的精准问答。 - [11:37](https://www.youtube.com/watch?v=lP-Cpq16W28&t=697s) 视频本质由物体的时空运动构成并伴随大量静态背景冗余，朴素的密集逐帧采样会造成灾难性 Token 膨胀，在工业级长视频分析中不可行。 - [11:37](https://www.youtube.com/watch?v=lP-Cpq16W28&t=697s) LongVU 创新性地结合多样性关键帧采样、SigLIP 与 DINO 双编码器、查询感知分辨率提升及静态 Token 置零压缩，仅用约 7,000 Token 就在长视频评测中超越 GPT-4V。 - [15:01](https://www.youtube.com/watch?v=lP-Cpq16W28&t=901s) 创业企业 CreativAI 旨在构建海量视频资产的类 Perplexity 智能查询引擎，能在 4 小时内解析以往需 24 小时处理的大规模视频库并构建底层知识图谱。 - [15:01](https://www.youtube.com/watch?v=lP-Cpq16W28&t=901s) CreativAI 提出了“X-Rows（时空事件定位行）”与“X-Columns（多维属性提取列）”体系，将非结构化的长视频影像直接转换为可供 BI 工具分析的关系型数据。 - [17:49](https://www.youtube.com/watch?v=lP-Cpq16W28&t=1069s) CreativAI 现已开放开发者 API，能针对《鲨鱼坦克》等大规模长视频快速完成人物筛选、商业决策判定与属性抽取，实现端到端的业务数据自动化。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=lP-Cpq16W28&t=4s) 缘起与先驱：从 2013 年零样本物种识别到 CLIP 的线性雏形 Mohamed Elhoseiny 回顾了自己在视觉与语言（Vision-Language）交叉领域深耕逾 15 年的学术与创业历程。在创立 CreativAI 之前，他曾先后在硅谷的多家顶尖研究机构工作，包括在 Facebook AI Research 与 Yann LeCun 合作，以及在斯坦福大学和 Adobe Research 推动前沿 AI 研究，目前担任阿卜杜拉国王科技大学（KAUST）副教授兼 CreativAI CEO。这项研究热情的起点可以追溯到他在 2013 年发表的一篇论文（距今已有 12 年以上）。当时的研究灵感来源于人类如何通过语言描述来发现和认知新物种。例如，若直接展示一种名为“bracket”（或罕见鸟类）的图片，大多数人无法辨识；但若给出一段语言描述——“这是一种体型小巧的鸟，拥有短小的橙色喙，羽毛上方呈深色、下方呈白色”——人们的大脑中便会立刻构建出该鸟的视觉形象，从而能在从未见过该鸟类的情况下从一堆照片中迅速将其识别出来。 ```mermaid flowchart LR accTitle: 2013年文本描述零样本分类算法流程图 subgraph Input ["多模态输入"] A["文本类别描述 T (200+ 鸟类物种)"] B["未标注鸟类图像 I"] end subgraph Embedding ["联合嵌入与分类器生成"] C["线性映射权重 W"] D["类别分类器 Classifier = T × W"] end subgraph Matching ["跨模态对齐"] E["同类别拉近 / 异类别推远"] F["零样本/未见类别精准预测"] end A --> C --> D B --> E D --> E --> F ``` 基于“人类能从语言描述中学习，机器也应如此”的动机，Elhoseiny 在 12 年前收集了 200 多种鸟类的文本描述数据集，并设计了一种联合嵌入学习算法：将类别级别的文本描述与鸟类图像配对，同类样本在嵌入空间中相互拉近，异类样本相互推远。只要将语言表征 $T$ 与一个简单的神经网络权重矩阵 $W$ 相乘，便能直接为任意未见过的类别生成分类器。Elhoseiny 指出，OpenAI 后来大获成功的 CLIP 模型在其论文中明确引用了这项工作作为首创先驱；CLIP 本质上是该算法在海量数据和深层网络加持下的现代延伸，而他在 2013 年提出的方法正是“CLIP 的线性版本（linear version of CLIP）”。 - 2013 年的先驱论文构建了首个包含 200+ 鸟类物种语言描述的细粒度零样本分类基准。 - 核心算法通过构建图文联合嵌入空间，使得未见类别的文本特征可直接映射生成视觉分类器权重。 - OpenAI 的 CLIP 沿用了这一对比学习与联合嵌入框架，依靠海量互联网数据和更深的模型架构将其发扬光大。 --- ### [03:44](https://www.youtube.com/watch?v=lP-Cpq16W28&t=224s) 从跨模态检索迈向细粒度推理：VisualGPT 与 MiniGPT-4 的爆发尽管基于检索的 CLIP 模型在图文对齐上表现优异，但它本质上只是跨模态检索器，缺乏细粒度的“推理机器（reasoning machine）”。Elhoseiny 团队在后续预印本研究中发现，当物种规模扩展到 10,000 种以上时，CLIP 类检索机制的效果会大幅下降。因为在细粒度物种识别中，区分两种近缘鸟类的唯一依据可能只是头顶一根微微弯曲的羽毛或鸟喙颜色的微小差异，这需要具备深层推理能力的视觉大语言模型（Visual LLM）。为了将模型推向具备指令遵循能力的视觉系统，团队在 2021 年率先研发了 VisualGPT，这是最早将纯语言模型接入图像描述任务的大模型之一，但它当时只能执行“为该图像提供简短描述”这类隐式指令，无法按要求撰写营销文案或复杂长文。转折点出现在 2023 年 4 月，团队正式开源了 MiniGPT-4，这甚至早于 OpenAI 正式发布其多模态图像能力。几乎在同一时期（仅相隔数天至一周），威斯康星大学等机构也发布了 LLaVA（Visual Instruction Tuning）。 MiniGPT-4 在开源社区引发了轰动，其 GitHub 仓库迅速斩获超过 25,000 颗 Star，成为该领域最具影响力的开源项目之一。用户第一次惊喜地发现，开源模型能够理解诸如“在冰冻湖面上出现一棵仙人掌有什么不寻常之处？”这类复杂常识问题，能够根据随手拍摄的马克杯照片自动生成引人入胜的广告文案，或是根据一盘刚出炉的海鲜大餐照片直接反推烹饪食谱（精确给出“准备 4 只龙虾、2 汤匙橄榄油”等详尽配料）。 --- ### [06:54](https://www.youtube.com/watch?v=lP-Cpq16W28&t=414s) MiniGPT-4 的技术精髓与多模态模型家族演进 MiniGPT-4 之所以能以极高效率实现强大的多模态对齐，其核心在于精巧的两阶段训练架构。团队选用了一个与语言领域高度对齐的预训练视觉编码器，并仅在其后连接了一个简单的线性映射层。基于这一设计，模型对齐所需的训练开销极小。最初仅在通用数据集上预训练时，约有 30% 的生成结果存在语言复读或幻觉等缺陷；随后团队仅引入了 5,000 对自建的高质量图文指令遵循微调数据进行第二阶段优化，便彻底消除了生成瑕疵，带来了质的飞跃。随着 MiniGPT-4 的成功，团队迅速拓展并构建了丰富的多模态模型家族： 1. **MiniGPT-v2**：强化了细粒度视觉定位（Visual Grounding）能力，例如当模型描述“一名身穿西装的男士正叠放着两个塑料杯”时，能同步高亮标出男士与塑料杯在图像中的精确边界框。 2. **MiniGPT-3D**：将多模态理解延伸至 3D 点云与空间场景。 3. **MiniGPT-4-video**：针对视频理解任务进行升级。通过对大语言模型进行 LoRA 微调，使系统不仅能从时序视频帧中抽取关键视觉表征，还能同时结合视频字幕（Subtitles）进行联合理解。模型能够为一段鲜花盛开的视频赋诗，或者精准提炼出女性赋权主题宣传视频的核心主旨。 --- ### [09:52](https://www.youtube.com/watch?v=lP-Cpq16W28&t=592s) Goldfish：基于分块检索的长视频大模型（突破 1 小时长视频问答）在处理数分钟的短视频之后，团队将目光投向了更具挑战性的长视频理解场景——例如理解整集经典美剧《老友记》（*Friends*）。当用户提出跨度极长、定位极细的问题（如“菲比（Phoebe）在与莫妮卡（Monica）交谈时向她展示了什么？”）时，传统的整视频端到端编码会遭遇上下文窗口与计算复杂度的硬性制约。 ```mermaid flowchart TD accTitle: Goldfish长视频理解与检索问答架构 subgraph Ingestion ["长视频分块与描述生成"] V["1小时长视频输入"] --> CH["时序分块 (Chunking)"] CH --> SVM["短视频模型 (Short Video Model)"] SVM --> CAP["生成局部语言描述/表征索引"] end subgraph Retrieval ["查询相关片段检索"] Q["用户提问: '菲比向莫妮卡展示了什么？'"] --> RM["检索模块 (Retrieval Module)"] CAP --> RM RM --> TOP["召回 Top-K 最相关视频片段"] end subgraph Generation ["上下文整合与推理"] TOP --> LLM["7B 参数 LLM 上下文推理"] Q --> LLM LLM --> ANS["输出精准答案与事件定位"] end ``` 为此，团队推出了名为 **Goldfish** 的长视频大语言模型，成功实现了对约 1 小时级别超长视频的流畅解析。Goldfish 采用分块检索机制：首先将超长视频切分为连续的短片段，通过短视频模型提取每个片段的语言级高维表征并建立索引；当接收到用户提问时，专用检索模块会动态计算问题与各视频片段的相关性，精准召回最关键的 Top-K 候选片段，送入 7B 参数大语言模型的上下文窗口中生成最终答案。实验表明，无论输入视频长度如何增加，Goldfish 均能保持极为稳定的准确率，在短视频与长视频基准评测中均取得了当时的 SOTA 性能。 --- ### [11:37](https://www.youtube.com/watch?v=lP-Cpq16W28&t=697s) LongVU：解构时空冗余与 Token 爆炸（ICML 工作详解）在发表于 ICML 的 **LongVU** 工作中，Elhoseiny 进一步深入探讨了视频的物理与几何本质。他指出，将视频简单视为一帧帧静态图像的堆叠是非常低效的；视频本质上是物体在时间轴上的位移与形变，伴随着海量的背景冗余——例如在演讲画面中，仅有主讲人的面部和手势在动态变化，绝大部分背景像素完全静止。朴素的单帧密集采样会导致灾难性的 Token 爆炸： - 1 帧静态图像约占用 144 个 Token； - 一段 20 分钟的普通视频将膨胀至 172,000 个 Token； - 一整天（24 小时）的视频数据更是高达 1,230 万个 Token； - 面对像 Netflix 拥有超过 100,000 小时库存的工业级场景，简单的帧采样完全不可行。 ```mermaid flowchart LR accTitle: LongVU多模态时空压缩流水线 subgraph Step1 ["时序关键帧采样"] A["输入长视频"] --> B["多样性感知关键帧选择 (最小化帧间相似度)"] end subgraph Step2 ["多尺度特征抽取"] B --> C["SigLIP 语义特征 + DINO 空间特征"] end subgraph Step3 ["查询感知与时空压缩"] C --> D["查询相关性高低分辨率双模提升 (High-Res / Low-Res)"] D --> E["时空 Token 压缩 (静态区域置零，保留动态 Token)"] end subgraph Step4 ["高效推理"] E --> F["仅 7,000 Tokens 传至 7B LLM (性能大幅超越 GPT-4V)"] end ``` 针对这一瓶颈，LongVU 提出了一套系统的技术创新方案： 1. **多样性感知采样（Diversity-Aware Sampling）**：筛选信息熵最高且帧间视觉相似度最小的关键帧集合。 2. **双编码器协同（SigLIP + DINO）**：同时提取 SigLIP 的深层多模态语义特征与 DINO 的细粒度空间几何特征。 3. **查询感知分辨率自适应（Query-Aware Uplifting）**：设立高分辨率（$H_H \times W_H$）与低分辨率（$H_L \times W_L$）两种视觉表征模式，仅对与用户提问高度相关的关键帧进行高分辨率特征提升。 4. **时空 Token 压缩（Spatiotemporal Token Compression）**：在空间坐标 $(x, y)$ 上沿时间轴追踪 Token 变化，对随时间保持静态的背景 Token 统一置零剔除，仅保留产生实质运动变化的动态区域。凭借这一流水线，LongVU 仅需约 7,000 个 Token 的极小开销，便能驱动 7B 参数模型在多项长视频评测中全面击败 GPT-4V。 --- ### [15:01](https://www.youtube.com/watch?v=lP-Cpq16W28&t=901s) 创业实践 CreativAI：打造面向海量视频的知识图谱与类 Perplexity 查询引擎在演讲的最后，Elhoseiny 介绍了其创立的企业 CreativAI。该公司的核心使命是构建“LLM 驱动的超大规模多模态视频查询引擎”，使用户面对无限量级（Infinite Video Data）的视频资产时，能够如同使用 ChatGPT 或 Perplexity 一样，在指尖快速提取结构化情报。 CreativAI 具备极高的数据吞吐效率，能在 4 小时内完成原本需要 24 小时处理的海量视频分析（支持成千上万小时的视频库解析），并底层原生集成了知识图谱（Knowledge Graph）与无代码 Agent 解决方案。 | 核心维度概念 | 工业级定义 | 典型应用场景示例 | |:--- |:--- |:--- | | **X-Rows（时空定位行）** | 负责在海量长视频库中自动定位符合特定语义条件的时序事件片段 | 在《老友记》第 1 至 3 季中精确定位所有“角色正在进餐”的场景片段；在《创智赢家》（*Shark Tank*）全集中定位“Mark Cuban 参与谈判”的访谈片段 | | **X-Columns（属性提取列）** | 负责从已定位的视频片段中多维度抽取离散/连续的结构化元数据与深度洞察 | 判定食物是否健康、菜系类别（中餐/法餐等）、出场人物统计、各主角就餐偏好交叉图表；提取创业者性别、是否达成投资协议、合投 Shark 成员名单并自动导出数据表 | 系统目前已开放 Beta 版本体验并提供标准 API。该系统旨在将非结构化的视频影像彻底转化为可供商业智能（BI）工具直接分析的关系型表格与知识网络，即使是轻量级模型，在此架构赋能下亦能在标准基准上媲美 GPT-4o 等顶级闭源大模型的表现。 ### [17:49](https://www.youtube.com/watch?v=lP-Cpq16W28&t=1069s) 面向开发者的 API 支持、结构化视频检索实战与总结演讲者在分享的最后介绍了目前已经开放使用的 API 能力，重点展示了如何通过该 API 提取视频中的特定片段与结构化信息。对于开发者而言，这一工具能够极大地简化长视频内容的定向挖掘与自动化分析。以知名商业真人秀节目《鲨鱼坦克》（*Shark Tank*）的视频合集为例，开发者可以直接调用接口，精准筛选并提取所有马克·库班（Mark Cuban）出场的视频片段，并围绕这些片段发起更深度的结构化查询与推理。通过该系统，用户可以自动化解析并回答一系列细粒度的业务问题，例如：登台推介的创业公司 CEO 的性别是什么、该公司最终是否成功获得了投资交易（Deal）、以及当时还有哪些其他“鲨鱼”（投资人）共同参与了该笔交易等。这种处理方式能够将原本非结构化的长视频素材转化为高价值、易于下游程序调用的结构化数据。演讲者以此例总结了模型在实际开发场景中的落地潜力，并向现场观众致谢结束了本次演讲。 - **开发者 API 功能**：支持针对大规模视频数据集进行特定人物定位、片段提取及多模态属性解析。 - **《鲨鱼坦克》（*Shark Tank*）案例查询维度**： - 特定投资人筛选（如马克·库班出场的所有片段）； - 创业者属性识别（如创始 CEO 的性别）； - 商业结果判定（公司是否达成投资意向及具体跟投的投资人名单）。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=lP-Cpq16W28&t=4s) 这是一种体型小巧的鸟，拥有短小的橙色喙，羽毛上方呈深色、下方呈白色 — Mohamed Elhoseiny > [00:04](https://www.youtube.com/watch?v=lP-Cpq16W28&t=4s) 人类能从语言描述中学习，机器也应如此 — Mohamed Elhoseiny > [00:04](https://www.youtube.com/watch?v=lP-Cpq16W28&t=4s) CLIP 的线性版本（linear version of CLIP） — Mohamed Elhoseiny > [03:44](https://www.youtube.com/watch?v=lP-Cpq16W28&t=224s) 在冰冻湖面上出现一棵仙人掌有什么不寻常之处？ — Mohamed Elhoseiny > [03:44](https://www.youtube.com/watch?v=lP-Cpq16W28&t=224s) 准备 4 只龙虾、2 汤匙橄榄油 — Mohamed Elhoseiny > [06:54](https://www.youtube.com/watch?v=lP-Cpq16W28&t=414s) 一名身穿西装的男士正叠放着两个塑料杯 — Mohamed Elhoseiny > [09:52](https://www.youtube.com/watch?v=lP-Cpq16W28&t=592s) 菲比（Phoebe）在与莫妮卡（Monica）交谈时向她展示了什么？ — Mohamed Elhoseiny > [15:01](https://www.youtube.com/watch?v=lP-Cpq16W28&t=901s) LLM 驱动的超大规模多模态视频查询引擎 — Mohamed Elhoseiny ## Mentioned Mohamed Elhoseiny, Yann LeCun, Facebook AI Research, 斯坦福大学, Adobe Research, 阿卜杜拉国王科技大学（KAUST）, CreativAI, CLIP, OpenAI, VisualGPT, MiniGPT-4, LLaVA, MiniGPT-v2, MiniGPT-3D, MiniGPT-4-video, Goldfish, LongVU, SigLIP, DINO, GPT-4V, GPT-4o, StoryGPT-V, Vgent, InfiniBench, SpookyBench, 《老友记》（Friends）, 《创智赢家》（Shark Tank）, Mark Cuban, Netflix, ChatGPT, Perplexity ## Video description Mohamed Elhoseiny, Associate Professor & CEO, KAUST & CreativAI Abstracts The rise of Vision-Language Models (VLMs) has opened new frontiers in video understanding, yet scaling these systems to handle long, complex, and structured visual narratives remains a fundamental challenge. In this talk, we explore a suite of recent advances aimed at achieving scalable and structured video comprehension through Large Vision-Language Models (Video LLMs). We begin by examining instructable models like MiniGPT-4 (image-based), MiniGPT-4-v2(image-based), MiniGPT-3D, and its video extension, which leverage multimodal instruction tuning for diverse video tasks. We then introduce Long Video LLMs—including the Goldfish and LongVU models—that tackle the token explosion problem with retrieval-augmented generation and spatiotemporal compression. Further, we address structured understanding via StoryGPT-V and Vgent, which model character consistency and entity-based graph reasoning. To rigorously evaluate progress, we present InfiniBench and SpookyBench, two novel benchmarks designed to probe long-form comprehension and temporal perception in state-of-the-art models. Finally, we extend the discussion to include multimodal capabilities in multilingual, emotional, and action-driven contexts, as well as exploratory work on bridging vision and brain signals. [Source](https://www.youtube.com/watch?v=lP-Cpq16W28)

Source quotes

[0:04] Mohamed Elhoseiny: 这是一种体型小巧的鸟，拥有短小的橙色喙，羽毛上方呈深色、下方呈白色 [0:04] Mohamed Elhoseiny: 人类能从语言描述中学习，机器也应如此 [0:04] Mohamed Elhoseiny: CLIP 的线性版本（linear version of CLIP） [3:44] Mohamed Elhoseiny: 在冰冻湖面上出现一棵仙人掌有什么不寻常之处？ [3:44] Mohamed Elhoseiny: 准备 4 只龙虾、2 汤匙橄榄油 [6:54] Mohamed Elhoseiny: 一名身穿西装的男士正叠放着两个塑料杯 [9:52] Mohamed Elhoseiny: 菲比（Phoebe）在与莫妮卡（Monica）交谈时向她展示了什么？ [15:01] Mohamed Elhoseiny: LLM 驱动的超大规模多模态视频查询引擎

Source quotes

[0:04] Mohamed Elhoseiny: linear version of CLIP [3:44] Mohamed Elhoseiny: provide a short description [3:44] Mohamed Elhoseiny: Visual Instruction Tuning [3:44] Mohamed Elhoseiny: four lobsters, two tablespoons of olive oil [6:46] Mohamed Elhoseiny: learning energy [9:53] Mohamed Elhoseiny: What does Phoebe show Monica while they are talking? [15:02] Mohamed Elhoseiny: Perplexity-like experience for video data [15:02] Mohamed Elhoseiny: Analyze food consumption in Friends

Original transcript

[0:04] So today I'm going to share with you a [0:05] little bit about my journey uh more than [0:07] a decade 15 years working on the [0:09] intersection between vision and language [0:11] mo mostly starting from academic [0:13] perspective and concluding with an [0:15] entrepreneurial uh like gig that I have [0:18] have started recently over the over a [0:20] year now. So uh towards scalable and [0:23] structural understanding of video. So a [0:25] little bit about my background. I have [0:26] worked in several several years at in [0:29] Silicon Valley including Facebook [0:30] research with uh with the Yanikon or uh [0:34] Stanford Adobe B research working on [0:37] advancing the state-of-the-art of uh uh [0:40] of AI specifically more and more [0:42] specifically related to vision language [0:43] modeling. So to start I would like to [0:46] share with you maybe the one of the [0:48] first works that I have built. It was [0:50] this journey. Um and this is where I my [0:54] excitement about this field has started. [0:56] This is a paper from 2013 more than 12 [0:58] years old paper that is inspired by or [1:02] by species discovery. So if I tell you [1:06] for example bracket uh I ask you to [1:08] identify it in an image. Maybe most of [1:10] us may not know what it what it is. But [1:13] if we describe that in language and I [1:14] tell you that that this is a small bird [1:16] that has a short orange peak and the [1:18] blemish of that that bird is short is is [1:21] a dark above and white below. You start [1:23] to imagine and plant an image of this [1:25] bird in your head. And then identifying [1:28] this bird among this different one [1:31] becomes an easy task even if you haven't [1:33] seen this bird before. So this actually [1:35] this shows that people can learn from [1:38] language descriptions and this motivates [1:40] that we may build machines that learns [1:42] like us. This is a this is basically the [1:44] slide of this 12 years old paper. So [1:47] back then I collected language [1:48] descriptions for more than 200 bird [1:51] species to aiming to identify or [1:54] enabling tasks just like this and that [1:56] was trained with a learning that was [1:58] done by running algorithm that I [2:00] developed back then that that picks [2:02] every text image pair. This text [2:04] description is defined at the class [2:06] level, bird class level, an image of a [2:08] bird. And if they belong to the same [2:11] class, I want to bold them together. And [2:13] if they are not from the same class, I [2:15] want to make them distant from another. [2:18] So I basically try to learn a joint [2:20] embedding space where that brings images [2:22] and text at the same time. This makes it [2:25] in fact conceptually you can basically [2:27] predict a classifier for an unseen class [2:29] given its language description by having [2:32] the langu representation of that class T [2:34] multiplied by a simple neural network [2:36] that which is we refer here to as as W [2:39] and once you do that you have uh right [2:42] away a classifier for that can apply to [2:44] arbitrary uh unseen class. uh this is [2:48] this has been revisited recently by the [2:50] clip paper if you heard about it the [2:52] openi paper which cited my work as the [2:55] first work that basically does this kind [2:58] of idea. So basically this openi paper [3:00] is is exactly my method but has been [3:04] revisited by training on large amount of [3:06] data and and this enables them to train [3:09] on a deeper model. So basically you can [3:10] think of the method that I just proposed [3:12] in 2013 as a linear version of clip [3:16] which is has is now the foundation of [3:18] many uh advances in the field. Back then [3:20] nobody really cared about image language [3:22] modeling but now this has been this same [3:25] very same learning algorithm has been [3:26] revisited uh and enabled by uh the [3:30] availability of large amount of data. So [3:32] this has been a journey that started [3:33] that just started back then and have [3:36] developed many methods over the years. [3:38] This is a QR code that shows basically a [3:40] journey in this specific problem. So [3:44] coming back to here right clip methods [3:46] is basically a retrieval uh mechanism a [3:48] crossmodel retrieval mechanism given a [3:50] language I want to get the a language [3:52] description of a bird I want to build a [3:54] bird that has uh a visual appearance [3:56] that is compatible with a description [3:59] and get this highly ranked but you don't [4:01] have a reasoning machine right uh if you [4:04] look at this this is just a paper we put [4:07] in archive recently that shows that this [4:09] clip retrieval based mechanism are not [4:11] really good at identifying more than [4:13] 10,000 species. Because if you look at [4:16] fine grain species, the only difference [4:18] that could identify one fine grain [4:20] species from another could be a curved [4:22] feather that comes from the top of the [4:23] head or a color of a beak or something. [4:25] This requires more fine grain reasoning [4:27] machine. And we showed in the last row [4:29] that this can be empowered by having [4:32] reason simplicit reasoning machines [4:34] which is powered by an instructible [4:37] large language model visual large [4:38] language model meaning large language [4:40] models that can operate on images. This [4:44] motivates our next station mini GBT [4:48] at noting that bridging towards [4:50] instructible visual machine. I built the [4:52] visual GBT in 2021. This is the first [4:55] large language model that can uh that [4:57] task with a a blind large language model [5:01] operating on images for the task of [5:04] image description. Uh but is this has an [5:07] implicit instruction of of something [5:09] like provide a short description of this [5:11] image. Not really something that you can [5:13] ask it to write let's say um let's say [5:17] an essay or write it ask it to write an [5:20] advertisement about a product that you [5:22] want to sell. And here's a picture of [5:23] the product. But it was enabling it was [5:26] much more efficient as an image [5:27] captioning system. Later we proposed [5:31] mini GBT4 in April 2023. That was even [5:33] before OpenAI released their first model [5:35] that can operate on images. This uh was [5:38] u in 2023 and was in parallel. There was [5:41] another uh paper that called visual [5:43] destruction tuning. People refer to as [5:46] lava. This would announce it few days [5:48] later to a week. This was the this uh [5:51] get uh got a lot of uh attention on [5:54] GitHub 20 more than 25,000 [5:57] stars. It's still the most starred [5:59] GitHub repository in this field of large [6:02] language modeling computing and images. [6:04] And it shows that you can actually for [6:06] the first time back then to ask it what [6:09] is wrong with this image? Is it why [6:11] cactus in the middle of frozen lake is [6:13] something unusual? Give it an image of a [6:15] of a for example a mug and you want to [6:17] advertise this mug and it describes you [6:20] this bonus it by describing for example [6:22] how can you advertise this mug. Give it [6:25] a picture of a meal that you want to [6:28] meal that they want to that you just ate [6:30] and I want to know how to make it at [6:31] home and it tells you bring me four [6:33] tablespoons like four lobsters two [6:35] tablespoon of olive oils and things like [6:37] that. So at that time that shows some [6:39] promising ability of having models that [6:41] can follow instruction in response to [6:43] images. Um so and we have that started [6:46] not just a one model mini GBT became [6:48] like a family of model. We have many GBT [6:52] this uh [6:54] this was [6:57] >> a model from your notebook and make a [7:01] well. [7:03] So see that time a lot of people got [7:05] excited in in in the social media [7:07] because that uh like open I said that [7:09] they are working on something like that [7:11] and as academic uh people we did this [7:14] before then so a lot of people got [7:15] excited that we have an open source [7:17] model that was capable of doing this. So [7:21] the technical details of this was quite [7:23] uh like simple. Given an image, we train [7:26] it like we we we picked carefully a [7:28] visual encoder that operates on the [7:32] image uh that's be aligned with language [7:34] domain and then we train the linear [7:36] layer with the assumption that the [7:39] learning energy that we want to spot [7:40] here if we pick this in visual image [7:43] encoder carefully uh will require small [7:45] amount of uh uh like learning energy. We [7:49] trained this on some initial data set [7:51] that is available. Uh that was not ideal [7:54] and then we had to collect some [7:56] instruction data that we had to collect [7:58] ourself. But luckily this was only a [8:00] small amount of 5,000 image text [8:02] response pair and it was enabling uh [8:06] when while we have maybe 30 of the [8:08] responses was glitching like this 60% [8:11] was good. after stage two improvement [8:13] which is part of the same method uh this [8:16] uh problem has heavily uh been addressed [8:18] and that led to uh the model performing [8:21] uh and lots of people resonating with [8:23] the with the response. So again we [8:27] developed a family of model one mini GBT [8:30] version two mini VT video mini GBT for [8:32] radiology this you can basically check [8:34] the the the journey and and this [8:36] repository but this for example is [8:38] enabling it to ground for example what's [8:40] happening in the image a man in a suit [8:43] holding two plastic cups on top of each [8:45] other it can can tell you also as it [8:47] speaks about what's in the image where [8:49] it is where is the man where is the [8:51] plastic cups and so on [8:53] video model so this This is also one of [8:55] the early model for processing videos. [8:58] Like you give it uh an im a video and [9:01] then it you you divide them into frames [9:04] and then from this you uh train uh a [9:08] model that can give it give it an [9:13] a video like this and ask it to write a [9:16] boom about this this uh video of flowers [9:20] or ask it to summarize this video about [9:23] uh that is about empowering women and [9:26] then ask it to maybe uh [9:30] get the extract the key message of this [9:32] of this video. [9:35] >> All right. So that the idea of this was [9:38] quite like we we we shifted from having [9:41] a frozen large language model to one [9:43] that is Laura fine-tuned and a model [9:45] that can access both the information in [9:47] the image and also and also in the [9:50] subtitles if exists. [9:52] um like this is um basically uh like [9:56] models that can process maybe few [9:59] minutes videos later we got interested [10:01] on basically processing longer videos. [10:04] Imagine that you have an episode of [10:06] friends and you want to ask questions [10:08] about what's happening in this video [10:10] like let's say what does FIPA FIP uh [10:13] show Monica while they are talking. So [10:16] we have a paper from a go called [10:18] goldfish u that can process about an [10:21] hour of video or something like that. [10:28] >> Well look no further. [10:31] >> So it basically tries to find the [10:34] segment of the video that you are [10:35] interested to uh ask questions bas still [10:38] you need to provide this uh video one [10:40] video at a time. Um the the model was [10:43] based on chunking. You chunk given the [10:44] video you chunk it and then you process [10:46] each one through the short video model [10:48] produce language description and you [10:50] have uh like a a retrieval module that [10:53] takes the question and then determine [10:56] its relevance to the list of segments [10:58] retrieve the top ones bring them to the [11:00] context of large language model and then [11:02] and then provide that uh context to an [11:05] LM to extract the final answer. [11:08] um that uh back then was a 7 billion [11:10] parameter model and did achieve the [11:13] state-of-the-art uh uh uh performance on [11:15] video LMS on short videos and also in a [11:18] in a long video evaluations data set [11:20] that we um we proposed. Um interestingly [11:25] the model has steer no matter how big [11:28] the video lens is. The performance is [11:29] still uh is still is more consistent [11:32] compared to other uh models that have [11:35] less favored behavior. [11:37] All right. So this is think I think I [11:40] would maybe roughly go over this paper [11:42] and then we'll talk about the company uh [11:44] in the last five minutes. So basically [11:47] long view is another like paper we that [11:50] we presented at ICML recently that pro [11:53] that basically addresses the very nature [11:55] of video. Videos as we may imagine are [11:59] not just a set of frames. They are the [12:02] this is not this is simplistic view of a [12:04] video right? It's objects that are [12:06] moving that they are and there are lots [12:08] of redundancy that as I speak right now [12:11] maybe only small portion of the video is [12:13] moving a lot lots of the backgrounds are [12:15] not moving. So just representing images [12:17] as set of frames is not an ideal or [12:20] token efficient representation. So a [12:23] long view is one of the the models that [12:25] we propose that addresses the very [12:27] nature of video by addressing this [12:30] speech to temporal redundancy. If you do [12:32] this for example, if you look at this [12:34] simplistic view of video, you will find [12:36] for example that one frame, one image is [12:39] about 144 tokens. 20 minute of video is [12:42] 172,000 tokens. One day video, if you [12:45] want to imagine something like this, [12:47] which is not hard to imagine like I [12:49] think um [12:51] u Netflix has more than 100,000 hours [12:54] for example, right? Uh this is 12.3 [12:57] million token, right? So we need [12:58] something that we can do better than [13:00] just sampling videos as set of frames. [13:03] So motivated by this we proposed a set [13:06] of like let's say technical uh [13:09] innovations. This starts by picking the [13:12] like a set selecting a subset of frames [13:14] that have the highest information set of [13:16] key frames that are mostly important and [13:20] then are not that that that somehow [13:23] reduces the simil like similarity uh or [13:25] minimizes similarity between it and uh [13:28] and and each other. So this you can [13:30] think think of this as diversity aware [13:32] sampling of the video frames and then [13:35] after that we represent this by cyc and [13:37] dynino features to make the model more [13:40] specially aware and also more semantic [13:42] aware which is captured by cichlip and [13:45] dino respectively and then after that we [13:48] have uh query aware uh like um [13:51] uplifting. So basically we bring images [13:55] like set of frames that are most [13:56] relevant to the question. We bring we [13:59] have two modes of of of visual [14:01] representation. One is high resolution [14:03] mode HW [14:07] uh and the other one HLWL which is low [14:09] resolution mode. So we uplift the [14:12] queries that are the frames that are [14:14] most relevant to the question uh that [14:17] based on how relevant they are to the [14:18] frame. The second the third step was [14:21] special to temporal special token [14:23] compression which is basically it looks [14:25] at every token and xy coordinate and if [14:27] it does not change over time it [14:29] represent it zeros out all the rest [14:31] because so that so that you don't uh [14:34] provide redundant information and those [14:38] regions that does change over time we [14:39] keep them the same all right this model [14:42] have achieved better performance and it [14:45] has it it actually outperforms with [14:47] GBT4V with uh with a a margin uh that [14:51] for the first time with only a 7 billion [14:53] parameter model and with a model that [14:54] have only 7,000 tokens [14:58] and it also have outperforms smaller [15:00] vision language model getting to the [15:02] company uh I would say spend the last [15:04] four minutes on in the company the the [15:06] company is about creating LM powered [15:08] query engine at scale so imagine talking [15:12] at infinite video data and you want to [15:14] extract information from it at your [15:17] fingertips [15:20] So the way we are doing right now is [15:22] that we can process large language with [15:24] in a chat GBT like way uh thousands of [15:28] hours hundreds thousand thousand of [15:30] hours we process in 24 hours in under [15:32] four hours we have knowledge graph [15:35] support no code solution agent and so [15:37] on. So this is to demonstrate how this [15:40] looks like and you can actually scan the [15:41] QR code now and and and actually request [15:44] the access. We are releasing a better [15:46] version that is ready to try now. Maybe [15:49] by by next week we will have the uh the [15:52] uh basically more details about the the [15:54] pricing and things like that. But right [15:56] now everyone can give it a shot. You [15:59] have like friends collection. You can [16:01] basically put pull to pull something [16:04] like this people eating food in seasons [16:06] uh one and three. And the model will [16:09] determine based on this the relevant [16:11] segments and we'll have a smart way to [16:13] represent the information that is [16:14] knowledge graph away aware and then you [16:16] can ask you can basically suggest [16:19] questions. So we have a concept here [16:20] called X rows and X columns that we uh [16:22] we are we we we are we innovated X rows [16:25] here is basically when you ask questions [16:27] you want to localize information or [16:29] video segments that are relevant to the [16:31] information you want to extract here [16:32] food eating uh exper food eating [16:35] experiences in seasons uh one and three [16:38] are localizing the extros which is uh [16:40] extracting visual information from this [16:42] video. The X columns here are the [16:44] information I want to extract like for [16:45] example I want and I want to know [16:46] whether this food is healthy or not. I [16:48] want to know uh what cuisine is this [16:51] right? I want to know what are the [16:52] characters that appeared in this uh [16:54] video series. I want to show the dist [16:55] the distribution of cuisines overall. I [16:58] want to blot the distribution of [16:59] cuisines that has certain characters. I [17:01] want to do that for every other friend [17:02] character. I want to blo them next to [17:04] each other. Imagine all these kind of [17:05] analytics that you want to extract. [17:08] Another example here. This is a shark [17:10] tank collection. In this case, [17:12] similarly, you want to extract [17:13] information about interviews that that [17:14] Mark Cuban made, whether he made a deal [17:17] or not, right? What are the other sharks [17:19] that has uh made a deal with him? Uh [17:21] what are the gender of the founders and [17:25] generate a table based on that to [17:27] extract some information, right? So this [17:29] is basically so we aim to have this kind [17:33] of like perplexity like experience for [17:36] video data that support uh knowledge [17:38] graph that is cheap. We have also model [17:40] that uh that even a standard benchmark [17:43] will even million parameter model PG [17:45] GBT40. Uh we have also support of API. [17:49] So this is also accessible today the API [17:51] support for extracting the extros. You [17:54] can think of this if you are a [17:56] developer. You can basically extract the [17:58] segments of shark tag collection where [18:01] basically Mark Cuban is there. Uh and I [18:05] want to know what gender is the like the [18:07] C CEO for example. what did the company [18:09] receive the deal or not? What other [18:12] sharks that were in the deal? And you [18:13] can process it in a way that is like [18:15] this. Um yeah. So basically this is it [18:19] and thank you very