The SQL Layer for Physical and Visual AI With KAUST & CreativAI

Ai4

AI-assisted summary of The SQL Layer for Physical and Visual AI With KAUST & CreativAI — original source. Check the original for full context.

Source published: 2026-08-17

Detailed notes

## Background Published on YouTube by Ai4 on August 17, 2026, this presentation features Mohamed Elhoseiny, Associate Professor and Founder at KAUST and CreativAI. The video's description outlines the concept of an SQL layer for physical and visual AI, presenting a unified interface designed to abstract large-scale multimodal streams into structured, database-like assets for fast exploration and analytics.

## TL;DR Mohamed Elhoseiny traces a decade of multimodal AI research, examining how milestones like contrastive pre-training, MiniGPT-4, and spatio-temporal video token compression evolved to handle complex visual data. He argues that despite these algorithmic leaps, current vision-language models remain too verbose and unstructured for enterprise workflows that require deterministic, verifiable operations. To resolve this industrial bottleneck, he presents CreativAI's 'pixels in, structure out' platform, which ingests raw camera feeds and converts them into relational databases. This SQL-like operational layer supports dynamic schema expansion, conversational analytical queries, and automated event dispatching for physical infrastructure and robotics.

## Atmosphere & style The presentation carries the poised, deliberate cadence of an academic lecture while steadily building toward an ambitious entrepreneurial pitch. Elhoseiny establishes an earnest, reflective tone as he candidly describes 'slamming into a wall' when attempting to translate theoretical breakthroughs into industrial reality. As the narrative pivots from technical token-compression diagrams to urgent, multi-camera airport scenarios, the delivery becomes brisk and focused, creating the vivid impression of an engineering puzzle finally being solved with architectural clarity.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=4s) Academic vision-language models frequently stall when transitioning to real-world industrial environments due to a lack of deterministic, structured outputs. - [01:01](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=61s) Contrastive image-language pre-training, first introduced in 2013 for zero-shot recognition, established the core embedding alignment architecture later scaled by models like OpenAI's CLIP. - [03:03](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=183s) Frameworks like VisualGPT and MiniGPT-4 pioneered bridging visual features into LLMs, introducing instructable multimodal reasoning and fine-grained spatial grounding. - [05:26](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=326s) Traditional video RAG relying on pre-computed text summaries discards raw visual features, making fine-grained query-time visual inspection impossible. - [07:45](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=465s) Managing video token explosion across long streams requires spatio-temporal token compression, including invariant background deduplication and query-guided token zooming. - [10:00](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=600s) Enterprise and industrial operations require structured, verifiable database records rather than the verbose, unstructured narratives generated by standard VLMs. - [10:00](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=600s) Over a billion optical cameras monitor global physical operations, but visual data remains largely unprogrammable and siloed in raw pixel formats. - [11:50](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=710s) CreativAI implements a 'pixels in, structure out' paradigm, turning raw live and archival video streams into relational database tables tied to exact timestamps and frames. - [11:50](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=710s) The SQL layer acts as a central cognitive brain for physical environments, instantly structuring incidents to automate workflows and dispatch robotic or human responses. - [14:02](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=842s) Dynamic schema expansion enables iterative visual analytics, automatically appending new columns for relational operations like GROUP BY without re-processing entire video datasets.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=4s) From Academic Vision-Language Research to Industrial Reality

Mohamed Elhoseiny introduces his career journey, reflecting on two decades dedicated to teaching machines to understand visual inputs through vision-language research. Throughout this journey, he describes repeatedly "slamming into a wall" whenever attempting to bridge cutting-edge academic models with real-world industrial and operational environments. To set the stage, he structures his presentation into two distinct acts: first, an overview of the foundational academic and algorithmic breakthroughs over the past decade, followed by an analysis of why current vision-language models fall short of enterprise adoption, ultimately motivating the creation of a dedicated SQL layer for physical and visual AI.

```mermaid flowchart LR accTitle: Evolution of Vision-Language Models to Industrial Query Systems accDescription: Diagram tracing the evolution from Contrastive Pre-training to LLM Bridging, Video Token Compression, and the Structured SQL Layer.

subgraph Foundations ["Foundational Image-Language AI"] A["Contrastive Pre-training (Zero-Shot Recognition)"] --> B["VisualGPT & MiniGPT-4 (LLM Bridging & Grounding)"] end

subgraph VideoScaling ["Video Scaling & Compression"] B --> C["Segment Chunking & RAG"] C --> D["Spatio-Temporal & Query-Aware Token Compression"] end

subgraph Industry ["Industrial Operational Layer"] D --> E["CreativAI SQL Layer (Structured Queries on Visual Data)"] end ```

[01:01](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=61s) The Roots of Zero-Shot Learning and Contrastive Pre-training

The foundational architecture underpinning modern vision-language models traces back to Elhoseiny’s 2013 ICCV paper on zero-shot recognition for unseen classes. To illustrate the concept, he presents an intuitive example: most people might not know what a specific bird species looks like in advance, but if given a linguistic description—such as having a short orange beak, darker plumage on top, and white plumage below—one can mentally synthesize the appearance and recognize the bird upon encountering it. Enabling this capability in machines requires cross-modal inference that directly connects natural language semantics to visual feature spaces.

To achieve this cross-modal alignment, Elhoseiny developed contrastive image-language pre-training in 2013. The algorithm pairs visual data with corresponding natural language descriptions, optimizing representations so that pairs belonging to the same semantic class exhibit high similarity and minimal distance in embedding space, while mismatched pairs are pushed far apart. While initially introduced using linear projections, this contrastive framework served as the direct precursor to modern web-scale models like OpenAI's CLIP, which scaled the core algorithm across massive image-text datasets using deep visual and textual encoders.

- **Zero-Shot Principle**: Recognizing unseen visual classes purely from structured textual descriptions via cross-modal semantic mapping. - **Contrastive Pre-Training Mechanics**: Maximizing embedding similarity for matching image-text pairs while minimizing similarity for mismatched pairs. - **Scaling Vector**: Modern breakthroughs like CLIP scale this fundamental contrastive learning paradigm across deep architectures and internet-scale datasets.

[03:03](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=183s) Bridging Images into LLMs: VisualGPT, MiniGPT-4, and Grounded Vision

While contrastive pre-training maps visual representations into a joint space, it does not inherently yield instructable, conversational intelligence. To bridge visual features into language models, Elhoseiny’s lab introduced VisualGPT in 2022, marking the first framework to bridge visual features directly into a large language model (GPT-2 at the time) to perform image captioning. Because early LLMs were not natively instruction-tuned, these initial architectures focused primarily on descriptive captioning rather than multi-turn dialogue.

Shortly thereafter, the lab developed MiniGPT-4, which became the first instructable vision-language model. The project experienced massive viral adoption, accumulating 10,000 GitHub stars within its first three days and eventually surpassing 26,000 stars. MiniGPT-4 utilized a surprisingly lightweight projection layer to bridge visual encoders with instruction-tuned LLMs, allowing users to input arbitrary images—such as a photo of a meal—and receive actionable, multi-step instructions, like generating a recipe specifying ingredients down to tablespoons of olive oil. Subsequent iterations introduced visual grounding (such as MiniGPT-v2), giving models spatial awareness to localize and bound objects dynamically in the image frame as it generates descriptive text, such as pinpointing the exact coordinates of children playing on a seesaw.

- **VisualGPT (2022)**: Pioneered bridging visual representations into LLMs for conditioned text generation. - **MiniGPT-4 (2023)**: Achieved viral breakout success (10k stars in 3 days, 26k+ overall) by enabling zero-shot instruction following on visual inputs. - **Spatial Grounding**: Integrated object-level coordinate prediction directly into generated natural language streams.

[05:26](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=326s) Scaling Modalities: Video Architectures, Chunking, and Query-Time Limits

Scaling from static images to video introduces massive computational bottlenecks, even though video comprises over 80% of all internet traffic. Elhoseiny’s lab expanded their architecture into MiniGPT4-video, encoding sequences of video frames alongside contextual subtitles. However, processing long-form temporal data—such as a full television episode of *Friends*—reveals fundamental challenges when querying specific narrative moments, such as locating the exact scene where Phoebe and Monica are conversing in a restaurant.

To handle long videos, their ACCV paper introduced a temporal chunking and retrieval-augmented generation (RAG) framework:

```mermaid flowchart TD accTitle: Temporal Chunking vs Query-Native Video Processing accDescription: Flow showing the ACCV segment-chunking RAG pipeline compared to native query-time visual processing.

subgraph ACCV_Pipeline ["Pre-Computed Video RAG Pipeline"] V1["Raw Long Video Stream"] --> S1["Temporal Chunking / Segments"] S1 --> S2["Pre-Generated Text Summaries"] S2 --> DB[("Text Index / Vector DB")] Q1["User Query"] --> RET["Text-based Retrieval"] DB --> RET end

subgraph QueryNative ["Query-Native Native Visual Processing"] Q2["User Query"] --> COMP["Query-Guided Frame & Token Compression"] V2["Raw Video Tokens"] --> COMP COMP --> VLM["Direct VLM Spatial-Temporal Reasoning"] end ```

While effective at extending context windows, this pre-captioning approach suffers from a major architectural limitation: at query time, the system lacks native access to raw visual features. Because the natural language descriptions of video segments are pre-computed prior to receiving the user's question, fine-grained visual details not captured in the initial summary are permanently lost to the model.

[07:45](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=465s) Spatio-Temporal Token Compression and Entity Indexing

Overcoming the query-time disconnect requires models to operate natively on raw video tokens without exceeding context budgets. Standard video encoding generates approximately 144 tokens per frame; at that rate, a 20-minute video consumes 172,000 tokens, and a full 24-hour surveillance feed explodes to 12.3 million tokens. To combat this token explosion, Elhoseiny’s team engineered advanced spatio-temporal compression techniques:

```mermaid flowchart TD accTitle: Spatio-Temporal Video Compression Strategies accDescription: Flowchart illustrating the three compression strategies: diversity selection, temporal background deduplication, and query-aware token pooling.

subgraph CompressionStrategies ["Video Token Optimization"] direction TB A["Raw Video Frame Stream (144 tokens/frame)"]

A --> B["Temporal Diversity Selection (Sample dynamic moments, drop identical frames)"] A --> C["Spatial Background Deduplication (Encode static backgrounds once across time)"] A --> D["Query-Guided Token Zooming (Dense tokens for query regions, sparse for irrelevant)"]

B & C & D --> E["Compressed Visual Token Stream (High efficiency, full visual fidelity)"] end ```

In addition to static background deduplication and query-informed token allocation—detailed in work presented at ECCV featuring a lightweight VLM selector—the researchers integrated entity-level abstraction layers. By indexing moving objects and human actors as structured entities across temporal segments, models can answer precise tracking queries, such as identifying the exact timestamp and location where a user last opened their laptop.

- **Token Explosion Metrics**: Standard 144 tokens/image leads to 172k tokens for a 20-minute video and 12.3 million tokens for a 24-hour feed. - **Spatial Deduplication**: Identifying and compressing invariant background tokens across consecutive frames. - **Dynamic Query Zooming**: Allocating high token density to query-relevant spatio-temporal regions while downsampling background contexts. - **Entity-Level Indexing**: Tracking persistent physical entities across time intervals for exact physical-world recall.

[10:00](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=600s) The Industrial Dilemma: Launching CreativAI and the SQL Layer

Reflecting on the decade-long evolution of vision-language models, Elhoseiny underscores a fundamental barrier: foundational VLMs are fundamentally "verbose," generating expansive, unstructured textual narratives that do not align with the structured, verifiable requirements of industrial enterprise applications. Industrial settings do not need verbose conversational summaries; they need deterministic, structured querying over visual and physical processes. This gap led to the founding of CreativAI to establish a standardized "SQL layer for physical and visual AI."

Elhoseiny introduces the company’s launch video, which captures the scale of the global visual infrastructure:

The presentation highlights that over one billion optical cameras currently monitor real-world environments across hospitals, manufacturing plants, logistics warehouses, and retail spaces. With the rapid deployment of autonomous robots and embodied AI systems, this visual sensor footprint is expanding exponentially. Despite capturing how global physical operations run every second, visual feeds—which represent over 80% of all internet traffic—remain locked in raw, unstructured formats without a programmable interface to query, verify, and automate operational decisions.

[11:50](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=710s) Pixels In, Structure Out: The Vision of CreativAI

Visual and physical AI has long had the ability to observe the world through lenses, but it has lacked a unified cognitive architecture to interpret and coordinate those observations. CreativAI is designed to act as the "SQL layer for physical and visual AI," operating on the foundational principle of "pixels in, structure out." Rather than relying on secondary metadata, human tags, or text captions, the system directly ingests live video streams as they occur, alongside archival footage dating back years. It decomposes raw video into structured relational databases formatted as rows and columns, where every entry links directly back to the exact video frame and moment in time it originated from. This turns disparate feeds from security cameras, aerial drones, autonomous vehicles, and robotic platforms into a synchronized, single source of truth that humans, autonomous agents, and robots can query and operate against simultaneously.

```mermaid flowchart TD accTitle: CreativAI Data Pipeline Architecture subgraph Ingest ["1. Multi-Lens Ingest"] A["Cameras, Drones, Robots & Vehicles"] end subgraph Engine ["2. Structuring Engine"] B["Raw Pixels Analysis"] C["Relational DB: Events, Entities & Behaviors"] end subgraph Act ["3. Unified Action Layer"] D["Humans: Natural Language"] E["AI Agents: REST / Query APIs"] F["Robotics: Real-Time Dispatch"] end A --> B B --> C C --> D C --> E C --> F ```

The practical power of this architecture is highlighted through high-stakes, high-density environments like an international airport—an ecosystem spanning thousands of cameras and tens of thousands of active passengers. CreativAI converts raw visual pixels into actionable structured events the instant an incident unfolds. When a liquid spill occurs in Concourse B, the system immediately generates a structured record containing the substance type, precise location, surface area, and severity level, automatically dispatching a nearby cleaning robot to the site. Similarly, if a passenger collapses near gate B12, the platform logs the individual, the location, and an urgency rating, routing the emergency directly to the nearest airport staff member and elevating it above every other item in the queue.

- **Three Core Jobs**: - **Ingest**: Ingests optical feeds from any device equipped with a lens, including fixed cameras, drones, mobile robots, and vehicles. - **Structure**: Automatically parses raw pixel streams into discrete entities, behaviors, and events as they unfold in real time. - **Act**: Provides an operational query layer where humans communicate in natural language, AI agents query via APIs, and physical robots receive automated instructions. - "Visual and physical AI has eyes. What it never had is a structured brain."

---

[14:02](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=842s) Conversational Visual Analytics and Dynamic Schema Expansion

Whether deployed by individual researchers, software developers, or large enterprise organizations, the underlying platform is built to scale elastically across any camera count and data volume. The infrastructure supports multiple deployment models, running on public clouds, on-premises setups, or isolated private clouds. The core objective is a "structure-first" analytical workflow, allowing users to converse directly with massive camera networks just as they would with a traditional relational database. For instance, an operator monitoring municipal feeds across New York City can ask in plain language for the helmet compliance ratio among cyclists during the day versus at night. The engine autonomously scans the video, identifies the corresponding cyclist segments, and generates a structured data table where the rows represent video snippets and the columns track lighting conditions (daytime vs. nighttime) and whether the cyclist is wearing a helmet, producing the comparative ratio instantly.

```mermaid flowchart LR accTitle: Dynamic Visual Table Expansion A["Query 1: Cyclist Helmet Compliance"] --> B["Initial Table: Segments | Day/Night | Helmet Compliance"] B --> C["Query 2: Intersection Breakdown"] C --> D["Expanded Table: Appends 'Intersection' Column for GROUP BY"] ```

When users iterate with follow-up queries, the system exhibits contextual memory rather than recomputing the visual corpus from scratch. If an analyst subsequently asks how helmet compliance varies between street intersections and non-intersections, the engine recognizes the preexisting table and dynamically appends an intersection column to the schema. This allows operators to immediately run familiar analytical aggregations like `GROUP BY` on the newly enriched dataset. Beyond urban monitoring, this approach applies across domains such as professional sports analytics, where an analyst can provide a photo of Lionel Messi and query how many left-corner kicks versus right-corner kicks he executed across World Cup tournaments. Backed by filed patents on structured-knowledge transformation from visual media, the platform ranks number one on visual retrieval benchmarks, bringing visual computing into parity with standard enterprise SQL workflows.

- **Infrastructure & Scale**: Dynamically auto-scales compute resources based on ingested data volume; supports public cloud, on-premises, and private cloud deployments. - **Dynamic Table Construction**: Converts open-ended video queries into schema-backed rows and columns, enabling subsequent relational operations (`GROUP BY`, filtering, and cross-tabulation). - **Benchmark Standing & IP**: Holds the top ranking on multimodal visual retrieval benchmarks, supported by proprietary patents centering on the transformation of raw unstructured video into SQL-retrievable relational formats.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=4s) slamming into a wall — Mohamed Elhoseiny

> [10:00](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=600s) verbose — Mohamed Elhoseiny

> [10:00](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=600s) SQL layer for physical and visual AI — Mohamed Elhoseiny

> [11:50](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=710s) pixels in, structure out — Mohamed Elhoseiny

> [11:50](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=710s) Visual and physical AI has eyes. What it never had is a structured brain. — Mohamed Elhoseiny

> [14:02](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=842s) structure-first — Mohamed Elhoseiny

> [11:50](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=710s) Ingest — Mohamed Elhoseiny

> [11:50](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=710s) Structure — Mohamed Elhoseiny

> [11:50](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=710s) Act — Mohamed Elhoseiny

## Mentioned ACCV, Ai4, CLIP, CreativAI, ECCV, Friends, Git, GitHub, GPT-2, ICCV, KAUST, Lionel Messi, MiniGPT-4, MiniGPT-v2, MiniGPT4-video, Mohamed Elhoseiny, OpenAI, VisualGPT

## Video description Mohamed Elhoseiny - Associate Professor & Founder, KAUST & CreativAI

The SQL layer for Physical and Visual AI is a unified interface that lets developers query, filter, and retrieve structured insights from large-scale multimodal data such as images, video, and sensor streams. It abstracts the complexity of visual and physical-world data into a familiar query paradigm, enabling fast exploration and analytics without custom pipelines. This turns raw perception data into a programmable, database-like asset for building and scaling AI applications.

[Source](https://www.youtube.com/watch?v=WlZdiBcAzs4)

中文简报

面向物理与视觉AI的SQL操作层：KAUST与CreativAI Mohamed Elhoseiny回顾了多模态AI研究十年的发展历程，探讨了对比预训练、MiniGPT-4以及时空视频Token压缩等里程碑如何演进以处理复杂的视觉数据。他指出，尽管算法取得了这些跨越式进展，但当前的视觉语言模型对于需要确定性、可验证操作的企业工作流而言仍然过于冗长且缺乏结构化。为了解决这一工业落地瓶颈，他展示了CreativAI的“像素输入，结构输出”（pixels in, structure out）平台，该平台可摄取原始摄像头视频流并将其转换为关系型数据库。这种类似SQL的操作层支持动态模式（Schema）扩展、对话式分析查询，以及面向物理基础设施与机器人的自动化事件分发。由于缺乏确定性的结构化输出，学术界的视觉语言模型在向现实工业环境过渡时往往陷入停滞。 (0:04) 对比图文预训练于2013年首次被引入用于零样本识别，它奠定了核心的嵌入对齐架构，后来被OpenAI的CLIP等模型进一步拓展放大。 (1:01) VisualGPT和MiniGPT-4等框架开创了将视觉特征衔接至大语言模型的先河，引入了可遵循指令的多模态推理与细粒度空间定位能力。 (3:03) 依赖预先计算的文本摘要的传统视频RAG会丢弃原始视觉特征，使得查询时细粒度的视觉核查变得无法实现。 (5:26) 应对长视频流中的Token爆炸需要进行时空Token压缩，包括恒定背景去重和查询引导的Token缩放聚焦。 (7:45)

Reading summary

Mohamed Elhoseiny argues that foundational vision-language models fail in enterprise settings because their verbose, unstructured outputs cannot support automated industrial decision-making. To bridge this gap, his research and startup CreativAI introduce an operational layer that converts raw video feeds directly into queryable, relational databases. Evolution from Contrastive Learning to Grounded VLMs (1:01) The conceptual foundation for modern vision-language models originated with Elhoseiny’s 2013 ICCV research on zero-shot recognition, which enabled models to identify unseen classes through linguistic descriptions alone. This framework utilized contrastive pre-training to map image and text embeddings into a shared semantic space—pulling matched pairs together while repelling mismatched pairs—establishing the architectural mechanism later scaled by OpenAI's CLIP across web-scale data. To move beyond static classification into generative language, Elhoseiny’s laboratory released VisualGPT in 2022, connecting visual encoders to GPT-2 for image captioning. In 2023, the team developed MiniGPT-4, using a lightweight projection layer to connect vision encoders to instruction-tuned large language models. This enabled zero-shot instruction following, such as generating ingredient-level recipes from food photos. Subsequent iterations, such as MiniGPT-v2, introduced spatial grounding, allowing the model to predict bounding coordinates for physical objects directly within generated textual narratives. Tackling Video Token Explosion and Query Limits (5:26) Extending vision-language models to temporal data introduces steep computational challenges. Standard video encoding produces approximately 144 tokens per frame, totaling 172,000 tokens for a 20-minute video and scaling to 12.3 million tokens for a continuous 24-hour feed. Initial work addressing long-form video relied on temporal chunking and pre-computed segment summarization indexed in vector databases. However, this retrieval-augmented pipeline strips away raw visual details prior to receiving a user's question, leaving models unable to inspect fine-grained visual evidence at query time. To maintain raw visual fidelity under context constraints, Elhoseiny's team developed spatio-temporal token compression techniques. The system eliminates redundant static backgrounds across temporal sequences, implements query-guided zooming to allocate high token density only to relevant visual regions, and uses entity-level indexing to track persistent physical objects and human actors across discrete time frames. Operationalizing Pixels into Relational Databases (10:00) Elhoseiny contends that while search engines indexed text and relational databases structured tabular data, the physical world remains locked in raw video feeds across more than one billion optical cameras. Foundational models generate descriptive narrative paragraphs, but physical operations require deterministic, verifiable data structures. CreativAI was launched to implement a "pixels in, structure out" paradigm, decomposing live and archival video streams into structured relational databases where every row and column links back to verifiable video timestamps. In operational environments like international airports, this architecture converts continuous camera feeds into autonomous real-time actions. For example, the system can detect a floor spill, record its substance type, exact location, and surface area, and dispatch an automated cleaning robot. Similarly, the platform can register a medical incident, calculate an urgency rating, and prioritize alerts to the nearest operational personnel. Conversational Analytics and Dynamic Schema Expansion (14:02) The platform's analytical interface allows operators to converse with large-scale camera networks using natural language, which the underlying engine translates into structured relational schemas. For instance, an urban analyst querying municipal video can request cyclist helmet compliance ratios across day and night conditions, generating a structured table with light conditions, helmet presence, and corresponding video snippets. When analysts pose iterative questions, the engine preserves context and dynamically expands the existing schema. Asking how compliance changes at street intersections causes the system to append an intersection column to the preexisting dataset, enabling standard aggregations like `GROUP BY` without recomputing the video corpus from scratch. The infrastructure scales across public, private, and on-premises clouds, relying on patented video-to-structure transformations that Elhoseiny reports hold top ranks on visual retrieval benchmarks. CreativAI frames structured relational indexing as the missing operational brain for camera networks and embodied robotics, though its long-term viability hinges on maintaining low-latency token compression and reliable schema generation across complex, multi-camera physical environments.

精炼阅读摘要

Mohamed Elhoseiny 认为，传统视觉语言模型生成冗长且非结构化文本的特征，难以满足工业场景对确定性与可验证性的需求；为此，CreativAI 借鉴十余年多模态与视频压缩技术积累，构建了面向物理与视觉 AI 的专用“SQL 层”，旨在将海量视频流实时转换为标准关系型数据以支持精准分析与自动化调度。从零样本对比学习到指令化视觉模型 (1:01) 现代多模态大模型的底层机制可追溯至针对未见类别的跨模态语义对齐研究。Elhoseiny 在 2013 年提出基于图文配对的对比预训练算法，通过在嵌入空间中拉近匹配图像与文本的距离、推远不匹配项，实现了无需直接样本的零样本识别，该范式随后成为 OpenAI 的 CLIP 等大规模多模态模型的核心基石。为进一步赋予视觉特征以交互对话能力，其团队于 2022 年推出 VisualGPT，首次将视觉特征接入 GPT-2 实现受控图像描述。 2023 年推出的 MiniGPT-4 则通过轻量级投影层将视觉编码器连接至指令微调语言模型，具备零样本图文指令遵循能力，在开源前三天即获得超 1 万星标（累计超 2.6 万）。随后的迭代版本（如 MiniGPT-v2）进一步引入空间定位能力，使模型在生成结构化自然语言的同时，能够动态输出目标物体的精确像素坐标与空间包围框，完成了从单一描述到空间感知推理的跨越。视频Token爆炸与原生特征丢失瓶颈 (5:26) 将视觉模型由静态图像拓展至时序视频时面临严峻的算力约束。在常规视频编码中，每帧图像产生约 144 个 Token，这导致 20 分钟视频需要消耗 17.2 万 Token，而全天 24 小时的监控数据则会膨胀至 1230 万 Token。早期针对长视频的处理方案主要依赖时序分段与检索增强生成（RAG），但由于文本摘要是在查询发生前预先生成的，系统在推理时失去了对原始底层视觉细节的访问能力，极易永久丢失细粒度特征。为突破这一瓶颈，研究团队开发了时空压缩与实体级索引方案。该机制一方面通过时域多样性采样剔除重复帧并消除时序上不变的静态背景，另一方面结合轻量级模型根据用户具体查询在关键区域实施动态 Token 聚焦。配合将运动对象和人物抽离为独立实体的时序追踪机制，系统既能大幅压缩上下文开销，又能精准响应特定时刻与实物交互的细粒度查询。像素进结构出的工业级视觉数据引擎 (11:50) 全球目前有超过 10 亿台光学摄像头分布在工厂、物流仓储、医院与零售场景中，但这些占互联网流量 80% 以上的视频流长期缺乏标准化的可编程接口。不同于通用模型输出非结构化叙述性文本的模式，CreativAI 确立了“像素进，结构出”的核心架构，直接将摄像头、无人机和机器人获取的连续像素解析为具备行列关系的数据表，并将每项指标直接锚定至原始视频帧与具体时间戳。该平台贯穿“摄取、结构化、行动”三大链路，为人类操作员提供自然语言界面，为软件系统提供 REST API，并向自动化设备下发控制指令。以高密度运营的大型国际机场为例，当候机区发生液体泄漏时，系统能即时生成包含物质分类、精确位置、波及面积与严重等级的结构化事件记录，并自主派发邻近清洁机器人执行处置；若旅客发生突发昏迷，系统则会迅速提取关键要素并自动将其提升为最高优先级工单，调度地勤人员前往支援。动态模式扩展与交互式多维分析 (14:02) 该系统支持在公有云、本地及私有云环境中根据吞吐量弹性伸缩，核心优势在于支持针对庞大监控网络的连续交互式分析。管理人员可以使用自然语言下达统计指令，例如分析特定城市路段中白天与夜晚骑行者的头盔佩戴合规率；计算引擎在扫描视频后，会自动生成以视频切片为行、以光照环境与佩戴状态为列的结构化数据集，并即时输出对比统计结果。当分析人员进一步追问路口与非路口的差异时，系统具备上下文记忆能力，无需重新从底层视频展开计算，而是动态向现有数据表追加“交叉路口”字段，允许直接执行类似于 SQL 中的 `GROUP BY` 聚合分组操作。该技术不仅能用于城市治理与交通分析，也可推广至专业体育分析等场景（例如依据照片统计梅西在历届世界杯中的角球分布）。团队指出，该方案已在多模态视觉检索基准测试中取得领先成绩，并为其视觉转结构化数据的核心技术申报了专利保护。 Mohamed Elhoseiny 总结强调，物理与视觉 AI 领域长期具备采集图像的“眼睛”，却缺少将其转化为标准化决策资产的“结构化大脑”。将视频流直接解析为关系型数据为实现自动化操作提供了确定性基础；然而，面对超大规模摄像头并发流的实际部署，如何在持续降低长时序推理成本的同时，确保复杂开放场景中事件抽取与因果关联的绝对精准，仍是该技术由理论验证走向工业常态化运行所面临的核心挑战。

Detailed notes — zh

## Background 本视频由 Ai4 频道于 2026 年 8 月 17 日在 YouTube 平台发布，主讲人为 KAUST 副教授兼 CreativAI 创始人 Mohamed Elhoseiny。视频简介阐述了面向物理与视觉 AI 的 SQL 查询层，它作为统一接口能够从图像、视频和传感器流等海量多模态数据中检索结构化洞察，将原始感知数据转化为可编程的数据库资产。 ## TL;DR 演讲系统梳理了从 2013 年早期零样本跨模态对比学习、VisualGPT 到 MiniGPT-4 及长视频理解的技术演进历程，剖析了长视频原生处理面临的 Token 爆炸与信息固化等瓶颈。面对现有视觉语言模型过于冗长、无法直接支撑确定性工业执行的困境，演讲者推出了 CreativAI 及其专属的“物理与视觉 AI SQL 层”。该系统践行“输入像素，输出结构”的理念，将全球数十亿摄像头捕获的原始视觉流实时转化为结构化数据表，实现了从多源采集、毫秒级事件解析到自然语言与 SQL 查询、API 派单及机器人调度的完整闭环。 ## Atmosphere & style 整场演讲兼具扎实的学术纵深与前瞻的产业开拓气息，主讲人以二十年研究生涯中屡次遭遇的落地壁垒为引子，开篇便奠定了真实而引人深思的基调。叙事节奏从容且层次分明，既有对长视频时空压缩等硬核算法瓶颈的严谨推演，又有借助概念宣传短片展现十亿级镜头互联时的宏大叙事。主讲人沉稳自信的讲解与生动的现实案例（如机场应急派单、交通合规统计）交织，让观众深刻感受到视觉大模型从“会说话的生成器”向“可计算的结构化大脑”跨越的产业变革力量。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=4s) 前沿视觉语言模型在工业落地中面临严重鸿沟，难以直接嵌入以确定性和可执行性为导向的工业流程，促成了物理与视觉 AI 专属 SQL 查询层的构想。 - [01:01](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=61s) 早期的零样本图文对比预训练奠定了 CLIP 等模型的理论基础，而 VisualGPT 则首次验证了将图像表征轻量化桥接至纯文本语言模型的可行性。 - [03:41](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=221s) MiniGPT-4 推动了指令跟随能力在多模态领域的爆发，并进一步引入细粒度空间定位（Visual Grounding），实现了逐字生成描述时的精确坐标标定。 - [05:20](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=320s) 长视频处理中预先切片生成文本摘要再检索的 RAG 方案存在固有缺陷，在查询阶段会丢失对特定问题至关重要的底层原生视觉细节。 - [07:43](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=463s) 长视频全量编码会引发严重的 Token 爆炸（24 小时监控可达 1230 万 Token），必须借助多样性筛选、查询自适应缩放与背景去重等时空压缩手段大幅降低计算负载。 - [07:43](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=463s) 通过引入实体级索引抽象与轻量级模型剪枝，系统能够快速定位关键事件的时空区间，大幅提升复杂视频的理解与检索效率。 - [10:00](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=600s) 全球数十亿视觉终端正在持续记录现实世界的运转，但非结构化的原始像素长期处于沉睡状态，缺乏将其转化为可直接指导行动的结构化数据流手段。 - [11:50](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=710s) 现有大模型偏向于生成冗长的大段自然语言描述，而 CreativAI 的核心定位是实现“输入像素，输出结构”，将视频数据直接转化为标准数据表行列。 - [11:50](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=710s) 基于“采集-结构化-行动”架构，系统可在毫秒级提取事件、实体与行为，支持人类自然语言查询、AI Agent 接口调用及机器人自动化调度执行。 - [14:02](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=842s) “结构化优先”设计允许用户用日常对话向海量摄像头提问，并支持基于对话上下文自动扩展数据列以进行多维度的 Group By 聚合统计分析。 - [14:02](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=842s) 将非结构化视频流转化为支持类 SQL 检索的标准资产，能够无缝复用企业在关系型数据库领域沉淀数十年的工具生态与成熟工作流。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=4s) 从学术探索到工业落地：视觉语言模型的演进困境演讲者 Mohamed Elhoseiny 回顾了自己过去二十年在计算机视觉与多模态领域的学术生涯。他长期致力于教会机器理解视觉世界，但在尝试将前沿的视觉语言（Vision-Language）研究成果对接到工业实际应用时，却屡屡碰壁，仿佛“一直撞在一堵无形的墙上”。本场演讲的核心脉络正是探讨如何将学术界多模态基础模型的进展，真正转化为高密度视频行业在运营卓越性（Operational Excellence）上的生产力。学术界的突破固然令人振奋，但在面对企业级和工业化部署时却显得力不从心。这一现实鸿沟直接催生了为“物理与视觉 AI”（Physical and Visual AI）打造专属 SQL 查询层的构想。接下来的内容将分为两部分：首先系统梳理视觉语言模型的底层学术演进，进而深入剖析现有模型在工业落地中暴露出的根本缺陷。 --- ### [01:01](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=61s) 视觉语言跨模态基础：从零样本对比学习到 VisualGPT 现有多模态大模型的技术底座可以追溯到演讲者在 2013 年发表于 ICCV 的一篇经典论文。当时该研究聚焦于零样本学习（Zero-Shot Recognition）中的未见类别识别问题。演讲者以一种名为长尾鹦鹉（Parakeet）的鸟类为例：即便观众从未见过这种鸟，但只要给出文字描述——“拥有短橙色鸟喙、羽毛深色且下腹为白色”，人类大脑就能构想出其形态并在现实中准确识别。这种跨模态推理要求算法必须建立语言与视觉表征之间的关联映射。 ```mermaid flowchart LR accTitle: 跨模态对比学习与视觉语言桥接演进 subgraph S1["2013 对比学习基础"] A["图像输入"] --> C["计算特征相似度 / 距离"] B["文本描述"] --> C end subgraph S2["CLIP 规模化演进"] C --> D["大规模图文对 + 深度编码器"] end subgraph S3["LLM 跨模态桥接"] D --> E["VisualGPT (2022) 图像字幕"] E --> F["MiniGPT-4 (2023) 指令跟随"] end ``` 为了实现跨模态对齐，演讲者早在 2013 年便提出了图文对比预训练算法的核心思想：若图片与其物种描述相匹配，则拉近二者的特征距离、提高相似度；反之则推远距离。这一开创性工作正是后来风靡业界的 CLIP 模型的理论基石。CLIP 本质上是将这种对比算法在海量图文对与更深的神经网络上进行工程化缩放（Scaling），使得视觉特征可以直接投影映射到大语言模型（LLM）的表示空间中。在此基础上，团队在 2022 年推出了 VisualGPT，成为首个将图像信号成功桥接到纯文本语言模型（如当时的 GPT-2）的算法架构，初步打通了图像描述生成（Image Captioning）的通路。 - **零样本识别（Zero-Shot Recognition）**：通过自然语言属性描述建立未见类别的视觉假设。 - **对比图文预训练（Contrastive Pre-training）**：优化同类图文嵌入的高相似度与异类图文的远距离度量。 - **VisualGPT（2022）**：早期探索性架构，训练轻量化桥接层以赋予纯文本 LLM 理解单张图像并生成字幕的能力。 --- ### [03:41](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=221s) 指令级多模态爆发：MiniGPT-4 与空间定位能力的引入随着大语言模型指令微调技术的成熟，团队在 VisualGPT 之后迅速推出了开源界极具影响力的 MiniGPT-4。作为最早具备指令遵循（Instructable）能力的视觉语言模型之一，MiniGPT-4 凭借其极其简洁而高效的跨模态桥接机制引发了广泛关注——在开源发布的前三天内便斩获了 10,000 颗 GitHub Star，随后迅速攀升至 26,000 颗 Star。用户只需给模型上传一张美食照片并提问“如何在家做这道菜？”，模型就能有条不紊地列出食材清单与制作步骤：“准备龙虾、两汤匙橄榄油等”。随后，研究团队进一步增强了模型的细粒度空间定位能力（Visual Grounding）。不仅能生成全局语言描述，模型还能在输出文字的同时指出画面中各个物体所在的具体坐标位置。例如当画面中呈现一个男孩和一个女孩在玩跷跷板、且女孩的位置高于男孩时，模型在逐字生成描述的过程中能精准标定出“男孩在哪里”以及“女孩在哪里”，为图像级多模态交互打下了坚实的技术基础。 --- ### [05:20](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=320s) 视频模态的挑战与长视频检索优化然而，仅支持单张图像的多模态能力远不足以应对真实世界的需求。视频占据了当今互联网数据流量的 80% 以上，是信息承载的核心模态。为了支持视频输入，团队研发了 MiniGPT-4-Video，将多帧图像序列与字幕（Subtitles）信息统一联合编码。但这种方式在处理诸如整集《老友记》（Friends）这样的长视频时遭遇了严重瓶颈：长视频中包含大量错综复杂的情节分支与主题转换。 ```mermaid flowchart TD accTitle: 长视频处理与动态检索架构 subgraph S1["视频预处理"] A["长视频流输入"] --> B["视频分段切片 (Chunking)"] B --> C["生成各分段语言摘要"] end subgraph S2["早期检索方案 (ACCV)"] C --> D["文本 RAG 检索"] D --> E["定位目标片段 (如：菲比与莫妮卡在餐厅)"] end subgraph S3["局限性反思"] E -.->|"痛点: 预生成文本丢失原生视觉细节"| F["原生多模态查询时间计算需求"] end ``` 在发表于 ACCV 的工作中，团队提出了一种针对长视频的切片索引策略：将长视频切分为多个短片段，预先为每个片段生成文本描述，随后在上层构建类似 RAG（检索增强生成）的检索系统。比如当用户询问“在对话过程中，菲比在什么时候向莫妮卡展示了某件物品？”，系统无需逐秒扫描全局，而是直接定位到两人在餐厅交流的具体片段。但这种“分段生成字幕再检索”的方案存在致命缺陷：在查询发起时（Query Time），模型无法原生访问底层的视觉信号，因为文本描述在用户提问前就已经固化生成，极易丢失对特定查询至关重要的视觉细节。 --- ### [07:43](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=463s) 时空压缩与实体感知：突破视频 Token 爆炸瓶颈长视频原生处理面临的最严峻挑战是“Token 爆炸”。按照标准视觉编码标准，每张图像约消耗 144 个 Token。一场仅 20 分钟的学术演讲视频就需要处理约 172,000 个 Token；而一段长达 24 小时的监控视频，Token 总量将飙升至令人望而却步的 1230 万。如果全量输入大模型，计算开销将呈指数级失控。 ```mermaid flowchart TD accTitle: 视频时空动态压缩机制 A["高帧率原始视频输入"] --> B{"多维度自适应压缩"} subgraph Compress["时空压缩策略"] B --> C["多样性筛选 (Diversity Selection)"] B --> D["查询自适应缩放 (Query Zoom-in / Zoom-out)"] B --> E["空间背景去重 (Spatial Token Deduplication)"] end Compress --> F["小体量 VLM 引导剪枝 (ECCV 方案)"] F --> G["实体级索引构建 (Entity-Level Indexing)"] ``` 为了解决算力与上下文窗口的限制，团队研发了一系列时空自适应压缩技术： - **多样性帧筛选**：基于画面表征的多样性剔除高度冗余的连续相似帧。 - **查询相关性动态缩放**：根据用户 Query 的关注点，对相关性高的片段和 Token 分配高分辨率细节（Zoom-in），对无关背景片段进行稀疏化降采样（Zoom-out）。 - **空间背景去重**：以固定机位的演讲场景为例，演讲者的背景在长时间内几乎保持不变。算法会识别并追踪这种时空静止区域，仅对背景 Token 编码一次而非持续重复编码。 - **端到端轻量模型剪枝（ECCV 最新成果）**：利用小型视觉语言模型在查询引导下对各个片段进行动态 Token 筛选，仅保留关键特征送入主模型。 - **实体级索引抽象（Entity-Level Indexing）**：在视频片段上抽象出实体层级标签。当用户提问“你上次是在哪里打开笔记本电脑的？”时，系统能依据实体级索引快速锁定事件发生的精确时空区间。 --- ### [10:00](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=600s) 工业级痛点与 CreativAI 愿景：物理与视觉 AI 的 SQL 层尽管学术界在长视频理解与 Token 压缩方面取得了突破，但这依然无法直接满足工业与企业级落地的严苛要求。现有的视觉语言模型本质上是“冗长而啰嗦的生成器（Verbose）”，擅长生成大段自然语言描述，却无法直接嵌入到以确定性、可执行性为导向的工业流程中。为了彻底打破这一壁垒，演讲者创立了 CreativAI，致力于打造面向物理与视觉 AI 的结构化查询层（SQL Layer）。演讲现场播放了 CreativAI 的官方发布短片，揭示了全球正在经历的物理感知数据爆炸： - **十亿级视觉终端部署**：在此时此刻，全球的医院、工厂、仓储中心和零售门店中运行着超过 10 亿个摄像头；随着机器人与具身智能（Embodied AI）的快速普及，物理空间中的感知镜头数量还在以更快的速度攀升。 - **数据孤岛与行动断层**：“十亿个镜头记录着现实世界的实际运转，但没有任何数据转化为可以被直接行动采纳的依据。”工人持续劳作，摄像头持续拍摄，机器人持续移动，但这些海量物理事件从未被统一转化为可计算、可结构化的数据流。 - **数据类型的历史演进规律**：正如发布视频所强调——文档数据催生了搜索（Search），表格数据孕育了 SQL，代码数据成就了 GitHub。每一种混乱复杂的数据形态最终都通过结构化工具实现了“可查询、可验证、可执行”。 - **视觉数据的长期缺位**：作为人类创造的最大数据类型、占据互联网流量 80% 以上的视觉数据，至今仍处于原始、非结构化且封闭在底层存储中的瘫痪状态。CreativAI 的使命正是为这些物理与视觉数据构建一套统一的查询与分析层，使其从沉睡的原始像素转变为能够被现代企业直接编程与调用的数据资产。 ### [11:50](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=710s) 物理与视觉 AI 的 SQL 层：从原始像素到结构化中枢演讲视频展示了 CreativAI 的核心定位——为物理世界与视觉 AI 打造专属的“SQL 层”。系统的核心理念是“输入像素，输出结构”（Pixels in, structure out）。面对摄像头、机器人和人员协同作业的复杂物理场景，CreativAI 并非依赖简单的字幕或元数据，而是直接将摄像头所捕获的画面本身结构化为标准的数据表行列。无论是实时发生的视频流，还是企业沉淀多年的历史存档，都能被统一转化为可索引、可查询的知识库。当所有终端都从这一统一样本中读取数据时，整个组织系统便能如同一体般高效协同。以全球运营最为复杂的场景之一——机场为例，数以万计的旅客与数千个摄像头同时运作，各种突发状况瞬息万变。CreativAI 能够在事件发生的瞬间将原始像素转化为结构化记录：B 候机厅出现液体泄漏，系统立即提取出物质类型、具体位置、覆盖面积以及严重程度，清洁机器人随即自动前往处理；若在 B12 登机口有旅客晕倒，系统会迅速判断人员身份特征、地点与紧急级别，优先将其派单给距离最近的机场员工。每一行数据均能精确回溯至其产生的视频瞬间。 ```mermaid flowchart TD accTitle: CreativAI 物理与视觉 AI 架构流程 subgraph Ingest["1. 采集 Ingest"] A["多源镜头输入 (摄像头 / 无人机 / 机器人 / 车辆)"] end subgraph Structure["2. 结构化 Structure"] B["原始像素转换 (事件 / 实体 / 行为)"] C["结构化数据表 (行与列 / 关联视频片段)"] end subgraph Act["3. 统一行动 Act"] D["人类团队 (自然语言查询)"] E["AI Agent (API 接口调用)"] F["机器人与硬件 (自动化调度执行)"] end A --> B B --> C C --> D C --> E C --> F ``` 整个系统的作业流分为三个清晰步骤： - **采集（Ingest）**：兼容任何带有镜头的设备，包括固定摄像头、无人机、移动机器人及车辆。 - **结构化（Structure）**：在事件发生的毫秒级瞬间，将原始像素解构为事件、实体和具体行为，生成带有时空索引的结构化行与列。 - **行动（Act）**：构建通用的可查询中枢，支持人类以自然语言查询、AI Agent 通过 API 检索，以及机器人直接读取指令执行动作。正如演讲中所强调的：“视觉与物理 AI 早已拥有了双眼，但它们此前从未拥有过一个结构化的大脑。”CreativAI 所构建的正是连接摄像头感知与团队/机器人行动的这一关键桥梁。 --- ### [14:02](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=842s) 结构化优先查询与应用场景落地 Mohamed Elhoseiny 进一步阐述了该技术的底层架构设计与灵活性。无论是个人开发者构建下一代创新应用，还是大型企业释放视觉数据资产的商业价值，该系统均支持弹性自动扩缩容，能够根据处理的数据规模自适应扩展，并支持公有云、私有云以及本地（On-premises）部署。系统最大的突破在于贯彻了“结构化优先（Structured-first）”的设计思维，使用户能够直接用日常对话的方式向摄像头群提问。例如，在接入纽约市交通摄像头的场景中，管理人员可以直接提问：“请统计骑自行车人群在白天与夜间的头盔佩戴合规率对比。”系统在理解语义后，会自动检索所有相关视频片段并生成一张数据表：每一行代表一段自行车骑行片段，列分别记录时间段（白天/夜间）以及是否存在未佩戴头盔的违规行为，进而直接计算出合规比例。如果用户紧接着追问：“在十字路口与非十字路口，这一合规率的发生频率如何？”系统能理解先前的上下文，并在已有数据表基础上自动扩展新的一列（是否位于路口），从而支持用户直接基于路口属性进行分组统计（Group By）。 - **跨领域通用性**：在体育分析领域，系统同样游刃有余。例如输入一张梅西的照片，可以直接查询“梅西在世界杯比赛中主罚左侧与右侧角球的次数与分布”。 - **性能与专利优势**：CreativAI 在视觉数据检索相关的多项基准测试（Benchmarks）中均位列第一，其基于结构化知识（Structured Knowledge）的技术路线已申请核心专利保护。 - **与企业级技术栈融合**：企业界在关系型数据库和 SQL 上沉淀了数十年成熟的管理工具与工作流，将视觉数据转化为支持 SQL 检索的标准资产，彻底打通了非结构化视频流与现有企业数据库体系的壁垒。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=4s) 一直撞在一堵无形的墙上 — Mohamed Elhoseiny > [01:01](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=61s) 拥有短橙色鸟喙、羽毛深色且下腹为白色 — Mohamed Elhoseiny > [03:41](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=221s) 如何在家做这道菜？ > [03:41](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=221s) 准备龙虾、两汤匙橄榄油等 > [05:20](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=320s) 在对话过程中，菲比在什么时候向莫妮卡展示了某件物品？ > [07:43](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=463s) 你上次是在哪里打开笔记本电脑的？ > [10:00](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=600s) 冗长而啰嗦的生成器（Verbose） — Mohamed Elhoseiny > [10:00](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=600s) 十亿个镜头记录着现实世界的实际运转，但没有任何数据转化为可以被直接行动采纳的依据。 — CreativAI > [11:50](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=710s) 输入像素，输出结构 — CreativAI > [11:50](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=710s) 视觉与物理 AI 早已拥有了双眼，但它们此前从未拥有过一个结构化的大脑。 — CreativAI > [14:02](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=842s) 请统计骑自行车人群在白天与夜间的头盔佩戴合规率对比。 > [14:02](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=842s) 在十字路口与非十字路口，这一合规率的发生频率如何？ > [14:02](https://www.youtube.com/watch?v=WlZdiBcAzs4&t=842s) 梅西在世界杯比赛中主罚左侧与右侧角球的次数与分布 ## Mentioned Mohamed Elhoseiny, CreativAI, KAUST, Ai4, ICCV, CLIP, GPT-2, VisualGPT, MiniGPT-4, MiniGPT-4-Video, ACCV, ECCV, 老友记 (Friends), GitHub, SQL, 梅西 (Lionel Messi) ## Video description Mohamed Elhoseiny - Associate Professor & Founder, KAUST & CreativAI The SQL layer for Physical and Visual AI is a unified interface that lets developers query, filter, and retrieve structured insights from large-scale multimodal data such as images, video, and sensor streams. It abstracts the complexity of visual and physical-world data into a familiar query paradigm, enabling fast exploration and analytics without custom pipelines. This turns raw perception data into a programmable, database-like asset for building and scaling AI applications. [Source](https://www.youtube.com/watch?v=WlZdiBcAzs4)

Source quotes

[0:04] Mohamed Elhoseiny: 一直撞在一堵无形的墙上 [1:01] Mohamed Elhoseiny: 拥有短橙色鸟喙、羽毛深色且下腹为白色 [3:41] 如何在家做这道菜？ [3:41] 准备龙虾、两汤匙橄榄油等 [5:20] 在对话过程中，菲比在什么时候向莫妮卡展示了某件物品？ [7:43] 你上次是在哪里打开笔记本电脑的？ [10:00] Mohamed Elhoseiny: 冗长而啰嗦的生成器（Verbose） [10:00] CreativAI: 十亿个镜头记录着现实世界的实际运转，但没有任何数据转化为可以被直接行动采纳的依据。 [11:50] CreativAI: 输入像素，输出结构 [11:50] CreativAI: 视觉与物理 AI 早已拥有了双眼，但它们此前从未拥有过一个结构化的大脑。 [14:02] 请统计骑自行车人群在白天与夜间的头盔佩戴合规率对比。 [14:02] 在十字路口与非十字路口，这一合规率的发生频率如何？ [14:02] 梅西在世界杯比赛中主罚左侧与右侧角球的次数与分布

Source quotes

[0:04] Mohamed Elhoseiny: slamming into a wall [10:00] Mohamed Elhoseiny: verbose [10:00] Mohamed Elhoseiny: SQL layer for physical and visual AI [11:50] Mohamed Elhoseiny: pixels in, structure out [11:50] Mohamed Elhoseiny: Visual and physical AI has eyes. What it never had is a structured brain. [14:02] Mohamed Elhoseiny: structure-first [11:50] Mohamed Elhoseiny: Ingest [11:50] Mohamed Elhoseiny: Structure [11:50] Mohamed Elhoseiny: Act

Original transcript

[0:04] I spent my career teaching machines to uh understand what they see in the case of [0:09] vision language research and the the whole time I kept slamming into a wall that when [0:14] it comes to connecting these two industrial applications. So I will first cover a bit [0:19] about research I've been doing over the over the past two uh decades and then transition uh from as [0:26] uh as the slides saying uh how we convert the vision language foundation program [0:33] uh models of progress from academia to real uh progress in uh operational excellence in video [0:39] heavy industries. Um so I'm going to talk in that perspective. The first station will be [0:44] mostly academic and technical work and the other part is is explaining why this is not enough from [0:50] industrial adoption perspective and that's that's the that's the motivation behind the [0:55] sec what we call the secl layer of physical and visual AI let's talk a bit about the background [1:01] of how existing uh vision language foundation model works the key foundations this back to [1:07] uh like actually a paper I wrote in 2013 in ICV one of the conference in computer vision [1:13] uh where I it was initially introduced as the recognition problem in zero shot for recognizing [1:19] unseen classes. For example, maybe most of you may not know what perkid is but if I describe that in [1:25] language and say that is a short orange beak is a bird that has a short orange peak and the blemish [1:30] is darker and white below. From that description you may imagine how this bird may look like and [1:35] then based on that you can it it becomes easy for you to recognize this bird even if you haven't [1:40] seen it before. In order to enable this to happen, you need to c you need crossmodel inference. [1:45] You need to understand language and related to computer vision. In order to do that, you need to [1:50] uh develop an algorithm called contrastive image language re training. So I introduced this paper [1:55] uh back in 2013. And the key idea is to have an image uh an image and a corresponding language [2:01] description. In this case, we have for every bird a language descri language description that [2:06] describe the species and image that correspond to that species. If the image and the language [2:11] the description correspond to the same class you want the similarity to be high and distance to [2:16] be small and if otherwise you want them to be to be distant and this is the foundation what [2:21] called what's called contrastive language and image pre-training if some of you heard about [2:26] clip clip basically is is scaling this algorithm instead of having a linear linear version of this [2:33] it is just has been scaled on large amount of uh image text pairs and deep deeper image and [2:39] text models which enable learning the underlying visual and and and image representation. So this [2:44] is basically the enabling layer that enables image to be mapped to the LM space but this doesn't give [2:51] us instructible LMS. So this gives me gives us an ingredient that enables LMS to be bridged [2:57] into the language space and that motivates other uh technological innovation. I'm sharing also you [3:03] uh visual GPT. This is a paper also we introduced back in 2022 and it was the first adaptation the [3:09] first algorithm that enables images to be bridged into a large language model at that time GBT2 [3:15] which was a language only model but we wanted to train a layer a train the bridging mechanism from [3:22] the image to the LLM space so that the LLM can operate on image data. In that context it was [3:29] uh for image captioning and at that time like these models were not good enough for following [3:34] instructions. So the focus has been on image captioning. Um not not long later like not much [3:41] time later we introduced many GBT4. This is the first instructible uh vision. um it reached the [3:50] uh 26,000 stars 10 reached 10,000 star in the first three days when we uh posted it and it it [3:56] was surpris surprisingly very simple mechanism that bridges a an unstructable LLM. So we there [4:03] were there was at that time language only model but it's they cannot operate on visual data. So [4:09] this models has been enabled at that time for for the first time to give it a picture of your meal [4:15] for example like this and ask it how can I make it at home and it and then it it produced text like [4:21] bring uh for lobster two two tablespoons of olive oils and so on and a lot of progress has been made [4:29] since then but that was a key key u enabling aspect that enables large language model to [4:35] operate on visual data having having a realigned language representation image representation [4:40] that is aligned with language and bridging that's through an LLM. Um this made we made since then a [4:47] lot of progress. We developed um as a conversion that that made the uh the access to that enables [4:56] the system also to uh ground information in the image. So for example when you ask it to produce [5:02] a a language description of a of an image it then can tells you where each each object is. So if you [5:08] can imagine a picture of a person on a boy and a girl on a seesaw where the girl is higher than the [5:14] boy, it tells you where the boy is as it speaks and where the girl is as it speaks. Uh that this [5:20] gives us a good uh foundation for most image LLMs. But this is not sufficient if you are operating on [5:26] a large v bigger modality like videos which uh constitutes 80% of internet traffic. It's very [5:33] important modality. So how can we adapt this? Uh uh we need to do develop progress on enabling this [5:40] uh large language model to operate on um visual encoders uh in video encoders. And that was a [5:48] motivation of the mini GPT for video uh which uh in instead of uh encoding only one frame in [5:56] which corresp it it it encodes instead um a set of [snorts] um frames that corres as as well as the [6:05] uh subtitles uh in in a way that enables you to to process the video. But this is still an issue [6:14] that if you have a longer video like a for example friends episode or something like that you this [6:20] can produce really a lot of topics. So we need to develop better a way to represent the information [6:26] in the video so that you can ask questions where like this where does Phoebe show Monica when [6:32] um when they were talking and in order to that this question does not relate to all segments [6:37] in the video. There is a specific moment where Phoi and Monica were in a in in a restaurant or [6:42] something like that and um in order to do that you need this one's yours for a Christmas tree. [6:50] So in order to do that we need to chunk the video into segments and for every one of these segments [6:55] the system the the paper at that time at ACCV was producing a language description of every segment [7:01] and then you have a raglike solution on top of this and that enabled us to process really wrong [7:05] videos for the first time. Um but video and and and we there are results supports that compared to [7:12] the previous approaches but there the underlying problems in video is much more than that because [7:18] there's a limitation here in this method is that at query time uh you don't have access native [7:24] access to the visual data. Uh because the language description for every segments it get pre-produced [7:30] before it has seen the query. So we need to do better than that. We need to have at time systems [7:36] that can operate natively on the visual data and this requires us to process process the visual [7:43] data in a way that is a bit more smarter. If we look at the standard way to encode a video there [7:49] are 144 tokens per image. If you want to process 20 minute video it's just length of this talk is [7:55] 172,000 tokens. One day video is 12.3 million tokens. If you we want to get this further, [8:01] you can imagine the problem will escalate. Uh so in order to do that, we developed techniques that [8:08] can do some special temporal uh compression uh by selective selecting the images that you want [8:15] to encode based on how diverse they are. They representing the video uh they are based on how [8:21] related they are to the query query. For example, frames and segments that are and tokens that are [8:28] more related to the query are more uh zoomed in and others ones are are zoomed out like mean [8:35] presented with less tokens. And the last thing is we we call special uh a special token compression [8:42] which means that if I have a background of of if I'm speak as I'm speaking right now my background [8:48] does not change and hence the if I track it over time this token is keeps keeps repeating. So how [8:56] about I represent this token only once instead of keep repeating it so so that I compress the [9:00] information in the video and hence the system becomes more token uh efficient. So there are [9:05] we have developed uh I think I hope that you get the point. We have even smarter compressions like [9:10] right now we have a paper at the upcoming ECV in September one of the top conferences in computer [9:15] vision that does this does this even in a more smarter way. So we have a smaller vision language [9:20] model that compress the information for in a query uh informed way for every uh segment so that every [9:27] se segment depending on how relevant it is there's a token it selects the tokens that needs to be [9:32] processed. Um so and and so on. There are also other techniques that can perform some layers of [9:38] abstraction producing entities over video segments so that when you ask questions where did you last [9:44] open your laptop or something like that you have entity level indexing of the information of the [9:49] video and this enables the model to be much more entity aware and hence the performance of [9:54] the model become uh better. So the the long story short I wanted to share with you is that maybe a a [10:00] quick recap a quick uh um recap of the journey of of that has worked on over the past uh decade uh [10:10] in vision language model and also to share with you that this is not sufficient for industrial [10:16] adoption or enterprise adoption and that led to the creation of what what I referred to today as [10:22] a creative AI which is what we call as aql for physical and visual AI vision language model [10:29] are verbose. we produce a lot of text but what not in a way that relate to industrial adoption [10:34] if you if we uh if we say that uh I want to um basically start this by sharing a 3minut video [10:43] uh that represent the launch of uh of of the of the company and it it it tells a story bit [10:51] uh quite concisely and I will share this one and I will comment on it right now at this [10:56] exact second there are over a billion cameras as on Earth in hospitals, factories, warehouses, [11:03] and stores. And as robots and embodied AI move into those same spaces, the numbers climbs faster. [11:14] A billion lens is capturing how operations actually run every day and none of it becomes [11:19] something that anyone can act on. The people work, the camera sees, the robot moves, and none of it [11:25] happens as one bite. Every kind of data got its breakthrough. Documents got search. Tabular data [11:31] got SQL. Code got GitHub. Each one gave messy data a structure. Something you could query, [11:37] verify, and act on. Visual data never got that. The largest data type we produce, [11:44] over 80% of internet traffic still raw, [music] still unstructured, still locked inside the [11:50] pixels. That's why we built Critivi, the SQL layer for physical and visual AI. Pixels in, [11:57] structure out. Visual data finally structured [music] one connected picture your cameras, [12:03] robots and people all work from. People ask what makes us different. We structure what the camera [12:09] sees into rows and columns. The video itself, not the captions, not the metadata, live streams as [12:15] they happen, archives as you have had for years. All of it becomes a knowledge base that you can [12:20] produce and query. And when everything reads from that knowledge base, it starts moving as one. [12:26] An airport, one of the most complex operations in the world. Tens of thousands of people, thousands [12:32] of cameras, [music] everything happening at once. Creative AI turns raw pixels into structure the [12:38] instant something happens. A spill in concourse B. Substance, location, area, severity, and [music] a [12:45] cleaning robot is already on its way. A passenger collapses at gate B12. [music] Whom, where, how [12:52] urgent. routed to the nearest staff member and prioritized above everything else in the queue. [12:58] Every row points back to the moment it came from three jobs [music] ingest from anything with a [13:05] lens, cameras, drones, robots, vehicles. [music] Structure raw pixels into events, [13:12] entities, and behaviors the [music] moment they happen. ACT, a queryable layer that humans, [13:18] AI [music] agents, and robots can all read from. Humans query it in plain language. [music] AI [13:24] agents query it through APIs. Robots act on it. One reliable source of truth. Humans, [13:30] agents, and robots working as one team. This is ambient intelligence for the physical world. A [13:36] structured understanding moving into every space. One shared record acting and sensing [13:41] in the background. Visual and physical AI has eyes. What it never had is a structured brain. [13:47] Something that carries what the camera sees to the robots and the team that act on it. [13:51] That's the layer we're building and with it people and robots finally move as one. [14:02] So as we k I will just emphasize with the uh technology here uh whether you are individual [14:08] exploring AI or developer building the next generation application in or or an enterprise [14:14] unlocking the value of visual AI. uh the system is aimed to make it make me be flexible and to [14:20] scale no matter how many cameras you are. The infrastructure itself is scaled automatically [14:25] depending on the volume of data you need to process and also can be deployed on cloud [14:29] in on on your on premises on private cloud as as well and if you are interested or uh happy [14:36] to uh to help there are more um like um there are more sorry what I want to share is that the [14:45] system itself aims to structure the knowledge in the visual data. So imagine for example you have [14:51] um uh you have New York City cameras. You can basically converse with your cameras and ask it [14:57] tell me the helmet compliance ratio in daytime versus nighttime in for people riding bicycle [15:03] and the system will understand this and from that finds all the segments by itself. It syncs [15:08] in a structured first way. uh find people who are riding uh bicycles and tell and and [15:13] basically produce a table whose rows are those segments and columns are basically whether they [15:18] are happening in daytime or night time and the second column would be the what hel if there's [15:23] helmet compliance issues and based on this table is it's straightforward to produce the helmet [15:28] compliance ratio for daytime or night time and maybe you can ask the system another follow-up [15:33] question for example I want to know how frequent this does this happen in an intersection versus [15:38] non-intersection So the system will understand that you have previously produced that table [15:43] and will produce an additional columns. Uh in this case it will just tell you if this is at [15:47] an intersection or not and hence you can group by again the helmet compliance statistics based on [15:53] whether it's at an intersection or not. Um that's that's basically there are a lot of [15:58] uh examples that are in our website in sports you can basically ask it how many uh times does does [16:04] this person Messi for example give it a picture of Messi did uh left versus right corner kicks [16:10] in the world cup or something like that. uh you can ask it other like questions like uh [16:16] uh like this. Uh the we are number one in benchmarks and like retrieval compared to existing [16:22] u uh u models existing works and we are the only solution that is based on the structured knowledge [16:30] concept which because this is part of the patent we we filed uh that that focuses on how we make [16:36] this system. We have invented SQL for many years and enterprise can adopt a lot of data if it's [16:42] in that structured format. So why don't we make visual data structured in a way that makes it easy [16:49] to integrate with existing structured database systems. And this system literally does enable [16:55] the production or the transformation of visual data into SQL uh um like like uh it makes it [17:03] basically SQL retrievable. That's it. Thank you so much for your attention. [applause]