MIT's Fan Embeds Math Barriers in AI to Prevent Robot Crashes
Stanford Online
Autonomous systems fail under standard reinforcement learning because soft penalties prompt agents to freeze or crash, requiring hard neural barrier functions and active Bayesian falsification to guarantee real-world safety.
AI-assisted summary of Stanford Robotics Seminar ENGR319 | Autumn 2026 | Bridging Rigorous Methods with Machine Learning — original source. Check the original for full context.
Source published:
Key ideas
- MIT researcher Chuchu Fan presents a closed-loop framework uniting formal synthesis (designing controllers with hard mathematical guarantees) and Bayesian falsification (actively uncovering rare failure modes). (5:06)
- Neural constraint value functions formulated as Graph Control Barrier Functions (GCBFs) enable purely simulation-trained decentralized multi-agent policies to scale zero-shot to 1,000 agents and transfer directly to real hardware. (14:41)
- Falsification is reframed from brittle optimization into Bayesian MCMC sampling over energy-based distributions, cutting testing evaluation costs tenfold across spaces reaching 250,000 dimensions. (39:07)
- Retrospective failure analysis of the December 2022 Southwest Airlines meltdown isolated airplane reserve depletion at warm-weather hubs like Las Vegas and Dallas, disproving the official winter-storm narrative. (54:57)
Detailed notes
## Background This seminar was published by Stanford Online on YouTube on October 29, 2026, featuring MIT Associate Professor Chuchu Fan for the Stanford Robotics Seminar ENGR319. According to the page description, the session explores why learning-based controllers struggle with real-world deployment, introduces a framework bridging synthesis and falsification to guarantee safety as a structural policy property, and presents adaptive Bayesian active learning techniques to discover failure modes and stress-test autonomous systems.
## TL;DR Chuchu Fan outlines a unified methodology for autonomous robotics that pairs safe policy synthesis with active failure falsification. On the synthesis side, neural constraint value functions and temporal logic are converted into Control Barrier Functions and decomposed Bellman equations, enabling zero-shot scaling across multi-agent drone swarms, heterogeneous delivery systems, and autonomous vehicles without relying on soft penalty tuning. On the falsification side, she recasts safety testing from traditional optimization into high-dimensional Bayesian active inference and MCMC sampling to map multimodal risk landscapes. The talk illustrates how these testing frameworks uncover edge cases in autonomous driving, stress-test foundation models on disaster response triage, and conduct retrospective root-cause analysis on complex systemic disruptions like the 2022 Southwest Airlines meltdown.
## Atmosphere & style The presentation is intellectually rigorous, authoritative, and paced with academic clarity. Fan delivers her insights with a measured, matter-of-fact tone, punctuated by candid observations on real-world engineering realities, such as how poorly tuned agents learn to freeze rather than move safely. The visual rhythm shifts smoothly from dense mathematical control formulations to dynamic hardware demonstrations and real-world failure post-mortems, creating an engaging, deeply technical atmosphere focused on solving the fundamental fragility of modern autonomous systems.
## Bulletin - [05:06](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=306s) MIT researcher Chuchu Fan presents a closed-loop framework uniting formal synthesis (designing controllers with hard mathematical guarantees) and Bayesian falsification (actively uncovering rare failure modes). - [14:41](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=881s) Neural constraint value functions formulated as Graph Control Barrier Functions (GCBFs) enable purely simulation-trained decentralized multi-agent policies to scale zero-shot to 1,000 agents and transfer directly to real hardware. - [39:07](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=2347s) Falsification is reframed from brittle optimization into Bayesian MCMC sampling over energy-based distributions, cutting testing evaluation costs tenfold across spaces reaching 250,000 dimensions. - [54:57](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=3297s) Retrospective failure analysis of the December 2022 Southwest Airlines meltdown isolated airplane reserve depletion at warm-weather hubs like Las Vegas and Dallas, disproving the official winter-storm narrative.
## Key takeaways - [00:09](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=9s) Classical control theory guarantees safety and stability on explicit models but hits scalability bottlenecks, while modern machine learning scales to high dimensions but remains opaque and empirically unsafe. - [05:06](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=306s) Scalable provable autonomy requires two complementary pillars: rigorous design via formal synthesis and empirical evidence from automated falsification testing. - [13:16](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=796s) Soft penalties and Lagrangian relaxations in standard reinforcement learning cause catastrophic failures or degenerate behaviors like freezing when safety objectives conflict with task progress. - [14:41](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=881s) Formulating infinite-horizon safety as a constraint value function satisfies Bellman equations, mathematically proving that learned neural value functions can serve as valid Control Barrier Functions. - [14:41](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=881s) Graph Control Barrier Functions enable decentralized execution using local sensor graphs, scaling policies trained on 8 simulated agents up to 1,000 agents without centralized coordination. - [23:07](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=1387s) Temporal logic formulas paired with large language model translation decompose complex missions into graphs of atomic Hamilton-Jacobi value functions trained independently or in parallel. - [27:37](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=1657s) Real-world demonstrations show hard temporal constraints reliably enforce complex traffic rules at stop signs and yield zero-shot cooperative driving through unseen roundabouts. - [30:42](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=1842s) Formal certificates inside design models cannot eliminate real-world failures due to unmodeled dynamics, necessitating active adversarial testing to discover latent blind spots. - [37:07](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=2227s) Failures in autonomous systems span five distinct dimensions: distributional, reality (sim-to-real gap), contextual, normative, and retrospective. - [39:07](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=2347s) Standard optimization algorithms fail to locate failure modes because failure landscapes are highly multimodal and non-convex, trapping optimizers in single local attractors. - [44:23](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=2663s) Bayesian active inference guided by an acquisition function balancing Information Gain and User Utility maps complex topological risk landscapes under limited real-world test budgets. - [49:39](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=2979s) Contextual edge cases synthesized by combining causal factors from historical crashes expose safety failures across leading autonomous driving stacks and multimodal foundation models. - [53:24](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=3204s) Stress-testing multimodal LLMs on paired satellite imagery revealed that newer models like GPT-5.2 exhibited significantly higher output variance and sensitivity to minor perturbations than older models like GPT-4.1. - [58:53](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=3533s) Conflicting hard constraints must be managed through formal priority hierarchies where physical collision avoidance strictly overrides nominal operational rules such as lane markings.
## Detailed walkthrough ### [00:09](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=9s) Bridging Rigorous Methods with Machine Learning
Chuchu Fan introduces the overarching focus of her lab: building safe and intelligent autonomy by creating a bridge between classical, rigorous methods and machine learning. She clarifies that highlighting "rigorous methods" is not meant to imply that machine learning lacks rigor, but rather to point out the distinct engineering trade-offs inherent in how both paradigms are traditionally formulated. Autonomous systems designers today—whether in robotics or engineering more broadly—are faced with a choice between classical approaches (such as control theory, formal methods, and optimization) and purely data-driven methods (such as deep reinforcement learning and large foundation models). While the boundaries between these fields are increasingly blending, engineers often find themselves having to trade off critical capabilities:
```mermaid flowchart TD accTitle: Classical vs Data-Driven Methods Tradeoff accDescr: Comparison of properties between classical formal approaches and modern data-driven machine learning methods. subgraph Classical["Classical Methods"] C1["Safety & Stability: By Design"] C2["System Model: Explicitly Required"] C3["Scalability: Limited by Complexity"] C4["Reasoning: Trackable & Interpretable"] C5["Guarantees: Mathematical Proofs"] end subgraph DataDriven["Data-Driven / ML Methods"] D1["Safety & Stability: Empirical Only"] D2["System Model: Learned from Data / Model-Free"] D3["Scalability: High Capacity"] D4["Reasoning: Opaque / Black-Box"] D5["Guarantees: Rare or Lacking"] end Classical <-->|"Trade-off Gap"| DataDriven ```
In classical control, core guarantees such as stability and safety hold *by design* based on the underlying theory and an assumed model of the system. In purely data-driven frameworks, however, safety metrics are largely empirical. While empirical performance may suffice in low-stakes scenarios, it is inadequate for safety-critical settings such as physical robots collaborating alongside humans. Conversely, classical methods suffer from severe scalability bottlenecks and are bounded by the fidelity of human-engineered models—recalling the adage that all models are wrong and diverge from reality. Machine learning excels precisely where classical methods fail by remaining model-free and learning directly from high-dimensional data, but it lacks interpretable reasoning. As Fan explains:
> “So this reasoning is not the reasoning mode of your large model. Although I do acknowledge that is reasoning. Although that reasoning is more of the datadriven based reasoning where we don't really know what they're reasoning about there they show you something but you if we use the classical approaches we have a way of understanding which components is supposed to do what what's the bottleneck when a problem arises there's a way for us to track now where these issues happen so I would say reasoning is tractable in classical methods but it's an emerging topic in machine learning and still people are defining what he means by reasoning and when it comes to being able to provide proofs and guarantees uh they are often there in the classical approaches however um in datadriven approaches we see that it's really rare that people are talking about proofs and guarantee they are harness but what I'm talking about this mathematically rigorous objects describing the guarantees we can have so the what we are looking we're looking at is this critical gap how do we provide build pro safety at scale in the sense like we don't want to compromise between the two.”
[05:06](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=306s) Two Complementary Paths: Rigorous Design and Falsification Testing
To eliminate the forced compromise between classical guarantees and data-driven scalability, Fan’s lab structures their autonomous systems research around two complementary pillars:
1. **Safety and Mission Satisfaction by Design (Synthesis):** Incorporating formal mathematical constraints directly into the learning framework. Rather than treating safety violations as soft cost penalties that can be traded off during optimization, constraints are embedded as hard mathematical requirements that the learned policy cannot violate. 2. **Understanding Failures by Testing (Falsification and Active Learning):** Recognizing that real-world deployment inevitably encounters edge cases not anticipated in any design-time model. To establish genuine trustworthiness, the system automatically discovers failure cases through targeted testing, identifying whether breakdowns stem from unmodeled environmental dynamics, missing requirements, or intrinsic physical limitations.
As Fan summarizes the core philosophy:
> “So in short the uh the rigorous uh the provable safe at scale is done by rigorous design and evidence from testing al together.”
[06:58](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=418s) Formulating Safe Synthesis as Constrained Optimal Control
Addressing the first pillar—safety and mission success by design—Fan formulates the autonomous mission as a constrained optimal control problem. The objective is to discover a policy that maximizes a chosen performance metric (or minimizes cost) while strictly adhering to safety specifications, even when the underlying robot system dynamics remain unknown to the designer at training time:
$$\max_{\pi} \quad J(\pi) \quad \text{subject to task and safety specifications, with unknown dynamics } \dot{x} = f(x, u)$$
To solve this class of problems at scale without relaxing constraints into soft penalties, Fan’s approach combines two elements: - **Hard Constraint Enforcement:** Enforcing safety and mission requirements throughout policy learning, refusing to allow constraint violations in exchange for higher rewards. - **Automated Task Decomposition:** Systematically breaking down complex, temporally extended tasks into modular components with formal reasoning to verify that compositionally fulfilling each subtask guarantees the satisfaction of the full overarching mission.
[08:52](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=532s) Motivating Scenario: Heterogeneous Autonomous Warehouse Logistics
To illustrate the challenge of enforcing complex, continuous constraints over an infinite time horizon, Fan presents an autonomous warehouse delivery environment involving heterogeneous multi-robot coordination. The setup pairs fast-moving aerial drones with ground-based quadruped robots to execute recurring supply pickups and drop-offs:
```mermaid flowchart LR accTitle: Heterogeneous Warehouse Workflow accDescr: Quadruped ground robots enter the warehouse to fetch packages and meet aerial drones at corridors for drop-offs. subgraph Warehouse["Warehouse Interior (No-Fly Zone)"] W1["Ground Quadruped fetches incoming packages"] end subgraph Corridor["Transfer Corridor"] C1["Quadruped carries package on back"] C2["Aerial Drone rendezvouses with Quadruped"] C1 -->|"Hand-off"| C2 end subgraph Delivery["External Delivery Route"] D1["Aerial Drone flies fast to deliver payload"] end W1 --> C1 C2 --> D1 ```
The system is governed by strict, persistent constraints that must hold indefinitely rather than for a single episode: - **Collision Avoidance:** Aerial drones and ground robots must continuously avoid physical collisions with obstacles and each other. - **Geographic Restrictions & No-Fly Zones:** The ground quadrupeds are physically restricted to specific transit lanes (akin to bus routes), while the warehouse interior is a strict no-fly zone forbidden to aerial drones. - **Infinite-Horizon Recurrence:** Because packages arrive non-stop, ground quadrupeds must repeatedly re-enter the warehouse, haul cargo to designated transfer points, rendezvous with drones, and hand off payloads. Drones must repeatedly track down suppliers, retrieve cargo, and complete deliveries. - **Resource Constraints (e.g., Battery Lifetimes):** Ground and aerial units must monitor their state-of-charge, cycling to charging pads or swapping roles with fully charged robots before depletion.
Fan emphasizes that these requirements cannot be treated as optional soft objectives. The goal of their design framework is to synthesize a purely learning-based policy that operates directly within these constraints, guaranteeing that the mission succeeds without ever breaching a safety threshold.
[11:02](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=662s) Mathematical Abstraction: Two Core Challenges in Safe Multi-Agent Autonomy
To bridge rigorous control with learning-based techniques, two fundamental mathematical challenges must be resolved. The first is enforcing safety and stability as hard constraints throughout the learning and control process. The second is managing complex objectives or mission goals where hand-crafting a monolithic reward function that permanently satisfies delivery constraints and dynamic tasks is impractical.
```mermaid flowchart TD accTitle: Two Challenges in Safe Multi-Agent Learning accDescr: Flowchart illustrating the decomposition into safety constraints and mission goals for decentralized agents.
subgraph Problem["Mathematical Problem Formulation"] C1["Challenge 1: Enforce Hard Safety & Stability"] C2["Challenge 2: Complex Task Objectives"] end
subgraph MultiAgent["Decentralized Multi-Agent System"] A1["Agent 1 Policy: Local Observations"] A2["Agent 2 Policy: Local Observations"] AN["Agent N Policy: Local Observations"] end
Problem --> MultiAgent MultiAgent --> JointGoal["Joint Optimization: Maximize Objective subject to Hard Constraints"] ```
Scaling this problem requires a multi-agent formulation. In realistic deployments, robotic fleets are heterogeneous, composed of distinct robot types that must execute local sub-missions while collaboratively pursuing a shared global objective. Centralized controllers cannot scale to large numbers of robots. Instead, agents must execute decentralized, distributed policies driven strictly by local observations. The system must jointly maximize a shared performance objective or reward function subject to safety constraints defined over local observations, operating over unknown underlying system dynamics:
- Multi-agent scale requires decentralized execution where each agent acts solely on local observations (e.g., cameras or LiDAR). - A shared task objective must be maximized jointly across heterogeneous platforms without centralized coordination. - Safety constraints are modeled as functions over local observations, allowing flexibility up to and including human-labeled safe/unsafe states. - The underlying physical dynamics of the agents and environment are treated as unknown during the formulation.
---
[13:16](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=796s) Why Soft Constraints and Lagrangian Relaxation Fail in Safety-Critical Robotics
In conventional reinforcement learning (RL), safety constraints are routinely converted into soft penalties or addressed via Lagrangian duality methods. This converts hard boundaries into multi-part reward or cost functions. In practice, tuning the associated hyperparameter weights is notoriously brittle: coefficients that succeed in one operating condition completely fail when environmental dynamics shift.
```mermaid flowchart LR accTitle: The Failure Mode of Reward Penalty Formulations accDescr: Diagram showing how soft penalties lead to freezing behaviors due to conflicting objectives.
subgraph StandardRL["Conventional RL Approach"] Hard["Hard Safety Constraint"] --> Soft["Soft Penalty / Lagrangian Dual"] Soft --> Monolithic["Multi-Part Cost Function"] end
subgraph FailureMode["Deployment Bottleneck"] Monolithic --> Hyper["Hyperparameter Tuning Gridlock"] Hyper --> Conflict["Mission vs. Safety Conflict"] Conflict --> Freeze["Degenerate Policy: Agent Freezes or Crawls"] end ```
This structural flaw becomes severe when safety conflicts directly with mission progress. Chuchu Fan emphasizes this trade-off:
> “The best way to be safe is to not move.”
In autonomous driving, for instance, heavily penalizing collisions causes the vehicle to stop entirely or crawl at impractical speeds, severely degrading task performance. To overcome this, hard constraints must be preserved structurally rather than treated as soft penalties.
---
[14:41](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=881s) Neural Constraint Value Functions as Graph Control Barrier Functions
Rather than penalizing collisions softly, the system leverages a specific formulation known as the constraint value function. Defined mathematically as the minimum of a safety margin function $h$ over an infinite time horizon, this value function satisfies a Bellman equation. This property allows it to be solved via traditional optimal control when a model is known, or via RL value iteration when dynamics are unknown. Crucially, this constraint value function is theoretically proven to be a Control Barrier Function (CBF).
```mermaid flowchart TD accTitle: Control Barrier Function Operation accDescr: Pipeline of a Control Barrier Function mapping states to scalar values and filtering control actions.
subgraph Evaluation["CBF Evaluation"] State["State / Sensor Inputs"] --> CBF["Constraint Value Function (CBF / GCBF)"] CBF --> Margin["Scalar Margin: Positive = Safe, Negative = Unsafe"] end
subgraph ActionFiltering["Action Space Filtering"] Margin --> Bound["Action Boundary Computation"] Bound --> Action["Safe Permissible Actions: e.g., Velocity Limits"] end ```
In control theory, a Control Barrier Function takes a high-dimensional state—including environmental context—and outputs a scalar value that is positive in safe regions and negative in unsafe regions. By monitoring this boundary, the CBF computes the exact permissible action space (such as safe velocity envelopes for a vehicle avoiding a hazard) that guarantees the existence of a safe continuation path. For partially observable, decentralized multi-agent fleets, this value function generalizes into a Graph Control Barrier Function (GCBF):
- The constraint value function computes $\min_{t \ge 0} h(x(t))$, directly evaluating whether safety holds across an infinite horizon. - It satisfies a Bellman equation, enabling solution via dynamic programming or reinforcement learning value iteration without analytical system models. - For decentralized swarms, the formulation yields a Graph Control Barrier Function (GCBF). - Each agent evaluates safety locally via its immediate neighborhood graph, while their joint execution provably maintains global system-level safety invariants.
---
[17:48](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=1068s) Zero-Shot Scalability and Robust Sim-to-Real Hardware Transfer
Formulating safety as a Graph Control Barrier Function yields strong generalization across multi-agent configurations. Because policies rely entirely on local observations within a decentralized graph, policies trained in simulation on small groups transfer directly to substantially larger fleets and entirely new objects without retraining.
```mermaid flowchart TD accTitle: Training to Deployment Generalization accDescr: Generalization of decentralized policies from small-scale simulation training to massive swarms and hardware deployment.
subgraph SimulationTraining["Simulation Training"] Sim["Small Fleet: 3 to 8 Agents, Co-carrying Objects"] end
subgraph ScaledEvaluation["Zero-Shot Scaling"] Scale1["Hardware: 6 Heterogeneous Agents Carrying New Payloads"] Scale2["Simulation: 1,000+ Agents Navigating Dense Obstacles"] F16["Full-Scale F-16 Aircraft Close-Formation Flight"] end
Sim --> Scale1 Sim --> Scale2 Sim --> F16 ```
In multi-drone collaborative transport—where aerial vehicles must co-carry payloads that exceed individual capacity while avoiding mutual aerodynamic downwash, maintaining distance bounds, and dodging obstacles—policies trained on only 3 to 5 drones generalize seamlessly to 6 or more drones carrying novel payloads. In large-scale navigation, agents trained exclusively with 8-agent setups scaled up to 1,000 agents in simulation, navigating dense obstacle fields while maintaining state-of-the-art success rates.
The safety barrier abstraction functions as an inherent margin of robustness against unmodeled dynamics, enabling zero-shot simulation-to-real transfer:
> “We actually train everything in simulation. We never collect real data to do training. Everything was trained in simulation. Of course, the model will not be accurate. But when we deploy them on hardware, the robustness that we’re able to build will be able to handle like the wind disturbance or other adversarials.”
- **Cooperative Drone Transport:** Fleets maintain precise inter-agent separation to avoid mutual aerodynamic downwash and obstacle collisions while moving shared payloads. - **Extreme Swarm Scaling:** Policies trained on 8 agents generalize to thousands of agents simultaneously navigating complex obstacle environments. - **Target Tracking & Inspection:** Dual-robot systems maintain continuous camera visibility of moving targets within strict inspection zones. - **High-Fidelity Aerospace Applications:** Pairs of full-scale F-16 jet models execute close-formation maneuvers—reminiscent of safety boundaries from *Top Gun*—ensuring trailing aircraft remain within safe following envelopes without colliding into engine exhaust or jet wash. - **Physical Hardware Robustness:** Drone policies trained strictly in simulation without real-world telemetry autonomously reject physical wind gusts, track dynamic trajectories, and execute precision landings on moving platforms.
[21:20](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=1280s) Enforcing Hard Constraints and the Limits of Existing Methods
Once a safety certificate is represented and integrated directly into the machine learning pipeline, reinforcement learning (RL) algorithms can be guided to enforce hard constraints rather than treating safety as a soft penalty. The resulting value function serves as a portable, explicit safety certificate that can even be transferred to other systems. However, this alone does not resolve the challenge of expressing and executing highly complex mission objectives. Returning to real-world tasks like autonomous package delivery, the field still struggles to systematically encode complex goals:
```mermaid flowchart TD accTitle: Approaches to Complex Task Objectives accDescr: Flowchart contrasting Classical Control, Standard RL or MPC, Large Models, and the Value Decomposition approach.
Goal["Complex Mission Objective"] --> CC["Classical Control / Temporal Logic"] Goal --> RL["Standard RL / MPC"] Goal --> LM["Large Models / Vision-Language Models"] Goal --> VD["Proposed: Value Decomposition"]
CC --> CC_Fail["Fails in high-dimensional state spaces"] RL --> RL_Fail["Costly reward tuning; objective misalignment"] LM --> LM_Fail["Requires vast data; unavailable for novel tasks"] VD --> VD_Success["Decomposes rigorous logic into parallel Bellman tasks"] ```
Classical control uses formal languages like temporal logic, but these methods break down when applied to high-dimensional systems. Standard RL or Model Predictive Control (MPC) requires extensive trial-and-error reward shaping or cost tuning, which incurs heavy financial costs and frequently introduces reward misalignment—where the policy optimizes the surrogate reward rather than the true mission intent. As for foundation models or vision-action models (VLMs), they demand massive datasets that simply do not exist for rare or specialized autonomous missions. As Chuchu Fan emphasizes: “What if you don't have data at all? How should I start? Then I at least need have something to generate the data for me.” The solution lies in identifying the exact mathematical object being learned—the value function—and decomposing it systematically.
[23:07](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=1387s) Temporal Logic and Value Decomposition
To specify intricate multi-stage tasks, missions can be formulated in temporal logic, a rigorous symbolic framework capable of expressing infinite horizons, strict timing windows, logical dependencies, and recurrent behaviors (such as continuously picking up and delivering packages). It precisely captures complex rules: agents revisiting drop-off locations, ground delivery vehicles returning to warehouses, and vehicles respecting no-fly zones. Translating natural language into formal temporal logic used to be a bottleneck, but modern large language models handle the translation seamlessly. Fan notes that while she published research three years earlier on translating English instructions into temporal logic using models like GPT-3.5, “at that time I already knew that that work probably wouldn't have any long-term meaning because the newer generation of the model would be more powerful and now I think you can pretty confidently ask large language model to do the translation for you.”
With the temporal logic specification established, the task is addressed via **value decomposition**. Because temporal logic possesses well-defined syntactic and semantic grammar rules, any specification can be recursively broken down into a hierarchy of fundamental sub-tasks:
```mermaid flowchart TD accTitle: Value Decomposition Graph Architecture accDescr: Diagram illustrating how a high-level temporal logic formula is decomposed into parallel atomic Hamilton-Jacobi value functions.
TL["Temporal Logic Mission Formula\n(e.g., Reach A, Reach B, Always Avoid Walls)"]
subgraph Decomp ["Grammar-Based Decomposition"] D1["Disjunction / Conjunction Rules"] D2["Reach-Avoid Tasks"] D3["Recurrent Tasks (Global / Until)"] end
subgraph Atomic ["Atomic Hamilton-Jacobi Value Nodes"] A1["Node 1: Atomic Bellman Equation"] A2["Node 2: Atomic Bellman Equation"] A3["Node 3: Atomic Bellman Equation"] end
Policy["Distributed Execution Policy\n(Provable Satisfaction Guarantee)"]
TL --> Decomp Decomp --> A1 Decomp --> A2 Decomp --> A3 A1 & A2 & A3 -->|"Simultaneous Parallel Learning & Bootstrapping"| Policy ```
- **Reach-Avoid Tasks:** Directing the agent to achieve a target set while steering clear of unsafe sets. - **Recurrent Tasks:** Mandating behaviors that must repeat indefinitely over an infinite horizon. - **Disjunction Handling:** Disjunctive requirements permit parallel evaluation; satisfying either branch fulfills the clause. - **Atomic Hamilton-Jacobi (HJ) Values:** Leaf nodes resolve into atomic value functions governed by known Bellman equations, enabling standard RL algorithms to train them independently or search them in parallel. - **Cross-Node Bootstrapping:** Implicit value relationships bootstrap across graph nodes during simultaneous training; during execution, agents query the decomposed graph to execute control policies that provably guarantee specification satisfaction. - **Scalability:** In simulation benchmarks, agents scale up to tracking and visiting 100 sequential targets while maintaining near-perfect success rates.
[27:37](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=1657s) Real-World Demonstrations: Shepherding, Drone Delivery, and Autonomous Driving
To demonstrate the versatility of value decomposition, Fan presents three distinct robotic applications:
1. **Multi-Robot Shepherding:** A heterogeneous team consisting of a slower ground quadruped and a agile aerial drone acts as "sheepdogs." The "sheep" are non-cooperative drones programmed simply to flee from the dogs. The dogs execute a distributed policy where active constraints dynamically highlight: they cooperatively squeeze the sheep through a boundary gate, drive them into a circular pen, and hold them there indefinitely. 2. **Infinite-Horizon Ground-Air Delivery:** In physical hardware experiments, a large autonomous ground vehicle patrols warehouse zones while smaller delivery drones repeatedly rendezvous with it. The drones touch down to collect cargo, launch toward designated delivery zones, and cycle back to the ground vehicle indefinitely, maintaining robustness across fluctuating physical environments and hardware parameters. 3. **Autonomous Driving under Formal Traffic Rules:** Every official traffic rule can be translated mathematically into a temporal logic formula. By embedding these rules directly as hard constraints during single-car training over diverse road geometries, the policy learns strict compliance with stop signs and traffic signals:
> “Believe it or not, stop sign is actually a pretty complicated rule to enforce: you have to stop, you have to look at all your surroundings, and decide when you are safe to pass.”
When interacting with surrounding vehicles displaying aggressive, conservative, or rule-violating behaviors, the autonomous agent preserves absolute rule compliance. Remarkably, the policy exhibits zero-shot generalization to environments outside its training distribution. Although the policy was never trained on a traffic circle (roundabout), the combined rules of avoiding rear-end collisions and managing right-of-way allowed the vehicles to safely and cooperatively negotiate roundabouts.
[30:42](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=1842s) The Inevitability of Failures
The first phase of the synthesis framework demonstrates that mission objectives, safety envelopes, and stability requirements can be formulated as structured, computable value functions. By breaking down complex formulas into atomic Bellman equations, policies are synthesized with rigorous mathematical certificates—such as Control Barrier Functions (CBFs), Lyapunov functions, and temporal logic guarantees.
Yet, this mathematical completeness does not mark the end of the engineering problem. Fan cautions that formal guarantees inside a model are never a silver bullet in the physical world: “However, it's not the end of the story, because no matter how hard we try, failures will [occur].” Real-world uncertainty, distribution shifts, and unmodeled environmental interactions inevitably expose safety gaps, necessitating an equally rigorous framework to actively search for, test, and falsify policies.
[31:43](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=1903s) Mathematical Formulation of Testing: Design Parameters vs. Exogenous Parameters
Chuchu Fan begins by emphasizing a fundamental reality of autonomous engineering: no system is completely immune to breakdowns. As she puts it:
> “I don't think there is a system that never fail.”
Instead of fruitlessly chasing impossible perfection, her approach shifts perspective to understand precisely when and why systems fail. Mathematically, whether dealing with an isolated component, a complex system-of-systems, a simulator, or physical hardware, the system's behavior is governed by two distinct classes of inputs:
1. **Design parameters**: These are the internal, tunable variables that engineers choose and optimize (for instance, using synthesis methods) to minimize a broad cost function $J$ representing goals, constraints, and physical limits. 2. **Exogenous parameters**: These are external environmental factors that cannot be directly controlled—such as terrain, lighting, weather conditions, or unpredictable humans who actively disregard safety protocols. While engineers can enforce internal constraints when dynamics are known, there is no mathematical guarantee against an uncooperative agent who refuses to comply.
In this formalization, policy design aims to find the design parameters that minimize cost $J$ under assumptions about the outside world. Conversely, testing (or falsification) searches for the exogenous parameters that maximize $J$, pushing the system into violation of safety limits, uncompleted missions, or catastrophic failure. Because design assumptions are never flawless, testing identifies the blind spots that designers overlooked.
```mermaid flowchart LR accTitle: System Parameter Interaction accDescr: Diagram contrasting how design parameters minimize system cost while exogenous parameters are searched during testing to maximize cost and expose failure modes.
subgraph Inputs["System Parameter Inputs"] DP["Design Parameters<br/>(Tunable: policies, controller gains)"] EP["Exogenous Parameters<br/>(External: weather, terrain, agents)"] end
SYS["Autonomy System / Simulator<br/>Cost Function J(·)"]
subgraph Objectives["Dual Objectives"] MIN["Design Goal:<br/>Minimize Cost J"] MAX["Testing Goal:<br/>Maximize Cost J (Induce Failure)"] end
DP --> SYS EP --> SYS SYS --> MIN SYS --> MAX ```
---
[34:06](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=2046s) The Landscape of Verification, Validation, and Testing (V&V / T&E)
Engineers rely on Verification and Validation (V&V) and Testing and Evaluation (T&E) to evaluate autonomous performance. Across the literature, existing approaches navigate a three-way tradeoff between robustness, flexibility, and efficiency:
* **Formal methods (e.g., reachability analysis, optimization-based verification):** These techniques offer high robustness because they compute rigorous over-approximations of system behavior to formally prove safety. However, they lack flexibility, typically scaling only to small polynomial dynamical models, failing on black-box systems, and demanding prohibitive computational cost. * **Black-box learning and testing:** Popularized by large-model and reinforcement learning communities, this paradigm uses black-box optimization or adversarial RL agents without making structural assumptions about the underlying model. While highly flexible, it sacrifices robustness; because it relies on best-effort heuristics, failing to find a crash yields no rigorous certificate of safety. * **White-box and domain-specific simulators:** By taking advantage of differentiable simulation platforms, these tools strike a practical compromise, speeding up gradient-based search while preserving domain physics.
Fan points out the real-world consequence of relying on naive testing without formal robustness:
> “Well, if you don't have a robustness, you ended up driving a self-driving car forever on the road hoping that you will see a a tricky case.”
Her group's goal is to break these tradeoffs: preserving flexibility for complex black-box systems, achieving efficiency by generating high-coverage test cases at low computational cost, and ensuring robustness by quantifying residual risk and diagnosing failure boundaries rather than relying on luck.
---
[37:07](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=2227s) Five Levels of Failure in Autonomous Systems
To systematically diagnose autonomous systems, Fan categorizes failures across five progressive dimensions:
* **Distributional failure:** Answering *"Where does it fail?"* by mapping out the statistical distribution and parameter regions where failures cluster. * **Reality failure (Sim-to-Real gap):** Evaluating whether a failure identified in simulation or on a lower-fidelity model actually manifests on physical hardware. * **Contextual failure:** Addressing the ambiguous question of *"What counts as a failure?"* For some operators, failure means failing to reach a destination in time; for others, it means bumping into a forbidden obstacle. Formulating precise semantic failure criteria is itself a significant hurdle. * **Normative failure:** Calibrating system behavior across multi-stakeholder groups to decide whose preferences and safety margins take precedence during testing and deployment. * **Retrospective failure:** Performing root-cause analysis after an incident has occurred. This entails isolating hidden environmental or state variables that triggered the crash, allowing engineers to monitor these precursor signals online or synthesize targeted test benches.
---
[39:07](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=2347s) Recasting Failure Discovery: From Optimization to MCMC Sampling
Standard falsification treats finding a failure as a mathematical optimization task: searching for the exogenous parameter vector $Y$ that maximizes the failure cost function $J$. However, optimization routines reliably break down in practice because failure landscapes are fundamentally multimodal. As Fan explains:
> “Failure is a multiode case. failure could be caused by many different reasons and they they can appear in different ways and if I look at this optimization problem to generate test cases I'm really having a non-convex like a super non-convex with multiple optima if I use an optimization based approach even if there are hyper heristics to help you get rid of the getting stuck at local optima you will eventually get this what we call the region of attraction meaning where you start to test will always land to that particular ular failure and you will be bland about the other failures that could happen.”
When gradient-based or local heuristic optimizers search a non-convex failure landscape, their starting points trap them within specific regions of attraction, blinding the tester to distinct, alternative failure modes.
To overcome this limitation, Fan recasts falsification from an optimization problem into a probabilistic sampling problem over a posterior distribution. By treating test case generation as sampling from a posterior failure distribution, the search avoids getting permanently trapped in a single local optimum; different sampling paths retain non-zero probability of discovering separate modes across the parameter space.
Formally, the framework uses Bayesian inference: * The **prior** incorporates the designer's initial domain beliefs about which parameter spaces are risky or benign. * The **likelihood** measures how readily a parameter set produces high cost, commonly approximated via an energy-based exponential formulation: $$\text{Likelihood} \propto \exp(J)$$ * **Markov Chain Monte Carlo (MCMC):** Starting from the prior and updating through simulated system rollouts, MCMC chains generate diverse samples directly from high-cost, high-probability failure regions of the posterior, leveraging simulator gradients whenever available.
[42:32](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=2552s) High-Dimensional Failure Generation and Racing Car Obstacle Avoidance
When exploring failure spaces across design and exogenous parameters, the availability of gradient information shapes the sampling strategy: gradient-based methods accelerate the discovery process, while model-free sampling variants handle scenarios where gradients are inaccessible. Chuchu Fan explains that this methodology has been evaluated across a wide cluster of applications where the design parameters ($X$) and exogenous parameters ($Y$) scale into the thousands or reach up to nearly a quarter-million dimensions. Across these high-dimensional benchmarks, the active sampling approach cuts testing costs by an order of magnitude: “able to see that our method can achieve a 10 times lower cost compared to other approaches to generate the diverse failure cases.” Furthermore, when those generated failures are fed back into repairing the underlying system, the process yields a five-fold boost in system robustness.
```mermaid flowchart LR accTitle: Racing Car Double-Obstacle Failure Scenario accDescr: A vehicle evades an initial obstacle but gets channeled into an unavoidable second collision. subgraph Scenario ["Racing Scenario"] A["Obstacle 1 Avoidance"] --> B["Vehicle Swerves Right"] B --> C["Unavoidable Collision with Obstacle 2"] end subgraph Analysis ["Risk Profiling"] C --> D["Failure Probability Map"] D --> E["Quantified Risk Landscape"] end ```
To ground this setup, Fan presents a concrete example involving a racing car guided by a sampling-based controller. Because sampling-based controllers are intrinsically stochastic, backpropagating gradients through them to perform optimization-based falsification is impossible. The testing objective is to determine precisely where obstacles can induce collisions for this well-tuned system. Rather than merely flagging isolated collision points, the algorithm maps out regions accompanied by explicit failure probabilities. In one demonstrated scenario featuring two obstacles, the vehicle swerves to evade the first obstacle, only to be forced into an unavoidable wreck against the second. Assigning failure probabilities to each spatial configuration produces a comprehensive risk profile for evaluating the autonomy stack.
- Design ($X$) and exogenous ($Y$) parameter spaces scale up to approximately 250,000 dimensions in benchmark testing. - Generating failure suites with this active formulation achieves a 10x reduction in evaluation cost compared to baseline techniques. - Incorporating these discovered failure modes into system repair yields a 5x improvement in operational robustness.
[44:23](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=2663s) Why Classical Testing Fails: The Move to Bayesian Active Inference
Despite these successes, Fan emphasizes that standard posterior sampling relies on shaky assumptions: “A lot of people will immediately say that um well, I don't know a J, right?” In physical deployments, practitioners rarely have access to an accurate prior, simulation-to-real gaps distort system models, collecting real-world failure data is prohibitively expensive, and formalizing a scalar cost function ($J$) is notoriously hard. When working with engineers on complex systems like mobile manipulators, design requirements are buried in dense documentation where balancing task success, clearance margins, dexterous handling, and requests for human intervention cannot be collapsed into an analytical objective. Classical falsification treats testing as a rigid search for counterexamples or witnesses against a predefined metric, producing static compliance checklists that offer false security: “but we know passing those tests does not mean that you are guaranteed to be safe.”
```mermaid flowchart TD accTitle: Closed-Loop Active Inference Testing Cycle accDescr: The iterative loop between querying a system, assessing outcomes, learning risk models, and generating test cases. subgraph Execution ["1. Execution & Evaluation"] SYS["System / Simulator / LLM"] -->|"Output Trajectory / Answer"| ASSESS["Assessment Engine (Human or VLM/LLM)"] end subgraph Modeling ["2. Learning & Optimization"] ASSESS -->|"Evaluated Risk Labels"| SURR["Risk Prediction Model / Surrogate"] SURR -->|"Active Inference"| ACQ["Acquisition Function Optimization"] end subgraph Generation ["3. Test Selection"] ACQ -->|"Information Gain + User Utility"| GEN["Generate Next Test Scenario"] GEN -->|"Query"| SYS end ```
To break this impasse, the problem is reframed from a search process into an active Bayesian inference framework. Instead of seeking binary violation witnesses, the algorithm samples from a posterior distribution conditioned on existing system knowledge to trace an entire risk landscape showing where failures cluster and what triggers them. The architecture establishes a closed test-assess-learn-generate loop. A query is submitted to the target system—which can be a simulator, a physical robot, or a large foundational model—producing an outcome such as a trajectory or textual response. An evaluator assesses this output, providing objective or subjective feedback on perceived risk. That assessment updates a failure prediction surrogate model $P(\text{risk} \mid \text{parameters})$, which in turn guides the next acquisition query.
- Classical testing outputs discrete counterexample witnesses; inference-based testing constructs an entire topological risk landscape. - To bypass hand-crafted or inaccurate prior distributions, the framework leverages variational inference. - The active acquisition function balances two complementary objectives: - **Information Gain:** Drives the selection of queries that maximize uncertainty reduction regarding system boundary limits. - **User Utility:** Directs the search toward failure regimes that human operators or safety engineers prioritize. - The overarching pipeline transmutes sparse initial evidence (simulations, past incident reports, sparse hardware trials) into deployable proof of operational limits.
[49:39](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=2979s) Contextual Failures, Surrogate Models, and LLM Disaster Decision Testing
Many real-world failures are inherently contextual and subjective. In robotic manipulation, for instance, defining failure mathematically is tricky: an aggressive grip may guarantee dynamic stability while simultaneously risking structural damage to a delicate object. Because human operators cannot spend all day labeling complex edge cases, Fan notes a key operational substitution: “Actually in all of our work we replace that human with large models because we think they are pretty good at uh providing assessment about if this is a failure they want to see or if you there are particular things about the failure they want to talk about.” These evaluations train surrogate models across distinct failure modes, allowing the scenario generator to systematically cover combinations of failure sources—such as compounding sensor noise, extreme lighting changes, and camera depth misinterpretations—at varied severity levels.
```mermaid flowchart LR accTitle: Disaster Response LLM Decision Profiling accDescr: Inputs of tweets and satellite imagery are evaluated by LLMs for disaster triage, guided by an acquisition utility. subgraph Ingestion ["Disaster Data Ingestion"] T["Social Media Tweets"] --> LLM["Disaster Response LLM"] S["Satellite Imagery"] --> LLM end subgraph Evaluation ["Response Assessment"] LLM -->|"Mitigation Directives"| EVAL["Difficulty & Confidence Assessment"] EVAL -->|"Utility Metric"| MAP["Decision Boundary Mapping"] end ```
In autonomous driving, this methodology synthesizes novel edge cases by recombining ingredients from separate real-world incidents. Fan highlights a scenario that synthesizes the causal factors of five historical autonomous vehicle crashes into a single synthesized challenge that none of today's leading self-driving stacks could clear, despite the exact combined sequence never having occurred in history. Extending this beyond robotics, the team evaluated large language models deployed for disaster response triage. Tasked with reading satellite images and social media tweets to allocate constrained humanitarian resources under high pressure, the system does not measure failure as a simple binary crash. Instead, mission utility tracks decision difficulty against model confidence. Clear tweets loaded with actionable facts produce high-confidence, low-difficulty decisions, whereas ambiguous, contradictory, or fragmented emergency feeds expose critical blind spots in automated operational guidance.
- Real-world autonomous vehicle failures are synthesized by stitching together causal vectors drawn from multiple recorded incident reports. - Foundation models serve as surrogate evaluators, replacing human annotators in scoring nuanced, non-binary robotic failures. - In disaster management stress testing, the acquisition utility categorizes prompt regimes into high-confidence easy choices versus low-confidence ambiguous inputs to map the reliability envelope of foundational models.
[53:24](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=3204s) Stress-Testing Multimodal LLMs in Disaster Response
Professor Chuchu Fan examines how large language models handle complex disaster response scenarios. Under certain disaster conditions, an LLM might exhibit high confidence when generating actionable mitigation plans, while in other challenging cases, the situation is characterized by high difficulty and low confidence. In those ambiguous scenarios, incoming data indicates that urgent rescue interventions are necessary, yet determining the exact actionable steps remains uncertain. By feeding pre-disaster and post-disaster satellite imagery into the pipeline, the framework evaluates both low-difficulty/high-confidence and high-difficulty/low-confidence cases.
Using real satellite imagery from Hurricane Florence, the model can generate high-confidence action recommendations. However, the framework is not limited to real historical archives; it can also artificially synthesize paired pre- and post-disaster satellite imagery specifically crafted to stress-test the LLM and uncover cases where its outputs should not be trusted. As Fan warns, “We can generate pre and post satellite images to tell you this will really stress test your LLM and in this case you shouldn't trust what LLM is going to say.”
Evaluating different frontier models reveals surprising variations across generations. While comparing model behaviors, Fan notes that more advanced models such as GPT-5.2 are not necessarily more trustworthy, as their recommended actions can become highly sensitive to slight input perturbations and scatter broadly across the action space. In contrast, older iterations like GPT-4.1 demonstrate lower variance, reliably offering consistent recommendations across different types of inputs.
- Synthetic and real pre- and post-disaster satellite imagery are used to evaluate action suggestions (highlighted as purple recommendations). - Advanced models can display extreme output variance and high sensitivity to input perturbations, undermining real-world trustworthiness. - Consistent output distribution across varied inputs is critical before deploying LLMs for disaster triage and physical rescue planning.
---
[54:57](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=3297s) Retrospective Failure Analysis: The 2022 Southwest Airlines Meltdown
Turning to retrospective failure analysis, Fan examines real-world system collapses where operators have no simulator or access to rerun the system, leaving only observational historical data to audit. A relatable example is widespread flight cancellations and delays, where passengers are stranded without clear insight into what triggered the breakdown or who made the operational calls. Fan grounds this challenge in the infamous December 2022 Southwest Airlines scheduling crisis, during which thousands of flights were canceled across Christmas week, stranding over two million travelers across the United States.
```mermaid flowchart TD accTitle: Southwest Airlines Meltdown Causal Analysis accDescr: Diagram illustrating how the initial Denver storm led to cascading nationwide reserve shortages in Las Vegas and Dallas.
subgraph Trigger["Initial Trigger"] A["Denver Winter Storm<br>(De-icing Delays)"] end
subgraph Impact["Observed Effects"] B["Denver Airport Reserve:<br>Sharp Dip, Rapid Recovery"] C["Las Vegas & Dallas Reserves:<br>Cascading Unrecovered Shortages"] end
subgraph Outcome["System Breakdown"] D["System-wide Cancellations<br>(2M+ Stranded Passengers)"] end
A --> B A --> C C --> D ```
Southwest leadership attributed the nationwide cascade entirely to an initial winter storm and de-icing delays in Denver. However, Denver was the only airport facing severe weather, while the rest of the country enjoyed clear operating conditions. Meanwhile, the Southwest Airlines Pilots Association (SWAPA) blamed executive leadership, and media outlets pointed to legacy scheduling software networks.
Without an internal simulator, Fan's team applied their test-case and failure-generation methodology directly to the historical flight data to uncover the hidden operational bottleneck. Their analysis pinpointed airplane reserve—the number of aircraft each airport maintains overnight to service the following day’s schedule—as the critical latent variable. While Denver’s airplane reserves dipped and then rebounded quickly following the storm, major hubs such as Las Vegas and Dallas that experienced clear weather saw their reserves steadily deteriorate over the entire week. This hidden depletion starved the downstream network of aircraft and triggered the nationwide collapse.
- Over thousands of flights were cancelled and two million passengers stranded during Christmas week 2022. - Airplane reserve was mathematically inferred across major hubs because actual reserve targets were never made public. - Denver's fleet reserve stabilized rapidly post-storm, whereas unimpacted hubs (Las Vegas and Dallas) suffered compounding reserve shortages that directly drove cancellations.
---
[57:51](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=3471s) Unifying Rigorous Verification with Data-Driven Machine Learning
Fan synthesizes the seminar's core message: building dependable autonomy requires unifying two complementary paradigms. On one side is safety and mission satisfaction by design (synthesizing control policies with mathematical safety guarantees); on the other is generating empirical counterexamples through active, adaptive testing. While each paradigm probes different facets of system dynamics and environmental parameter spaces, integrating them bridges classical formal methods with modern data-driven learning.
This unified approach allows roboticists to build large-scale autonomous systems that operate with mathematical guarantees rather than unverified empirical hopes. Fan concludes her talk by stating that combining these approaches will “enable us to build systems at large scale that are safe by design, scalable by architecture and reliable by proof.”
- Safety by design guarantees that policies satisfy known bounds and certificates mathematically. - Active falsification discovers unmodeled parameter spaces and rare-event failures that violate baseline assumptions. - Bridging formal methods with reinforcement learning grounds empirical neural models in structural stability.
---
[58:53](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=3533s) Q&A: Handling Contradictory Hard Constraints and Unavoidable Adversaries
During the question-and-answer session, an audience member raises a practical dilemma regarding hard safety constraints: what happens when safety constraints become contradictory in real-world edge cases? The audience member illustrates this using an autonomous driving scenario where an oncoming car veers into the ego vehicle’s lane. To avoid a catastrophic head-on crash, the vehicle must swerve into the opposing lane, thereby violating lane-keeping rules. The attendee asks how the framework arbitrates situations where not all hard constraints can be simultaneously upheld.
Fan clarifies the boundary between mathematical specification and design assumptions. Hard constraints strictly enforce whatever the engineer explicitly defines as invariant within the model: “when we say the safety are hard constraints there are hard constraints on whatever you define as safety right like as I said why the reason we need testing is there are always cases we cannot imagine at the design phase so we have to understand when they fail because certain certain assumptions we make on the design phase got broke.”
If both collision avoidance and strict lane-keeping are hard-coded with equal priority, the mathematical solver will find the problem infeasible when an unavoidable hazard appears, leaving no escape trajectory. To resolve this, safety constraints should be structured hierarchically. Severe hazards (such as physical collisions) are assigned top priority, while secondary rules (such as lane discipline) are treated as relaxable constraints when strict coexistence is physically impossible.
Finally, Fan emphasizes physical realism: if an adversary moves toward the vehicle at the speed of light, avoiding impact is physically impossible regardless of the control policy. Rigorous testing is essential precisely because it exposes these extreme scenarios, allowing designers to map the operational boundary where assumptions hold versus where environmental conditions make failure unavoidable.
- Hard constraints will be strictly satisfied mathematically only up to the exact specifications programmed into the controller. - Unanticipated conflicts require hierarchical safety specifications, treating collision avoidance as primary and road rules (like lane-keeping) as secondary relaxable constraints. - Active testing delineates the operational design domain by finding adversarial boundaries where no safe control input physically exists.
## Notable quotes > [27:37](https://www.youtube.com/watch?v=ot_whQ3xyJQ&t=1657s) Believe it or not, stop sign is actually a pretty complicated rule to enforce: you have to stop, you have to look at all your surroundings, and decide when you are safe to pass. — Chuchu Fan
## Mentioned Chuchu Fan, MIT, Department of Aeronautics and Astronautics (AeroAstro), Laboratory for Information and Decision Systems (LIDS), GPT-3.5, GPT-4.1, GPT-5.2, F-16, Top Gun, Southwest Airlines, Southwest Airlines Pilots Association (SWAPA), Denver International Airport, Las Vegas, Dallas, Hurricane Florence
## Video description For more information about Stanford’s graduate programs, visit: https://stanford.io/grad-ed-engr319-1002
October 2, 2026 This seminar covers: • Why learning-based controllers struggle with real-world deployment: we can’t guarantee a learned policy is safe, and we can’t efficiently find the rare scenarios where it fails • A new approach that closes the loop between synthesis (building safe policies) and falsification (finding where they break) • How neural certificates can be formulated as value functions in reinforcement learning, producing control policies with built-in safety and stability guarantees — turning safety into a structural property of the policy rather than a soft penalty • An adaptive test-case generation method using Bayesian active learning to discover failure scenarios that are both likely to reveal problems and useful for broader coverage • How this method can predict failures from limited hardware demonstrations • How it supports scalable, evolving experimental design for system-level testing
Follow along with the seminar schedule: https://stanfordasl.github.io/robotics_seminar/archive/
Chuchu Fan is an Associate Professor (pre-tenure) in the Department of Aeronautics and Astronautics (AeroAstro) and the Laboratory for Information and Decision Systems (LIDS) at MIT.
[Source](https://www.youtube.com/watch?v=ot_whQ3xyJQ)