Sentdex Replaces GTA V Engine With Playable Nvidia GameGAN Model

sentdex

Generative neural networks can eliminate conventional 3D game engines and hardcoded physics pipelines by directly learning physical dynamics from video data.

AI-assisted summary of Playing a Neural Network's version of GTA V: GAN Theft Auto — original source. Check the original for full context.

Source published: 2021-06-18

Key ideas

Sentdex and Daniel Kukieła created 'GAN Theft Auto', a playable simulation where a neural network directly acts as the game engine, generating frames in real time from player keyboard inputs without traditional physics code or 3D rendering pipelines. (0:04)
Data harvesting was accelerated using 12 automated bots running simultaneously in a single Grand Theft Auto V instance with collisions and mutual visibility disabled, generating driving data along a shared highway stretch. (5:19)
Model training was powered by an NVIDIA DGX Station A100 featuring 320 GB of VRAM across four 80 GB GPUs, cutting down training times for both GameGAN and roughly 30 custom super-sampling models. (5:19)
The neural engine exhibited emergent real-world behaviors, including barrier scraping physics, background parallax of distant mountains, third-person camera horizon tilt, and the initiation of PIT maneuvers during car collisions. (10:05)

Detailed notes

## Background Published on YouTube by sentdex on June 18, 2021, this video showcases "GAN Theft Auto," an interactive recreation of Grand Theft Auto V built using a Generative Adversarial Network. Based on a fork of NVIDIA's GameGAN research, the project replaces the traditional game engine entirely so that the neural network itself serves as the playable environment.

## TL;DR Sentdex and collaborator Daniel Kukieła trained a GameGAN model to generate a fully playable driving slice of Grand Theft Auto V directly from player keyboard inputs and previous frame states. To harvest sufficient training data under tight deadlines, the team deployed 12 parallel, rules-based bot drivers in a single game instance before training the network on an NVIDIA DGX Station A100 equipped with 320 GB of VRAM. Paired with a custom 8x super-sampling model to sharpen the raw, low-resolution output, the neural engine demonstrated emergent behaviors including realistic wall scraping, specular lighting, dynamic drop shadows, background parallax, and partial vehicle collisions. Sentdex reflects on the future of entirely AI-driven engines trained on real-world footage and releases the weights and starter dataset to GitHub for community experimentation.

## Atmosphere & style The video carries an earnest, enthusiastic, and inquisitive tone, driven by sentdex's infectious geeky delight as he watches complex rendering behaviors emerge spontaneously from raw neural weights. His delivery is conversational, methodical, and humble, treating technical bugs like splitting police cars or vanishing geometry with amused curiosity rather than frustration. Watching the demonstration feels like peering through a fuzzy, dreamlike digital haze where a familiar game world slowly crystallizes into coherent reality through pure matrix multiplication.

## Bulletin - [00:04](https://www.youtube.com/watch?v=udPY5rQVoW0&t=4s) Sentdex and Daniel Kukieła created 'GAN Theft Auto', a playable simulation where a neural network directly acts as the game engine, generating frames in real time from player keyboard inputs without traditional physics code or 3D rendering pipelines. - [05:19](https://www.youtube.com/watch?v=udPY5rQVoW0&t=319s) Data harvesting was accelerated using 12 automated bots running simultaneously in a single Grand Theft Auto V instance with collisions and mutual visibility disabled, generating driving data along a shared highway stretch. - [05:19](https://www.youtube.com/watch?v=udPY5rQVoW0&t=319s) Model training was powered by an NVIDIA DGX Station A100 featuring 320 GB of VRAM across four 80 GB GPUs, cutting down training times for both GameGAN and roughly 30 custom super-sampling models. - [10:05](https://www.youtube.com/watch?v=udPY5rQVoW0&t=605s) The neural engine exhibited emergent real-world behaviors, including barrier scraping physics, background parallax of distant mountains, third-person camera horizon tilt, and the initiation of PIT maneuvers during car collisions.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=udPY5rQVoW0&t=4s) GAN Theft Auto replaces traditional hand-crafted 3D graphics and collision engines with a pure tensor-based neural network model executing in Python. - [00:04](https://www.youtube.com/watch?v=udPY5rQVoW0&t=4s) The runtime pipeline pairs raw low-resolution GameGAN predictions with an 8x super-sampling upscaling model to produce a smooth, playable visual display. - [02:05](https://www.youtube.com/watch?v=udPY5rQVoW0&t=125s) The architecture builds on NVIDIA's GameGAN research, which features distinct memory modules for static and dynamic objects to retain world state across navigation. - [02:05](https://www.youtube.com/watch?v=udPY5rQVoW0&t=125s) Early experiments with CartPole failed due to minimal motion dynamics, prompting the creation of a custom driving game called 'Vroom' to provide continuous turning feedback. - [05:19](https://www.youtube.com/watch?v=udPY5rQVoW0&t=319s) Training at scale was facilitated by an NVIDIA DGX Station A100 equipped with four 80 GB GPUs, offering 320 GB of combined VRAM. - [05:19](https://www.youtube.com/watch?v=udPY5rQVoW0&t=319s) Twelve rules-based bot agents drove simultaneously inside a single GTA V instance with collision clipping disabled to rapidly capture the necessary training trajectories. - [07:24](https://www.youtube.com/watch?v=udPY5rQVoW0&t=444s) Initial models broke down upon hitting obstacles because the automated collection bots had originally been programmed never to crash into barriers. - [07:24](https://www.youtube.com/watch?v=udPY5rQVoW0&t=444s) Retraining with intentional wall impacts and police cruiser interactions successfully taught the network barrier resistance, scraping friction, and multi-vehicle dynamics. - [07:24](https://www.youtube.com/watch?v=udPY5rQVoW0&t=444s) Complex physical and lighting properties emerged naturally from the model, including dynamic specular highlights on windshields and authentic vehicle drop shadows. - [10:05](https://www.youtube.com/watch?v=udPY5rQVoW0&t=605s) Advanced spatial dynamics emerged in later iterations, such as background mountain parallax and camera horizon tilt corresponding to steering maneuvers. - [12:11](https://www.youtube.com/watch?v=udPY5rQVoW0&t=731s) Multi-car collision handling occasionally manifested bizarre visual artifacts, such as vehicles duplicating into two objects, blurring, or vanishing into thin air. - [12:11](https://www.youtube.com/watch?v=udPY5rQVoW0&t=731s) Steering into the rear quarter of a police vehicle demonstrated authentic physical resistance and initiated an emergent PIT maneuver rotation. - [13:53](https://www.youtube.com/watch?v=udPY5rQVoW0&t=833s) Sentdex envisions future video game engines and operating systems becoming largely neural-based, with real-world video potentially training realistic 'IRL GAN' physics. - [15:22](https://www.youtube.com/watch?v=udPY5rQVoW0&t=922s) The project repository was made publicly available on GitHub with base weights, the super-sampler, and a starter dataset for community experimentation.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=udPY5rQVoW0&t=4s) GAN Theft Auto: Playing Inside a Neural Network Environment

Sentdex introduces "GAN Theft Auto," a project where the player drives directly inside an interactive world generated entirely by an artificial neural network. The network was trained by observing an AI bot drive along a specific stretch of highway in *Grand Theft Auto V*, learning how to generate each subsequent visual frame in real time from prior frames paired with keyboard steering inputs. Unlike recent deep learning research focused on post-processing GTA V to look photorealistic, this neural network does not sit on top of a traditional 3D graphics pipeline. Instead, it replaces the game engine entirely: there are no pre-programmed collision boundaries, physics calculations, or engine rules coded by Rockstar Games or the project creators. Every visual element—from player-controlled steering and wall impacts to traffic behavior, distant scenery shifts, and dynamic lighting reflections—is generated on the fly.

```mermaid flowchart LR accTitle: GAN Theft Auto Runtime Pipeline accDescr: Diagram showing player inputs feeding into the GameGAN model and super-sampling upscale. subgraph Input ["Player Actions"] K["Keypresses (A / D / W / S)"] P["Previous Frame State"] end subgraph Model ["Neural Engine"] G["GameGAN Model (Python / PyTorch)"] end subgraph Output ["Display Pipeline"] N["Native Low-Res Frame"] S["8x Super-Sampling Model"] D["Playable Output Display"] end K --> G P --> G G --> N N --> S S --> D N -.->|"Recirculated State"| P ```

The system operates simply as a Python model running on a GPU, ingesting keyboard presses from the user and streaming out raw pixel arrays. To make the native low-resolution feed more visually appealing, an integrated super-sampling model upscales the imagery, while the raw, blocky output remains visible in a smaller preview box at the bottom of the screen. Because the entire simulation is essentially pure tensor computation, GAN Theft Auto is platform-agnostic and can run on virtually any operating system capable of executing matrix mathematics on an accelerated graphics card. As sentdex observes when contrasting this setup with conventional gaming AI:

> “So we're used to seeing AI playing within environments, but here the AI is the environment; it's just a model that we're running with Python that takes key presses from me, the player, and outputs pixel values.”

- **Engine Architecture**: Completely neural-based; eliminates traditional hand-crafted physics, hardcoded boundary geometry, and standard rendering pipelines. - **Hardware Portability**: Runs across operating systems provided the machine supports matrix math, display output, and GPU acceleration. - **Display Setup**: Features an upscaled primary viewport paired with a native-resolution picture-in-picture window at the bottom-center.

---

[02:05](https://www.youtube.com/watch?v=udPY5rQVoW0&t=125s) Origins in NVIDIA's GameGAN and Early Experiments with CartPole and Vroom

The initiative began as a joint effort between sentdex and Daniel Kukieła, building directly on NVIDIA's original GameGAN research. GameGAN represents a significant evolution beyond naive generative adversarial networks that map control inputs to video frames. Crucially, the architecture incorporates distinct memory modules for static versus dynamic objects, allowing the simulation to retain world state when the player leaves an area and later returns. Sentdex had been eager to interact with this technology ever since NVIDIA demonstrated GameGAN recreating *Pac-Man*, but the source code, pre-trained weights, and training datasets remained unavailable for an extended period. When NVIDIA eventually published the code, it shipped without pre-trained checkpoints or training corpora, forcing the team to build their own pipelines from scratch.

```mermaid flowchart TD accTitle: Research Evolution from Pac-Man to Custom Environments accDescr: Flowchart tracing development from NVIDIA Pac-Man demo down to the custom Vroom environment. A["NVIDIA GameGAN Paper & Pac-Man Demo"] --> B["Code Released (No Models or Datasets)"] B --> C["CartPole Experiment (OpenAI Gym)"] C -->|"Inconclusive / Minimal Motion"| D["Daniel Builds 'Vroom' Environment"] D --> E["Training on BOXX Station (2x Quadro RTX 8000)"] E --> F["Successful Track Memory & Car Steering"] ```

Their initial training experiment attempted to replicate the classic OpenAI Gym CartPole environment, where a player balances a vertical pole on a moving cart. The results were disappointing from start to finish, which sentdex hypothesized might stem from the nature of CartPole itself—the objective requires moving as little as possible, providing scarce dynamic variance, or perhaps the recording bot failed to supply diverse gameplay data. To overcome this limitation, Daniel Kukieła developed a bespoke driving game called "Vroom," visually inspired by OpenAI Gym's CarRacing environment. Written from the ground up to feature perpetual movement and heavy cornering, Vroom gave GameGAN continuous visual feedback. Training began on BOXX workstations equipped with dual NVIDIA Quadro RTX 8000 GPUs. Early Twitter demos showed grainy, foggy results, but the car turned realistically and its wheels rotated on keypresses. Within a few days of additional compute, the network accurately modeled track departures, off-road driving across grass, and returning to the exact same persistent raceway.

> “From the moment I saw the Pac-Man video, I just had to play in the environment, I wanted it so bad, but the code and the models and even the data to train a model was not really made available for quite some time.”

- **Early Hardware**: Initial training runs took place locally using BOXX workstation setups powered by two Quadro RTX 8000 GPUs. - **CartPole Pitfalls**: Static target dynamics made it poorly suited for GameGAN’s motion-dependent learning mechanisms. - **Vroom Testbed**: Built from scratch by Daniel Kukieła to guarantee continuous turning, steering, and camera displacement.

---

[05:19](https://www.youtube.com/watch?v=udPY5rQVoW0&t=319s) Scaling Up: The DGX Station A100 and Automated GTA V Data Harvesting

Following the success of the Vroom proof of concept, NVIDIA reached out and offered sentdex access to a DGX Station A100 equipped with four 80 GB A100 GPUs, providing an unprecedented pool of 320 GB of high-bandwidth VRAM. Armed with enterprise-grade compute, sentdex decided the logical next frontier was modeling *Grand Theft Auto V*, coining the project title "GAN Theft Auto." Because access to the DGX Station was time-limited, gathering hundreds of hours of manual gameplay driving was out of the question, as it would have taken months to collect the required volume of frame data.

```mermaid flowchart TD accTitle: Parallelized Data Harvesting Architecture accDescr: Architecture showing 12 automated bots running in parallel inside a single GTA V instance. subgraph GTA ["Single GTA V Game Instance"] B1["Bot Agent 1"] B2["Bot Agent 2"] B3["Bot Agent 3"] B12["...up to 12 Parallel Bots"] end subgraph Sync ["World State Modifications"] M1["Shared Highway Stretch"] M2["Invisibility Toggled"] M3["Collision Clipping Disabled"] end subgraph DGX ["DGX Station A100 (320 GB VRAM)"] T["GameGAN Batch Training"] end GTA --> Sync Sync -->|"Concurrent Video & Action Streams"| DGX ```

Daniel Kukieła stepped in once again to automate data generation by programming a deterministic, rules-based bot to navigate GTA V's highways. To maximize throughput against the clock, Daniel configured 12 autonomous bots to operate simultaneously inside the exact same GTA V game instance along the identical highway corridor. To prevent traffic jams and interference, the bots were rendered invisible to one another with car-to-car collision clipping disabled, allowing all 12 vehicles to record independent driving trajectories across the same asphalt in parallel. After gathering the initial dataset, the team ran a 6-epoch training pass on the DGX Station. Despite heavy native pixelation, the generated world was instantly identifiable as GTA V, accurately reproducing the player's real-time keyboard inputs, cast shadows beneath the vehicle, light glinting off the rear windshield, and the distant background scrolling accurately through turns.

> “Playing GTA manually to collect the data really wasn't going to be an option because it would take us months to get enough, so again Daniel stepped in to bend GTA to his will and created a rules-based AI to drive around GTA and collect the data.”

- **Compute Specifications**: NVIDIA DGX Station A100 containing 4 × 80 GB A100 GPUs (320 GB total VRAM). - **Parallel Collection**: 12 bot drivers ran concurrently in a single game instance with collision clipping disabled to accelerate dataset creation. - **Epoch 6 Milestones**: Real-time steering response, accurate directional shadow casting, background parallax, and window glare.

---

[07:24](https://www.youtube.com/watch?v=udPY5rQVoW0&t=444s) Incorporating Wall Collisions, Police Chases, and 8x Super Sampling

The baseline 6-epoch model possessed a critical flaw: it lacked environmental constraints. Because the autonomous collection bots had been programmed never to crash, the model had never observed wall collisions during training. Consequently, whenever sentdex attempted to steer into a roadside barrier during inference, the neural network rapidly desynchronized and became confused. Recognizing that GameGAN is theoretically capable of understanding rigid obstacles, the team redesigned the data generation routines to force bot collisions into concrete highway dividers and introduced secondary vehicles in the form of police cruisers to teach the network multi-car interactions. Daniel modified GTA V scripts to reliably spawn police vehicles into the bots' recording sessions, overcoming engine glitches to capture realistic vehicle impacts and roadside friction.

```mermaid flowchart LR accTitle: Native Output Upscaling Comparison accDescr: Visual pipeline comparing low-resolution native GameGAN output to the chosen 8x super-sampled visual frame. subgraph ModelOutput ["Raw Generation"] R["Native GameGAN Output (Low-Resolution / Heavy Pixelation)"] end subgraph SuperSampling ["Model Selection"] E["Evaluated 30 Distinct Upscaling Models"] C["Selected Best-Performing 8x Model"] end subgraph Final ["Inference Display"] F["Enhanced High-Fidelity Feed with Smooth Lighting & Edges"] end R --> E E --> C C --> F ```

To resolve the severe graininess of the raw GameGAN output, sentdex and Daniel trained 30 separate super-sampling models, eventually selecting an 8x upscaling model that produced the cleanest visual clarity. The resulting combined pipeline exhibited striking emergent physical and lighting properties. When driven into concrete highway barriers, the car no longer broke the model; instead, the neural network forced the car straight, realistically simulating friction as the vehicle scraped along the wall. Furthermore, the network accurately simulated complex specular lighting, rendering changing sunbeams striking the matte paint of the car body and shifting across the rear glass as the vehicle changed heading relative to the sky.

> “The wall collisions did eventually work to the point at least where the car approaches the wall, hits the wall, and then is like forced, you know, to turn straight and kind of slide against the wall, so I think that modeling of physics is actually kind of cool; we didn't really know if that would work or not, and it turns out it would.”

- **Collision Physics**: The model learned barrier resistance, forcing steering realignment and visual vehicle scraping upon impact. - **Vehicle Diversity**: Limited strictly to player vehicles and standardized police cruisers to avoid overwhelming the model's visual memory. - **Super-Sampling Scale**: Selected the best out of 30 experimental models to achieve an 8× resolution enhancement over native GameGAN output. - **Lighting Fidelity**: Successfully rendered dynamic specular reflections across matte body panels and transparent rear windshield glass relative to directional sunlight.

[10:05](https://www.youtube.com/watch?v=udPY5rQVoW0&t=605s) Emerging Physics, Parallax, and Vehicle Dynamics

Sentdex highlights that the generative model does not simply reproduce static frames; it organically captures nuanced physics concepts that typically require dedicated game engines. Beyond dynamic light casting, the vehicle's shadow updates accurately beneath the car as it maneuvers. While these shadow calculations are standard in modern 3D titles, having them emerge purely from an artificial neural network remains deeply impressive.

After training roughly 30 super-sampler models and cycling through multiple GameGAN architectures to incrementally improve visual fidelity, Sentdex noticed an unexpected breakthrough in one of the later iterations: genuine background parallax. As the car travels forward along the highway, distant mountain ranges approach at the correct speed relative to the vehicle's forward motion. Sentdex marveled: “I just I never expected that that would be modeled at all, but it was and it like just really blew me away.”

In addition to parallax, the model recreated subtle third-person camera dynamics. Whenever the car turns, the horizon line rolls slightly to mirror the vehicle's body roll and perspective shift, matching Grand Theft Auto V's native camera feel. Sentdex reflected on the achievement: “Overall I think it did really well here, I mean this is definitely Grand Theft Auto 5, only it's completely generated for us by an AI and we get to play within it and that is just incredibly awesome to me.”

- Trained approximately 30 super-sampler models alongside numerous GameGAN candidate models. - Emergent behaviors modeled: dynamic vehicle drop shadows, accurate distance scaling for distant mountain terrain, and camera horizon tilt during sharp steering. - The neural network generates the entire interactive visual and physical environment from scratch without relying on traditional 3D polygon renderers.

---

[12:11](https://www.youtube.com/watch?v=udPY5rQVoW0&t=731s) Environmental Limits and Vehicle Collision Quirks

The training dataset was restricted to a relatively short stretch of highway, leaving the upper limits of the environment's roaming range untested. Sentdex noted that systematically widening the roam boundary stood out as an obvious next step, though time constraints prevented them from pushing the model size to see where it broke down.

Handling dynamic interactions with other traffic proved far more erratic. In one bizarre glitch, GAN Theft Auto resolved a front-end crash with a police vehicle by cleanly splitting the police car into two separate objects. While other cars frequently blurred into visual noise or vanished entirely into thin air, certain test runs showed surprisingly coherent physical boundaries. Sentdex demonstrated a side-by-side view featuring the raw GameGAN output on the left (complete with live keypress overlays in the upper left corner) and an upscaled stream on the right.

In this collision sequence, Sentdex repeatedly steered hard left into a police car, but the neural network refused to let the player phase through it. Instead, the player's car dragged straight along the police cruiser's flank. When struck closer to its rear axle, the cruiser visibly rotated into the beginning of an authentic PIT maneuver before destabilizing and blinking out of existence altogether.

```mermaid flowchart TD accTitle: Vehicle Interaction Flow in GAN Theft Auto accDescr: Pipeline showing player input, neural network collision simulation, and visual failure modes. subgraph Inputs["Player Controls"] A["Steering & Throttle Inputs"] end

subgraph Simulation["GAN Environment Modeling"] B["GameGAN Prediction"] C{"Collision Check"} D["Emergent Physical Resistance<br>(Car Dragging & Pivot)"] end

subgraph FailureModes["Model Instability"] E["Car Splits into Two"] F["Entity Blurs Out"] G["Entity Vanishes Entirely"] end

A --> B B --> C C -->|"Coherent Contact"| D C -->|"Uncertain Geometry"| E C -->|"Latent Fade"| F C -->|"State Collapse"| G ```

- Training scope was confined to a single highway section; scaling to an open-world map remained untried. - Collision attempts produced bizarre artifacts, ranging from car duplication to abrupt disappearance. - Partial collision mechanics manifested naturally: physical obstruction prevented clipping, and rear-quarter impacts initiated genuine PIT maneuver vehicle rotation.

---

[13:53](https://www.youtube.com/watch?v=udPY5rQVoW0&t=833s) Future Prospects of AI Game Engines and Real-World Simulators

Sentdex and his collaborator pulled the entire project together in approximately two months of cumulative effort, starting with very little prior background in Generative Adversarial Networks. Given that steep learning curve, Sentdex expressed genuine pride in the results and voiced optimism about the broader implications for interactive software, stating: “I kind of think there's no reason that we won't have a future where many game engines are entirely or even mostly AI-based like this, or even things like operating systems or in other programs.”

Looking beyond recreating pre-rendered video games, Sentdex floated the idea of building an “IRL GAN” or “NVGAN” trained directly on real-world footage. While traditional video game development has historically struggled to craft truly believable real-time physics, an AI model could theoretically bypass synthetic physics engines entirely by directly learning real-world physical dynamics from camera data.

Sentdex closed by thanking NVIDIA for publishing the original GameGAN research and for providing the DGX Station A100 (equipped with 80GB GPUs), which enabled them to conduct rapid model iteration at “warp speed.”

- Development timeline: roughly two months total, built by developers new to GAN architectures. - Speculative next step: an “IRL GAN” or “NVGAN” trained on real-world footage to learn organic physical laws directly from reality. - Compute hardware: NVIDIA DGX Station A100 (80GB).

---

[15:22](https://www.youtube.com/watch?v=udPY5rQVoW0&t=922s) Model Release, Inference Performance, and Community Collaboration

Sentdex made the base GAN Theft Auto model, the accompanying super-sampler, and a starter slice of the training data publicly available on GitHub for viewers to test on their own machines. Because the code was strictly built for proof-of-concept experimentation, no optimization passes were made to streamline inference speeds; consequently, player frame rates will vary widely depending on their local GPU setup.

Due to bandwidth constraints, the massive full training dataset could not be hosted publicly. However, Sentdex issued an open invitation to the community: anyone who builds a promising architecture or fork using the starter data is encouraged to get in touch, and Sentdex’s team may train that promising model on their full dataset using the DGX Station.

- Artifacts hosted publicly via GitHub: base GameGAN weights, super-sampler model, and sample training slices. - The release code features unoptimized inference; frame rates depend heavily on consumer hardware. - The complete dataset remains unhosted due to raw file size, but community members with promising architectures can request full-scale training runs on Sentdex's DGX Station hardware.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=udPY5rQVoW0&t=4s) So we're used to seeing AI playing within environments, but here the AI is the environment; it's just a model that we're running with Python that takes key presses from me, the player, and outputs pixel values. — sentdex

> [13:53](https://www.youtube.com/watch?v=udPY5rQVoW0&t=833s) I kind of think there's no reason that we won't have a future where many game engines are entirely or even mostly AI-based like this, or even things like operating systems or in other programs. — sentdex

## Mentioned GAN Theft Auto, GameGAN, NVIDIA, Grand Theft Auto V, Rockstar Games, Python, PyTorch, sentdex, Daniel Kukieła, Pac-Man, CartPole, OpenAI Gym, Vroom, BOXX, Quadro RTX 8000, DGX Station A100, GitHub

## Video description GAN Theft Auto is a Generative Adversarial Network that recreates the Grand Theft Auto 5 environment. It is created using a GameGAN fork based on NVIDIA's GameGAN research.

With GAN Theft Auto, the neural network *is* the environment and you can play within it.

Github: https://github.com/sentdex/GANTheftAuto/

Unboxing and reviewing the DGX Station A100 80GB: https://www.youtube.com/watch?v=0mAesfFt4us

Neural Networks from Scratch book: https://nnfs.io

Channel membership: https://www.youtube.com/channel/UCfzlCWGWYyIQ0aLC5w48gBQ/join Discord: https://discord.gg/sentdex Reddit: https://www.reddit.com/r/sentdex/ Support the content: https://pythonprogramming.net/support-donate/ Twitter: https://twitter.com/sentdex Instagram: https://instagram.com/sentdex Facebook: https://www.facebook.com/pythonprogramming.net/ Twitch: https://www.twitch.tv/sentdex

[Source](https://www.youtube.com/watch?v=udPY5rQVoW0)

中文简报

Sentdex利用可玩的英伟达GameGAN模型替代GTA V游戏引擎生成式神经网络可以通过直接从视频数据中学习物理动力学，从而取代传统的3D游戏引擎和硬编码物理管线。 Sentdex与合作者Daniel Kukieła训练了一个GameGAN模型，能够直接根据玩家的键盘输入和上一帧状态，生成完全可玩的《侠盗猎车手5》（Grand Theft Auto V）驾驶片段。为了在紧迫的时间内采集足够的训练数据，该团队在单个游戏实例中部署了12个并行的规则驱动机器人驾驶员，随后在一台配备320 GB显存的NVIDIA DGX Station A100上训练了该网络。结合用于锐化原始低分辨率输出的定制8倍超采样模型，该神经引擎展现出了涌现行为，包括逼真的蹭墙、镜面光照、动态阴影、背景视差以及部分车辆碰撞。Sentdex思考了完全由真实世界画面训练的AI驱动引擎的未来，并将权重和初始数据集发布到了GitHub供社区实验。 Sentdex与Daniel Kukieła打造了名为“GAN Theft Auto”的可玩模拟程序，其中神经网络直接充当游戏引擎，无需传统物理代码或3D渲染管线即可根据玩家键盘输入实时生成画面。 (0:04) 数据采集通过在单个《侠盗猎车手5》实例中同时运行12个自动化机器人得到加速，这些机器人关闭了碰撞和互相可见性，在同一段高速公路上生成驾驶数据。 (5:19) 模型训练由一台配备4块80 GB GPU、共计320 GB显存的NVIDIA DGX Station A100提供动力，大幅缩短了GameGAN以及约30个定制超采样模型的训练时间。 (5:19) 该神经引擎展现出了涌现的现实世界行为，包括刮擦护栏的物理机制、远山的背景视差、第三人称视角的水平倾斜，以及汽车碰撞时触发的精准致瘫术（PIT maneuvers）。 (10:05)

Reading summary

Sentdex and Daniel Kukieła developed "GAN Theft Auto," an interactive driving simulation where an artificial neural network completely replaces the conventional 3D game engine, rendering visual frames and simulating physics in real time from keyboard inputs. Neural Simulation Replacing the Game Engine (0:04) Unlike neural approaches that apply photorealistic post-processing to existing game graphics pipelines, GAN Theft Auto eliminates traditional hand-crafted physics engines, hardcoded boundary geometry, and standard polygon rendering pipelines entirely. Running as a Python model on a GPU, the neural engine ingests player keypresses (W, A, S, D) along with the prior frame state and streams out raw pixel arrays. Because the simulation functions strictly as matrix mathematics, the system is platform-agnostic across operating systems with graphics acceleration. Iterative Development and Data Harvesting (2:05) The project originated from NVIDIA's GameGAN paper, which sentdex and Kukieła adapted after finding that OpenAI Gym's CartPole lacked dynamic motion. Kukieła then created "Vroom," a custom driving environment used to validate track memory on dual Quadro RTX 8000 GPUs before scaling to Grand Theft Auto V. NVIDIA subsequently granted access to a DGX Station A100 featuring four 80 GB A100 GPUs (320 GB total VRAM). To harvest training data rapidly under time limits, Kukieła automated 12 rules-based driving bots operating simultaneously in a single GTA V instance along a shared highway segment, with car collisions and visibility disabled to log parallel trajectory data. Emergent Physical Mechanics and Upscaling (7:24) Because the initial dataset featured non-crashing bots, early iterations desynchronized upon contacting roadside geometry. The team retrained the system on intentional divider collisions and police cruiser interactions. After testing approximately 30 super-sampling configurations, an 8x upscaling model was paired with the native low-resolution GameGAN feed to improve fidelity. Remarkably, the retrained network learned organic barrier resistance, forcing the vehicle straight and depicting scraping friction rather than clipping. The system also exhibited emergent lighting and spatial dynamics without hardcoded rules, such as dynamic vehicle drop shadows, specular sun glare on matte paint and rear glass, horizon camera tilt, and accurate background parallax as distant mountains approached. Simulation Instability and Future Scope (12:11) Despite emergent physics, dynamic entity handling demonstrated notable failure modes. Collisions with secondary vehicles produced erratic visual artifacts, including police cruisers splitting into two separate objects, dissolving into visual noise, or vanishing abruptly, though rear-quarter impacts occasionally initiated genuine PIT maneuver rotations. The simulation scope remained restricted to a single highway stretch due to hardware timeline limits. Sentdex raised the prospect that future interactive software and game engines could operate largely on neural networks, proposing an "IRL GAN" trained on real-world footage to learn organic physics directly from video data. The base GameGAN model weights, the 8x super-sampler, and a starter dataset have been published on GitHub for community testing, though runtime performance remains unoptimized and the full training archive was withheld due to file size constraints.

精炼阅读摘要

在实验项目“GAN Theft Auto”中，开发者 sentdex 与 Daniel Kukieła 基于 NVIDIA GameGAN 架构，构建了一个完全由深度神经网络替代传统 3D 渲染管线与物理引擎的游戏环境，使玩家输入的控制指令能实时驱动并生成《侠盗猎车手 5》（GTA V）的交互画面。完全由神经网络驱动的纯张量游戏引擎 (0:04) 该项目的核心机制并非对既有游戏画面进行后处理滤镜渲染，而是用运行在 GPU 上的 Python 与 PyTorch 神经网络模型彻底取代传统游戏引擎。整个环境内不存在 Rockstar Games 或开发团队预先编写的几何碰撞体积、刚体动力学算法或传统图形渲染流水线。系统仅通过接收玩家实时的键盘控制信号（如 W、A、S、D）并结合上一帧画面的循环状态，端到端直接推算并输出连续的原生像素阵列。由于底层运算完全基于张量矩阵计算，只要硬件具备图形加速与矩阵运算能力，该仿真系统理论上可在各类操作系统间跨平台迁移。为了改善原生 GameGAN 输出分辨率极低且带有重度像素化马赛克的问题，研发团队设计了双重视图流水线：玩家主屏幕通过专门训练的超采样（super-sampling）模型将画面放大并平滑呈现，而屏幕底部居中的子窗口则同步保留并显示原生的低分辨率推理阵列，直观印证了“AI 本身即是整个运行环境”的技术架构。从自制赛车试错到 12 机器人并行数据采集 (2:05) 项目雏形源于 NVIDIA 发布的 GameGAN 技术，该架构具备分离静态场景与动态物体的独立记忆模块，允许玩家在离开某个场景后重新返回时保留世界状态。早期团队在未获得官方预训练权重的情况下尝试复现，首先选用了 OpenAI Gym 的 CartPole（倒立摆）环境，但因其目标为保持静止、动态画面信息极少而告失败。随后 Daniel Kukieła 自研了高频转弯的简易赛车游戏“Vroom”，并在搭载两张 NVIDIA Quadro RTX 8000 的工作站上耗时数日完成训练，成功验证了模型对草地偏离、赛道记忆以及轮胎转向物理规律的学习能力。在获得配备 4 张 80 GB A100 GPU（共计 320 GB 显存）的 NVIDIA DGX Station A100 支持后，团队转向模拟 GTA V 的公路场景。由于算力租借时间有限且人工驾车采集海量数据耗时过长，Daniel 编写了基于规则的自动驾驶脚本，在同一游戏实例内并发运行 12 个采集机器人。开发团队禁用了车辆间的物理碰撞体积并将彼此设为不可见，使 12 辆车得以在同一段高速公路上互不干扰地并行记录不同轨迹，极大地加速了训练集构建，并在 6 个 epoch 的训练后初步复现了车身阴影、转弯视差与挡风玻璃反光。碰撞物理补全、镜面光照与远景视差涌现 (7:24) 初版 6-epoch 模型存在严重缺陷：由于采集机器人被设定为永不撞车，缺乏碰撞数据的模型在玩家撞向路边路障时会陷入几何失真与状态崩溃。为此，团队调整了数据采集策略，强制机器人在巡航中撞击混凝土中央隔离带，并通过脚本规律性生成警车以提供多车交互样本。数据扩充后，网络展现出令人惊讶的涌现物理特性：当车辆撞向墙体时，模型不再崩溃，而是强制车辆回正方向并模拟出车身贴墙滑行的摩擦阻力感。在视觉表现方面，团队测试评估了 30 个独立的超采样模型，最终选定了一款 8 倍放大模型来平衡画面细节与锐度。伴随模型迭代，神经网络自主推演出了多项复杂的图形物理规律，包括随太阳入射角动态变化的哑光车身与后车窗镜面高光、车体下方的动态投射阴影，以及急转弯时地平线自动倾斜的第三人称跟随视角。更为显著的是，模型在完全没有 3D 深度缓冲的情况下涌现出了真实的远景视差，远处的山脉能够根据车辆行驶车速以符合透视规律的精确速度向前逼近。交互失效伪影与纯数据驱动仿真的未来前景 (12:11) 尽管系统在受限环境下表现亮眼，但其仍存在明显局限。整个训练集仅局限于一段特定高速公路，未能检验其扩展至开放世界大地图时的容量边界；同时，多实体交互仍处于不稳定状态，在与警车相撞时常伴随异常视觉伪影，例如警车因几何预测崩溃而分裂成两半、逐渐模糊成噪点或凭空消失。不过在部分测试中，模型仍展现出实体阻挡特性，甚至在玩家顶撞警车后侧时模拟出了类似执法车辆致偏拦截（PIT 机动）的真实车体旋转动态。全套流程由缺乏 GAN 经验的团队在约两个月内搭建完成，这让 sentdex 对生成式技术的前景持乐观态度。他推测未来的游戏引擎、模拟软件甚至操作系统可能大部分由 AI 直接构建；此外，他提出了构建以现实世界摄像素材为训练集的“IRL GAN / NVGAN”设想，认为该路径有望绕过繁琐的手动物理编码，直接从真实影像中学习自然界的物理规律。目前，该项目的基础权重与超采样模型已在 GitHub 开源，但因文件庞大并未公开全部数据集，且代码未针对消费级显卡的推理性能进行帧率优化。 GAN Theft Auto 证明了仅凭张量计算与生成对抗网络替代传统 3D 渲染与物理引擎的可行性，模型在端到端训练中自发涌现了视差、光照和摩擦阻力；但该系统当前高度依赖受限的道路数据切片，面对复杂多车交互时频繁产生实体分裂或消失的伪影，且推理效率未经过工程优化，距离成为通用可玩的开放世界架构仍有显著差距。

Source quotes

[0:04] sentdex: So we're used to seeing AI playing within environments, but here the AI is the environment; it's just a model that we're running with Python that takes key presses from me, the player, and outputs pixel values. [13:53] sentdex: I kind of think there's no reason that we won't have a future where many game engines are entirely or even mostly AI-based like this, or even things like operating systems or in other programs.

Original transcript

[0:04] welcome everybody to [0:05] gan theft auto what you see here is me [0:08] playing [0:09] inside of a neural networks environment [0:13] this neural network was trained by [0:15] watching a bot play grand theft auto 5 [0:18] on this particular stretch of highway [0:19] learning how to generate new frames [0:21] from previous frames and player input [0:25] so every pixel you see here is generated [0:28] from a neural network while i play [0:31] while generating the visuals or just [0:33] improving them [0:34] is interesting too and there has been [0:36] some interesting work lately with neural [0:38] networks improving the visuals with [0:39] grand theft auto 5 in particular [0:42] in attempts to make it look more [0:43] photorealistic [0:45] this is something much more than that [0:47] the neural network in this case [0:49] is the entire game it's handling what [0:53] happens when you press a key [0:55] to turn left or right there are no rules [0:58] written here [0:59] by us or the original grand theft auto [1:01] engine [1:02] the neural network is controlling all of [1:05] this along with what happens when we try [1:07] to turn [1:07] into another car when we hit a wall or [1:10] what objects in the background should do [1:12] what the light reflections should do [1:14] it's doing all of this based on our [1:16] inputs [1:17] it's the visuals the controls the [1:20] physics [1:21] it's everything we are playing entirely [1:25] within a neural network's [1:26] representation of grand theft auto 5. [1:29] so we're used to seeing ai playing [1:32] within environments but here the ai [1:34] is the environment it's just a model [1:36] that we're running with python that [1:38] takes key presses from me [1:39] the player and outputs pixel values [1:42] we've also added on top some super [1:44] sampling to make things look a little [1:45] less pixelated than they are natively [1:47] so you can see the native in the kind of [1:49] bottom middle screen there [1:51] the other neat thing here is gan theft [1:52] auto runs pretty much on any machine [1:54] that does matrix math [1:55] and has a display hooked up it doesn't [1:57] really matter what your operating system [1:59] is or other underlying software [2:01] beyond a gpu acceleration so where did [2:05] this all come from this is a project of [2:07] daniel kukiela [2:08] and myself based on nvidia's gamegan [2:10] i'll put a link to all this in the [2:12] description [2:13] but besides this page also the paper [2:16] itself [2:17] from the initial research this gamecan [2:20] research is kind of a step beyond purely [2:22] using [2:23] gans you know to take key presses and [2:25] generate frames there's other things [2:27] going on here like static versus dynamic [2:29] things [2:30] being able to leave an environment and [2:31] come back and have the state [2:34] remembered so we've kind of just barely [2:36] scratched the surface based on what we [2:38] did but we were just [2:39] just trying to play around so from the [2:41] moment i saw [2:43] uh the the pacman video [2:46] i just i had to play in the environment [2:50] i wanted it so bad but the code and the [2:52] models and [2:53] even the data to train a model was not [2:56] really made available for quite some [2:57] time and then finally the code came out [2:59] but [3:00] there were no models or data [3:04] so with no data nor model we needed our [3:07] own we started off trying to mimic the [3:09] open ai gym cart pull environment where [3:11] you just balance a pole [3:13] on a cart results here started off [3:16] questionable [3:16] and ended that way too the model had a [3:18] hard time learning this one we [3:20] thought maybe it was because the point [3:21] of the game is to move as little as [3:23] possible so the poll [3:24] moves as little as possible but later we [3:27] thought it might be actually how the bot [3:29] played and generated the data for us for [3:31] training [3:32] so we might wind up revisiting this one [3:34] later just to find out the answer to [3:35] that question [3:36] so we needed something probably a little [3:38] more custom suited to our needs and we [3:40] didn't really know of any out of the box [3:42] environment to use but we did have a [3:44] daniel and he proceeded to make a game [3:46] that we called vroom [3:48] based pretty obviously visually on open [3:50] ai's car racing environment but this was [3:52] written entirely from scratch for our [3:54] specific needs and purpose [3:56] but this would be probably a much better [3:58] environment since there would always be [3:59] movement [4:00] and a lot more turning going on and [4:02] probably just be easier for [4:03] for game gan to understand [4:06] so actually training these models is [4:09] fairly gpu [4:10] memory intensive so we started off [4:12] training on the box machines [4:14] to rtx 8000s and it went actually okay [4:18] i posted some of the preliminary results [4:20] to twitter [4:22] and this was again like one of the very [4:24] first outputs so it's yes it's a little [4:26] foggy and grainy and stuff but we're [4:28] we are pressing keys the car is turning [4:30] even the little wheels are turning [4:32] uh we're very excited to see this it let [4:34] us know that we are indeed [4:36] on the right track pun intended so then [4:38] a couple of days later after some [4:39] further training i posted the following [4:41] results [4:42] the pavement sort of moves a little bit [4:43] the grass sort of moves a little bit [4:45] we're definitely still improving a lot [4:47] less fogginess [4:49] we can leave the track we can turn [4:50] around come back on the track the wheels [4:52] are still [4:53] turning we could even we go pretty far [4:55] off track then turn around and come back [4:56] to the same track so that was really [4:58] cool and [4:59] overall we were just really ecstatic to [5:02] see this again this is [5:03] this is us playing inside of a neural [5:06] network's environment [5:07] it is dictating where the road is all [5:10] the pixel values the turning [5:12] everything is dictated by this neural [5:14] network so this is just [5:16] i don't know it's just so cool to see so [5:19] around this time and video reached out [5:21] and [5:22] suggested that possibly i could check [5:24] out their new dgx station a 180 [5:27] gigabytes [5:28] which has four of the a 180 gigabyte [5:31] cards for a total of 320 [5:33] gigabytes of vram so it became clear [5:35] that hey we could [5:36] we could maybe take this a step further [5:38] so the question became [5:40] okay what is the next reasonable step [5:43] that we could take from here [5:44] and of course as you probably could [5:46] guess the next reasonable step was [5:48] modeling grand theft auto 5. i came up [5:51] with the name [5:52] gan theft auto and basically [5:55] with a name like that it really can't [5:58] fail [5:59] so we just needed a lot of data and as [6:01] quickly as possible i wasn't going to [6:03] have the dgx station for [6:04] forever so we needed to figure out how [6:07] to make the best of the time that we had [6:09] and playing gta manually [6:11] to collect the data really wasn't going [6:12] to be an option because it would take us [6:14] months to to get enough so again daniel [6:17] stepped in to bend [6:19] gta to his will and created a [6:22] rules-based ai [6:23] to drive around gta and collect the data [6:26] but again time was still an issue [6:27] so not just one as many as we could fit [6:31] so here we have 12ai all driving around [6:34] in the same game instance [6:36] and actually on the same stretch of road [6:38] they're all here together [6:40] it's just the other cars are invisible [6:41] and clipping through each other so here [6:43] you can see with all the cars visible [6:45] uh what's going on [6:49] so once we had a good helping of data to [6:51] train with we trained our first model [6:53] and here's the result after six epochs [6:55] of training so [6:57] while this is indeed very pixelated this [6:59] is very clearly [7:00] grand theft auto 5. you can see the [7:03] action in the upper left this is the [7:05] live key presses as i play the game in [7:07] real time notably we also see shadows of [7:10] the car [7:10] light reflections on the back window and [7:13] even the background [7:14] moves as we turn things are looking [7:17] really impressive so impressive that i [7:19] was like okay i guess we gotta ratchet [7:21] this up even further [7:24] the first model that we trained didn't [7:26] include any real boundaries so [7:28] in the actual training data we didn't [7:29] hit walls or anything like that [7:31] so if you drove into a wall in the [7:34] inference [7:35] the model would just get very confused [7:36] very fast but in theory gamegan [7:39] should be able to learn these hard [7:41] boundaries and how to properly react [7:43] when hitting something like a wall or [7:45] this like barrier on the highway [7:48] given our successes up to this point [7:49] along with adding colliding into walls [7:52] we thought why not also add [7:54] other cars including some collisions [7:56] even with those we thought it was [7:58] probably best to keep the number of [8:00] types of cars minimal so we went with [8:02] just police cars and daniel once again [8:04] went to work showing gta [8:06] who was boss by figuring out how to [8:08] spawn in police cars [8:11] but this wasn't without some struggle [8:15] but eventually daniel continued with his [8:17] gta 5 [8:18] domination and we were back on track [8:21] with some more [8:22] training data and this is what created [8:25] our latest [8:25] gann theft auto model which is what you [8:27] saw in the beginning though of course [8:28] this is still quite a bit more pixelated [8:31] so what's the difference [8:32] the difference is super sampling so this [8:34] is the initial output from the model [8:36] and then some quick initial training [8:38] steps [8:39] and then the rest of the training in [8:41] much quicker succession [8:42] so this is actually one of the 30 [8:45] super sampling models that we ended up [8:47] training this is just the one that we [8:49] ended up choosing as [8:50] what we thought was the best looking one [8:52] of them all [8:53] this is times 8 super sampling so pretty [8:56] considerable super sample going on here [8:59] so it's pretty impressive considering [9:00] the base image that we had to [9:02] start from this is the original image [9:07] and this is the up-sampled version of [9:10] that so that's a [9:11] pretty considerable change [9:14] altogether we have at least our first [9:17] version of [9:18] gan theft auto completed some things to [9:21] note that i think are [9:22] pretty cool or the wall collisions did [9:24] eventually [9:25] work to the point at least where the car [9:27] approaches the wall hits the wall [9:28] and then is like forced you know to turn [9:31] straight and and kind of [9:32] uh slide against the wall so i think [9:34] that modeling of physics is actually [9:36] kind of cool we didn't really know if [9:38] that would work or not and it turns out [9:40] it would [9:41] so that was pretty neat to see other [9:43] things are like the [9:44] window sun reflections like we already [9:46] kind of pointed out and just the general [9:48] lighting [9:48] of the area around where the sun is so [9:51] it's not just the window you can [9:52] actually see [9:53] that the sunlight hitting that kind of [9:56] matte finish on the car [9:57] is also being modeled and it moves as [10:00] the car moves [10:01] that is that's just so cool and [10:05] uh along the lines of this kind of light [10:07] casting and all that you also have the [10:08] car's shadow and again this shadow works [10:11] as you would expect so again these are [10:13] kind of like [10:14] physics concepts that aren't really cool [10:16] you know or impressive in your typical [10:18] video game these days but the fact that [10:20] a [10:20] neural network model is doing this and [10:24] doing it pretty accurately [10:26] uh is i think pretty impressive to to [10:28] see that it's doing all these dynamic [10:30] things so [10:31] another thing that it's doing is in the [10:33] earlier models this wasn't happening and [10:35] then i don't know exactly which model so [10:37] just like we trained like 30 super [10:38] samplers we trained [10:40] quite a few game gan models to try to [10:42] get you know improve results and so on [10:45] and uh in one of the later models i [10:47] noticed that [10:48] wow as you travel along the road the [10:51] background [10:52] so the mountains and stuff in the [10:53] background actually [10:55] is coming closer to you just at the [10:58] correct pace [10:59] right um again this is really cool i [11:02] just i [11:03] never expected that that would be [11:04] modeled um at all [11:06] but it but it was and it like just [11:08] really blew me away [11:09] like that we were able to um again just [11:12] have this neural network model [11:14] model that because yeah that's that's [11:16] not really impressive and your [11:18] traditional kind of game engine in an [11:20] environment that you might build with [11:21] 3d environments are very well advanced [11:24] but in terms of [11:26] doing this with a neural network model [11:28] um [11:29] but that's really cool um [11:32] another neat thing is just the vehicle [11:34] roll so when you turn you know the [11:35] horizon kind of rolls a little bit so [11:37] it's you know [11:38] obviously we're looking at this car in [11:40] third person but in grand theft auto [11:42] when you kind of turn the car [11:43] it gives you you know as you're playing [11:45] the perspective that the car [11:47] kind of rolls a little bit it's just [11:48] this little subtle detail [11:50] but again this was picked up by the gan [11:53] theft auto model here [11:54] and it worked quite well so overall i [11:58] think [11:58] i think it did really well here i mean [12:00] this is definitely [12:01] grand theft auto 5. only it's completely [12:04] generated for us [12:05] by an ai and we get to play within it [12:08] and that is just incredibly awesome [12:11] to me our training data only consisted [12:14] of a small section [12:15] of that highway we never actually found [12:17] any limitation there in terms of how big [12:19] the roam range [12:20] could be i would have liked to try [12:24] uh training on a much larger range or [12:26] just you know just incrementally [12:27] increase it [12:28] so that we could find what what is the [12:30] roaming range here like how big of an [12:32] environment could we model [12:33] uh before maybe we have to make the [12:35] model bigger make some changes and so on [12:37] so we really don't know [12:38] um i would have liked to have tested [12:40] that along with doing more work on car [12:42] collisions and just [12:43] having other cars work we had a pretty [12:45] hard time with that [12:46] so so more work there for sure but i [12:49] definitely think adding a bigger roaming [12:50] range was kind of a low-hanging fruit [12:52] that we just didn't have time to do [12:54] on the topic of vehicles and vehicle [12:56] collisions we had fairly interesting [12:58] results including one example where the [13:00] gann handled a collision with a police [13:01] car by simply splitting [13:03] the police car into two for the most [13:06] part the model is clearly confused and [13:08] fuzzy with what to do with cars often [13:10] just simply disappearing them but there [13:13] were examples where we could interact [13:14] with other cars [13:15] somewhat correctly so here you can see [13:17] two windows the left is the default gan [13:19] theft auto output the right is an up [13:21] sample [13:21] you can see the live key presses at the [13:23] top left of that gan output window [13:26] and you can see here how we're unable to [13:28] turn through the other car when it is in [13:30] the way despite trying to [13:32] go left we continue to just kind of be [13:34] dragged along [13:35] straight we can even see that the police [13:37] car rotates a bit if i hit it [13:39] closer to the rear wheel the you know [13:42] pit maneuver doesn't complete but we can [13:44] see it's starting to go in the correct [13:46] direction [13:47] but eventually the police car just [13:49] disappears all together [13:53] i think there's probably a lot more that [13:55] could be done here and arguably [13:57] you know we really only worked on this [13:59] project for maybe like a total of two [14:01] months [14:01] put together and cumulatively and [14:04] neither of us came to this knowing much [14:06] at all about working with gans so [14:08] overall [14:09] um i'm super pleased with with our [14:11] outcome here [14:12] but i definitely think there's a lot of [14:14] more interesting things that [14:16] can happen here and i mean i kind of [14:18] think there's no reason [14:20] that we won't have a future where many [14:23] game engines [14:23] are entirely or even mostly ai based [14:26] like this or even things like operating [14:28] systems or in other programs [14:31] i even have some like real life [14:32] simulating that i would like to test [14:35] with this kind of model structure maybe [14:38] call it mvgan or some something other [14:39] than gamegan [14:41] so maybe stay tuned for that at some [14:43] point but but things like [14:44] modeling physics so like this model [14:47] learned to model [14:48] grand theft autos modeled physics [14:51] but could it have just as well learned [14:54] to model [14:55] maybe some basics uh some basic real [14:57] life [14:58] physics that would be really cool [15:01] because [15:02] video game physics is kind of one of [15:04] those areas that we are kind of weak at [15:06] so anyways lots of cool things and maybe [15:08] stay tuned for [15:09] some irl gan or something like that [15:12] a huge thank you to nvidia for putting [15:14] the research out there and also lending [15:16] us the awesome dgx station [15:18] to help us do the r d at like [15:22] warp speed and finally like me like i [15:25] said i'm sure that many of you will wish [15:27] to play with the models that we've we've [15:29] got here we're going to be hosting the [15:31] base scan theft auto model [15:33] um so the one that i'm actually playing [15:34] here along with [15:36] the super sampler we really made no [15:39] effort to make this run [15:41] super efficiently in inference so your [15:43] mileage [15:44] may vary in the frame rate that you're [15:46] able to actually get but i think many of [15:48] you will be capable of playing [15:50] uh the current version so there will be [15:53] a link in the description [15:54] to the github and the models and some [15:57] basic [15:57] training data for you guys to try your [16:00] own models if you want the full training [16:02] data set [16:03] is a bit large for me to uh host [16:06] but if you can train something that [16:09] looks promising share it with us we'd [16:11] probably be happy to to train it for you [16:14] on the full data set so if you do have [16:16] something interesting uh feel free to [16:18] reach out to us [16:19] anyway that is all for now i hope you [16:21] all found this as [16:22] fascinating as we did and maybe more on [16:25] this concept [16:26] to come in the future if you guys want [16:27] to see more of it thanks for watching [16:29] and [16:30] until next time [17:07] you