TensorFlow Integrates Swift to Eliminate C++ Rewrites in Deep Learning

TensorFlow

Integrating automatic differentiation directly into the Swift compiler eliminates the industry's two-language divide by letting engineers build models and performance-critical algorithms without C++ glue code.

AI-assisted summary of Swift for TensorFlow: The Next-Generation Machine Learning Framework (TF Dev Summit '19) — original source. Check the original for full context.

Source published: 2019-03-07

Key ideas

Swift for TensorFlow integrates first-class automatic differentiation directly into the Swift language and compiler rather than packaging it as an external library. (15:39)
The framework provides native, zero-overhead interoperability with Python, enabling developers to import libraries like NumPy, Matplotlib, and OpenAI Gym inside pure Swift code. (12:08)
Swift addresses the two-language problem by allowing low-level algorithms like Monte Carlo tree search and high-level neural networks to execute in the same language without C++ rewrites. (26:42)
Fast.ai announced a collaboration with the Swift for TensorFlow team, including an upcoming deep learning course co-taught by Jeremy Howard and Chris Lattner alongside the release of version 0.2. (28:45)

Detailed notes

## Background Presented by Distinguished Engineer Chris Lattner and Software Engineer Brennan Saeta at the TensorFlow Dev Summit 2019, this video was published on the official TensorFlow YouTube channel on March 7, 2019. The session introduces Swift for TensorFlow, focusing on language-integrated automatic differentiation and next-generation tooling built to enhance researcher productivity beyond TensorFlow 2.

## TL;DR To overcome the friction of bridging dynamic Python prototyping with C++ production systems, the TensorFlow team developed Swift for TensorFlow, integrating automatic differentiation directly into the Swift programming language and compiler. The presentation showcases interactive Google Colab workflows featuring familiar Keras-style architectures, custom layers with compile-time error detection, and gradient accumulation for simulating distributed training. The speakers demonstrate seamless bidirectional interoperability with Python libraries like NumPy, Matplotlib, and OpenAI Gym, alongside first-class differentiable programming across custom user types. Concluding with a live demo of a MiniGo player running Monte Carlo tree search and neural network evaluations entirely in pure Swift, the team announces their 0.2 release and an educational partnership with Fast.ai.

## Atmosphere & style The presentation carries an energetic, forward-looking developer-conference atmosphere, blending ambitious language-design philosophy with practical, rapid-fire live coding. Chris Lattner and Brennan Saeta share an easy, natural stage rapport, trading off between architectural vision and interactive Google Colab demonstrations punctuated by lighthearted jokes about quick typing and calculus fundamentals. Watching the talk feels intellectually invigorating and pragmatic, moving from the pain points of the classic 'two-language problem' to tangible, browser-based breakthroughs that spark genuine optimism for the future of differentiable programming.

## Bulletin - [15:39](https://www.youtube.com/watch?v=s65BigoMV_I&t=939s) Swift for TensorFlow integrates first-class automatic differentiation directly into the Swift language and compiler rather than packaging it as an external library. - [12:08](https://www.youtube.com/watch?v=s65BigoMV_I&t=728s) The framework provides native, zero-overhead interoperability with Python, enabling developers to import libraries like NumPy, Matplotlib, and OpenAI Gym inside pure Swift code. - [26:42](https://www.youtube.com/watch?v=s65BigoMV_I&t=1602s) Swift addresses the two-language problem by allowing low-level algorithms like Monte Carlo tree search and high-level neural networks to execute in the same language without C++ rewrites. - [28:45](https://www.youtube.com/watch?v=s65BigoMV_I&t=1725s) Fast.ai announced a collaboration with the Swift for TensorFlow team, including an upcoming deep learning course co-taught by Jeremy Howard and Chris Lattner alongside the release of version 0.2.

## Key takeaways - [00:00](https://www.youtube.com/watch?v=s65BigoMV_I&t=0s) Existing machine learning workflows frequently stall at production boundaries, requiring C++ glue code to bridge dynamic eager training with deployment. - [00:00](https://www.youtube.com/watch?v=s65BigoMV_I&t=0s) Swift's open language evolution enables modifying compiler internals directly to embed deep learning primitives and automatic differentiation at the language level. - [02:12](https://www.youtube.com/watch?v=s65BigoMV_I&t=132s) Model definitions adopt a familiar Keras-style syntax while prioritizing handwritten training loops for algorithmic control. - [02:12](https://www.youtube.com/watch?v=s65BigoMV_I&t=132s) Native Google Colab integration enables zero-setup Swift kernel execution and rapid iterative development in the browser. - [06:35](https://www.youtube.com/watch?v=s65BigoMV_I&t=395s) Static type checking and compiler analysis catch typographical errors and tensor shape mismatches prior to heavyweight runtime execution. - [09:05](https://www.youtube.com/watch?v=s65BigoMV_I&t=545s) Custom micro-batch gradient accumulation inside handwritten training loops lets individual researchers simulate large-batch cluster execution on a single accelerator. - [10:33](https://www.youtube.com/watch?v=s65BigoMV_I&t=633s) Bidirectional Python interoperability enables incremental codebase migration and direct calls to existing Python libraries without manual C-API wrappers. - [15:39](https://www.youtube.com/watch?v=s65BigoMV_I&t=939s) The @differentiable attribute directs the compiler to derive analytical derivatives automatically for standard numerical functions and user-defined structs conforming to Differentiable. - [15:39](https://www.youtube.com/watch?v=s65BigoMV_I&t=939s) Discrete types like Int trigger compile-time errors when differentiated, enforcing mathematical validity directly in the type system. - [21:13](https://www.youtube.com/watch?v=s65BigoMV_I&t=1273s) Custom gradients bridging foreign C functions or custom operations are expressed by returning the forward output alongside a pullback closure. - [21:13](https://www.youtube.com/watch?v=s65BigoMV_I&t=1273s) Developers can build user-space debugging tools, such as in-line gradient inspectors, without relying on specialized framework runtime hooks. - [24:42](https://www.youtube.com/watch?v=s65BigoMV_I&t=1482s) Native compilation, automatic TensorFlow graph extraction, and the absence of a Global Interpreter Lock give Swift a high-performance eager execution model. - [26:42](https://www.youtube.com/watch?v=s65BigoMV_I&t=1602s) Pure Swift implementations of systems like MiniGo demonstrate that search-heavy reinforcement learning pipelines can eliminate the performance bottlenecks of Python without C++ rewrites. - [28:45](https://www.youtube.com/watch?v=s65BigoMV_I&t=1725s) The release of version 0.2 delivers foundational compiler and runtime infrastructure as the project works toward broader production readiness.

## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=s65BigoMV_I&t=0s) Innovating Beyond TensorFlow 2 with Swift

Chris Lattner and Brennan Saeta introduce a new initiative from the TensorFlow team designed to push the boundaries of machine learning beyond the newly minted TensorFlow 2. While TensorFlow 2 incorporates years of deep learning lessons, existing workflows still hit significant roadblocks when moving from research to production. Specifically, while eager mode streamlines dynamic model training, deployment often forces developers to write boilerplate C++ code just to drive execution. Similarly, integrating machine learning into larger production applications still requires heavy C++ glue code. Lattner argues that addressing these persistent friction points requires re-examining the entire stack, reaching past runtime libraries down into the programming language and compiler itself. As Chris Lattner explains, “We think that applying new solutions to old problems can help push machine learning even further than before.”

To achieve this, the team turned to Swift, an open-source, cross-platform language built for clarity, safety, and expressive API design. Swift combines static typing with type inference, giving developers the syntax conciseness of dynamic languages while catching programming errors early at compile time. Because Swift features an open language evolution process, the TensorFlow team can evolve the language and its compiler primitives directly—specifically to support first-class automatic differentiation and reusable mathematical types.

```mermaid flowchart TD accTitle: Swift for TensorFlow Stack Innovation accDescr: Diagram showing how Swift for TensorFlow integrates the language, compiler, and runtime to eliminate C++ deployment glue. subgraph LanguageCompiler["Language & Compiler Layer"] Swift["Swift Language & Open Evolution"] Diff["First-Class Automatic Differentiation"] StaticCheck["Static Analysis & Type Inference"] end subgraph Execution["Execution & Runtime Layer"] Model["Dynamic Model Definition"] Opt["Direct Optimization & Execution"] end Swift --> Diff Swift --> StaticCheck Diff --> Model StaticCheck --> Model Model --> Opt ```

- Eager mode simplifies dynamic training but traditionally leaves a gap at deployment time, requiring C++ glue code. - Integrating models into larger end-user software systems frequently demands rewriting or bridging through C++. - Swift’s open evolution model allows direct language-level modifications to support deep learning primitives rather than treating the language as an external wrapper.

---

[02:12](https://www.youtube.com/watch?v=s65BigoMV_I&t=132s) Keras-Style Model Definition and Browser Workflow in Google Colab

Chris Lattner illustrates how model definitions in Swift for TensorFlow feel immediately familiar to existing practitioners. The syntax closely mirrors Keras, where layers are declared and then composed within a forward function in a clean, sequential flow. Rather than hiding the mechanics behind rigid training abstractions, the framework encourages writing training loops by hand. Lattner notes that crafting handwritten training loops provides researchers with complete control and flexibility to experiment with unconventional architectures and algorithmic variations.

Transitioning from slides to a live development environment, Brennan Saeta demonstrates how easy it is to start writing Swift for TensorFlow code without local installation hassles. The framework is integrated natively into Google Colab, allowing users to run interactive Swift Jupyter notebooks directly inside a web browser tab. Saeta instantiates the demo model, sets up a stochastic gradient descent (SGD) optimizer, feeds in synthetic random training data, and runs the training loop, verifying that the training loss decreases over successive steps.

- The model declaration mirrors the familiar Keras API to lower the barrier to entry for incoming practitioners. - Handwritten training loops are prioritized to give researchers maximal freedom over the forward and backward execution mechanics. - Swift for TensorFlow is pre-installed in Google Colab, enabling zero-setup experimentation in browser-hosted notebooks.

---

[04:09](https://www.youtube.com/watch?v=s65BigoMV_I&t=249s) Iterating in 90 Seconds: Skip Connections and Residual Networks

Brennan Saeta demonstrates the rapid iteration cycle made possible by the interactive Colab environment. In machine learning research, initial models rarely fit the final objective without structural tuning. Drawing on modern deep learning literature, Saeta decides to enhance the baseline model by making it deeper and adding residual skip connections to stabilize gradient propagation. He issues a live challenge to implement the structural changes, update the forward pass, and retrain the model in under 90 seconds.

Using Tab autocomplete directly in the browser, Saeta quickly adds an extra dense layer, adjusts tensor shapes to match dimensions for addition, and updates the forward execution logic. The forward pass captures the output of the initial flattening layer into a temporary variable, pushes it through the first dense layer, and performs an element-wise addition of the skip connection directly before invoking the second dense layer. Re-instantiating and executing the network immediately demonstrates a substantially lower training loss compared to the initial sequential baseline.

- Autocomplete in the Colab Swift kernel assists with rapid typing, shape alignment, and parameter completion. - The forward pass uses basic tensor arithmetic (`dense2.applied to tmp + tmp2 in context`) to assemble residual pathways. - The model with skip connections exhibits significantly faster loss reduction over identical synthetic batches.

---

[06:35](https://www.youtube.com/watch?v=s65BigoMV_I&t=395s) Custom Layer Abstraction and Static Error Checking

Moving beyond off-the-shelf sequential layers, Brennan Saeta shows how researchers can invent new building blocks from scratch. To illustrate this, he drafts a custom "double bias dense layer" that augments the standard weight matrix and single bias vector with an additional bias parameter. With a deadpan joke about typing the full implementation in 15 seconds, Saeta reveals the layer structure: an initializer that accepts input and output dimensions, explicit parameter fields ($W$, $B_1$, $B_2$), and a forward call that multiplies the inputs by weights and adds both biases.

Saeta then integrates the custom layer into a new model and runs it against a handwritten training loop. Here, the Swift compiler's static analysis surfaces an immediate advantage by pinpointing human errors before long training jobs fail at runtime. Saeta purposefully exposes a typo in the loss call, where `softmaxCrossEntropy` was passed a misspelled argument label. The compiler immediately flags the exact line and guides the fix. Although this particular experimental layer yields worse loss than the previous model, it highlights how painless it is to explore and debug unorthodox mathematical hypotheses. As Saeta remarks, “Swift for TensorFlow was designed for researchers. And researchers often need to do more than just change models and change the way the architecture fits together.”

- Custom layers define internal parameters directly as properties, with initializers handling tensor allocation. - The forward pass operates as a standard mathematical function without cumbersome layer-wrapping overhead. - Compile-time checking catches argument label typos and mismatched dimensions before executing heavyweight compute graphs.

---

[09:05](https://www.youtube.com/watch?v=s65BigoMV_I&t=545s) Simulating Large-Batch Cluster Training with Gradient Accumulation

Saeta broadens the discussion to infrastructure and training dynamics, highlighting the gap between high-end industrial hardware and the everyday developer setup. Modern state-of-the-art breakthroughs frequently rely on massive TPU superpods or large GPU clusters that demand very large batch sizes to stay computationally saturated. However, individual researchers often have to work with limited hardware, such as a single workstation GPU under a desk, making true large-scale distributed runs inaccessible during early prototyping.

To bridge this gap, Saeta demonstrates how the flexibility of handwritten training loops allows researchers to simulate massive data-parallel cluster execution on a single card. Rather than being restricted by an opaque, rigid `model.fit()` engine, a developer can simply add an inner loop inside each training epoch to accumulate gradients across multiple smaller micro-batches before calling the optimizer. As Brennan Saeta points out regarding the realities of experimental research, “Swift for TensorFlow is, again, designed for researchers. And researchers need to be able to customize everything, right? That's the whole point of research.”

- Large distributed clusters and TPU pods require scaling up batch sizes to maintain hardware efficiency. - Individual researchers can simulate massive-batch distributed regimes on a single machine by implementing custom inner forward-backward accumulation loops. - Handwritten control flow in Swift makes altering standard gradient calculation behavior straightforward without overriding complex framework internals.

[10:33](https://www.youtube.com/watch?v=s65BigoMV_I&t=633s) Simulating Distributed Training and Swift’s Interoperability Vision

Brennan Saeta demonstrates how straightforward it is to express distributed training patterns in Swift for TensorFlow. By wrapping the model execution inside a four-iteration loop, taking gradients at each step, and aggregating them into a single `grads` variable, the code simulates execution across four independent accelerators—such as four GPUs or TPUs—operating in a data-parallel fashion over a batch four times larger than normal. Once aggregated, the optimizer updates the model weights in a synchronous data-parallel process. Brennan emphasizes that this capability provides researchers with immense flexibility without requiring massive distributed infrastructure code.

Chris Lattner returns to highlight the broader developer experience, noting that tooling advantages like early compile-time error detection and code completion are not just about typing fewer characters, but are essential for discovering APIs naturally. He explains how Swift’s heritage influenced their architectural choices: Swift has long featured native, zero-overhead C interoperability, allowing developers to import C header files and call C functions directly without manual wrappers or boilerplate code. The TensorFlow team ported this exact design philosophy directly to Python, bridging the two languages without clunky glue code. As Chris Lattner puts it: “And one of the cool things about this is that that allows you to combine the power of Swift for TensorFlow with all the advantages of the Python ecosystem.”

[12:08](https://www.youtube.com/watch?v=s65BigoMV_I&t=728s) Seamless Python Interoperability: NumPy, Matplotlib, and OpenAI Gym

Brennan Saeta demonstrates how Swift interoperates with Python, acknowledging that the Python data science ecosystem is extraordinarily rich and that researchers shouldn't have to give up their favorite libraries when adopting Swift. In a Jupyter notebook running Swift, Brennan imports `TensorFlow` alongside the `Python` module. Once imported, the `Python` object provides an entry point to import any installed Python library dynamically. He imports `matplotlib.pyplot` and `numpy as np`, seamlessly calling `np.linspace`, `np.sin`, and `np.cos`, and then passing those values directly into `pyplot` to render an inline chart. Brennan jokes that while the snippet looks exactly like the Python code developers write every day, it is completely pure Swift under the hood.

```mermaid flowchart LR accTitle: Swift and Python Interoperability Architecture accDescr: Diagram showing how Swift code communicates through the Python bridge to standard Python libraries and OpenAI Gym. subgraph SwiftRuntime["Swift for TensorFlow Runtime"] Code["Swift Script / Notebook Cell"] Bridge["Python Interop Bridge"] end subgraph PythonEcosystem["Python Ecosystem"] NP["NumPy & Matplotlib"] Gym["OpenAI Gym (CartPole-v0)"] end Code --> Bridge Bridge -->|"Call linspace, sin, plot"| NP Bridge -->|"Step environment & get state"| Gym Gym -->|"Return observation & reward"| Bridge ```

To prove that the bridge handles serious workloads beyond basic array math, Brennan loads OpenAI Gym to train a reinforcement learning agent directly in the browser. Using the CartPole-v0 environment, he constructs a two-layer sequential neural network in Swift, defines hyperparameters, and implements an episode training loop. The loop uses Gym's Python environment to step through actions and extract observations, computes model gradients using Swift, applies updates through a Swift optimizer, and records mean rewards across episodes. The resulting training metrics are immediately plotted using Python’s NumPy and Matplotlib.

Chris Lattner notes that this interoperability is designed for bidirectional workflows, making it possible to combine Swift for TensorFlow with Python TensorFlow code so that teams can migrate existing codebases incrementally rather than performing risky rewrites. Chris jokes about the math foundations of ML: “Now, I think it's fair to say that calculus is an integral part of machine learning.”

- Importing Python in Swift requires only `import Python` and calling `Python.import("package_name")`. - The OpenAI Gym demonstration trains an agent on the `CartPole-v0` control environment using a simple sequential dense network written in pure Swift. - Python variables and methods in Swift can be invoked with standard dot-notation and familiar argument conventions without writing manual C-API bindings.

[15:39](https://www.youtube.com/watch?v=s65BigoMV_I&t=939s) First-Class Language Differentiable Programming

Chris Lattner and Brennan Saeta pivot to what they view as Swift for TensorFlow’s core technical innovation: language-integrated automatic differentiation. Rather than treating differentiation as an external library call or an operation bound strictly to a monolithic tensor class, Swift builds differentiable programming directly into the grammar and compiler of the language itself. Brennan breaks down the mechanics by defining a standard mathematical Swift function called `myFunction` that takes two standard 64-bit floating-point numbers (`Double`), performs products, quotients, and sums, and returns a `Double`. Simply attaching the `@differentiable` attribute instructs the Swift compiler to derive the analytic derivative automatically at compile time.

To query the computed derivative, developers call `gradient`, passing a closure and the evaluation point. Brennan demonstrates taking partial derivatives with respect to specific arguments as well as retrieving the full gradient vector. For deep learning routines where forward predictions are needed alongside backpropagation—such as computing training accuracy or logging losses—Swift provides `valueWithGradient`, which evaluates the function once and returns a tuple containing both the forward output value and the gradient. Brennan highlights that Swift tuples feature named fields rather than mere positional indices, which makes debugging output clean and readable. As Brennan Saeta points out regarding the types involved: “It's just plain old double. This is because automatic differentiation is built right into the language in Swift for TensorFlow.”

```mermaid flowchart TD accTitle: Swift Automatic Differentiation Flow accDescr: Flowchart illustrating how functions annotated with differentiable compile down to gradients and named tuples. Func["Function definition: myFunction(a, b)"] --> Attrib["Annotate with @differentiable"] Attrib --> Compiler["Swift Compiler Derives Derivative"] Compiler --> GradCall["Call gradient(at: point)"] Compiler --> ValGradCall["Call valueWithGradient(at: point)"] GradCall --> GradOnly["Output: Gradient Vector / Partial"] ValGradCall --> NamedTuple["Output: Named Tuple (value, gradient)"] ```

Brennan emphasizes that automatic differentiation is fully extensible to custom, user-defined data structures. He creates a 2D geometric `Point` struct containing `x` and `y` coordinates and conforms it to the `Differentiable` protocol. While the compiler can often synthesize differentiation rules automatically, Brennan advocates explicitly annotating helper methods with `@differentiable` to document intent and trigger compile-time validation. He shows how the compiler catches errors: annotating an integer-returning helper function with `@differentiable` immediately throws a compilation error explaining that `Int` does not conform to `Differentiable`, since integers are discrete values that cannot support infinitesimal steps.

Finally, Brennan writes an extension on `Point` to compute Euclidean vector magnitude. Wanting to illustrate how Swift interfaces with external C code, he intentionally bypasses Swift's native math libraries to invoke the standard C library's double-precision square root function (`sqrt`). When he marks the magnitude function `@differentiable`, the compiler instantly halts with an error: it cannot differentiate through the external C `sqrt` function because it lacks automatic differentiation metadata, setting up the need for custom derivative registration.

- The `gradient(at:...)` function accepts an evaluation point and the mathematical closure to differentiate. - `valueWithGradient` returns both the computed forward value and gradient in a structured, named tuple. - Non-continuous or discrete types like `Int` fail type checking when marked `@differentiable` because they do not satisfy mathematical differentiability requirements. - External foreign-function interfaces (such as C's `sqrt`) cannot be automatically differentiated by the compiler without explicit derivative registration.

[21:13](https://www.youtube.com/watch?v=s65BigoMV_I&t=1273s) Custom Gradients and Building an In-Code Gradient Debugger

Continuing his live demonstration in Colab, Brennan Saeta encounters an error when differentiating a function that calls `sqrt`. He explains that `sqrt` is an external C function compiled by a standard C compiler, and because standard C compilers cannot automatically generate derivatives, Swift for TensorFlow flags that it cannot differentiate it out of the box. Rather than seeing this as an obstacle, Brennan welcomes the error: “This is excellent, because it gives me a great excuse to show you how to write custom gradients.” He defines a wrapper called `mySqrt` that invokes the underlying C function during the forward pass. For the backward pass, the function returns a tuple: the first element is the normal evaluated value, while the second element is a pullback closure that defines the backward pass and captures whatever values are required from the forward pass. Substituting `mySqrt` into the earlier `magnitude` method resolves the error immediately.

Brennan then demonstrates how effortlessly custom types and operations can be combined by writing what he terms a “silly function” marked with the `@differentiable` attribute. This function accepts two `Point` instances and a `Double`, evaluates dot products and magnitudes, and returns a `Double`. Swift for TensorFlow differentiates smoothly across these mixed types, producing full gradients and partial derivatives with respect to individual arguments. Taking this capability a step further, Brennan illustrates how users can build their own debugging tools directly in user-space Swift code. Often, model developers need visibility into the exact gradient values flowing through intermediate stages of a computation. To solve this, Brennan writes a simple `gradientDebugger` function. In the forward pass, it acts as an identity function that takes a `Double` and returns it unmodified; in the backward pass, it intercepts the gradient value, prints it to the console, and returns it downstream. When inserted into the `silly function` right after `a.dot(b)`, running the backward pass prints the exact gradient (`3.80`) directly inside the notebook.

```mermaid flowchart LR accTitle: Custom Gradient and Pullback Flow accDescr: Diagram showing how a custom differentiable function splits into forward computation and a backward pullback closure.

subgraph ForwardPass["Forward Pass"] X["Input x"] --> Core["Compute mySqrt(x)"] Core --> Out["Forward Value: y"] end

subgraph BackwardPass["Backward Pass (Pullback)"] Capture["Captured Values from Forward Pass"] -.-> PB["Pullback Closure"] GradIn["Incoming Gradient dy"] --> PB PB --> GradOut["Propagated Gradient dx"] end

Core --> Capture ```

- When bridging C functions into differentiable Swift code, users implement a custom derivative returning the forward output alongside a pullback closure. - Swift enables mixing and matching arbitrary differentiable data types—such as custom structs (`Point`) alongside primitive floats (`Double`)—within the same differentiable function. - Custom gradient debuggers can be implemented as lightweight identity wrappers that print intermediate gradient values during backpropagation without dedicated runtime hooks.

[24:42](https://www.youtube.com/watch?v=s65BigoMV_I&t=1482s) Language Integration, Performance, and the Two-Language Problem

Taking the stage again, Chris Lattner reflects on the historical foundations of automatic differentiation, observing that while the underlying algorithms date back to the 1970s, it required deep language integration to make them truly practical for modern machine learning workflows. He emphasizes that this unified design opens the door to brand-new research paradigms and refers attendees to extensive online design documents for those wanting to explore the implementation details. Turning to execution speed, Chris explains that Swift is fast because of its native low-level compiler optimization, its absence of a Global Interpreter Lock (GIL) to impede concurrent execution, and sophisticated compiler passes that automatically extract TensorFlow computation graphs behind the scenes. Bringing these elements together, Chris asserts: “The consequence of all this together is that we think Swift has the world's most advanced eager mode.”

Chris then addresses why raw performance and language unity matter beyond simple benchmarks, noting an increasingly painful industry bottleneck: teams prototype neural networks in high-level frameworks but must orchestrate them within larger, performance-critical host applications. Traditionally, this requires exporting computation graphs and writing complex C++ glue code to drive the models. DeepMind's AlphaGo Zero illustrates this friction vividly. The project blends three major components: deep neural networks, Monte Carlo tree search (MCTS) algorithms to explore game states, and massive distributed execution across Tensor Processing Unit (TPU) clusters. While world-class teams can successfully link high-level Python models with C++ tree search, Chris stresses that the friction holds back broader scientific progress: “And if you're an advanced team like DeepMind, you can totally do this. But it's much more difficult than it should be. And we think that breaking down barriers like this can lead to new breakthroughs in science.”

[26:39](https://www.youtube.com/watch?v=s65BigoMV_I&t=1599s) Live Demo: MiniGo and Pure Swift Monte Carlo Tree Search

To show how Swift eliminates the need to rewrite search algorithms in C++, Chris hands the presentation back to Brennan Saeta to showcase MiniGo, an open-source Go engine inspired by AlphaGo Zero. Brennan notes that while the MiniGo team initially wrote their system in standard TensorFlow and achieved great results on a single machine, they ran into severe performance walls when attempting to scale across large TPU pods. Because Python could not keep up with the accelerators, the team had to rewrite critical components like Monte Carlo tree search in C++.

Brennan reveals that the Swift for TensorFlow team reimplemented MCTS and the entire self-play pipeline of MiniGo entirely in pure Swift, running interactively right inside a Colab notebook. In the live demo, Brennan runs a game loop where a game configuration sets up two participants: a neural-network-backed Monte Carlo tree search player competing against a random-move player. The cell executes immediately, displaying the white and black players exchanging moves directly on screen. Brennan highlights that because Swift handles both high-level neural networks and low-level search logic seamlessly, researchers no longer face the artificial split between prototyping and performance: “We think that Swift for TensorFlow is going to unlock whole new classes of algorithms and research, because of how easy it is to do everything in one language with no barriers, no having to rewrite things into C++.”

[28:45](https://www.youtube.com/watch?v=s65BigoMV_I&t=1725s) Fast.ai Collaboration, the 0.2 Release, and the Road Ahead

Chris Lattner returns to highlight the significance of running an end-to-end AlphaGo-style pipeline inside a standard workbook, reiterating that removing engineering barriers will unlock novel hybrid techniques across the research community. However, he insists that Swift for TensorFlow is not designed solely for elite researchers: Swift is widely used across the world to teach introductory programming, making educational accessibility a foundational pillar of the project. To that end, Chris introduces a video message from Jeremy Howard, co-founder of Fast.ai. Jeremy describes Fast.ai's mission to democratize state-of-the-art deep learning—pointing to their sub-$25 ImageNet training speed records and world-class NLP classifiers—before announcing that their upcoming course will feature a major Swift component co-taught by someone who “knows Swift pretty well.”

On stage, Brennan quips, “Chris, I think he means you,” to which Chris laughs, “Yeah, we'll see how this goes.” Chris shares his enthusiasm for co-teaching the next generation of students while working directly with Jeremy to shape high-level, human-centered APIs for Swift for TensorFlow. Wrapping up the session, Chris emphasizes that Swift for TensorFlow remains pure TensorFlow at its core, fully integrated with the broader ecosystem and tooling. He summarizes the project's availability: - The project is fully open source on GitHub with an active community. - All session demonstrations and tutorials are publicly accessible and runnable immediately in Google Colab. - The project has officially shipped its 0.2 release, establishing the foundational compiler and runtime infrastructure. - Chris notes that high-level APIs are actively under development and the platform is not yet intended for production use, inviting advanced researchers to jump in early, test the programming model, and help shape its future.

[31:12](https://www.youtube.com/watch?v=s65BigoMV_I&t=1872s) Session Outro and Closing Music

The presentation comes to an end as the outro music plays, concluding the Swift for TensorFlow technical session at the TensorFlow Dev Summit 2019.

## Notable quotes > [26:39](https://www.youtube.com/watch?v=s65BigoMV_I&t=1599s) We think that Swift for TensorFlow is going to unlock whole new classes of algorithms and research, because of how easy it is to do everything in one language with no barriers, no having to rewrite things into C++. — Brennan Saeta

> [06:35](https://www.youtube.com/watch?v=s65BigoMV_I&t=395s) Swift for TensorFlow was designed for researchers. And researchers often need to do more than just change models and change the way the architecture fits together. — Brennan Saeta

## Mentioned AlphaGo Zero, Brennan Saeta, Chris Lattner, DeepMind, Fast.ai, Google Colab, Jeremy Howard, Keras, Matplotlib, MiniGo, NumPy, OpenAI Gym, Python, Swift, Swift for TensorFlow, TensorFlow, TensorFlow 2, Terra Monk

## Video description TensorFlow is about innovation. In this Swift for TensorFlow session, you will learn about language-integrated automatic differentiation, and tooling optimized for your productivity.

See the revamped dev site → https://www.tensorflow.org/

Watch all TensorFlow Dev Summit '19 sessions → http://bit.ly/TFDS19Sessions Event homepage → http://bit.ly/TFDS19

Subscribe to the TensorFlow YouTube channel → https://bit.ly/TensorFlow1

Speakers: Chris Lattner, Distinguished Engineer Brennan Saeta, Software Engineer

Music by Terra Monk → http://bit.ly/TerraMonkTFDS Event Photo Album → http://bit.ly/TFSummit19

event: TensorFlow Dev Summit 2019; re_ty: Publish; product: TensorFlow - General; fullname: Chris Lattner;

[Source](https://www.youtube.com/watch?v=s65BigoMV_I)

中文简报

TensorFlow集成Swift以消除深度学习中的C++重写将自动微分直接集成到Swift编译器中，使工程师无需编写C++胶水代码即可构建模型和性能关键型算法，从而消除了行业中的双语言隔阂。为了消除动态Python原型开发与C++生产系统之间衔接的摩擦，TensorFlow团队开发了Swift for TensorFlow，将自动微分直接集成到Swift编程语言和编译器中。本次演讲展示了交互式Google Colab工作流，涵盖熟悉的Keras风格架构、支持编译时错误检测的自定义层，以及用于模拟分布式训练的梯度累积。演讲者展示了与NumPy、Matplotlib和OpenAI Gym等Python库的无缝双向互操作性，以及跨自定义用户类型的一等可微编程。团队最后现场演示了一个完全使用纯Swift运行蒙特卡洛树搜索和神经网络评估的MiniGo对弈程序，并宣布发布0.2版本以及与Fast.ai建立教学合作关系。 Swift for TensorFlow将一等自动微分直接集成到Swift语言和编译器中，而不是将其封装为外部库。 (15:39) 该框架提供了与Python的原生、零开销互操作性，使开发人员能够在纯Swift代码中导入NumPy、Matplotlib和OpenAI Gym等库。 (12:08) Swift通过允许蒙特卡洛树搜索等底层算法与高层神经网络在同一种语言中执行，解决了双语言问题，无需使用C++重写。 (26:42) Fast.ai宣布与Swift for TensorFlow团队合作，随着0.2版本的发布，Jeremy Howard和Chris Lattner将共同教授即将推出的深度学习课程。 (28:45)

Detailed notes — zh

## Background 本内容由 TensorFlow 官方频道于 2019 年 3 月 7 日发布在 YouTube 平台上，记录了 TensorFlow Dev Summit 2019 期间的分会演讲。视频由 Chris Lattner 和 Brennan Saeta 共同主讲，官方简介指出本场分享聚焦于 Swift for TensorFlow 这一全新机器学习框架，重点介绍语言集成的自动微分机制以及旨在提升开发者生产力的工具链。 ## TL;DR 演讲围绕下一代机器学习框架 Swift for TensorFlow 展开，旨在打破传统“Python 原型训练加 C++ 繁琐部署”的割裂范式。团队通过将自动微分作为一等公民直接内置进 Swift 语言与编译器底层，实现了高阶类型安全检查、无需胶水代码的极速手写训练循环与自定义梯度。同时，该框架原生支持无缝调用 Python 生态（如 NumPy、Matplotlib 及 Gym）并免除了 GIL 的并发限制，使得像 MiniGo 这类复杂树搜索系统得以完全在单一高级语言中高效实现。大会最后联合 Fast.AI 宣布了教学合作，并正式发布了面向开源社区的 0.2 版本。 ## Atmosphere & style 整场演讲充满前瞻探索与极客工程美感，两位讲者配合默契，将深奥的编译器底层原理讲得举重若轻。现场穿插了密集而流畅的 Google Colab 实时编码挑战，从 90 秒内重构残差网络到故意留 Bug 验证静态报错，台下频繁响起掌声与心领神会的欢笑。随着连线 Jeremy Howard 并宣布 0.2 版本开源，气氛走向高潮，整体传达出一种邀请全球开发者共同参与定义未来机器学习编程范式的热忱与开放情怀。 ## Bulletin - [00:00](https://www.youtube.com/watch?v=s65BigoMV_I&t=0s) TensorFlow 团队联合 Swift 创始人 Chris Lattner 正式发布 Swift for TensorFlow 0.2 版本，将自动微分与机器学习底层支持直接深入到编译器与编程语言层级。 - [15:39](https://www.youtube.com/watch?v=s65BigoMV_I&t=939s) Swift for TensorFlow 引入语言级可微编程，开发者只需添加 `@differentiable` 注解即可让编译器自动合成导数，且微分能力覆盖原生基础类型，摆脱框架张量限制。 - [12:08](https://www.youtube.com/watch?v=s65BigoMV_I&t=728s) 框架支持无缝双向桥接 Python 生态，开发者无需编写胶水代码即可在 Swift 中以原生语法直接调用 NumPy、Matplotlib 及 OpenAI Gym 等工具库。 - [24:36](https://www.youtube.com/watch?v=s65BigoMV_I&t=1476s) 借助无全局解释器锁（GIL）优势与原生低延迟特性，团队展示了纯 Swift 实现的 MiniGo 围棋引擎，打破了过去训练在 Python、树搜索部署在 C++ 的双语言屏障。 - [28:45](https://www.youtube.com/watch?v=s65BigoMV_I&t=1725s) Fast.AI 创始人 Jeremy Howard 宣布与项目达成深度合作，不仅参与高层 API 设计，还将与 Chris Lattner 联合在其深度学习课程中引入 Swift 教学模块。 ## Key takeaways - [00:00](https://www.youtube.com/watch?v=s65BigoMV_I&t=0s) 现有开发范式存在明显断层：动态图模式虽然便于训练，但部署与复杂集成阶段仍被迫依赖繁琐的 C++ 代码，必须从语言和编译器底层进行重构。 - [01:40](https://www.youtube.com/watch?v=s65BigoMV_I&t=100s) 选择 Swift 的核心原因在于其类型系统兼备静态安全与脚本式表达力，且开放的演进机制允许团队直接针对机器学习需求改造语言本身。 - [01:40](https://www.youtube.com/watch?v=s65BigoMV_I&t=100s) 高层 API 沿袭直观的 Keras 声明风格，同时团队极力提倡手写训练循环，以赋予科研人员最大的架构创新自由度。 - [03:03](https://www.youtube.com/watch?v=s65BigoMV_I&t=183s) Swift for TensorFlow 原生集成于 Google Colab，配合 Tab 键补全与即时反馈，研究者可在几十秒内完成跳跃连接（Residual Connections）等架构重构。 - [06:41](https://www.youtube.com/watch?v=s65BigoMV_I&t=401s) 强类型系统与静态分析在编译期即刻捕获参数名拼写等低级错误，避免了动态脚本语言在耗费数小时训练后才报出运行时异常的痛点。 - [09:05](https://www.youtube.com/watch?v=s65BigoMV_I&t=545s) 通过在手写循环中多步累加梯度，单机本地 GPU 即可精准模拟多加速器集群下的大 Batch 数据并行训练逻辑。 - [12:08](https://www.youtube.com/watch?v=s65BigoMV_I&t=728s) 基于原生 C 互操作优势，Swift 进一步实现了与 Python 生态的无缝调用，可在纯 Swift 环境中驱动 OpenAI Gym 强化学习并利用 Matplotlib 绘图。 - [15:39](https://www.youtube.com/watch?v=s65BigoMV_I&t=939s) 可微编程（Differentiable Programming）成为语言一等公民，`valueWithGradient` 函数以带具名元组的形式同时返回计算值与梯度，大幅改善调试体验。 - [18:40](https://www.youtube.com/watch?v=s65BigoMV_I&t=1120s) 任何遵循 `Differentiable` 协议的自定义类型均可直接参与求导，而诸如 `Int` 等离散非连续类型在编译期就会被严格驳回，在概念层阻断数学错误。 - [21:13](https://www.youtube.com/watch?v=s65BigoMV_I&t=1273s) 面对未提供导数规则的底层外部 C 函数，开发者可通过拉回闭包（pullback）手动定义正反向计算，甚至能用纯 Swift 编写无侵入的梯度调试器。 - [24:36](https://www.youtube.com/watch?v=s65BigoMV_I&t=1476s) Swift 摆脱了 Python 的 GIL 性能瓶颈，无需切换至 C++ 即可在单一语言内高效协同运行蒙特卡洛树搜索（MCTS）与深度神经网络。 - [28:45](https://www.youtube.com/watch?v=s65BigoMV_I&t=1725s) 0.2 版本已在 GitHub 全面开源并在 Colab 提供交互环境，虽然高层 API 尚未达到生产就绪阶段，但已向全球科研社区全面开放探索。 ## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=s65BigoMV_I&t=0s) 迈向 TensorFlow 2 之后的新纪元 Chris Lattner 与 Brennan Saeta 登台开场，向与会者介绍一种全新的机器学习设计思路。Chris 指出，TensorFlow 团队的职责是不断推进机器学习技术的最前沿。在过去的几年里，团队从深度学习的实践中汲取了大量经验，并将绝大部分成果凝聚到了 TensorFlow 2 当中。然而，他们的视野并未止步于此，而是把目光投向了更长远的未来。 Chris 剖析了现有开发范式所遭遇的瓶颈：动态图（Eager mode）虽然让动态模型的训练变得异常轻松，但一旦进入模型部署阶段，开发者往往仍被迫编写大量 C++ 代码来驱动模型，这种体验亟待改进；对于那些希望将机器学习模型深度嵌入大型复杂应用程序的研究人员来说，编写 C++ 胶水代码同样是难以避开的负担。此外，机器学习需要更灵活、更富表现力的自动微分机制，比如定义一套可复用的自定义类型并直接无缝参与微分运算。为了从根本上改变现状，必须在技术栈的各个层面展开彻底革新，直接深入到编译器和程序语言本身。正如 Chris Lattner 所言：“And to do that, we need to be able to innovate at all levels of the stack. This includes the compiler and the language. And that's what Swift for TensorFlow is all about.” - **深度创新的技术栈层级**：不仅在库（Library）层面修补，而是深入到编译器（Compiler）与程序语言（Language）底层。 - **开发工作流痛点**：消除模型从 Python 训练向 C++ 部署或宿主应用集成时的割裂感，实现更早期的源码级错误排查并缩短迭代反馈周期。 ```mermaid flowchart TD accTitle: Swift for TensorFlow 核心驱动架构 accDescr: 展示了从语言与编译器层面的革新，如何向上支撑自动微分、可复用类型以及统一的研发部署工作流。 subgraph Core["底层创新 (Innovation at the Stack)"] L["现代编程语言 (Swift)"] C["底层编译器深度融合"] end subgraph Features["核心能力 (Core Capabilities)"] AD["语言级自动微分 (Language-Integrated AD)"] RT["原生可复用类型 (Reusable Types)"] TI["类型推断与静态校验 (Type Inference)"] end subgraph Workflow["开发者体验 (Workflow)"] Dev["高效调试与即时迭代"] Dep["免写繁琐 C++ 的无缝部署"] end Core --> Features Features --> Workflow ``` --- ### [01:40](https://www.youtube.com/watch?v=s65BigoMV_I&t=100s) 认识 Swift：模型定义与手写训练循环 Chris 解释了为何选择 Swift 作为创新的载体。Swift 是一门现代、跨平台的编程语言，设计初衷便是易学易用。它的强类型系统不仅能帮助开发者在编码阶段及早捕获错误，还能自然促成优秀的 API 设计；配合强大的类型推断功能，代码依然能保持类似脚本语言的优雅与轻便。更为关键的是，Swift 完全开源且拥有开放的语言演进机制（Evolution process），这让团队能够直接在语言层面上针对机器学习需求进行定制改造。随后，Chris 展示了 Swift for TensorFlow 的实际代码风格。在定义一个简单网络时，开发者声明模型包含的各层，并通过 `applied(to:)` 编写线性连接的前向传播函数。整个结构与 Keras 高度相似，其意图正是为了让熟悉 Keras 的开发者能够零成本迁移直觉与使用习惯。在模型训练部分，开发者完成模型实例化、指定优化器（如 SGD）并准备输入数据后，便可以直接手写自定义训练循环。Chris 强调，团队之所以格外提倡手写训练循环，是因为它赋予了研究者最大的自由度去尝试各种新奇的构造型态。 - **Keras 风格的 API 继承**：沿袭直观的声明与前向连接范式，降低学习门槛。 - **手写训练循环的灵活性**：避开被高阶封装黑盒限制，完全掌控前向、反向与参数更新的每个细节。 --- ### [03:03](https://www.youtube.com/watch?v=s65BigoMV_I&t=183s) 浏览器即开即用：Google Colab 中的交互式迭代与残差连接为了让上手门槛降到最低，团队让 Swift for TensorFlow 原生内置在基于浏览器的 Google Colab（托管型 Jupyter Notebook）中。Brennan Saeta 接过演示，直接在 Colab 浏览器标签页中运行 Chris 刚刚展示的简单网络模型。配合 SGD 优化器和随机输入数据，单元格执行后几秒内便输出了持续下降的训练损失。紧接着，Brennan 模拟了研究者在日常科研中频繁调整架构的场景：“But if you're ever like me and whenever I try and use machine learning in any application, I start with a simple model. And I've got to iterate. I've got to tweak the model to make it fit better to the task at hand.” Brennan 现场发起了一项“90秒内重构模型”的挑战：遵循深度学习“做得更深”的直觉以及现代顶尖论文推崇的跳跃连接（Skip / Residual Connections），他在模型中追加了一个新的 `Dense` 全连接层，并修改前向传播逻辑——利用临时变量暂存 `Flatten` 层的输出，再将经第一层卷积/全连接处理后的特征与其相加（`tmp + tmp2`），一并输入第二个全连接层。在敲代码的过程中，Colab 提供了顺畅的 Tab 键自动补全。修改完成并重新触发训练循环后，输出的损失值明显比上一版基础模型降低得更快、数值更小，生动展示了语言与工具链协同带来的高效迭代体验。 --- ### [06:41](https://www.youtube.com/watch?v=s65BigoMV_I&t=401s) 自定义层实现与静态分析纠错 Swift for TensorFlow 专为探索未知的研究人员打造，而科研工作绝不仅限于拼装现成层，更需要从零构建全新的数学抽象。Brennan 随即演示了如何在十几秒内手写一个非标层——“双重偏置全连接层”（Double Bias Dense Layer）。该层在维护神经元权重矩阵 $W$ 的同时，额外引入了两个独立的偏置参数 $B_1$ 与 $B_2$；其初始化器接收输入输出维度，前向运算则极其直观地表达为矩阵乘法加上两个偏置项（`matmul(input, weight) + b1 + b2`）。在调用这个全新自定义层组装模型并准备训练时，Brennan 故意在手写循环的代码中留下了一个打字错误（拼错 `softmaxCrossEntropy` 的参数标签）。Swift 的编译器静态分析能力即刻介入，在单元格运行时精准报错并指出需要修改为 `labels:`。修改正确后，模型顺利收敛，尽管最终的损失表现证明“双重偏置”在这个玩具任务上并非一个好点子，但这恰恰证明了该框架能让科研人员用极小的时间成本迅速验证新假设并安全避坑。 - **极简的自定义算子/层抽象**：只需声明权重与运算公式，无需繁复的底层样板代码。 - **编译期与静态分析守护**：告别动态语言运行时跑了数小时才遭遇参数名拼写错误的崩溃，让类型系统在运行前排查低级 Bug。 --- ### [09:05](https://www.youtube.com/watch?v=s65BigoMV_I&t=545s) 单机突破：模拟超大规模分布式批量训练进入分享的高潮阶段，Brennan 将讨论引向更大维度的科研挑战：当前前沿突破往往依赖海量 GPU 集群或庞大的 TPU Pods，而为了喂饱这些超大规模算力池，必须大幅提高训练的批次大小（Batch Size）。然而，绝大多数研究者在日常工作中不可能随时独占昂贵的集群算力，通常手头只有工位桌子底下的一两块 GPU。正如 Brennan Saeta 所强调的：“And researchers need to be able to customize everything, right? That's the whole point of research.” 为了在本地探究大规模 Batch 尺寸下的网络收敛特性，研究者需要一套具备高度定制自由的训练流程。Brennan 开始在 Colab 中展示如何仅用几行精简代码，在原本 1 到 10 个 Epoch 的常规循环内嵌套一层内层循环（Inner Loop），通过多步累积梯度或模拟前向运算，仅凭单张本地显卡就精准模拟出分布式并行集群的大 Batch 训练逻辑。 ### [10:33](https://www.youtube.com/watch?v=s65BigoMV_I&t=633s) 数据并行模拟与 Swift 的 C/Python 互操作性 Brennan Saeta 继续展示上一节的代码逻辑：模型在一次循环中运行四次（“whoops—four times”），并在每个步骤中获取梯度，统一累加到 `grads` 变量中。这一操作在逻辑上模拟了在四个独立加速器（4 张 GPU 或 4 块 TPU）上以数据并行（Data Parallel）模式运行一个 4 倍大小的 batch，随后通过优化器（Optimizer）基于聚合后的梯度更新模型，模拟同步数据并行训练过程。Saeta 总结道：“That's it. That's all there is to it. We're really excited by this sort of flexibility and capabilities that Swift for TensorFlow brings to researchers.” ```mermaid flowchart LR accTitle: 数据并行模拟流程 accDescr: 展示四次模型前向与反向传播获取梯度，梯度累加后交由优化器统一更新权重的过程。 subgraph DataParallel ["数据并行模拟 (4x Accelerators)"] direction TB Step1["Step 1 前向/反向"] --> G1["梯度 1"] Step2["Step 2 前向/反向"] --> G2["梯度 2"] Step3["Step 3 前向/反向"] --> G3["梯度 3"] Step4["Step 4 前向/反向"] --> G4["梯度 4"] end G1 & G2 & G3 & G4 --> Agg["累加到 grads 变量"] Agg --> Opt["优化器更新模型参数"] ``` Chris Lattner 接过发言权，指出尽早捕获错误（Catching errors early）和代码补全（Code completion）等生产力工具不仅是为了自动打字，更核心的价值在于“API 的发现（discovery of APIs）”。接着，Lattner 强调了 Swift 本身具备极强的 C 语言互操作能力（Interoperability）：在 Swift 中可以直接 `import` 一个 C 头文件，无需任何包装器（wrapper）或样板代码（boilerplate），直接调用 C 语言符号。TensorFlow 团队将这一核心理念迁移到了 Python 生态，让研究人员既能享受 Swift for TensorFlow 的底层性能与类型安全，又能无缝调用整个成熟的 Python 生态系统。 - 模型通过 4 次步进与梯度聚合，在单机上模拟 4 块 GPU/TPU 的同步数据并行训练。 - Swift 具备原生 C 互操作性，直接导入 C 头文件即可调用底层符号，无需胶水代码。 - TensorFlow 团队为 Swift 打造了无缝对接 Python 的能力，兼具两者的生态优势。 ### [12:08](https://www.youtube.com/watch?v=s65BigoMV_I&t=728s) 无缝桥接 Python：从 NumPy、Matplotlib 到 OpenAI Gym 强化学习 Brennan Saeta 现场演示了 Swift 与 Python 之间“无缝桥接（Seamless Python interoperability）”的能力。首先，在 Swift 代码中执行 `import TensorFlow` 和 `import Python`，系统会提供一个特殊的 `Python` 对象，允许导入任意 Python 库。Saeta 导入了 `matplotlib.pyplot` 和 `numpy`（并赋值给 `np`），随后直接在 Swift 中以原汁原味的 Python 语法调用 `np.linspace`、正弦 `sin`、余弦 `cos`，并将计算结果直接传给 `pyplot` 画图。运行单元格后，图表顺畅渲染出来，迎来了现场观众的热烈掌声。Saeta 笑着解释，虽然这段代码长得跟 Python 一模一样，但它百分之百是纯 Swift 代码。随后，Saeta 指出科学计算画图只是“玩具级示例（toy example）”，为了展现实际研究场景中的威力，他引入了 OpenAI Gym 强化学习框架。他们要在浏览器的 Swift 笔记本环境中，利用 OpenAI Gym 训练一个控制倒立摆（CartPole-v0）的强化学习 Agent： ```mermaid flowchart TD accTitle: Swift 与 OpenAI Gym 交互流程 accDescr: 展示在 Swift 代码中导入 Gym 环境、定义模型、采集动作梯度并调用 Python 进行可视化的完整流程。 subgraph SwiftS4TF ["Swift for TensorFlow (宿主环境)"] GymEnv["OpenAI Gym: CartPole-v0"] PolicyNet["两层 Dense Sequential 神经网络"] TrainLoop["训练循环 (采集 episode / 计算梯度 / 更新参数)"] end subgraph PythonBridge ["Python Bridge"] NumPyPlot["Python NumPy & Matplotlib"] end GymEnv --> TrainLoop PolicyNet --> TrainLoop TrainLoop -->|"记录平均奖励 (mean rewards)"| NumPyPlot ``` Saeta 导入 `gym`，定义超参数，构建了一个简单的两层 Dense Sequential 神经网络模型及优化器。训练循环直接驱动 Gym 模拟器采集交互回合（episodes），执行模型前向推断，提取梯度并应用到优化器，同时记录平均奖励（Mean rewards），最后再借助 Python 的 NumPy 将奖励曲线绘制出来。Chris Lattner 随后补充说明，团队的长远目标不仅是调用 NumPy 这类流行库，更是在探索如何把 Swift for TensorFlow 与现有的 Python 版 TensorFlow 代码无缝混合，从而为开发者提供一条平滑、渐进式迁移现有代码库的演进路径。 - 语法层面完全保持 Python 习惯，例如 `np.linspace`、`plt.plot`，背后由 Swift 类型系统驱动。 - 在 CartPole-v0 强化学习任务中，模型网络、优化器及循环均在 Swift 中编写，环境仿真与图表绘制调用 Python Gym 和 Matplotlib。 - Lattner 强调正在推进 Swift 与 Python TensorFlow 代码的双向互通，帮助开发者平滑过渡。 ### [15:39](https://www.youtube.com/watch?v=s65BigoMV_I&t=939s) 语言级可微编程（Differentiable Programming） Chris Lattner 幽默地切入下一个核心议题：“Now, I think it's fair to say that calculus is an integral part of machine learning.”（微积分是机器学习不可或缺的一部分，此处 integral 兼有“微积分中的积分”和“必不可少”的双关），引发全场大笑。他指出，由于微分对现代 AI 至关重要，团队选择直接将**可微编程（Differentiable Programming）**作为一等公民内置进 Swift 语言本体，从而为自定义导数计算提供极高的表达自由度与灵活性。 Brennan Saeta 接着拆解底层实现原理。在代码中定义一个常规数学函数 `myFunction`，输入两个 `Double` 浮点数，执行加法、乘法与除法运算，并返回一个 `Double`。只需在函数前添加 `@differentiable` 属性注解，Swift 编译器就会在编译期自动推导并生成其对应的导数实现： ```mermaid flowchart LR accTitle: Swift 语言级自动微分机制 accDescr: 展示普通函数通过 @differentiable 注解生成导数，并通过 gradient 或 valueWithGradient 进行调用的机制。 Func["myFunction(a, b) 双精度计算"] -->|"@differentiable 属性"| Compiler["Swift 编译器推导导数"] Compiler --> GradCall["调用 gradient(at: a) {... }"] Compiler --> ValGradCall["调用 valueWithGradient(at:...)"] ValGradCall --> OutTuple["返回具名元组: (value:..., gradient:...)"] ``` 在调用端，可以直接使用 `gradient` 函数，传入目标闭包以及求导点的输入值（例如某个偏导变量或特定参数），即可精准求得该点处的导数值。对于神经网络训练，研究人员通常既需要反向传播的梯度来更新权重，又需要前向传播预测的损失或标量值来评估准确率（Accuracy）和进行调试。为此，框架提供了 `valueWithGradient` 函数，其执行后返回一个包含计算值与梯度的元组（Tuple）。因为 Swift 支持**具名元组（Named parameters in tuples）**，返回值能以带有字段名的方式清晰打印和解构，显著提升了调试与可读性体验。Saeta 特别提醒大家注意：“These aren't tensor of something. It's just plain old double. This is because automatic differentiation is built right into the language in Swift for TensorFlow.” - `@differentiable`：让 Swift 编译器在编译期自动合成函数的微分实现。 - `gradient`：接收待求导闭包及自变量取值点，输出精确导数或偏导数。 - `valueWithGradient`：同时返回前向输出与梯度，返回类型为具有良好可读性的 Swift 具名元组。 - 自动微分支持纯基础类型（如原生 `Double`），而非必须绑定到特定的 `Tensor` 框架类型对象。 ### [18:40](https://www.youtube.com/watch?v=s65BigoMV_I&t=1120s) 类型系统、编译期检查与外部 C 算子求导挑战为了展示自动微分的极致定制性与扩展性，Saeta 演示了如何在二维空间中从零构建一个几何代数类型。他声明了一个包含 `x` 和 `y` 属性的 `Point` 结构体，并直接声明其遵循 `Differentiable` 协议（`struct Point: Differentiable`）。在这个结构体上，开发者可以自由定义 `dot` 点积等辅助方法。虽然 Swift 编译器在很多上下文中能自动推断求导需求，但官方强烈推荐开发者显式标记 `@differentiable` 来声明代码意图，其关键收益在于**利用强类型系统在编译早期拦截数学逻辑错误**。现场演示中，当开发者尝试在一个返回 `Int`（整数）类型的辅助函数上标注 `@differentiable` 时，编译器立刻报错拦截，明确指出：只有返回值遵循 `Differentiable` 协议的函数才允许被求导，而 `Int` 并不遵循该协议。Saeta 解释道，梯度优化要求变量可以在空间中进行无穷小步长（Infinitesimally small steps）的连续变化，而整数是高度离散的（discrete），因此编译器直接在敲下代码的瞬间阻止了此类数学概念上的不合理操作。随后，Saeta 试图通过为 `Point` 扩展欧几里得距离的 `magnitude`（模长）计算函数，来进一步结合底层 C 语言能力。代码假装 Swift 没有内置开平方函数，特意直接调用 C 语言标准库中的 `sqrt` 函数对 `Double` 进行计算。然而，当他在该函数上添加 `@differentiable` 并运行该单元格时，编译器抛出了错误：“And it totally just-- no, it doesn't quite work.” 原因是 `magnitude` 依赖了外部的 C `sqrt` 函数，而这个来自 C 语言的外部原生符号并没有被编译器标记为可微函数，自动微分系统无法自动穿透并推导出该 C 函数的导数规则。 - 自定义类型只需遵循 `Differentiable` 协议，即可无缝融入语言级自动微分体系。 - 离散类型（如 `Int`）因数学上无法连续求导，在被标记 `@differentiable` 时会被编译器类型检查直接驳回。 - 直接调用未标注可微信息的底层 C 函数（如 C 标准库 `sqrt`）会导致编译期微分链断裂，引发求导失败报错。 ### [21:13](https://www.youtube.com/watch?v=s65BigoMV_I&t=1273s) 自定义梯度与纯 Swift 梯度调试器在演示中遇到一个编译错误：当尝试对向量范数（magnitude）求导时，底层调用的 C 语言标准数学库函数 `sqrt` 无法被自动微分。Brennan Saeta 借机展示了如何编写自定义梯度（custom gradients）。因为 C 编译器至今无法自动为代码生成导数，Swift for TensorFlow 会直接报错提示无法求导。开发者可以通过定义一个包装函数 `mySqrt` 来解决：在正向传播（forward pass）中，它直接调用 C 函数计算平方根；在反向传播（backwards pass）中，它返回一个二元元组（tuple），第一项为正向传播的计算结果，第二项则是一个拉回闭包（pullback closure），用于捕获正向计算所需的值并显式定义反向梯度逻辑。替换原函数后，针对自定义数据类型 `Point` 的 `dot`（点积）与 `magnitude`（模长）计算顺利通过自动微分。 ```mermaid flowchart LR accTitle: Custom Gradient Wrapper Flow accDescr: 演示 Swift 中通过拉回闭包自定义正向与反向梯度的执行流程 subgraph Forward["正向计算 (Forward Pass)"] A["输入 Double"] --> B["C 语言 sqrt 函数"] B --> C["返回正向值"] end subgraph Backward["反向求导 (Pullback Closure)"] C -. 捕获正向状态.-> D["自定义拉回逻辑"] E["传入反向微分量"] --> D D --> F["计算并返回梯度"] end ``` 这种机制支持将可微类型自由组合。Brennan 编写了一个接受两个 `Point` 和一个 `Double` 的函数，展示了不仅能求出函数标量值，还能对各个单独入参求偏导或计算全梯度。为了展示开发者对底层机制的绝对控制权，Brennan 进一步演示了如何用纯 Swift 代码手写一个“梯度调试器（gradient debugger）”：该函数在正向传播时充当恒等函数（identity function）直接透传数值，但在反向传播闭包中捕获梯度并将其 `print` 打印到控制台，从而让开发者无需任何黑盒工具即可在模型内部任意位置观察导数流动。 - Brennan Saeta 指出：“This is excellent, because it gives me a great excuse to show you how to write custom gradients.” - 通过 `pullback` 机制，开发者可以在反向传播中直接捕获并自由处理前向变量。 - 自制梯度调试器在测试计算中精准输出了 $a \cdot b$ 对应位置的梯度数值（3.80）。 --- ### [24:36](https://www.youtube.com/watch?v=s65BigoMV_I&t=1476s) 突破性能瓶颈：无 GIL 限制与纯 Swift 版 MiniGo 案例 Chris Lattner 接过话题，感叹这些自动微分算法核心早在 1970 年代就已提出，但直到通过语言深度集成，才真正带入现代机器学习世界。针对性能问题，Chris 强调 Swift 拥有优异的底层执行效率，且完全不存在 Python 那样的全局解释器锁（GIL）来阻碍高并发。配合高级编译器技术自动识别并提取计算图，Swift 打造了“世界上最先进的 Eager 模式”。在当前业界趋势中，研究团队往往在 Python 中构建神经网络，但在整合到大型应用时，必须导出计算图并编写繁琐的 C++ 代码来负责加载与调度。例如 DeepMind 的 AlphaGo Zero，它将深度学习、蒙特卡洛树搜索（MCTS）和大规模 TPU 集群加速三者融合，但工程实现难度极大。 ```mermaid flowchart TD accTitle: Unified Architecture vs Traditional Two-Language Problem accDescr: 对比传统 Python 加 C++ 架构与 Swift 统一语言架构 subgraph Traditional["传统工作流 (双语言屏障)"] T1["Python: 神经网络定义"] --> T2["导出 Graph 静态图"] T2 --> T3["C++: 重写 MCTS 与调度逻辑"] T3 --> T4["TPU / GPU 集群运行"] end subgraph S4TF["Swift for TensorFlow (单语言统一)"] S1["纯 Swift 代码"] S1 -->|"语言内嵌求导"| S2["神经网络"] S1 -->|"原生低延迟 / 无 GIL"| S3["蒙特卡洛树搜索 (MCTS)"] S2 & S3 --> S4["直接调度 TPU 加速"] end ``` 为了证明语言整合如何打破壁垒，Brennan 演示了社区开源围棋引擎 MiniGo（受 AlphaGo Zero 启发）。MiniGo 团队起初使用标准 TensorFlow 实现，但在大规模 TPU 集群上遇到了性能瓶颈，不得不把蒙特卡洛树搜索用 C++ 重写。而现在，团队使用纯 Swift 重写了 MCTS 以及 MiniGo 的整套自我博弈（self-play）逻辑。在 Jupyter Colab 笔记本中，黑白双方棋手（基于神经网络驱动的 MCTS 与随机选点玩家）直接在 Swift 运行时中流畅对弈，完全不需要 C++ 胶水代码。 - Chris Lattner 强调：“The consequence of all this together is that we think Swift has the world's most advanced eager mode.” - 谈及跨语言工程成本，Chris 评价道：“Now, this is possible today. And if you're an advanced team like DeepMind, you can totally do this. But it's much more difficult than it should be.” - 纯 Swift 实现的 MiniGo 证明了复杂树搜索算法能与神经网络在同一门高级语言中高效协同，消除了系统迁移成本。 --- ### [28:45](https://www.youtube.com/watch?v=s65BigoMV_I&t=1725s) 联手 Fast.AI 与 0.2 版本生态发布 Chris 表示，Swift for TensorFlow 的目标不仅限于顶尖研究员，也极其重视编程教育，因为 Swift 本身也是面向初学者友好的入门语言。随后，Chris 宣布与 Fast.AI 创始人 Jeremy Howard 达成重磅合作。Jeremy 通过视频远程连线发言，回顾了 Fast.AI 在降低前沿技术门槛方面的成就——例如以仅 25 美元的计算成本创下 ImageNet 训练速度纪录，以及构建顶尖的文档分类器。Jeremy 宣布 Fast.AI 的下一期深度学习课程将包含大规模的 Swift 教学模块，并由一位“极其懂 Swift 的神秘人”共同讲授；Brennan 随即打趣点出这正是 Swift 之父 Chris Lattner 本人。除了联合教学，Jeremy 还将深入参与 Swift for TensorFlow 高层 API 的设计。Chris 总结道，Swift for TensorFlow 归根结底依然属于 TensorFlow 大家庭，深度整合了 TensorFlow 生态的底层能力。团队正式发布了包含核心底层设施的 0.2 版本，相关代码全部开源在 GitHub 上，发布会中展示的所有交互式 Demo 和教程均可在 Google Colab 中即刻运行。尽管高层 API 仍在快速演进、尚未达到生产就绪（production ready）状态，团队诚挚邀请广大高阶研究人员参与开源社区，共同探索并塑造这一新一代机器学习编程模型。 - Jeremy Howard 表示：“We think that with Swift for TensorFlow, we can go even further. So we're announcing today that our next course will include a big Swift component co-taught by someone that knows Swift pretty well.” - Chris Lattner 坦言项目阶段：“So this is not ready for production yet as you could guess. But we're very excited about shaping this future, building this out, exploring this new programming model.” - 核心进展：正式发布 0.2 版本，全面开源并提供完整的 Colab 交互式学习与评测环境。 ### [31:12](https://www.youtube.com/watch?v=s65BigoMV_I&t=1872s) 演讲结束与片尾本场演讲至此全部结束。在 Chris Lattner 和 Brennan Saeta 围绕 Swift for TensorFlow 的核心理念、语言级可微编程支持、与 Python 生态的深度互操作性以及开发工具链的讲解之后，现场进入尾声。视频伴随着背景音乐（Terra Monk 制作的音乐）淡出收尾，TensorFlow Dev Summit 2019 的本场分会正式闭幕。 - 演讲以现场背景音乐与活动致谢画面结束。 - 官方在描述中提供了 TensorFlow 官网更新链接、峰会演讲列表合集以及 Swift for TensorFlow 的相关开源资源入口。 ## Notable quotes > [09:05](https://www.youtube.com/watch?v=s65BigoMV_I&t=545s) And researchers need to be able to customize everything, right? That's the whole point of research. — Brennan Saeta > [00:00](https://www.youtube.com/watch?v=s65BigoMV_I&t=0s) And to do that, we need to be able to innovate at all levels of the stack. This includes the compiler and the language. And that's what Swift for TensorFlow is all about. — Chris Lattner ## Mentioned Swift for TensorFlow, Chris Lattner, Brennan Saeta, TensorFlow 2, Swift, Google Colab, Keras, Python, C++, NumPy, Matplotlib, OpenAI Gym, Fast.AI, Jeremy Howard, MiniGo, AlphaGo Zero, DeepMind ## Video description TensorFlow is about innovation. In this Swift for TensorFlow session, you will learn about language-integrated automatic differentiation, and tooling optimized for your productivity. See the revamped dev site → https://www.tensorflow.org/ Watch all TensorFlow Dev Summit '19 sessions → http://bit.ly/TFDS19Sessions Event homepage → http://bit.ly/TFDS19 Subscribe to the TensorFlow YouTube channel → https://bit.ly/TensorFlow1 Speakers: Chris Lattner, Distinguished Engineer Brennan Saeta, Software Engineer Music by Terra Monk → http://bit.ly/TerraMonkTFDS Event Photo Album → http://bit.ly/TFSummit19 event: TensorFlow Dev Summit 2019; re_ty: Publish; product: TensorFlow - General; fullname: Chris Lattner; [Source](https://www.youtube.com/watch?v=s65BigoMV_I)

Source quotes

[9:05] Brennan Saeta: And researchers need to be able to customize everything, right? That's the whole point of research. [0:00] Chris Lattner: And to do that, we need to be able to innovate at all levels of the stack. This includes the compiler and the language. And that's what Swift for TensorFlow is all about.

Source quotes

[26:39] Brennan Saeta: We think that Swift for TensorFlow is going to unlock whole new classes of algorithms and research, because of how easy it is to do everything in one language with no barriers, no having to rewrite things into C++. [6:35] Brennan Saeta: Swift for TensorFlow was designed for researchers. And researchers often need to do more than just change models and change the way the architecture fits together.

Original transcript

[0:00] [MUSIC PLAYING] [0:08] CHRIS LATTNER: Hi, everyone. [0:09] I'm Chris. [0:09] And this is Brennan. [0:10] And we're super excited to tell you about a new approach [0:13] to machine learning. [0:14] So here in the TensorFlow team, it [0:16] is our jobs to push the state of the art [0:18] in machine learning forward. [0:19] And we've learned a lot over the last few years [0:21] with deep learning. [0:23] And we've incorporated most of that all into TensorFlow 2. [0:25] And we're really excited about it. [0:27] But, here, we're looking a little bit further [0:29] beyond TensorFlow 2. [0:31] And what do I mean by further? [0:33] Well, eager mode makes it really easy to train a dynamic model. [0:37] But deploying it still requires you take that and then write [0:41] a bunch of C++ code to help drive it. [0:43] And that could be better. [0:45] Similarly, some researchers are interested in taking machine [0:48] learning models and integrating them into larger applications. [0:51] That also often requires writing C++ code. [0:54] We always want more flexible and expressive autodifferentiation [0:58] mechanisms. [0:59] And one of things we're excited about [1:00] is being able to define reusable types that [1:04] then can be put into new places and used [1:07] with automatic differentiation. [1:09] And we always love improving your developer workflow. [1:12] We want to make you more productive [1:14] by taking errors in your code and bringing them [1:16] to your source and also by just improving your iteration time. [1:21] Now, what we're really trying to do here [1:22] is lift TensorFlow to entirely new heights. [1:24] And to do that, we need to be able to innovate [1:26] at all levels of the stack. [1:28] This includes the compiler and the language. [1:31] And that's what Swift for TensorFlow is all about. [1:34] We think that applying new solutions to old problems [1:37] can help push machine learning even further than before. [1:40] Well, let's jump into some code. [1:42] So first, what is Swift? [1:43] Swift is a modern and cross-platform programming [1:47] language that's designed to be easy to learn and use. [1:51] Swift uses types. [1:52] And types are great, because they can [1:53] help you catch errors earlier. [1:55] And also, they encourage good API design. [1:58] Now, Swift uses type inference, so it's really easy to use [2:01] and very elegant. [2:03] But it's also open source and has an open language evolution [2:07] process, which allows us to change the language [2:09] and make it better for machine learning which is really great. [2:12] Let's jump into a more relevant example. [2:14] This is how you define a simple model in Swift for TensorFlow. [2:18] As you can see, we're laying out our layers here. [2:21] And then we can find a forward function, which composes them [2:24] together in a linear sequence. [2:26] You've probably noticed that this looks a lot like Keras. [2:28] That's no accident, of course. [2:30] We want you to be able to take what you know about Keras [2:32] and bring it forward into this world as well. [2:35] Now, once we have a simple model, let's train it. [2:38] How do we do that? [2:39] All we have to is instantiate our model, [2:42] pick an optimizer and some random input data, [2:44] and then pick a training loop. [2:46] And, here, we'll write it by hand. [2:47] One of the reasons we like writing by hand [2:49] is that it gives you the maximum flexibility [2:51] to play with different kinds of constructs. [2:53] And you can do whatever you want, which is really great. [2:56] But some of the major advantages of Swift for TensorFlow [2:58] are the workflow. [3:00] And so instead of telling you about it, what do you think, [3:02] Brennan, should be show them? [3:03] BRENNAN SAETA: Let's do it. [3:06] All right, the team has thought long and hard [3:08] about what's the easiest way for people to get started [3:11] using Swift for TensorFlow. [3:13] And what could be easier than just opening up a browser tab? [3:16] This is Google Colab, hosted Jupyter notebooks. [3:21] And it comes with Swift for TensorFlow built right in. [3:24] Let's see it in action. [3:26] Here is the layer model, the model [3:28] that Chris just showed you a couple of slides ago. [3:31] And we're going to run it using some random training [3:34] data right here in the browser. [3:36] So we're going to instantiate the model. [3:39] We're going to use the stochastic gradient descent SGD [3:41] optimizer. [3:43] And here we go. [3:45] We have now just trained a model using [3:46] Swift for TensorFlow in our browser on some training data [3:50] right here. [3:51] Now, we can see the training loss is decreasing over time. [3:54] So that's great. [3:56] But if you're ever like me and whenever I try and use [3:58] machine learning in any application, [4:01] I start with a simple model. [4:02] And I've got to iterate. [4:03] I've got to tweak the model to make it fit better [4:06] to the task at hand. [4:09] So since we're trying to show you the workflow, [4:11] let's actually edit this model. [4:13] Let's make it more accurate. [4:16] So here we are. [4:17] Now, let's think a little for a moment. [4:19] What changes do we want to make to our model? [4:21] Well, this is deep learning after all. [4:23] So the answer is always to go deeper, right? [4:26] But you may have been following the recent literature in state [4:30] of the art in that not just sequential layers, [4:32] but skip connections or residual connections [4:35] are a really good idea to make sure your model continues [4:37] to train effectively. [4:39] So let's go through and actually add an extra layer [4:42] to our model. [4:44] Let's add some skip connections. [4:46] And we're going to do it all right now in under 90 seconds. [4:49] Are you ready? [4:50] All right, here we go. [4:51] So the first thing that we want to do [4:53] is we need to define our additional layer. [4:55] So we're going to fill in this dense layer. [4:58] Whoops. [4:59] Flow. [5:01] And one thing you can see is that we're [5:02] using Tab autocomplete to help fill [5:04] in code as we're trying to develop and modify our model. [5:09] Now, we're going to fix up the shapes right here really [5:12] quick, so that the residual connections will all work. [5:17] If I can type properly, that would go better. [5:21] All right, great. [5:21] We have now defined our model with the additional layers. [5:26] All we need to do is modify the forward pass, [5:28] so that we add those skipped connections. [5:30] So here we go. [5:32] The first thing we need to do is we [5:33] need to store in a temporary variable [5:38] the output of the flattened layer. [5:40] Then we're going to feed the output of the flattened layer [5:43] to our first dense layer. [5:44] So dense.applied to tmp in context. [5:51] Now, for the coup de grace, here is our residual connection. [5:56] So dense2.applied to tmp + tmp2 in context. [6:04] Run that. [6:05] And, yes, that works. [6:07] We have now just defined a new model [6:10] that has residual connections and is [6:12] one additional layer deeper. [6:14] Let's see how it does. [6:16] So we're going to reinstantiate our model [6:18] and rerun the training loop. [6:20] And if you recall from the loss that we saw before, [6:22] this one is now substantially lower. [6:24] This is great. [6:26] This is an example of what it's like to use Swift [6:28] for TensorFlow to develop and iterate as you apply models [6:32] to applications and challenges. [6:35] But Swift for TensorFlow-- thank [6:36] [APPLAUSE] [6:41] But Swift for TensorFlow was designed for researchers. [6:44] And researchers often need to do more than just change models [6:48] and change the way the architecture fits together. [6:51] Researchers often need to define entirely [6:53] new abstractions or layers. [6:56] And so let's actually see that live right now. [6:59] Let's define a new custom layer. [7:03] So let's say we had the brilliant idea [7:05] that we wanted to modify the standard dense layer that [7:07] takes a weights and biases and we [7:09] wanted to add an additional bias set of parameters, OK? [7:14] So we're going to define this double bias dense layer right [7:18] here. [7:19] So I'm going to type this really quickly. [7:21] Stand by 15 seconds. [7:23] Here we go. [7:24] [LAUGHTER] [7:26] Woo, all right, that was great. [7:29] So let's actually walk through the codes [7:31] that you can see what's going on. [7:33] So the first thing that we have is we define our parameters. [7:37] So these are a W, like our weights for our neurons, [7:40] and B1, bias one, and B2, our second bias. [7:44] We defined an initializer that takes [7:46] an input size and an output size just like dense does. [7:50] We use that to initialize our parameters. [7:54] The forward pass is very simple to write. [7:56] So here's just applied to. [7:58] And we just take the matrix multiplication of input [8:01] by our weights, and we add in our bias terms. [8:04] That's it. [8:06] We've now just defined a custom layer right in Colab [8:09] in just a few lines of code. [8:12] All right, let's see how it goes. [8:13] Here's model two. [8:15] And so we're going to use our double bias dense layer. [8:18] And we're going to instantiate and. [8:19] We're going to train it using, again, our custom [8:22] handwritten training loop. [8:24] Here's an example of another way that we [8:26] think Swift for TensorFlow makes your life easier. [8:29] Because Swift for TensorFlow can statically analyze your code, [8:32] it can be really helpful to you. [8:34] I don't know about you, but I regularly put typos in my code. [8:38] I don't if you saw me typing earlier. [8:39] And Swift for TensorFlow here is helping you out, right? [8:42] It's saying, look, you mistyped softmaxCrossEntropy. [8:46] This should be labels, OK? [8:49] All right, so we run it. [8:50] We train it. [8:51] And our loss isn't as good. [8:53] This was not the right idea. [8:55] But this is an example of how easy [8:57] it is for researchers to experiment with new ideas [9:00] really easily in Swift for TensorFlow. [9:05] But let's go deeper. [9:08] Swift for TensorFlow is, again, designed for researchers. [9:12] And researchers need to be able to customize everything, right? [9:15] That's the whole point of research. [9:17] And so let's show an example of how [9:20] to customize something other than just a model or a layer. [9:24] So you may have heard that large GPU clusters or TPU super pods [9:28] are, like, delivering massive breakthroughs in research [9:32] and advancing the state of the art in certain applications [9:35] and domains. [9:36] And you may have also heard that, as you scale up [9:39] to effectively utilize these massive hardware pools, [9:42] you need to increase your batch size. [9:45] And so let's say you're a researcher, [9:46] and you want to try and figure out [9:48] what are the best ways to train deep neural networks at larger [9:52] batch sizes. [9:54] Well, if you're a researcher, you probably [9:56] can't buy a whole GPU cluster or rent a whole TPU super pod all [9:59] the time for your experiments. [10:01] But you often have a GPU under your desk. [10:04] So let's see how we can simulate running on a super large data [10:07] parallel GPU or TPU cluster on a single machine. [10:13] We're going to do it all in a few lines of code right here. [10:17] So here's our custom training loop. [10:19] Well, here's the standard part, right? [10:21] This is 1 to 10 training epics. [10:24] And what we're going to do is, instead of just applying [10:27] our model forward once, we have an additional inner loop, [10:30] right? [10:31] So we're going to run our forward pass. [10:33] We're going to run our model-- [10:35] whoops-- four times. [10:38] And we're going to take the gradients for each step. [10:40] And we're going to aggregate them in this grads variable. [10:43] OK? [10:44] This simulates running on four independent accelerators, four [10:48] GPUs or four TPUs in a data parallel fashion [10:51] on a batch that's actually four times as large [10:53] as what we actually run. [10:55] We're going to then use our optimizer [10:56] to update our model along these aggregated gradients, [11:00] again simulating a data parallel synchronous training process. [11:05] That's it. [11:06] That's all there is to it. [11:07] We're really excited by this sort of flexibility [11:09] and capabilities that Swift for TensorFlow [11:12] brings to researchers. [11:13] Back over to you, Chris. [11:15] CHRIS LATTNER: Thanks, Brennan. [11:16] [APPLAUSE] [11:21] So I think that the focus on catching errors early and also [11:24] productivity enhancements like code completion [11:27] can help you in a lot of ways. [11:28] And it's not just about, like, automating typing of code. [11:32] But it can also be about discovery of APIs. [11:35] So another thing that's really cool about Swift as a language [11:37] is that it has really good interoperability with C code. [11:40] And so in Swift, you can literally just [11:42] import a C header file and call symbols directly [11:46] from C without wrappers, without boilerplate or anything [11:48] involved. [11:49] It just works. [11:51] So we've taken this approach. [11:52] In the TensorFlow team, we've taken this approach [11:54] and brought it to the world of Python. [11:56] And one of the cool things about this [11:58] is that that allows you to combine the power of Swift [12:00] for TensorFlow with all the advantages of the Python [12:03] ecosystem. [12:04] How about we take a look? [12:08] BRENNAN SAETA: Thanks, Chris. [12:11] The Python data science ecosystem [12:13] is incredibly powerful and vibrant. [12:15] And we wanted to make sure that, as you start [12:17] using Swift for TensorFlow, you didn't [12:19] miss all your favorite libraries and utilities that you [12:21] were used to. [12:23] And so we've built a seamless Python interoperability [12:26] capability to Swift for TensorFlow. [12:28] And let's see how it works in the context [12:30] of my favorite Python data science library, NumPy. [12:34] So the first thing you need to do [12:35] is import TensorFlow and import Python. [12:38] And once you do that, that defines this Python object [12:41] that allows you to import arbitrary Python libraries. [12:44] So here we import pyplot from the matplotlib library [12:48] and NumPy. [12:49] And we assign it to np, OK? [12:52] After that, we can just use np just as if we were in Python. [12:56] So, here, we call linspace. [12:57] We're going to call sine and cosine. [12:59] And we're going to pass those values to pyplot. [13:03] When we run the cell, it just works exactly as you'd expect. [13:08] [APPLAUSE] [13:09] Thank you. [13:10] [APPLAUSE] [13:13] Now, this sort of kind of looks like the Python code [13:15] you're used to writing, but this is actually pure Swift. [13:19] It just works seamlessly. [13:21] But this is maybe a bit of a toy example. [13:23] So let's see this a little bit more in context. [13:28] OpenAI has done a lot of work in the area [13:31] of reinforcement learning. [13:32] And in order to help that along, they [13:34] developed a Python library called OpenAI Gym. [13:39] Gym contains a collection of environments [13:42] that are very useful when you're trying [13:44] to train a reinforcement learning agent [13:46] across a variety of different challenges. [13:48] Let's use OpenAI Gym to train a reinforcement learning [13:52] agent in Swift for TensorFlow right now our browsers. [13:56] So the first thing we need to do is we need to import Gym. [14:00] We're going to define a few hyperparameters here. [14:02] And, now, we define our neural network. [14:04] In this case, we're going to pick [14:06] a simple two-layer dense network. [14:08] And it's just a sequential model, OK? [14:11] After that, we have some helper code [14:13] to filter out bad or short episodes and whatnot. [14:17] But here's the real meat of it. [14:20] We're going to use Gym to instantiate [14:22] the CartPole v0 environment. [14:25] So that's our env. [14:26] We're going to then instantiate our network right [14:28] here and our optimizer. [14:30] And here's our training loop. [14:32] There we go. [14:33] So we're going to get a bunch of episodes. [14:35] We're going to run our model, get the gradients. [14:38] And we're going to apply those to our optimizer. [14:40] And we're going to record the mean rewards as we train, OK? [14:43] It's all very simple, straightforward Swift. [14:46] And here you can see us training a Swift for TensorFlow model [14:49] in an OpenAI Gym environment using the Python bridge, [14:54] totally seamless. [14:55] And of course, afterwards, you can [14:57] keep track of the parameters of the rewards. [15:01] In this case, we're going to plot the mean rewards [15:04] as the model trained using Python NumPy, totally seamless. [15:08] You can get started using Swift for TensorFlow using [15:11] all the libraries you know and love [15:12] and take advantage of what Swift for TensorFlow [15:16] brings to the table. [15:17] Back over to you, Chris. [15:19] CHRIS LATTNER: Thanks, Brennan. [15:20] So one of the things that I love about this is it's [15:22] not just about being able to leverage [15:24] big important libraries like NumPy. [15:26] We're working on the ability to integrate Swift for TensorFlow [15:29] and Python for TensorFlow code together, [15:31] which we think will provide a nice transition [15:34] path to make you able to incrementally move code [15:37] from one world to the other. [15:39] Now, I think it's fair to say that calculus [15:42] is an integral part of machine learning. [15:45] [LAUGHTER] [15:45] And we think that differentiable programming [15:48] is so important that we've built it right into the language. [15:51] This has a number of huge advantages, [15:53] including enabling more flexible and custom work [15:56] with differentiables, with derivatives. [16:01] And we think this is really cool. [16:02] So I'd like to take a look. [16:06] BRENNAN SAETA: So we've been using [16:07] Swift for TensorFlow's differential programming [16:09] capabilities throughout all of our demos so far. [16:12] But let's really break it down and see what's going on [16:15] at a fundamental level. [16:17] So here we define my function that [16:19] takes two doubles and returns a double based on some products, [16:24] and sums, and quotients. [16:27] If we want Swift for TensorFlow to automatically compute [16:30] the derivative for us, we just annotate it at differential. [16:36] Swift for TensorFlow will then derive the derivative [16:40] for this function right when we run the cell. [16:43] To use this autogenerated derivative, use gradient. [16:48] So gradient takes two things. [16:49] It takes a closure to evaluate and a point that you want [16:53] to evaluate your closure at. [16:56] So here we go. [16:58] This is what it is to take the derivative of a function [17:01] at a particular point. [17:02] So we can change it surround. [17:04] This one's my favorite tasty number. [17:07] And that works nicely. [17:09] Now, one thing to note, we've just [17:11] been taking the partial derivatives of my function [17:14] with respect to a. [17:15] But, of course, you can take the partial derivatives [17:17] and get a full gradient of my function, like so. [17:22] Often with neural networks, however, you [17:24] want to get not just the gradients for your network [17:27] as you're trying to train it and optimize your loss function. [17:30] You often want what the network predicted, right? [17:32] This is really useful to compute accuracy or other debugging [17:36] sort of information. [17:37] And for that you can use value with gradient. [17:40] And that returns a tuple containing [17:43] both the value and the gradient, shockingly enough. [17:46] Now, one thing to note, in Swift, [17:48] tuples can actually have named parameters. [17:50] They aren't just ordered. [17:52] And so you can actually see that it prints out really nicely. [17:56] And you can access values. [17:57] We think this is, again, another nice little thing that helps [18:00] makes writing and debugging code and, more importantly, [18:03] reading it and understanding it later a little bit easier. [18:07] But the one thing that I want to call out [18:10] is that throughout this we've been using just normal types. [18:13] These aren't tensor of something. [18:16] It's just plain old double. [18:18] This is because automatic differentiation [18:20] is built right into the language in Swift for TensorFlow. [18:23] It makes it really easy to express your thoughts [18:26] very clearly. [18:29] But even though it's built into the language, [18:31] we've actually worked very hard to make sure [18:33] that automatic differentiation is totally flexible so that you [18:36] can customize it to whatever it needs you have. [18:40] And instead of telling you about that, let's show you. [18:44] So let's say you want to define an algebra in 2D space. [18:48] Well, you're certainly going to need a point data type. [18:51] So here we define a point struct with x and y. [18:54] And we just market differentiable. [18:57] We can define helper functions on it like dot or other helper [19:01] functions. [19:02] And Swift for TensorFlow, when you try and use your code, [19:04] will often automatically infer when [19:06] you need gradients to be automatically computed [19:10] for you by the compiler. [19:13] But often, it's a good idea to document your intentions. [19:17] And so you can annotate your helper functions [19:19] as @differentiable. [19:23] The other reason why we recommend doing this [19:25] is because this helps catch errors. [19:28] So here, Swift for TensorFlow is actually [19:31] telling you that, hey, you can only differentiate functions [19:35] that return values that conform to differentiable. [19:39] But int doesn't conform to differentiable, right? [19:41] What this is telling you is that my helper function [19:44] returns an int. [19:45] And int is all about taking infinitesimally small steps [19:48] as you optimize and take gradients, right? [19:50] And integers just are very discrete. [19:53] And so Swift for TensorFlow is helping to catch errors, [19:57] you know, right when you write the code very easily [19:59] and tell you what's going on. [20:01] So the solution, of course, is just [20:03] to not mark that as @differentiable. [20:05] The cell runs just fine. [20:09] But let's say we also wanted to go beyond just defining the dot [20:13] product. [20:14] Let's say we also wanted to define the magnitude helper [20:17] function. [20:20] That is the magnitude of the vector defined by the origin [20:22] to the point in question. [20:25] So to do that, we can use the distance formula [20:28] if you're going to do Euclidean distance. [20:30] And we can define an extension on point that does this. [20:35] But we're going to pretend for a moment [20:37] that Swift doesn't include a square root function, [20:40] because I want a good excuse for you [20:43] to see the interoperability with C, OK? [20:46] So we're actually going to use C's square root function that [20:50] operates on doubles, OK? [20:53] So based on the definition of Euclidean distance, [20:55] we can define the magnitude. [20:57] And it totally just-- [20:59] no, it doesn't quite work. [21:00] OK. [21:00] Let's see what's going on. [21:03] So we wanted magnitude to be differentiable. [21:05] And it's saying that you can't differentiate the square root [21:08] function, because this is an external function that hasn't [21:11] been marked as differentiable. [21:13] OK. [21:14] What's that saying? [21:15] Well, the square root, it's a C function. [21:17] It was compiled by the C compiler. [21:19] And as of today, the C compiler can't automatically [21:22] compute derivatives for you. [21:23] So Swift for TensorFlow is saying like, hey, this [21:26] isn't going to work. [21:28] This is excellent, because it gives me a great excuse [21:30] to show you how to write custom gradients. [21:32] All right, so here we define a wrapper function, [21:38] mySqrt square root, that just calls down [21:39] in the forward pass to the C square root function. [21:43] In the backwards pass, we take our double [21:47] and we return a tuple of two values. [21:50] Rather, the first element in the tuple [21:52] is the normal value in the forward pass. [21:55] And the second is a pullback closure. [21:59] And this is where you define the backwards pass [22:02] capturing whatever values you need from the forward pass. [22:05] OK? [22:06] So we're going to run that. [22:08] We're going to go back up to our definition of magnitude [22:10] and change it from square root to my square root, [22:13] rerun the cell, and it works. [22:17] We've now defined point and two methods [22:19] on it, dot and magnitude. [22:21] And we can now combine these in arbitrary [22:23] other silly differentiable functions. [22:25] So, here, I've defined the silly function. [22:28] And we've marked it as differentiable. [22:30] And we're going to take two points. [22:32] We're also going to take a double, right? [22:34] You can mix and match differentiable data types [22:37] totally fluidly. [22:38] We're going to return double, and we're [22:40] going to take magnitudes and do dot products. [22:42] It's a silly function after all. [22:44] And we can then use it, compute the gradient of this function [22:49] at arbitrary data points. [22:52] Just like you'd expect, you can get the value of the function, [22:57] get full gradients in addition to partial derivatives [22:59] with respect to individual values. [23:02] That's been a quick run through of how [23:04] to use customization, custom gradients, custom data [23:08] types with a language integrated automatic differentiation built [23:11] into Swift for TensorFlow. [23:14] But let's go one step further. [23:16] Let's put all this together and show [23:18] how you can write your own debuggers [23:20] as an example of how this power is all in your hands. [23:25] So often when you're debugging models, [23:30] you often want to be able to see the gradients [23:32] at different points within your model. [23:34] And so here, we can just define in regular Swift code [23:37] a gradient debugger. [23:39] Now, it's going to take as input a double. [23:43] And it's going to return it just like normal [23:45] for the forward pass, right? [23:46] It's an identity function. [23:47] On the backwards pass, we're going to get the gradient. [23:50] We're going to print the gradient. [23:51] And then we're going to return it. [23:53] So we're just passing it through just printing it out. [23:55] Now that we've defined this gradient debugger ourselves, [23:58] we can use it in our silly function [24:00] to see what's going on as we take derivatives. [24:03] So gradient debugger, there we go. [24:10] We can rerun that. [24:12] And when we take the gradients, we can now see that for that [24:16] point in the silly function of a dot b, the gradient is 3.80. [24:24] That's been a brief tour through how automatic differentiation [24:26] works in Swift for TensorFlow and how it's customizable [24:30] so that you can harness the power in whatever abstractions [24:33] or systems you need to build. [24:36] Back over to you, Chris. [24:37] CHRIS LATTNER: Thanks, Brennan. [24:38] [APPLAUSE] [24:42] So the funny thing about all this [24:43] is that the algorithms that we're building on [24:45] were defined back in the 1970s. [24:47] And so it really took language integration [24:49] to be able to bring these things forward [24:50] into the world of machine learning. [24:53] There's a tremendous amount of depth here. [24:55] And I'm really excited to see what you all can do with it. [24:57] And we think that this is going to enable new kinds of research [24:59] which we're very excited about. [25:00] There's also a ton of depth. [25:02] And if you're interested in learning more, [25:03] we have a bunch of detailed design documents [25:04] available online. [25:06] Let's talk about performance a little bit. [25:08] Now, Swift is fast. [25:10] And this comes from a number of different things, one of which [25:12] is that the language itself has really good low level [25:14] performance. [25:16] There's also no GIL to get in the way of concurrency. [25:19] Swift for TensorFlow also has some advanced compiler [25:21] techniques to automatically identify graphs for you [25:24] and extract them. [25:25] So you don't have to think about that. [25:27] The consequence of all this together [25:28] is that we think Swift has the world's [25:30] most advanced eager mode. [25:32] Now, you may not care that much about performance. [25:34] You may wonder, like, why do we care about this stuff? [25:37] Well, we're seeing various trends [25:38] in the industry where people are defining neural nets [25:41] and then want to integrate them into other larger applications. [25:44] And typically what this requires is this requires you to export [25:48] graphs and then write a bunch of C++ code to load [25:51] and orchestrate them in various ways. [25:53] So let's take a look at an example of this. [25:56] AlphaGo Zero is really impressive work [25:59] that combines three major classes of techniques. [26:02] Of course, you have deep learning on the one hand. [26:04] But it also drives it through Monte Carlo tracers to actually [26:08] find and evaluate these spaces. [26:11] And then it runs them at scale on industry [26:13] leading TPU accelerators. [26:15] And so it's the combination of all three of these things [26:17] that make AlphaGo Zero possible. [26:19] Now, this is possible today. [26:22] And if you're an advanced team like DeepMind, [26:24] you can totally do this. [26:25] But it's much more difficult than it should be. [26:27] And we think that breaking down barriers like this [26:29] can lead to new breakthroughs in science. [26:31] And we think that this is what can drive progress forward. [26:34] So instead of talking about it again, let's take a look. [26:39] BRENNAN SAETA: MiniGo is an open source go [26:41] player inspired by DeepMind's AlphaGo Zero project. [26:46] It's available on GitHub. [26:47] And you can certainly check out the code. [26:48] And I encourage you to. [26:49] They're also going to be here. [26:51] And they have some other presentations tomorrow. [26:55] But the MiniGo project, when they started out, [26:57] they were getting everything in normal TensorFlow. [26:59] And it was working great until they [27:01] started trying to run at scale on large clusters of TPUs. [27:05] There, they ran into performance problems and had to rewrite [27:08] things like Monte Carlo tree search into C++ in order [27:12] to effectively utilize modern accelerators. [27:15] Here, we've reimplemented Monte Carlo tree [27:18] search and the rest of the MiniGo self-play in pure Swift. [27:25] And we're going to let you see it running right here in Colab. [27:29] So here we define a helper function [27:31] where we take in a game configuration [27:32] and a couple of participants. [27:34] These are our white and black players. [27:36] And we're going to run, basically, play [27:39] the game until we have a winner or loser. [27:43] And so let's actually run this. [27:45] Here, we define a game configuration. [27:46] We're going to play between a Monte Carlo tree search powered [27:50] by neural networks versus just a random player just [27:53] to see how easy it is to flip back and forth [27:56] or mix and match between deep learning [27:58] and other arbitrary machine learning algorithms [28:03] right here in Swift. [28:05] So here you go. [28:06] You can see them playing white, black, playing different moves [28:10] back and forth. [28:12] And it just goes. [28:14] We think that Swift for TensorFlow is going to unlock [28:16] whole new classes of algorithms and research, [28:19] because of how easy it is to do everything in one language with [28:22] no barriers, no having to rewrite things into C++. [28:25] Back over to you, Chris. [28:26] CHRIS LATTNER: Thank, Brennan. [28:28] The cool thing about this, of course, [28:29] is that you can actually do something [28:31] like this in a workbook, which is pretty phenomenal. [28:34] And we've seen many different families of new techniques [28:37] that can be combined together and fused in different ways. [28:40] And bringing this to more people we think [28:42] will lead to new kinds of interesting research. [28:45] Now, our work on usability and design [28:47] is not just about high-end researchers. [28:50] So we love them, but Swift is also [28:53] widely used to teach new programmers how to code. [28:56] And education is very close to our hearts. [28:58] And so I'm very excited to announce a collaboration [29:00] that we're embarking on with none other than Jeremy Howard. [29:03] But instead of talking about this, [29:05] I'd rather have Jeremy speak about it now. [29:09] JEREMY HOWARD: At Fast.AI, we're always [29:11] looking to push the boundaries of what's [29:12] possible with deep learning, especially [29:15] pushing to make recent advances more accessible. [29:18] We've been involved with setting image net speed [29:20] records at a cost of just $25 and building the world's best [29:24] document classifier. [29:26] Hundreds of thousands have become deep learning [29:28] practitioners through our courses [29:30] and are producing state of the art results with our library. [29:33] We think that with Swift for TensorFlow, [29:36] we can go even further. [29:37] So we're announcing today that our next course will include [29:41] a big Swift component co-taught by someone [29:44] that knows Swift pretty well. [29:48] BRENNAN SAETA: Chris, I think he means you. [29:50] CHRIS LATTNER: Yeah, we'll see how this goes. [29:54] So I'm super excited to be able to help [29:56] teach the next generation of learners. [29:58] But I'm also really excited that Jeremy [30:00] will be bringing his expertise in API design [30:02] and helping us shape the high level [30:03] APIs in Swift for TensorFlow. [30:06] So we've talked about many things. [30:08] But the most important part is Swift [30:10] for TensorFlow is really TensorFlow at its core. [30:13] And we think this is super important, [30:14] because we've worked really hard to make sure that it integrates [30:17] with all the things going on in the big TensorFlow family. [30:21] And we're very excited about that. [30:23] Now, you may be wondering where you could get this. [30:26] So Swift for TensorFlow is open source. [30:28] You can find out it on GitHub now. [30:29] And you can join our community. [30:31] It also works great in Colab as you've seen today. [30:34] We have tutorials. [30:35] We have examples. [30:36] And all the demos you saw today are available now [30:38] in Colab, which is great. [30:40] We've also released our 0.2 release, which [30:43] includes all the basic infrastructure [30:44] and underlying technology to power these demos and examples. [30:47] And we're actively working on high level APIs right now. [30:51] So this is not ready for production [30:53] yet as you could guess. [30:55] But we're very excited about shaping this future, [30:58] building this out, exploring this new programming model. [31:00] And this is a great opportunity for advanced researchers [31:04] to get involved and help shape the future of this platform. [31:06] So we'd love it for you to try it out and let [31:09] us know what you think. [31:10] Thank you. [31:10] [APPLAUSE] [31:12] [MUSIC PLAYING]