Expert to Expert: Rich Hickey and Brian Beckman - Inside Clojure

Expert to Expert: Rich Hickey and Brian Beckman - Inside Clojure — jasonofthel33t

jasonofthel33t

AI-assisted summary of Expert to Expert: Rich Hickey and Brian Beckman - Inside Clojure — original source. Check the original for full context.

Source published:

Key ideas

  • Rich Hickey designed Clojure around four foundational pillars: predominantly functional programming with default immutability, a Lisp foundation with syntactic homoiconicity, a concurrency model engineered to tame multi-threaded state, and first-class hosting on the JVM and CLR. (1:17)
  • Clojure rejects object-oriented state management by arguing that OOP conflates identity with mutable memory addresses; instead, Clojure models identity as an explicit container managing a timeline of timeless, immutable values. (37:15)
  • Persistent data structures in Clojure are implemented using shallow trees such as 32-way branching Hash Array Mapped Tries (HAMTs), achieving near-constant-time appends and updates via structural sharing without degrading access performance on older snapshots. (32:13)
  • State management constructs like Atoms (uncoordinated compare-and-swap), Refs (Software Transactional Memory coordinated by dosync), and Agents (asynchronous background execution) strictly require persistent data structures to allow speculative retries without corrupting data. (46:02)

Detailed notes

## Background This video was published on YouTube by the channel jasonofthel33t on March 12, 2013, cross-posted from Microsoft's Channel 9 "Going Deep" series. The show notes describe an in-depth interview between software architect Brian Beckman and Clojure creator Rich Hickey, exploring Clojure as a dynamic, hosted functional Lisp dialect targeting the JVM and CLR, built around immutable persistent data structures, software transactional memory, and seamless host interop.

## TL;DR Rich Hickey outlines the core philosophy behind Clojure, detailing how its four pillars—functional programming, Lisp homoiconicity, hosted platform interop, and first-class concurrency—address the severe pitfalls of traditional multithreaded object-oriented programming. At the whiteboard, Hickey demonstrates how Clojure's persistent collections use shallow tree structures and structural sharing to guarantee near-constant-time updates without destructive mutation. The discussion culminates in Clojure's state model, which strictly decouples timeless, immutable values from time-managing reference constructs such as atoms, transactional refs (`dosync`), and asynchronous agents. Brian Beckman connects these mechanisms to broader systems concepts, including hardware compare-and-swap, Reactive Extensions, and the Time Warp operating system's speculative execution model.

## Atmosphere & style The conversation carries the relaxed yet rigorous atmosphere of two senior software architects deep in the weeds at a whiteboard. Beckman plays the role of an intellectually playful, enthusiastic interlocutor, frequently drawing sharp analogies to Lisp history, physics, and distributed systems, while Hickey delivers clear, disciplined, and pragmatic architectural explanations. It feels less like a standard product pitch and more like a masterclass in language design, with palpable mutual respect and Beckman's growing, contagious excitement as each piece of Clojure's state and data model clicks into place.

## Bulletin - [01:17](https://www.youtube.com/watch?v=wASCH_gPnDw&t=77s) Rich Hickey designed Clojure around four foundational pillars: predominantly functional programming with default immutability, a Lisp foundation with syntactic homoiconicity, a concurrency model engineered to tame multi-threaded state, and first-class hosting on the JVM and CLR. - [37:15](https://www.youtube.com/watch?v=wASCH_gPnDw&t=2235s) Clojure rejects object-oriented state management by arguing that OOP conflates identity with mutable memory addresses; instead, Clojure models identity as an explicit container managing a timeline of timeless, immutable values. - [32:13](https://www.youtube.com/watch?v=wASCH_gPnDw&t=1933s) Persistent data structures in Clojure are implemented using shallow trees such as 32-way branching Hash Array Mapped Tries (HAMTs), achieving near-constant-time appends and updates via structural sharing without degrading access performance on older snapshots. - [46:02](https://www.youtube.com/watch?v=wASCH_gPnDw&t=2762s) State management constructs like Atoms (uncoordinated compare-and-swap), Refs (Software Transactional Memory coordinated by dosync), and Agents (asynchronous background execution) strictly require persistent data structures to allow speculative retries without corrupting data.

## Key takeaways - [02:50](https://www.youtube.com/watch?v=wASCH_gPnDw&t=170s) Clojure targets the JVM and CLR runtimes directly, utilizing dedicated syntax like the dot operator for transparent host member access rather than an opaque foreign function interface. - [03:56](https://www.youtube.com/watch?v=wASCH_gPnDw&t=236s) Lisp was chosen as Clojure's foundation because its homoiconic nature and macro system treat code as native data structures, enabling users to extend syntax and language constructs directly. - [07:11](https://www.youtube.com/watch?v=wASCH_gPnDw&t=431s) Clojure adopts an applicative-order strict evaluation engine, but sequence abstractions like map and filter are lazy by default to support infinite data generation and stream processing. - [10:48](https://www.youtube.com/watch?v=wASCH_gPnDw&t=648s) Unlike pure functional languages that isolate side effects via static type systems and monads, Clojure adopts a dynamic typing model that avoids wrapping host APIs in monadic layers to preserve seamless platform interoperability. - [12:58](https://www.youtube.com/watch?v=wASCH_gPnDw&t=778s) Clojure expands classical Lisp homoiconicity by introducing literal syntax for persistent vectors, maps, and sets alongside traditional singly linked lists. - [15:10](https://www.youtube.com/watch?v=wASCH_gPnDw&t=910s) Arbitrary precision arithmetic is supported by default, automatically promoting integers to BigInt on overflow and maintaining exact ratios to prevent premature loss of numerical precision. - [18:45](https://www.youtube.com/watch?v=wASCH_gPnDw&t=1125s) Persistent vectors and maps act as callable functions of their indices or keys, allowing expressions like (v 0) to retrieve elements directly. - [21:28](https://www.youtube.com/watch?v=wASCH_gPnDw&t=1288s) The Clojure reader operates as an explicit parsing stage that transforms source text directly into live in-memory data structures before passing them to the compiler or evaluator. - [25:15](https://www.youtube.com/watch?v=wASCH_gPnDw&t=1515s) The classical Lisp cons operation is generalized to conj (conjoin), inserting elements where computationally optimal for each collection type, such as appending to vectors and prepending to lists. - [30:17](https://www.youtube.com/watch?v=wASCH_gPnDw&t=1817s) Functional persistence guarantees that older versions of data structures remain fully accessible and continue to meet identical runtime complexity guarantees indefinitely through structural sharing. - [32:13](https://www.youtube.com/watch?v=wASCH_gPnDw&t=1933s) Clojure's persistent vectors adapt Phil Bagwell's Hash Array Mapped Trie (HAMT) concepts into shallow, 32-way branching trees where appending modifies only a localized leaf array and root-path spine. - [40:04](https://www.youtube.com/watch?v=wASCH_gPnDw&t=2404s) Identity is defined as an entity tracked across a timeline, decoupled from the timeless, immutable snapshot values it references at any specific point in time. - [42:47](https://www.youtube.com/watch?v=wASCH_gPnDw&t=2567s) Concurrent reads in Clojure never block writes or require synchronization locks, because readers hold stable references to immutable value snapshots while identity boxes update atomically. - [46:02](https://www.youtube.com/watch?v=wASCH_gPnDw&t=2762s) Atoms provide uncoordinated state updates through compare-and-swap (CAS) loops, retrying state transformations automatically if another thread commits first. - [47:15](https://www.youtube.com/watch?v=wASCH_gPnDw&t=2835s) Software Transactional Memory (STM) coordinates updates across multiple Refs inside a (dosync ...) block, rolling back and retrying transactions atomically if conflicts occur. - [49:15](https://www.youtube.com/watch?v=wASCH_gPnDw&t=2955s) Agents enable asynchronous, fire-and-forget state mutations by dispatching transformation functions onto background thread pools. - [50:04](https://www.youtube.com/watch?v=wASCH_gPnDw&t=3004s) Clojure reference types cannot safely function with mutable data structures because speculative execution and retry loops would leave side effects half-applied.

## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=wASCH_gPnDw&t=0s) Introduction and Rich Hickey's Background

Brian Beckman introduces himself as an interviewer and introduces Rich Hickey, the independent creator of Clojure. Hickey shares his background, noting that he has worked as a professional programmer for a couple of decades. His career focused heavily on broadcast automation and scheduling systems, the National Exit Poll, and various machine listening problems. Beckman characterizes this domain as hardcore real-time engineering with heavy concurrency demands, which Hickey confirms involved high-stress throughput.

Beckman jokes that he is merely a "country physicist who dabbles in programming," though he has a long history with functional languages dating back to discovering Scheme around 1978 or 1979. Beckman shares that he has deployed Scheme in strange environments over the decades, citing his crowning glory as embedding Scheme directly inside the operating system kernel to manage page table replacement policies.

[01:17](https://www.youtube.com/watch?v=wASCH_gPnDw&t=77s) What Clojure Is: The Four Pillars

Before diving deeper into technical nuances, Beckman asks Hickey to define Clojure. Hickey explains that Clojure is a dynamic programming language and a dialect of Lisp, noting that Lisp is unusual in how naturally it supports and encourages dialects. He states that Clojure is founded on four core pillars:

```mermaid flowchart TD accTitle: The Four Pillars of Clojure accDescr: Diagram illustrating the four foundational design principles of the Clojure programming language.

C["Clojure Design"] C --> P1["1. Functional Language<br/>(Immutable by default, not purely enforced)"] C --> P2["2. Dialect of Lisp<br/>(Homoiconic, syntactic macro extensibility)"] C --> P3["3. Concurrency Focus<br/>(Designed to tame multi-threaded state)"] C --> P4["4. Designed to be Hosted<br/>(JVM & CLR first-class platform integration)"] ```

1. **Functional Programming:** It is predominantly functional, emphasizing immutability rather than mathematical purity. 2. **Lisp Foundation:** It retains the distinctive syntactic extensibility of Lisps, which remains unique even after mainstream languages adopted other Lisp features. 3. **Concurrency Focus:** It was born out of Hickey's frustrations with multi-threaded programming in industry: > “I've worked in all these broadcast automation systems were multi-threaded and they were done in C++ and Java and C# and you know basically that's too hard for me, so I was trying to make that easier.” 4. **Designed to be Hosted:** Clojure targets runtimes like the JVM and the CLR, not merely as an incidental compilation target (like ports of Common Lisp or Scheme), but as a first-class design choice meant to directly exploit host libraries, VMs, and runtime ecosystems transparently.

[02:50](https://www.youtube.com/watch?v=wASCH_gPnDw&t=170s) Interoperability and Prefix Host Syntax

Beckman asks whether hosting on the CLR means exposing reflection and an underlying object model. Hickey clarifies that Clojure makes host interop transparent rather than burying it behind a complex, foreign foreign-function interface (FFI). Clojure provides dedicated syntax and a core `.` (dot) special operator to reach directly into the host runtime, allowing programmers to instantiate classes via `new`, call methods, and access fields.

Macros wrap the dot operator to preserve idiomatic Lisp style. Regular function invocation retains standard prefix parenthesized notation `(foo x y z)`, while host member invocation retains the same prefix structure: `(.foo x y z)`.

[03:56](https://www.youtube.com/watch?v=wASCH_gPnDw&t=236s) Why Lisp: Homoiconicity and Syntactic Extension

Beckman inquires why Hickey chose Lisp instead of inventing a brand-new dynamic syntax for the JVM and CLR. Hickey responds that Lisp holds an enduring, unmatched advantage in syntactic extensibility. In conventional programming languages, if a developer wishes for a missing language construct—such as a `for-each` loop—they are stuck waiting for the language designers to implement it. In Lisp, developers do not have to wait: they can write the construct themselves.

Beckman connects this to metaprogramming, recalling Eugene Kohlbecker’s introduction of hygienic macros for Scheme. Hickey notes that many developers conflate Lisp macros with C-preprocessor macros, which are merely crude textual string substitutions. Real programmatic macros are programs that transform programs into other programs.

This transformation is enabled by Lisp being homoiconic (or auto-iconic): code is represented using the language's native data structures. Consequently, programs can manipulate source code using the exact same libraries and collection-manipulation tools used for ordinary data. While pure functional languages like ML, Miranda, and Haskell historically eschewed homoiconicity—following early academic arguments, such as Philip Wadler's paper suggesting calculating is preferable to "scheming"—they turned metaprogramming into an external topic. In Clojure, the language’s evaluation strategy explicitly provides an open hook:

> “The evaluation strategy incorporates a step that says at this point I will call your program to help me evaluate, so macros are a hook that's built into compilation or evaluation that allows your program—everybody can write a program that transforms another program, transforms source code—the trick is that that's part of the evaluation strategy of the language.”

[07:11](https://www.youtube.com/watch?v=wASCH_gPnDw&t=431s) Evaluation Strategy: Strictness vs. Laziness

Beckman asks about Clojure's evaluation strategy, specifically whether it uses strict or lazy evaluation. Hickey explains that Clojure’s core execution model is strict and applicative-order, meaning function arguments are evaluated before being passed into the function.

However, Clojure provides a hybrid approach through sequence abstractions: - The execution engine itself is strictly evaluated. - Sequences (`seq`) can be implemented lazily. - Core sequence transformations like `map` and `filter` are lazy by default.

Hickey and Beckman agree that generating infinite data structures and stream processing represent the primary practical utility of laziness in everyday software engineering, avoiding the global performance overhead of a purely lazy language like Haskell.

[08:06](https://www.youtube.com/watch?v=wASCH_gPnDw&t=486s) Pragmatic Functional Programming and Immutability

When asked to clarify how Clojure is functional without being "pure," Hickey outlines the spectrum of meanings attached to "functional programming": - **Haskell's definition:** Pure mathematical functions applied universally to all expressions, mediated by a type system. - **Historical Lisp definition:** Functions being first-class citizens that can be stored in variables and passed as values. - **Clojure's middle ground:** Emphasizing programming with pure functions that are free of side effects, accept immutable values as arguments, and return new values as output.

While traditional Lisps made immutability an optional programming convention, Clojure forces the issue structurally. Clojure’s core collections—including the data structures used to represent source code—are immutable and persistent. Because developers cannot mutate a collection in place, they must write functions that return new data values.

Hickey notes that Clojure does not strictly outlaw side effects; if a developer chooses to execute I/O inside a function, the dynamic type system will not stop them:

> “It's a dynamically typed language which just respects the programmer's right to do whatever they please.”

Beckman reflects on historical dynamically typed functional languages such as SASL and KRC, contrasting them with Miranda and Haskell. He observes that once languages adopted strong static type systems and type inference, functional constraints became far more manageable—pointing out that without type classes and an overloaded bind operator, using abstractions like monads quickly becomes an oppressive way to program.

[10:48](https://www.youtube.com/watch?v=wASCH_gPnDw&t=648s) Purity, Monads, and the Lisp Lineage

Brian Beckman and Rich Hickey begin by contrasting purely functional paradigms with Clojure's pragmatic design. Beckman notes that while pure functional programming offers algebraic cleanliness, it forces programmers to cart around extra arguments or navigate the complexities and pitfalls of monads, such as composition headaches and Haskell's monomorphism restriction when combining two monads. Beckman observes:

> “So Clojure is definitely coming from the Lisp tradition, walking towards a more functional style of programming; it's definitely not from the Haskell-Miranda tradition.”

When Beckman asks whether it is easy to write pure functions in Clojure and whether the language helps you detect when you leave that pure world, Hickey responds candidly:

> “It's easy to write a functional program or write portions of your program functionally, and there's no help whatsoever in terms of knowing when you've done that correctly or not.”

Because Clojure is dynamically typed, it possesses no type system to enforce the presence or absence of side effects or I/O. Developers can freely instantiate and manipulate mutable Java or.NET objects, retaining both their capabilities and their traditional pitfalls. Hickey explains that attempting to police purity dynamically would be counterproductive: it would require flagging virtually all existing Java and.NET frameworks as dangerous by default. Because seamless interop with host platforms is a primary virtue of Clojure, forcing users to wrap standard APIs in monads would severely undermine that practical advantage.

```mermaid flowchart TD accTitle: Pure Functional Typing vs Clojure Practical Interop accDescr: Contrasts Haskell's strict monadic isolation with Clojure's unmonitored functional core and host interop.

subgraph Haskell ["Haskell / Miranda Tradition"] A["Pure Functional Core"] --> B["Monads & Type System"] B --> C["Guaranteed Purity / Explicit I/O"] end

subgraph Clojure ["Clojure / Lisp Tradition"] D["Idiomatic Functional Core"] --> E["Dynamic Typing / No Effect Tracking"] E --> F["Direct Host Interop: Java /.NET"] F --> G["Practical I/O & Framework Reuse"] end ```

---

[12:58](https://www.youtube.com/watch?v=wASCH_gPnDw&t=778s) Persistent Data Structures and Platform I/O

Clojure achieves practical functional programming not through restrictive type mechanics, but by establishing immutable data structures as the default, idiomatic choice. Hickey points out that traditional Lisps and Schemes primarily privileged singly linked lists as their core recursive, structural type; other structures, such as Common Lisp hash tables, were mutable in-place structures identical to Java collections. Clojure radically expands Lisp's homoiconicity by introducing first-class literal representations for vectors, maps (key-value associations), and sets alongside lists.

All these collections are persistent and immutable, implemented using shallow tree structures that support efficient structural sharing when creating modified versions. Because literal syntax and standard library operations revolve around these immutable types, developers naturally remain in a pure calculation model for the core logic of their programs. When developers eventually need to render graphics, persist records to disk, query a database, or serialize XML, they can reach directly into the Java and.NET standard libraries, where the vast majority of platform utility resides in I/O plumbing.

---

[15:10](https://www.youtube.com/watch?v=wASCH_gPnDw&t=910s) Numerics and Symbolic Computation

Beckman transitions to Clojure's numerical capabilities, prompting Hickey to outline its arithmetic foundations:

- **Arbitrary precision arithmetic**: Integers do not wrap around on overflow; they automatically promote to big numbers (`BigInt`). - **Exact rationals**: Ratios like `22/7` are preserved in exact fractional form rather than immediately converting to floating-point numbers. - **Common Lisp tradition**: While Scheme enforces a strict distinction between exact and inexact numbers across its numeric tower, Clojure adheres closer to Common Lisp, ensuring by default that calculations do not lose precision.

Hickey clarifies that while symbolic simplification (such as algebraic reduction found in Macsyma) is not built directly into the core runtime, Clojure's code-as-data architecture makes implementing symbolic engines straightforward. As an example, he highlights Incanter, a statistical computing and data analysis environment for Clojure modeled after the R language. Because Clojure provides transparent macro extensibility, domain languages like Incanter integrate smoothly with standard Clojure without awkward bridging layers or foreign function interfaces.

---

[16:55](https://www.youtube.com/watch?v=wASCH_gPnDw&t=1015s) Whiteboarding: Literal Syntax and Data Literals

Beckman suggests moving from abstract principles to concrete syntax, inviting Hickey to the whiteboard to illustrate Clojure expressions for those unfamiliar with Lisp. Hickey begins with prefix notation:

```clojure (+ 1 2) ;; => 3 ```

Operators appear first, followed by arguments, with no need for commas or semicolons. Hickey then diagrams the core collection literals and basic types supported natively in syntax:

| Literal Form | Type | Example | |:--- |:--- |:--- | | `(...)` | List | `(1 2 3)` | | `[...]` | Vector | `[1 2 3]` | | `{...}` | Map | `{:a 1:b 2}` | | `#{...}` | Set | `#{:a:b:c}` |

Hickey emphasizes that maps are key-value associations, sets naturally eliminate duplicate elements without preserving order, and all these composite structures are arbitrarily nestable (e.g., maps containing vectors, or vectors containing maps).

---

[18:45](https://www.youtube.com/watch?v=wASCH_gPnDw&t=1125s) Lists vs. Vectors, and Functions as Collections

Beckman asks Hickey to contrast lists with vectors. Hickey explains that the difference lies in their operational complexity:

- **Lists (`List`)**: Standard singly linked lists. Adding an item to the head using `cons` (construct) is $O(1)$. However, accessing an arbitrary element (e.g., the 14th item) or appending to the end requires traversing the chain, running in $O(n)$ linear time. This structure naturally suits recursive decomposition (`first` and `rest`). - **Vectors (`Vector`)**: Persistent array-like structures that provide efficient access by index and efficient append operations at the end.

When Beckman asks how index lookups are expressed syntactically, Hickey highlights one of Clojure's defining design choices: vectors and maps act as callable functions of their keys or indices. A developer can query the first element of vector `v` simply by invoking it as `(v 0)` or by calling `(get v 0)`.

```mermaid flowchart LR accTitle: List vs Vector Algorithmic Characteristics accDescr: Compares the insertion and lookup complexity between Clojure lists and vectors.

subgraph ListOps ["Persistent List (Linked)"] L1["Prepend (cons)"] -->|"O(1)"| L2["Head of List"] L3["Lookup by Index / Append"] -->|"O(n)"| L4["Linear Traversal"] end

subgraph VectorOps ["Persistent Vector (Tree)"] V1["Append to Tail"] -->|"O(log32 n) ~ O(1)"| V2["Vector End"] V3["Indexed Lookup: (v idx)"] -->|"O(log32 n) ~ O(1)"| V4["Direct Retrieval"] end ```

---

[20:33](https://www.youtube.com/watch?v=wASCH_gPnDw&t=1233s) Symbols, Keywords, and Strings

The discussion concludes with Hickey clarifying the distinctions among Clojure's primitive identifiers on the whiteboard:

- **Symbols (`a`)**: Used to name program entities, variables, and bindings. If evaluated directly, a symbol resolves to the value it names; to treat it as a pure symbolic identifier, it must be quoted (`'a`). - **Keywords (`:a`)**: Prefixed with a colon, keywords evaluate to themselves. Hickey succinctly summarizes the fundamental distinction:

> “A symbol might stand for something else in use in a program; a keyword only ever designates itself.”

- **Strings (`"a"`)**: Standard string literals, which in Clojure are direct instances of underlying host platform strings (e.g., `java.lang.String`).

[21:28](https://www.youtube.com/watch?v=wASCH_gPnDw&t=1288s) Literals, Abstract Syntax Trees, and the Reader

Rich Hickey and Brian Beckman explore how Clojure handles strings and literal data structures. In Clojure, strings are native host types—a Java string on the JVM and a CLR string on the.NET runtime—with zero wrapper overhead. Although a string does not turn into a linked list of characters, it can still be treated sequentially, allowing functions like `map` to operate directly across it. Beckman observes that Clojure's map literal syntax represents an arbitrary discrete mapping where both keys and values can be any arbitrary value, rather than being restricted to trivial name-value pairs.

Beckman points out that Lisp’s prefix notation provides a direct, literal representation of abstract syntax trees (ASTs). When expressions nest, they map straight onto trees without ceremony. He recalls the industry reaction when XML emerged: “when XML came out the old-timers said this is nothing but Lisp with right well noisy Lisp right and the XML-er says what's this list thing that's that right hiding the structure but it's just trees right.” Hickey explains that writing a Clojure program fundamentally consists of writing data structures—primarily lists, but also vectors, maps, and sets.

The core distinction of Lisp lies in having a two-phase syntax rather than defining language syntax purely in terms of text characters. Hickey emphasizes: “it's a two-phase syntax there is your source code right right then there's something called a reader and the source code is text what the reader outputs is data.” Source code passes into the reader, which constructs live in-memory data structures—such as a list containing the symbol `+` and the numbers `1` and `2`. That data structure is then handed to the evaluator or compiler. Because of this separation, programs can generate data structures dynamically in memory and pass them directly to the evaluator without touching a disk file, or pass data to macros that transform and return new data structures before evaluation.

```mermaid flowchart LR accTitle: Clojure Two-Phase Syntax Pipeline accDescr: Diagram illustrating source text passing through the Reader into Data Structures, interacting with Macros, and entering the Evaluator or Compiler.

subgraph Phase1["Textual Phase"] A["Source Text<br/>File / REPL"] end

subgraph Phase2["Data Structure Phase"] B["Reader"] C["In-Memory Data<br/>Lists, Vectors, Maps"] M["Macro Expansion<br/>Data to Data"] end

subgraph Phase3["Execution Phase"] D["Compiler / Evaluator<br/>Bytecode Generation"] end

A --> B B --> C C --> M M --> C C --> D ```

- When the evaluator encounters a list, it treats the first item as an operator or function call and the remainder as arguments. - Vectors, maps, and sets evaluate directly to themselves as literal data. - While a list can be treated as literal data by quoting it with a tick mark (`'`), idiomatic Clojure uses vectors (`[...]`) for generic data grouping. - Because data structures print and read symmetrically, Clojure code needs no external serialization formats like JSON or XML.

---

[25:15](https://www.youtube.com/watch?v=wASCH_gPnDw&t=1515s) Persistent Collections, conj, and Generic Abstractions

Hickey details the behavior of Clojure collections, noting that vectors provide fast lookup by index and fast append at the end. Because these collections are persistent, exploratory and recursive algorithms can build new vectors conditionally and discard them freely without mutating earlier versions. Backtracking requires no cleanup steps because the original collection remains untouched.

Clojure abstracts classical Lisp operations: the traditional `cons` operation, which attached an element to the head of a singly linked list, has been broadened into `conj` (conjoin). Rather than forcing everything to behave as a head-inserted list, `conj` adds elements to a collection at the position most natural and efficient for that data structure's concrete type. Conjoining to a list prepends to the front; conjoining to a vector appends to the end; conjoining to a map inserts key-value pairs; and conjoining to a set adds unique elements.

Generic higher-order operations build on these primitives. Hickey demonstrates `into`, which pours elements from one collection into another by performing a recursive `reduce` using `conj`:

```clojure (into target-collection (map f source-collection)) ```

This uniform abstraction model functions across all collections because every Clojure collection implements a sequence interface. Under the hood, Clojure structures are defined against standard JVM or CLR host interfaces rather than opaque engine internals. Hickey explains that Clojure establishes an abstraction hierarchy with distinct interfaces for associativity (shared by maps and vectors), constant-time countability, and sequentiality. The platform ships with concrete implementations—such as red-black trees for sorted structures and hash array mapped tries (HAMTs) for hashed collections—while allowing user-defined data structures implementing these interfaces to interoperate transparently with `into`, `map`, and `reduce`.

---

[27:37](https://www.youtube.com/watch?v=wASCH_gPnDw&t=1657s) Host Interoperability and the Tension Between Mutable and Immutable

Beckman asks if this interface-driven design makes it easy for JVM and CLR hosts to invoke the Clojure evaluator and achieve full, bi-directional interop. Hickey confirms this, noting that Clojure's foreign function interface is "not so far" because it compiles directly to host bytecode and talks to the host without runtime barriers.

Hickey grabs the whiteboard eraser to show host syntax—joking about “violating immutability here” as he wipes the board. He demonstrates standard syntax shortcuts for calling host APIs: - Invoking instance methods: `(.toUpperCase "hello")` - Accessing static members: `(Math/PI)` - Instantiating host classes: `(new MyClass)` or `(MyClass.)`

Beckman points out that interacting directly with host objects immediately brings programmers face-to-face with the friction between mutable and immutable paradigms. Clojure's native data structures are persistent and immutable—transformations produce new values rather than destroying the source—whereas standard host objects are perforce mutable and observably side-effecting. Beckman remarks that working in an immutable world requires a completely different mindset, and expresses a strong desire to explore how Clojure navigates both worlds:

> “I'd like to understand the Clojure concept of identities and how you how we navigate the waters of the the mutable world which has one sort of feel style intuition that's appropriate for it in the immutable world which is less familiar to the average programmer but has its own sort of conceptual clarity that is very attractive to some people and Clojure is doing both and how do we how do we navigate between those waters and where does this identity concept which I don't understand I want to understand the identity concept first of all and then I want to understand how it fits into these these mixed waters that I also understand so I like the Blue Nile flowing into the White Nile area.”

---

[30:17](https://www.youtube.com/watch?v=wASCH_gPnDw&t=1817s) Defining Persistent Data Structures and Complexity Guarantees

Before tackling identity, Hickey insists on laying down the exact computer science definition of a *persistent data structure*. He notes that simply telling programmers that something cannot be changed causes immediate skepticism: “I think it's one thing to tell somebody something is immutable and they can't change it in which case they say that doesn't seem very useful to hang of course and and to give them a recipe for working with immutable things that's both efficient and convenient.”

Hickey clarifies that "persistent" here has nothing to do with saving data to a disk or relational database. In functional programming, persistence means that when a new version of a collection is produced: 1. **Accessibility:** The previous version remains fully accessible and completely unchanged. 2. **Complexity Guarantees:** Producing the new version adheres strictly to the algorithmic complexity guarantees of that collection type. If inserting into a vector is specified as constant time, generating the new vector must run in constant time. 3. **No Full Copies:** Because creating a new collection must meet constant-time complexity, the implementation cannot simply duplicate the entire structure, which would be an $O(N)$ linear operation; it must rely on structural sharing. 4. **No Version Decay:** Both old and new versions retain their runtime complexity guarantees indefinitely.

Hickey emphasizes that Clojure rejects implementation shortcuts that compromise older references: “there's no cheating going on in Clojure data structures they're really persistent and so they meet the complexity guarantees even old versions.” He notes that previous historical attempts at persistent structures often cheated by optimizing only for the latest version while allowing access to older snapshots to degrade in performance. Clojure guarantees that all versions—past and present—remain equally fast and fully functional.

[32:13](https://www.youtube.com/watch?v=wASCH_gPnDw&t=1933s) Persistent Vectors and Hash Array Mapped Tries

Rich Hickey explains how to build a persistent vector where appending an element takes near constant time. It cannot be represented as a large monolithic array; it must be implemented as a tree. Aside from the classic singly linked list, almost all Clojure data structures are trees. While Clojure provides red-black trees for sorted collections, general hash maps and vectors demand constant or near-constant time performance. To achieve this, the underlying trees are extremely shallow with very high branching factors.

```mermaid flowchart TD accTitle: Structural Sharing in a 32-Way Persistent Tree accDescr: Diagram showing how an appended node copies only the spine path while sharing unchanged leaf arrays.

subgraph OldTree ["Original Vector Root"] R1["Old Root Node"] L1["Leaf Array (Indices 0..31)"] L2["Leaf Array (Indices 32..63)"] R1 --> L1 R1 --> L2 end

subgraph NewTree ["New Vector Root (Appended)"] R2["New Root Node"] L3["New Copied Leaf Array (Appended)"] R2 --> L3 end

R2 -.->|"structural sharing"| L1 ```

Brian Beckman immediately recognizes the concept, asking whether it functions like a trie, which Hickey confirms. Hickey cites Phil Bagwell's work on Hash Array Mapped Tries (HAMTs). Bagwell originally designed these non-persistently for mutable hash tables; Hickey adapted the approach to design fast, immutable versions. For Clojure's vectors, elements reside in arrays at the leaf nodes, indexed directly along the path down the tree. With a branching factor of 32, the trees remain exceptionally shallow.

When appending an element to a vector: - The last leaf array (bounded at a maximum size of 32 elements) is copied, which takes constant time. - The path from that leaf back up to the root is copied, creating a new root and a new right spine. - All other branches of the tree are retained through structural sharing without modification.

As Rich Hickey summarizes the efficiency: “If this thing has a million items in it, you're going to change, you know, three nodes on the right side to add something to the end, and that makes that efficient—very near constant.”

- Persistent data structures rely on structural sharing, making the creation of new modified versions inexpensive without changing existing data in place. - Programming shifts from in-place mutation to calling a transformation function that returns an updated reference while leaving original structures intact. - Immutability eliminates defensive cloning and synchronization concerns across concurrent threads. - Backtracking algorithms and undo operations become straightforward because applications can simply retain references to previous root nodes.

---

[37:15](https://www.youtube.com/watch?v=wASCH_gPnDw&t=2235s) The Flaw of Object Orientation: Conflating Identity and Value

The conversation moves to Clojure's rejection of traditional object-oriented programming. Hickey recounts that Clojure arose directly out of his struggles building large, multithreaded systems with OOP languages, where he found himself writing in an aggressively functional style inside C# purely as a defensive measure. In analyzing what makes object orientation break down in concurrent environments, Hickey argues that the paradigm fundamentally conflates identity with value.

In mainstream OOP, an object reference is merely an alias for a mutable block of memory. It couples the conceptual entity a developer wants to track over time with whatever mutable values happen to sit inside that memory chunk at a given instant. Brian Beckman points out that the core challenge of concurrent programming is coordinating access timing across threads. When an object is mutated, it inevitably risks corrupting or invalidating the view of any other observer examining that same memory address.

Hickey explains that identity is not the memory address, nor is it merely a string name like "Fred" or "Brian"—names can change while identity persists. Instead, an identity is a stable label assigned to a timeline of continuous states that an entity occupies. In Hickey's words: “Object orientation conflates identity and value, right? They're bound together. If you have some object, you have something you'd like to reference over time, but its value also could change over time. There's no distinction between, you know, Fred, the thing I want to remember, and the value of Fred at any particular instance of time.” Traditional OOP assigns a label directly to a mutable block of memory rather than treating the identity as a reference to a succession of timeless values across time.

---

[40:04](https://www.youtube.com/watch?v=wASCH_gPnDw&t=2404s) Separating Values from Identities with Reference Types

To fix the structural defects of mutable objects, Clojure cleanly decouples value types from reference types. Values are immutable snapshots: they have no internal behavior over time and cannot change under any circumstances, making them simple to reason about. Reference types, by contrast, act as explicit containers or "boxes" that designate identity.

```mermaid flowchart LR accTitle: Decoupling Identity and Value in Clojure accDescr: Shows an Identity Box atomically switching its pointer from one immutable value to another while a reading thread safely retains the older value snapshot.

subgraph Identity ["Identity (Time Management Box)"] Ref["Fred (Reference Type)"] end

subgraph Timeline ["Immutable Value Snapshots"] V1["Fred Value Snapshot 1<br>(Original DOB)"] V2["Fred Value Snapshot 2<br>(Corrected DOB)"] end

Ref -->|"Atomically updated to"| V2 ThreadReader["Reading Thread"] -.->|"Holds stable reference to"| V1 ```

Hickey defines these identity boxes as time management units: “And all they are are boxes, right? And the boxes always refer to values. Okay, so what is this box? This box is a time management unit.” Clojure provides multiple reference types offering distinct concurrency semantics to govern how the timeline of values progresses.

To modify an entity: - An update does not mutate memory in place. If an error is discovered in Fred's recorded birthdate, a transformation function takes the original immutable value snapshot (`Fred Value 1`) and returns an updated immutable snapshot (`Fred Value 2`). - Structural sharing ensures that generating the new snapshot is fast and does not disturb the original. - The reference box is atomically redirected to point to the new value snapshot.

Any thread or computation currently reading `Fred Value 1` remains entirely unaffected by the identity pointer update. A thread can retain its snapshot indefinitely with absolute consistency. As Beckman and Hickey jokingly observe, even if Fred's date of birth is updated because he previously lied about his age, any observer holding the earlier snapshot is completely isolated and cannot be compromised by the concurrent change.

[42:47](https://www.youtube.com/watch?v=wASCH_gPnDw&t=2567s) References as Time-Management Constructs

Rich Hickey clarifies the crucial separation of concerns in Clojure's state model: performing calculations on immutable values is completely independent of the reference that holds them. Whether calculating a new version of an entity involves a minor attribute tweak like changing an occupation or an elaborate, expensive data transformation, the calculation simply takes one immutable value and returns another. Crucially, readers can never be impacted by this process, nor do they block concurrent updates; reading the current value of an identity like "Fred" does not impede another thread from constructing the next version of Fred. All the reference needs to do is manage time properly. Brian Beckman asks whether it is fair to describe a reference as just a special kind of variable. Hickey agrees, noting that "variable" is an empty, semantically vague concept in conventional computing.

Hickey explains that a genuine time construct must ensure an atomic succession of immutable values. A reference will only ever hold a reference to a value, the succession from one value to the next must happen atomically, and the current value must remain visible to readers—which Clojure provides via the dereferencing syntax `@Fred`. Hickey observes that this design draws inspiration from Standard ML's `ref` type (which was essentially ML's only mutable construct besides arrays), but even ML's `ref` lacked clear concurrency semantics for time. When multiple threads concurrently attempt to modify state, there must be a defined mechanism to manage time. One of the simplest models is compare-and-swap (CAS), an uncoordinated, single-timeline management system ensuring that whoever generates the next state does so as a mathematical function of the immediately preceding state. Beckman points out that in the Windows world, this matches the `InterlockedCompareExchange` primitive (often related to underlying hardware test-and-set instructions). With CAS, reads observe a consistent snapshot at a specific point in time without holding locks, even if the value changes an instant later.

```mermaid flowchart LR accTitle: Clojure Value Succession Model accDescr: An atomic reference points to successive immutable values over time without blocking readers. subgraph Succession["Succession of Immutable Values"] V0["Value t0: {Fred, Job: Dev}"] V1["Value t1: {Fred, Job: Architect}"] V0 -->|"f(v0)"| V1 end subgraph Reference["Time Construct"] R["Reference: @Fred"] --> V0 R -.->|"atomic update"| V1 end Reader["Concurrent Readers"] -->|"deref (@)"| V0 ```

- When evaluating what a reference means, Rich Hickey emphasizes: “It is a special kind of variable because the because variable is another one of these empty concepts yeah right with no semantics right so what does it mean to be a time construct right right it means you have to ensure a succession of values all right so the first key is there's only going to be ever have in it a reference to a value right right that succession has to be atomic right the value inside has to be visible”.

[46:02](https://www.youtube.com/watch?v=wASCH_gPnDw&t=2762s) Atoms, Refs, and Coordinated Transactions (`dosync`)

To wrap compare-and-swap semantics cleanly, Clojure introduces the `Atom`. An atom is an uncoordinated reference type that handles low-level spin-swapping automatically. To mutate an atom, you call `(swap! my-atom some-function args...)`. The atom applies the provided function to its current state along with any arguments, producing the next state. If another thread updates the atom in the interim, the CAS fails and the atom automatically retries the operation with the newer value. However, because atoms are isolated and uncoordinated, they cannot safely manage operations across multiple independent identities simultaneously—such as transferring funds between two bank accounts.

For multi-reference coordination, Clojure provides transactional references (`Ref`s), managed by Software Transactional Memory (STM). Modifications to refs occur using `(alter my-ref some-function args...)`. If a developer invokes `alter` outside of a transaction, Clojure throws an error. Coordinated updates across multiple refs must be wrapped inside a `(dosync...)` block. If two transactions conflict or cannot obtain a synchronized snapshot of the involved refs, the STM system aborts and retries the transaction automatically. When `dosync` exits cleanly without exceptions, the entire atomic update has succeeded; if an unhandled application exception occurs, the transaction aborts and cleanly rolls back, leaving no half-applied mutations.

- Rich Hickey describes the mechanism of an atom's update: “Clojure has a has a construct that wraps this whole thing it's called an atom it's the simplest time management construct it uses CAS internally right it does the spin swap automatically so that every value is a guaranteed function of the immediate preceding value”. - On coordinating multiple refs, Rich Hickey explains: “So we say do sync around this that's night and now that succeeds either all succeeds together or not or an unusual track ACID kind of things except it's not”.

[49:15](https://www.youtube.com/watch?v=wASCH_gPnDw&t=2955s) Asynchronous Agents, Immutability Guarantees, and Beckman's Parallels

Beyond atoms and refs, Hickey introduces Clojure's asynchronous reference type: the `Agent`. Like atoms, agents manage a succession of values, but dispatching a state transformation via an agent returns immediately, scheduling the calculation to run on a thread pool in the background. While notification mechanisms exist, the agent philosophy mirrors Erlang's message-passing style, encouraging developers to embrace "fire-and-forget" asynchronous flows rather than trying to freeze the universe with heavyweight transactions. Additionally, all Clojure reference types provide uniform support for watches (change notifications) and user-defined validator functions that can reject illegal state transitions before they commit.

Beckman asks whether these reference constructs can or should be paired with mutable data structures. Hickey firmly rejects the idea: “You don't, you don't use them.” The entire reference architecture relies completely on persistent data structures. Because transactions and atomic swaps may execute speculatively and retry multiple times, mutating data in place would corrupt state and disrupt readers. Beckman concedes the point and muses about a hypothetical, ultra-strict "Clojure--" that enforces pure persistence without mutable Java interop. Beckman concludes by highlighting two deep conceptual touch points: 1. Microsoft's Reactive Extensions (`IObservable`/`IObserver` in.NET), which represent values changing over time via elaborate continuations/callbacks. 2. The Time Warp operating system—an architecture rooted in speculative execution, virtual time coordinates, and rollback mechanisms triggered by "time faults," creating an intriguing parallel to Clojure's retry-driven STM.

- Rich Hickey underscores why Clojure references require persistent data structures: “So they go together right there's no taking this cool reference identity thing and and making it work with something mutable”. - Brian Beckman connects the model to speculative execution architectures: “And the other is Time Warp which is one of my old hobby horses which was a whole operating system built on speculative execution okay so the sequences of operations specified with respect to virtual time with a mapping to real time that affected rollback and commitment uh-huh with a deep analogy to virtual memory”.

[53:23](https://www.youtube.com/watch?v=wASCH_gPnDw&t=3203s) Wrapping Up: Clojure Across .NET and the JVM

Reflecting on the depth of the discussion, Brian Beckman notes that exploring the remaining architectural layers—such as putting systems on top of Time Warp or vice versa—would easily require several more hours of conversation. Despite running out of time, Beckman expresses immense enthusiasm for what Rich Hickey has demonstrated and built, emphasizing that he completely grasps the model and its value. He praises Clojure as a tremendous achievement, observing that while he naturally approaches software from the perspective of a.NET partisan, Clojure's availability across both the.NET and Java Virtual Machine ecosystems makes it an exceptionally compelling tool.

Beckman's excitement is palpable as he concludes the interview, noting that Clojure is not merely an interesting theoretical exercise but something practical and ready for immediate adoption. As Beckman enthusiastically puts it, “It's the kind of thing that I will I would definitely use this this afternoon.” Hickey warmly accepts the praise with a quick word of thanks, bringing their whiteboard session to a close.

- Brian Beckman notes that fully connecting these concepts to Time Warp or deeper runtime mechanics would demand significantly more time than a single session allows. - Clojure's dual presence on both the CLR (.NET) and the JVM makes it a versatile, cross-platform functional weapon in a modern developer's toolkit. - The interview wraps up with Beckman eager to put Clojure into direct practice right away.

## Notable quotes > [37:15](https://www.youtube.com/watch?v=wASCH_gPnDw&t=2235s) Object orientation conflates identity and value, right? They're bound together. If you have some object, you have something you'd like to reference over time, but its value also could change over time. — Rich Hickey

> [10:48](https://www.youtube.com/watch?v=wASCH_gPnDw&t=648s) Clojure is definitely coming from the Lisp tradition, walking towards a more functional style of programming; it's definitely not from the Haskell-Miranda tradition. — Brian Beckman

> [20:33](https://www.youtube.com/watch?v=wASCH_gPnDw&t=1233s) A symbol might stand for something else in use in a program; a keyword only ever designates itself. — Rich Hickey

> [10:48](https://www.youtube.com/watch?v=wASCH_gPnDw&t=648s) It's easy to write a functional program or write portions of your program functionally, and there's no help whatsoever in terms of knowing when you've done that correctly or not. — Rich Hickey

## Mentioned Rich Hickey, Brian Beckman, Clojure, Java Virtual Machine (JVM), Common Language Runtime (CLR), .NET, Java, C#, C++, Lisp, Scheme, Common Lisp, Haskell, Miranda, ML, Standard ML, SASL, KRC, Erlang, Eugene Kohlbecker, Philip Wadler, Phil Bagwell, Incanter, Macsyma, R, Reactive Extensions (Rx), Time Warp operating system

## Video description Cross posted from http://channel9.msdn.com/Shows/Going+Deep/Expert-to-Expert-Rich-Hickey-and-Brian-Beckman-Inside-Clojure.

Clojure is a dynamic programming language created by Rich Hickey that targets both the Java Virtual Machine and the CLR. It is designed to be a general-purpose language, combining the approachability and interactive development of a scripting language with an efficient and robust infrastructure for multithreaded programming. Clojure is a compiled language - it compiles directly to JVM bytecode, yet remains completely dynamic. Every feature supported by Clojure is supported at runtime. Clojure provides easy access to the Java frameworks, with optional type hints and type inference, to ensure that calls to Java can avoid reflection.

Clojure is a dialect of Lisp, and shares with Lisp the code-as-data philosophy and a powerful macro system. Clojure is predominantly a functional programming language, and features a rich set of immutable, persistent data structures. When mutable state is needed, Clojure offers a software transactional memory system and reactive Agent system that ensure clean, correct, multithreaded designs.

Astrophysicist and Software Architect Brian Beckman interviews Rich Hickey to dig into the details of this very interesting language. If you don't know much about Clojure and the general problems it aims to solve, well, watch and listen carefully to this great conversation with plenty of whiteboarding and outstanding questions. Expert to Expert simply rocks! Thank you for spending time with us, Rich! Clojure is great!

[Source](https://www.youtube.com/watch?v=wASCH_gPnDw)

中文简报

专家对话:Rich Hickey与Brian Beckman——深入剖析Clojure Rich Hickey概述了Clojure背后的核心哲学,详细阐述了其四大支柱——函数式编程、Lisp同音异形性(代码即数据)、宿主平台互操作性以及一等公民级别的并发支持——如何解决传统多线程面向对象编程的严重缺陷。Hickey在白板前演示了Clojure的持久化集合如何利用浅树结构和结构共享,在无需破坏性突变的情况下保证近乎恒定时间的更新。讨论的高潮聚焦于Clojure的状态模型,该模型将永恒的不可变值与管理时间的引用结构(如atom、事务性ref(`dosync`)和异步agent)进行了严格解耦。Brian Beckman将这些机制与更广泛的系统概念联系起来,包括硬件比较并交换(CAS)、Reactive Extensions以及Time Warp操作系统的推测执行模型。 Rich Hickey围绕四大基石设计了Clojure:以默认不可变性为主的函数式编程、具备语法同形性的Lisp基础、专为控制多线程状态而设计的并发模型,以及对JVM和CLR的一流宿主支持。 (1:17) Clojure摒弃了面向对象的状态管理,认为OOP混淆了同一性(identity)与可变内存地址;相反,Clojure将同一性建模为一个显式容器,用于管理永恒、不可变值的时间线。 (37:15) Clojure中的持久化数据结构采用浅树实现,例如32路分支的哈希数组映射树(HAMT),通过结构共享实现近乎恒定时间的追加和更新,且不会降低旧快照的访问性能。 (32:13) Atom(无协调的比较并交换)、Ref(由dosync协调的软件事务内存)和Agent(异步后台执行)等状态管理结构严格依赖持久化数据结构,以允许推测性重试而不损坏数据。 (46:02)

Source quotes

[37:15] Rich Hickey: Object orientation conflates identity and value, right? They're bound together. If you have some object, you have something you'd like to reference over time, but its value also could change over time. [10:48] Brian Beckman: Clojure is definitely coming from the Lisp tradition, walking towards a more functional style of programming; it's definitely not from the Haskell-Miranda tradition. [20:33] Rich Hickey: A symbol might stand for something else in use in a program; a keyword only ever designates itself. [10:48] Rich Hickey: It's easy to write a functional program or write portions of your program functionally, and there's no help whatsoever in terms of knowing when you've done that correctly or not.

Original transcript

[0:00] hello hello hello my I'm Brian Beckman [0:04] and I'm here with rich Hickey today rich [0:06] is the creator of closure and rich your [0:10] unaffiliated or independent where I [0:12] learned about it yes tell you what some [0:15] of your background in in computing and [0:18] language design and that kind of thing [0:19] I've been working as a professional [0:21] programmer for a couple of decades [0:23] specializing mostly in broadcast [0:26] automation and scheduling systems also [0:29] the National exit poll and various [0:32] machine listening kinds of problems oh [0:36] very good so hardcore real-time yeah a [0:39] lot of time route but yeah okay good [0:43] well I'm I'm just a country physicist [0:46] who dabbles in programming but I've been [0:48] using Lisp for I don't know I think I [0:50] discovered scheme around 1978 or 79 and [0:54] fell in love with it and have deployed [0:56] it in more strange corners of computers [1:00] than you can possibly imagine Mya [1:02] I guess my crowning glory would be [1:04] sticking it in the kernel and using it [1:06] to do page table replacement policies [1:08] page replaces so I have a little bit of [1:14] exposure to Lisp and scheme and let me [1:17] ask you well first why don't you just [1:19] tell me what closure is Before we jump [1:21] into into into that a closure is a [1:25] dynamic language it's a dialect of Lisp [1:29] and one of the interesting things about [1:31] Lisp is that it supports dialects yes [1:33] just unusual compared to most other [1:35] languages and it's really founded on [1:39] four four ideas one is that it be a [1:43] functional language so it's primarily [1:47] functional it's not pure round but it [1:49] emphasizes immutability the other is [1:52] that it be a lisp and there are certain [1:54] things about lisps in particular this [1:56] syntactic extension capabilities of [1:58] Lists okay that make them still unique [2:01] even though most the other ideas from [2:02] Lisp have been adopted by other [2:03] languages right it has an emphasis on [2:07] concurrency so a lot of the problems [2:10] I've worked in all these broadcast [2:11] automation systems were multi-threaded [2:12] and they were done [2:13] in C++ and Java and c-sharp inthere you [2:16] know basically that's too hard for me [2:18] yes so I was trying to make that easier [2:20] and the fourth attribute of closure is [2:23] that it's designed to be hosted so while [2:26] it's implemented on top of the JVM and [2:28] the support to the CLR the fact that [2:32] it's on top of those is not just an [2:33] implementation detail for instance there [2:35] are ports of scheme and common list to [2:37] the JVM but that it's designed to be [2:41] hosted it's designed to let you touch [2:43] the VM or the environment on which it's [2:46] running so that you can leverage all the [2:48] libraries and everything else okay so [2:50] it's specifically got support for [2:52] calling Java objects and net objects [2:56] marvelous for interface which well but [2:58] it's a lot less foreign than it is in in [3:01] those languages so it's marvelous [3:03] tremely transparent now does this mean [3:05] that in the case of the CLR that you [3:07] have access to all the reflection [3:09] machinery and can that it sort of does [3:11] that under the hood under the hood ok so [3:13] but you can suppose to a lisp II Lisp [3:17] style model for CLR objects or uh you [3:21] know as well you you basically there's [3:23] syntax for calling methods and calling [3:26] new and there's a dot special operator [3:29] in closure which is designed to provide [3:31] a path to the host then there are macros [3:33] that make that look quite Lisp like okay [3:36] good so we list call would be open paren [3:39] you know foo XYZ yep and a.net call on [3:43] some object X would be dot foo XYZ it's [3:48] still prefix notation it's absolutely [3:49] still prefix notation and so yeah you [3:52] have access to objects and their methods [3:55] and fields why did you pick Lisp as a [3:57] way to to launch into something like [4:00] this because you could imagine a dynamic [4:01] language with with the support for [4:04] that's designed to be hosted with [4:06] support for calling into the CLR or the [4:11] Jade and that isn't Lisp right [4:13] why'd you pick list well Lisp is still [4:16] uniquely powerful in that syntactic [4:19] extensibility area a good so for [4:23] instance if you have a language that [4:24] you're using and you wish it had [4:26] some construct that it doesn't like for [4:30] each or something like that let's just [4:31] imagine language you didn't have it [4:32] right you would have to wait for the [4:34] authors of the language to edit yes and [4:37] in Lisp you don't have to wait enlist [4:40] you can write any construct like that [4:41] yourself right so that kind of power is [4:44] extremely useful it ends up that in Lisp [4:48] you you basically can write a custom [4:50] language for every problem that you [4:52] solve quite easily and a lot of people [4:55] in our world call this meta programming [4:56] right and I remember when Eugene CO [4:59] becquer created hygienic macros for [5:02] scheme right gosh knows how many years [5:04] ago now that really opened it up right [5:07] so all the macro systems are you know [5:10] the way that syntactic extension is [5:11] implemented in Lisp right right they're [5:13] called macros and the point of the [5:15] problems with the word macros is people [5:17] associate it with C macros yeah which [5:19] are sort of a very crude string [5:21] substitution system but real [5:23] programmatic and hygienic macros are [5:26] programs that transform programs into [5:29] other programs [5:30] right and it's this is enabled by the [5:31] unique feature of Lisp that some people [5:35] use the word Auto iconic right or homo [5:37] icon or homo iconic it just means that [5:39] the representation for the program is [5:41] the same as a representation for data [5:42] correct and you can transform the [5:46] program it's a program itself exactly [5:48] the way you would transform data [5:49] directly with the same libraries right [5:51] the same tools right correct and and [5:53] other languages that may be powerful in [5:55] the functional world or the real-time [5:57] world such as ml languages like ml and [6:01] Miranda and Haskell they explicitly [6:03] avoid auto iconic item based on a [6:07] probably possibly on a paper by Wadler [6:09] long ago that said calculating is better [6:11] than scheming or something like but it [6:14] does it does mean that meta programming [6:16] that is now a separate topic in white [6:18] languages whereas in in Lisp you can use [6:22] the language itself to modify itself [6:24] it's really wonder that's right and [6:25] there's really this two aspects one is [6:27] that code is data you know we call it [6:30] code is data the other is that the [6:33] evaluation strategy incorporates a step [6:36] that says at this point I will call your [6:39] program to [6:40] help me evaluate right so they're so [6:42] macros are a hook yeah that's built into [6:44] compilation or evaluation that allows [6:48] your program that can do it everybody [6:50] can write a program that transforms [6:52] another program right transform source [6:53] code the trick is that that's part of [6:56] the evaluation strategy of the language [6:57] saying I've done this I've made a [7:00] program that transforms a program and [7:01] when you see this please call it yes and [7:03] the compiler will do that means that the [7:05] evaluation is open yes to your programs [7:09] extension by your programs [7:11] now does closure have more than one [7:14] evaluation strategy such as lazy versus [7:16] strict or does it to have just one [7:19] closure is strict it is drum but closure [7:22] has sequences which are an abstraction [7:25] that can be implemented lazily right and [7:27] all of the closure core functions like [7:30] map and filter are implemented lazily [7:34] okay good so so it's sort of a hybrid [7:36] that way right it's not lazy evaluation [7:38] in the haskell sense on the other hand [7:41] the core algorithms for sequence [7:44] manipulation have been written in a lazy [7:45] style and that's the main the main [7:47] practical use of lace exactly for [7:49] infinite data structure writing or [7:51] streams streams right yes but it does [7:54] mean that if your evaluation strategy is [7:57] applicative order that means [7:59] specifically that you evaluate function [8:01] arguments before you pass them to the [8:03] phone correct ok strict good super now [8:06] in what sense is closure functional and [8:10] in what sense is you said at the [8:12] beginning that it's functional but not [8:13] pure can you tell me exactly what you [8:15] mean yes I think you know functional has [8:17] been applied you know for a whole [8:19] spectrum of things I think there's the [8:21] Haskell meaning of functional which is [8:22] sort of the pure notion of function [8:24] applied to absolutely everything in the [8:26] language right and then there's a notion [8:29] of functional which you know lisps were [8:31] claimed to be functional early on right [8:33] only for the simple reason that [8:35] functions were first class objects their [8:39] value passed them their values right [8:40] that you can pass around and so those [8:43] are the two ends somewhere in the middle [8:46] I think is is a style of programming [8:50] that emphasizes programming with pure [8:52] functions [8:53] functions that are free of side effects [8:55] that operate only upon their arguments [8:57] which are values yes and that only [8:59] produce values as results right so [9:02] that's a pure function so closure [9:04] emphasizes the style of programming [9:06] using pure functions where lisps [9:08] traditionally have not you could do it [9:10] but it was a choice the key thing in [9:14] forcing you to make that choice is to [9:17] have the core data structures be [9:19] immutable right so they now can be [9:21] treated as values in a way that mutable [9:23] things cannot so closure emphasizes [9:26] functional programming by providing a [9:29] set of core data structures including [9:31] those ones that you use to represent [9:32] code yeah they're all immutable in [9:34] closure and they're also persistent [9:37] which is a special characteristic we [9:39] could talk about that's interest um and [9:40] because of that it forces a functional [9:44] style of programming because you cannot [9:45] change them so you have to write a [9:47] function that produces a new value as a [9:50] result but there's no enforcement if you [9:53] write a function and you do i/o then [9:55] that's not a pure function anymore and I [9:57] okay I don't preclude it and there's no [10:00] type system to preclude it it's write [10:02] dynamically typed language which just [10:04] respects the programmers right to do [10:07] whatever they please [10:07] yes yes well that's the the last [10:12] dynamically typed language that I can [10:15] remember in the in the sort of Haskell [10:17] genealogy is probably SAS all I think [10:20] maybe KRC I don't remember where the KRC [10:24] had types by the time I got to Miranda [10:26] they had types and we were beginning to [10:27] do type inference which makes certain [10:29] kinds of pure functional restrictions [10:32] more palatable because now you can so [10:36] for example I bring out the dreaded M [10:37] word the monad word without type classes [10:41] and the ability to overload the the bind [10:43] operator monads become an incredibly [10:45] oppressive way they're they're [10:48] algebraically clean right you get you [10:51] get the you know the guarantees of pure [10:53] functionality but you've got a halt [10:55] around these extra arguments and just [10:57] pull them out and stick them [10:58] implementing oh yeah right or deal with [11:00] some restricted you know view of monads [11:02] where you know a monad can be active at [11:04] a time but they don't really compose as [11:06] well [11:07] right and then Haskell has some pitfalls [11:09] in there [11:10] if you if you're putting two monads [11:12] together there are places where it can't [11:14] is it doesn't know from the program text [11:17] which one you're trying to party in [11:19] right and I think it's the mono morphism [11:22] restrictions right they know so closure [11:24] is definitely coming from the Lisp [11:25] tradition yes walking you know towards a [11:29] more functional style program it's [11:32] definitely not from the Haskell Miranda [11:34] tradition right now would you say that [11:36] it is it's easy to write a pure [11:39] functional program if you want to and is [11:42] it easy to know when you are leaving the [11:44] world of pure functions in closure yes [11:49] it's easy to write a functional program [11:51] or write portions of your program [11:52] functionally right and there's no help [11:56] whatsoever in terms of knowing when [11:58] you've done that correctly or not [12:00] right right so being dynamically typed [12:02] there's no types enforcing the lack of [12:04] IO or the presence of IO or or anything [12:07] about that right you can freely use Java [12:10] mutable things and dotnet immutable [12:13] things and if you do they have the same [12:15] you know issues they always did okay and [12:18] it's important thing for people to be [12:20] able to do that so no it does not try to [12:24] track through a type system when things [12:26] are purely functional when they're not [12:27] right and not even through a dynamic [12:29] type system there there would be some [12:32] ways that you might be able to track [12:33] that dynamically but unfortunately doing [12:35] so would be would require sort of [12:38] flagging most of the dotnet yeah and [12:40] Java frameworks are dangerous by default [12:43] yes and reducing the utility I mean [12:45] reaching those frameworks and leveraging [12:47] those libraries is a big plus for [12:49] closure plus yeah and if you're in a [12:52] pure functional world you have to wrap [12:53] them in monads or something like that [12:55] yes yeah right so in practice what's [12:58] happened the the the thing that closure [13:02] does is it it gives you these really [13:05] easy to use immutable data structures as [13:08] the default thing so unlike lists [13:11] traditional lisps we're really the only [13:14] data structure you had in the syntax was [13:16] a list yeah most lisps or schemes also [13:19] had hash tag [13:20] and things like that but they they [13:22] didn't have recursive definitions like [13:24] lists to write write a list you know you [13:27] add to the front and you can share the [13:28] tail you know if you're not mutating it [13:30] but hash tables for instance in common [13:33] lisp or just like java ones you just [13:35] bang on them they change in place yep [13:37] so closure extends the homo iconicity of [13:41] lisp [13:41] to the other data types so there are [13:45] literal representations for maps and [13:48] sets and vectors oh very nice all of [13:50] which are immutable and all of which are [13:53] persistent Millard's they do have [13:55] recursive definitions right right you [13:56] are based on tree structures and they [13:58] support efficient making it efficiently [14:01] making new changed version right so so [14:05] you have syntax for all of that which is [14:06] nice a lot more like you know Python or [14:08] Ruby or you have syntax for for maps and [14:11] well they call them have when you say a [14:13] map do you mean our name value pair I [14:14] mean yeah like an association yeah right [14:17] so so this so there are literals for all [14:19] those things when you use them you end [14:21] up with something immutable right so [14:23] immediately you're handed all these [14:25] immutable things then you're handed this [14:27] library that knows how to manipulate [14:28] them and so using them is idiomatic so [14:31] automatically you start off and you're [14:33] doing this and and for most of the [14:37] calculation part of your program you can [14:40] happily stay there right all right and [14:42] then you have okay now I need to put [14:44] something on the screen I need to write [14:46] something to the disk I need to stick it [14:48] in the database those are all i/o things [14:50] right and you know if you look at the [14:52] libraries of Java and.net a substantial [14:55] portion of those libraries the value in [14:58] those libraries is i/o yes it's sealed [15:00] getting stuff in and out of XML getting [15:02] stuff in and out of these data formats [15:04] you know talking to a web server right [15:05] Oh [15:06] io plumbing so I mean that's all great [15:08] just you know use it what about numerix [15:11] do you have any special numeric [15:12] capabilities in closure it has the [15:14] traditional Lisp things which is the [15:16] older math is good you know big dumb big [15:19] gnomes [15:19] you know integer arithmetic doesn't wrap [15:23] around the integers will just continue [15:25] to grow there's a there's no real [15:28] rationals so you can say 22 over 7 and [15:31] that's as in turn at the point so good [15:33] exact trash it stays [15:34] right it stays exact so it's not the [15:36] same kind of numeric tower as scheme [15:38] scheme had a much stronger sense of [15:40] exact and inexact right then Common Lisp [15:42] it's more in the Common Lisp tradition [15:44] but basically by default you won't lose [15:47] any information but about any aesthetic [15:50] capability such as simplifying a [15:53] symbolic expression that's not built in [15:55] but like Lisp you could you know so the [15:57] maximum was written in Lisp not so hard [15:59] to add right right that's one of the [16:01] lovely do that Lisp is right and what's [16:03] it's part of what makes it suitable for [16:06] writing compilers and program [16:07] transformations it also makes it [16:09] suitable for doing symbolic calculations [16:11] in algebra and calculus right are you're [16:13] just doing symbolic calculations right [16:15] so there's already a project called in [16:18] Cantor for closure which is like an an [16:22] are like statistics language right and [16:25] so writing those kinds of languages in a [16:27] lisp is really easy and a lot of fun and [16:29] and because there's a transparent [16:32] extensibility to the underlying language [16:34] there's no bridging necessary you know [16:37] you can intermix in Cantor are like [16:40] statistics code and closure code is it [16:42] can't foresee int o are in Cantor I n CA [16:45] NT er er okay it could be a pun on the [16:49] the mathematician can no no I don't [16:51] think it is but what do you feel [16:55] comfortable writing some some stuff on [16:57] the board we've been talking abstract [16:59] because you and I both have pictures in [17:01] our minds of what this stuff might look [17:03] like but for those who don't know Lisp [17:05] let's write a few expression okay how [17:07] we've got some markers here all right [17:09] let's um look here we go yes so Lisp is [17:14] let's see what some stuff looks like a [17:16] first a list in a you know an ordinary [17:19] calculation and then I want to see what [17:21] maps and sets and all these other groovy [17:23] things look like okay so if we say plus [17:27] 1 2 we'll get three good there are there [17:31] no commas there's no semicolons or [17:34] anything like that it's writing prefix [17:35] notation yes so the operator comes first [17:39] and then any arguments follow great by [17:42] default the interpretation of things and [17:45] parens is a list yes so I've already [17:47] talked a little bit of [17:48] about evaluation but essentially this is [17:50] a list of three things right this is a [17:54] vector of three numbers okay okay this [17:58] is a map it's a curly this is a map with [18:06] two entries good this is a keyword and a [18:08] number a keyword and a number okay [18:10] you could write maps from strings to [18:14] vectors or any a vector to to their [18:20] arbitrarily nest able arbitrarily nests [18:22] table very nice like that okay so list [18:26] vector map there's also sets which [18:31] precede the curly with the sharp a B C [18:36] okay and now the prison these are the [18:38] usual set semantics where they're if you [18:41] put duplicates in there they get out and [18:43] there's no order there's no order now [18:45] how does that list differ from a vector [18:47] a list differ from differs from a vector [18:49] in the complexity characteristics of [18:53] different operations okay so a list is [18:55] the very traditional lisp list it's a [18:59] singly linked list where it's efficient [19:01] to add things to the front yes in an [19:03] operation called cons is construct yes [19:06] that's an old school so if we were to [19:08] cons onto the beginning of a list we [19:10] would get a list with that new thing on [19:12] the front right but looking up you know [19:16] looking up the 14th thing in a list is a [19:18] linear operation because you have to [19:20] walk the chain right so the complexity [19:23] characteristics of insertion at the [19:24] front is bigger one looking up in the [19:27] middle is Big O n it's linear time [19:29] adding at the end is linear time it's [19:31] terrible yeah you know it's just not [19:33] good for that right good for adding at [19:35] the front and other most of everything [19:37] else is linear but that also means it's [19:39] good for recursive definitions where you [19:41] say do something to the first and then [19:43] do something to the right but all of [19:45] these data structures are immutable and [19:49] persistent so vector has efficient [19:52] adding to the end in addition it has [19:55] efficient lookup by index what's the [19:57] syntax for looking up in a vector by the [19:59] way it [20:01] it's and well vectors are vectors and [20:05] maps are functions yes right right so [20:10] you just call it on its index beautiful [20:14] love it or you could say get or anything [20:19] there's a bunch of other shirts and a [20:20] little help there's a bunch of that's [20:22] the primary idea there it's just another [20:25] it's just a friend from an index is also [20:27] do this and get one this colon operation [20:33] is a little interest okay so there's a [20:35] couple of other primitives that we're [20:37] not talking about here this is a literal [20:39] for a number yes this is a literal for a [20:42] keyword okay which is essentially like a [20:45] symbol actually I should do these first [20:47] that's a literal for a symbol alright [20:48] which is like a string except you don't [20:50] have to put it in double quotes and it's [20:51] distinct from strings okay so the symbol [20:54] a is different from the string a is [20:56] different from the keyword a and you [20:58] don't have to put tick marks the way you [20:59] used to have to do in in old lisper well [21:02] you would if you asked to evaluate a [21:04] okay right then now the sudden you're [21:06] asking for the interpretation of ax I [21:08] see in which case if you just wanted the [21:10] symbol you could quote it alright good [21:12] so essentially the difference a symbol [21:15] might stand for something else in use in [21:17] a program mm-hm [21:19] a keyword only ever designates itself [21:21] good [21:22] Oh cause the distinction between [21:23] keywords and symbols and strings are [21:25] strings in fact this string is a Java [21:28] string on Java it's a CLR string on the [21:30] CLR nice there's no wrappers as it [21:33] doesn't turn into a list of Caring other [21:35] than that but you could treat it as one [21:37] for instance you you could map a [21:40] function across those string okay we're [21:43] skipping way ahead was all right so so [21:48] there are literals for all the data [21:50] structures all right and then there's a [21:51] and it and what happens is you write a [21:54] program by writing data structures well [21:57] this is very before we go there I want [22:00] to point out this is very things I [22:02] noticed one is this is very nice this is [22:04] is really it's not just name value pairs [22:07] it's an arbitrary discrete map where the [22:10] domain on the left the domain is any of [22:13] the value that is a directive [22:15] and the domain on the right is any of [22:16] the veggies you're allowed down correct [22:18] so that's that's lovely that's right the [22:19] second thing about the prefix notation [22:21] that really is a direct representation [22:24] of abstract syntax trees when you start [22:27] nesting those things right that turns [22:29] into a STS that's right and this is why [22:31] people when XML came out the old-timers [22:34] said this is nothing but Lisp with right [22:36] well noisy Lisp right and the XML er [22:38] says what's this list thing that's that [22:40] right hiding the structure but it's just [22:43] trees right so you write a program [22:46] mostly by writing literal [22:49] representations of lists in your program [22:52] ran some vectors as well are used in the [22:54] syntax in addition if you want to create [22:56] a map or a set you might use a literal [22:59] version of those but the important thing [23:01] about lisps is that the language is not [23:05] the term is not defined in terms of [23:08] syntax defined in terms of characters [23:11] it's a two-phase syntax there is your [23:15] source code right right then there's [23:19] something called a reader and the source [23:24] code is text what the reader outputs is [23:29] data right so the reader will read this [23:33] and will read [23:34] the symbol plus a list of the symbol [23:37] plus the number one and the number two [23:39] right and that's what the reader will [23:41] produce is actually a in-memory data [23:44] structure of that list right and that [23:47] data structure is then passed to the [23:49] evaluator or the compiler which will [23:54] maybe do something about it the critical [23:57] thing is so you can put your code in a [24:00] file and take the text and put it into [24:02] the reader get the data out and that's [24:04] what happens automatically if you say [24:05] load file all this stuff happens and [24:07] right disk gets handed to the compiler [24:09] but you could also have some program [24:10] that you wrote write produce data [24:15] directly yep and hand that and from that [24:18] point on you can't tell the difference [24:20] it cannot tell the difference it never [24:22] needs to be in a file in addition the [24:24] evaluator can go and say hey I've gotten [24:26] this data [24:28] what do you think about it mm-hmm and [24:30] get new data back and that's how macros [24:33] work right right yeah so it's a really [24:36] beautiful thing so we write our code in [24:38] terms of data structure literals [24:39] we don't need JSON or XML because we can [24:42] send all these things they all print and [24:44] they all read correctly right so this [24:48] this isn't text for literals is the [24:50] syntax for the language it happens to be [24:52] that when the evaluator sees a list it [24:54] tries to interpret the first thing in [24:56] the list as a function of some sort [24:57] right and and the rest of it as a call [24:59] to that function right it interprets a [25:02] literal vector as a literal vector and [25:04] maps as maps and writes most of the rest [25:06] of them as data right but treats the [25:08] lists as calls right and that's why you [25:10] need tick mark for something like this [25:12] if you wanted to treat this as data you [25:13] would do this yeah but it's idiomatic [25:15] closure to use a vector instead ok very [25:17] so vectors have a fast look-up by index [25:21] you have fast insertion at the end but [25:23] because they are persistent data [25:24] structures you can write a recursive [25:26] algorithm or say an exploratory [25:28] algorithm that conditionally builds [25:30] vectors and maybe you throw them away [25:32] right right not over backtracking is [25:34] easy because you're not actually [25:36] destroying it so there used to be an [25:39] operation called cons that added to the [25:41] beginning of the list that's been [25:43] abstracted into something called conch [25:45] enclosure which will add to anything in [25:49] the way that's appropriate to that thing [25:50] yes so you can con John to this we'll [25:52] add it at the front if you're cons onto [25:54] that it will add it at the end if you're [25:55] cons on to this [25:56] it'll add the key value pair and cons [25:58] onto the set so you can write generic [26:00] algorithms like given any kind of data [26:06] structure any one of these data [26:07] structures you can say into that data [26:11] structure some collection map a function [26:15] across some other collection right this [26:18] works for every single one of these into [26:21] is defined in terms of cons it's reduced [26:23] with cons so you can build collections [26:25] generically and and you know reduce is [26:29] done recursively right and map very nice [26:32] works on any collection so all the [26:34] collections can be treated as if they [26:36] were lists they aren't structurally [26:38] right but they support a sequence [26:40] interface yeah [26:41] so that gets the next part of closure is [26:43] that there are interfaces for all these [26:45] things and they are normal Java or.net [26:47] interfaces right so all of these are [26:50] lifted to an abstraction represented by [26:52] an interface so there's an interface for [26:54] associate emmitt II does it which [26:56] applies actually to both of these things [26:58] right because it's the same markup right [27:01] there's a interface for whether or not [27:05] things are countable in constant time [27:07] there's an interface for sequentiality [27:09] which is possessed by this but not that [27:12] right and and so there's a big hierarchy [27:15] of abstractions that are underneath this [27:17] which means it supports multiple [27:18] implementations I provide sorted you [27:21] know a red-black tree behind this and a [27:23] special hash array mapped try version of [27:27] this but you can write your own right [27:29] and the same code will work right for [27:31] you all you have to do is implement that [27:33] interface interact and then into then [27:35] this Worley's dudes will work correct [27:37] does that also mean that it's easy for [27:38] JVM and CLR programs to invoke the [27:42] evaluator yeah sure okay so you've got [27:44] complete bi-directional uh what we used [27:49] to call foreign function interface but [27:50] it's not so far very not far so so we [27:53] could look at that if if you want to say [27:55] you have two upper on a string I'm [27:58] running out of space here oh yeah let me [27:59] give you the eraser this side yeah [28:05] violating immutability here it's a to [28:10] upper hello birthdays yeah somebody [28:15] remind me of the syntax so there's some [28:21] there's some special syntax for a host [28:24] interrupts right so we can call a method [28:27] on an object using dot which is actually [28:32] connected to that symbol right we can [28:35] call you know some static thing [28:42] like that anything you can call new by [28:47] saying new you know my class or just my [28:52] class again to get back to sort of [28:54] treating it like a function right so [28:57] that's Interop yep and it it's very [29:01] transparent it also brings us into the [29:03] into the realm of mutable versus [29:06] immutable your basic data structures are [29:08] immutable meaning that functions that [29:10] perform transformations on them don't [29:12] destroy the data but these guys are of [29:15] course perforce mutable and very [29:17] observably so depends on which they are [29:19] but election isn't right but in in [29:22] general they're mutable actually and you [29:24] have to the style of programming is [29:27] completely different why don't we get [29:29] into get into that because one of the [29:32] things where I'd like to end up is I'd [29:33] like to understand the the closure [29:36] concept of identities okay and how you [29:39] how we navigate the waters of the the [29:42] mutable world which has one sort of feel [29:45] style intuition right that's appropriate [29:50] for it in the immutable world which is [29:52] less familiar to the average programmer [29:54] but has its own sort of conceptual [29:56] clarity that is very attractive to some [29:59] people and closures doing both and how [30:02] do we how do we navigate between those [30:04] waters and where does this identity [30:06] concept which I don't understand I want [30:08] to understand the identity concept first [30:10] of all and then I want to understand how [30:11] it fits into these these mixed waters [30:14] that I also understand so I like the [30:15] Blue Nile flowing into the White Nile [30:17] area well the first thing I would want I [30:19] would want to explain is what I'm [30:21] talking about when I say persistent data [30:23] straighter because I think it's one [30:26] thing to tell somebody something is [30:27] immutable and they can't change it in [30:29] which case they say that doesn't seem [30:31] very useful to hang of course and and to [30:34] give them a recipe for working with [30:35] immutable things that's both efficient [30:37] and convenient right and so what a [30:40] persistent data structure is is not [30:42] merely immutable a persistent data [30:44] structure has another and special [30:46] characteristics so we're not talking [30:47] about persistence meaning you can put it [30:49] on the disk in a database we're talking [30:51] about persistence in this case which [30:53] means essentially if you take some [30:55] immutable thing and you produce a new [30:57] version of it several things will be [31:01] true the old version will remain [31:03] accessible you can produce the new [31:07] version in compliance with whatever the [31:09] complexity guarantees are for that data [31:12] structure so if I say you can put [31:15] something at the end of a vector in [31:16] constant time [31:17] then doing that even though it's [31:19] producing a new vector must happen in [31:22] constant time right which implies [31:23] something immediately [31:25] it can't be copying the entire vector [31:27] right as if it were to be doing that it [31:29] wouldn't be meeting its complexity [31:30] guarantees it would be linear then right [31:32] so so many thing like they're sharing [31:34] going on it right so there's a bunch of [31:36] implications from this but the [31:37] guarantees are that the old version will [31:40] remain accessible that the new version [31:41] can be produced while meeting the [31:44] complexity guarantees that both the old [31:46] and new versions after making a new [31:49] version will also meet the complexity [31:50] guarantees because there are some people [31:52] who have attempted in the past to sort [31:54] of cheat to make versions where the the [31:57] latest version is is as fast as you [31:59] expect but older versions kind of decay [32:01] you know they and so there's no cheating [32:03] going on in closure data structures [32:05] they're really persistent and so they [32:09] meet the complexity guarantees even old [32:10] versions so so how could you possibly do [32:13] this how could you make a vector where [32:15] you could add something to the end in [32:17] constant time and the way you do it is [32:20] that it obviously can't be a big array [32:23] no right it has to be a tree yes [32:26] so the real secret is all of these data [32:29] structures except for the list which [32:30] which can meet us guarantees by just [32:32] being a singly linked list right are [32:34] trees and part of the work of designing [32:40] closure for me was to make versions of [32:43] hash maps so what you get by doing this [32:46] is a hash map right is not a red-black [32:48] tree okay I have red-black trees if you [32:51] want a sorted version you had that but [32:52] this is a hash map and this obviously [32:54] has constant time or near constant time [32:58] expectations right so making them fast [33:01] was a critical objective of closure so [33:03] the trees that underlie them are [33:06] extremely shallow they have very high [33:08] branching factors right so what it means [33:11] is to produce a new vector that's [33:12] represented as a tree means that vectors [33:15] are segmented right at the leaves of [33:18] these trees are arrays okay good okay so [33:22] now you can imagine like a try then it [33:24] is a try okay very good [33:26] you know some people pronounce a tree I [33:28] don't know how do you so yeah right see [33:30] they are ie hash array mapped rise and [33:39] the reference is Bagwell [33:42] so Bagwell good Bagwell did these [33:45] Bagwell did hash hash tables this way [33:49] non persistently non immutably right and [33:53] so what I do was make mutable versions [33:55] that were fast enough and then the [33:57] vectors are not really hash array maps [34:00] but they're they end up being in leaves [34:03] of a tree right well you can access them [34:06] by using their index to find the right [34:08] path right the branching factor is 32 so [34:11] they're extremely shallow yep but what [34:13] does it mean to add to the end of this [34:15] it means you're going to take the last [34:19] array right you're going to copy it [34:22] right and since it's got a bounded size [34:27] of 32 that's constant time yep right and [34:30] then you're going to copy the path to [34:33] the root okay you're going to end up [34:34] with a new root right and a new path [34:38] down the right side yeah this thing has [34:39] been added to this but the rest of this [34:42] will all share right this this tree this [34:46] new tree will share everything else [34:48] right so this thing has a million items [34:50] in it you're going to change you know [34:52] three nodes on the right side to add [34:54] something to the end right and that [34:57] makes that efficient very near constant [34:59] right right so that makes that efficient [35:01] and so that's called structural sharing [35:04] so persistent data structures have [35:05] structural sharing which means that you [35:08] can make a new version and it's [35:09] expensive to do so right and that's how [35:12] you program instead of saying change [35:13] this thing in place you say call this [35:17] transformation function which makes a [35:19] new thing and give me [35:20] yeah and of course that changes your [35:23] life because then you can share these [35:25] things in ways you could never share [35:27] objects right because an object you [35:29] don't know is this going to be okay can [35:32] I remember this do I need to clone it [35:33] yeah you don't know what's concurrency [35:36] involved it's a catastrophe all bets are [35:38] off right so that's not true you can [35:40] always use one of these because they are [35:42] immutable with no synchronization [35:44] whatsoever well let me stress this a [35:46] little bit when you say the old when you [35:51] say the old version is accessible what [35:52] that means practically is that if some [35:54] variable is bound to at that old version [35:57] and you reference that variable you will [35:59] still get the old things that means [36:01] you'll forget it right the computations [36:03] that you've already done right it means [36:04] the compiler can make deductions based [36:07] on those on simple textual analysis of [36:10] the program and say this this part over [36:11] here is safe that part over here is safe [36:13] and I can allocate one thread to one and [36:16] another thread to the other and they'll [36:17] never collide with more help it probably [36:20] could I mean theoretically be resistant [36:22] right in a system that that conveyed the [36:25] fact that this was immutable and I'm [36:27] handing it to you that way you could you [36:29] can continue that process right the in [36:32] practice closure does not do that right [36:35] it just gives you these data structures [36:37] and the confidence and you can call this [36:40] function it won't change what you've got [36:41] in hand you can freely alias things [36:43] right because they can't change realm [36:46] you can build recursive country undo [36:48] what could be easier now yeah let's undo [36:51] its just keep track of the old roots of [36:54] this tree and all the old backtracking [36:55] algorithms everything works beautifully [36:58] right so a lot of the old Lisp [36:59] algorithms which really could only work [37:00] on lists yeah now could work on sets [37:03] yeah you could you could do search with [37:05] sets path sets you know the old I set L [37:08] languages it just seems it's completely [37:10] incorporated right that kind of thing [37:12] you can do here yes yeah beautiful all [37:15] right now let's get into the if it's the [37:18] right thing to do if it's the right time [37:20] to get into that let's get into this [37:21] this these identities what okay these [37:23] critters okay so closure is not object [37:27] oriented right and it's not a closure [37:29] sort of a response to the challenges I [37:31] face trying to build these big [37:33] you know write multi-threaded systems [37:36] using object orientation I mean [37:38] eventually my programs became really [37:39] functional even c-sharp programs just as [37:42] a self-defense mechanism attic to do [37:46] that right and so in in my thinking [37:49] recently I tried to tease out a couple [37:52] of things that underlie object [37:54] orientation that I think aren't working [37:56] quite correctly and one is that object [38:00] orientation conflates identity and end [38:04] value okay right they're bound together [38:07] if you have some object you you have [38:12] something you'd like to reference over [38:13] time but its value also could change [38:16] over time there's no distinction between [38:18] you know Fred the thing I want to [38:21] remember mm-hmm and the value of Fred at [38:24] any particular instance of time there's [38:26] no distinct distinguishing those because [38:29] the object reference is essentially an [38:31] alias for a block of memory somewhere [38:32] right and when you look at it when you [38:34] reference it you don't have any idea no [38:36] I have no idea where you're going to go [38:38] and they'll be difficult the challenge [38:40] of concurrent programming is is timing [38:42] the access to things that's what way [38:45] does it yeah that right right so so you [38:48] know I have a picture in my in my [38:49] presentation where there's a bunch of [38:51] references to some objects you know [38:54] we'll call them Fred and and they call [38:58] them Fred they they like that yeah and [39:00] the problem is that there's no changing [39:03] Fred without potentially undermining [39:05] anyone observing it right they not only [39:08] have the identity right which is [39:11] important thing identity is an important [39:13] thing of Brian sure right well you're [39:15] not the same for a moment a moment [39:16] no yeah my public key might be something [39:19] about you is the same because we can [39:22] continue to refer to you that way right [39:23] so let's that's what identity is about [39:26] it's not a so your name you could change [39:27] your name but we could all remember well [39:29] you used to be called Brian now he's [39:30] calling himself Fred so it's not [39:33] actually the name it's it's the label we [39:35] we give to the continuous set of states [39:39] you're going to take on right it's [39:41] actually a label for a time line in a [39:43] way exactly yes it is a label for a time [39:46] line [39:46] so so this is broken okay it's broken in [39:53] a big way because it completes identity [39:55] with state this is a label associated [39:58] with this block of member to state [39:59] remember now that of a label associated [40:01] with the timeline right so so how can we [40:04] fix this what we can do is we can say [40:08] we're never going to let these things [40:10] change these are values but this is a [40:14] pile of memory it has really no logical [40:18] support all right it's not really a [40:20] value it's not really a time series of [40:23] values it's just cubbyholes [40:26] yeah somewhere right and and it's very [40:28] hard to think about it very so it's very [40:30] easy to think about a value all these [40:32] things they can't change right so we can [40:34] imagine some version of fret at some [40:36] point in time that's this value snapshot [40:39] every one and it could be a big [40:41] complicated thing like the absolutely [40:43] but its behavior over time is is this as [40:46] you know has no behavior over time right [40:48] so it doesn't be any simpler right it's [40:50] a value and instead what closure does is [40:53] going to say you can you can put your [40:55] identity concept over here okay you can [40:58] label this box yeah Fred and so closure [41:00] has a set of reference types okay which [41:03] are distinct from these value types [41:04] right and all they are are boxes okay [41:07] right and the boxes always refer to [41:10] values okay okay so what is this box [41:13] this box is a time management unit okay [41:16] good okay and so it ends up you can have [41:18] a variety of different semantics for [41:20] managing time for managing the [41:22] succession of values that you're going [41:24] to call Fred okay so let's say we want [41:28] to change Fred's something there's a [41:30] date of birth here you know and there's [41:34] some value we want to make we want to [41:36] change tangent we had it wrong yeah so [41:38] we need to change it so what we're going [41:39] to do is we're going to call some [41:40] function on this value yeah to produce [41:42] another value yeah okay which which will [41:46] be the Fred's value - yeah and because [41:51] we're going to use these persistent data [41:53] structures and the functions on them [41:54] producing this new value won't change [41:56] that one right so we can be doing this [41:59] at any time and anybody who is looking [42:00] at [42:01] is completely unaffected by that right [42:03] so we make we fix the date of birth here [42:05] and now it's better yep right at some [42:07] point we want to say all right this [42:08] should be the value that people use when [42:11] they when they refer to Fred right with [42:14] the new thing so what happens is we have [42:16] a way to atomically say this is now what [42:23] Fred is okay which means if you were [42:25] thinking about this what happened to you [42:26] well if I have a variable bound to it [42:29] that's still okay but if I'm referring [42:31] to it through Fred but you're not [42:33] I'm right have you if you're in the [42:34] middle of some computation using this [42:36] value everything's fine you're [42:37] completely unaffected by right you [42:39] cannot be affected by this right in fact [42:40] you can remember this forever yeah I [42:42] love the with Fred was born on the stage [42:43] yeah change his date of birth he's lying [42:45] and he was lying right yeah that wasn't [42:47] a good feature to change maybe is [42:49] occupation yeah we got yeah so these are [42:52] two independent things right so you do [42:54] your calculations on values yes and [42:56] producing this version maybe this is an [42:58] elaborate thing maybe it's such a set [42:59] date of birth equals something right new [43:01] but some elaborate transformation it [43:04] doesn't matter it's going to take one [43:05] value and return another right the [43:07] readers can never be impacted in [43:09] addition they don't hold up this [43:11] operation either the fact that someone's [43:13] reading this value doesn't impede [43:15] somebody making the next value of Fred [43:18] and so all you care about is that this [43:21] reference manages time well now is it [43:25] fair to say that the reference is just a [43:27] special kind of variable it is a special [43:29] kind of variable because the because [43:31] variable is another one of these empty [43:33] concepts yeah right with no semantics [43:36] right so what does it mean to be a time [43:38] construct right right it means you have [43:41] to ensure a succession of values all [43:50] right so the first key is there's only [43:52] going to be ever have in it a reference [43:54] to a value right right that succession [43:56] has to be atomic right the value inside [44:01] has to be visible okay there was I can [44:06] ask for what's infrared so there's a you [44:10] can say at Fred and close right time so [44:13] the so they're a lot like ml references [44:18] ml had one mutable thing allison erase [44:21] yeah which was a ref type and that's [44:24] what inspired this the problem is even [44:26] ml zareff type does not have semantics [44:28] for time right so the semantics for time [44:30] say how are you going to make sure this [44:31] is the case right if two people go in [44:33] and try to muck with this how we're [44:35] going to make sure that they know that's [44:37] going to work and it ends up that there [44:38] are multiple semantics for managing time [44:41] one of the simplest ones is castes just [44:44] compare and swap can Heron swap is a [44:47] single time line uncoordinated time [44:51] management system right that basically [44:53] says there's going to be a succession of [44:55] states here I'm going to make sure that [44:57] whoever's creating the next state is [44:59] doing so as a function of the [45:00] immediately previous state so I managed [45:03] the succession right in the old windows [45:06] Leo does interlock compare exchange I [45:09] should say all right [45:11] compare exchange just for those who are [45:13] in Windows good literate right I don't [45:18] think that's what I need [45:19] CPU calls it oh yeah no okay and then [45:23] you write test and said or you know yeah [45:25] okay so we can make sure that's atomic [45:28] and of course you can just read out of [45:30] it which means you're only going to see [45:31] the value at a particular point in time [45:32] you don't know immediately six after [45:35] that it could change right and and we [45:37] should become happy with this because in [45:39] the concurrent world there's a lot more [45:41] is going to work this way yeah so it [45:43] ends up you can do a lot with Kaz like [45:46] semantic originals for time right right [45:48] that that's way better than a variable [45:51] already because yes a variable you never [45:53] know that things are going to be atomic [45:55] you don't know that one value [45:56] it's a successor of another it hardly [46:00] changed and right it can be so closure [46:02] has a has a construct that wraps this [46:04] whole thing it's called an atom it's the [46:06] simplest time management construct it [46:08] uses Kaz internally right it does the [46:10] spin swap automatically so that every [46:13] value is a guaranteed function of the [46:16] immediate preceding value okay if it's [46:18] not your Kaz is going to fail and you [46:20] got to do over right but you can't do [46:23] things like transfer information between [46:25] accounts like that so now I'm [46:27] transactional yeah right well it's it's [46:29] it's it's only deals with a single thing [46:32] so it's isolated right right so a lot of [46:34] times you need to make changes to the [46:36] world that are coordinated so there's [46:38] also coordinated a reference type [46:40] enclosure which is called the ref okay [46:44] and that is transactional those [46:47] references can only be altered inside a [46:50] transaction okay so if I go and I have a [46:53] an Adam I can say swap and pass it a [46:56] function so so it's a beautiful thing [46:58] the way these things look oops important [47:01] part here yeah the way these things look [47:03] is if I have an Adam I can say swap my [47:08] Adam some function maybe some marks okay [47:15] right the function will be applied to [47:19] the folks will be applied to the current [47:21] value right with the args and that will [47:26] come the new value the next time yeah [47:29] okay so there are also references which [47:32] are transactional they look very similar [47:35] the you say alter alter some ref which [47:40] is now one of these transactional things [47:42] again you pass it a function and some [47:44] args right and maybe you won't do this [47:48] too you know two guys ultra ref one this [47:52] way ultra ref to some other way right [47:56] all right [47:57] okay so you do this that fails why you [48:03] didn't start a transaction [48:04] okay the coordination yes right so we [48:07] say do sync around this that's night and [48:11] now that succeeds either all succeeds [48:13] together or not or an unusual track acid [48:16] kind of things except it's not how do [48:18] you find out whether it succeeded or not [48:20] or does it just when - when do sync [48:23] returns it's guaranteed succeeded or it [48:25] can't be it's gotta you gotta have the [48:27] do sync the transactions will retry [48:30] automatic they'll retry automatic [48:31] correct [48:32] so Wendy sync is do sync is finished [48:34] without exceptions it's done with ok [48:36] exceptions is the way to find out [48:37] exceptions can flow out I mean if you [48:39] had an exceptional problem if your [48:40] problem was only that you couldn't get a [48:43] synchronized view of these two things [48:45] right that will automatically be retried [48:47] three tries for synchronization retries [48:49] for synchronization happen automatic but [48:51] if you get an exception out of here [48:52] there's a rollback you had something [48:54] right not it didn't happen good nothing [48:56] happen so so you have those you have [48:59] those semantics which looks very similar [49:01] so it's a model based around references [49:04] or time coordinators that ensure [49:08] succession of values that each value is [49:10] a function of the last right there are [49:12] different semantics for it coordinator [49:14] uncoordinated there's also an [49:16] asynchronous version which is a lot like [49:18] the atom except when you ask for the [49:20] change to be made that returns [49:21] immediately and the change happens [49:23] sometime later on a thread pool okay and [49:25] how do you pick up notification of the [49:27] change a core you don't oh I mean you [49:29] can't then you can but by default you [49:32] wouldn't it's it's a style of [49:33] programming like Erlang's [49:35] right where you want to stop caring [49:37] about that again your emphasis should be [49:39] away from the transactional the [49:41] transactional sort of old-school stop [49:43] the world I want to own the computer [49:45] right right it doesn't actually stop the [49:47] world it has it supports multiple [49:48] simultaneous transactions if they don't [49:50] conflict they can both succeed in [49:52] parallel right but you want to move as [49:56] things get more and more concurrent to a [49:58] world in which you want to care a lot [50:00] less right um you more fire-and-forget [50:04] more we have sort of fire-and-forget or [50:06] yes or you build communication [50:08] infrastructures but the beautiful thing [50:10] about these time units is that they [50:12] support like a uniform way to be [50:14] notified when something changes that [50:16] does exist and you can get it for any of [50:17] them right you can also in a uniform way [50:20] supply validators that say these are the [50:23] set of values I would accept here right [50:25] and provide an arbitrary thunking code [50:27] that will ensure that things that could [50:28] put there or okay by you now we went [50:33] over how these things are used with the [50:35] immutable or persistent data structures [50:37] in closure what what if you're trying to [50:39] use these with things that you you don't [50:41] you don't use them okay very good very [50:43] good very nice rigor because it's sort [50:45] of essential now I the reason why I [50:47] wanted to talk about the persistent data [50:49] structures is it's quite essential that [50:52] this being expensive and not disturb [50:55] that yeah because now we just saw well [50:57] it might get retry this might be done [50:59] speculatively right right um and and [51:02] also it needs to be it needs to be the [51:03] case that readers don't get you know [51:05] impacted by these things so they go [51:08] together right there's no taking this [51:10] cool reference identity thing and and [51:13] making it work with something mutable [51:14] well I could actually imagine a variant [51:17] of closure call it closure - - that is [51:20] purely functional has only these [51:21] persistent structures but also has this [51:23] reference idea and it doesn't have the [51:25] immutable data structures that you [51:26] definitely could yes oh you could it [51:28] would be a restricted thing that button [51:29] have the beautiful foreign function [51:30] interface or but it would all be clean [51:33] and yes yeah okay good yes well there [51:36] are a couple of areas that if we had a [51:38] couple more hours I would love to [51:39] explore and those would be we recently [51:43] have the new reactive framework in [51:46] dotnet which is AI observer and I [51:48] observable and I see some similarities [51:50] between the succession of values over [51:55] time and the and the I observer there [51:58] you know what how an i observer gets [51:59] notified mhm and I tend to think of [52:02] those as a elaborate callbacks but [52:04] they're really a variation of the [52:05] continuation monad or a variation of the [52:08] old continuation passing style which [52:10] 2-valve like a lead balloon it right in [52:12] the list world but still there there are [52:14] some very pretty ideas and I think I [52:16] observer and I observable have done a [52:18] beautiful job of fixing right in in [52:20] dotnet I'd love to be able to explore [52:22] that at length and the other is time [52:24] warp which is one of my old hobby horses [52:26] which was a whole operating system built [52:28] on speculative execution okay so the [52:32] sequences of operations specified with [52:38] respect to virtual time with a mapping [52:41] to real time that affected rollback and [52:43] commitment uh-huh with a deep analogy to [52:47] virtual memory so your entire program [52:49] was really just as space-time [52:51] coordinates azzaman which events took [52:54] place in in events were specified in [52:58] virtual space versus space and virtual [53:00] time-space but were realized in in real [53:03] time and in real address space by page [53:05] fault [53:06] time fault okay time fault being a [53:08] roll-back and it seems like there are [53:10] some great touch points that we could [53:13] definitely go until connected here I'm [53:16] not sure whether whether it would be a [53:19] cool idea to try to write time warp [53:21] enclosure or to try to write closure on [53:23] top of time warp or vice versa so that's [53:26] why I think we would need a couple more [53:27] unless the player that I'm I'm really [53:31] happy I get it I get it I love it I [53:34] think this is a another great piece of [53:38] work in the of course being a being a [53:41] part is Anna in the.net Arsenal but it's [53:43] also in the in the Java VM Arsenal and I [53:46] love it it's the kind of thing that I [53:47] will I would definitely use this this [53:51] afternoon alright great thank you very [53:54] much sure thanks