jasonofthel33t
AI-assisted summary of Expert to Expert: Rich Hickey and Brian Beckman - Inside Clojure — original source. Check the original for full context.
Source published:
Key ideas
- Rich Hickey designed Clojure around four foundational pillars: predominantly functional programming with default immutability, a Lisp foundation with syntactic homoiconicity, a concurrency model engineered to tame multi-threaded state, and first-class hosting on the JVM and CLR. (1:17)
- Clojure rejects object-oriented state management by arguing that OOP conflates identity with mutable memory addresses; instead, Clojure models identity as an explicit container managing a timeline of timeless, immutable values. (37:15)
- Persistent data structures in Clojure are implemented using shallow trees such as 32-way branching Hash Array Mapped Tries (HAMTs), achieving near-constant-time appends and updates via structural sharing without degrading access performance on older snapshots. (32:13)
- State management constructs like Atoms (uncoordinated compare-and-swap), Refs (Software Transactional Memory coordinated by dosync), and Agents (asynchronous background execution) strictly require persistent data structures to allow speculative retries without corrupting data. (46:02)
Detailed notes
## Background This video was published on YouTube by the channel jasonofthel33t on March 12, 2013, cross-posted from Microsoft's Channel 9 "Going Deep" series. The show notes describe an in-depth interview between software architect Brian Beckman and Clojure creator Rich Hickey, exploring Clojure as a dynamic, hosted functional Lisp dialect targeting the JVM and CLR, built around immutable persistent data structures, software transactional memory, and seamless host interop.
## TL;DR Rich Hickey outlines the core philosophy behind Clojure, detailing how its four pillars—functional programming, Lisp homoiconicity, hosted platform interop, and first-class concurrency—address the severe pitfalls of traditional multithreaded object-oriented programming. At the whiteboard, Hickey demonstrates how Clojure's persistent collections use shallow tree structures and structural sharing to guarantee near-constant-time updates without destructive mutation. The discussion culminates in Clojure's state model, which strictly decouples timeless, immutable values from time-managing reference constructs such as atoms, transactional refs (`dosync`), and asynchronous agents. Brian Beckman connects these mechanisms to broader systems concepts, including hardware compare-and-swap, Reactive Extensions, and the Time Warp operating system's speculative execution model.
## Atmosphere & style The conversation carries the relaxed yet rigorous atmosphere of two senior software architects deep in the weeds at a whiteboard. Beckman plays the role of an intellectually playful, enthusiastic interlocutor, frequently drawing sharp analogies to Lisp history, physics, and distributed systems, while Hickey delivers clear, disciplined, and pragmatic architectural explanations. It feels less like a standard product pitch and more like a masterclass in language design, with palpable mutual respect and Beckman's growing, contagious excitement as each piece of Clojure's state and data model clicks into place.
## Bulletin - [01:17](https://www.youtube.com/watch?v=wASCH_gPnDw&t=77s) Rich Hickey designed Clojure around four foundational pillars: predominantly functional programming with default immutability, a Lisp foundation with syntactic homoiconicity, a concurrency model engineered to tame multi-threaded state, and first-class hosting on the JVM and CLR. - [37:15](https://www.youtube.com/watch?v=wASCH_gPnDw&t=2235s) Clojure rejects object-oriented state management by arguing that OOP conflates identity with mutable memory addresses; instead, Clojure models identity as an explicit container managing a timeline of timeless, immutable values. - [32:13](https://www.youtube.com/watch?v=wASCH_gPnDw&t=1933s) Persistent data structures in Clojure are implemented using shallow trees such as 32-way branching Hash Array Mapped Tries (HAMTs), achieving near-constant-time appends and updates via structural sharing without degrading access performance on older snapshots. - [46:02](https://www.youtube.com/watch?v=wASCH_gPnDw&t=2762s) State management constructs like Atoms (uncoordinated compare-and-swap), Refs (Software Transactional Memory coordinated by dosync), and Agents (asynchronous background execution) strictly require persistent data structures to allow speculative retries without corrupting data.
## Key takeaways - [02:50](https://www.youtube.com/watch?v=wASCH_gPnDw&t=170s) Clojure targets the JVM and CLR runtimes directly, utilizing dedicated syntax like the dot operator for transparent host member access rather than an opaque foreign function interface. - [03:56](https://www.youtube.com/watch?v=wASCH_gPnDw&t=236s) Lisp was chosen as Clojure's foundation because its homoiconic nature and macro system treat code as native data structures, enabling users to extend syntax and language constructs directly. - [07:11](https://www.youtube.com/watch?v=wASCH_gPnDw&t=431s) Clojure adopts an applicative-order strict evaluation engine, but sequence abstractions like map and filter are lazy by default to support infinite data generation and stream processing. - [10:48](https://www.youtube.com/watch?v=wASCH_gPnDw&t=648s) Unlike pure functional languages that isolate side effects via static type systems and monads, Clojure adopts a dynamic typing model that avoids wrapping host APIs in monadic layers to preserve seamless platform interoperability. - [12:58](https://www.youtube.com/watch?v=wASCH_gPnDw&t=778s) Clojure expands classical Lisp homoiconicity by introducing literal syntax for persistent vectors, maps, and sets alongside traditional singly linked lists. - [15:10](https://www.youtube.com/watch?v=wASCH_gPnDw&t=910s) Arbitrary precision arithmetic is supported by default, automatically promoting integers to BigInt on overflow and maintaining exact ratios to prevent premature loss of numerical precision. - [18:45](https://www.youtube.com/watch?v=wASCH_gPnDw&t=1125s) Persistent vectors and maps act as callable functions of their indices or keys, allowing expressions like (v 0) to retrieve elements directly. - [21:28](https://www.youtube.com/watch?v=wASCH_gPnDw&t=1288s) The Clojure reader operates as an explicit parsing stage that transforms source text directly into live in-memory data structures before passing them to the compiler or evaluator. - [25:15](https://www.youtube.com/watch?v=wASCH_gPnDw&t=1515s) The classical Lisp cons operation is generalized to conj (conjoin), inserting elements where computationally optimal for each collection type, such as appending to vectors and prepending to lists. - [30:17](https://www.youtube.com/watch?v=wASCH_gPnDw&t=1817s) Functional persistence guarantees that older versions of data structures remain fully accessible and continue to meet identical runtime complexity guarantees indefinitely through structural sharing. - [32:13](https://www.youtube.com/watch?v=wASCH_gPnDw&t=1933s) Clojure's persistent vectors adapt Phil Bagwell's Hash Array Mapped Trie (HAMT) concepts into shallow, 32-way branching trees where appending modifies only a localized leaf array and root-path spine. - [40:04](https://www.youtube.com/watch?v=wASCH_gPnDw&t=2404s) Identity is defined as an entity tracked across a timeline, decoupled from the timeless, immutable snapshot values it references at any specific point in time. - [42:47](https://www.youtube.com/watch?v=wASCH_gPnDw&t=2567s) Concurrent reads in Clojure never block writes or require synchronization locks, because readers hold stable references to immutable value snapshots while identity boxes update atomically. - [46:02](https://www.youtube.com/watch?v=wASCH_gPnDw&t=2762s) Atoms provide uncoordinated state updates through compare-and-swap (CAS) loops, retrying state transformations automatically if another thread commits first. - [47:15](https://www.youtube.com/watch?v=wASCH_gPnDw&t=2835s) Software Transactional Memory (STM) coordinates updates across multiple Refs inside a (dosync ...) block, rolling back and retrying transactions atomically if conflicts occur. - [49:15](https://www.youtube.com/watch?v=wASCH_gPnDw&t=2955s) Agents enable asynchronous, fire-and-forget state mutations by dispatching transformation functions onto background thread pools. - [50:04](https://www.youtube.com/watch?v=wASCH_gPnDw&t=3004s) Clojure reference types cannot safely function with mutable data structures because speculative execution and retry loops would leave side effects half-applied.
## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=wASCH_gPnDw&t=0s) Introduction and Rich Hickey's Background
Brian Beckman introduces himself as an interviewer and introduces Rich Hickey, the independent creator of Clojure. Hickey shares his background, noting that he has worked as a professional programmer for a couple of decades. His career focused heavily on broadcast automation and scheduling systems, the National Exit Poll, and various machine listening problems. Beckman characterizes this domain as hardcore real-time engineering with heavy concurrency demands, which Hickey confirms involved high-stress throughput.
Beckman jokes that he is merely a "country physicist who dabbles in programming," though he has a long history with functional languages dating back to discovering Scheme around 1978 or 1979. Beckman shares that he has deployed Scheme in strange environments over the decades, citing his crowning glory as embedding Scheme directly inside the operating system kernel to manage page table replacement policies.
[01:17](https://www.youtube.com/watch?v=wASCH_gPnDw&t=77s) What Clojure Is: The Four Pillars
Before diving deeper into technical nuances, Beckman asks Hickey to define Clojure. Hickey explains that Clojure is a dynamic programming language and a dialect of Lisp, noting that Lisp is unusual in how naturally it supports and encourages dialects. He states that Clojure is founded on four core pillars:
```mermaid flowchart TD accTitle: The Four Pillars of Clojure accDescr: Diagram illustrating the four foundational design principles of the Clojure programming language.
C["Clojure Design"] C --> P1["1. Functional Language<br/>(Immutable by default, not purely enforced)"] C --> P2["2. Dialect of Lisp<br/>(Homoiconic, syntactic macro extensibility)"] C --> P3["3. Concurrency Focus<br/>(Designed to tame multi-threaded state)"] C --> P4["4. Designed to be Hosted<br/>(JVM & CLR first-class platform integration)"] ```
1. **Functional Programming:** It is predominantly functional, emphasizing immutability rather than mathematical purity. 2. **Lisp Foundation:** It retains the distinctive syntactic extensibility of Lisps, which remains unique even after mainstream languages adopted other Lisp features. 3. **Concurrency Focus:** It was born out of Hickey's frustrations with multi-threaded programming in industry: > “I've worked in all these broadcast automation systems were multi-threaded and they were done in C++ and Java and C# and you know basically that's too hard for me, so I was trying to make that easier.” 4. **Designed to be Hosted:** Clojure targets runtimes like the JVM and the CLR, not merely as an incidental compilation target (like ports of Common Lisp or Scheme), but as a first-class design choice meant to directly exploit host libraries, VMs, and runtime ecosystems transparently.
[02:50](https://www.youtube.com/watch?v=wASCH_gPnDw&t=170s) Interoperability and Prefix Host Syntax
Beckman asks whether hosting on the CLR means exposing reflection and an underlying object model. Hickey clarifies that Clojure makes host interop transparent rather than burying it behind a complex, foreign foreign-function interface (FFI). Clojure provides dedicated syntax and a core `.` (dot) special operator to reach directly into the host runtime, allowing programmers to instantiate classes via `new`, call methods, and access fields.
Macros wrap the dot operator to preserve idiomatic Lisp style. Regular function invocation retains standard prefix parenthesized notation `(foo x y z)`, while host member invocation retains the same prefix structure: `(.foo x y z)`.
[03:56](https://www.youtube.com/watch?v=wASCH_gPnDw&t=236s) Why Lisp: Homoiconicity and Syntactic Extension
Beckman inquires why Hickey chose Lisp instead of inventing a brand-new dynamic syntax for the JVM and CLR. Hickey responds that Lisp holds an enduring, unmatched advantage in syntactic extensibility. In conventional programming languages, if a developer wishes for a missing language construct—such as a `for-each` loop—they are stuck waiting for the language designers to implement it. In Lisp, developers do not have to wait: they can write the construct themselves.
Beckman connects this to metaprogramming, recalling Eugene Kohlbecker’s introduction of hygienic macros for Scheme. Hickey notes that many developers conflate Lisp macros with C-preprocessor macros, which are merely crude textual string substitutions. Real programmatic macros are programs that transform programs into other programs.
This transformation is enabled by Lisp being homoiconic (or auto-iconic): code is represented using the language's native data structures. Consequently, programs can manipulate source code using the exact same libraries and collection-manipulation tools used for ordinary data. While pure functional languages like ML, Miranda, and Haskell historically eschewed homoiconicity—following early academic arguments, such as Philip Wadler's paper suggesting calculating is preferable to "scheming"—they turned metaprogramming into an external topic. In Clojure, the language’s evaluation strategy explicitly provides an open hook:
> “The evaluation strategy incorporates a step that says at this point I will call your program to help me evaluate, so macros are a hook that's built into compilation or evaluation that allows your program—everybody can write a program that transforms another program, transforms source code—the trick is that that's part of the evaluation strategy of the language.”
[07:11](https://www.youtube.com/watch?v=wASCH_gPnDw&t=431s) Evaluation Strategy: Strictness vs. Laziness
Beckman asks about Clojure's evaluation strategy, specifically whether it uses strict or lazy evaluation. Hickey explains that Clojure’s core execution model is strict and applicative-order, meaning function arguments are evaluated before being passed into the function.
However, Clojure provides a hybrid approach through sequence abstractions: - The execution engine itself is strictly evaluated. - Sequences (`seq`) can be implemented lazily. - Core sequence transformations like `map` and `filter` are lazy by default.
Hickey and Beckman agree that generating infinite data structures and stream processing represent the primary practical utility of laziness in everyday software engineering, avoiding the global performance overhead of a purely lazy language like Haskell.
[08:06](https://www.youtube.com/watch?v=wASCH_gPnDw&t=486s) Pragmatic Functional Programming and Immutability
When asked to clarify how Clojure is functional without being "pure," Hickey outlines the spectrum of meanings attached to "functional programming": - **Haskell's definition:** Pure mathematical functions applied universally to all expressions, mediated by a type system. - **Historical Lisp definition:** Functions being first-class citizens that can be stored in variables and passed as values. - **Clojure's middle ground:** Emphasizing programming with pure functions that are free of side effects, accept immutable values as arguments, and return new values as output.
While traditional Lisps made immutability an optional programming convention, Clojure forces the issue structurally. Clojure’s core collections—including the data structures used to represent source code—are immutable and persistent. Because developers cannot mutate a collection in place, they must write functions that return new data values.
Hickey notes that Clojure does not strictly outlaw side effects; if a developer chooses to execute I/O inside a function, the dynamic type system will not stop them:
> “It's a dynamically typed language which just respects the programmer's right to do whatever they please.”
Beckman reflects on historical dynamically typed functional languages such as SASL and KRC, contrasting them with Miranda and Haskell. He observes that once languages adopted strong static type systems and type inference, functional constraints became far more manageable—pointing out that without type classes and an overloaded bind operator, using abstractions like monads quickly becomes an oppressive way to program.
[10:48](https://www.youtube.com/watch?v=wASCH_gPnDw&t=648s) Purity, Monads, and the Lisp Lineage
Brian Beckman and Rich Hickey begin by contrasting purely functional paradigms with Clojure's pragmatic design. Beckman notes that while pure functional programming offers algebraic cleanliness, it forces programmers to cart around extra arguments or navigate the complexities and pitfalls of monads, such as composition headaches and Haskell's monomorphism restriction when combining two monads. Beckman observes:
> “So Clojure is definitely coming from the Lisp tradition, walking towards a more functional style of programming; it's definitely not from the Haskell-Miranda tradition.”
When Beckman asks whether it is easy to write pure functions in Clojure and whether the language helps you detect when you leave that pure world, Hickey responds candidly:
> “It's easy to write a functional program or write portions of your program functionally, and there's no help whatsoever in terms of knowing when you've done that correctly or not.”
Because Clojure is dynamically typed, it possesses no type system to enforce the presence or absence of side effects or I/O. Developers can freely instantiate and manipulate mutable Java or.NET objects, retaining both their capabilities and their traditional pitfalls. Hickey explains that attempting to police purity dynamically would be counterproductive: it would require flagging virtually all existing Java and.NET frameworks as dangerous by default. Because seamless interop with host platforms is a primary virtue of Clojure, forcing users to wrap standard APIs in monads would severely undermine that practical advantage.
```mermaid flowchart TD accTitle: Pure Functional Typing vs Clojure Practical Interop accDescr: Contrasts Haskell's strict monadic isolation with Clojure's unmonitored functional core and host interop.
subgraph Haskell ["Haskell / Miranda Tradition"] A["Pure Functional Core"] --> B["Monads & Type System"] B --> C["Guaranteed Purity / Explicit I/O"] end
subgraph Clojure ["Clojure / Lisp Tradition"] D["Idiomatic Functional Core"] --> E["Dynamic Typing / No Effect Tracking"] E --> F["Direct Host Interop: Java /.NET"] F --> G["Practical I/O & Framework Reuse"] end ```
---
[12:58](https://www.youtube.com/watch?v=wASCH_gPnDw&t=778s) Persistent Data Structures and Platform I/O
Clojure achieves practical functional programming not through restrictive type mechanics, but by establishing immutable data structures as the default, idiomatic choice. Hickey points out that traditional Lisps and Schemes primarily privileged singly linked lists as their core recursive, structural type; other structures, such as Common Lisp hash tables, were mutable in-place structures identical to Java collections. Clojure radically expands Lisp's homoiconicity by introducing first-class literal representations for vectors, maps (key-value associations), and sets alongside lists.
All these collections are persistent and immutable, implemented using shallow tree structures that support efficient structural sharing when creating modified versions. Because literal syntax and standard library operations revolve around these immutable types, developers naturally remain in a pure calculation model for the core logic of their programs. When developers eventually need to render graphics, persist records to disk, query a database, or serialize XML, they can reach directly into the Java and.NET standard libraries, where the vast majority of platform utility resides in I/O plumbing.
---
[15:10](https://www.youtube.com/watch?v=wASCH_gPnDw&t=910s) Numerics and Symbolic Computation
Beckman transitions to Clojure's numerical capabilities, prompting Hickey to outline its arithmetic foundations:
- **Arbitrary precision arithmetic**: Integers do not wrap around on overflow; they automatically promote to big numbers (`BigInt`). - **Exact rationals**: Ratios like `22/7` are preserved in exact fractional form rather than immediately converting to floating-point numbers. - **Common Lisp tradition**: While Scheme enforces a strict distinction between exact and inexact numbers across its numeric tower, Clojure adheres closer to Common Lisp, ensuring by default that calculations do not lose precision.
Hickey clarifies that while symbolic simplification (such as algebraic reduction found in Macsyma) is not built directly into the core runtime, Clojure's code-as-data architecture makes implementing symbolic engines straightforward. As an example, he highlights Incanter, a statistical computing and data analysis environment for Clojure modeled after the R language. Because Clojure provides transparent macro extensibility, domain languages like Incanter integrate smoothly with standard Clojure without awkward bridging layers or foreign function interfaces.
---
[16:55](https://www.youtube.com/watch?v=wASCH_gPnDw&t=1015s) Whiteboarding: Literal Syntax and Data Literals
Beckman suggests moving from abstract principles to concrete syntax, inviting Hickey to the whiteboard to illustrate Clojure expressions for those unfamiliar with Lisp. Hickey begins with prefix notation:
```clojure (+ 1 2) ;; => 3 ```
Operators appear first, followed by arguments, with no need for commas or semicolons. Hickey then diagrams the core collection literals and basic types supported natively in syntax:
| Literal Form | Type | Example | |:--- |:--- |:--- | | `(...)` | List | `(1 2 3)` | | `[...]` | Vector | `[1 2 3]` | | `{...}` | Map | `{:a 1:b 2}` | | `#{...}` | Set | `#{:a:b:c}` |
Hickey emphasizes that maps are key-value associations, sets naturally eliminate duplicate elements without preserving order, and all these composite structures are arbitrarily nestable (e.g., maps containing vectors, or vectors containing maps).
---
[18:45](https://www.youtube.com/watch?v=wASCH_gPnDw&t=1125s) Lists vs. Vectors, and Functions as Collections
Beckman asks Hickey to contrast lists with vectors. Hickey explains that the difference lies in their operational complexity:
- **Lists (`List`)**: Standard singly linked lists. Adding an item to the head using `cons` (construct) is $O(1)$. However, accessing an arbitrary element (e.g., the 14th item) or appending to the end requires traversing the chain, running in $O(n)$ linear time. This structure naturally suits recursive decomposition (`first` and `rest`). - **Vectors (`Vector`)**: Persistent array-like structures that provide efficient access by index and efficient append operations at the end.
When Beckman asks how index lookups are expressed syntactically, Hickey highlights one of Clojure's defining design choices: vectors and maps act as callable functions of their keys or indices. A developer can query the first element of vector `v` simply by invoking it as `(v 0)` or by calling `(get v 0)`.
```mermaid flowchart LR accTitle: List vs Vector Algorithmic Characteristics accDescr: Compares the insertion and lookup complexity between Clojure lists and vectors.
subgraph ListOps ["Persistent List (Linked)"] L1["Prepend (cons)"] -->|"O(1)"| L2["Head of List"] L3["Lookup by Index / Append"] -->|"O(n)"| L4["Linear Traversal"] end
subgraph VectorOps ["Persistent Vector (Tree)"] V1["Append to Tail"] -->|"O(log32 n) ~ O(1)"| V2["Vector End"] V3["Indexed Lookup: (v idx)"] -->|"O(log32 n) ~ O(1)"| V4["Direct Retrieval"] end ```
---
[20:33](https://www.youtube.com/watch?v=wASCH_gPnDw&t=1233s) Symbols, Keywords, and Strings
The discussion concludes with Hickey clarifying the distinctions among Clojure's primitive identifiers on the whiteboard:
- **Symbols (`a`)**: Used to name program entities, variables, and bindings. If evaluated directly, a symbol resolves to the value it names; to treat it as a pure symbolic identifier, it must be quoted (`'a`). - **Keywords (`:a`)**: Prefixed with a colon, keywords evaluate to themselves. Hickey succinctly summarizes the fundamental distinction:
> “A symbol might stand for something else in use in a program; a keyword only ever designates itself.”
- **Strings (`"a"`)**: Standard string literals, which in Clojure are direct instances of underlying host platform strings (e.g., `java.lang.String`).
[21:28](https://www.youtube.com/watch?v=wASCH_gPnDw&t=1288s) Literals, Abstract Syntax Trees, and the Reader
Rich Hickey and Brian Beckman explore how Clojure handles strings and literal data structures. In Clojure, strings are native host types—a Java string on the JVM and a CLR string on the.NET runtime—with zero wrapper overhead. Although a string does not turn into a linked list of characters, it can still be treated sequentially, allowing functions like `map` to operate directly across it. Beckman observes that Clojure's map literal syntax represents an arbitrary discrete mapping where both keys and values can be any arbitrary value, rather than being restricted to trivial name-value pairs.
Beckman points out that Lisp’s prefix notation provides a direct, literal representation of abstract syntax trees (ASTs). When expressions nest, they map straight onto trees without ceremony. He recalls the industry reaction when XML emerged: “when XML came out the old-timers said this is nothing but Lisp with right well noisy Lisp right and the XML-er says what's this list thing that's that right hiding the structure but it's just trees right.” Hickey explains that writing a Clojure program fundamentally consists of writing data structures—primarily lists, but also vectors, maps, and sets.
The core distinction of Lisp lies in having a two-phase syntax rather than defining language syntax purely in terms of text characters. Hickey emphasizes: “it's a two-phase syntax there is your source code right right then there's something called a reader and the source code is text what the reader outputs is data.” Source code passes into the reader, which constructs live in-memory data structures—such as a list containing the symbol `+` and the numbers `1` and `2`. That data structure is then handed to the evaluator or compiler. Because of this separation, programs can generate data structures dynamically in memory and pass them directly to the evaluator without touching a disk file, or pass data to macros that transform and return new data structures before evaluation.
```mermaid flowchart LR accTitle: Clojure Two-Phase Syntax Pipeline accDescr: Diagram illustrating source text passing through the Reader into Data Structures, interacting with Macros, and entering the Evaluator or Compiler.
subgraph Phase1["Textual Phase"] A["Source Text<br/>File / REPL"] end
subgraph Phase2["Data Structure Phase"] B["Reader"] C["In-Memory Data<br/>Lists, Vectors, Maps"] M["Macro Expansion<br/>Data to Data"] end
subgraph Phase3["Execution Phase"] D["Compiler / Evaluator<br/>Bytecode Generation"] end
A --> B B --> C C --> M M --> C C --> D ```
- When the evaluator encounters a list, it treats the first item as an operator or function call and the remainder as arguments. - Vectors, maps, and sets evaluate directly to themselves as literal data. - While a list can be treated as literal data by quoting it with a tick mark (`'`), idiomatic Clojure uses vectors (`[...]`) for generic data grouping. - Because data structures print and read symmetrically, Clojure code needs no external serialization formats like JSON or XML.
---
[25:15](https://www.youtube.com/watch?v=wASCH_gPnDw&t=1515s) Persistent Collections, conj, and Generic Abstractions
Hickey details the behavior of Clojure collections, noting that vectors provide fast lookup by index and fast append at the end. Because these collections are persistent, exploratory and recursive algorithms can build new vectors conditionally and discard them freely without mutating earlier versions. Backtracking requires no cleanup steps because the original collection remains untouched.
Clojure abstracts classical Lisp operations: the traditional `cons` operation, which attached an element to the head of a singly linked list, has been broadened into `conj` (conjoin). Rather than forcing everything to behave as a head-inserted list, `conj` adds elements to a collection at the position most natural and efficient for that data structure's concrete type. Conjoining to a list prepends to the front; conjoining to a vector appends to the end; conjoining to a map inserts key-value pairs; and conjoining to a set adds unique elements.
Generic higher-order operations build on these primitives. Hickey demonstrates `into`, which pours elements from one collection into another by performing a recursive `reduce` using `conj`:
```clojure (into target-collection (map f source-collection)) ```
This uniform abstraction model functions across all collections because every Clojure collection implements a sequence interface. Under the hood, Clojure structures are defined against standard JVM or CLR host interfaces rather than opaque engine internals. Hickey explains that Clojure establishes an abstraction hierarchy with distinct interfaces for associativity (shared by maps and vectors), constant-time countability, and sequentiality. The platform ships with concrete implementations—such as red-black trees for sorted structures and hash array mapped tries (HAMTs) for hashed collections—while allowing user-defined data structures implementing these interfaces to interoperate transparently with `into`, `map`, and `reduce`.
---
[27:37](https://www.youtube.com/watch?v=wASCH_gPnDw&t=1657s) Host Interoperability and the Tension Between Mutable and Immutable
Beckman asks if this interface-driven design makes it easy for JVM and CLR hosts to invoke the Clojure evaluator and achieve full, bi-directional interop. Hickey confirms this, noting that Clojure's foreign function interface is "not so far" because it compiles directly to host bytecode and talks to the host without runtime barriers.
Hickey grabs the whiteboard eraser to show host syntax—joking about “violating immutability here” as he wipes the board. He demonstrates standard syntax shortcuts for calling host APIs: - Invoking instance methods: `(.toUpperCase "hello")` - Accessing static members: `(Math/PI)` - Instantiating host classes: `(new MyClass)` or `(MyClass.)`
Beckman points out that interacting directly with host objects immediately brings programmers face-to-face with the friction between mutable and immutable paradigms. Clojure's native data structures are persistent and immutable—transformations produce new values rather than destroying the source—whereas standard host objects are perforce mutable and observably side-effecting. Beckman remarks that working in an immutable world requires a completely different mindset, and expresses a strong desire to explore how Clojure navigates both worlds:
> “I'd like to understand the Clojure concept of identities and how you how we navigate the waters of the the mutable world which has one sort of feel style intuition that's appropriate for it in the immutable world which is less familiar to the average programmer but has its own sort of conceptual clarity that is very attractive to some people and Clojure is doing both and how do we how do we navigate between those waters and where does this identity concept which I don't understand I want to understand the identity concept first of all and then I want to understand how it fits into these these mixed waters that I also understand so I like the Blue Nile flowing into the White Nile area.”
---
[30:17](https://www.youtube.com/watch?v=wASCH_gPnDw&t=1817s) Defining Persistent Data Structures and Complexity Guarantees
Before tackling identity, Hickey insists on laying down the exact computer science definition of a *persistent data structure*. He notes that simply telling programmers that something cannot be changed causes immediate skepticism: “I think it's one thing to tell somebody something is immutable and they can't change it in which case they say that doesn't seem very useful to hang of course and and to give them a recipe for working with immutable things that's both efficient and convenient.”
Hickey clarifies that "persistent" here has nothing to do with saving data to a disk or relational database. In functional programming, persistence means that when a new version of a collection is produced: 1. **Accessibility:** The previous version remains fully accessible and completely unchanged. 2. **Complexity Guarantees:** Producing the new version adheres strictly to the algorithmic complexity guarantees of that collection type. If inserting into a vector is specified as constant time, generating the new vector must run in constant time. 3. **No Full Copies:** Because creating a new collection must meet constant-time complexity, the implementation cannot simply duplicate the entire structure, which would be an $O(N)$ linear operation; it must rely on structural sharing. 4. **No Version Decay:** Both old and new versions retain their runtime complexity guarantees indefinitely.
Hickey emphasizes that Clojure rejects implementation shortcuts that compromise older references: “there's no cheating going on in Clojure data structures they're really persistent and so they meet the complexity guarantees even old versions.” He notes that previous historical attempts at persistent structures often cheated by optimizing only for the latest version while allowing access to older snapshots to degrade in performance. Clojure guarantees that all versions—past and present—remain equally fast and fully functional.
[32:13](https://www.youtube.com/watch?v=wASCH_gPnDw&t=1933s) Persistent Vectors and Hash Array Mapped Tries
Rich Hickey explains how to build a persistent vector where appending an element takes near constant time. It cannot be represented as a large monolithic array; it must be implemented as a tree. Aside from the classic singly linked list, almost all Clojure data structures are trees. While Clojure provides red-black trees for sorted collections, general hash maps and vectors demand constant or near-constant time performance. To achieve this, the underlying trees are extremely shallow with very high branching factors.
```mermaid flowchart TD accTitle: Structural Sharing in a 32-Way Persistent Tree accDescr: Diagram showing how an appended node copies only the spine path while sharing unchanged leaf arrays.
subgraph OldTree ["Original Vector Root"] R1["Old Root Node"] L1["Leaf Array (Indices 0..31)"] L2["Leaf Array (Indices 32..63)"] R1 --> L1 R1 --> L2 end
subgraph NewTree ["New Vector Root (Appended)"] R2["New Root Node"] L3["New Copied Leaf Array (Appended)"] R2 --> L3 end
R2 -.->|"structural sharing"| L1 ```
Brian Beckman immediately recognizes the concept, asking whether it functions like a trie, which Hickey confirms. Hickey cites Phil Bagwell's work on Hash Array Mapped Tries (HAMTs). Bagwell originally designed these non-persistently for mutable hash tables; Hickey adapted the approach to design fast, immutable versions. For Clojure's vectors, elements reside in arrays at the leaf nodes, indexed directly along the path down the tree. With a branching factor of 32, the trees remain exceptionally shallow.
When appending an element to a vector: - The last leaf array (bounded at a maximum size of 32 elements) is copied, which takes constant time. - The path from that leaf back up to the root is copied, creating a new root and a new right spine. - All other branches of the tree are retained through structural sharing without modification.
As Rich Hickey summarizes the efficiency: “If this thing has a million items in it, you're going to change, you know, three nodes on the right side to add something to the end, and that makes that efficient—very near constant.”
- Persistent data structures rely on structural sharing, making the creation of new modified versions inexpensive without changing existing data in place. - Programming shifts from in-place mutation to calling a transformation function that returns an updated reference while leaving original structures intact. - Immutability eliminates defensive cloning and synchronization concerns across concurrent threads. - Backtracking algorithms and undo operations become straightforward because applications can simply retain references to previous root nodes.
---
[37:15](https://www.youtube.com/watch?v=wASCH_gPnDw&t=2235s) The Flaw of Object Orientation: Conflating Identity and Value
The conversation moves to Clojure's rejection of traditional object-oriented programming. Hickey recounts that Clojure arose directly out of his struggles building large, multithreaded systems with OOP languages, where he found himself writing in an aggressively functional style inside C# purely as a defensive measure. In analyzing what makes object orientation break down in concurrent environments, Hickey argues that the paradigm fundamentally conflates identity with value.
In mainstream OOP, an object reference is merely an alias for a mutable block of memory. It couples the conceptual entity a developer wants to track over time with whatever mutable values happen to sit inside that memory chunk at a given instant. Brian Beckman points out that the core challenge of concurrent programming is coordinating access timing across threads. When an object is mutated, it inevitably risks corrupting or invalidating the view of any other observer examining that same memory address.
Hickey explains that identity is not the memory address, nor is it merely a string name like "Fred" or "Brian"—names can change while identity persists. Instead, an identity is a stable label assigned to a timeline of continuous states that an entity occupies. In Hickey's words: “Object orientation conflates identity and value, right? They're bound together. If you have some object, you have something you'd like to reference over time, but its value also could change over time. There's no distinction between, you know, Fred, the thing I want to remember, and the value of Fred at any particular instance of time.” Traditional OOP assigns a label directly to a mutable block of memory rather than treating the identity as a reference to a succession of timeless values across time.
---
[40:04](https://www.youtube.com/watch?v=wASCH_gPnDw&t=2404s) Separating Values from Identities with Reference Types
To fix the structural defects of mutable objects, Clojure cleanly decouples value types from reference types. Values are immutable snapshots: they have no internal behavior over time and cannot change under any circumstances, making them simple to reason about. Reference types, by contrast, act as explicit containers or "boxes" that designate identity.
```mermaid flowchart LR accTitle: Decoupling Identity and Value in Clojure accDescr: Shows an Identity Box atomically switching its pointer from one immutable value to another while a reading thread safely retains the older value snapshot.
subgraph Identity ["Identity (Time Management Box)"] Ref["Fred (Reference Type)"] end
subgraph Timeline ["Immutable Value Snapshots"] V1["Fred Value Snapshot 1<br>(Original DOB)"] V2["Fred Value Snapshot 2<br>(Corrected DOB)"] end
Ref -->|"Atomically updated to"| V2 ThreadReader["Reading Thread"] -.->|"Holds stable reference to"| V1 ```
Hickey defines these identity boxes as time management units: “And all they are are boxes, right? And the boxes always refer to values. Okay, so what is this box? This box is a time management unit.” Clojure provides multiple reference types offering distinct concurrency semantics to govern how the timeline of values progresses.
To modify an entity: - An update does not mutate memory in place. If an error is discovered in Fred's recorded birthdate, a transformation function takes the original immutable value snapshot (`Fred Value 1`) and returns an updated immutable snapshot (`Fred Value 2`). - Structural sharing ensures that generating the new snapshot is fast and does not disturb the original. - The reference box is atomically redirected to point to the new value snapshot.
Any thread or computation currently reading `Fred Value 1` remains entirely unaffected by the identity pointer update. A thread can retain its snapshot indefinitely with absolute consistency. As Beckman and Hickey jokingly observe, even if Fred's date of birth is updated because he previously lied about his age, any observer holding the earlier snapshot is completely isolated and cannot be compromised by the concurrent change.
[42:47](https://www.youtube.com/watch?v=wASCH_gPnDw&t=2567s) References as Time-Management Constructs
Rich Hickey clarifies the crucial separation of concerns in Clojure's state model: performing calculations on immutable values is completely independent of the reference that holds them. Whether calculating a new version of an entity involves a minor attribute tweak like changing an occupation or an elaborate, expensive data transformation, the calculation simply takes one immutable value and returns another. Crucially, readers can never be impacted by this process, nor do they block concurrent updates; reading the current value of an identity like "Fred" does not impede another thread from constructing the next version of Fred. All the reference needs to do is manage time properly. Brian Beckman asks whether it is fair to describe a reference as just a special kind of variable. Hickey agrees, noting that "variable" is an empty, semantically vague concept in conventional computing.
Hickey explains that a genuine time construct must ensure an atomic succession of immutable values. A reference will only ever hold a reference to a value, the succession from one value to the next must happen atomically, and the current value must remain visible to readers—which Clojure provides via the dereferencing syntax `@Fred`. Hickey observes that this design draws inspiration from Standard ML's `ref` type (which was essentially ML's only mutable construct besides arrays), but even ML's `ref` lacked clear concurrency semantics for time. When multiple threads concurrently attempt to modify state, there must be a defined mechanism to manage time. One of the simplest models is compare-and-swap (CAS), an uncoordinated, single-timeline management system ensuring that whoever generates the next state does so as a mathematical function of the immediately preceding state. Beckman points out that in the Windows world, this matches the `InterlockedCompareExchange` primitive (often related to underlying hardware test-and-set instructions). With CAS, reads observe a consistent snapshot at a specific point in time without holding locks, even if the value changes an instant later.
```mermaid flowchart LR accTitle: Clojure Value Succession Model accDescr: An atomic reference points to successive immutable values over time without blocking readers. subgraph Succession["Succession of Immutable Values"] V0["Value t0: {Fred, Job: Dev}"] V1["Value t1: {Fred, Job: Architect}"] V0 -->|"f(v0)"| V1 end subgraph Reference["Time Construct"] R["Reference: @Fred"] --> V0 R -.->|"atomic update"| V1 end Reader["Concurrent Readers"] -->|"deref (@)"| V0 ```
- When evaluating what a reference means, Rich Hickey emphasizes: “It is a special kind of variable because the because variable is another one of these empty concepts yeah right with no semantics right so what does it mean to be a time construct right right it means you have to ensure a succession of values all right so the first key is there's only going to be ever have in it a reference to a value right right that succession has to be atomic right the value inside has to be visible”.
[46:02](https://www.youtube.com/watch?v=wASCH_gPnDw&t=2762s) Atoms, Refs, and Coordinated Transactions (`dosync`)
To wrap compare-and-swap semantics cleanly, Clojure introduces the `Atom`. An atom is an uncoordinated reference type that handles low-level spin-swapping automatically. To mutate an atom, you call `(swap! my-atom some-function args...)`. The atom applies the provided function to its current state along with any arguments, producing the next state. If another thread updates the atom in the interim, the CAS fails and the atom automatically retries the operation with the newer value. However, because atoms are isolated and uncoordinated, they cannot safely manage operations across multiple independent identities simultaneously—such as transferring funds between two bank accounts.
For multi-reference coordination, Clojure provides transactional references (`Ref`s), managed by Software Transactional Memory (STM). Modifications to refs occur using `(alter my-ref some-function args...)`. If a developer invokes `alter` outside of a transaction, Clojure throws an error. Coordinated updates across multiple refs must be wrapped inside a `(dosync...)` block. If two transactions conflict or cannot obtain a synchronized snapshot of the involved refs, the STM system aborts and retries the transaction automatically. When `dosync` exits cleanly without exceptions, the entire atomic update has succeeded; if an unhandled application exception occurs, the transaction aborts and cleanly rolls back, leaving no half-applied mutations.
- Rich Hickey describes the mechanism of an atom's update: “Clojure has a has a construct that wraps this whole thing it's called an atom it's the simplest time management construct it uses CAS internally right it does the spin swap automatically so that every value is a guaranteed function of the immediate preceding value”. - On coordinating multiple refs, Rich Hickey explains: “So we say do sync around this that's night and now that succeeds either all succeeds together or not or an unusual track ACID kind of things except it's not”.
[49:15](https://www.youtube.com/watch?v=wASCH_gPnDw&t=2955s) Asynchronous Agents, Immutability Guarantees, and Beckman's Parallels
Beyond atoms and refs, Hickey introduces Clojure's asynchronous reference type: the `Agent`. Like atoms, agents manage a succession of values, but dispatching a state transformation via an agent returns immediately, scheduling the calculation to run on a thread pool in the background. While notification mechanisms exist, the agent philosophy mirrors Erlang's message-passing style, encouraging developers to embrace "fire-and-forget" asynchronous flows rather than trying to freeze the universe with heavyweight transactions. Additionally, all Clojure reference types provide uniform support for watches (change notifications) and user-defined validator functions that can reject illegal state transitions before they commit.
Beckman asks whether these reference constructs can or should be paired with mutable data structures. Hickey firmly rejects the idea: “You don't, you don't use them.” The entire reference architecture relies completely on persistent data structures. Because transactions and atomic swaps may execute speculatively and retry multiple times, mutating data in place would corrupt state and disrupt readers. Beckman concedes the point and muses about a hypothetical, ultra-strict "Clojure--" that enforces pure persistence without mutable Java interop. Beckman concludes by highlighting two deep conceptual touch points: 1. Microsoft's Reactive Extensions (`IObservable`/`IObserver` in.NET), which represent values changing over time via elaborate continuations/callbacks. 2. The Time Warp operating system—an architecture rooted in speculative execution, virtual time coordinates, and rollback mechanisms triggered by "time faults," creating an intriguing parallel to Clojure's retry-driven STM.
- Rich Hickey underscores why Clojure references require persistent data structures: “So they go together right there's no taking this cool reference identity thing and and making it work with something mutable”. - Brian Beckman connects the model to speculative execution architectures: “And the other is Time Warp which is one of my old hobby horses which was a whole operating system built on speculative execution okay so the sequences of operations specified with respect to virtual time with a mapping to real time that affected rollback and commitment uh-huh with a deep analogy to virtual memory”.
[53:23](https://www.youtube.com/watch?v=wASCH_gPnDw&t=3203s) Wrapping Up: Clojure Across .NET and the JVM
Reflecting on the depth of the discussion, Brian Beckman notes that exploring the remaining architectural layers—such as putting systems on top of Time Warp or vice versa—would easily require several more hours of conversation. Despite running out of time, Beckman expresses immense enthusiasm for what Rich Hickey has demonstrated and built, emphasizing that he completely grasps the model and its value. He praises Clojure as a tremendous achievement, observing that while he naturally approaches software from the perspective of a.NET partisan, Clojure's availability across both the.NET and Java Virtual Machine ecosystems makes it an exceptionally compelling tool.
Beckman's excitement is palpable as he concludes the interview, noting that Clojure is not merely an interesting theoretical exercise but something practical and ready for immediate adoption. As Beckman enthusiastically puts it, “It's the kind of thing that I will I would definitely use this this afternoon.” Hickey warmly accepts the praise with a quick word of thanks, bringing their whiteboard session to a close.
- Brian Beckman notes that fully connecting these concepts to Time Warp or deeper runtime mechanics would demand significantly more time than a single session allows. - Clojure's dual presence on both the CLR (.NET) and the JVM makes it a versatile, cross-platform functional weapon in a modern developer's toolkit. - The interview wraps up with Beckman eager to put Clojure into direct practice right away.
## Notable quotes > [37:15](https://www.youtube.com/watch?v=wASCH_gPnDw&t=2235s) Object orientation conflates identity and value, right? They're bound together. If you have some object, you have something you'd like to reference over time, but its value also could change over time. — Rich Hickey
> [10:48](https://www.youtube.com/watch?v=wASCH_gPnDw&t=648s) Clojure is definitely coming from the Lisp tradition, walking towards a more functional style of programming; it's definitely not from the Haskell-Miranda tradition. — Brian Beckman
> [20:33](https://www.youtube.com/watch?v=wASCH_gPnDw&t=1233s) A symbol might stand for something else in use in a program; a keyword only ever designates itself. — Rich Hickey
> [10:48](https://www.youtube.com/watch?v=wASCH_gPnDw&t=648s) It's easy to write a functional program or write portions of your program functionally, and there's no help whatsoever in terms of knowing when you've done that correctly or not. — Rich Hickey
## Mentioned Rich Hickey, Brian Beckman, Clojure, Java Virtual Machine (JVM), Common Language Runtime (CLR), .NET, Java, C#, C++, Lisp, Scheme, Common Lisp, Haskell, Miranda, ML, Standard ML, SASL, KRC, Erlang, Eugene Kohlbecker, Philip Wadler, Phil Bagwell, Incanter, Macsyma, R, Reactive Extensions (Rx), Time Warp operating system
## Video description Cross posted from http://channel9.msdn.com/Shows/Going+Deep/Expert-to-Expert-Rich-Hickey-and-Brian-Beckman-Inside-Clojure.
Clojure is a dynamic programming language created by Rich Hickey that targets both the Java Virtual Machine and the CLR. It is designed to be a general-purpose language, combining the approachability and interactive development of a scripting language with an efficient and robust infrastructure for multithreaded programming. Clojure is a compiled language - it compiles directly to JVM bytecode, yet remains completely dynamic. Every feature supported by Clojure is supported at runtime. Clojure provides easy access to the Java frameworks, with optional type hints and type inference, to ensure that calls to Java can avoid reflection.
Clojure is a dialect of Lisp, and shares with Lisp the code-as-data philosophy and a powerful macro system. Clojure is predominantly a functional programming language, and features a rich set of immutable, persistent data structures. When mutable state is needed, Clojure offers a software transactional memory system and reactive Agent system that ensure clean, correct, multithreaded designs.
Astrophysicist and Software Architect Brian Beckman interviews Rich Hickey to dig into the details of this very interesting language. If you don't know much about Clojure and the general problems it aims to solve, well, watch and listen carefully to this great conversation with plenty of whiteboarding and outstanding questions. Expert to Expert simply rocks! Thank you for spending time with us, Rich! Clojure is great!
[Source](https://www.youtube.com/watch?v=wASCH_gPnDw)