Rich Hickey Splits Database Architecture to Eliminate Read Locks

Rich Hickey重构数据库架构以彻底消除读锁 将事务写入与不可变的分段读取解耦，使应用服务器能够在本地执行查询并不受协调限制地永久缓存数据。 Hickey解构了传统单体数据库，对原地更新突变、查询处理与事务协调的强耦合以及历史信息的丢失提出了批判。借鉴函数式编程和持久化数据结构的思想，他提出了一种以不可变原子事实（称为datom）为中心的信息模型，这些事实被保存在只追加的时间线中。由此产生的架构将数据库解耦为不同角色：用于协调的单写事务器（transactor）、用于持久化存储的通用存储服务，以及在本地针对缓存的不可变段运行丰富查询的对等应用服务器。该设计消除了读锁，实现了零成本的时间旅行和假设性推演查询，并支持查询工作负载的弹性水平扩展。 传统数据库混淆了过程（写入和协调新事实）与感知（查询和索引），将两者与具有破坏性的原地更新绑定在同一个单体服务器中。 (9:58) Datomic将数据库架构解构为三个解耦的角色：专用的单写事务器（transactor）、通用键值存储（如DynamoDB或RDBMS），以及在应用层内部执行的无状态对等查询引擎。 (38:36) 由于存储被处理为结构化为持久树的不可变索引段，索引数据可以无限期缓存在应用内存或Memcached中，而无需缓存失效逻辑。 (43:21) 将时间作为原子事实（[实体, 属性, 值, 事务]）中的一等公民维度加以保留，可实现零成本的历史查询（as-of）、假设性分支推演，以及基于固定数据库基准的协调异步工作流。 (17:28)

Rich Hickey contends that traditional databases introduce pervasive architectural complexity by conflating write coordination with query evaluation and overwriting historical records. He proposes deconstructing the database into decoupled services using an accretive model of immutable facts, separating write processing from read perception to achieve elastic scalability and native temporal querying. The Limits of Place-Oriented Database Monoliths (3:02) Traditional relational databases treat storage as a mutable "place" where block-oriented B-trees overwrite data in place, discarding historical context and requiring heavy locking coordination. Hickey identifies this architectural flaw as the source of the "basis problem": querying a mutable database over sequential network round trips yields shifting, inconsistent baselines. This instability forces developers into bad software designs, such as bundling unrelated business decisions and reporting logic into monolithic queries to ensure they operate on the same data reality. While distributed NoSQL engines address horizontal scalability through key-value storage, Hickey observes that they often force applications to sacrifice ACID guarantees and rich declarative queries. Concurrently, manual relational workarounds—such as tracking time via custom timestamp columns—introduce brutal index contention and tuning friction under online write loads, while reactive eventing continues to depend on inefficient polling. An Accretive Information Model of Datoms (17:28) To replace place-oriented overwriting, Hickey advocates an information model built on atomic facts termed "datoms." Each datom consists of an entity ID, an attribute, a value, and a transaction-based temporal marker ($t$). Transactions are treated as first-class entities capable of recording identity, provenance, and audit metadata alongside committed assertions. Instead of bundling unrelated attributes into composite rows or documents, this structure isolates facts at the lowest granular level. Under this model, database state expands outward through accretion, analogous to tree rings adding wood without altering interior layers. Novelty is represented purely through assertions and retractions. Retracting a fact does not erase its historical validity; it merely asserts that the statement ceased to apply at a designated transaction point. By reifying process as an explicit, cumulative timeline of facts, the database preserves the historical context necessary for auditing and longitudinal analysis. Separating Write Process from Read Perception (25:54) Hickey deconstructs the conventional monolithic database server into two distinct halves: process and perception. Process handles novelty coordination and is assigned to a dedicated transactor. To guarantee consistency without contention, the transactor acts as a single writer that sequences transactions, executes deterministic transaction functions atomically for compound operations like bank balance updates, logs writes to an append-only log, and broadcasts incoming changes to connected nodes. Perception is entirely offloaded to application servers acting as "peers." Peers run query engines in-process and evaluate data locally against immutable index snapshots and live memory buffers. Because reading immutable data requires no distributed locking, read coordination disappears entirely. Durable persistence is delegated to pluggable distributed storage services (such as DynamoDB or SQL engines), isolating storage durability from application query compute. Durable Persistent Structures and Temporal Querying (58:37) To index data immutably without destructive updates, Datomic applies functional programming's persistent data structures to disk through structural sharing and path copying. Query engines on application peers merge historical index segments fetched from storage with buffered in-memory novelty using on-the-fly merge joins. Because historical index segments are strictly immutable, caching becomes universal and automatic: peer object caches and intermediate Memcached tiers retain segments indefinitely without developer-managed invalidation. This architecture unlocks native temporal capabilities. Applications can issue `as-of` queries to inspect past states without altering query logic or schemas, and can evaluate speculative "what-if" transactions entirely in local memory without altering shared storage. Furthermore, distributed workflows can coordinate reliably by passing an explicit database basis $t$, ensuring distinct services inspect the exact same historical reality across asynchronous execution steps. Hickey concludes that deconstructing the database preserves ACID transactions and declarative relational queries while tapping the elasticity of modern distributed storage. However, this architecture deliberately relies on a single transactor to sequence novelty, trading away arbitrary write scalability in order to protect consistency and simplify the programming model.

Rich Hickey 提出，传统单体数据库将存储、事务协调与查询执行紧密绑定，并采用就地覆写更新机制，严重破坏了信息的时间维度并引发基准不一致问题；通过将数据库解构为基于不可变事实的值模型，系统能够将写入处理、读取感知和持久化存储解耦，实现无协调的弹性查询与历史可回溯性。 单体数据库的局限与基准问题 (3:02) Hickey 认为，软件复杂性主要根源于状态与控制，传统客户端-服务器关系型架构虽然提供了优秀的声明式查询能力，却将数据视为一个不断在原地发生改变的“位置”（place），从而引发“基准问题”（basis problem）。当应用程序在不同时间点发出连续查询时，底层的物理数据可能早已被并发事务修改，迫使开发者为了避免数据在查询间隙消失或变动，将业务逻辑与展示查询强行打包为庞大而复杂的单一查询。此外，传统数据库依赖就地更新（update-in-place）的 B 树结构，抹去了历史事实和因果关系；为了追踪时间，团队往往被迫引入时间戳字段，但在高并发写入场景下，针对“当前有效记录”的查询调优面临极高锁争用与索引性能瓶颈。面对扩展性难题，2012 年前后的部分 NoSQL 架构虽换取了弹性伸缩，却普遍牺牲了 ACID 事务保证与声明式查询能力。 原子事实与不可变增积信息模型 (17:28) 为消除就地修改的缺陷，Hickey 提出应当建立以不可变值（value）为核心的信息模型。传统的关系行或文档结构将多个不相关的属性强行捆绑在一起，而 Datomic 将数据拆解为细粒度的原子事实——“datom”，每个 datom 由实体 ID、属性、属性值以及事务时间维度（$t$）共同构成。在这种模型下，业务变化不再通过覆盖物理位置来实现，而是像树的年轮一样不断向外增积（accretion）：新发生的事情意味着占用新空间，而已经记录的事实永远不可更改。过程的演变被具象化（reified）为基础的断言（assertions）与撤回（retractions）。即使某个事实在后续时刻被撤回，它在过去某个时间区间内为真的历史事实依然成立，从而使系统无需借助离散的历史备份快照，就能在单一数据视图中直接进行跨任意时间窗口的纵向趋势分析。 解构架构：处理、感知与存储三权分立 (25:54) Datomic 在物理实现上打破了单体数据库的垄断，将核心职责解构为三个独立角色：负责推进过程的事务协调器（Transactor）、负责感知的对等节点（Peers，即应用服务器端）以及底层的通用存储引擎（Storage）。Transactor 采用单写者架构，专注于保证事务一致性、日志持久化并将新产生的数据实时广播给各节点，由于其彻底剥离了查询负担，计算资源全部集中于写入排序。底层的存储系统被抽象为不可变持久化数据结构（durable persistent data structures），仅作为通用键值黑盒运作，能够无缝插拔 Amazon DynamoDB、传统 SQL 数据库或内存键值集群，存储只追加日志和由树节点构成的不可变索引段。由于节点段完全不可变，新旧索引树之间可以通过路径复制实现大规模结构共享，从而使得历史索引段被缓存在本地内存或 Memcached 时永远不需要失效处理。 边缘查询、动态合并与事务函数 (43:21) 通过将查询引擎作为库直接嵌入在应用对等节点（Peers）的 JVM 进程内，系统将计算推向了网络边缘。由于历史索引段全局不可变，读取操作完全不需要与 Transactor 协调，读事务争用从架构中彻底消失。当对等节点执行查询时，它只需要按需懒加载工作集数据，并将本地内存中暂存的近期增量索引与来自底层存储的持久化历史索引段进行高效的归并连接（merge join）。针对具有前后依赖关系的原子更新（例如银行账户余额增加），Datomic 引入了事务函数，允许将确定性函数与当前数据库值传入 Transactor 中执行，在事务内部原子化求值并递归展开为底层的断言与撤回操作；对于无冲突场景，客户端亦可采用乐观预计算来避免占用集中处理资源。 历元状态赋能历史回溯与假设分析 (58:37) 将数据库状态视为显式的不可变值，催生了基于历元（epochal）的状态转换模型：数据从一个完全一致的状态点跃迁到另一个状态点，不存在任何脏读或中间态。开发者在调试生产环境历史异常时，不再受限于数据被后续操作破坏，只需通过 `as-of` 查询将数据库句柄锚定在特定历史时间点或事务基准 $t$，便能以零额外存储开销重现当时的现实，且原有查询语句无需附加任何时间筛选逻辑。此外，对等节点可以在本地内存中基于当前状态推演投机性的事务数据，直接在离线状态下执行“假若分析”（what-if analysis），而无需影响中心数据库。在响应事件时，Transactor 广播的事件直接包含变更前后的数据库不可变值与增量事实，彻底终结了轮询机制以及事件触发后因二次并发写入导致的视图漂移。 Rich Hickey 总结指出，将数据库解构为不可变值不仅能够显著降低系统的认知与维护复杂度，还赋予了架构在物理拓扑上自由重配组件的能力；不过，该架构依赖单写者模型来维持强一致性，其对写入吞吐量极限和大规模对等节点广播机制的适应能力，构成了该设计在特定超高写入负载场景下的权衡约束。

## Background 该视频由 InfoQ 于 2012 年 8 月 23 日发布在 YouTube 平台上。视频记录了 Clojure 的缔造者兼 Datomic 设计师 Rich Hickey 在 JaxConf 2012 上的技术演讲，核心内容是探讨并呈现一种颠覆传统的全新数据库架构视角。 ## TL;DR Rich Hickey 深入剖析了传统单体关系型数据库的弊端，指出就地更新（In-place update）不仅抹杀了时间历史，还因缺乏稳定基准（Basis）而导致应用缓存和查询协调变得极其复杂。他提出将单体数据库彻底“解构”，把系统拆分为负责追加新事实与协调写入的 Transactor、作为不可变黑盒的键值存储，以及嵌入在应用进程内部的 Peer 节点。通过将数据库视为一个由原子事实（Datom）不断向外累加构成的不可变纯“值”（Value），系统彻底消除了读协调与缓存失效难题，使历史回溯（As Of 查询）、本地推演（What-If）与查询算力的弹性横向扩展变得轻而易举。 ## Atmosphere & style 整场演讲充满批判性思考与哲学思辨，Rich Hickey 举重若轻地用幽默风趣的比喻（如树木年轮、黑板上的奇迹、以及下午五点抓狂找 Bug 的程序员）化解了枯燥复杂的系统架构难题。现场互动生动，观众频频被讲者对传统数据库痛点的尖锐剖析引发强烈共鸣。整体氛围如同一次酣畅淋漓的思想重塑，听众在层层递进的解构与逻辑推导中，真切体会到底层设计哲学从“繁杂纠缠”走向“优雅极简”所带来的震撼与解脱。 ## Bulletin - [40:28](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=2428s) Rich Hickey 提出解构传统单体数据库的架构方案，将核心能力拆解为负责写入协调的 Transactor、底层的键值存储黑盒以及内嵌在应用中的查询节点 Peer，使读写路径彻底解耦。 - [38:36](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=2316s) 基于不可变性与“事实累加（Accretion）”模型，读事务及其分布式协调开销被完全消除，系统可以在 Peer 节点或共享 Memcached 中激进缓存且永不过期。 - [58:37](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=3517s) 将数据库状态抽象为不可变的值（Values），让历史追溯（As-Of 查询）、本地无副作用的假设分析（What-If）以及确定性的历史 Bug 重现变得极为轻量和原生。 - [30:14](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=1814s) Datomic 底层存储只需满足简单的 Key-Value 读写与一致性读，可直接插件化对接 Amazon DynamoDB、传统 SQL 数据库或 Infinispan 内存网格。 ## Key takeaways - [03:02](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=182s) 传统数据库将状态、控制与就地覆写（In-place update）绑定，导致业务因缺乏不可变的“基准（Basis）”而被迫写出高度耦合的臃肿查询以防基准漂移。 - [30:14](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=1814s) 现代硬件下“磁盘局部性已死”，跨机网络与磁盘 I/O 的速度差距大幅缩窄，因此数据库应将磁盘抽象为纯粹的存储黑盒，转而在内存中保障局部性。 - [14:20](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=860s) 关系代数虽然严密，但传统数据库的原地更新破坏了真实世界的“信息模型”；完整的信息应以不可变事实（Datom）累加记录，并天然内嵌时间维度。 - [22:22](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=1342s) 系统状态的推进通过原子操作“断言（Assertion）”与“撤回（Retraction）”具象化表达，撤回不代表抹除历史，而是记录事实在当前不再成立。 - [28:13](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=1693s) 通过引入函数式编程的持久化数据结构（Persistent Data Structures）与路径复制技术，Datomic 将不可变结构共享推广到了外存耐用性存储层。 - [38:36](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=2316s) 架构将新事实（Novelty）的处理与已有事实的观察（Perception）彻底拆分，查询操作无需经过中心协调器，因而传统“读事务”在系统中完全消亡。 - [47:30](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=2850s) 查询引擎作为库内嵌于各个 Peer 节点中，读取通过将本地内存的实时索引（Live Index）与存储中的历史持久化索引执行轻量合并（Merge Join）完成。 - [43:21](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=2601s) 不可变数据使得缓存失效（Cache Invalidation）难题不复存在，存储段一旦写入便永久有效，系统可在堆内及 Memcached 实施下沉到底层的无痛缓存。 - [51:15](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=3075s) 对于依赖当前状态的复杂写逻辑，系统通过事务函数（Transaction Functions）递归展开为基础事实，并提供客户端乐观计算与服务端原子执行的双重权衡。 - [56:19](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=3379s) 不同于部分 NoSQL 牺牲强一致性换取写入扩展，Datomic 坚持单写入者的纪元式状态模型，在保障强一致的同时，查询算力可通过动态扩容 Peer 节点实现线性弹性扩展。 - [58:37](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=3517s) 数据模型仅定义属性（Attribute）维度，原生支持零成本的“As-Of”时间旅行查询，避免了在业务 SQL 的 JOIN 条件中手工混杂时间戳过滤条件的痛苦。 - [01:01:06](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=3666s) Peer 节点支持在本地对拟提交变更进行分支推演（What-If 分析），使开发者无需向数据库提交事务即可就地运行完整查询来验证猜想。 - [01:02:01](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=3721s) Transactor 广播事件流时同时附带变更前数据库值、事务数据与变更后数据库值，为应用感知系统变化提供了确定性、零竞态的完整上下文。 ## Detailed walkthrough ### [00:01](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=1s) 解构数据库：打破软件复杂性的根源 Rich Hickey 开场介绍了演讲主题《解构数据库》（Deconstructing the Database）。他阐明，这不仅是对数据库架构全新视角的探索，同时也是 Datomic 背后架构基础的揭示——Datomic 是他当时已研发两年的数据库系统。但他随即强调，本场演讲绝非推销产品的商业宣讲，其核心在于深入剖析这些设计抉择背后的底层思想。 ```mermaid flowchart TD accTitle: 编程复杂度的根源与解法困境 accDescr: 展示 Out of the Tar Pit 论文归纳的两种复杂度，以及纯函数式与关系模型所忽略的过程更新问题 subgraph Causes["编程复杂度的两大根源"] C1["状态 (State)"] C2["控制 (Control)"] end subgraph Proposed["经典理想方案"] P1["函数式编程 (Functional)"] P2["声明式编程 (Declarative)"] P3["应用内关系数据模型 (Relational)"] end subgraph Missing["缺失的关键闭环"] M1["外界变化如何进入系统？<br/>(进程/变化更新机制)"] end Causes --> Proposed Proposed -.->|"如同黑板漫画：<br/>'此处发生奇迹'"| M1 ``` 解构数据库所要解决的核心问题，是编程中的“复杂度”。面对台下听众，Rich Hickey 抛出一个带有些许调侃的反问：“有多少人认为与数据库打交道是一件轻松且毫无烦恼的事？”台下鸦雀无声，他笑着点评“太好了，绝大多数人都不这么认为”。 关于软件系统复杂度的本质，经典论文《走出泥潭》（*Out of the Tar Pit*）曾指出，编程中的所有复杂度最终都可以归结为两类：一类与状态（State）有关，另一类与控制（Control）有关。该论文的作者提议通过采纳函数式编程、声明式编程，以及在应用程序内部引入关系数据模型来彻底消除这种复杂性。Rich Hickey 赞同该论文极具启发性，但他指出该方案存在一个致命的漏洞：作者们擅长函数式与声明式推理，却在系统如何接纳真实世界变化的问题上“打了退堂鼓”。这就像那幅经典的数学家漫画一样——黑板上写满了推导公式，但在核心步骤处却写着“然后，此处发生奇迹”。 该模型假定应用内部存在一个可供查询的关系模型，却完全回避了数据是如何被更新的、外界新发生的事件如何被并入系统。Rich Hickey 将这一被回避的核心议题定义为程序中的“进程（Process）”问题：“在其他部分都尽量保持纯函数式的前提下，面对现实世界中不断涌现的新事物，这些新信息到底该安放于何处？” - 论文《Out of the Tar Pit》归纳的复杂度根源：状态（State）与控制（Control）。 - 论文提出的理想化路径：函数式编程 + 声明式编程 + 应用内关系模型。 - 理论落地时的断层：完全忽略了变化（更新过程/进程）如何发生，留下了“奇迹发生”的逻辑空白。 ### [03:02](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=182s) 声明式编程的优势与“基准（Basis）”缺失的隐患 在重新审视数据库时，Rich Hickey 指出首要目标是全面拥抱声明式编程。日常开发中，对于非人工智能领域的研究人员来说，最常见的声明式工具正是 SQL。声明式编程在操纵数据方面，远优于常规命令式语言甚至函数式语言；其背后的高阶集合逻辑（higher order set logic）非常卓越。他犀利地指出，面向对象语言在处理数据上表现糟糕：“我们之所以使用面向对象语言，仅仅是因为我们手头恰好拥有它们，而不是因为它们在这方面做得更好；实际上它们在处理数据时要糟糕得多。”然而在传统 Client/Server（C/S）数据库架构下，声明式逻辑被隔绝在远端服务器上，成了与应用程序脱节的异物。 这种远端黑盒机制直接导致了灾难性的“缺乏基准（No Basis）”问题。如果在应用中想要基于数据库进行某项计算，决策所依赖的基准到底是什么？若数据库整体无时无刻不在就地修改，开发者就永远无法确知当前计算所面对的状态到底是什么，只能无奈地说：“我不知道，它大概是我上周二运行这个计算时看到的样子，但我现在根本无法准确重现当时的基准。” 这一困境在客户端与数据库的多次往返请求（Round Trips）中被进一步放大。人们往往误以为抗拒网络往返只是为了性能，其实根本原因在于对“基准漂移”的恐惧： > “如果我需要做一个复合决策，我能否在一段时间内向数据库提出三个互不相关的独立问题，然后得出我的最终答案？不能。为什么？因为在这些调用之间，那个被称为‘数据库’的地方已经发生了变化。” 为了规避这种风险，开发者被迫写出极不健康的耦合代码——最典型的便是将业务决策逻辑与前端展示报表逻辑强行混在一起。例如，程序原本应该先查出哪些商品在做活动，随后由展示模块再去独立查询渲染所需的具体字段；但在现实中，人们往往把这两道查询捆绑成一条臃肿的大 SQL，唯恐分开查询会导致两次结果集对应不上。这种设计上的倒退，完全是被传统数据库缺乏不可变基准所逼出来的妥协。 ### [06:00](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=360s) 分布式权衡、模式僵化与被遗忘的时间维度 谈及近年来 NoSQL 技术的兴起，Rich Hickey 剖析了传统数据库在一致性与横向扩展之间的挣扎。由于传统数据库服务器在设计上默认是单体架构（monolithic），难以应对大规模扩展。从 Amazon Dynamo 到 Google Bigtable，业界开始拥抱分布式技术与弹性伸缩，但这往往伴随着沉重的代价：开发者为了换取横向扩展能力，不得不放弃一致性，从而引入了全新的系统复杂度。重新审视数据库架构的核心命题在于：在吸收分布式系统的任意可扩展性（arbitrary scalability）与弹性（elasticity）的同时，我们究竟能保留多少传统数据库的核心价值主张？ 除了扩展性，传统关系型数据库在灵活性上的僵化也积弊已深。繁重的矩形表格（二维表）、必须人为设立的多对多关联表（intersection tables），这些机械的底层物理构造反向污染了业务逻辑。当面对稀疏数据（sparse data）、不规则数据或层级结构（hierarchical data）时，关系模式显得笨拙而脆弱，直接扼杀了敏捷开发的灵活性。 更严重的问题在于对“信息与时间”的割裂。计算机诞生之前的真实世界中，记账和档案从来都是只追加不覆盖的，具备真正的记忆（real memory）与可审计性。然而在当代数据库中，覆盖修改破坏了历史。很多团队不得不手动在表结构里塞入时间戳（timestamp）字段，自行管理有效区间，Rich Hickey 描述了由此带来的工程折磨： > “有多少人曾经自己写过查询，从那张表里把属于‘当前’（now）的数据捞出来？是的。有多少人对那个查询做过性能调优？是的，那简直是一场噩梦。有人喜欢那种查询吗？调优那种查询是极其残暴的，尤其是在线上联机系统里，资源争用（contention）可怕至极。” ### [08:45](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=525s) 感知与反应：告别低效的轮询机制 ```mermaid flowchart LR accTitle: 传统轮询与事件驱动机制对比 accDescr: 展现传统轮询模式的资源浪费，以及触发器机制因缺乏时间点基准导致的竞态困境 subgraph Polling["传统轮询机制"] direction TB App1["客户端应用"] -->|"低效定时轮询 (Poll)"| DB1["单体数据库<br/>(状态就地覆盖)"] end subgraph Triggers["触发器机制与基准缺失"] direction TB DB2["数据库触发事件"] -->|"通知：数据已变更"| App2["监听服务"] App2 -.->|"回查详情：此时数据<br/>可能已被再次篡改！"| DB2 end ``` 在演讲第一部分的尾声，Rich Hickey 提出了对数据库更高层次的诉求：建立健全的“感知与反应”（Perception and Reaction）模型。“感知”指的是能够获取一个稳定不变的基准来进行决策；而“反应”则涉及对真实世界事件与状态变化的响应。 当被问及在传统 C/S 数据库中如何观察数据变化时，Rich Hickey 打趣道，人们使用的是一个以 P 开头、以 L 结尾的四个字母的下流词汇——轮询（Poll），这种方式非常粗鄙恶劣（very gross）。现代系统需要一种不依赖轮询的响应式机制，而且这种机制必须能够提供完全一致的世界视图。 开发者即使手工利用数据库触发器（Triggers）搭建事件通知体系，依然无法跳脱“基准缺失”的梦魇。系统收到“某条记录发生了变更”的通知后，若试图回头再去查询数据库以获取详情，此时的数据可能早就被其他事务再次篡改，网络中正处于在途（in flight）状态的操作变得不可捉摸。现有的系统机制根本无法向调用者保证：“这次变更精确对应数据库在某个特定时间点的状态，你可以随时安全地回到那个时间点发起问询。”这种缺乏时间锚点的设计缺陷，彻底切断了响应式架构所需的一致性前提。 ### [09:58](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=598s) 传统单体数据库的解构与架构局限 当系统试图理解正在发生的事情时，如果仅仅被动接收到“某物发生了变更”的通知，开发者根本无法推断出导致该变更的原因是什么、它将产生何种后续影响，或者它与真实世界中的其他事物究竟有何联系。为了彻底改进这一现状，Rich Hickey 指出必须将数据库“拆解”（take a database apart），首先审视传统数据库是如何拼装在一起的。虽然各家实现细节略有差异，但传统数据库的整体布局基本一致：核心始于最底层的持久化需求——数据库必须具备耐用性（durability）。因此，传统设计通常围绕“一块磁盘”展开，将其封装在一个专属的机器（Box）里，由该机器全权掌控磁盘的一切，所有操作都经由这一 I/O 子系统进行调度。 在磁盘 I/O 之上，数据库主要提供两项依托于底层存储的核心服务：其一是处理事务（transactional component），负责接收外界产生的新增变化（novelty）并将其融合进当前的世界视图中；其二是支持查询（query support component），负责接收外界提出的问题并基于内部存储的数值返回答案。Rich Hickey 强调，查询引擎的威力完全取决于存储的组织方式：“如果把进入数据库的所有数据仅仅追加（append）到一个扁平文件中，那查询引擎的效率能有多高？显然不会很高。”数据库之所以具有极高的杠杆效应（leverage），是因为它建立了索引，将数据以排序视图的形式妥善存放，使得查询引擎能够迅速解答疑问。反观如今大行其道的键值存储（Key-Value Stores），几乎丧失了这种索引带来的杠杆收益，但人们仍然称其为数据库。Rich Hickey 讽刺道，在数据库诞生之前人们就早已拥有了文件系统等各种存储机制，当时并没有人把它们叫作数据库，如今却为何要把几乎没有查询杠杆的键值存储强行冠以“数据库”之名？ ```mermaid flowchart TD accTitle: Traditional Monolithic Database Architecture accDescr: Diagram illustrating the client-server monolithic layout of traditional databases including queries, transactions, caching, and disk storage. subgraph Clients ["客户端层 (Clients)"] App1["应用客户端 A"] App2["应用客户端 B"] end subgraph CacheTier ["应用缓存层 (Caching over Database)"] Cache["独立缓存 (Cache / ORM)"] end subgraph Monolith ["单体数据库服务器 (Monolithic DB Box)"] direction TB subgraph Services ["核心服务层"] Tx["事务组件 (Transaction Engine)<br/>接收并整合 Novelty"] Query["查询引擎 (Query Engine)<br/>回答声明式问题"] end Idx["索引与结构化存储 (Indexing)"] IO["磁盘 I/O 子系统 (Disk I/O)"] Services --> Idx --> IO end Clients -->|"SQL 字符串查询 / 事务"| Monolith Clients <-->|"读写缓存 (失效由应用自行负责)"| CacheTier CacheTier -.->|"缓存未命中时回源"| Monolith ``` ### [12:40](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=760s) 客户端通信的割裂与应用层缓存的困境 传统数据库历来是一个庞大而复杂的单体进程（monolith），由于当年的计算硬件、所需内存以及承载它的机器都极其昂贵，因此它是一个必须被集中托管的“特权资源”——系统中通常只有这么一台数据库服务器。相较之下，应用客户端则轻量得多。客户端与数据库之间必须通过一种“外语”进行通信：“你怎么跟一个 SQL 数据库打交道？就是把一段外语字符串发过去，然后它去执行某些动作。”查询也是如此，客户端送出一段字符串，随后在 Java 等语言层面上拿到一个看似 ResultSet 的返回结果。 随着业务规模扩大、应用数量激增，这台唯一的数据库资源很快就会不堪重负。由于绝大多数应用的负载都是以读取为主（read-dominated），频繁的查询会榨干这台独占机器的算力。于是，几乎所有系统最终都会被迫在数据库之上再增加一层缓存（Cache）：“如果我提问的成本非常昂贵，我就会把这些问题的答案存进缓存里。这样下次再想问同样的问题时，我会先查缓存，查不到再去承受访问服务器的高昂开销。”然而，这带来了无休止的维护泥潭： - 究竟什么样的数据该放入缓存？ - 缓存数据应该以何种形态呈现？ - 这些缓存条目在什么时候应当失效（invalidate）？ Rich Hickey 直言：“这些问题归根结底全变成了你的问题，或者你去购买某种花哨的 ORM，它在你的问题之上又套了一层，现在你就有了两个问题。”这种凌驾于数据库之上的缓存模式（caching over the database），本质上是把服务器原本应该解决的数据一致性与视图管理责任强行甩锅给了应用程序开发者。 ### [14:20](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=860s) 数据模型、状态模型与信息模型的缺失 深入剖析传统数据库的内部概念，会发现它不仅包含存储与 API 机制，还绑定了特定的“数据模型”（data model）。以 SQL 数据库为例，其建立在关系代数（relational algebra）的坚实数学基础之上，这赋予了它出色的声明式编程能力。然而，传统数据库同时内置了一套极其简陋且成问题的“状态模型”（state model）。Rich Hickey 尖锐地指出，关系代数本身是完美的数学推导，它假设数据库存在一个既定的状态，所有代数运算在此状态上都是严密的；但是，数据库到底是如何获得新状态的？在传统模型里，“那就像是发生了一个奇迹，然后你就得到了一个新的关系世界，但 Update 操作根本谈不上有什么数学模型。” 更关键的是，传统数据库通常完全缺失了一套真正的“信息模型”（information model）： - **原地更新（Update in place）**：传统状态模型普遍依赖就地覆盖，用新值暴力抹去旧值，彻底丢弃了历史演进轨迹。 - **事实与时间的脱节**：真正的“信息模型”应当具备存储不可变“事实”（facts）的能力，不让新数据覆盖旧数据，并且每一笔事实都必须具备内在的时间维度（temporal notion）。 回顾历史，Rich Hickey 指出 30 多年前数据库刚被发明时，硬件资源极端匮乏，磁盘容量小得可怜，设计者根本不可能采取“保留一切”的策略。为了在微小的磁盘上运转，先驱们发明了大量围绕就地更新的技术：B 树索引、磁盘块的就地覆写与重复利用，以及深度侵入计算机内存管理的复杂机制。为了在就地更新的前提下同时服务多个客户端，系统不得不引入极其庞大的并发协调开销（coordination overhead），这也正是导致传统数据库运行迟缓、扩展困难的罪魁祸首。 ### [16:28](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=988s) 解构数据库的三大核心原则与信息模型落地 面对传统数据库在扩展性、声明式表达、信息保真度等方面的重重阻碍，Rich Hickey 提出了自下而上重构数据库架构的指导原则。他特别重申了主题演讲中的核心论断：拥抱以“值”（values）为核心的不可变信息模型，并不仅仅是一种编程风格的偏好，它能够直接带来巨大的系统架构红利： > “采用基于值的模型，在架构层面能够带来实实在在的、非常巨大的好处。”（Rich Hickey: “adopting a value oriented model has architecture benefits really substantial ones.”） 为了将庞杂的单体数据库彻底拆解，Rich Hickey 归纳出以下几个核心解构支柱： 1. **转向信息模型（Move to an information model）**：底层数据模型必须全面转向以记录不可变事实（facts）为核心。 2. **拆分流转过程与感知认知（Split process and perception）**：将新状态的接纳流转（处理写与事务）与对已确立事实的观察感知（查询与读取）在架构上彻底剥离。 3. **将底层存储视作不可变存储（Treat storage immutably）**：数据一旦写入存储介质，就永不变更、永不覆写。 4. **内存化管理近期增量（Manage novelty in memory）**：为了应对持续发生的世界演进，在将数据刷盘变为长期不可变结构之前，必须在内存中维持一个用于吸收最新增量（novelty）的时间窗口。 在探讨如何构建事实模型时，Rich Hickey 强调必须把传统数据模型中过度封装的结构剥离出来。在关系型数据库的一整行（row）或文档数据库的一个文档（document）中，并没有纯粹的“事实粒度”。如果数据库里存储了一行关于某人的复合记录，当这个人更新了其中的电子邮件地址字段时，整行数据本身并不是单一事实，它其实是一组事实的粗暴聚合。只有下沉到最纯粹的单项事实（single facts），才能从根本上简化系统并释放极致的存储与检索效率。 在尝试确立原子事实的标准表达时，语义网领域的 RDF 规范曾提出基于“主体-谓词-客体”（Subject-Predicate-Object）的三元组（triples）设计，但 Rich Hickey 认为三元组仍有重大缺陷：由于缺少时间维度，它依旧无法完整表征事实的发生与更迭。为此，Datomic 团队将其扩展并定义为原子事实基元——**Datom**。之所以采用“Datom”这一特殊拼写，是为了在口语交流中与复数形式容易引起混淆的“datum / data”区分开来。一个 Datom 包含了四个核心要素：实体（Entity）、属性（Attribute）、值（Value）以及一个显式的不可逆时间要素（Temporal component），通过用事务元数据而非单一裸时间戳来绑定时序，为数据库构建起稳固的事实底座。 ### [20:23](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=1223s) 事务的一等公民属性与最小事实单元 在 Datomic 的设计中，事务本身是一等公民（First-class entity）。当一条数据作为事务的一部分进入系统时，系统不仅可以对事务进行排序，还能将时间戳直接附加到事务上。不仅如此，由于事务本身就是一个实体，还可以附带记录诸如“谁提交了该操作”、数据来源、是否通过审计等各类溯源（Provenance）信息与元数据。Rich Hickey 指出，在本次演讲中可以简化理解：事务中的关键维度 $t$ 就是时间，它是通往“何时发生”的路径，构成了系统中最微小的事实单元（Smallest fact）。 随后，Rich Hickey 抛出了数据库核心的“状态问题”（Database State Problem）：如果从根本上希望数据库是一个不可变的值（Immutable value），那该如何面对现实世界中不断涌现的“新事物”（Novelty）？业务在持续运转，企业会销售更多商品、引入新产品、增加新客户，这些新增信息必须有存放之处。这看似与“不可变值”的概念自相矛盾。为了化解这一冲突，Rich Hickey 提出了经典的“树轮类比”（Tree ring analogy）。 ```mermaid flowchart LR accTitle: Tree Ring Fact Accretion accDescr: 数据库通过事实累加向外生长，如同树木年轮，内层永不改变。 subgraph S1["旧有事实 (历史)"] direction TB R1["核心树轮: t0 事实"] R2["内层树轮: t1 事实"] end subgraph S2["新增事实 (新空间)"] direction TB R3["外层树轮: t2 事实断言 / 撤回"] end R1 --> R2 R2 --> R3 ``` 正如树木逐年生长一样，数据库可以被视作一个不断向外延展的值：它从不在原地（in place）更新，而仅仅向外增生，通过累积事实（Accretion of facts）来实现增长。系统绝不回头去改动内层年轮，只需在外层增加新的年轮。因为过去绝不改变，构建系统所依赖的底层核心也始终恒定不变，这完全契合了人们对“值”的期待——“凡是见过的，绝不改变”。Rich Hickey 强调，摆脱对物理存储“位置”（Places）的执念，转向为新事物分配新空间，能为架构带来极大的自由度。 - 事务具备丰富的一等公民属性：不仅包含时间戳（$t$），还能容纳来源、审计信息与行为人（Provenance）。 - 树木年轮类比：数据库通过事实累加（Accretion）向外扩张，永不在原处覆写已有历史。 - 核心设计哲学：“新事物意味着新空间”（New stuff means new space），以此彻底摆脱面向原地内存/磁盘位置更新的束缚。 --- ### [22:22](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=1342s) 具象化过程：断言与撤回 既然不再采用“更新某地址”或“按主键修改某记录”的原地更新语义，那么表达变化的最小粒度究竟是什么？Rich Hickey 指出，过程与新事物的最底层表示，可以精炼为事实的“断言”（Assertion）与“撤回”（Retraction）。当系统记录变化时，只需声明：“这个新事实成立”或者“之前成立的那个事实现在不再成立了”。即使某个事实被撤回，它“在过去某个时间段内曾经成立”这一事实依然永恒为真。 针对这一抽象，Rich Hickey 提出了具象化过程（Reifying process）的概念，并将其与事件溯源（Event Sourcing）的思想相联系。在传统的原地更新数据库中，系统运行一段时间后，如果想追问“系统是如何演变成当前状态的”，除非硬着头皮去解析底层的物理事务日志，否则毫无办法；而传统日志往往只是一串晦涩的物理位置修改指令，极难还原业务语境。而一旦将过程实体化为一个可存储、可读取的对象，历史变迁就一目了然：添加了新客户 Sally、撤回了旧邮箱、新增了新邮箱、售出了商品。Rich Hickey 强调： > “过程就是断言与撤回事实。”（“Process is just assertions and retractions of facts.”） 这种事实累积过程意味着数据库包含完整的历史，而不是一组彼此孤立的周二、周三快照切片。任何时刻观察数据库，内部都包裹着完整的过去。这对于现代信息系统至关重要，因为决策分析往往需要回答“在这段时间窗口内发生了多少变化”或“统计该期间内特定事件发生的频次”，这些必须在一个统一的时间感知模型中进行。 - 变更的最小表述：系统仅需两种基本原子操作——断言（Assert）与撤回（Retract）。 - 具象化（Reification）：将过程本身变成实体数据进行存储，摆脱黑盒式的物理日志回放。 - 全历史感知：数据库任何时刻的视图都自然内嵌所有历史，无需维护互相脱节的分立时间切片。 --- ### [25:54](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=1554s) 解构单体：分离“过程”与“感知” 在模型层面，Rich Hickey 将传统数据库服务器中混杂的索引、事务、查询以及磁盘 I/O 等能力进行了拆解，明确划分为两个完全独立的核心维度：过程（Process）与感知（Perception）。“过程”关注新事物的处理，包括事务排序、存储写入以及是否即时索引；而“感知”则是对数据库的理解与使用，包括提出查询、利用索引以及从存储读取数据。在传统架构中二者被死死绑定在一个进程或服务器内，但依托于不可变信息模型，这两者完全可以解耦。 ```mermaid flowchart TD accTitle: Process and Perception Separation accDescr: 将数据库的过程（写入与协调）与感知（查询与决策）解耦拆分。 subgraph PROC["过程 (Process)"] direction TB TX["事务协调 (最小协调)"] ST["不可变存储系统"] TX --> ST end subgraph PERC["感知 (Perception)"] direction TB APP1["应用节点 / 分析服务 1"] APP2["应用节点 / 分析服务 2"] APP3["应用节点 / 分析服务 3"] end APP1 -.->|"提交事务"| TX ST ==>|"读取不可变数据与索引"| APP1 ST ==>|"读取不可变数据与索引"| APP2 ST ==>|"读取不可变数据与索引"| APP3 ``` 基于这种解构，Datomic 构建了一种新的分布式协作模型：全面赋能应用端（如应用服务器、分析服务器）。应用端应当具备独立感知变更、本地记忆历史、自主计算并做出决策的能力，只在必须产生新事实时才与共享系统交互。在这个体系中，整个集群所依赖的共享中心资源被压缩至极简：仅剩负责写入裁决的事务协调者（Coordinator）和底层的共享存储。当业务对查询能力的需求爆发时，完全无需扩展脆弱的写中心，只需水平增加无状态的感知应用节点即可。 Rich Hickey 解释了这种设计的扩展性优势： > “如果我们想要更强的查询能力，我们需要做什么？只要增加更多的应用节点即可，我们完全不需要关心协调中心变大，因为我们根本没让它承担多少工作，仅仅是一点点协调而已。”（“If we want more query capability what do we need to do? Just add more of these guys. We don't really care about this growing because we're not asking it to do much, just some coordination.”） - 职责分离：将负责写入与状态推进的“过程”（Process）同负责读取与决策分析的“感知”（Perception）彻底拆分。 - 赋能应用端：应用节点直接承担感知与查询计算职责，不再向传统单体数据库倾倒查询负载。 - 极低协调开销：系统瓶颈不再压在单体数据库上，查询能力的水平扩展只需简单增加应用实例。 --- ### [28:13](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=1693s) 持久化数据结构与结构共享 从理论回到实现层面，如何使用具体的数据结构来表达这种“不可变且持续向外延展的树轮状态”？Rich Hickey 指出，任何底层结构都必须按某种方式组织排序以高效支撑查询。函数式编程早已为此提供了成熟的技术解法：持久化数据结构（Persistent Data Structures）。在此语境下，“Persistent”并非指磁盘上的持久化耐用性（Durable），而是指数据结构在发生逻辑修改后，依然能够完整保留旧版本并保持不可变性。 无论表示的是数组、映射（Map）还是有序集合（Sorted Set），系统都可以用一棵树来组织。持久化数据结构的核心机制在于“结构共享”（Structural Sharing）。当需要向树中某个叶子节点插入新数据以生成集合的新版本时，系统不会原地修改该节点，而是分配一个新的叶子节点，并沿着树向上复制该叶子到根节点的路径上的节点（Path copying）。 ```mermaid flowchart TD accTitle: Structural Sharing in Persistent Trees accDescr: 通过复制路径节点生成新版本，同时大部分旧节点保持共享。 subgraph Shared["未修改的共享子树"] N1["不变节点 A"] N2["不变节点 B"] end subgraph OldPath["旧版本路径 (保持不变)"] R_OLD["旧根节点"] --> P_OLD["旧路径中间节点"] P_OLD --> L_OLD["旧叶子节点"] P_OLD --> N1 R_OLD --> N2 end subgraph NewPath["新版本路径 (路径复制)"] R_NEW["新根节点"] --> P_NEW["新路径中间节点"] P_NEW --> L_NEW["新增叶子节点 (含新数据)"] P_NEW --> N1 R_NEW --> N2 end ``` 这种机制创造出了一棵代表最新状态的新树，而新树与旧树在内存中共享绝大部分未发生变化的子树分支。之所以能够安全地大规模共享结构，其底层唯一的前提就是不可变性（Immutability）。Rich Hickey 总结道，这种结构分享技术早已广泛应用于各类函数式语言的内存实现中；而 Datomic 所做的突破性一步，正是将这种纯内存的持久化数据结构移植到外部存储介质上，构建出真正具备磁盘持久耐用性的“耐用持久化数据结构”（Durable Persistent Data Structures）。 - 术语辨析：函数式编程中的“Persistent”（版本保留与不可变）不同于传统存储领域的“Durable”（落盘耐用）。 - 路径复制（Path copying）：生成新版本时只重新分配新增节点及其至根节点的一条路径，其余节点完全共享。 - 跨界移植：Datomic 将函数式编程内存中的结构共享机制，推广并实现到了磁盘存储层。 ### [30:14](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=1814s) 磁盘局部性已死：将 I/O 与磁盘封装为“存储”黑盒 Rich Hickey 回顾了传统数据库架构图：一台服务器塞满了 I/O 接口和物理磁盘。他直言不讳地表示自己“再也不想了解任何关于 I/O 和磁盘的事情了”。他提到最近看到的一篇论文，直截了当地指出“磁盘局部性（Disk Locality）已无足轻重”，这是一套正在走向消亡的陈旧观念。在过去，如果一台机器本地没有挂载磁盘，在计算上就会处于巨大劣势；因为带盘机器可以通过总线卡以极高的速度将数据加载进内存，拥有对磁盘的特权访问。如果另一台机器试图跨网络访问该磁盘上的数据，就要付出高昂的开销代价。然而，当今的网络速度已经大幅飙升，与此同时，磁盘与内存之间的性能鸿沟依然如天堑一般不可逾越。Rich 尖锐地总结道：“It ends up that anybody that needs to access the disk is losing, and the guy who has the disk in the same box is only losing very very slightly less than anybody else would lose so it doesn't matter anymore.” 既然只要碰磁盘大家都在输，那么那种紧密绑定磁盘的局部性就绝不该是现代系统架构的设计基石。系统真正应该关注的是把数据加载到内存中，并确保在内存层面拥有优良的局部性。因此，新架构直接将 I/O 和磁盘打包成一个黑盒，统称为“存储”（Storage）。这个存储服务只负责承载两件事：一是针对所有新产生事实（Novelty）的顺序追加日志（Append-only Log）——无论发生了一笔销售还是记录了一个邮箱，系统都像铲雪一样尽可能快地把这些事实塞入存储；二是类似于传统单体数据库放在磁盘上的 B 树，但在此处它们演变为位于存储中的持久化树（Persistent Trees），其树节点一旦写入便永久不可变。 - 对存储服务（Storage Service）的接口需求被精简到了极致，实际上只需要两个能力： - 一个简单的键值（Key-Value）接口：将索引树的某个代码块映射为一个不可变的数据段（Blob）。 - 少量可变存储空间以及一致性读（Consistent Read）：用于保存并更新指向整个数据库树根节点（Root）的指针。 - 架构层面不再关心底层存储的具体实现：无论是 Amazon DynamoDB，还是把关系型 SQL 数据库当作存放 Blob 的键值库来用，只要满足 Key-Value 存储与一致性读，就能直接作为底层存储。 ```mermaid flowchart LR accTitle: Storage Service Abstraction accDescr: 存储服务抽象图展示只追加日志与不可变树节点如何存放在统一的键值存储中 subgraph Inputs["写入与结构"] N["新增事实 (Novelty Log)"] T["不可变索引树节点 (Tree Segments)"] R["根节点指针 (DB Root)"] end subgraph Storage["存储黑盒 (Storage Service)"] KV["Key-Value 存储接口"] CR["一致性读接口 (Consistent Read)"] end N -->|"只追加写入"| KV T -->|"按 Blob 存储"| KV R -->|"指针更新与获取"| CR ``` ### [34:01](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=2041s) 细粒度索引树与列式存储的天然统一 Rich Hickey 解释说，所谓的索引结构在概念上非常简单：它就是一棵树，根节点指向内部节点，内部节点再指向叶子段（Leaf Segments）。这些叶子段内部只是一大块按照特定顺序排好序的事实基准数据（Datum）。数据的排序方式直接取决于业务访问模式：如果按照“实体-属性-值-事务”（Entity-Attribute-Value-Transaction，即 EAVT）中的“实体”（Entity）优先排序，它就能以极高的效率拉取看似对象或文档的数据视图；而如果把排序维度倒过来，以“属性”（Attribute）优先排序，比如把所有的电子邮箱地址存放在一起，把所有的电话号码聚合在一处，那么整个数据形态便瞬间具有了列式存储（Column Store）的特质。列式存储是做大数据分析（Analytics）的强力武器，而由于基本属性排序组合有限，数据库只需针对同一份不可变数据维护少数几种不同维度的树状副本即可。 当数据量极其庞大时，整棵索引树也会随之膨胀。因为树的节点完全不可变，任何新数据的引入都需要执行路径复制（Path Copying）。Rich 提醒台下，绝对不能每来一条新事实就重写一次树路径，那样做将是一场吞吐量的灾难。传统做法试图在存储层对索引做“实时排序维护”，但在不可变存储模型下这种开销是无法承受的。 - 实体优先排序（EAVT）：天然贴合文档和对象获取模式，便于单实体全属性聚合。 - 属性优先排序（AEVT 等）：具备列式存储特性，适合大规模分析与属性聚合查询。 - 节点不可变性：杜绝就地覆写（In-place Mutation），数据段在存储中天然具备幂等性与安全性。 ### [35:11](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=2111s) 内存聚合与动态合并：规避存储实时排序的代价 为了高效维护不可变索引，Rich 借鉴并改进了 Google BigTable 等系统的设计思路。在这些系统里，新来的数据已经先行记录在持久化日志中以保证持久性（Durability），因此将数据合入大型索引的频率完全与持久性脱钩。BigTable 的做法是在内存中不断累积变更，比如攒满 64MB 后直接刷出到磁盘生成扁平文件，之后由独立后台进程将新文件与之前全量有序的扁平文件执行归并排序（Merge Sort）生成新文件。虽然旧文件同样保持不变，但 Rich 指出 BigTable 模型的短板在于：每次生成的新扁平大文件与旧文件之间没有任何结构共享，文件过于庞大，既缺乏细粒度的寻址能力，也无法切成小块进行缓存。 Datomic 架构则将索引拆分成许多小尺寸的树状数据段（Segments）。这种设计不仅带来了细粒度的可寻址性，更带来了天然的“结构共享”（Structural Sharing）：当后台合并生成新版索引树时，它可以复用旧索引中的绝大多数子树节点；只要客户端缓存了这些小节点段，就永远不必担心它们会失效，因为它们在存储里是永久不变的。 平时系统在处理事务时，写入节点会把新事实一方面追加进持久化日志（确保哪怕宕机也能恢复），另一方面直接塞进内存中维护的有序集合——“实时索引”（Live Index）。由于这是纯内存操作，维护有序集合的代价极其微小。当系统需要查询当前世界最新状态时，只需在读取端做一次轻量的动态归并连接（Dynamic Merge Join），将内存中的实时索引与存储中的冷索引拼接在一起。等到特定时间间隔，后台合并进程将内存增量批量压入存储生成新树，所有人便能立即清空内存中对应的实时索引，周而复始。 ```mermaid flowchart TD accTitle: Index Merging and Lifecycle accDescr: 展示事实如何写入日志和实时内存索引，并定期与持久化存储合并 subgraph Ingestion["新事务写入 (Novelty)"] TX["写入事务"] Log["持久化日志 (Storage Log)"] Live["内存实时索引 (Live Index in Memory)"] end subgraph StorageTrees["存储索引树"] OldTree["存储中旧索引树"] NewTree["合并后新索引树 (结构共享)"] end TX -->|"立刻追加确保持久性"| Log TX -->|"轻量插入有序集"| Live Live -.->|"定期批量合并"| Merger["后台合并进程"] OldTree -.-> Merger Merger -->|"生成并共享子树"| NewTree ``` ### [38:36](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=2316s) 读事务的消亡与系统新角色：Transactor 与 Peer 在展示了感知端（Perception）如何查看数据库之后，Rich 指出了一个极为震撼却常被忽略的变化：无论是要执行查询、执行分析统计，还是单纯读取某个实体，读取方只需要两样东西——访问内存中的实时索引，以及访问底层键值存储。感知者完全不需要靠近事务处理系统，甚至不需要与事务协调系统发生任何交互。Rich 激动地向观众提问：在这个过程中，传统的“读事务”（Read Transactions）究竟去哪儿了？答案是它们彻底消失了。Rich 强调了这一核心设计理念：“They just disappeared, because a correct implementation of perception does not require coordination. Perception in the real world does not require coordination.” 在现实物理世界中，人看一眼眼前的物体从来不需要和周围所有人发起分布式两阶段提交；同理，在一个由不可变值构成的数据库系统中，纯粹的感知与读取本就不该背负协调的锁与代价。顺着这个拆解逻辑，Rich 正式定义了新架构中的几个核心角色： - **存储服务器（Storage Servers）**：底层的冗余存储服务，通常采用现代具备分布式冗余能力的成熟键值存储系统。 - **事务处理者（Transactor）**：专门负责接收新事实、写入持久化日志、处理冲突并推进全局版本更新的独立协调进程。 - **对等体（Peer）**：运行在用户自己的应用程序服务器进程内部。每个 Peer 都是极其强大的平等节点，完整具备“感知、记忆、决策、反应”的计算能力，本地拥有缓存，能够自由进行查询与计算，彼此之间以及与 Transactor 之间在查询时完全零协调开销。 ### [40:28](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=2428s) 架构解耦：Transactor、存储服务与 Peer 节点 Rich Hickey 解释道，只要不把底层存储系统当成传统数据库对待，而是作为单纯的 Key-Value 存储，它们就具备极其惊人的工程优势：“它们具有冗余性、高可靠性、高可用性、可扩展且天然分布式，那里蕴含着巨大的能力，我们理应充分利用它。”随后，他展示了将传统数据库各项职能彻底打散、重新编排后的架构图。在目前的实现中，事务处理器（Transactor）凭借自身充裕的计算周期和实现的便利性，会兼任索引构建的工作；但如果需要，完全可以将索引任务剥离到一台独立的机器上运行。 整个写入与同步流程清晰明了：当应用产生“新事实”（novelty）时，它会与 Transactor 通信；Transactor 接收后立刻将其写入日志（log），装入自身的内存实时索引（live index），同时将该新增事实重新广播（rebroadcast）给集群中的所有对等节点（Peers）。这样一来，每个 Peer 节点都能即时同步维护自己本地的实时索引。 底层存储服务（Storage service）在整个架构中完全被当作黑盒处理。它只需要是一个能够支持一致性读取（consistent read）的键值存储即可。当前该架构支持 Amazon DynamoDB、PostgreSQL 等任意关系型数据库，甚至支持 Infinispan 这类大型内存网格（memory grid）。而在 Peer 节点一侧，不仅具备与 Transactor 通信的组件，本地内存中驻留着实时索引，还拥有直接与底层存储通信的能力。Rich Hickey 指出，这种设计打破了传统数据库的“数据局部性”假设：没有任何计算节点比其他节点更贴近存储，Transactor 离底层存储的距离并不会比 Peer 更近；既然存储本身就是一个独立的远端服务，这就意味着集群中的每个节点都可以直接独立读取存储。 ```mermaid flowchart TD accTitle: Datomic 整体读写解耦架构 accDescr: 展示应用 Peer、Transactor 与底层 Key-Value 存储之间的写入、广播与查询交互链路 subgraph WritePath["写入路径"] APP["应用 / Peer 节点"] TX["Transactor (事务处理器)"] LOG["写入持久化 Log"] end subgraph StorageLayer["底层存储黑盒 (Key-Value)"] STORE["存储服务\n(DynamoDB / Postgres / Infinispan)"] INDEX_TREE["持久化索引树 (不可变数据块)"] end subgraph ReadPath["读取与查询路径"] PEER["Peer 节点 (包含查询引擎)"] L_INDEX["本地内存实时索引\n(Live Index)"] CACHE["本地缓存 / 共享 memcached"] end APP -->|"提交 novelty"| TX TX --> LOG LOG --> STORE TX -->|"实时广播 novelty"| L_INDEX PEER -->|"合并查询"| L_INDEX PEER -->|"查询未命中时读取"| CACHE CACHE -.->|"缓存缺失"| STORE STORE -.-> INDEX_TREE ``` - 事务处理器（Transactor）负责接收应用变更并记录 Log，同时负责向外广播增量事实。 - 架构中仅存在单个负责串行事务协调的 Transactor，避免了分布式冲突处理的复杂度。 - 存储层作为黑盒，只需保证键值读取的一致性，已支持 DynamoDB、关系型数据库（如 PostgreSQL）及 Infinispan 内存网格。 --- ### [43:21](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=2601s) 查询下放、不可变性与“无痛缓存” 因为所有节点都可以直接访问存储和索引，这彻底改变了查询引擎的归宿。Rich Hickey 提问：“查询引擎真正需要的是什么？它需要访问组织好的数据，即访问索引。”由于任何节点都能拿到索引，查询逻辑便可以放置在系统的任何地方，完全不需要昂贵的专用查询服务器。在 Datomic 中，查询引擎被做成了一个随宿主 Java/JVM 应用一同运行的库文件，内嵌查询能力与实时索引。 针对存储位于远端网络、必须付出网络跳数开销的质疑，Rich Hickey 提出了核心应对方案：缓存。他向观众抛出问题：“从这个存储中，缓存什么是安全的？”当台下试探性地给出回答时，他直截了当地揭示了不可变架构的威力：“一切内容都是安全的。为什么？因为它不可变。它什么时候过期？永远不过期。作为架构师，这正是你最梦寐以求的答案——这太美妙了。我们可以在任何地方毫无顾忌地进行激进缓存。”开发者不仅可以在 Peer 本地设置堆内缓存，还能轻松架设一套被多个 Peer 共享的 memcached 集群。 尤为关键的是，这种缓存完全发生在系统底层（caching under / caching inside），而不是应用层的负担。在传统系统中，开发者必须在业务代码里手写大量的读取判断、缓存穿透防护和失效淘汰逻辑；而在 Datomic 中，应用发出查询指令后，如果本地缓存命中则直接使用，未命中则自动从共享缓存或底层存储拉取。整个过程纯属机械运作，没有任何业务特例，开发者只需启动 memcached 并将其地址告诉系统即可。面对现场关于“是否需要全量复制数据”的提问，Rich Hickey 澄清这绝非全量同步，而是完全按需驱动（on-demand）的 Working Set 缓存：一个用于价格分析的应用只会加载并缓存它关心的分析切片；而一个负责展示商品详情的 Web 节点则按需加载商品数据。得益于浅层索引树设计（整棵树至多仅有三层），一旦节点缓存了树的顶层结构，获取任何从未见过的全新数据，都只需进行一次精准的读取跳转。 - 节点无需全量复制数据，仅按各自的运行负载（Working Set）按需加载与填充缓存。 - 浅层树状索引使得顶层节点极易被长久保留在缓存中，穿透查找深层叶子节点通常只需一次存储读取或 memcached 读取。 - 缓存失效难题被彻底消解：“因为它不可变。它什么时候过期？永远不过期。”Rich Hickey 称这正是系统架构设计中最为省心的特质。 --- ### [47:30](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=2850s) 增量数据与持久化索引的合并查询机制 为了让观众彻底看清读取路径与写入路径的交汇细节，Rich Hickey 详细拆解了系统如何在节点本地完成“时间差”查询。存储中的索引可以理解为一颗根据历史日志构建并排序的不可变树，而日志则是纯粹的追加写流（Append-only）。假设系统最后一次构建全局索引是在一小时之前，那么在过去这一个小时内发生的所有新增事实，都仅追加在了 Log 中，尚未刷入持久化索引树，但它们已被 Transactor 广播到了每个 Peer 的本地内存实时索引中。 假设此时客户端要求渲染用户 Sally 的个人主页，而 Sally 恰好在过去一小时内修改了自己的电子邮箱。面对这个场景，查询引擎执行的是一次本地的合并连接（Merge Join）：一边在本地内存实时索引中定位 Sally（获取这过去一小时内的新事实与变更），另一边在持久化索引树中定位 Sally（获取一小时以前的所有历史事实），然后将两者归并输出。如果这一个小时内 Sally 执行了“撤销/修改”（retraction）操作，合并逻辑会使得旧事实被掩盖。查询引擎从两份数据源的叠加中，完美且即时地拼凑出 Sally“当前时刻”的完整状态。 当增量变更积累到一定阈值（例如满一个小时），Transactor 或索引构建任务便会生成一颗融入了最新数据的新索引树，该树在物理上与旧树共享大部分未修改的数据块。新树构建完毕并持久化后，系统会通知所有 Peer 节点，各节点随之清空自己的这一阶段的本地增量日志窗口，重新开始记录下一个周期的变更。针对现场观众对“全量广播是否会导致千万节点性能崩溃”的担忧，Rich Hickey 解释道，Peer 节点在角色定位上是应用服务器（App Server）而非直接面向终端访客的 Web 前端连接器，集群规模通常不会达到上万个；当前系统虽未引入多播（Multicast）网络基础设施，但即使面对海量分发，多播也是现成的成熟演进路径。 ```mermaid flowchart LR accTitle: Peer 节点本地合并查询机制 accDescr: 展示查询引擎如何将内存中的最新增量事实与底层持久化历史索引合并为一致的最新视图 subgraph Memory["Peer 本地内存"] LIVE["实时索引 (Live Index)\n最近 1 小时内的新事实/撤销\n例如：Sally 修改后的新邮箱"] end subgraph StorageIndex["持久化索引 (不可变树)"] PERSIST["历史索引树\n1 小时前的所有基线数据\n例如：Sally 的旧档案数据"] end ENGINE["Peer 本地查询引擎\n执行 Merge Join"] RESULT["当前时刻完整视图 (Sally 的最新主页)"] LIVE -->|"Seek to Sally"| ENGINE PERSIST -->|"Seek to Sally"| ENGINE ENGINE -->|"合并增量与基线\n应用 Retraction"| RESULT ``` > “当我想去查询并要求‘告诉我你所知道的关于 Sally 的一切，因为我要展示她的资料主页’时，查询引擎会执行一次合并连接：先在这里定位 Sally，再在持久化索引里定位 Sally，然后将它们合并在一起。” —— Rich Hickey --- ### [50:43](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=3043s) 超越简单事实：事务函数的引入 在梳理完断言（Assertion）与撤销（Retraction）的机制后，Rich Hickey 将讨论引向更为复杂的业务场景。尽管纯粹的数据断言和撤销足以表达状态的变迁，但在现实业务中，许多数据转换逻辑无法以孤立的新事实直接表达。例如“向银行账户里充值 10 美元”这一操作，它所产生的最终事实（即充值后的总余额），严格依赖于该账户在事务发生那一瞬间的现有数值。 为了在无锁、解耦的分布式环境中正确处理这类依赖当前状态的逻辑转换，系统必须引入一种机制，将计算逻辑紧密贴合到状态提交的关口。Rich Hickey 指出，解决这一难题的答案就是“事务函数”（Transaction Functions）——它是一种接受当前数据库状态作为参数并安全产出新事实的纯函数。 > “在逻辑处理上我想做的是‘给你的银行账户增加 10 美元’，但这一过程最终产生的新事实，完全取决于你账户里原有的余额数值。” —— Rich Hickey ### [51:15](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=3075s) 事务函数与事务数据的多级展开机制 Rich Hickey 深入讲解了事务数据的构成与执行原理。在此之前，人们通常认为事务数据仅仅由断言（assertion）与撤销（retraction）组成；但在 Datomic 中，事务数据不仅包括事实的断言与撤销，还包含了**事务函数（transaction functions）及其参数**。这意味着开发者可以编写一个事务函数并传入参数，由系统在事务内部调用该函数并注入数据库的当前值。由于数据库本身就是一个纯值（value），因此这里的事务函数是针对“真实值”调用的纯函数。该函数在事务内部可以执行任意查询——比如查询 Sally 当前的银行账户余额——然后返回更多的事务数据。举例来说，若当前余额为 100 美元，函数计算出新余额为 110 美元后，就会展开为一条具体事实断言：“Sally 的新余额是 110 美元”。 事务函数的展开过程具有递归性：任何事务都是由断言、撤销和潜在的事务函数调用组成的。事务函数被调用后，可以进一步展开为对其他事务函数的调用，但最终都会层层展开为基础的事实断言与撤销。这一展开过程会反复进行，直到只剩下断言和撤销为止。这样一来，开发者便可以在数据库的数据上施加任意的业务转换和查询逻辑。更重要的是，Datomic 赋予了开发者架构上的独立决策权： - 如果写入的是“纯粹的新增事实”（pure novelty），开发者完全无需将其置于事务函数内部，直接提交新事实即可。 - 如果预计并发冲突概率极低，开发者可以采取乐观策略（optimistic）：基于本地视角的余额（100 美元）计算出新余额（110 美元），并附加一个先决条件“若当前余额不是 100 则失败”。对于开销高昂且冲突概率低的计算，这种在客户端完成计算的方式往往效率更高。 - 如果需要绝对无干扰的原子保证，则采用“传统老派做法”：将函数发送到事务内部（Transactor），让其在事务内以原子方式执行。 ### [53:35](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=3215s) Transactor 运作流水线与 Peer 节点双层缓存 Rich Hickey 梳理了 Transactor 的端到端执行流水线以及 Peer 节点的存储架构。Transactor 的职责高度聚焦：它接收由断言、撤销和事务函数组成的数据，对函数进行展开，将其应用到 Transactor 内部的内存数据库视图中，写入日志（log），向提交方确认完成（acknowledge），最后将变更广播给所有订阅者（Peer）。在后台，系统会周期性地生成新索引。由于采用了不可变树状结构，每当生成新索引时，旧索引的根节点与许多分支节点就变成了废弃数据；因此，类似垃圾回收（GC）机制的存储清理工具会跟进清理这些垃圾存储块。 在读取端，Peer 服务拥有直接访问底层存储的权限，内置独立的查询引擎和实时的内存索引，并能自行在本地执行内存与存储的合并操作。在 Peer 的查询引擎之下，设计了精密的两层缓存机制：当发起查询且数据未在本地命中时，系统会优先查找本地的对象级缓存；如果对象缓存未命中（Miss），若配置了 Memcached，则前往 Memcached 读取数据段（segments）；若仍未命中，最后才回退到底层存储并随后缓存在上层。 ```mermaid flowchart TD accTitle: Transactor Pipeline and Peer Caching Architecture accDescr: Pipeline of transaction expansion and commit alongside Peer node caching tiers. subgraph TransactorFlow["Transactor 事务提交流水线"] T1["接收事务数据\n(断言 / 撤销 / 函数)"] --> T2["递归展开事务函数\n直至全为基础事实"] T2 --> T3["应用到内存视图\n并写入日志 (Log)"] T3 --> T4["向客户端确认 (Ack)\n广播新事实到各 Peer"] end subgraph PeerRead["Peer 节点读取与两层缓存"] P1["Peer 独立查询引擎"] --> P2{"本地对象级缓存\n是否命中?"} P2 -->|"是"| P3["直接合并内存与索引返回"] P2 -->|"否"| P4{"Memcached 数据段缓存\n是否命中?"} P4 -->|"是"| P3 P4 -->|"否"| P5["回退读取持久化存储\n(并填充两级缓存)"] end T4 -.->|"广播新状态"| P1 ``` 在系统内部，存在着作为实时索引的内存持久化数据结构（Persistent Data Structure），它在内存中本身就是不可变的，系统只是从一个不可变版本切换到下一个不可变版本。整个系统中**唯一可变的东西（the only mutable thing）**，就是包裹着指向当前数据库值结构体指针的“盒子”（Box/Identity）。每当内存索引更新或索引任务完成，盒子内的指针就会从一个不可变结构原子地指向另一个。除此以外，系统内再无任何可变数据。正如 Rich Hickey 所言： > “The only mutable thing in the system is the fact that the contents of this box will move from one of these immutable structures to another whenever we've updated the memory index or whenever an indexing job is completed.” 这意味着，一旦某个线程获取了当前数据库值的指针并开启了一个耗时极长的查询，哪怕同进程内的其他线程正在并发执行更新，该查询所持有的底层数据也绝不会被篡改。 ### [56:19](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=3379s) 读写分离带来的可扩展性决策与弹性 Peer 查询 针对此前关于是否使用 Lamport 逻辑时钟或向量时钟（Vector Clocks）的提问，Rich Hickey 阐明了将“感知”（perception）与“处理/进程”（process）分离所带来的本质优势：系统可以针对可用性（availability）与可扩展性（scalability）独立做出技术选型。在事务处理侧，Datomic 选择了贴合传统商业市场需求的单写入者（single writer）强一致性模型；而在 Transactor 身上，除了处理写入和事务协调外，没有任何其他的查询开销，因而能将单写性能发挥到极致。Rich Hickey 犀利地指出，许多 NoSQL 数据库强制开发者用放弃事务和丰富查询来换取任意写入扩展，这种权衡对许多企业而言并不明智： > “If you want arbitrary write scalability, you're going to have to give up transactions and queries and everybody who's adopting NoSQL databases that make that choice for you is getting that trade-off, and I think it's not a great trade-off for a lot of companies.” 而在读取和查询侧，Datomic 则充分享受了分布式架构的红利： - 底层存储插件化：可以将 DynamoDB 放在底层存储槽位中，利用 AWS 强大的存储冗余与读写吞吐旋钮；也可以直接构建在现有的 SQL 数据库之上。 - 查询弹性扩缩容（Elastic Query Scaling）：查询执行被下放到了每一个 Peer 节点，而不是集中在单台数据库主节点上。增加查询能力不再像维护复杂数据库集群配置那样沉重，而是可以直接配合 AWS Auto Scaling：随着业务负载增加动态拉起 Peer 实例，查询吞吐随之线性增加；负载下降则自动缩容。 ### [58:37](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=3517s) 极简属性模式与免受时间参数污染的“As Of”查询 在数据模型上，Datomic 采用了极简的 Schema 设计。系统中最基础的单元是“事实”（Fact），而围绕事实唯一需要配置的只有**属性（Attribute）**的定义，例如属性名称、值类型以及基数（Cardinality，单值或多值）。不存在比属性更高层级的表或实体 Schema 抽象。在应用层使用时，这种模型让开发体验极其接近于操作嵌套的多重映射（Multimaps）——每个实体都表现为一个由键值对组成的集合，每个键的值天然支持多值化，既灵活又直观。 不仅如此，由于数据库是一个包含了全部历史的不可变纯值，系统天然具备了时间维度的查询能力。开发者可以对当前持有的数据库说一句：“假装你是上个月的数据库”（pretend it's last month）。这一操作不需要复制、修改或丢弃任何数据，底层查询在遍历时只要发现晚于该时间点的数据直接忽略即可。这意味着针对历史时间点的“As Of”查询几乎是零开销的。 Rich Hickey 询问现场听众有多少人曾为了支持时间审计，不得不自己在业务系统里维护时间戳。他指出传统方式的实现极为痛苦： > “It's brutal right because what do you have to do you have to flow around that time right if I want to say as of last week last week has to be part of every query has to be part of every join and it's nasty right you have to do Max and all that very very tough stuff.” 在传统关系模型中，时间戳必须强行注入到业务的每一个 JOIN 条件与聚合子查询（如求 MAX）中，不仅污染了查询逻辑，极易出错，而且让系统维护变成噩梦。而在 Datomic 中，开发者无需修改 SQL 或查询模板：同一个计算销售总额的纯业务查询，既可以直接运行在“当前”的数据库值上；也可以向系统获取一个“上周”的数据库值，并将完全相同的查询直接施加在该值上。查询本身完全解耦了时间参数，时间不再出现在 JOIN 逻辑中。此外，系统还天然支持跨时间段比较以及滑动时间窗口（windows since a point in time）的分析查询。 ### [01:01:06](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=3666s) 进程内的推演能力：“假设分析”（What-If） 当数据库被视作不可变的值并且查询运行在进程本地时，开发者便获得了一种全新的能力：就地进行推演与假设分析。Rich Hickey 指出，你完全可以在本地内存中提出各种“如果……会怎样”（What-if）的问题，例如“如果我加入了这些数据会怎样？”或“如果我提交了这个事务，数据库会变成什么样？某个查询的结果又会是什么？”为了验证这些猜想，客户端根本不需要向服务器发起请求。因为本地所持有的内存数据结构本身就是持久化数据结构（Persistent Data Structure），你可以直接基于当前数据派生出一个包含了拟提交事务数据的新版本。 这个新版本同样是一个货真价实的数据库值（Database Value），虽然新增的数据完全存在于本地内存中，但它依然能无缝结合底层存储系统中的既有数据。这种设计让应用可以在没有任何外部依赖、完全不打扰任何人的情况下执行查询并验证逻辑。Rich Hickey 强调了这种工作流的自由度：“I don't have to put that into the database to ask answer the question I still have query capability.”开发者既可以利用这种能力在真正提交事务前验证查询是否正常工作，也可以将其纯粹用于决策支持的推演分析，即使这些临时数据永远不写入集中式存储，也不会丧失任何强大的查询能力。 ```mermaid flowchart LR accTitle: 本地推演工作流架构图 accDescr: 展示如何在本地持久化数据结构上叠加未提交事务数据生成新的数据库值并执行查询 subgraph StorageLayer["底层存储"] A["持久化数据快照 (Storage)"] end subgraph LocalPeer["Peer 进程本地内存"] B["当前数据库值 (In-Memory DB Value)"] C["拟提交的变更数据 (Novelty / Tx Data)"] D["推演数据库值 (What-If DB Value)"] E["本地查询引擎 (Query Engine)"] end A -.-> B B --> D C --> D D --> E E --> F["即时推演分析结果"] ``` --- ### [01:02:01](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=3721s) 感知与响应：基于事件流与数据库值的反应模型 当谈到系统的“感知与响应”（Perception and Reaction）时，Rich Hickey 表示对系统状态的感知已经非常直观，因为不可变的数据结构使得所有查询本质上都是某种形式的感知。而在这个新架构下，对系统变化的“响应”也变得极其简单。负责写入的协调节点（Transactor）会将所有新增的变化数据（Novelty）实时广播推送给所有的查询对等端（Peers），这意味着在 Peer 节点上捕获并生成“有新数据到达”的事件通知变得轻而易举。 更为优雅的一点在于，事件推送不仅包含变动本身，还直接附带了完整的上下文状态。Rich Hickey 举例说，假设系统中发生了一个事件：“Sally 修改了她的电子邮件地址”。在传统架构中，如果你想知道这个新地址是否与历史地址相同，往往需要大费周章地排查和比对。但在将数据库视为值的架构中，伴随着事件通知一起发送给 Peer 的，包括了变更发生前的数据库值、变更本身的数据，以及变更发生后的数据库值。 - **三位一体的上下文**：每次数据变动事件都同时提供变更前数据库（Before）、变更数据本身（Tx Data）以及变更后数据库（After），支持在任意视角执行查询。 - **全内存查询引擎的灵活性**：Peer 内置的查询引擎不仅能够针对持久化数据库运行，还能直接对内存数据结构以及二者的混合体执行查询，开发者只需编写普通查询语句即可完成高精度的事件过滤。 --- ### [01:03:16](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=3796s) 纪元状态模型与可重现的“时间旅行”调试 Rich Hickey 总结道，将系统彻底解耦带来的最核心收益就是数据库状态模型的极度简化。他将这种模型定义为“纪元式”（Epochal）状态模型：数据库的状态永远只从一个有效、一致的点跳转到下一个有效、一致的点，绝不存在任何中间状态，也不需要任何读取协调机制，所有的协调开销仅仅存在于 Transactor 处理写入过程的那一瞬间。因此，每次只要在同一个数据库基础上执行相同的查询，得到的必然是完全一致的结果，为系统决策提供了坚如磐石的稳定基准。 谈及排查生产 Bug 的痛苦经历，Rich Hickey 幽默地唤起了全场共鸣：“在下午五点左右，查询突然开始返回某些诡异的结果；而这时候每个人都想下班回家，所以大家商量‘我们明天再看吧’。结果第二天回来一查，查询又完全恢复正常了，仅仅是因为系统里后来写入了更多的新数据。那你到底该怎么查出昨晚出了什么错？你彻底完蛋了。”然而在纪元式模型下，时间旅行让问题变得极为简单，开发者只需要要求系统“以昨天下午五点的状态来运行该查询”，就能精准重现当时的诡异结果并快速定位根因。此外，这一机制也极大方便了节点间的协作，上游系统向工作人员或下游服务分派任务时，不再只是扔出一句模糊的“去数据库里看看”，而是可以精确指定在“数据库逻辑时间 $T$”的基础上查看当时的数据视图。 > “Every time you ask the same query you get the same results you have a stable basis for decision making.” —— Rich Hickey --- ### [01:04:45](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=3885s) 架构解构的收益：物理自由度、极简性与横向扩展 在架构的物理部署层面，将数据库拆解为独立构件带来了极大的重定位自由度。系统不再硬性绑定查询在哪里执行、存储位于何处或协调器在哪里运行。开发者可以自由组合基础设施，例如把持久化数据存放在云端的 DynamoDB 上，将整个计算与查询系统运行在私有局域网（LAN）内，并在本地局域网部署 Memcached 缓存层，从而完全屏蔽可能由公网延迟带来的影响。 Rich Hickey 总结了将数据库解构、将数据库视为值所带来的最终成果。这种做法赋予了系统真正的“信息模型”（Information Model），大幅消除了不必要的复杂性： 1. **更易理解与演进**：各个组件职责单一，可以轻松替换存储后端或迁移物理组件。 2. **读写分离与高扩展性**：读取操作不存在跨节点协调，系统若需要更强的计算能力与思考大脑，只需启动更多的 Peer 进程即可近乎线性地横向扩容。 3. **保留全量信息**：信息模型完整记录了发生过的一切事实，不仅满足了商业机构需要回溯历史进行决策的刚需，还提供了极高的数据塑形灵活性，能够根据业务需要轻松投影成任意所需形态。 > “The net result of deconstructing the database is that you're able to treat the database as a value... you have a system that's substantially less complex.” —— Rich Hickey ## Notable quotes > [16:28](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=988s) adopting a value oriented model has architecture benefits really substantial ones. — Rich Hickey > [22:22](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=1342s) Process is just assertions and retractions of facts. — Rich Hickey > [01:03:16](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=3796s) Every time you ask the same query you get the same results you have a stable basis for decision making. — Rich Hickey > [01:04:45](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=3885s) The net result of deconstructing the database is that you're able to treat the database as a value... you have a system that's substantially less complex. — Rich Hickey > [56:19](https://www.youtube.com/watch?v=Cym4TZwTCNU&t=3379s) If you want arbitrary write scalability, you're going to have to give up transactions and queries and everybody who's adopting NoSQL databases that make that choice for you is getting that trade-off, and I think it's not a great trade-off for a lot of companies. — Rich Hickey ## Mentioned Rich Hickey, Datomic, Clojure, Out of the Tar Pit, SQL, Amazon Dynamo, Amazon DynamoDB, Google Bigtable, RDF, PostgreSQL, Infinispan, Memcached, AWS, AWS Auto Scaling ## Video description Rich Hickey, author of Clojure, and designer of Datomic presents a new way to look at database architectures in this talk from JaxConf 2012. ** For more presentations from JaxConf 2012, head to http://mrkn.co/txtch [Source](https://www.youtube.com/watch?v=Cym4TZwTCNU)

[16:28] Rich Hickey: adopting a value oriented model has architecture benefits really substantial ones. [22:22] Rich Hickey: Process is just assertions and retractions of facts. [63:16] Rich Hickey: Every time you ask the same query you get the same results you have a stable basis for decision making. [64:45] Rich Hickey: The net result of deconstructing the database is that you're able to treat the database as a value... you have a system that's substantially less complex. [56:19] Rich Hickey: If you want arbitrary write scalability, you're going to have to give up transactions and queries and everybody who's adopting NoSQL databases that make that choice for you is getting that trade-off, and I think it's not a great trade-off for a lot of companies.

[43:21] Rich Hickey: What's safe to cache? Everything. Why? It's immutable. When does it expire? Never. These are the things you want to hear as an architect, right? [56:19] Rich Hickey: If you want arbitrary write scalability you're going to have to give up transactions and queries and everybody who's adopting NoSQL databases that make that choice for you is getting that trade-off and I think it's not a great trade-off for a lot of companies. [35:49] Rich Hickey: By using trees you get fine-grained addressability and you get these nice small chunks which are good for caching. You also have the potential for structural sharing.

[0:01] [Music] [0:11] talk is deconstructing the database and [0:15] uh what it is is it's it's a look at uh [0:17] a new way to look at database [0:19] architectures um it it happens to also [0:22] be a look at um the underpinnings of the [0:25] architecture of datomic which is the [0:27] database I've been working on for the [0:28] last two years um um but it's not a [0:30] sales pitch for datomic it's really [0:32] about the ideas underly the design [0:36] choices so why do we want to deconstruct [0:38] the database what are we trying to [0:39] accomplish what problem are we trying to [0:41] solve I think the fundamental problem [0:43] we're trying to solve is uh the problem [0:47] of [0:48] complexity in programming I mean how [0:50] many people think dealing with databases [0:52] is easy and trouble-free [0:56] nice uh most people don't um and there's [1:00] number of sources of complexity uh [1:02] there's this great paper out of the [1:04] tarpit U and in it the author sort of [1:07] identified a bunch of problems uh [1:09] related to complexity in programming and [1:11] and they basically said that all [1:13] complexity boils down to you know two [1:15] two flavors one of uh one has to do with [1:18] State and the other has to do with [1:20] control and uh the authors didn't [1:23] implement it but they suggested that um [1:27] by adopting functional programming and [1:29] declarative program programming um and a [1:32] relational model for data inside our [1:35] applications we could get rid of get rid [1:37] of this complexity uh but the and it's a [1:41] great paper I really recommend you read [1:42] it but one of the problems with the [1:43] paper is that um while they had a good [1:46] grip on the functional programming and [1:48] declarative programming part and also [1:51] possibly on using a a relational model [1:55] for data inside your [1:57] applications um they really sort of [2:00] punted everybody seen the cartoon where [2:02] the mathematician has this chalkboard [2:04] and it's full of full of stuff and then [2:06] in the bottom corner it says and then a [2:08] miracle occurs right and then there's [2:10] the answer and the big thing that was [2:12] missing from their picture of the world [2:13] was they imagined there would be this [2:15] relational model of your data that you [2:17] could access in your application and [2:19] that somehow it got [2:21] updated like somehow there something [2:23] happened in the world and it was [2:25] different uh and and that all the all [2:27] the ick related to state had to do with [2:30] with however that got updated but they [2:31] didn't say how it would and I I would [2:34] call that updating uh and the problem [2:37] that they avoided talking about really [2:39] the problem of process in our programs [2:41] in other words we know there's going to [2:43] be novelty in the world there are [2:45] programs are going to encounter um where [2:46] does that go in a model that's otherwise [2:49] functional right or as functional as we [2:52] could make [2:53] it so some of the things some of the [2:55] other problems we're trying to solve are [2:56] things we want to obtain in in re in [2:59] looking at the database in a fresh way [3:02] uh I think is to embrace declarative [3:04] programming I agree with the paper we [3:06] want declarative programming what's the [3:08] best example of declarative programming [3:09] we encounter most [3:11] often if we're not artificial [3:13] intelligence [3:16] researchers SQL actually is the most [3:20] declarative thing most of us encounter [3:21] on a on a day-to-day basis and and and [3:24] it ends up that declarative programming [3:27] uh is much better at manipulating data [3:29] than what we do in our languages even [3:31] functional languages right in our [3:33] languages we sort of go through stuff [3:34] but we do not have this very nice higher [3:37] order set Logic for dealing with data [3:39] and it is superior to dealing with data [3:42] even in a functional language and way [3:44] Superior than dealing with data in an [3:46] object-oriented language we use [3:48] object-oriented languages because we [3:49] have them not because they're better at [3:51] this they really are much worse at this [3:54] um one of the problems we have is with a [3:56] client server database that declarative [3:58] programming is something that's alien to [4:00] us it's sort of over there um the other [4:03] problem related to the model these guys [4:04] were espousing is that there's no basis [4:07] right um that you could rely on if I [4:09] want to do a I want to calculate [4:12] something related to a database what's [4:13] the basis for that calculation if the [4:16] whole database is sort of changing all [4:17] the time we're back to the problems I [4:19] talked about in the keynote right the [4:20] the database in its entirety is a place [4:23] and we have a problem of saying what's [4:25] the basis for our decision- making well [4:26] I don't know it was whatever I saw last [4:28] Tuesday when I ran this computation but [4:30] I can't can't tell you now what that was [4:33] exactly there are problems with [4:35] databases related simply to the client [4:36] server nature about them being over [4:39] there the basis problem is one of them [4:41] um the other is that our fear of round [4:44] trips um and we're we're fear we're [4:46] afraid of round trips I think most often [4:48] for performance reasons but actually the [4:50] biggest roundt trip problem is that same [4:52] basis problem right what if I have a [4:55] composite decision to make can I ask the [4:57] database three independent questions [4:59] over time and then make get my [5:02] answer no why not because stuff has [5:05] happened to that place in between those [5:07] calls and it makes us do weird stuff in [5:10] particular one of the things I think it [5:12] makes us do is couple questions with [5:14] reporting right so let's say you have [5:17] your application and your application [5:18] makes a decision about what entities [5:20] we're going to put on sale or display on [5:22] this web page um do we ever you know do [5:25] we send the query to find out what those [5:26] entities are and then later send a query [5:28] to you know gather the data we need for [5:31] display no a lot of times we piggyback [5:33] those two things together because we're [5:34] afraid of the result sets not matching [5:37] up anymore that's actually a fear that's [5:39] born of the lack of basis again and it's [5:42] the biggest problem with round trips U [5:44] and of course from a design perspective [5:46] we know those two pieces of logic should [5:47] be separate they should be independent [5:49] decisions one part of my app has knows [5:51] the logic for deciding what should be [5:53] displayed and the other one knows about [5:55] what what does the screen look like and [5:57] what you know what do we want to show on [5:58] it [6:00] uh we have problems related to [6:01] consistency and scale I don't know if [6:03] anybody saw any of the nosql talks uh [6:06] this week but you know a lot of times we [6:08] have difficulty scaling servers that are [6:12] monolithic by default and uh you know [6:15] we've seen no SQL we've seen the Dynamo [6:17] paper and some of these other [6:18] Technologies and I think one of the [6:19] questions we have in revisiting the [6:21] architecture of a database is what's [6:23] possible Right how much of the value [6:25] propositions of databases can we retain [6:28] while tapping into some of the new value [6:30] propositions of distributed systems in [6:32] particular their arbitrary scalability [6:35] and [6:36] elasticity um also I think people are [6:39] sort of adopting um these distributed [6:42] systems and getting a bunch of [6:44] complexity as a result because they're [6:46] trading off uh distribution and scale [6:49] for consistency they're losing [6:51] consistency in the in the in the [6:54] tradeoff but we have things like Dynamo [6:57] and big tables how do we use them [7:00] other problems we have in general when [7:02] we talk about traditional databases are [7:04] flexibility problems you know everybody [7:07] knows the rigidity of relational [7:08] databases and the big rectangles and you [7:11] know the the artifice you have in having [7:14] to form intersection record tables and [7:17] things like that um things that you [7:19] really shouldn't have to know about and [7:20] your application ends up uh becoming [7:23] rigid because it it does know about [7:25] those uh in addition lots of things are [7:28] difficult to represent in a traditional [7:30] relational model like sparse data or [7:32] regular data hierarchical data um and [7:36] things like that so we want to be more [7:38] flexible we want to be more agile in our [7:40] development and we want to try to avoid [7:42] this rigidity seeping [7:45] in and of course again related to the [7:48] talk earlier another thing we want to [7:50] try to get right if we revisit database [7:52] architecture is information and time in [7:54] particular we want a database uh that we [7:57] can use to represent information we can [8:00] use to obtain real memory and real [8:03] record keeping like we used to have [8:04] before we had computers and there's lots [8:07] of good reasons for this it helps [8:09] support decision- making as I said in my [8:11] talk before and auditing there's plenty [8:14] of domains in which it's a requirement [8:16] and people are doing this manually on [8:17] top of systems that don't really [8:19] understand that that's what you're [8:19] trying to do how many people have ever [8:21] added timestamp field themselves to [8:23] tables and manage it all themselves like [8:26] people have right how many people have [8:28] written the query that get you now out [8:29] of that table yeah how many people have [8:32] tuned that yeah that's a nightmare [8:35] anybody like that query tuning that [8:37] query is brutal right the contention uh [8:41] is terrible especially if it's a it's if [8:43] it's also an online [8:45] system and the last thing I think we'd [8:47] like from databases uh that maybe we [8:50] don't think about now because we don't [8:52] connect the to necessarily I think is a [8:54] is a strong model for perception and [8:56] reaction right perception is part of [8:58] what I was talking about before getting [8:59] getting that stable basis for decision [9:01] making reaction is more like [9:04] Eventing right things are changing in [9:06] the world how do we see change in a [9:08] traditional client server [9:09] database what do we [9:12] do it's the big four-letter word begins [9:15] with p and ends with [9:16] o poll we poll very gross right so we'd [9:21] like to be able to make reactive systems [9:23] that don't pull uh and we'd like those [9:25] systems to get consistent views of the [9:27] world which is another difficult thing [9:28] even if you build even if you build [9:30] manually say a trigger based Eventing [9:32] system right people have done that right [9:34] so triggers and they say oh something [9:36] changed it's like okay great well in [9:38] between when they told you something [9:40] changed you and you're wanting to make a [9:42] decision on the basis of it maybe you're [9:43] going to go back to the database what's [9:45] the basis now for that did it change [9:47] again between when you you know what's [9:49] in Flight you have no way to know this [9:52] changed and that change was related to [9:55] the database at this point in time and [9:57] you can go back and ask questions to [9:58] figure out what was going on and what [10:00] either caused that change or what the [10:02] effects of that change should be or how [10:04] it relates to the rest of the world you [10:05] have no way to do that you just were [10:07] told something changed and maybe a value [10:09] about it but not where that's situated [10:11] relative to the rest of the world so we [10:14] want to do that [10:16] better so if we're going to uh take a [10:19] database apart we have to look at how [10:20] it's put together and this is just this [10:23] is not a particular database we're [10:24] talking about here this should be [10:25] familiar though to anybody who's dealt [10:27] with traditional databases and how how [10:29] they're laid out I'm sorry if this spont [10:31] is too [10:32] small um and the and the guts the meat [10:35] the meat of it is at the bottom so we'll [10:36] start at the bottom uh a database [10:40] certainly is something in general we [10:42] expect to be durable so most of the [10:45] traditional databases are built around [10:47] there is this disc and we're going to [10:49] we're going to put that disc in a box [10:51] and that box is going to be in charge of [10:53] the disc and everything comes from there [10:55] so there's some IO subsystem that deals [10:58] with the dis [11:00] right and then there's sort of three [11:02] fundamental well two fundamental uh sets [11:06] of services the database will provide [11:08] they both rely on the this third the [11:10] first is there's a trans transactional [11:13] component that accepts novelty and [11:15] integrates Novelty into the view of the [11:18] world right and then there's a query [11:20] support component that accepts questions [11:24] and gives you back answers based upon [11:25] the values here and both especially the [11:29] query side it needs to have storage like [11:31] if you just took everything that came [11:33] into a database and you just appended it [11:34] to a flat file how good would the query [11:37] engine [11:38] be not very right right so leverage [11:42] comes from indexing leverage comes from [11:45] organizing the way we store the data [11:47] such that a query engine has um sorted [11:50] views of things that it can use to [11:52] answer questions uh quickly and that's [11:56] the leverage of a database I think we've [11:58] gone to key value stores that have [12:00] almost no leverage and we're still [12:02] calling them databases but uh in my mind [12:05] uh this is what made a database a [12:07] database otherwise we had file systems [12:09] and all other kinds of things before we [12:10] had databases and we didn't call them [12:12] databases um why are we calling key [12:14] value stores databases now uh so in [12:18] general traditionally this was a big [12:20] monolithic thing there was a big [12:23] sophisticated or complicated process [12:25] that knew about all this stuff and had [12:27] an integrated view of how they would [12:28] work [12:29] and you know because this was expensive [12:32] and the memory it needed was expensive [12:33] and the Box on which it ran was [12:35] expensive this was a very special thing [12:38] you had one of them and then you had [12:41] clients which were somewhat more [12:42] lightweight and they communicate using [12:45] usually a foreign foreign language right [12:47] so how do you communicate with a SQL [12:49] database strings in a foreign language [12:52] you send it over and it it does [12:54] something right same thing how do you [12:55] communicate [12:57] queries you send strings over in a [13:01] foreign language and you get back well [13:04] who knows right maybe it maybe the API [13:07] makes it look like a result set to you [13:09] you know up at the Java level and then [13:12] we know as we get a lot of apps going [13:15] this unique resource gets taxed right [13:18] it's getting everybody's as everybody's [13:20] putting all the data in there and [13:21] everybody's asking questions there we [13:22] know the questions dominate uh in most [13:26] applications uh are read read oriented [13:28] so so most applications eventually end [13:31] up adding another tier right so if it if [13:34] it's very costly for me to ask questions [13:36] I'm going to store the answers to those [13:38] questions in a cach so maybe next time I [13:41] want to ask that question I'll check the [13:43] cash first otherwise I'll incur the cost [13:45] of going all the way to the server and [13:47] what goes in the [13:48] cache what form does it [13:52] take when does it get [13:55] invalidated whose problem are all these [13:57] questions [14:00] yours your problem or maybe you buy into [14:02] some fancy omm that's makes it your [14:05] problem with another layer on top of [14:07] your problem now you have two problems [14:10] yeah it's up to you there's no there's [14:12] no who knows it's definitely not the [14:15] server job right we this I would call [14:18] caching over the [14:20] database and there are some other things [14:22] that database comes with we don't [14:24] necessarily uh think about uh certainly [14:27] most of these databases most databases [14:30] have a data model um it can be a really [14:33] lowlevel thing that is about how things [14:36] are stored um or an API kind of thing or [14:39] can be a relatively high level thing I [14:41] mean it's certainly a great trait of SQL [14:44] databases that they're based upon a [14:46] mathematical foundation in relational [14:48] algebra that is a proper data model with [14:50] a bunch of great characteristics um that [14:53] allow you to write that those [14:54] declarative programs um but they also [14:58] contain a a state model uh and in fact [15:01] relational algebra is a lot like that [15:03] old paper right relational algebra is [15:05] like perfect it says there is this state [15:07] of the database and all this algebra [15:09] applies it's math it's great right how [15:12] do you get a new state of the [15:14] database well a miracle occurs and then [15:18] you have a new relational world and then [15:20] you have that but update is not [15:23] mathematical there's there's not the [15:25] same model behind it um so there is a [15:28] statement model and in general not all [15:30] the time that's an update in place model [15:33] that's subject to all the kind of [15:34] criticisms I gave in uh in the [15:37] keynote what's usually missing is an [15:41] information model and here I mean [15:43] something precise that I said before uh [15:47] um is anybody not in the keynote because [15:48] I'm just referring to it everybody is [15:50] okay great uh and and by an information [15:53] model I mean the ability to store facts [15:56] um to not have things replace other [15:58] things in place to have some temporal [16:00] notion to what's being stored that's [16:01] what I would consider a true information [16:04] model that's usually missing from the [16:06] databases so we want to we want to solve [16:09] all of this we want more scalability we [16:11] want to try to leverage these new [16:12] systems we'd like to have more [16:14] declarative programming in our [16:16] applications we'd like to have a proper [16:18] information model um maybe we don't want [16:21] to program with strings anymore what are [16:23] the challenges we're going to face um if [16:25] we Tred to do that with this approach [16:28] the biggest one by by far is definitely [16:30] the the state model the fact that it's [16:32] update in place and again as I said in [16:35] my talk there's a great reason why [16:39] traditional databases work the way they [16:41] do because when they were invented 30 or [16:44] more years ago these things these [16:46] resources were scarce you couldn't make [16:48] a database that said I'll just keep [16:50] everything you know because you had like [16:52] this tiny little disc uh so they [16:56] invented all this update and place [16:58] technology usually inside a database [17:00] there are these be trees they use blocks [17:02] on the disc they'll reuse the blocks [17:04] they'll fill the blocks they're [17:06] rewriting them they're usually [17:07] interacting at a pretty intimate level [17:09] with the memory management on the [17:11] computer and because they're updating in [17:13] place and they're trying to serve [17:15] multiple clients they have a huge amount [17:17] of coordination overhead uh to do that [17:22] and that slows them down [17:25] significantly so the approach we're [17:27] going to take in trying to break things [17:28] apart [17:29] is going to be based on these sort of [17:31] four three three principles one is to [17:34] move to an information model now I made [17:37] claims during my talk that using values [17:40] and and having an information model has [17:42] architectural [17:44] implications and and if you take away [17:46] nothing from this talk I would I would [17:48] hope you would take away that this is a [17:50] real example of that um that in action [17:55] uh that adopting a value oriented model [17:58] has architecture benefits really [18:00] substantial ones so we're going to move [18:02] to an information model we'll see how [18:03] that plays out we're going to split [18:06] process and perception and I I have a [18:08] diagram later that will make that [18:10] clearer um we're going to treat our use [18:14] of storage immutably in other words [18:17] we're going to store stuff but once [18:19] we've stored it we're not going to [18:20] change [18:21] it and the other half of doing that is [18:25] that in order to deal with process we're [18:26] going to have to manage no novelty in [18:29] memory uh for for a window of time and [18:31] I'm going to break all these down so to [18:35] move to an information model means to [18:36] move to uh a data model that is [18:39] fundamentally about facts so we're going [18:41] to say we're going to have a database of [18:43] facts and we're going to that means [18:44] sucking the structure out right because [18:47] when you look at a relational row or or [18:49] a document there's nothing fact like [18:52] about that right it doesn't say when and [18:55] the granularity of the fact is this [18:57] composite thing so you know if I had a [19:00] whole row for you and you changed your [19:01] email and email is one of the columns [19:03] you know where the the row is not a fact [19:06] it's not the granularity of a fact it's [19:08] bigger than a fact right maybe it's a [19:09] set of facts um so we want to get down [19:12] to single facts that's going to be [19:14] important for efficiency reasons but it [19:16] also dramatically simplifies stuff so [19:19] how many people know what rdf [19:21] is not too many okay so rdf is is an [19:24] attempt to um have a universal schema [19:27] for information and they use something [19:29] called triples which are subject [19:31] predicate object um I I I argued during [19:35] the [19:36] speaker um Summit Sunday that that's not [19:39] enough because it doesn't let you [19:42] represent facts because it doesn't have [19:43] any temporal aspect so if you take a [19:45] step back from that and say that's [19:47] generally a good idea it seems Atomic um [19:50] we really do want Atomic facts uh we [19:53] label them we call them datums um and we [19:56] just spell it differently so we can say [19:58] datum because if we spelled it D AUM the [20:01] plural would be data and then we'd be [20:02] into people don't know what that means [20:04] even though they say it all the time so [20:06] we have datum which is an atomic fact [20:08] and datums are more than one fact and [20:10] it's just an entity an attribute of [20:12] value and then some temporal component [20:15] it ends up that you could just put the [20:17] time stamp there but that doesn't give [20:20] you a lot of power If instead you say uh [20:23] well when this thing this thing came in [20:25] as part of a transaction I can sort the [20:27] transaction there [20:29] if your transactions are first class and [20:31] they are in this system you can put the [20:33] time on the transaction but you could [20:35] also put who said it on the transaction [20:37] or where it came from or whether or not [20:39] it's been audited or any other kind of [20:41] Providence or other characteristics so [20:43] that's what we do but you can read this [20:45] for the purposes of this talk because [20:46] I'm not going to talk a lot about that [20:47] other stuff as that t part is time it's [20:51] a path to when and that's the smallest [20:55] fact okay [20:59] so now we have the problem of the state [21:01] the database State problem we say we [21:03] want to have the database fundamentally [21:05] be a value we want it to be immutable um [21:07] it seems to be a contradiction in terms [21:09] because we know there's going to be [21:10] novelty right our business is going to [21:12] run and we're going to sell more stuff [21:14] or get new products or have new [21:15] customers and that novelty that newness [21:18] has to go somewhere so how does that [21:20] jive with the notion of a value and the [21:23] best analogy I could come up with was a [21:24] tree ring one right if you think of the [21:27] database has an ever expanding value so [21:30] it never updates in place it only [21:32] expands it only grows outward right that [21:35] it grows by ation of facts we're just [21:37] going to add more facts we never go back [21:40] inside just like the tree rings you [21:41] don't go back inside the tree rings you [21:43] just add you add more Rings um we end up [21:46] with something that really smells like a [21:49] value and will function as a value right [21:51] we're only ACC creating the past doesn't [21:54] change so the core upon which we're [21:56] building never changes and that's really [21:59] the key um characteristic we expect of a [22:01] value that anything I've seen before [22:03] will never change um the implications [22:06] are as we have novelty you know new [22:09] stuff means new space like I said in my [22:11] talk and this this is sort of Super Key [22:15] we're going to get a lot of Freedom by [22:18] moving away from places right at this [22:22] point the other problem we have is how [22:24] do we represent change so we're gonna we [22:26] said we're going to accrete facts so [22:28] what is what is the granularity of [22:30] change right we're used to saying update [22:32] this place and here's the address of the [22:33] place or here's the primary key of the [22:35] place you know go do something there if [22:38] we don't want to say that anymore if we [22:40] just want to accrete facts then what is [22:42] the fundamental unit of novelty you know [22:45] if I have a new customer what am I going [22:46] to say to the [22:47] database and what we're going to say is [22:49] that at the bottom we can represent this [22:52] process right so this is the problem [22:54] we're trying to solve right the novelty [22:56] problem we're going to say we can [22:57] represent novelty by just as assertions [23:00] or retractions of facts right this new [23:03] thing is true this new thing is true [23:05] that thing that was true is not true [23:07] anymore okay still a fact that it was [23:10] true from then to here it was true and [23:15] this ends up being the minimal possible [23:17] representation of process right with [23:19] this you can build on you can express [23:22] anything in these terms and so we'll say [23:24] that all the other Transformations will [23:26] expand into this and show you that a [23:29] little bit [23:29] later the other key thing we want to do [23:32] with process is we want to reify it how [23:34] many people have heard of like event [23:35] sourcing or or anything like that so one [23:37] of the ideas behind it is that um if you [23:40] talk about a database and you just look [23:42] at a database um that's been running for [23:44] a while and that's how a lot of activity [23:46] when you look at it and you want to know [23:48] what happened how do you figure that [23:51] out how do how did it become what it [23:55] is you have no resources for doing that [23:57] right unless you know how to read the [23:59] logs right maybe there's a transaction [24:00] log and and maybe there's a way to to [24:02] read read that but a lot of times you're [24:04] going to have to sort of Replay that [24:06] because that's just a re that's just a [24:08] successive set of modifications to [24:11] places it's really hard to read that and [24:13] understand what happened if instead we [24:15] say we're going to reify process when [24:16] you add a new customer there's going to [24:17] be something that says there is a new [24:19] customer that customer's name is Sally [24:22] that's what's going to be in a reified [24:24] version of process that says process is [24:26] just assertions and retractions of facts [24:29] so that's great so we want to make a [24:30] thing out of that because that's [24:31] something that we could store we can [24:33] look at we can understand when we look [24:35] at it this this is the this change [24:37] happened we added this we retracted that [24:39] email we added a new email we sold this [24:43] we did this we did this fact fact fact [24:45] fact fact these things happened right [24:47] it's an information system fact fact [24:49] fact fact fact um and that's going to be [24:52] great because that's going to let us do [24:53] some other cool things later like [24:56] events the accretion proc process one of [24:59] the um things that uh is important to [25:02] understand is that it really does add to [25:05] what's already there so that means that [25:07] if you ever look at the view of the [25:09] database at any point in time you will [25:11] be able to access the past it's still [25:13] inside just like the inner tree rings [25:15] are still there it's not like there's a [25:17] snapshot from last Tuesday and then one [25:19] from you know Wednesday and one from [25:21] Thursday and one from Friday and each [25:22] one um has more stuff in it uh it it's [25:26] as if anytime you look at the database [25:28] it includes all of history inside of it [25:31] and this is important for an information [25:33] model where we want to give people [25:34] decision-mak capabilities that say how [25:36] much have things changed in this window [25:38] of time or count how many of those that [25:41] happened over a window of time we need [25:43] all the time in one place we don't want [25:45] a bunch of Independent Records here's [25:47] Tuesdays facts and Wednesdays and [25:48] Thursdays separately so we have this [25:51] growing tree ring thing all right so [25:54] that's our plan how do we do it we're [25:57] talking more now at the [25:59] uh at the model level right we want to [26:02] deconstruct this now we're looking just [26:04] the server component right indexing [26:06] transactions query IO and disk and I [26:09] think you can divide this up into halves [26:11] right what I said before we want to [26:13] separate process from [26:15] perception right because there's a [26:17] process part of this right novelty [26:19] processing I have a new thing it has to [26:21] go through a transaction processing [26:23] thing maybe indexing happens on it then [26:25] or not don't know it's an open question [26:27] at this point in the talk and then this [26:28] output to the storage [26:30] system right completely independent of [26:33] that is a perception characteristic to [26:36] the use of databases right I have a [26:38] question want to ask the question maybe [26:40] that leverages indexes almost definitely [26:42] it does and there's going to be input [26:45] this is all process relative input to me [26:47] right as I read back from Storage we can [26:51] separate these two things out and we can [26:53] only because we've adopted an [26:55] information model and a [26:57] mutability so the model we're we're [26:59] trying to get to and again this is not [27:01] yet a physical model is one where we can [27:03] Empower applications and you can read [27:06] this as application servers or analytics [27:08] servers or anything you want it's not [27:09] like a desktop application right we want [27:12] to empower um independent applications [27:15] with as much of these capabilities as we [27:17] can we want them to be able to perceive [27:19] change we want them to be able to react [27:21] to it we want them to be able to [27:23] independently remember anything that's [27:24] important to their decision making [27:26] process and we want them to make [27:27] decision [27:29] and then possibly affect the process [27:32] that they're sharing what we want to do [27:34] and obviously there's going to be some [27:36] shared resources here right there's got [27:39] to be some coordination related around [27:41] change right and there's going to have [27:42] to be some shared resource around [27:45] storage right we want to minimize the [27:48] coordination that's necessary to support [27:51] this but that's the model we want [27:53] because now if we can do this if we want [27:55] a more powerful system what do we have [27:57] to [27:59] do if we want more query capability what [28:02] do we need to do just add more of these [28:05] guys we don't really care about this [28:07] growing because we're not asking it to [28:08] do much just some [28:10] coordination so if we revisit that whole [28:13] tree ring thing when now we're into [28:14] implementation details um how do we [28:17] represent State how do we represent this [28:19] immutable expanding value right we know [28:23] one thing whatever representation we use [28:25] it has to be organized to support query [28:28] right has to be sorted in some way [28:31] that's that's really sort of the [28:32] fundamental leverage capability we have [28:34] is sorting things and uh it ends up that [28:37] there's a technique that can be used you [28:38] know it's used in functional programming [28:40] quite often uh called persistent data [28:42] structures and the word persistent there [28:44] does not mean durable it's a different [28:47] notion of persistence and it has to do [28:49] with the fact that you can represent a [28:51] large mutable structure and it doesn't [28:53] matter whether it's supposed to [28:54] represent an array or a uh [28:58] map uh you can represent almost [29:01] anything or sorted set would be another [29:03] example we're going to use as a [29:05] tree and you can and you can represent [29:08] that immutably by using something called [29:11] structural sharing so this tree could [29:13] represent anything it could represent [29:15] for instance a sorted set and this is [29:17] the view of it we have right now and it [29:19] contains all these nodes if we want to [29:21] add another child to this node so we [29:25] want to make a new version of the set we [29:27] don't want to updated in place we're [29:30] going to need to allocate a new node for [29:32] that leaf and then copy the path to the [29:35] root then we have a new tree that has [29:39] this new piece of information in it and [29:41] substantially share structure with the [29:43] old tree because and we can do that [29:45] because [29:47] why it's all immutable right so this is [29:50] the underpinnings of what are called [29:51] persistent data structures so we can do [29:53] this in memory it's done in memory by [29:55] most functional programming languages [29:57] what we're going to start to do though [29:59] is do this on disk so we'll have a [30:02] really persistent or persistent [30:03] persistent or I'd rather say durable [30:06] persistent data structures that have [30:09] this kind of [30:11] shape so um you saw in the earlier [30:14] diagram right we had a server and it had [30:17] IO and a disk right I don't want I don't [30:20] want to know about iOS and discs anymore [30:23] um there was a paper recently that came [30:25] out it said dis locality considered [30:27] irrelevant [30:28] right it's another one of these old [30:30] Notions that's now [30:33] dying right it used to be boy if you're [30:36] not the machine that has the discs [30:39] you're at a tremendous disadvantage from [30:42] a computational perspective because that [30:44] machine it's got you know it's got a [30:45] card that's attached to the dis can get [30:48] the data up into memory it's lightning [30:50] fast that machine has privileged access [30:53] to that dis if you try to access that [30:55] data on the dis from another machine [30:58] you're going to be paying a huge [30:59] overhead well now how much faster are [31:02] networks way faster and what's the [31:05] differential between disc and memory [31:08] huge it ends up that anybody that needs [31:10] to access the dis is losing and the guy [31:13] who has the dis in the same box is only [31:16] losing very very slightly less than [31:18] anybody else would lose so it doesn't [31:20] matter anymore that kind of locality is [31:23] not the way we should be aring [31:24] architecting systems we don't care about [31:26] it care a lot more about uh putting data [31:29] into memory and having good locality in [31:31] the way we do that but if we actually [31:33] have to touch the dis we're we're losing [31:35] anyway so we're just going to wrap up [31:37] both the io and the and the disk and say [31:40] that's something we're going to call a [31:41] blackbox called [31:43] storage what we're going to put in [31:45] storage are two things one is a log of [31:48] every all the novelty as it comes in and [31:51] that's really a an append kind of job [31:53] right somebody says this I sold this [31:56] Sally gave me an email fact fact fact [31:59] we're just to shovel those facts as fast [32:00] as we can into storage the other thing [32:02] that will be in storage are much the [32:05] same shaped things that we used to have [32:07] locally right when we had a database [32:09] that was monolithic it had be trees on [32:12] disk we're now going to have those [32:14] persistent trees in [32:16] storage with nodes that we just don't [32:18] change but otherwise it's the same kind [32:20] of idea we've moved away from disc to [32:23] storage and we're storing um index [32:26] segments that were now going to mutate [32:29] uh but the key thing now is that we [32:31] we're treating storage and it can with a [32:34] with a simple interface it's just a key [32:36] value interface we say this block of the [32:38] index tree maps to this segment which is [32:41] a blob of stuff and it's going to be [32:43] immutable so all we need from Storage is [32:45] this sort of key value thing it's a lot [32:48] like the old databases used to say this [32:50] block on the dis contains these you know [32:54] bytes now we're just lifting it up to [32:56] sort of a systems level cooperating [32:58] systems level so of the storage we need [33:00] a key value interface where we can store [33:02] blocks of index under Keys we also do [33:06] need a little bit of modifiable storage [33:10] for the roots so what's the current root [33:13] of the whole database tree is something [33:15] we're going to need to point at new [33:17] versions of the tree and the other [33:20] characteristic we need from Storage is [33:22] that we have to be able to obtain that [33:24] using consistent read so we're now sort [33:26] of getting a set of definitions for [33:27] storage service right what are the [33:29] requirements of a storage service must [33:31] support key value storage and every now [33:33] and then we're going to ask it for a [33:35] consistent read these are the two things [33:36] we need out of storage otherwise I don't [33:38] care how it works or where it is don't [33:41] want to know I'm not getting [33:43] architectural flexibility from doing [33:46] that um and there's lots of things that [33:48] can satisfy this so you know we can sit [33:50] on top of um Dynamo DB or a SQL database [33:53] can satisfy those two things right you [33:55] can treat a SQL database as a key Value [33:57] Store stick blobs in it and uh it offers [33:59] consistent [34:01] read so an index is very simple thing [34:03] it's a tree right there's some rout it's [34:06] got pointers to Inner nodes which have [34:09] then got pointers to Leaf nodes and all [34:11] that's in these Leaf segments are sorted [34:13] datam so it's a big block of datum [34:16] sorted in a particular order you can [34:17] imagine the orders right we said entity [34:19] attribute value transaction so you're G [34:22] to have an a sort by [34:24] entity right that's going to give you a [34:26] great way to pull out what look like [34:28] objects or documents right because it's [34:30] entity oriented but you also have a sort [34:32] that's oriented by attribute first right [34:36] same data just sorted a different way [34:38] second copy of it sorted a different way [34:40] one that's driven by attributes is going [34:42] to feel like what kind of a [34:45] database going to feel like a column [34:47] store right column stor store just email [34:50] addresses all together and then they [34:52] store just uh uh phone numbers Al [34:56] together and column stor are very [34:58] powerful tools for [35:01] analytics and you can store other other [35:03] flavors but there's only you know six [35:06] ways to sort [35:08] them so then there's the the actual job [35:11] of [35:12] indexing right so we get novelty in we [35:16] want to incorporate this we're going to [35:17] have these trees obviously once we have [35:19] a lot of data those trees are going to [35:20] be what huge and we said they're [35:24] immutable so we said anytime you want to [35:26] incorporate new information [35:28] into that tree we're going to have to do [35:29] that path copying job do we want to do [35:32] that every time we get a new [35:35] fact no that's a disaster can't do that [35:40] uh so I would call that maintaining the [35:42] sort live in storage it's not something [35:45] you can do efficiently if you're going [35:46] to treat storage immutably so big data [35:49] Google's big data is an example of a [35:51] solution to this problem that other [35:53] people have used as well and the idea is [35:56] everything that's new you're already [35:58] logging to storage so this is not a [36:00] durability question it just has to do [36:02] with how often do you integrate novelty [36:05] into that big relatively expensive to [36:07] create [36:09] index uh the way big table works is it [36:12] accumulates novelty in memory like until [36:15] it's got 64 Megs of novelty and then it [36:17] blows that out onto disk and then a [36:19] separate process later takes that 64 [36:22] Megs and the big sorted flat file that's [36:25] everything it knew before and it does a [36:27] merge sort and produces a new flat file [36:30] none of the files ever get changed so [36:32] it's the same idea they're all immutable [36:33] the biggest difference between that and [36:35] what I've been talking about is that we [36:37] have trees of smaller segments and can [36:40] share stuff when they create a new flat [36:42] file it shares nothing with the older [36:44] flat file and it's huge so it's not [36:47] particularly addressable and it's not [36:49] easily cached in [36:51] chunks by using trees you get fine [36:54] grained addressability and you get these [36:56] nice small chunks which are good for [36:57] caching you also have the potential for [36:59] structural sharing right I can integrate [37:02] new stuff into the next version of the [37:03] index it could share a whole bunch of uh [37:06] nodes with the old index and if they've [37:08] been cached they're still good right [37:10] because we know they're not going to [37:12] change uh so we accumulate a memory [37:15] anytime we want a current view of the [37:17] world we're going to need to merge [37:19] dynamically what we have in memory with [37:21] what's coming from Storage just do a [37:23] dynamic merge join between those two [37:25] things and every now and then something [37:28] has to go and and integrate what's in [37:29] memory into storage and as soon as [37:31] that's been done everybody can drop that [37:33] from memory we don't care about that [37:35] we're going to start it fresh because we [37:37] know now that's in the index that's in [37:40] storage so that just looks like this [37:43] whatever is handling transactions this [37:45] is still pretty much a logical model but [37:47] whatever is handling transactions is [37:49] going to take novelty and immediately [37:51] log it right that's where you get your [37:53] durability so if the thing dies it's [37:56] somewhere but that's not organized in a [37:58] leverageable way it's going to put it [38:00] into an index in memory we'll call the [38:02] live Index this is sorted it's very [38:05] inexpensive to create you know a sorted [38:07] set in memory right and then sometime [38:10] later and occasionally some other [38:13] process is going to go and take this and [38:15] whatever is there and and merge but [38:18] there's going to be a lot more [38:19] efficiency to that because now it's got [38:20] a whole bunch of novelty and it's going [38:22] to make a new tree that incorporates the [38:25] new novelty and there's a lot of [38:26] efficiency to sharing that job as [38:29] opposed to doing it for every new new [38:33] transaction so that's the process side [38:36] the perception side is really [38:37] straightforward right if I want to see [38:40] what's going on it doesn't matter what [38:42] this is this could be a query it could [38:43] be an analytics thing it could just be [38:45] an ordinary you know get me an entity if [38:47] I want to see the current view of the [38:48] world I have to somehow have access to [38:50] the live index and access to [38:54] storage look at all the stuff I don't [38:56] have to have access to I don't need to [38:58] be near the transaction processing [38:59] system I don't need to even have [39:01] anything to do with it is there any [39:02] coordination associated with doing [39:05] this [39:06] no something has happened in the a [39:09] couple of slides here what happened to R [39:15] transactions they're [39:18] gone where did they [39:20] go they just disappeared because a [39:24] correct implementation of perception [39:26] does not requ require coordination [39:28] perception in the real world does not [39:30] require [39:31] coordination so we do are left one with [39:34] one lingering question which is how does [39:37] this get updated if this is actually [39:39] local to me where does that come from [39:41] I'll answer that in a second okay so [39:44] we're going to have just a couple of [39:45] names that may be new as we look at the [39:48] the real architecture um one is the [39:50] transactor so we've broken stuff down [39:53] now we have storage separate um we have [39:56] perceivers right who might be doing [39:58] queries they're separate um we're going [40:01] and then something that processes and [40:02] coordinates transactions we're going to [40:04] call that the [40:05] transactor we're going to call anyone [40:07] who has all those capabilities the [40:09] perceive remember decide react stuff [40:12] appear right they're equal they're very [40:14] powerful equals in this [40:16] system um and they're your own [40:18] application servers and then finally we [40:20] have some sort of storage servers [40:21] ideally it would be a redundant store of [40:23] one of these new fangled storages right [40:25] that do distributed redundancy they're [40:27] really have some great properties as [40:28] long as you don't try to treat them like [40:29] a database if you treat them like a key [40:31] value store they are awesome um they're [40:35] redundant they're highly reliable [40:36] they're highly available they're [40:37] scalable [40:38] distributed there's a lot of power there [40:40] we want to use [40:42] that so this is um this is the sort of [40:46] the same jobs [40:48] all [40:50] rearranged um for the purposes of um [40:53] talking about this I'm going to say that [40:54] a transactor will independent will [40:56] occasionally do the indexing job and and [40:59] it does you know in in the in the [41:00] current implementation as well just [41:02] because it has the spare cycles and it's [41:05] convenient to do so but anytime you want [41:06] to you could move this to a separate box [41:10] um so we'll start we'll start uh with [41:12] some novelty your app has some novelty [41:14] it needs to communicate with the [41:16] transactor right the transactor is going [41:18] to log that right [41:20] away um it has its own live index you [41:23] can just imagine that's here um it's [41:25] going to put it in there and the other [41:27] thing it's going to do is it's going to [41:28] transmit that novelty to any of the [41:32] peers and that's how the live indexes [41:34] are kept together right it's just a re [41:38] it's just a [41:40] rebroadcast um of the of the novelty and [41:42] the novelty only this is only the [41:46] novelty that comes through and I'll talk [41:47] a little bit more about this process in [41:49] a second Right Storage service we don't [41:53] care a whole lot about it's very much a [41:55] black box it needs to be key value store [41:57] that can support consistent read right [42:00] this works on top of Dynamo DB it works [42:02] on top of postgress or any relational [42:04] database it works on top of infinispan [42:06] right if you want a big memory grid [42:08] behind that you don't actually ever want [42:09] diss you put infin span behind it now [42:12] you have a big memory grid and basically [42:15] anything that can support that will [42:17] eventually support but those are ones we [42:18] support right now uh if we look up at [42:23] AER uh obviously have a Communications [42:25] component so they can talk to the [42:26] transac [42:27] they're going to have this live index in [42:31] memory uh and then they have to have [42:33] some ability if you just ignore the [42:35] caching for the moment some ability to [42:38] talk directly to storage and this is the [42:40] other critical thing right we said well [42:42] now who has locality to [42:46] storage nobody actually no one has [42:49] locality storage in this except whatever [42:51] the storage infrastructure this guy is [42:53] no closer to storage than these guys [42:57] because there's no advantage to being [42:58] close to storage so this is just a [43:00] service or a server somewhere else but [43:03] once it is that it means that everybody [43:05] can read directly [43:08] yes [43:12] con no there's one [43:15] transactor I'll talk about that I'll [43:17] talk about that in a second yeah [43:21] uh okay so so everybody has access to [43:24] storage it also means that where does [43:26] query look [43:28] live query can live anywhere you want [43:30] right what does query need what does a [43:32] query engine actually need needs access [43:35] to the organized data access to the [43:39] index since anybody can get access to [43:42] the index we can put query anywhere we [43:44] want there's no special machines for [43:45] query so in the case of datomic you end [43:49] up with this Library you put in your [43:51] Java app or jvm app uh that has a Corey [43:55] engine built into it and it's got the [43:57] live index built into it and I'll show [43:59] you what that looks like in a second the [44:02] the other thing that's really critical [44:03] about this is so we have the storage [44:04] it's remote it's at least a network hop [44:07] we said the network hop isn't costing [44:09] you nearly as much as the disc would if [44:11] you actually had to touch the disc but [44:12] it is still a network hop can we [44:14] alleviate [44:16] it what what what's what is safe to cash [44:19] from this this [44:22] storage but what's safe to cash I mean [44:24] that's what you might choose to cash but [44:26] what's what's safely cash [44:28] everything why it's immutable when does [44:31] it expire never this is a these are the [44:35] things you want to hear as an architect [44:37] right this is sweet we can cash this [44:39] stuff relentlessly anywhere we want you [44:42] can have a local cash you can set up a [44:44] mem cash cluster right and cash stuff [44:47] there so when guys don't find it in [44:49] their local cash they can pull it from [44:51] storage and put it in a cash that a [44:53] whole bunch of peers could share this is [44:55] very powerful but what's different about [44:57] this whose problem is this putting this [44:59] looking it up in storage and then [45:01] putting it in the cash if it's not there [45:03] whose problem is [45:04] it no it's my problem it's the system's [45:09] problem it's this library's problem it's [45:11] not your problem you never see that you [45:15] effectively you ask a query the query [45:16] tries to find the data it's either got [45:18] in the cach OR it goes and figures out [45:21] how to get it but it's under this is [45:23] caching under or caching inside it's not [45:26] an application Level problem this [45:28] caching because this caching is [45:29] mechanical there's no special logic [45:32] around or anything else right the system [45:34] can do this caching or for you all you [45:37] have to do is start up M cach and tell [45:39] it about it yes sounds like Lambda [45:43] ARS but are youn it all so you don't [45:46] have to [45:49] recate you're bring everything no no you [45:52] don't bring anything in right so so this [45:54] is a this is an app what does it do I [45:57] don't know maybe it does analytics on [45:59] pricing so it's going to be interested [46:01] in a certain portion of this data that's [46:03] all it's ever going to cash this other [46:05] app this guy underneath him he's the [46:08] website he's putting up product pages [46:11] and he's reading all different kinds of [46:12] stuff from here this is not a [46:14] replication of everything it's not even [46:16] proactive replication of anything in [46:18] specific right everybody is pulling in [46:21] their working sets depending on the work [46:23] they're doing and they don't care they [46:25] never need to pull in anything else ever [46:27] now of course being trees they're all [46:29] going to have a really nice set of the [46:31] top part of that tree that they'll all [46:32] cach and they'll get a lot of efficiency [46:34] for that and in fact it ends up that [46:36] that diagram where I showed you three [46:38] tiers that's it it doesn't get any [46:40] deeper than that which means that once [46:42] you've cached the top part of that you [46:44] can find any piece of information you've [46:46] never seen before in one read and [46:50] potentially that's one read from mcash [46:53] although some of these storage services [46:55] like Dynamo is really fast [46:58] um so you're you're only caching it's [47:01] it's a it's a it's a on demand driven [47:04] cache it's filled on demand it's filled [47:05] with your working set so different peers [47:08] will have different working sets and [47:09] different amounts of stuff cached yeah I [47:11] think this whole example total justs [47:15] this is only discussing no this part is [47:18] rights this is the right path right I [47:20] have some novelty I send it to a [47:22] transaction it writes it in here so [47:24] that's that's the right part this is all [47:26] read [47:28] yeah [47:30] as [47:32] you're [47:33] b key index [47:36] you let's just talk let's just talk [47:39] through a little bit right so the let's [47:41] just say that everybody understands that [47:42] indexing um takes let's just imagine it [47:45] takes the log it doesn't actually but [47:46] let's imagine it takes the log and turns [47:48] it into the sorted tree so log is just a [47:51] pend to pend to pend to pend and the [47:53] index in here in storage is the sorted [47:55] tree let's say it last did that an hour [47:59] ago everything that's happened since it [48:01] last did that is right here right it [48:04] came in it said boom it got logged but [48:07] it's not in the index yet but it got [48:09] reflected back out so it's here so now [48:12] you want to answer a question what's I [48:14] want to put up Sally's profile page and [48:17] in the last hour Sally told me her email [48:20] address had changed that fact is here [48:23] all the facts Sally's told us in the [48:25] past prior to an hour ago are in here [48:29] when I want to go and ask a question [48:30] saying tell me everything you know about [48:32] Sally because I need to show her profile [48:33] page I'm going to do well the query [48:36] engine is going to do a merge join it's [48:38] going to say Seek to Sally in here seek [48:42] to Sally in there merge them together [48:45] right so if there's retractions in here [48:47] they'll they'll make it seem like the [48:50] older facts here are invisible but what [48:52] you'll get is the sum of information [48:54] from the two and that looks like now [48:57] eventually let's say we allow this to [48:59] build up for an hour and we say now's a [49:01] good time to index we're going to make a [49:04] new tree here it's going to share a lot [49:06] with the old tree but it will now [49:07] incorporate the Sally's new email [49:09] address when that job is done everyone [49:11] will be informed and this will turn to [49:13] empty and we'll start accumulating the [49:16] next window of change that's how it [49:19] works that yeah yes the live index is [49:23] actually local in memory to any Pier [49:31] Pier that's [49:32] correct yes every change is broadcast [49:35] every [49:41] paer well so the thing is that these are [49:44] these are [49:45] servers right you have to these are [49:47] these are not something you're going to [49:48] have 10,000 [49:50] of you this this pier is really [49:53] something like an app server [49:57] okay that's the way to think of it it in [49:59] turn is probably serving other people [50:01] the way your app servers do today right [50:03] it's not necessarily the web layer it's [50:06] not necessarily your web tier although [50:08] 10,000 players in your web tier would be [50:10] a lot um we don't currently have a um [50:14] multicast infrastructure for making that [50:16] propagation to if you wanted tens of [50:18] thousands but that would be a way to [50:20] approach doing that let me keep moving [50:22] forward and then I'll try to answer [50:23] questions [50:25] later it definitely that that does [50:27] happen I just didn't want to draw that [50:29] line because it makes it all messy the [50:31] that stuff does come over here but it's [50:33] uh it's not the same as what's [50:36] there it's not it's not organized [50:39] yet the organization happens here okay [50:43] let me keep moving and then I'll I'll [50:44] take questions at the end so process [50:47] itself it ends up that I said we can [50:49] boil everything down to assertions and [50:50] retractions but you can certainly think [50:52] of Transformations that you couldn't [50:54] Express that way like I want to add $10 [50:57] to your bank account right that's the [51:00] logical process I want to make but it [51:01] ends up that the fact that ends up from [51:04] that process is dependent upon the [51:06] existing value in your bank account so [51:08] how do you do that and the way you do [51:10] that is with transaction functions a [51:12] transaction function is a function of [51:15] the database and some [51:18] arguments and what it yields is [51:20] transaction data so we said transaction [51:22] data is assertions and retractions of [51:24] facts now we're going to say transaction [51:27] data is assertions and retractions of [51:29] facts or transaction functions and [51:32] arguments which means that we can have a [51:34] transaction function that is ADD uh that [51:38] takes and it will be past the database [51:40] we say add uh $10 to Sally's bank [51:44] account that function will be run inside [51:47] the [51:48] transaction it will be given the current [51:51] value of the database because the [51:52] database really is a value so this is a [51:54] real function of a real value [51:57] it can perform any queries at once [51:59] including looking up Sally's current [52:01] balance and then we'll yield more [52:05] transaction data so if it was that [52:07] simple a thing it would go that function [52:09] could go look it up find out the balance [52:11] is 100 say it's 110 and what it would [52:14] expand into is the fact Sally's new [52:17] balance is 110 but it could do some more [52:19] involved things I have a picture in in a [52:21] second so you can imagine any [52:24] transaction is assertions and [52:25] retractions and potentially calls to [52:27] transaction functions those transaction [52:29] functions can expand into calls to other [52:31] transaction functions but eventually [52:33] will expand into assertions and [52:35] retractions this expansion happens over [52:38] and over again until it's all assertions [52:40] and retractions and this allows you to [52:42] do any arbitrary transformation on the [52:44] data in the database you can ask [52:45] questions you can do anything you want [52:48] but the cool thing is you have a [52:49] decision again architectural [52:50] Independence as to when you do this [52:52] right if you know you're adding pure [52:54] novelty you don't need to do it inside [52:55] the transaction you just say I have new [52:57] facts I know they're new if if you have [53:00] something where you think you have very [53:01] little [53:03] contention um you can say well my local [53:06] view of Sally's bank account is 100 I'm [53:08] going to make it 110 and I'll just put [53:10] that in with a with a condition that [53:12] says if it wasn't 100 fail that's more [53:15] optimistic but if it was an expensive [53:17] calculation and the chance of collision [53:18] was really low that's the more efficient [53:21] way to do it or finally you can do it [53:23] this sort of old school way which is [53:24] send the function all the way into the [53:26] transaction which is going to get run [53:28] automically inside a transaction with no [53:33] interference so we call that uh [53:35] expansion so this is what the transactor [53:37] does it accepts transactions [53:39] transactions are just data fact fact [53:41] fact you know assert assert assert [53:43] retract retract and then even if you [53:45] want have a function it still Express [53:46] this data it says call the you know [53:49] update salary function with this [53:52] argument and this this target entity um [53:55] it expands them it eventually will apply [53:58] them to the uh inmemory view of the [54:00] database inside the transactor it will [54:02] log it right and finally acknowledge the [54:06] uh transaction to you and then broadcast [54:07] it to everybody else every now and then [54:10] this indexing will occur and I as I said [54:12] in my talk this the storage equivalent [54:15] of garbage and Garbage Collection comes [54:17] out of doing it this way right because [54:18] as you make a new index and now no one [54:20] else is going to care about the old [54:21] index the root and a bunch of nodes of [54:23] the old index are junk in storage and so [54:27] we have to clean that up but the analogy [54:29] to memory is is really [54:32] good these Pier servers uh have direct [54:36] access to storage they have their own [54:37] query engine they have their own live [54:40] memory index and they do that merge [54:42] themselves um there's also this two tier [54:45] cache going on and this is inside right [54:48] under right when you ask a query if it [54:50] doesn't have the data it gets it um and [54:52] it will look for a local cache which is [54:55] at the object level if that um is a miss [54:58] it will go and look for segments in [54:59] memcached if you've configured it [55:01] otherwise it'll go to storage and then [55:03] it will cach [55:06] it so I'm not going to talk too much [55:08] about this because it's sort of involved [55:09] but this is just picture of what's [55:11] happening right there's a inmemory [55:13] persistent data structure that's the [55:14] live index it's really immutable in [55:16] memory too we're just moving from one to [55:17] the next there's an infrastructure for [55:19] allowing you to find the data in storage [55:24] it goes through the cache and then find [55:26] a storage and its trees the roots of [55:28] which will almost definitely be cached [55:30] in [55:31] memory this value is just a pointer [55:35] right it's a struct right that points to [55:36] these things it's inside a [55:40] box the only mutable thing in the system [55:43] is the fact that the contents of this [55:46] box will move from one of these [55:47] immutable structures to another whenever [55:51] we've updated the memory index or [55:52] whenever an indexing job is completed [55:55] there's nothing else m able in the [55:57] system this one identity which is the [55:59] database as of right [56:01] now which means if you've obtained this [56:04] and you started running a long running [56:05] query everything that you've got won't [56:08] change underneath you even amongst [56:10] threads in your same process right [56:13] you're really working with the [56:16] value okay so what are some of the [56:19] characteristics well the first thing is [56:20] and somebody asked about it before [56:21] what's the um you know does it use the [56:25] Lamport clocks or what right or vector [56:27] clocks and one of the cool things about [56:29] separating perception and process is you [56:32] now can make independent decisions about [56:35] availability and scalability for those [56:37] two things because they're completely [56:39] separate so the decision that's made by [56:42] by datomic because I think it's a market [56:44] need and has a lot of value to companies [56:47] is to make a traditional decision about [56:51] the process side it's [56:53] transactional right it's a consistent [56:55] view of the the world it's a single uh [56:59] single writer system the cool thing [57:02] though is that that transactor it's not [57:04] doing anything else it's not serving [57:06] queries or anything all it has to do is [57:08] handle rights right if you want [57:11] arbitrary rights scalability you're [57:13] going to have to give up transactions [57:15] and queries and everybody who's adopting [57:17] no SQL databases um that make that [57:19] choice for you um is getting that [57:22] tradeoff and I think it's not a great [57:24] trade-off for a lot of companies and [57:26] that's why I'm in the space trying to [57:27] give them this hybrid Solution that's [57:30] the combination of two on the read side [57:32] though we get all the benefits of [57:34] distribution right if you put Dynamo DB [57:36] in that storage slot you have arbitrary [57:39] scaling you know all of all of Amazon's [57:42] goodness about storing stuff in [57:43] different places you have knobs on [57:45] throughput for reads and wrs tremendous [57:48] service approach to storage um or you [57:51] can just put you know you're already [57:52] running a SQL database you can leave it [57:54] there and just start adding data in this [57:56] format on top of it U but it's an [57:58] independent decision uh if you do choose [58:01] a uh a redundant scalable storage [58:04] subsystem though you will get the [58:05] benefits of having done that um so you [58:08] get scalable reads and obviously query [58:10] scale right because the queries are not [58:12] in one box the quaries are in every peer [58:14] box so you not only have scalability of [58:17] that but you have very elastic [58:18] scalability of that it's not like adding [58:20] a new box to a cluster configuration [58:23] this is kind of a big job this could [58:25] this can skip scale up and down with you [58:27] know Amazon's Auto scaling right more [58:30] peers come up and down due to load you [58:32] have more query capability so it's [58:35] elastic um I'm going to hurry up through [58:37] the last couple here you definitely have [58:39] more flexibility the schema model is [58:41] extremely small all you define are the [58:43] attribute definitions you know the name [58:45] and type of an attribute what its [58:46] cardinality is and things like that [58:48] there's no higher level thing right the [58:50] the fundamental unit is a fact the only [58:53] configurable aspect of the fact is the [58:55] attribute so that's the only schema that [58:57] exists and there is schema um and the [59:00] net result is that it feels like whoops [59:02] it feels like you're programming with [59:04] multimaps if you take an entity approach [59:06] to this it feels like every entity has [59:09] is a is a key value set where each key [59:12] each value could be multivalued if you [59:14] wanted um so it's very convenient it's [59:16] very flexible and and uh easy to use [59:19] programming model down at the [59:21] bottom the other huge benefit you get [59:23] from this is you get time right the [59:26] database is really a value it does [59:27] include all the past you can take any [59:30] database that you've got and say pretend [59:33] it's as of last [59:35] month you can just say the database [59:37] pretend it's last month that doesn't do [59:40] anything it doesn't change stuff it [59:41] doesn't throw stuff away it just says [59:43] pretend it's last month now you can ask [59:44] queries of that [59:45] database right when you ask the queries [59:47] if it finds anything newer than last [59:49] month it just doesn't it doesn't use it [59:52] and it can answer those questions which [59:54] means you can do as of questions against [59:57] the current database for a past point in [59:59] time they don't cost anything uh to do [60:02] that the other critical thing is how [60:03] many people ever built systems with Tim [60:05] stamps and had to do as of [60:08] quaries it's brutal right because what [60:10] do you have to do you have to flow [60:12] around that t that time right if I want [60:15] to say as of last week last week has to [60:17] be part of every query has to be part of [60:19] every join and it's nasty right you have [60:21] to do Max and all [60:23] that very very tough stuff what if just [60:26] say about this I walk up to a database [60:28] and I say as of last week so I can have [60:30] a query which is get me the you know [60:33] sales sub totals that query works [60:35] against now I can tell the database give [60:37] me yourself as of last week I can take [60:39] that same query run it against that [60:41] database value it's not parameterized by [60:44] time anymore right the database knows [60:46] I'm supposed to be pretending it's last [60:47] week now the same query you can ask it [60:49] doesn't have any time not time not part [60:51] of the join of course you can also ask [60:54] Cory where you want to compare times and [60:55] things like like that so you can do as [60:57] of a point in time you can do windows [60:59] since a point in time the other thing [61:01] you can do because the database is [61:02] actually a local value right you have [61:04] that memory component but it is really [61:06] local in your process you can do as if [61:09] what if what if I added this stuff what [61:11] if I committed this transaction what [61:13] would the database look like what would [61:14] the answer to this query look like do I [61:16] need to go to the server to do [61:18] that no I can just take the inmemory [61:21] data structure which is persistent right [61:23] I can make a new version of it that [61:25] includes the transaction data I'm [61:26] thinking about committing that's now [61:28] another database value it's all in [61:30] memory but it can can include stuff from [61:33] Storage right but the novelty is the [61:35] stolen memory now I can take the same [61:37] query and ask the query I don't need [61:39] anybody help anybody's help I'm not [61:40] bothering anybody right I can do what if [61:43] and then I can say well that query still [61:45] works so good now I'll try to commit it [61:47] or maybe I'm just doing you know what if [61:49] analysis for people and I never commit [61:51] it you know we're just trying to make [61:53] decisions what would happen if we made [61:54] this decision what would the world look [61:56] like I don't have to put that into the [61:57] database to ask answer the question I [62:00] still have query [62:01] capability perception and reaction [62:04] obviously perception is straightforward [62:06] right we have this immutable thing all [62:07] the queries have been flavors of [62:09] perception but the reaction now is easy [62:12] because we have this live feed right the [62:14] transactor is sending all novelty to all [62:16] the peers which means get what what's [62:18] easy to do it's easy to make an event on [62:20] the peer that says some novelty came in [62:23] but the other thing that's beautiful [62:24] about it is you can say here here's some [62:26] novelty you know Sally changed her email [62:29] address and if it was important you to [62:31] say wow is it the same as what it was [62:34] can you do that sure because the [62:36] database as a value means that you can [62:38] capture the database value when that [62:40] change was made and in fact it gets sent [62:42] to you with the event as well here's the [62:43] database before the change here's the [62:45] data of the change here's the database [62:47] after the change ask any queries you [62:49] want and if you want to have [62:51] sophisticated Eventing you just have to [62:53] query that data right you just say I got [62:57] this event feed and I'm only interested [62:58] in these certain kinds of changes you [63:00] just query the feed and now you get that [63:02] because the query engine is in memory [63:04] and it operates not only against the [63:05] database but it operates on memory data [63:07] structures too combinations of your own [63:10] memory and what's coming from the [63:12] database so you can filter that way so I [63:16] think you get a lot out of this in [63:18] particular what you get is Simplicity in [63:20] the database the state model is what I [63:22] would call epical right it only moves [63:25] from valid consistent point to valid [63:27] consistent Point there's never any in [63:28] between there's no read coordination or [63:31] anything else there's only coordination [63:33] for the process and that's done in [63:34] transactor every time you ask the same [63:36] quer you get the same results you have a [63:38] stable basis for decision making if you [63:40] think there was a problem with the [63:42] system in the middle of the day and [63:44] you're not going to get to look at it [63:45] till next week how many people have ever [63:46] had that happen the quer is returning [63:47] some weird results at like 5 o'clock [63:49] everybody wants to go home so we'll like [63:51] look into this tomorrow and then you [63:52] come back tomorrow and the quer is fine [63:55] because just because more data was added [63:57] into the system how are you ever going [63:59] to find out what was wrong your toast [64:02] with this it's straight forward right we [64:04] can say ask that query as if it was five [64:06] o'clock because I know at five o'clock [64:07] that query was screwy and giving me [64:09] weird results and you could figure out [64:11] the answer to your problem uh because [64:13] you can get a basis anytime you want and [64:15] the transaction is really well defined [64:17] what a transaction is is a function of [64:19] the database [64:21] value um the other things that are [64:24] important is you can communicate a base [64:26] right if I make a decision or I want to [64:28] give you work to do I have a way to [64:29] communicate it I'm not going to say go [64:31] look in the database I'm going to say [64:33] there's some work you have to do at [64:35] database time t uh and you can go and [64:38] look at exactly what I was seeing so you [64:41] know what to [64:42] do um we've seen already architecturally [64:45] because we've broken stuff apart we have [64:47] freedom to relocate things we don't care [64:49] where the queries are run we don't care [64:50] where the storage is we don't care where [64:52] the transactor is running you have a [64:54] whole bunch of flexibility about where [64:56] you put things you can put you know data [64:58] up on dynamodb and run the whole system [65:00] in your land and put mcash in your land [65:02] and isolate yourself from the fact that [65:04] the internet's actually potentially in [65:06] between um we saw the time travel stuff [65:09] and we saw the event processing so the [65:12] net result of deconstructing the [65:14] database uh is that you're able to treat [65:16] the database as a value um in in [65:20] precisely the way I was talking about in [65:21] my talk you have a real information [65:24] model you have a system that's [65:25] substantially less complex right it's [65:28] easier to understand it's easier to [65:31] change right you want to change your [65:33] storage you can do that it's easy to [65:35] replace components it's easy to relocate [65:37] things um it's more powerful uh it's [65:41] more [65:42] scalable right you want more brains you [65:45] start more [65:46] peers there's less coordination which [65:48] also leads to the scalability aspects um [65:52] and the information model not only lets [65:53] you remember everything which is [65:55] important for your businesses because [65:56] they want to make decisions but also uh [65:59] gives you more flexibility because [66:01] that's a model that's easy to turn into [66:03] any shape that you [66:04] want and uh with that I'll wrap and [66:07] answer any questions but thanks [66:10] [Applause] [66:16] [Music]