AWS Engineer Houlihan Uses Single-Table DynamoDB to Replace RDBMS Joins

AWS工程师Houlihan采用单表DynamoDB替代关系型数据库联接 将规范化关系型模式应用于云NoSQL会导致性能严重骤降，因而需要单分区预聚合以消除大规模下的高昂计算联接成本。 演讲者Rick Houlihan阐述了如何从传统关系型模式转向优先考虑访问模式的NoSQL建模，以针对云经济效益和可预测的扩展能力进行优化。他吸取了Amazon内部从数千个Oracle数据库迁移的经验教训，展示了如何将异构数据预聚合到单表设计中，从而在不产生计算密集型读取开销的情况下模拟关系型联接。本场演讲深入探讨了复合键、全局二级索引重载、写入分片以及变更数据捕获流水线等具体技术，并通过一个涵盖23种访问模式的端到端员工门户和实际载荷优化案例进行了说明。 Amazon将大约12,000个内部服务从Oracle迁移到NoSQL，并通过围绕可预测的OLTP访问模式而非规范化关系模式重构系统，退役了超过3,000个Oracle实例。 (0:00) 现代云经济学颠覆了催生关系型数据库的历史成本模型：存储成本目前每GB仅需数美分，而计算则成为主导支出，这使得横向扩展反规范化远比运行时联接更具经济性。 (2:16) DynamoDB通过将异构实体预聚合到共享物理分区中来取代高昂的读取时关系联接，从而允许在单次网络往返中检索父记录和子记录。 (10:26) DynamoDB物理分区强制执行约1,000 WCU和3,000 RCU的上限；超过这些阈值的负载需要通过计算后缀和客户端分散-聚集逻辑进行写入分片。 (14:45) 在某保险报价负载中，通过存储轻量级增量记录而非完整的50 KB文档，写入容量消耗从1,000 WCU大幅降至50 WCU，将数据组装工作转移至下游客户端设备。 (53:19)

Rick Houlihan, Principal Technologist for NoSQL at AWS, argues that NoSQL data is inherently relational and that achieving high scale requires shifting from normalized relational schemas to access-pattern-driven, pre-joined data models in managed engines like Amazon DynamoDB. Economic Drivers and the Misconception of Non-Relational Data (2:16) Houlihan explains that traditional relational database management systems normalized data to optimize for expensive disk storage at the cost of compute. In modern distributed systems, data modeling must optimize for CPU efficiency by denormalizing data into distributed hash structures. Applying normalized relational modeling to NoSQL architectures produces severe performance degradations and cost overruns at scale. Single-Table Design and Pre-Joined Heterogeneous Partitions (10:26) Rather than performing runtime relational joins with computational complexities of O(log N) or worse, DynamoDB persists data directly into pre-aggregated partitions queried in O(1) constant time. A single partition defined by a partition key can co-locate heterogeneous entity types—such as customer records, timecards, and orders—sorted by composite keys. Scaling Write Throughput and Index Overloading (14:45) Because individual DynamoDB physical partitions enforce an upper throughput boundary of roughly 1,000 Write Capacity Units (WCUs) per second, high-velocity workloads must use write sharding (salting) to scatter writes evenly across partitions. Centralized data access layer APIs can then execute asynchronous scatter-gather queries across the sharded key space. Client-Side Computation and Payload Optimization (45:51) Complex operational workflows can significantly reduce infrastructure expenses by pushing triage and evaluation to client hardware. In a multi-access-pattern employee portal design, room reservation logic is resolved on client devices by fetching room metadata and active reservations over lightweight queries, cutting down application server fleet sizes. Houlihan maintains that NoSQL does not deprecate relational engines, but rather divides workloads by design fit: high-scale OLTP systems operate most economically on cloud-native NoSQL using rigorous entity-relationship design, while relational engines remain the ideal choice for exploratory, ad-hoc OLAP workloads.

AWS NoSQL 首席技术专家 Rick Houlihan 指出，现实中的业务数据天然具备关联性，在分布式 NoSQL 中应摒弃关系型引擎运行时的昂贵 Join，通过单表设计与预聚合模型直接满足可预测的访问模式。 技术演进与计算存储经济学转变 (2:16) Houlihan 回顾了从生物记忆、手工账本、打孔卡到现代分布式数据库的数据处理演进史，强调这一历程始终受制于“数据压力”——即系统无法足够快或足够低成本地执行所需操作。关系型数据库在 1980 年代兴起时，存储硬件极度昂贵，规范化设计通过消除冗余来节省磁盘空间，代价是消耗 CPU 在查询时执行表连接；而在现代分布式云架构中，存储成本大幅下降，CPU 计算却成为核心瓶颈。将关系型数据库的规范化模式硬套在分布式 NoSQL 上，会导致大规模扩展时性能严重下降并产生高昂成本。在设计 NoSQL 时，必须从“先定义 Schema”转向“以访问模式为先”，利用反规范化布局在单次网络往返中检索数据。 预聚合单表建模与关系表达 (24:07) 针对 NoSQL 是“非关系型”的行业误解，Houlihan 强调数据建模本质上都是关于实体间关联的表达。在关系型数据库中，即使是高效的嵌套循环 Join，其复杂度也会随数据集增大上升至 O(log N) 甚至 O(N log M)；当面临极大规模数据时，优化器常退回到构建哈希表的 Hash Join。DynamoDB 本质上是分布式哈希表，通过在同一个物理分区内预聚合异构实体，直接实现 O(1) 复杂度的常数级读取。例如在一个分区中同时存储客户元数据与订单记录，单次范围查询即可一次性获取父子实体，避免了多表遍历与应用层聚合开销。对于多对多图关系，可以通过在全局二级索引（GSI）上对调主键与排序键（分区键与排序键互换），将单向有向图转换为双向无向图，同时在边实体上反规范化存储不变属性，实现高效双向检索并避免数据过度冗余。 索引过载与高吞吐写分片架构 (14:45) 在 DynamoDB 单表设计中，推荐使用通用的 PK 和 SK 命名主键，并在 GSI 上实施“索引过载”（Index Overloading），将不同实体类型的访问路径汇聚到同一个泛型索引中。这种做法避免了为每个查询模式单独配置独立 GSI 带来的预置容量碎片化、独立 CloudWatch 告警监控成本以及非并发时段的资源闲置。在存储引擎底层，DynamoDB 物理分区存在约每秒 1,000 WCU 的硬件写入上限；当应用对特定热点键（如全量销售渠道或热门商品）产生高频写入时，必须在 GSI 分区键后添加计算后缀（如伪随机数或哈希取模）实施写分片（Salting），将流量分散到多个物理节点。为避免调用方感知底层分片，系统应在数据访问层 API 内部实现并发 Scatter-Gather 并行查询与结果归并，使存储节点监控热图呈现均匀分布的负载状态。 复合键过滤与基于 CDC 的聚合流 (39:38) 在单分区内过滤密集数据时，依赖 DynamoDB Filter Expression 会导致读取并计费整个扫描范围的所有项目后再剔除非匹配项，从而在大型分区中造成昂贵的读取开销。Houlihan 指出应将多属性拼接为复合排序键（如将状态与日期拼接为 pending#2019-12-07），利用 begins_with 条件在物理存储节点直接执行精准范围读取。针对 Top-N、计数、平均值等分析性指标，DynamoDB 原生 API 并不支持，强行在数据库层运算容易导致扩展瓶颈。推荐模式是基于 DynamoDB Streams 构建变更数据捕获（CDC）流水线，通过事件触发无状态且具备幂等性的 AWS Lambda 函数，异步更新预聚合汇总表，或将数据转储至 Elasticsearch 与 S3/Athena 以满足全文检索和即席 SQL 分析。 企业级复杂访问与客户端计算卸载 (41:33) Houlihan 通过一个包含 23 种访问模式的企业员工门户架构证明，单表设计完全能支撑复杂的企业级业务流。该模型将大楼会议室、员工档案、项目进度、工单记录组织在统一表中，利用 GSI 1 通过单次按员工邮箱与时间窗口的范围查询，同时拉取包含会议、工时单和工单的统一门户视图。对于会议室预订这类需要比对现有配置与冲突占用的逻辑，架构采用两次轻量查询分别获取房间元数据与时间段内活跃预订，将两组结果直接推送到客户端完成差分可用性计算，而非在应用服务器层循环合并，从而显著缩减了应用服务器集群规模。在保险报价版本控制的真实调优案例中，架构师将原本每次修改重写 50 KB 完整对象的做法改为仅持久化增量 Delta 对象并在前端组装，成功将预置容量从 1,000 WCU 骤降至 50 WCU。 Houlihan 总结认为，NoSQL 并没有完全废弃关系型数据库，合理的架构分工应是将已知、高并发且要求预测性延迟的 OLTP 工作负载部署在云原生 NoSQL（如 DynamoDB）上，而将不可预测的探索性即席查询与复杂 OLAP 分析留给关系型引擎；同时，团队应借助 NoSQL Workbench 等可视化工具加速实体关系向单表键值结构的迭代演进。

## Background 本内容由 AWS Events 于 2019 年 12 月 7 日在 YouTube 平台上发布，记录了 AWS re:Invent 2019 大会上编号为 DAT403-R1 的深度技术演讲。视频描述指出，该分享面向 Amazon DynamoDB 的进阶用户，系统总结了包括亚马逊零售业务在内的大型客户的最佳实践与架构实现，重点剖析全局二级索引（GSI）分片与重载、物化查询图计算、复合键关系建模及事务工作流等高级设计模式。 ## TL;DR AWS NoSQL 首席技术专家 Rick Houlihan 结合亚马逊将 12,000 多个微服务从 Oracle 迁移至 NoSQL 的实战经验，深度解析了高并发分布式场景下的数据建模方法论。他指出硬件成本结构反转使得反范式化成为必然，并在阐明 OLTP 与 OLAP 边界的基础上，详细演示了如何通过单表设计（Single-Table Design）、模型连接（Model Joins）、泛型键与 GSI 重载来替代代价高昂的关系型 JOIN。演讲还结合员工门户（支撑 23 种访问模式）与保险报价系统重构等硬核案例，展示了利用写分片、复合排序键及计算下推客户端等手段，实现极端负载下的极致性能与数十倍成本优化。 ## Atmosphere & style 整场演讲节奏明快、信息密度极大，充满了资深架构师举重若轻的自信与辛辣幽默。主讲人以极具煽动力的舞台表现力打破了传统技术布道的枯燥感，时不时抛出调侃友商或戏谑业界的犀利金句，引得全场笑声与惊呼不断。听众不仅能感受到来自万级微服务实战打磨出的绝对工程底气，更能在层层递进的架构推演与反直觉结论中体会到认知被颠覆的智力冲击。 ## Bulletin - [00:00](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=0s) 亚马逊通过将 12,000 多个内部微服务从 Oracle 迁移至 NoSQL 并下线 3,000 多个实例，验证了现代计算环境中“反范式化以节约 CPU 算力”远优于传统为节省昂贵磁盘而设计的关系型范式。 - [08:26](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=506s) DynamoDB 在 2019 年 Prime Day 期间实现了单表 5,450 万 TPS 的极限吞吐，且借助请求路由集群本地缓存机制，高并发压测下甚至呈现出“负载越高、响应越快”的性能特征。 - [14:45](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=885s) 单个逻辑分区的硬性物理写入上限为 1,000 WCU/秒（读取上限 3,000 RCU/秒），面对超高密度聚合场景必须使用加盐写分片（Write Sharding）来分散热点。 - [09:41](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=581s) 通过在单一主表中使用通用主键（PK/SK）和二级索引重载（GSI Overloading），开发者可以实现真正的多实体“预联接（Pre-join）”，用单次查询高效支撑数十种异构访问模式。 - [53:19](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=3199s) 某保险报价系统通过将“全量 50 KB 对象版本复制”重构成“仅存增量并在客户端计算最新视图”，将预置写入容量从 1,000 WCU 暴降至 50 WCU，服务端集群规模缩减约 5%。 ## Key takeaways - [02:17](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=137s) 硬件经济学的逆转推动了 NoSQL 的崛起：1980 年代存储昂贵促使 RDBMS 用 CPU 算力进行范式化去重，而当下存储极廉价、CPU 算力昂贵，使得反范式化预聚合成为高性价比架构。 - [04:35](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=275s) OLTP 与 OLAP 边界明确：可预测且高频的联机事务处理（OLTP）应使用围绕访问模式建模的 NoSQL；不可预测的即席查询与分析（OLAP）才适合关系型数据库。 - [09:41](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=581s) 引入排序键（Sort Key）的复合主键允许在单分区内构建异构集合，通过模型层面的预联接（Model Join）彻底消除了运行时昂贵的关系型多表关联。 - [12:58](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=778s) 全局二级索引（GSI）投影属性需严格遵循按需原则（KEYS_ONLY 或精简 INCLUDE），无脑选择 ALL 会成倍增加主表写入时的 WCU 消耗并加倍存储成本。 - [14:45](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=885s) 应对高频写入热点必须采用写分片（Write Sharding / Salting），且跨分片的扇出查询（Scatter-Gather）逻辑应完全封装在数据访问层 API 内，对业务透明。 - [17:22](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=1042s) 采用泛型键（PK/SK）进行 GSI 重载（Index Overloading），可以在单个索引结构中容纳多种异构实体的访问模式，大幅削减预置容量与运维告警开销。 - [20:53](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=1253s) DynamoDB 请求路由集群在持续承接高流量时会将表拓扑元数据热缓存至本地内存，从而展现出“负载越高、延迟越低”的反直觉特性。 - [23:09](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=1389s) DynamoDB 提供 5 分钟突发容量缓冲池（Burst Bucket），能借用闲置配额平稳吸收短时数百万 WCU 的瞬时流量洪峰，避免传统自建集群的雪崩离线。 - [26:09](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=1569s) 不存在所谓的“非关系型数据”：所有业务数据天生具备关联性，NoSQL 的精髓在于通过键翻转（Inverted Index）构建无向图，低成本承载多对多关系遍历。 - [31:07](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=1867s) 文档数据库与宽列数据库在建模本质上完全等价：非索引属性应打包为单一顶层 JSON Blob，仅将需参与索引过滤的字段提升至顶级属性以优化落盘与查询性能。 - [03:51:28](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=13888s) DynamoDB Stream 配合 Lambda 提供至少一次处理的 CDC 管道，是维护 Top N、Sum、Count 等实时指标聚合以及异构同步到 Elasticsearch 或 S3/Athena 的最佳架构模式。 - [03:53:38](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=14018s) 利用拼接的复合排序键配合 begins_with 前缀查询可在高密度分区内实现细粒度精确读取，避免 Filter Condition 扫描整个大分区造成的巨量吞吐浪费与计费惩罚。 - [03:55:33](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=14133s) 单表设计（Single-Table Design）足以优雅支撑 23 种以上复合业务访问模式，并在业务演进时通过追加属性与数据层 ETL 持续演进，无需全盘推翻重构。 - [45:51](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=2751s) 充分利用客户端充沛的 CPU 算力进行资源差集计算（如房间可用性比对）和增量数据拼装，可直接下放服务端压力并显著缩减应用服务器集群规模。 - [50:11](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=3011s) 在层级结构建模中，使用路径拼接（如 /City/Building/Floor）配合 GSI 上的 begins_with 查询，可以高效实现任意深度的组织架构或分类子树聚合。 ## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=0s) 开场：告别 Oracle 与 NoSQL 数据建模的崛起 演讲者 Rick Houlihan（AWS NoSQL 首席技术专家/Principal Technologist）在拉斯维加斯的现场向观众致意，并幽默地调侃道，在拉斯维加斯这个充满各色娱乐活动的夜晚，大家竟然愿意抽出时间来听一场关于 NoSQL 数据建模的技术分享，实在令人意外。他介绍自己在 AWS 专注于 NoSQL 数据建模多年，曾协助建立北美的专业实践团队，并在内部的“黑带团队”（Blackbelt Team）工作，参与主导了将亚马逊超过 12,000 个微服务从 Oracle 数据库迁移至 NoSQL 的庞大工程。 在这次里程碑式的技术转型中，亚马逊共关闭了超过 3,000 个 Oracle 数据库实例。Rick Houlihan 强调，团队在这一迁移历程中积累了大量的实战教训与经验，这也是本次演讲的核心内容。虽然部分基础内容在去年的大会上有所提及，但本场分享会进一步深入探讨业界鲜有人知的 NoSQL 复杂数据建模技术。他声明本场分享不会过多关注基础 API、功能和特性列表，而是聚焦于真实的业务应用、真实的访问模式（Access Patterns）以及如何在经济效益最大化的前提下构建高可用的生产系统。 - 主讲人：Rick Houlihan（AWS NoSQL 首席技术专家） - 亚马逊内部迁移规模：将约 12,000 个内部服务从 Oracle 迁移至 NoSQL，共关闭 3,000+ 个 Oracle 实例 - 核心主题：NoSQL 复杂数据建模与真实业务访问模式设计 --- ### [02:17](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=137s) 数据处理简史：从记账账本到关系型数据库的演进 ```mermaid flowchart LR accTitle: 数据处理架构演进历史 accDescr: 展示从人工大脑到账本、打孔卡、关系型数据库再到现代 NoSQL 的演化过程，揭示数据压力驱动的硬件成本权衡。 subgraph Early["早期与机械时代"] A["人类大脑 (单用户/无容灾)"] --> B["账簿会计记账 (延用数千年)"] B -->|"1880年普查耗时8年"| C["打孔卡机 (1890 赫尔曼·何乐礼)"] end subgraph Modern["计算机与云时代"] C --> D["磁带与块存储系统"] D -->|"1980年代: 存储昂贵/CPU廉价"| E["关系型数据库 (范式化以去重)"] E -->|"今天: 存储廉价/CPU昂贵"| F["NoSQL (反范式化以省CPU)"] end ``` 理解为什么要转向 NoSQL，必须回顾数据处理技术的发展史。Rick Houlihan 将整个数据处理历史概括为“数据压力”（Data Pressure）驱动下的技术触发曲线。所谓“数据压力”，即系统被要求处理的工作负载、处理速度与执行成本；当现有技术无法以足够快或足够便宜的方式满足需求时，高数据压力就会触发新技术的发明。 最早的“数据库”是人类的大脑——虽然眼睛睁开时高可用，但持久性极差、毫无容错机制，本质上只是个单用户系统，无法承载企业级应用。随后人类发明了账簿会计记账法（Ledger Accounting），这一结构化存储方法支撑了公共和私营部门数千年的运转。直到 1880 年的美国人口普查，统计员赫尔曼·何乐礼（Herman Hollerith）用了整整 8 年时间才处理完上一个十年的数据。面对系统崩溃的边缘，何乐礼在 1890 年发明了机器可读的穿孔打孔卡及打孔卡分拣机，极大缩短了处理周期。此后，纸带、磁带、分布式块存储以及随机访问文件系统相继诞生。 到了 20 世纪 80 年代，关系型数据库（RDBMS）应运而生。Rick Houlihan 明确指出，人们发明关系型数据库的根本原因是为了减轻数据中心存储子系统的压力： > “When we got to the 1980s, it's very important to understand why we invented the relational database was to decrease the pressure on the storage subsystems in the data center.” 在 1980 年代，存储极其昂贵，而 CPU 相对便宜；关系型数据库通过范式化（Normalized Data Model）将数据在磁盘上高度去重，以牺牲 CPU 算力（进行复杂的表连接）来节约宝贵的磁盘空间。然而 30 到 40 年后的今天，情况发生了彻底反转：磁盘存储成本降至每千兆字节（GB）几美分，而 CPU 算力则昂贵得多。继续使用一套为几十年前硬件环境优化的系统架构，在现代数据中心里已不再具备经济合理性。 - 1880 年美国人口普查：纯人工账簿统计耗时 8 年（普查周期为 10 年） - 1890 年：赫尔曼·何乐礼发明穿孔打孔卡系统，大幅降低处理时间 - 1980 年代痛点：磁盘存储成本高昂，CPU 闲置率高，推动了基于去重的范式化 RDBMS 诞生 - 现代现状：存储成本极低（每 GB 仅数美分），CPU 算力成本高昂，促使架构向反范式化的 NoSQL 转移 --- ### [04:35](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=275s) 技术采纳曲线与思维模式转换：OLTP 对比 OLAP 在将业务系统向 NoSQL 迁移的过程中，亚马逊也曾踩过严重的陷阱。团队在最初迁移小型工作负载时感觉非常顺利，但随着将规模庞大的关键系统搬上 NoSQL，系统性能却出现了断崖式下跌——运行效率既不高，成本也没有下降。经过深度剖析，团队得出了一个惨痛教训：“在小规模时把事情做烂很容易，而且你还会误以为自己做得很好。”根本问题在于开发团队缺乏对新技术的正确理解，直接生搬硬套了关系型数据库的范式化建模思路。 Rick Houlihan 强调，在分布式 NoSQL 中使用范式化数据模型，一旦业务规模横向扩展，系统必将遭遇灾难性的性能瓶颈。技术采纳曲线表明，当新技术刚出现时，缺乏经验的使用者往往体验极差；但随着行业认知成熟与技能普及，NoSQL 建模并不会比传统关系型建模更难，它仅仅需要一种完全不同的思考方式。 关系型数据库与 NoSQL 各自适用的业务边界非常清晰： 1. **OLAP（联机分析处理）**：适合关系型数据库或即席查询引擎（Ad-hoc Query）。当业务无法预知具体的访问路径，需要通过灵活的查询拼装执行统计分析时，关系型技术优势明显。 2. **OLTP（联机事务处理）**：适合 NoSQL。这类系统具有高度可预测且重复出现的访问模式。NoSQL 通过反范式化（Denormalization）将数据紧密围绕查询模式进行编排，极大地简化了数据访问路径并消除了运行时的昂贵 Join。由于开发者所编写的应用中有 90% 均服务于日常业务流程（属于 OLTP 范畴），因此掌握 NoSQL 建模是现代开发者最为关键的技能之一。 > “If you use a normalized data model, you are going to have pain at scale when you expand and you scale out your operations.” --- ### [07:11](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=431s) 全托管服务价值与 DynamoDB 的极致规模能力 在传统自建 NoSQL（如 MongoDB 分片集群）架构下，运维代价十分巨大。Rick Houlihan 举例指出，在自建集群规模扩大到 15 至 20 个分片（Shards）后，仅仅为了给集群扩容上线一个新的分片，可能就需要耗费数周甚至数月的时间来进行数据重新平衡与节点同步。全球许多大型企业在面临这种自建痛点后，纷纷转而寻求全托管的云原生解决方案。 Amazon DynamoDB 的核心价值在于其“云原生”与“全托管”属性。用户完全不需要耗费工程资源去管理底层节点、修复物理存储损坏设备，或是处理复杂的分片扩展逻辑。对于企业而言，将高昂的人力资源从基础设施维护中解放出来，投入到核心业务的开发中，具有无可比拟的经济优势。 在扩展性与性能表现方面，DynamoDB 无论面对何种规模的数据与请求，都能保持一致的个位数毫秒级响应延迟。Rick Houlihan 透露，在真实的生产环境中，DynamoDB 单表即可轻松支撑超过每秒 700 万次事务请求（TPS），展现出了自建传统系统难以匹敌的超大规模吞吐能力。 - 传统自建 NoSQL 痛点：MongoDB 等系统在达到 15–20 个分片后，扩容新分片往往需要数周至数月的数据同步时间 - 全托管核心收益：无需管理物理节点、无需维护存储驱动、全自动化分区与数据分发 - 极限吞吐能力：单表生产级实测吞吐量超过 7,000,000 TPS（Transactions Per Second），且保持低延迟与强一致性 ### [08:26](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=506s) 超大规模并发与 NoSQL 核心原语解析 演讲者首先以令人震撼的业务数据强调了 DynamoDB 的水平扩展极限。在 2019 年的亚马逊会员日（Prime Day）期间，Amazon CDO 表的处理能力达到了令人惊叹的每秒 5450 万次事务（54.5 million TPS）。不仅如此，在紧随其后的黑色星期五（Black Friday）和网络星期一（Cyber Monday），这项纪录被彻底打破，流量规模甚至更为庞大。演讲者直言，这种级别的超大规模扩展是“传统 NoSQL 供应商根本无法企及的”。在确立了技术护城河之后，讨论迅速切入 NoSQL 的核心建模原语。 在 NoSQL 数据建模中，最基础的构造就是表（Table），即存放对象的仓库（Object Repository）。放入表中的条目（Item）必须具有一个强制属性——分区键（Partition Key），除此之外的其他属性均非必需。如果只定义了分区键，表本质上就是一个键值（Key-Value）存储：以特定分区键发起查询，系统只会返回一个单一对象。这与文档型数据库（例如 Amazon DocumentDB）极为相似，在 DocumentDB 的集合（Collection）中存入文档，依靠 `_id` 属性唯一标识并执行键值读取。然而，宽列模型（Wide-column Model）与纯文档模型的根本分水岭在于：能否在表中引入排序键（Sort Key）。 - 2019 年 Prime Day 期间，Amazon CDO 核心表承载了高达 5450 万 TPS 的峰值请求。 - 仅有分区键时，表退化为严格的 Key-Value 访问模式，一次等值查询仅返回一条记录。 - 排序键的引入构成了复合主键（Composite Primary Key），彻底拓宽了数据查询的维度与表达力。 ### [09:41](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=581s) 复合主键：用模型连接代替关系型 Join 一旦在分区键的基础上增加了排序键，该分区就演变成了一个多对象集合（Collection）。此时，分区键界定了一个逻辑对象组，而排序键则在该分区内部唯一标识具体对象，并天然支持各种范围查询（Range Query）。演讲者举了一个经典业务场景：将分区键设为客户 ID（Customer ID），排序键设为订单创建时间（Order Date）。当需要查询该客户最近 30 天的订单时，只需指定分区键等于特定客户、排序键大于 30 天前的时间戳，数据库就能在单一分区内高效检索出这一批订单，无论该分区下沉淀了多少条历史数据。更深层次的转变在于：该集合不仅可以容纳同构数据（Homogeneous Collections），更能存放异构数据（Heterogeneous Collections）。 演讲者指出了关系型开发者对 NoSQL 的普遍误解，并对系统架构原理进行了透彻阐述。在传统关系型数据库处理未建立索引的多表连接时，底层也是动态构建哈希表，通过外层循环逐条在哈希表中做查找；而 DynamoDB 本质上就是预先将数据组织并落盘在分布式哈希表（Distributed Hash Table）的结构中。 “In essence what we've done is joined objects into a partition and we're going to start to decorate these objects with additional attributes that we're going to index to create additional aggregations or grouping of objects and so if you think about what I'm describing here I'm really describing the ability to join objects so this is what this is what we do in NoSQL it's not that we can't join objects we create model joins right we model our data so that we can group these objects into partitions on the table partitions on the GSIs and we're going to decorate these objects with common attributes that are going to tie them together when I query on various access patterns we're going to show you how to do this but this is not so different than the way actually relational databases work when they actually have to join data that's not indexed right they create a hash table and then they do lookups against the hash table as they iterate through the inbound collection or the or the outer collection so what we're really doing is storing data in that in that distributed hash table structure ok” ```mermaid flowchart TD accTitle: 单分区内的模型连接架构 accDescr: 展示客户元数据、订单及订单明细如何在单一分区键下利用排序键进行高效范围聚合 subgraph Partition["单一物理/逻辑分区 (Partition Key: Customer ID)"] direction TB M["客户元数据 (Customer Metadata)"] O1["订单 #1 (Order Date: 10天前)"] O1_Item["订单项明细 (Order Items)"] O2["订单 #2 (Order Date: 20天前)"] O3["旧订单 (Order Date: 60天前)"] end Query["单次查询 (Query: Order Date > 30天前)"] --> Partition Partition --> Results["返回：客户元数据 + 最近30天订单及商品明细"] ``` ### [11:46](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=706s) 单分区异质聚集消除多表复杂编排 在现实业务设计中，异构实体（Heterogeneous Items）的混合存储发挥着巨大威力。演讲者展示了一个设计范例：分区键为客户 ID，排序键由时间戳结合前缀组成。在同一个客户分区内，不仅存放着带有时间前缀的订单记录和订单明细条目，甚至还包括不带时间戳前缀的客户基础元数据。当系统下发一条“获取 30 天以内所有数据”的 Query 请求时，该范围条件能够一次性捕获客户基础属性、该期间生成的所有订单，以及这些订单所属的具体明细。 这种建模彻底颠覆了传统关系型架构的查询模式。在传统模式下，应用端必须发起极其繁重的编排调用：先查客户表读取元数据，再查订单表获取列表，最后还要遍历订单结果集去订单明细表查具体商品。这一系列多轮往返网络请求（Round-trips）带来了极大的复杂度和延迟开销。而在 DynamoDB 中，只要把不同维度的异构对象预先收拢并装配到同一个分区内，仅凭单次物理查询即可完整提取出整套业务上下文。这种模式极大地压缩了查询路径，也是 NoSQL 高并发架构的核心精髓。 “This is the crux of NoSQL design: what I'm trying to do is create collections of multiple objects or multiple types of objects that I can query with single queries.” - 传统多表模型缺陷：需要按层级串行或并发多次查询，并在应用内存中完成拼接，消耗大量 I/O 和网络开销。 - 单表异质聚集优势：通过合理设计排序键前缀，一条 Query 语句直接返回父项与子项的所有联动信息。 ### [12:58](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=778s) 全局二级索引（GSI）机制与投影开销控制 当业务需要从另一个完全不同的维度检索数据时，全局二级索引（GSI）便派上用场。演讲者将 GSI 的实质形象地概括为一个“随着主表数据插入和变更而由系统自动异步复制的辅助表”。AWS 为 GSI 数据复制提供了 100% 的 SLA 保证。尽管 GSI 保持的是最终一致性（Eventual Consistency），但在配置得当的情况下，其 P99 复制延迟能够稳定控制在 10 毫秒以内。演讲者借此对比了其他开源竞品（如 Apache Cassandra），指出 Cassandra 官方甚至建议用户避免使用二级索引，因为它们难以维持全局一致性；而 DynamoDB 的索引一致性机制则是其最核心的差异化优势之一。 伴随 GSI 而来的另一个重要设计决策是属性投影（Projections）。在构建 GSI 时，开发者可以指定将主表中的哪些属性投射到索引中：可以是全部属性（ALL）、部分属性（INCLUDE），或者仅仅是主键本身（KEYS_ONLY）。演讲者在此发出了严肃的成本警告：投影并非零代价。若选择投射全部属性，不仅意味着存储成本直接翻倍，更致命的是，向主表执行写入操作时所需的写容量单位（WCU）也会翻倍同步消耗。因此，严谨的控制成本策略是精细化分析访问模式，只投射 downstream 操作绝对必须的字段，绝不多投射冗余数据。 “If I only need to know the items that matched, then only project the keys; if I need to know additional data, then project what you need, but don't project more.” - GSI 复制延迟在合理架构下 P99 可达 10ms，具备极强的高吞吐承载能力。 - 投影 ALL 属性会成倍放大主表写入的 WCU 消耗，并导致底层存储成本翻倍增长。 - 遵循“按需投影”准则：若只用于查找定位对象，仅投影键（KEYS_ONLY）；若需辅助读取，精准限定投影字段列表（INCLUDE）。 ### [14:45](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=885s) 写分片（Write Sharding）应对超高密集度聚合 在深入讲解底层分区物理机制时，演讲者指明：DynamoDB 依靠无序哈希索引将主键空间打散，并在需要更大存储容量或更高吞吐量时对键空间进行切分。然而，集群中的每个底层物理存储节点（Storage Node）本身所能承载的吞吐量存在物理上限。为了在面对海量吞吐时依然能无缝扩容，核心就在于将写负载均匀分散。对于极高密集度的聚合场景（Highly Dense Aggregations），必须引入写分片（Write Sharding / Key Salting）技术。 演讲者以亚马逊自身的零售系统为例：假设业务方需要建立一个 GSI 来实时聚合并查询“来自某一渠道来源（Source，例如线上渠道 Online）的所有订单”。在超大规模电商活动中，这类全局来源字段极度密集。而在 DynamoDB 中，单个分区键的写入容量上限大约是每秒 1000 WCU。如果简单地直接拿“Online”作为 GSI 的分区键，超高并发的写入流量会瞬间打满该热点分区的 1000 WCU 限制，导致系统产生限流和严重节流（Throttling）。解决这一死锁的唯一手段是在建索引时对分区键进行“加盐”（Salting），即追加随机或散列后缀（如 `Online_1`, `Online_2`... `Online_N`），将单点写压力分流到多个分片上。 在实施写分片之后，读路径自然变得复杂，因为客户端必须跨多个分片并发扇出读取（Scatter-Gather）才能组装完整数据。演讲者强烈建议将这套复杂的写分片和跨分片并行查询逻辑完整封装在数据访问层 API（Data Layer API）内部，坚决对业务应用层透明。演讲者借此揭秘了 AWS 内部多项基础设施的运维设计：用户向 AWS 提工单申请调高某些资源配额（如 IAM 策略或配置项上限）时，后台之所以需要审批，往往正是因为该客户的写入量逼近了单个物理分区的 1000 WCU 红线，底层必须介入实施写分片改造以避免容量枯竭。 ```mermaid flowchart LR accTitle: 写分片（Write Sharding）分流与数据层并行读取 accDescr: 展示高频写入如何通过加盐分流到多个分片，以及数据层如何并行查询组装给客户端 subgraph Ingestion["高频写入端 (Write Traffic > 1000 WCU)"] W["来源字段 (Source: Online)"] --> Salt{"分片散列 (Salting)"} end subgraph GSI_Shards["二级索引分片 (GSIs)"] S1["Online_1 (上限 1000 WCU)"] S2["Online_2 (上限 1000 WCU)"] SN["Online_N (上限 1000 WCU)"] end Salt --> S1 Salt --> S2 Salt --> SN subgraph DataLayer["数据访问层 API (Data Layer API)"] ParallelQuery["并行读取与聚合 (Parallel Processing)"] end S1 -.-> ParallelQuery S2 -.-> ParallelQuery SN -.-> ParallelQuery ParallelQuery --> Client["业务客户端 (Client Application)"] ``` - DynamoDB 单个分区键的硬性写入瓶颈为 1000 WCU/秒，密集聚合极易诱发热点瓶颈。 - 写分片通过附加后缀哈希（Salting）将热点聚合分散到多个独立分区。 - 业务架构应当在数据访问层（Data Layer API）内部封装读写分片逻辑，屏蔽跨分片扇出聚合的底层复杂度。 ### [17:22](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=1042s) GSI 重载与泛型键模式 在处理全局二级索引（GSI）的设计时，通常需要应对分片与多种查询需求。当用户传入某些查询条件时，底层 API 负责处理写入分片，或从多个分片中跨片提取数据。如同主表一样，GSI 也应采用泛型键（Generic Keys，例如通用的 PK 和 SK）。使用泛型键的核心优势在于可以在同一个 GSI 中存入多种不同类型的项目，实现索引的高效复用。 讲师指出，业务中通常存在多种索引访问模式，例如按订单来源查询，或者按商品 ASIN 查询订单明细。如果为每种访问模式分别创建一个 GSI，运维成本与容量开销会成倍增加：必须为每个索引单独配置预置吞吐量桶、设置独立的 CloudWatch 告警，而且不同访问模式很少完全并发发生，彼此独立的容量无法共享。讲师极力推荐通过“GSI 重载”（Index Overloading）复用同一个全局二级索引，以此减少线上需要维护的基础设施数量：“It's more efficiency to reuse the GSI so I reuse a lot of GSIs to reduce the number of constructs we have to maintain in production.” 在查询重载的 GSI 时，返回哪些数据完全由分区键（Partition Key）和排序键（Sort Key）的条件决定。不同业务实体是否会发生键冲突，取决于键的设计。如果两种实体的分区键存在冲突风险，可以通过添加实体类型前缀（如 `ORDER#` 或 `ITEM#`）进行隔离；但在当前案例中，ASIN 和订单来源的取值空间完全正交，根本不会产生碰撞。使用来源作为分区键时，查询拿到的是订单项；使用 ASIN 作为分区键时，查询返回的则是包含该 ASIN 的具体订购明细。 ```mermaid flowchart TD accTitle: GSI Overloading Pattern accDescr: 多个应用查询请求复用同一个使用泛型键 PK 和 SK 的 GSI 物理表 subgraph Clients["应用访问请求"] Q1["查询请求: 按来源 (Source) 检索订单"] Q2["查询请求: 按商品 (ASIN) 检索明细"] end subgraph GSI["单个重载的 GSI (Generic Keys)"] G_PK["GSI-PK (存储 Source 或 ASIN)"] G_SK["GSI-SK (存储状态、时间戳等)"] end subgraph Results["返回的数据结果"] R1["订单主信息列表"] R2["订单商品明细列表"] end Q1 -->|"无冲突路由"| G_PK Q2 -->|"无冲突路由"| G_PK G_PK --> G_SK G_SK -->|"条件: Source 匹配"| R1 G_SK -->|"条件: ASIN 匹配"| R2 ``` --- ### [19:06](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=1146s) 分布式热键与均匀打散负载 NoSQL 的横向扩展本质上依赖将数据和流量均匀打散到整个键空间。如果未能将数据分散，系统就会出现极端的热键（Hot Keys）现象。讲师展示了一张服务节点的热力图：Y 轴是服务器编号，X 轴是时间线。在反模式的图谱中，11 号服务器被高密度的访问打得通红、满负荷运转，而其他节点几乎处于完全闲置状态。这种高频访问打在极少数甚至单个键上的情况，是典型的设计缺陷。 为了从 DynamoDB 或任何分布式 NoSQL 数据库中榨取最大性能，核心在于让尽可能多的存储节点和分片参与工作。虽然理想情况下流量应在时间和键空间上平滑分布，但现实中往往会遭遇“惊群效应”（Thundering Herds），如黑色星期五（Black Friday）和网络星期一（Cyber Monday）的流量洪峰。讲师展示了一个正向范例的热力图：约 450 台服务器在整个键空间内持续承载 40,000 到 50,000 WCU（写入容量单位），热力图分布极其均匀，如同老式电视机切台时满屏跳动的雪花白噪声。讲师笑着调侃道：“This looks like the white noise on the TV when you change between channels, right?... I have an old man, they can't help.” - **反模式现象**：单个节点处理全量请求，其余节点空闲，极易触发单个分区物理上限导致节流。 - **最佳实践**：通过合理的分区键设计将写入分散到 450+ 节点，均匀承载 40,000 至 50,000 WCU。 - **监控工具**：CloudWatch 现已提供特定功能用于观测承压键（Hot Keys）的指标，辅助排查局部倾斜问题。 --- ### [20:53](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=1253s) 云原生按需弹性与越压越快的性能特异性 在资源调配方面，传统的自建或托管型数据库（如关系型数据库，或 MongoDB、Cassandra 等旧一代 NoSQL）要求运维人员必须按照业务峰值预置资源。然而现实中，企业内部数据中心的平均资源利用率仅在 10% 左右，这意味着长年累月有大量的服务器成本被白白浪费。相比之下，开启了 Auto Scaling 的 DynamoDB 能够动态贴合业务负载曲线。讲师以 Amazon 某个真实履约中心（Fulfillment Center）的业务服务为例，展示了开启自动伸缩前后的显著对比：“The amount of cost that they're saving is around 40 percent. I think most of our fulfillment center applications are saving around 40%.” 除了成本节省，DynamoDB 在大规模高负载下的表现打破了传统数据库的常识。讲师强调了一个看似反直觉的性能特异性：“I don't know what other database you're going to look at and say it gets faster when you throw more load at it.” 在针对 DynamoDB 进行的人工合成压测中，负载一路攀升至每秒超过 100 万次请求（1M+ RPS），延迟反而持续下降并稳定在极低的毫秒级基线。 这一现象的底层原理在于其路由架构：DynamoDB 拥有庞大的分布式请求路由集群（Request Router Fleet）。当路由节点初次接触某张表的请求时，必须先访问内部的配置服务，加载该表的分区策略、存储节点物理拓扑等元数据；而随着海量请求持续涌入，这些元数据被高密度地缓存在每个路由节点的本地内存中，后续请求免去了跨服务查询，处理路径大幅缩短，整体响应随之提速。许多客户在 APM 监控中看到这个曲线时甚至误以为是监控打点出错，专门跑来向 AWS 团队确认其真实性。 ```mermaid flowchart LR accTitle: DynamoDB Request Router Caching Mechanism accDescr: 请求路由集群从首次访问需要查询配置服务，转变为缓存命中后直接访问存储节点 subgraph Routing["请求路由层 (Request Routers)"] RR1["冷请求 (无缓存)"] RR2["热请求 (高并发压测)"] end subgraph Internal["元数据与后端"] CS["配置服务 (Config Service)<br/>存储策略与分区拓扑"] SN["存储节点 (Storage Nodes)"] end RR1 -->|"1. 额外网络开销: 拉取元数据"| CS CS -.->|"写入本地缓存"| RR1 RR1 -->|"2. 路由请求"| SN RR2 ==>|"直接命中本地内存元数据<br/>(延迟大幅下降)"| SN ``` --- ### [23:09](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=1389s) 突发容量桶与“非关系型数据”的本质探讨 DynamoDB 还具备一项名为“突发容量桶”（Burst Bucket）的独特机制，允许实例在短时间内交付远超预置配额的吞吐量。讲师分享了一个短信服务提供商在超级碗（Super Bowl）期间的真实故事：该客户原本预置了接近 600 万 WCU 的容量，但在比赛哨响的瞬间，峰值吞吐瞬间飙升突破了 1000 万 WCU。令人惊讶的是，客户在当天完全没有察觉到流量超标，直到第二天复盘 CloudWatch 监控时才惊恐地发现吞吐曲线超额，并急切询问为何没有被限流节流。 原因正是 DynamoDB 具备一个 5 分钟的突发容量缓冲池，它允许客户借用过去 5 分钟未被占满的闲置容量来抵御瞬时尖峰。讲师对比道，若换作 MongoDB、Cassandra 或传统关系型数据库面对瞬间暴涨数百万的突发吞吐，系统往往会直接崩溃离线：“What happens with MongoDB when you do this? What happens with Cassandra? What happens with your relational database when you do this, right? It goes offline... it goes bye-bye.” 内存压力瞬间见顶，响应延迟彻底失控。 随后，话题切入到数据建模的核心：实体关系建模。讲师坦言，无论开发何种系统（社交网络、文档管理或 IT 运维系统），业务的核心始终由实体间的关系（Relationships）所驱动。那种认为切换到 NoSQL 就可以无脑塞入巨大 JSON Blob 文档的想法是非常糟糕的。甚至最纯粹的 Key-Value 模式，也是一种严格的一对一关系。讲师甚至开玩笑反问全场是否存在真正意义上的“非关系型数据”： > “I haven't found it as a matter of fact. I would love to have a non-relational banking application because I would log in over and over again until I saw the numbers that I liked, right? And then I would write my checks. I'd go, okay, hopefully it's Jeff Bezos' account, I'll be really happy with that.” - **突发容量能力**：支持借用前 5 分钟闲置的吞吐量，成功承接从 6M 到 10M+ WCU 的超级碗瞬时洪峰。 - **大文档反模式**：在 NoSQL 中直接存储巨型 JSON 文档是严重的设计误区。 - **建模本质**：数据天生具备关联性，在关系型数据库中我们通过范式化、复杂多表 JOIN 和高昂的 CPU 开销动态拼装视图；而在 NoSQL 中，必须围绕访问模式直接对数据层级进行针对性建模。 ### [26:08](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=1568s) 关系型查询的复杂度困境与 NoSQL 的“预连接”设计 讲者带领大家分析了关系型数据库在组装复杂数据时的代价。当我们在关系型数据库中执行跨表查询时，每一层关联都在急剧拉高系统的计算开销。最简单的一对一关系可能只是 $O(\log N)$ 或 $O(M \log N)$ 的时间复杂度，但随着连接表的增多，复杂度会直线上升。即便建立了完备的索引且只走高效的嵌套循环连接（Nested Loop Join），避免了恶劣的哈希连接（Hash Join）或合并连接（Merge Join），复杂度的攀升依然无法逆转。 ```mermaid flowchart TD accTitle: 关系型哈希连接与 NoSQL 预连接对比 accDescr: 对比关系型数据库在查询时动态创建哈希表的高昂代价与 NoSQL 事先按哈希分区组织数据的预连接优势。 subgraph RDBMS["关系型数据库运行时开销"] R1["扫描大表与输入集合"] --> R2["在排序列上动态构建哈希表"] R2 --> R3["反复迭代查询哈希表获取匹配项"] end subgraph NoSQL["NoSQL 事先预连接"] N1["按访问模式设计 Partition Key"] --> N2["将关联行作为 Item 放入同一分区"] N2 --> N3["单次查询常数时间 O(1) 获取完整数据集"] end ``` 针对关系型数据库在高负荷下的极端场景，讲者剖析了哈希连接的本质：当数据库面对大表关联时，它会先全表扫描，在排序列上临时构建一个哈希索引，接着不断迭代输入集合去查询该哈希表以拉取匹配项。讲者反问道：既然数据库引擎本身都认为“哈希表是支持反复查询最高效的数据结构”，为什么我们不从一开始就将数据存入哈希结构？NoSQL 本质上就是构建在分布式哈希表之上的系统，其查询在理论上具备恒定的常数时间复杂度（Time Constant）。通过将关系型表中的行数据事先“预连接”（Pre-join）并组织进特定的分区（Partition）中，系统彻底省去了查询时的动态计算开销。 - 关系型多表关联即使在最优索引配置下，时间复杂度也会随着关联表数量的增加而持续恶化。 - 关系型哈希连接不仅耗费大量 CPU 和内存去临时构建哈希表，还需执行全表扫描。 - NoSQL 分布式哈希表消除了查询时的计算惩罚，查询单本书籍或单张唱片时，仅凭分区键（Partition Key）即可在常数时间内返回结果。 --- ### [28:18](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=1698s) 基于反向索引（GSI）的有向图向无向图演进 通过具体的电影与演员模型，讲者展示了如何在 DynamoDB 中优雅实现图结构和多对多关系。在基础表设计中，电影分区（Movie Partition）内部包含了多个由演员姓名排序的条目，这构成了一个典型的“有向图”（Directed Graph）——电影节点知道自己关联了哪些演员，但演员自身的分区此时却无法反向得知其参演的电影。在电影分区下查询时，第一条数据描述电影自身（以导演姓名排序），随后的每条数据代表一个关联的演员边缘（Edge），记录其姓名、年龄、性别以及所饰演的角色。 讲者在此强调了投影（Projection）与反规范化（Denormalization）的取舍哲学：对于出生日期和性别这类相对不可变的数据（Immutable Data），完全可以直接冗余到边缘条目中以优化读取效率；讲者开玩笑道：“我倒是希望我能改自己的出生年月日，但这根本不可能。”而对于像个人简介（Bio）这样频繁变动的属性，则绝不能冗余到边缘条目中，否则日后演员每次修改简介，系统都必须痛苦地去全量更新其出演过的所有角色记录。 ```mermaid flowchart LR accTitle: 主表与 GSI 的键翻转多对多模型 accDescr: 展示主表以电影为分区键查询参演演员，而反转后的全局二级索引以演员为分区键查询参演电影的双向查询架构。 subgraph BaseTable["主表（电影维度：有向图）"] PK1["PK: 电影名称 (Movie)"] --> SK1["SK: 演员/导演名称 (Actor/Director)"] SK1 --> R1["单次查询: 电影详情 + 演员名单、角色、年龄"] end subgraph GSI["GSI 反向索引（演员维度：无向图）"] PK2["PK: 原 SK (演员/导演名称)"] --> SK2["SK: 原 PK (电影名称)"] SK2 --> R2["反向查询: 演员参演的所有电影 / 导演执导的所有作品"] end ``` 要将这种单向关系打造成完整的“无向图”（Undirected Graph），核心秘诀在于翻转键结构构建全局二级索引（GSI）：将主表的排序键（Sort Key）变成 GSI 的分区键，而主表的分区键则变成 GSI 的排序键。这一翻转瞬间解锁了全新的访问维度——我们不仅能查出演员参演的所有电影、音乐家制作的所有唱片、作家写的所有书籍，还能查询某位导演执导过的全部作品。对于这种建模手段，讲者明确指出： > “This is how we model many-to-many relationships in NoSQL and avoid massive duplication of data.” - 只有不可变数据（如出生日期、性别）才适合反规范化投影至边缘条目，以此优化读取吞吐。 - 可变属性（如履历、简介）严禁复制到各个关系节点，避免产生跨分区同步更新的高昂写放大。 - 主表满足一侧实体的查询需求，翻转键后的 GSI 承载另一侧实体的访问模式，以极小的数据冗余代价支撑双向多对多遍历。 --- ### [31:07](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=1867s) 文档数据库与宽列模型的本质等价性 针对市面上对于“文档数据库（Document DB）与宽列数据库（Wide-column DB）适用场景大相径庭”的炒作，讲者直截了当地予以反驳，指出二者在底层数据建模上根本没有本质区别，唯一的差异仅在于展现形式和索引机制。在文档数据库（如 MongoDB）中，默认主键索引通常是 `_id`，而如果在插入文档时不显式提供 `_id`，系统会自动生成一个 UUID。但讲者指出，UUID 主键在绝大多数业务中都是无意义的废索引，因为没有任何真实客户会在致电客服中心时报出自己的 UUID，客服问的第一句话永远是“您的邮箱地址是什么”。因此，主键必须具备明确的业务语义。 随后，讲者展示了如何将一个带有建筑与楼层复合索引（用于支持如“查询某建筑 7 层所有员工”的子树聚合）的文档模型直接平移到 DynamoDB 的宽列结构中。在 DynamoDB 中查询返回的实体本质上同样是 JSON 结构。更重要的是，DynamoDB 原生支持 JSON 属性，若想追求极致的查询性能，最佳实践是将所有无需建立索引的属性打包成一个整体的嵌套 JSON 属性。 ```mermaid flowchart TD accTitle: 文档模型向 DynamoDB 宽列模型的映射与存储优化 accDescr: 解释将需要建立索引的属性提升到顶级属性，并将海量无索引数据打包为单一 JSON Blob 的高性能存储模式。 subgraph DocModel["逻辑文档属性"] D1["业务主键 (Email / Employee ID)"] D2["层级字段 (Building / Floor)"] D3["非索引扩展属性群 (Preferences, Details...)"] end subgraph DynamoModel["DynamoDB 存储优化结构"] P1["PK: 业务主键"] P2["SK / 属性: 顶级索引键"] P3["单个 JSON Blob 属性: 包含所有非索引数据"] end D1 --> P1 D2 --> P2 D3 --> P3 ``` 底层存储引擎在落盘时必须逐一处理每一个顶层属性。如果将上百个属性打散平铺在 Item 中，引擎的处理开销会显著增加；反之，将非索引属性作为一个大 JSON 存入，DynamoDB 只需将其当成单个 Blob 属性处理，只需将需要作为索引条件的键提升到顶层即可，这种模式在性能和存储经济性上表现最佳。 - 坚决杜绝在文档数据库或 NoSQL 中滥用无业务意义的随机 UUID 作为主键，主键应当能够直接映射关键业务查询。 - DynamoDB 在客户端与底层传输层面依然全面使用 JSON，文档与宽列表格在建模逻辑上完全是同一回事。 - 在 DynamoDB 中提升查询性能的关键架构设计是：仅将参与索引构建的字段提至顶级属性，其余大量元数据全部装入单一的 JSON 对象属性中。 --- ### [33:33](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=2013s) 索引覆盖与投影成本：DocumentDB 与 DynamoDB 的底层机制差异 虽然两者在数据建模上是相通的，但在查询引擎与索引的底层实现上却有显著分歧。文档数据库（如 Amazon DocumentDB 或 MongoDB）具备独立的查询规划器（Query Planner），由它来决定走哪张索引。讲者结合自身在 MongoDB 的工作经历提到，他在实际写查询时经常被迫使用 `$hint` 操作符，因为绝不能任由规划器做出愚蠢的选择去走不对的索引，更要严防出现全集合扫描（Collection Scan）。在文档数据库中执行 `explain` 时，核心评估指标是两个数字：返回文档数（Documents Returned）与扫描文档数（Documents Scanned）。 如果扫描文档数为 0 且返回文档数大于 0，说明该查询是一次完全由索引满足的“覆盖查询”（Covered Query），即所有所需属性全部存在于索引定义中。但如果扫描文档数等于返回文档数，则意味着系统虽然命中了索引，但不得不回表（Collection Fetch）捞取未包含在索引中的扩展属性。在文档数据库中，为了实现属性投影以避免回表，开发者必须把这些属性全部塞进索引定义中，但这会导致索引体积膨胀并严重拖慢写入速度（Insert Expense）。而这也正是 DynamoDB 这类宽列数据库在架构设计上的强大优势所在。 - 文档数据库高度依赖查询规划器，在生产高并发场景下往往需要强制使用 Hint 来规避索引漂移与集合扫描风险。 - `documentsScanned = 0` 是文档数据库覆盖查询的黄金指标，代表无须产生额外的回表数据读取。 - 文档数据库若要实现属性投影，必须将对应字段强制塞入复合索引定义中，这直接增加了每次数据写入的存储与索引维护代价。 ### [03:49:53](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=13793s) 文档数据库与 DynamoDB 索引设计的本质同构 在索引设计与数据写入层面，主讲人指出向索引中添加投影属性并不会让单次写入操作在系统层面变得更加复杂；虽然投影更多属性可能会消耗更多写容量单位（WCU），但系统底层的 CPU 负载并不会因此陡增。随后，他将文档数据库（Document Database）与 DynamoDB 的索引工作机制进行了深度对比。在查询文档数据库时，必须把分片键（Shard Key）作为复合索引的第一项属性，并在查询语句中显式附带该分片键条件，以便将查询精准路由到特定的存储节点上。如果缺失了分片键条件，查询请求就必须广播分发至集群中的所有节点。尽管某些场景下确实需要全局扫描，但绝大多数时候这毫无必要，且大部分节点返回的空结果只会白白浪费集群的宝贵带宽。 与文档数据库依赖复合索引首字段进行路由不同，DynamoDB 要求开发者直接针对具体的访问模式（Access Pattern）定制索引，并在执行查询时使用分区键（Partition Key）的等值条件匹配。这种硬性约束强制开发者在一开始就写出“正确”的查询语句，把流量精准定位到目标存储节点，而无需开发者手动操心底层的路由判断。 主讲人总结道，这两类 NoSQL 数据库在本质上并无二致：在文档数据库中，人们使用复合索引进行性能优化；在 DynamoDB 中，则是利用投影（Projections）预先将所需字段加载进索引。无论是把对象建成文档格式还是宽列（Wide-column）数据结构，底层的核心思想完全相通。正如主讲人所言：“So anyone who tells you that there's a difference between these systems is wrong there's just not okay.” - 文档数据库索引要求：必须定义复合索引（Compound Index），且分片键必须作为第一属性，查询必须携带分片键条件以避免全节点广播。 - DynamoDB 索引要求：基于具体访问模式定义全局二级索引（GSI），强制使用分区键等值匹配精准命中存储节点。 - 架构本质：两种体系底层都是对象建模与定向索引，不存在根本性的范式对立。 --- ### [03:51:28](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=13888s) 基于流（Streams）与 CDC 实现聚合指标和异构同步 针对企业应用普遍需要的 KPI、Top N、Last N、平均值（Average）、计数（Count）以及求和（Sum）等复杂查询需求，DynamoDB 自身的基础 API 并未原生提供支持。为了高效处理这类业务逻辑，必须引入变更数据捕获（Change Data Capture, CDC）管道。DynamoDB Stream 是这一机制的物理基石，它作为一个连续的增量变更日志，精准记录了表上发生的每一次更新与写入操作。一旦数据落表，该条目的变更就会立即流入数据流，并触发 AWS Lambda 函数进行异步处理。 主讲人回忆了 Amazon 内部早期的架构演进历程：在尚未考虑全面转向 NoSQL 之前，工程师们为了扩展关系型数据库的性能瓶颈，选择构建运行时的汇总聚合表来分摊复杂查询的算力开销。然而，随着大家意识到在关系型数据库中频繁执行 Count、Max、Min 和 Average 会带来难以承受的高昂计算成本，团队顿悟了——如果必须在后台异步维护这类汇总聚合，那这正是彻底转向 NoSQL 的明确信号。文档数据库（如 MongoDB）内置的聚合框架在面对大规模高并发场景时同样无法线性扩展，最终开发者依旧得重构数据模型，依赖 CDC 管道去维护汇总指标。 除了计算业务汇总报表，CDC 架构的另一个常见用途是异构同步。当应用面临复杂的全文检索、地理位置查询（Geospatial）或索引交集（Index Intersections）等 DynamoDB 不支持的高级特性时，便可以通过 Stream 将数据增量同步至 Elasticsearch（Amazon OpenSearch）。 主讲人重点强调了 Lambda 消费数据流的契约特性：触发器保证“至少一次处理（At-least-once processing）”。鉴于容器故障崩溃、在更新完目标但尚未提交位点等罕见临界状态下容器重新拉起重试的情况，主讲人告诫开发者：“Make sure you understand that it's at least once code your ID your lambda functions be idempotent because there are rare conditions where container failure might occur.” 此外，利用 CDC 管道还可以把变更实时分发至 Amazon Kinesis Data Firehose，以 Parquet 列式存储格式批量归档到 Amazon S3 中，再通过 Amazon Athena 执行交互式 SQL 实时分析。由于整套链路全异步运行，主讲人幽默地将其比作“世界上最优秀的存储过程引擎”——它永远不会因为某段写得稀烂的存储过程而拖垮前台核心数据库。他开玩笑说，虽然有人觉得帮客户省钱会减少云厂商账单收入，但自己在 Amazon 的日常工作就是千方百计帮大家降低成本，而且他对此非常擅长。 ```mermaid flowchart LR accTitle: 基于 DynamoDB Streams 的 CDC 架构图 accDescr: 展示从 DynamoDB 写入触发 Stream，流转至 Lambda 进行聚合以及同步到多种下游系统的流程。 subgraph Core ["主存储层"] DDB["DynamoDB 表"] DDBStream["DynamoDB Stream<br/>(变更数据捕获)"] end subgraph Compute ["无服务器计算"] Lambda["AWS Lambda<br/>(幂等处理)"] end subgraph Targets ["下游数据目标"] Agg["回写汇总表<br/>(Top N, Count, Sum)"] ES["Elasticsearch<br/>(全文 / 地理位置检索)"] Firehose["Kinesis Firehose"] S3["Amazon S3<br/>(Parquet 格式)"] Athena["Amazon Athena<br/>(即席 SQL 分析)"] end DDB -->|"写入 / 更新"| DDBStream DDBStream -->|"至少一次消费"| Lambda Lambda -->|"异步回写"| Agg Lambda -->|"异构索引同步"| ES Lambda -->|"实时归档"| Firehose Firehose --> S3 S3 --> Athena ``` --- ### [03:53:38](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=14018s) 复合排序键与高密度分区读取优化 在讲解具体的 NoSQL 数据建模技术时，主讲人阐述了复合键（Composite Keys）的核心价值，即构建具有业务过滤意义的排序键（Sort Key）。他以游戏会话（Game Sessions）系统为例：该系统需要查询特定用户处于待处理状态（Pending）的游戏会话，并按日期排序。最初的设计方案是在会话表上建立索引，以 `UserId` 作为分区键，以 `SessionDate` 作为排序键，同时配置一个针对扩展属性 `Status` 的 Filter Condition（过滤条件）来匹配 `Status = 'Pending'`。 然而，这种设计隐藏着极大的吞吐量浪费陷阱。Filter Condition 是在 DynamoDB 存储节点读取数据后才在应用返回前进行的二次筛选。如果一个分区内包含 3 条记录，读取 3 条并过滤返回 2 条，计费和性能差异尚不明显；但假设该分区沉淀了 10,000 条历史记录，系统必须把这 10,000 条数据全部扫描计费，最后仅向客户端返回 2 条，读取效率将极度恶化。 解决方案是将两个离散属性拼接为复合排序键——每次写入数据时，将状态与日期拼接生成名为 `StatusDate` 的新属性（例如 `PENDING#2019-12-01`）。查询时，利用排序键支持的 `begins_with('PENDING')` 条件表达式，即可在存储引擎层面精准检索出待处理项。主讲人指出，通过这种方式，可以在承载数万条高密度数据的分区中实现极其精准的细粒度窄读取（Selective Reads）。 针对外界“不要往单一逻辑分区放入过多条目”的误区，主讲人坚决驳斥，指出重点从来不在于分区内存储的数据量，而在于读写的速率与速度（Velocity）。在系统底层，单个逻辑分区所能承载的物理上限非常明确：每秒支持 1,000 WCU（每秒 1,000 次 1KB 的写入），以及每秒 3,000 RCU（每秒 3,000 次 4KB 的强一致性读取）。只要业务的读写访问速率没有冲破单分区的物理阈值，在一个分区内存储任意多的条目完全没有问题。 | 特性 / 维度 | 基础过滤条件模式（Filter Expression） | 复合排序键模式（Composite Sort Key） | |:--- |:--- |:--- | | **设计方式** | 排序键为纯日期 `SessionDate`，过滤条件为 `Status = 'pending'` | 拼接复合属性 `StatusDate`（如 `PENDING#<Date>`）作为排序键 | | **I/O 机制** | 扫描分区内所有历史记录载入内存，过滤后剔除无效条目 | 存储引擎基于 B-Tree 索引范围直接执行 `begins_with` 精准定位 | | **计费与吞吐** | 按扫描的全部条目容量计费（大分区下可能读取数万条仅返回几条） | 仅针对命中的待处理条目消耗并支付 RCU | | **分区限制真相**| 单逻辑分区无容量上限；单分区性能物理硬顶为 1,000 WCU/s 与 3,000 RCU/s | --- ### [03:55:33](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=14133s) 企业员工门户的复杂关系建模（23 种访问模式） 主讲人转向了 NoSQL 设计中最常被诟病的核心痛点：如何在杜绝关系型范式（Normalized Data Models）的前提下处理高复杂度关联关系。传统开发者习惯于依赖多表 Join 和查询查找表（Lookup Tables），但在超大规模分布式集群下，这种跨表关联的时间复杂度极高。主讲人严肃警告：如果让高度优化的关系型数据库引擎执行多表关联都感到吃力，那么如果试图在应用服务器代码中通过多次遍历、循环拉取结果集来“模拟 Join”，系统的网络开销与处理延迟将会彻底失控。 为了展示单表设计（Single-Table Design）如何驾驭复杂现实系统，主讲人抛出了一个包含 23 种不同访问模式的企业员工门户（Employee Portal）系统。他开玩笑地提到，去年大会上他演示的一个交付服务案例只包含了 12 种访问模式，而这次搬出 23 种访问模式的案例就是要“羞辱”那些宣称 DynamoDB 无法应对业务演进的质疑者。面对“业务需求演化时单表设计是否难以添加新访问模式”的常见疑虑，主讲人指出，这个案例最初也仅仅是一张轻量级的员工档案表（类似于前文提到的简单文档模型），但随着业务拓展，团队有节奏地在同一个表中陆续融入了工单（Tickets）、项目（Projects）、会议日程等多个不同实体。 在该员工门户的实际表结构中，团队在同一张物理表上规划了多种异构的逻辑分区（Partitions），并把不同类型的业务条目打包载入各自所属的分区类型中： - **建筑分区（Building Partitions）**：以特定建筑物作为分区边界，内部混合存储会议室定义（Room Definitions）以及会议室预订记录（Meeting Reservations）。 - **员工分区（Employee Partitions）**：以员工编号作为分区键，集合存储员工核心元数据（Employee Metadata）以及其收到的会议邀请对象（Meeting Objects）。 - **项目分区（Project Partitions）**：用于维护具体项目实体及其内部流转的业务状态。 通过在同一张物理表中混合多维度的业务条目，系统在单次往返请求中便能直接拉取关联紧密的多实体集合，从而在避免应用层多次往返拼接的同时，彻底替代了高代价的传统关联查询。 ### [43:25](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=2605s) 主表的多实体建模与基础访问模式 在复杂的业务场景中，一个 DynamoDB 表往往需要容纳多种实体类型。当前演示的数据模型中包含了员工提交的项目工时卡（Time Card）、员工在各项目中的角色、项目的元数据（Project Metadata），以及工单系统（Ticket）。工单实体不仅包含创建工单时的基础项，还附带了相关留言消息（Messages）、工单创建人（Owner）和经办人（Assignee）。设计目标是通过单张主表高效支撑 9 种基础访问模式。 主表上的核心查询模式包括： 1. **按员工 ID 查询会议日程**：使用员工 ID 查询其分区，结合日期范围条件；为了排除分区中同属于日期范围但不是会议的其他项，附加一个 `> 0` 的 Filter 表达式。由于大多数员工分区内的数据量并不大，即便采用这种过滤方式进行非强选择性读取（less than selective read），性能和成本也完全在可控范围内。 2. **按大楼查询指定时间段内的所有会议**：在大楼分区中通过 Building ID 进行查询，同样传入时间范围条件。这里讲到了一个 DynamoDB API 的细节限制：无法在排序键（Sort Key）上直接使用 `contains` 操作符。讲师指出：“在 DynamoDB API 中，不知出于何种原因，你无法在排序键上使用 contains 条件；因此，如果确实需要该条件，就必须对排序键创建一个副本属性，并在该副本属性上使用 contains 作为过滤条件。”该副本属性编码了大楼、楼层和房间号等信息，借此完成匹配。 3. **按员工 ID 获取员工元数据**：每个员工都有一条以字母 `E` 开头的元数据项，查询时使用员工 ID 作为分区键，并在排序键上附加 `begins_with('E')` 条件即可精确取出。 4. **获取完整工单历史**：直接指定 Ticket ID 命中工单分区，一次性拉取整个工单的创建记录及全部追加消息。 5. **获取项目信息与工时记录**：通过项目名称查询，当排序键等于项目名称时仅返回项目元数据；若在项目分区上施加日期范围条件，则能拉出所有员工在该项目上提交的工时卡；如果进一步追加角色过滤条件（例如“角色 = 业务技术产品经理”），则能在相同查询下过滤出特定岗位在特定时间段内的工时记录。 --- ### [45:51](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=2751s) 会议室预订工作流：将业务计算下推至客户端 在讨论会议室与资源预订系统时，讲师分享了一个在 Amazon 内部系统及外部大客户中广泛使用的架构模式。通常情况下，系统并不能直接存储“哪些房间空闲”的静态列表，因为空闲状态是由“已有预订”和“全部可用资源”计算得出的动态结果。传统做法往往是应用服务器分别拉取数据，在服务端进行交叉比对后再响应前端，这在流量高峰时会消耗大量的服务器计算资源。 优化后的预订工作流将该逻辑完全下推给了客户端： - 用户打开预订页面时，客户端发起第一条查询，获取整栋大楼的所有房间元数据（数据量极小）。 - 用户选择目标时间段后，客户端发起第二条查询，拉取该大楼及对应楼层在指定时间窗口内的全部“已有预订”记录。 - 最终的差集计算（即哪些房间目前可用）直接交由客户端设备执行，因为客户端早已缓存了大楼的房间列表。 ```mermaid flowchart LR accTitle: "客户端与 DynamoDB 预订交互流程" accDescr: "展示客户端如何通过两阶段读取元数据和预订记录，并在本地计算空闲资源" subgraph ClientStage["客户端（拥有充沛 CPU）"] C1["打开预订页面"] C2["选择时间段"] C3["本地差集计算<br/>（房间列表 - 已预订）"] end subgraph DBStage["DynamoDB 表"] D1["查询大楼房间元数据"] D2["查询指定时间范围内的预订记录"] end C1 -->|"1. 请求房间数据"| D1 D1 -->|"返回房间列表"| C1 C1 --> C2 C2 -->|"2. 查询占用区间"| D2 D2 -->|"返回已有预订"| C3 ``` 讲师强调，这种模式能带来显著的基础设施收益：“这导致的结果是，在面临大规模流量时——我们有好几家客户采用了这种设计——他们反馈说他们实际上能够缩减应用服务器集群的规模，因为应用服务器不再需要处理这些数据的分流计算，而是把所有数据直接推给客户端，由客户端自行解析。”现代客户端（如浏览器、移动设备）拥有充裕的计算能力（CPU），将这类逻辑剥离出应用服务器不仅极大地减轻了服务端压力，还能显著降低运行成本。随后，讲师也提及类似的思想还可以用于版本控制与差异同步工作流，即只向客户端发送增量数据（deltas），避免反复往返服务端。 --- ### [47:26](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=2846s) GSI-1 索引设计：多实体聚合预联接与复合区间过滤 随着索引维度的展开，表在索引上的体积通常会逐步缩小，因为并非所有实体都需要被全局索引，部分项目仅需在主表的第一维度被检索。GSI-1 承担了另外 9 种访问模式，其核心思路是利用单次查询解决跨实体的聚合关联。 最典型的模式是员工工作台视图。当员工登录内部系统时，页面需要一次性加载其最近 30 天内的所有动态，包括参加的会议、提交的工时卡、相关的工单以及在工单下的互动留言。在 GSI-1 中，这些对象统一以员工邮箱（Email）作为分区键： - 客户端只需对该员工的邮箱分区发起一次查询，指定排序键大于 30 天前的时间戳； - 这一条查询便能跨越员工元数据、工单记录、工时卡及会议日程等 5 种不同的业务实体，全部打包返回。 讲师指出，这正是 NoSQL 建模的精髓所在：“这正是我们在 NoSQL 数据库中所追求的：我们会尝试构建能够同时检索多种对象类型的查询。当业务需要时，将这些对象归组到同一个分区中以支持这种‘联接（Join）’——说白了，这就是一种纯粹的联接。” 此外，GSI-1 还解决了一个极为特殊的预算审查场景：客户需要查出“在过去 3 个月内启动、且将在未来 3 个月内交付的所有项目”，以此决定哪些处于危险期之外的项目可以削减预算。虽然这在直觉上需要两个排序条件，但在 DynamoDB 中可以通过排序键与 Filter 表达式结合实现： - 使用排序键条件匹配项目在最近 3 个月内启动（`StartTime >= 3个月前`），从分区中执行相对宽松的读取； - 配合过滤条件判断交付目标日期（`TargetDelivery <= 3个月后`），将不符合时间窗的项目剔除。 - 讲师调侃道：“这是一个类似预算审批的用例：‘我需要知道哪些项目是要被砍掉的，所以我绝不想动那些即将在未来 3 个月内交付的项目，但在此之后的项目都得拿出来讨论。’我虽然不喜欢这种砍项目的场景，但有时候你不得不这么做。” --- ### [50:11](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=3011s) GSI-2 与 GSI-3：经办人维度、汇报线与层级子树聚合 在更高维度的二级索引（GSI-2 和 GSI-3）中，承载的实体范围进一步收敛，专门应对更加细分的组织架构与运营监控需求。 #### GSI-2 的组织与指派查询 - **按经办人邮箱获取工单历史**：在主表或 GSI-1 中，工单通常挂载在工单编号或创建人（Owner）名下；为了支持“经办人（Assignee）查看自己处理过的所有工单及留言”，系统在向工单和留言对象写入数据时，会额外修饰（decorate）经办人属性，将其投影至 GSI-2，方便以 Assignee Email 为键进行聚合检索。 - **按经理查询直属员工**：员工元数据项在 GSI-2 上以经理 ID（Manager ID）作为索引维度，只需查询目标经理的 ID，即可一次性列出其团队管辖下的所有下属员工。 #### GSI-3 的层级路径与积压工单挖掘 - **城市/大楼/楼层的子树聚合（Subtree Aggregation）**：这是层级树建模的典型手段。如果需要查询“某座城市、某栋建筑或某一层楼中的所有员工”，可以在 GSI-3 中将层级结构拼接为一个完整的路径字符串（例如 `/City/Building/Floor`），查询时利用 `begins_with` 前缀匹配即可无缝完成整棵子树的数据聚合。讲师强调这在电商（获取某类目下的所有商品）或办公资产管理中属于高频刚需。 - **滞留/积压工单（Languishing Tickets）的监控**：组织内部往往有海量的活跃工单，讲师最后引入了对长时间未更新、处于停滞状态的工单的提取模式，以便运维和管理团队能够从海量消息流中准确抓取被冷落的工单。 ### [51:47](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=3107s) 写入分片、工单升级与数据建模的迭代流程 为了处理工单系统中那些在过去 24 小时内未被触碰并需要升级的工单，演讲者演示了如何构建写入分片（Write-sharded）分区。系统中的某张工单可能落入分片 7，根据系统处理的工单吞吐量，整体可以设置大约 20 个分片。由于每个物理分区可提供 1,000 WCU，若将写入分散在 10 个分区上，就能轻松支撑每秒 10,000 张工单的写入吞吐量，这足以满足极大规模的工作负载。 在完成这一系列建模后，最终会沉淀出一套清晰的访问模式映射表：记录具体的访问模式、查询所需的条件、目标查询表或 GSI、以及需要应用的过滤条件。演讲者强调，这是他与每一个工程团队合作时必走的标准流程： 1. 先绘制实体关系图（ERD），明确手头的数据结构； 2. 梳理并定义具体的访问模式，搞清楚数据将如何被查询与存储； 3. 设计数据模型，并制定产出目标结果所需的精确查询条件。 “I started with one or two access patterns and started adding more more and more you know with real data sometimes you might have to you know do an ETL on the existing table and decorate the table with the distant existing items with new attributes you might need to change the values and existing attributes but the reality is you're not going to have to throw the baby out with the bathwater right this is an evolution and it's not dissimilar to what you go through with the relational database it's just we're much more familiar with that.” 演讲者以此指出，DynamoDB 的建模是一个持续迭代演进的过程，即便后续需要对现有表进行 ETL 或补充新属性，也绝不需要全盘推翻重来。 --- ### [53:19](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=3199s) 针对常见模式优化：某保险报价系统的架构重塑 在处理常见设计模式时，核心问题之一是评估何时使用大 Item（Big Items），以及大 Item 为何在特定场景下会成为系统瓶颈。演讲者分享了一家近期咨询的保险报价服务客户的实战案例。该客户业务发展迅速，峰值达到每分钟 800 次报价，在 DynamoDB 表上预置了 1,000 WCU。 最初，他们的数据模型设计逻辑如下： - 用户进入系统创建一份新报价，随后在交互过程中不断修改条件（例如通勤里程、家庭住址变更等），每次修改都会产生一个新版本。 - 用户每份报价通常会修改两到三次。每次变更，系统都会复制整份报价的所有数据，生成一个带有递增版本号的新版本。 - 其表结构将 `CustomerID` 设为分区键（PK），排序键（SK）类似于 `QuoteID_Version`。 - 当前台需要展示最新报价时，系统通过 `ScanIndexForward = false` 且 `Limit = 1` 查询到最新版本；若用户在前台翻阅历史版本，前端则每次都向数据库发起一次查询。 ```mermaid flowchart LR accTitle: 保险报价系统更新模式对比 accDescr: 对比原始模型中写入整份大对象的开销与重构后仅写入增量属性并下推组装逻辑到前端的方案。 subgraph LEGACY["原始设计 (1,000 WCU)"] direction TB L1["用户更新报价属性"] --> L2["复制全量数据 (50 KB)"] L2 --> L3["写入新版本 Item\nQuoteID_Version"] L3 --> L4["翻阅历史版本\n每次回源请求数据库"] end subgraph OPTIMIZED["优化设计 (50 WCU)"] direction TB O1["用户更新报价属性"] --> O2["仅存储变动 Delta\n版本化属性片段"] O2 --> O3["写入轻量更新\n保留在同一 Partition"] O3 --> O4["客户端一次性拉取版本对象\n浏览器本地组装渲染历史"] end ``` 该团队向演讲者寻求优化建议。经分析发现，报价变更时改动的数据量极小，实际上 90% 的报价字段（如姓名、地址、电话号码等）都是稳定不变的。然而他们此前每次创建新版本，都会将所有静态数据全部重新写入一次，单次更新达到 50 KB。 演讲者给出的优化策略是将差异（Deltas）直接存放在该分区的表结构内，微调 Schema： - 以 `QuoteID` 为主标识，将那些版本化的属性作为报价不同板块的独立版本属性存储。 - 当查询报价时，系统一次性下发这些版本化对象，并将拼接“最新视图”的业务逻辑直接推送到终端（浏览器/前端）执行。 这一改造带来了极具突破性的性能与成本收益： - 表的预置写入容量（WCU）直接从 1,000 WCU 暴跌至 50 WCU，因为系统绝大部分流量都是轻量的小更新，只有极少数情况下才会创建全新的完整报价。 - 突发的写入尖峰完全可以依托 DynamoDB 的突发容量池（Burst Bucket）平稳吸收，即使短时间内同时并发创建四五份新报价也不会发生限流。 - 在应用架构层面，许多采用类似架构的客户反馈其应用服务器集群规模缩减了约 5%，因为服务端不再需要承担数据拼接与转换的繁重工作，直接透传给前端即可。 - 在用户体验方面，用户在浏览器中点击“上一页”、“下一页”翻看报价版本历史时，无需再向后端 API 发送任何请求，所有数据已在本地就绪。 --- ### [56:30](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=3390s) 工具推荐：NoSQL Workbench for DynamoDB 在总结前，演讲者隆重推荐了 AWS 发布的工具 —— NoSQL Workbench for DynamoDB。该工具能够直观生成演讲中所展示的那些聚合表视图（Aggregate Views），是当前市场上针对 NoSQL 数据库进行数据建模的最优工具。 - **样本数据可视化**：支持直接定义 JSON 格式的样本数据并导入建模器，通过在不同属性上定义索引，系统会即时可视化展现样本数据的具体分布排布方式。 - **关系型数据库连接与迁移**：具备从关系型数据库加载（Load from a relational database）的功能。 - **代码生成器**：内置代码生成器，能直接辅助生成应用程序所需的代码。 - **协同设计**：越来越多的企业客户直接使用 NoSQL Workbench 完成初步建模后再与 AWS 专家沟通，专家能直接加载该模型并快速调整优化。 --- ### [57:32](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=3452s) 总结要点、常见误区与学习资源 在全场总结中，演讲者重点纠正了一个长期存在的行业认知偏见。他极力反对使用“非关系型数据”（Non-relational Data）这一概念： “It's a marketing term that was invented by marketing people who tried describe technology they don't understand to other people who don't understand it and they came up with this it's not relational it must be non relational don't use that don't fall in that trap data is all relational or doesn't matter the ERD still matters.” 演讲者指出，世间所有有价值的数据本身都具备关联关系，关系依然至关重要，因此实体关系图（ERD）在建模中始终不可或缺。 针对数据库选型与架构设计，演讲者提出了明确指导： - **并非替代关系**：关系型数据库并没有被 NoSQL 淘汰。二者应分工协作 —— 在需要极高扩展性的 OLTP 系统或决策支持系统中采用 NoSQL；在传统的 OLAP 场景中继续使用关系型数据库。 - **云原生优先（Cloud Native First）**：AWS 计算云的弹性极为出色，而诸如 DynamoDB 这类云原生托管服务在横向扩展能力上是无与伦比的，设计时应优先考虑。 最后，演讲者介绍了进一步深入学习 AWS 数据库服务的渠道，包括 AWS 官方培训与认证（AWS Training and Certification）提供的线上数字课程，以及合作伙伴 Linux Academy 刚刚上线的全新 DynamoDB 建模实战课程。在全场热烈的掌声中，演讲者幽默地向观众拉票：“如果大家不喜欢这场演讲，请在评分表上记成 DAT450；如果喜欢，请给本场 session —— DAT403 打个好评！” ## Notable quotes > [04:35](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=275s) If you use a normalized data model, you are going to have pain at scale when you expand and you scale out your operations. — Rick Houlihan > [11:46](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=706s) This is the crux of NoSQL design: what I'm trying to do is create collections of multiple objects or multiple types of objects that I can query with single queries. — Rick Houlihan > [02:17](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=137s) When we got to the 1980s, it's very important to understand why we invented the relational database was to decrease the pressure on the storage subsystems in the data center. — Rick Houlihan > [12:58](https://www.youtube.com/watch?v=6yqfmXiZTlM&t=778s) If I only need to know the items that matched, then only project the keys; if I need to know additional data, then project what you need, but don't project more. — Rick Houlihan ## Mentioned Amazon DynamoDB, Rick Houlihan, Oracle, AWS, Herman Hollerith, MongoDB, Amazon DocumentDB, Apache Cassandra, Amazon CloudWatch, AWS Lambda, Amazon OpenSearch, Elasticsearch, Amazon Kinesis Data Firehose, Amazon S3, Amazon Athena, NoSQL Workbench for DynamoDB, Linux Academy ## Video description This technical session is for advanced users of Amazon DynamoDB. The patterns and data models discussed in this presentation summarize a collection of implementations and best practices used by large customers—including Amazon retail businesses—to deliver highly scalable solutions for a wide variety of business problems. We delve into strategies for global secondary index sharding and index overloading, scalable graph processing with materialized queries, relational modeling with composite keys, executing transactional workflows on DynamoDB, and more. [Source](https://www.youtube.com/watch?v=6yqfmXiZTlM)

[4:35] Rick Houlihan: If you use a normalized data model, you are going to have pain at scale when you expand and you scale out your operations. [11:46] Rick Houlihan: This is the crux of NoSQL design: what I'm trying to do is create collections of multiple objects or multiple types of objects that I can query with single queries. [2:17] Rick Houlihan: When we got to the 1980s, it's very important to understand why we invented the relational database was to decrease the pressure on the storage subsystems in the data center. [12:58] Rick Houlihan: If I only need to know the items that matched, then only project the keys; if I need to know additional data, then project what you need, but don't project more.

[20:53] Rick Houlihan: You are provisioning for peak, you are hoping you are consuming a high percentage of that capacity that you have provisioned, but the reality is the average data center utilization in the enterprise is 10 percent.

[0:00] I welcome everybody now thanks for [0:02] taking this time out of this evening [0:05] here in Las Vegas where there's many [0:07] many other things to do to listen to me [0:09] talk a little bit about data modeling [0:10] and no SQL who knew that topic would be [0:13] so popular huh my name is Rick Houlihan [0:16] I am a principal technologist for no SQL [0:19] for Amazon Web Services I've done a lot [0:22] of no SQL data modeling over the years I [0:24] worked on the specialist team I helped [0:26] build the north-american practice there [0:28] and I've worked on the blackbelt team [0:31] internally to help migrate over 12,000 [0:34] Amazon Services or some some odd number [0:37] around there over several years as well [0:40] to from Oracle to know SQL how many [0:44] people have heard of that migration [0:45] we've shut down 3,000 Oracle instances [0:48] all right few you folks out there and in [0:51] the course of doing all this we learned [0:53] a lot of things and that's what I'm here [0:55] to talk to you a little bit about [0:56] tonight and some of these things might [0:58] be surprising to some folks and maybe [0:59] less so to others a lot of this stuff we [1:01] talked a little bit about last year [1:03] we're going to get more into some of the [1:05] complex modeling that we can actually do [1:07] with with no SQL that people might not [1:09] be so aware of so the first thing I like [1:11] to talk about in all of these sessions [1:12] is a brief history of data processing [1:14] why are we looking at no SQL I know a [1:16] lot of you have already heard this but [1:18] some of you haven't and the real reason [1:20] then the reason I do this is because if [1:21] you want a message to be understood you [1:23] have to repeat it it has to be broadcast [1:25] so for those of you who've heard it I [1:26] ask you for a minute or two for those of [1:28] you haven't hopefully you'll get [1:29] something out of that conversation we'll [1:31] get into a little bit of an overview of [1:33] Dinah mode to be more from a you know [1:35] structural perspective what is no SQL [1:36] I'm not going to talk a lot about api's [1:38] features and functions that's not what [1:39] we're here for really here to talk about [1:41] NoSQL data modeling that's what this [1:44] session is really about how do I [1:45] actually model data how do I do things [1:47] in the real world with real apps in a [1:50] way that actually makes sense and works [1:52] for my application and works for me from [1:54] an economic efficiency perspective and [1:57] so that's what we'll talk a lot about is [1:59] data modeling most examples I use might [2:01] be synthetic but they're going to use [2:02] real access patterns patterns that we [2:04] use in real application services and [2:06] I'll actually call those out as we talk [2:08] through those and we're going to get [2:09] into some of the common patterns and [2:11] we'll talk about some of the real-world [2:12] applications [2:13] we built that really again those access [2:15] patterns that we use so again the first [2:17] thing we talked about a little bit is [2:19] the history why are we looking at what [2:20] we're this new technology because we've [2:23] had this relational technology it's [2:24] worked so well for so long and to [2:26] understand this we really need to [2:27] understand that if the history of data [2:29] processing was laid out for us to look [2:31] at it would be a series of peaks and [2:33] valleys and data pressure right and data [2:34] pressure is what we define as what we're [2:37] asking the system to do and how fast and [2:39] how cheap it's happening if it's not [2:40] happening fast enough or cheap enough to [2:42] cheap enough then we've got a high high [2:44] data pressure situation and we're going [2:45] to do a lot of different things it's a [2:46] technology trigger we're going to invent [2:48] a lot of new things to fix that and we [2:50] have over the years the first one that [2:51] we had again was the one between our [2:53] ears it's a great database highly [2:56] available when my eyes are open it's [2:57] online but not necessarily you know [3:00] highly durable fault tolerance is [3:02] questionable single user system the [3:04] whole nine yards not exactly something [3:05] I'm going to build an enterprise [3:06] database on top of so we actually had to [3:09] start learning how to do different [3:10] things and we learned how to write [3:11] things down right we created a system of [3:13] ledger accounting it was the first [3:15] method of storing structured data [3:17] it ran public and private sector [3:19] workloads for millennia until the 1880 [3:22] US Census came along and a man named [3:23] Herman Hollerith was tasked with [3:26] processing all the data that was [3:27] collected it took him eight years of the [3:29] ten-year cycle using a ledger accounting [3:31] method to do so and he realized he had [3:33] to do something new and he did he [3:35] invented the machine readable punch card [3:36] in the punch card sorting machine so in [3:38] 1890 it took him a lot less time now [3:40] this actually was an interesting turning [3:42] point in the evolution of data [3:43] processing we started inventing many [3:45] technologies paper tape magnetic tape [3:47] distributed block storage random access [3:49] file systems and when we got to the [3:51] 1980s it's very important to understand [3:53] why we invented the relational database [3:55] was to decrease the pressure on the [3:57] storage subsystems in the data center [3:59] right this is really the reason we [4:00] invented the relational database because [4:02] it uses a normalized data model and that [4:04] normalized data model deduplicate it's [4:06] the data on disk it makes things nice [4:08] and efficient when your storage is [4:09] expensive we don't really care about CPU [4:11] in 1980 not so much fast forward 30 40 [4:14] years and the opposite is now true right [4:16] when I deploy applications data center [4:19] today the storage is pennies per [4:21] gigabyte the CPU is dollars for CPU [4:24] minute I don't want to use technologies [4:26] optimized for yesteryears you know [4:28] situations right they want to optimize [4:30] for today's situations and this is why [4:32] we use are going to know SQL database so [4:35] now it's under it's important to [4:36] understand when you use new technologies [4:38] that we need to understand how to use [4:40] them before we deploy them or we will [4:41] not have good experiences we found this [4:43] out very quickly at Amazon as we started [4:46] to migrate many of our workloads and saw [4:48] what we thought was great success then [4:51] we started moving bigger workloads and [4:52] found out that you know what it's really [4:53] easy to do small things badly and think [4:55] that you're doing okay right as you move [4:59] those bigger workloads you start to find [5:00] out that wow this thing's not working [5:02] very well we're not having the [5:03] experiences we expected it's not cheaper [5:05] it's not faster why is this happening [5:08] you start to analyze the system and you [5:10] really when you figure out you didn't [5:11] know how to use the new technologies you [5:12] tried to use them the same way you use [5:13] the old technologies in the case of no [5:16] SQL that's about data models if you use [5:18] a normalized data model you are going to [5:21] have pain at scale when you when you [5:23] when you expand and you scale out your [5:26] operations so this technology this chart [5:28] really kind of like this because it [5:29] talks about the adoption curve in the [5:31] beginning we have a technology trigger [5:32] people are running around trying to [5:34] solve problems in our case it's data [5:36] pressure we invent this new technology [5:38] people go to use it [5:39] some people have success others run to [5:41] the new technology and they have a [5:42] miserable experience because they [5:43] haven't figured out how to use it now as [5:45] the technology skills evolve in the [5:47] market and distribute across the market [5:49] we have better experiences because [5:51] people understand how to use the [5:52] technology today we understand I use [5:54] relational technology tomorrow we will [5:56] all understand how to use no SQL [5:57] technology and modeling data and no SQL [5:59] is not going to be any harder tomorrow [6:01] than it is today to model your [6:02] relational data I get this all the time [6:04] it's so hard it's so hard to use these [6:06] models no it's not it's actually just [6:08] learned a different way of thinking [6:09] about your data and hopefully this [6:10] conversation today is going to help you [6:12] get there when we talk about no SQL it's [6:15] important to understand it's not good [6:16] for everything right it's good for a [6:19] certain class of applications and that [6:21] class of applications are those that [6:23] have repeatable access patterns and the [6:24] reason why is because I need to [6:26] denormalize my data model if I'm going [6:29] to be able to simplify my access [6:30] patterns we'll get into this when we [6:32] talk about the modeling but that [6:33] denormalize model is very tightly [6:35] coupled to the access pattern in the [6:37] case of a relational database the ad hoc [6:39] query [6:39] gives me some flexibility if I don't [6:41] understand how I'm going to be accessing [6:43] the data then it can be very beneficial [6:45] for me to have an ad-hoc query engine [6:47] and that's really suitable for an OLAP [6:48] type of workload online analytics this [6:51] is a really not a good workload for no [6:53] SQL so understand the difference here [6:55] good for us in know SQL and that 90% of [6:58] the applications we write are written to [7:00] support common business processes which [7:02] represent OLTP applications so it's [7:05] actually one of the most relevant [7:06] technologies you can learn as a [7:09] developer today going forward all right [7:11] DynamoDB again we're not going to get [7:12] too much into you know the top level of [7:15] what is dynamo it's a fully managed no [7:18] SQL database and this is the biggest [7:20] value ad you get from dynamo it's cloud [7:22] native you don't have to spend resources [7:24] managing nodes rebuilding storage [7:27] devices [7:28] you know replicating in new shards [7:30] figuring out how long is it gonna take [7:31] me to get the shard online and believe [7:33] me that can take a long time [7:35] we have customers the largest customers [7:37] in the world using legacy no SQL [7:39] technologies now working with us because [7:41] you know it takes them months literally [7:43] to add new shards you know a MongoDB [7:45] cluster when you get up to 1520 shards [7:48] and you need more it starts taking weeks [7:50] months to get the new capacity online [7:52] and this is a common problem that people [7:54] have run into it's scale no only that [7:56] the amount of resources you're going to [7:57] dedicate to managing your new SQL [8:00] cluster can be much better spent [8:01] actually building your business so no [8:04] SQL databases we'll get into the [8:06] performance aspects of this have a lot [8:08] of advantages over your legacy no SQL [8:09] technologies that you're just never [8:10] going to be able to realize by rolling [8:12] your own if you want to talk about scale [8:14] it's fast and consistent at any scale [8:16] scales to support any workload we have [8:19] examples of single tables in excess of [8:21] seven million transactions per second [8:23] like I challenge you to bring me another [8:26] no SQL database that's scaled out to [8:27] that level 50 4.5 million transactions [8:30] per second was what the Amazon CDO [8:32] tables did on prime de 2019 I heard we [8:35] shattered that record over cyber monday [8:38] Black Friday I don't know what those [8:39] numbers are but they're massive you know [8:41] this is just not something you see from [8:42] the legacy no SQL providers all right [8:45] when you get into modeling data and no [8:47] SQL it's important to understand a [8:50] couple of constructs the first thing is [8:52] a table [8:52] as an object repository it's where I'm [8:54] gonna push items items can exist on the [8:57] table they must have a a single [8:59] attribute called the partition key as [9:00] far as that any other attribute is not [9:03] required in this particular [9:05] configuration what I have is a [9:06] collection of objects that identified [9:08] uniquely by those partition keys and [9:09] supports a key value access pattern if I [9:12] query this table with the partition key [9:14] equals X I'm gonna get a single object [9:16] back if I define and this is actually [9:18] you know this is cut similar to a [9:20] document database right in document DB [9:22] I'm going to define a collection in the [9:24] collection I'm going to push documents [9:25] those documents are gonna have underbar [9:26] ID as a unique attribute and I'm going [9:29] to query the system on a key value [9:30] access pattern give me the document with [9:31] an underbar ID value of X right I will [9:34] talk a little bit about the difference [9:35] between document models and wide column [9:39] models but one of the biggest difference [9:41] between the document wide column model [9:43] is the ability to add a sort key to the [9:44] table it's one of the part on this [9:46] collection now of objects what I've done [9:48] is I've defined a partition key that [9:50] defines a collection or a group of [9:52] objects and inside of that partition now [9:54] the sort key uniquely identifies the [9:57] objects in that partition okay and I can [9:59] use range queries to query those objects [10:01] selectively out of that partition so [10:03] example here might be a partition key as [10:05] a customer ID a sort key is an order [10:08] date and I'm going to query for [10:09] customers orders by date let's say [10:11] everything in the last 30 days write a [10:13] date range condition of greater than 30 [10:15] days partition key condition equals [10:17] customer X brings back that list of [10:19] orders for that particular customer [10:21] right and I could have a thousand orders [10:23] in that partition but those of the ORS [10:24] are going to come back conversely I can [10:26] also create other constructs we'll talk [10:28] about that can I put other types of [10:30] items they don't have to be homogeneous [10:32] collections of items they can be [10:33] heterogeneous collections of items but [10:35] what we really have done is we've [10:37] created now this object repository where [10:39] our primary table creates holds a [10:43] aggregation of objects that satisfies a [10:46] certain collection of access patterns so [10:48] in essence what we've done is joined [10:50] objects into a partition and we're going [10:52] to start to decorate these objects with [10:53] additional attributes that we're going [10:55] to index to create additional [10:57] aggregations or grouping of objects and [11:00] so if you think about what I'm [11:01] describing here I'm really describing [11:03] the ability to join objects so this is [11:05] what [11:05] this is what we do in no SQL it's not [11:07] that we can't join objects we create [11:10] model joins right we model our data so [11:12] that we can group these objects into [11:14] partitions on the table partitions on [11:16] the GSIS and we're going to decorate [11:17] these objects with common attributes [11:20] that are going to tie them together when [11:22] I query on various access patterns we're [11:24] going to show you how to do this but [11:26] this is not so different than the way [11:27] actually relational databases work when [11:29] they actually have to join data that's [11:31] not indexed right they create a hash [11:32] table and then they do lookups against [11:34] the hash table as they iterate through [11:36] the inbound collection or the or the [11:37] outer collection so what we're really [11:39] doing is storing data in that in that [11:42] distributed hash table structure ok so [11:46] we talked about being able to since cert [11:48] multiple items into the table this is [11:50] how we're going to do that this is an [11:51] example of a table where again we have a [11:53] partition key which is a customer ID we [11:56] have a sort key which is indexed by time [11:57] we're and it's actually got several [12:00] items in there some items have a time [12:02] prefix others don't and if I query this [12:05] particular partition with the query that [12:07] says give me everything that's greater [12:09] than 30 days ago I'm going to get all [12:11] the orders all the order items as well [12:12] as the customers metadata because that's [12:14] going to satisfy that query condition as [12:15] well so I don't actually have to query [12:17] multiple tables this is the crux of no [12:19] SQL design what I'm trying to do is [12:21] create collections of multiple objects [12:23] or multiple types of objects then I can [12:25] query with single queries I don't have [12:27] to go to versing multiple tables to get [12:28] that I don't have to issue a query [12:30] against the customers table to get the [12:31] customers metadata a query against the [12:33] orders table to get the orders metadata [12:34] then iterate through the orders result [12:37] set and then go get all the items for [12:38] those orders from another table this is [12:40] an amazingly complex workflow that I'm [12:42] avoiding by grouping these items into a [12:45] partition like this right so this is the [12:46] kind of games we're gonna play they'll [12:48] satisfy those queries right so now when [12:51] I go get those results sets oops sorry I [12:56] have a much simpler access pattern right [12:58] so again with indexes this is what we're [13:01] going to do to support a secondary [13:03] access patterns right with a with a and [13:06] you can think of an index in DynamoDB as [13:08] a a table that is replicated as you [13:12] insert and change data on the parent [13:14] table right we take care of that [13:16] replication there's a hundred percent [13:17] SLA guarantee [13:18] on GSI replication these GSIS are [13:21] eventually consistent but they're 10 [13:23] millisecond latency p99 when properly [13:25] configured so it's a very very good [13:27] system to be able to support secondary [13:31] access patterns in off-the-shelf no SQL [13:34] databases you can't necessarily do this [13:35] with some of these guys right like [13:37] Cassandra they even tell you don't don't [13:38] use indexes because there's no way to [13:40] maintain consistency on their indexes [13:42] DynamoDB indexes are guaranteed to be [13:44] consistent so this is one of the biggest [13:46] value ads of dynamodb others over some [13:49] of the other no SQL providers out there [13:50] one of the other things we do with [13:52] DynamoDB index is we support projections [13:54] so if you need to have an access pattern [13:57] let's say that needs to identify the [13:59] items that match on the table and when [14:01] identify those items I also need some of [14:03] the other attributes those items contain [14:05] I can choose to project all some or none [14:08] of those attributes on to the GSI now [14:11] remember when you do this you're [14:13] increasing the storage cost essentially [14:15] I could be duplicating the data if I [14:17] project all the attributes I've [14:19] duplicated or double the storage cost [14:21] and doubled the wcu cost of the insert [14:24] of any inserts I make to the table so [14:27] and you know this is a good way to [14:28] understand when you're getting to [14:30] understanding your access patterns you [14:31] want to start to control costs by [14:34] controlling what you project to the [14:35] indexes if I only need to know the items [14:37] that matched and only project the keys [14:38] if I only need if I need to know [14:40] additional data then project what you [14:42] need but don't project more okay so [14:45] partition keys in no SQL are used to [14:47] spread the data out so to speak and [14:49] dynamodb we create an unordered hash [14:51] index we spread those items out across [14:53] this arbitrary key space and we chopped [14:54] this key space up as we need to add [14:57] additional capacity either storage or [14:59] throughput this is important because an [15:01] individual storage node on the array [15:03] only has so much throughput so when we [15:05] designed for DynamoDB or for any no SQL [15:07] database we need to take into account [15:08] the idea that we need to get that [15:10] workload spread out if we want to scale [15:12] in scale the system in throughput and [15:15] the way we're going to do this in [15:18] DynamoDB is to use a technique we call [15:20] right sharding for highly dense [15:23] aggregation so the example here using [15:25] our customers partition key [15:28] and the data that we have laid out might [15:29] be that I want a secondary index that's [15:31] gonna tell me give me all of the orders [15:33] that were made from a given source right [15:36] so if you think about Amazon retail when [15:38] they want to get all the orders from the [15:41] odd that we're sourced online that could [15:43] be a very dense aggregation in DynamoDB [15:45] you only get about a thousand WCU's per [15:47] second so if the on the GSI I'm trying [15:49] to aggregate by source and all I'm doing [15:51] is using a source as a partition key [15:53] then the only way to increase throughput [15:55] is to create additional partition keys [15:57] by salting those partition keys I'm [15:59] indexing so in this case I'm aggravated [16:01] the orders by source I'm aggravated the [16:04] items by product Aysen and I'm looking [16:07] for additional data right the state of [16:09] these products in shipment I'm looking [16:12] for the source of the orders and support [16:14] additional access patterns now when I do [16:16] this on the read side what I'm going to [16:18] do behind some sort of data layer API is [16:20] create a parallel processing system that [16:23] allows me to query across those [16:24] partitions assemble the data for my [16:26] clients and produce a nice collection if [16:29] users have to query across those [16:31] partitions to gather all of that data [16:32] right but I'm not gonna I'm not gonna [16:34] require all my developers to go out [16:36] there and understand how things are [16:37] right sharted I'm gonna do this at the [16:39] data layer and the reason why is because [16:40] oftentimes when I'm designing for [16:42] DynamoDB I need to write shard some of [16:46] my customers maybe but not all of them [16:48] right and if you look at Amazon tables [16:51] and aw s tables configuration tables [16:53] sometimes you need to come to us and ask [16:54] us to increase the number of [16:56] configuration items or where I am [16:58] policies or or whatever it is you're [17:01] using and some of those reasons wire [17:02] because maybe because we need to do some [17:04] DynamoDB write charting if you go beyond [17:06] X you're gonna exceed a thousand WCU's [17:08] on one of our indexes or maybe even on [17:10] the primary table based on our estimated [17:13] write traffic if you go beyond a given [17:15] point so you're going to want to conceal [17:17] all of this from your users by creating [17:19] some sort of data layer API that's going [17:22] to handle the right sharding behind a [17:23] request to go get the data users provide [17:26] some conditions the API then shards the [17:29] data or [17:31] retrieves the data from multiple shards [17:33] now just like the table we want to use [17:36] generic keys because when we use generic [17:38] keys we can install it we can slip we [17:41] can insert multiple items into comment [17:43] into the GSIS right like on the table we [17:46] have multiple types of partitions the [17:47] weight and multiple types of items and [17:49] the way we make this happen is we use [17:51] these generic keys called PK and SK well [17:53] I showed you the two patterns we want to [17:55] index right by orders by source items by [17:58] a state I want to be able to maybe reuse [18:02] my GSI for both of those access patterns [18:04] because if I create two GSIS I got [18:06] allocate to capacity buckets got a set [18:08] up to come no sets of alerts and alarms [18:10] and I can't reuse the capacity that's [18:12] allocated to one when these access [18:14] patterns are probably not acts not [18:16] happening in parallel so it's more [18:18] efficiency to reuse the GSI so I reuse a [18:20] lot of GS is to reduce the number of [18:22] constructs we have to maintain in [18:24] production now when I query the GSI it's [18:26] the sort key and/or the partition and [18:28] sort key conditions that define what it [18:30] is that returns I don't have to worry [18:33] about those objects colliding so to [18:35] speak it's but you know if they have the [18:37] same potential for collision on the [18:38] partition key I can prefix the partition [18:40] key with the item type to avoid that but [18:43] you know in this particular use case I [18:45] don't because there's no chance for [18:46] collision between an ace in and a source [18:48] for an order when I query with the [18:50] source is the partition key I'm going to [18:52] get those order items when I query with [18:54] an ace in as the partition key I'm going [18:56] to get the order items that are actually [18:59] parts of those orders right so depending [19:01] on the use case depending on the access [19:02] pattern the keys are good to find the [19:04] items that come back all right so when [19:06] gets into scaling noise ql again it's [19:08] about getting that data spread out when [19:10] you don't spread the data out this is [19:11] this is what happens we get these what [19:13] we call hot keys this is a heat map we [19:16] now have a cloud watch feature that [19:19] allows people to look at the the these [19:23] keys we don't you don't get heat map [19:24] views like this but you can look at the [19:26] keys that are being experiencing [19:28] pressure like this in this case we have [19:30] a number of servers on the y-axis we [19:33] have time on the x-axis and you can see [19:36] no server number 11 there is doing a [19:37] whole bunch of work and where everybody [19:39] else is doing nothing we want to avoid [19:41] patterns like this this is an anti [19:42] patterns the high velocity [19:44] access patterns it's firing on a small [19:46] number of keys or a single key so the [19:48] way to get the most out of Modi be out [19:50] of any no SQL database as a matter of [19:52] fact is to spread the workloads out [19:54] right as we want to scale the system we [19:56] want to get more and more shards more [19:57] and more storage nodes [19:59] you know participating in the work and [20:01] so we're going to spread that data out [20:02] across the key space and we're going to [20:05] hopefully have our requests kind of line [20:07] up well over time sometimes that's not [20:10] the way things work right we get [20:11] thundering herd's we had Cyber Monday [20:13] and Black Friday just recently so we [20:15] know a little bit about that if you do [20:17] it right this is what ends up happening [20:19] you get pictures like this this is a [20:22] great heat map this is about 450 servers [20:26] they're pumping about 40 to 50,000 WCU's [20:30] consistently over time across their key [20:32] space and you can see how that access [20:34] pattern is just lighting up like a [20:35] Christmas tree up and down the key space [20:37] this looks like the white noise on the [20:38] TV when you change between channels [20:41] right that's the I don't know if any but [20:44] that is even happening or does it okay I [20:49] have an old man they can't help alright [20:53] so these are the things that happen when [20:54] you go to cloud native I had no SQL [20:58] services you get the ability to kind of [21:01] just in time provision your tables it's [21:03] been in both dimensions storage and [21:05] capacity and throughput in this case [21:08] what we're looking at on the left hand [21:09] side there is your relational database [21:11] that's your legacy no SQL technologies [21:14] right this is MongoDB this is Cassandra [21:17] you are provisioning for peak you you [21:19] are hoping you are consuming a high [21:20] percentage of that capacity that you [21:22] have provision but the reality is the [21:23] average data center utilization in the [21:25] enterprise is 10 percent if you're doing [21:27] better than that over time then you're [21:28] doing really good so the reality is [21:30] you're wasting a lot of money on the [21:32] right hand side this is dynamodb this is [21:34] cloud native database technology with [21:36] auto scaling this is a real service it [21:38] runs in one of our fulfillment centers [21:40] and you can see the difference between [21:41] they turned on auto scaling and before [21:44] it's night and day [21:46] the amount of cost that they're saving [21:47] is around 40 percent I think most of our [21:49] fulfillment center applications are [21:51] saving around 40% and it works very well [21:53] when you have that kind of nicely shaped [21:55] demand curve [21:56] for applications we have other pricing [21:58] models if you don't and we can talk a [22:00] little bit about those I'm not really [22:01] going to talk too much about pricing [22:02] models but I like to talk about [22:03] performance and performance at scale [22:04] because nothing else performs like [22:06] dynamodb or other cloud native no SQL [22:10] databases when you look at in dynamodb I [22:14] don't know what other database you're [22:15] going to look at and say it gets faster [22:16] when you throw more load at it all right [22:19] that's like counterintuitive right what [22:21] most databases this is a synthetic [22:23] workload ramping up over a million [22:25] requests per second and what you can see [22:26] is the low latency consistent low [22:28] latency at the peak utilization and the [22:31] reason why is because we have this [22:32] massively distributed request router [22:34] fleet and that request router fleet [22:35] needs to know about your table to [22:37] service your request so if it doesn't [22:39] know about your table it has to hit a [22:40] configuration service to understand your [22:42] policies your your stores node locations [22:45] and all the various information needs to [22:47] know to factually process the request [22:49] over time as that information cache is [22:52] in the request router we don't have to [22:53] go and back and look at that data so [22:55] it's actually going to get faster and [22:56] that's what happens over and over again [22:57] and we have many customers they have [22:59] come to us and said is this real you [23:02] know their APM instrumentation may be [23:03] telling them things that they don't [23:05] believe and we can say yes it's a common [23:07] phenomena the other thing we see with [23:09] dynamodb that's really amazing is the [23:11] ability to deliver more than you have [23:12] provisioned we have this thing called [23:14] the burst bucket this is an example of a [23:16] message service provider that around the [23:18] Superbowl they were provision close to 6 [23:20] million WCU's at the Superbowl when it [23:24] ended they peaked at over 10 million [23:26] WCU's they didn't notice this until the [23:29] next day they came back after looking at [23:31] their cloud watch metrics and oh my gosh [23:33] what happened here how come we didn't [23:34] throttle and the reason why is because [23:36] you have a five minute burst bucket with [23:37] dynamodb you can go to this database and [23:39] you can say hey look I know you weren't [23:41] busy five minutes ago can you give me [23:43] that now please and what happens with [23:45] with MongoDB when you do this what [23:47] happens with Cassandra what happens with [23:49] your relational database when you do [23:50] this right it goes offline okay that's [23:53] what happens it goes bye-bye and so and [23:55] you're going to start to see your [23:56] requests and response times go through [23:58] the roof as you put the memory pressure [23:59] on those systems right not so with [24:00] DynamoDB it's built to be elastic to [24:03] buffer that throughput and give it to [24:05] you when you need it okay so let's talk [24:07] about data modeling that's really what [24:08] we're here for [24:10] all right and we talked about some of [24:12] the basic data modeling but it's really [24:14] about relationships when we talk about [24:15] data modeling okay and relationships I [24:18] Drive every application I've ever worked [24:20] on I've never been able to build an [24:22] application without understanding the [24:23] relational model the entity relationship [24:25] model and that doesn't change when you [24:27] get into no SQL right then you know what [24:29] what changed was us thinking that we can [24:31] create these big giant blobs of data [24:32] that hold these relational structures [24:34] right the idea that hey these giant JSON [24:37] documents are good ideas you know what [24:39] that big documents are bad ideas we'll [24:40] talk a little bit about that as we get [24:42] into the modeling but it it doesn't [24:43] matter what type of application you're [24:45] building social networking document [24:46] management IT systems management [24:48] anything you're working on requires us [24:50] to build relationships even if it's a [24:52] key/value access pattern that's that's a [24:54] relationship there's a one-to-one [24:55] relationship between the key and the [24:57] values that are associated to it and and [25:00] you know if there's not I want to know [25:02] please tell me is there any [25:03] non-relational data that you're aware of [25:04] I haven't found it as a matter of fact I [25:07] would love to have an on relational [25:08] banking application because I would log [25:10] in over and over again until I saw the [25:12] numbers that I liked right and then I [25:14] would write my checks [25:15] I'd go okay hopefully as Jeff Bezos [25:18] account I'll be really happy with that [25:22] all right maybe not that's okay all [25:30] right so this is what we've done when we [25:31] model relational data we use a [25:32] normalized data model this is a example [25:35] I've been using for years it's a simple [25:36] product catalog there's a one-to-one [25:38] relationship between products and books [25:40] albums and videos one-to-many between [25:42] albums and tracks many many between [25:44] videos and actors and and the various [25:46] queries we have to run are quite complex [25:49] we're gonna dive into that in a second [25:50] but what we might do with the relational [25:52] database in this way where we would use [25:54] ad-hoc queries to join these tables and [25:56] a lot of CPU power to assemble the views [25:58] we're looking for we're actually going [25:59] to go ahead with no SQL and this might [26:01] be a simple approach of doing this would [26:02] be to kind of create that those views so [26:05] to speak or there's hierarchies of data [26:07] that we're commonly querying by [26:08] assembling them into documents you can [26:10] think of all the rows of these [26:11] relational tables is sticking them into [26:13] these documents or partitions as we will [26:16] demonstrate in a second which can be [26:17] very interesting and how we're going to [26:19] access them but let's take a look at the [26:21] complexity of those queries right so [26:22] let's populate some data [26:23] tables let's take a look at the various [26:25] queries we're running so the first query [26:26] we're gonna run is a single one-to-one [26:28] relationship not terrible the time [26:30] complexity you know that particular [26:32] query is a login and end login or M [26:35] login can't really see n log M whatever [26:38] don't have my glasses on when we when [26:41] you start to add more tables that time [26:43] complexity just gets worse right there's [26:45] no there's no adjusting this there's no [26:47] fixing this and this is assuming that we [26:49] have all the proper indexes and that [26:51] we're we're running you know nested loop [26:53] joins and we're not having to do [26:55] terrible things like Elmer's joins and [26:57] hash joins but but but we're actually [26:59] doing things efficiently that time [27:00] complexity just goes up it goes up it [27:02] goes up the more the more tables I join [27:04] in it goes up when we start to look at [27:06] no SQL right now SQL is basically built [27:08] on distributed hash table so no SQL [27:10] queries theoretically have no time [27:13] constant right so this is taking all [27:15] those same joins right and creating [27:19] partitions they're more or less pre join [27:21] the data now if you think about the [27:23] worst case scenario a relational [27:24] database what is it doing it's probably [27:26] going to hash join right I've got a [27:27] large input set I've got a large table [27:30] let me go ahead and table scan create a [27:32] hash index on the sort dimension that I [27:34] need right let's go ahead and then start [27:36] to iterate my input collection I'm going [27:38] to query the hash table repeatedly to [27:40] draw out the the items that match right [27:43] well this is why not do that first [27:45] because obviously the database believes [27:46] that a hash table is the actual you know [27:48] the most efficient database data [27:50] structure to query when I need to do [27:52] repeated query so let's go ahead and do [27:53] that and that's really what we're doing [27:55] here we're creating partitions these [27:56] partitions have the rows or the items [27:58] from our relational structures and as we [28:00] query through this in the particular [28:02] case here I'm querying for a book it's a [28:04] time constant if I query for the album I [28:07] query by the album title it's again time [28:10] constant know sort condition so I'm just [28:12] going to the databases give me [28:13] everything with this partition key I [28:14] give me everything with the movie [28:16] partition key now the interesting thing [28:18] about querying the movie here is that [28:20] what I've done is I have both movie and [28:21] actor partitions and the movies are [28:24] related to the actor partitions through [28:25] those items that are sorted on the [28:28] actors name so this is a graph this is a [28:30] directed graph I have movie partitions [28:32] those are movie nodes I have actor know [28:34] I have all these items inside of that [28:36] movie node that or that are in that [28:38] movie partition that describe the [28:39] relationships between the movie and the [28:41] various actors the the movie partition [28:43] understands what it's related to in this [28:45] particular data structure that the actor [28:47] doesn't yet we'll get into that but when [28:50] I query this partition for the movie I'm [28:53] going to get a set of items the one the [28:54] first one describes the the movie it [28:56] describes the movie and it's sorted on [28:58] the directors name if I query if I get a [29:01] list of actors I can put together [29:03] summary view here that shows me the [29:06] movie and the actors that are in the [29:07] movie what their names are what their [29:09] ages are what their genders are what [29:11] role they played and I'll project the [29:13] dates of their birth dates and their [29:15] genders because that's relatively [29:17] immutable data it doesn't change and I [29:20] wish I could change my birthdate but [29:23] that's not happening the in any case [29:28] it's okay to protect the mutable data on [29:30] these edges to make the read more [29:31] efficient I'm optimizing for the read [29:33] but I won't protect the bio because the [29:34] bio changes I don't want to have to [29:36] update the bio across all the actors [29:37] roles and whatnot but son I'm using [29:40] those edges to describe to tell me some [29:42] things about these actors if I go to [29:44] this to create an index and I flip the [29:47] partition in the sort key around right [29:49] so I take the sort key and it becomes [29:51] the partition key of the GSI and [29:52] petition key becomes the sort key now I [29:55] can actually query on additional [29:57] dimensions right I can query for all the [30:00] books that an author wrote I can query [30:02] for all of the albums that a song shows [30:05] up on I could query for all of the [30:08] movies that an actor's been in well now [30:10] what have I done I've completed that [30:12] many-to-many this is now an undirected [30:14] graph both sides understand what they're [30:16] related to I'll query the table for one [30:18] side I'll query the GSI for the other [30:19] right this is how we model many to many [30:22] relationships in no SQL and avoid [30:24] massive duplication of data right the [30:26] only data where I'm duplicating is [30:27] denormalized across those edges and I [30:29] can decide what I'm going to project [30:31] onto those edges oftentimes those edges [30:34] need the extended metadata describes [30:35] what is the relationship what is the [30:37] role what are the permissions what are [30:39] the things about this association to [30:41] this object that this person has or that [30:43] this other object has in my system I can [30:46] query for the movies that [30:48] director has directed I could query for [30:51] the albums of the musician has produced [30:54] so on and so forth right this is how [30:57] we're going to model additional access [30:58] patterns one side of access patterns on [31:00] the table another set of access patterns [31:02] on our indexes and that's the primary [31:04] way that we'll do it okay let's get into [31:07] documents versus wide column and this is [31:10] really important because there's a lot [31:11] of hype out there there's a lot of [31:13] people talking about how it's all [31:14] different and there's use cases for [31:16] document there's use cases for wide [31:18] column in and I'll tell you when you're [31:19] getting to actually modeling the data no [31:21] it's not there's not a lot of [31:23] differences as as matter of fact there's [31:25] none we're gonna get into that let's [31:26] talk about this document model we have [31:28] an employee item this this employee item [31:31] might be indexed on a couple of [31:32] dimensions in a document database under [31:34] par ID is a default index and we're [31:37] gonna be able to query it on that key [31:38] value access pattern we're going to use [31:41] a meaningful value for underbar ID I [31:43] know a MongoDB if you insert an object [31:45] without underbar idea gives you a UUID [31:48] what that really does is creates a data [31:50] index for most practical purposes most [31:52] applications I don't care about an [31:54] object reference by UUID [31:56] maybe some do but I no one calls up [31:59] customer support and says I'm customer [32:00] UUID right they call up customer support [32:02] and say I'm so-and-so what's your email [32:03] address is the first question so use [32:05] something that's meaningful as a key as [32:07] a primary key and then we're going to [32:10] create a secondary index a compound [32:11] index on the building dot floor to [32:13] support subtree aggregations of our [32:15] employees in this particular example as [32:17] what we're trying to produce [32:18] all right so query where you know the [32:20] building ID equals x and the floor [32:23] starts with you know four seven give me [32:25] everybody out seventh floor of this [32:26] given building or so forth so on and so [32:27] forth [32:28] so let's take that data model we're [32:29] going to move that data model into a [32:30] wide column data structure and we can [32:32] see how different this is and really [32:34] it's just a different way to display the [32:36] data as a matter of fact when I query [32:38] the data in dynamodb what comes back is [32:40] how many folks have queried DynamoDB but [32:42] what I see is JSON right it's just a [32:45] data structure okay data structures can [32:47] be flat like this they can be [32:49] hierarchical and DynamoDB we support [32:51] JSON attributes so you can have [32:52] hierarchies of data as a matter of fact [32:54] the fastest way to query your DynamoDB [32:56] table is to put your unindexed [32:58] attributes in JSON [33:00] all right so if you have lots of [33:01] attributes in your documents create a [33:03] big JSON document insert that in is one [33:05] attribute and then project above into [33:08] the root of the item all of your your [33:10] indexable attributes because we can't [33:13] index the attributes inside of a JSON [33:14] object but DynamoDB will treat that JSON [33:17] object like a big blob a single [33:19] attribute okay we actually have to store [33:21] each attribute in these storage engines [33:23] so if you create a JSON attribute it's [33:24] one attribute if you split that all out [33:26] into a lot of attributes it actually [33:28] takes a little bit more time and there's [33:29] a blog post out there for it you could [33:31] look at all right getting into the [33:33] differences then what we're really [33:34] talking about is indexing and how do we [33:37] query write the system and that's it's [33:40] really the difference in that kind of [33:41] dimension it has nothing to do with the [33:43] data model in document DB there's a [33:45] default index on underbar ID and [33:47] DynamoDB the partition key defines that [33:49] default index in document DB we've got a [33:53] query planner that we can actually go [33:55] ahead and query it chooses the index you [33:58] know when I was at MongoDB I used a lot [33:59] of dollar hints on my queries because I [34:01] don't want that thing to use indexes [34:02] that I don't spend I want to use the [34:04] index I specified I certainly don't want [34:06] to collection scan for me when you query [34:09] and document DB you're gonna see two [34:11] very relevant metrics on the result that [34:14] you get back when you do and explain [34:15] it's going to be documents returned and [34:17] document scanned if documents scanned is [34:19] zero that means that and document [34:21] returns greater than zero it means that [34:24] your index covered the query okay every [34:26] attribute you had defined on the index [34:28] was covered when you made the query if [34:30] it says documents returned to document [34:32] scanned or equivalent then it means that [34:34] well I found that the documents on the [34:36] index but I had to go back to the [34:38] collection to get the extended [34:39] attributes remember I talked about [34:40] projections right in order to project [34:42] attributes in document DB I need to [34:44] include those attributes in the index [34:46] definition that actually makes your [34:48] inserts more expensive okay so that's [34:50] one of the nice things about a wide [34:52] column database like dynamo DB with [34:53] projections on the index I don't have to [34:55] you know those those those those inserts [34:57] don't become more complex they may take [34:59] more wcu that doesn't cost more CPU to [35:01] do those inserts when we go into [35:04] indexing and defining indexes when I [35:08] query a document database I better make [35:11] sure that the shard key is included [35:13] as the first attribute of my index [35:15] because that's what's going to be used [35:17] to turn where to route my query and I [35:19] better make sure when I make the query [35:20] that I include a shard key condition [35:22] otherwise the query gets sent to every [35:24] node now you may want that but you know [35:26] most of the time you don't and most of [35:28] those queries would be coming back empty [35:30] has just wasted bandwidth in the system [35:32] so you need to find compound indexes and [35:34] include that shard key as the first [35:35] metric and include the shard key values [35:38] when you make the query to be able to [35:39] isolate your queries to given nodes in [35:41] the system with dynamodb [35:43] I chose the index I design the index for [35:45] a specific access pattern that's the one [35:47] I want to use I will then use a [35:50] partition key quality condition every [35:51] time I query that GS higher every time I [35:53] query that table that routes me exactly [35:55] to the right storage node I don't have [35:57] to worry about making sure I do it right [35:59] it just forces you to do it right again [36:01] I'm gonna optimize with compound indexes [36:04] in document database I'm gonna use [36:06] projections to preload my indexes in [36:08] DynamoDB those are the differences there [36:11] are no other differences I'm going to [36:12] model this thing the same I'm going to [36:14] create objects I'm gonna index them I'm [36:16] going to create those objects as [36:17] documents I'm going to create those as [36:19] wide column data structures one or the [36:20] other but that's what I'm really doing [36:22] with no SQL so anyone who tells you that [36:24] there's a difference between these [36:25] systems is wrong there's just not okay [36:28] we're getting to complex queries it's [36:30] about understanding KPIs and metrics in [36:33] the system that are useful to the [36:35] application because most applications [36:36] require things like top in last in [36:39] average counts sums you know all these [36:42] types of complex queries that don't [36:43] really get supported in the DynamoDB API [36:46] so how do I support those queries and we [36:48] do this with change data capture [36:49] processing right I'm going to create a [36:51] pipeline off of what we call the dynamo [36:53] DB stream the stream is a running change [36:55] log of data update and write operations [36:59] that occur on the database every time [37:03] you do write to the table you're going [37:04] to see that object on the stream and [37:06] will be able to process that with lambda [37:08] what do people do with their lots of [37:10] different things the primary thing that [37:11] we're going to do with this is we're [37:13] going to do roll-ups [37:14] write summary metrics kpi's things that [37:19] I you know with I kind of need in my [37:21] application right one of the things we [37:22] found at Amazon we started to work out [37:25] you know how [37:26] we're gonna make these migrations and [37:27] actually how to scale our relational [37:28] databases sorry before we even [37:29] determined two migrations I was we [37:31] offloaded the cost of these complex [37:33] queries by creating you know summary [37:35] aggregation tables and once we started [37:37] doing this it was a clue right we said [37:39] you know it's too expensive to run the [37:41] query to get the count the summary the [37:42] average the max whatever the min and [37:44] we're and we're maintaining these is [37:46] running aggregations maybe what we [37:48] should do is no SQL and that's what we [37:49] do here right this is how you get [37:50] aggregation to scale anyone who's using [37:52] aggregation framework in MongoDB or in [37:55] document database is gonna find really [37:56] quickly as you scale your database that [37:58] those queries don't scale with you you [38:00] can end up remodeling your data you're [38:02] getting implement change data capture [38:03] pipelines like this to maintain those [38:04] summary metrics summary aggregations the [38:07] other thing people do is they're going [38:08] to go ahead and you know sink to [38:10] elasticsearch right if you have indexing [38:12] you know requirements that we don't [38:14] support in dynamo DB index intersections [38:17] geospatial full-text things that [38:20] DynamoDB doesn't support let's use [38:21] change data capture with streams and [38:23] lamda again one of the nice things about [38:25] cloud native this is a guaranteed [38:27] contract that happens with dynamodb [38:29] between the table and the lamda process [38:31] when you use lamda to process the stream [38:33] there's a trigger that occurs it's [38:34] guaranteed at least once processing make [38:36] sure you understand that it's at least [38:38] once code your ID your lambda functions [38:40] be idempotent because there are rare [38:41] conditions where container failure might [38:43] occur you're in the middle of processing [38:45] maybe I just processed the item I didn't [38:47] update the configuration tables yet [38:48] container refires Landa function picks [38:50] back up right where it thinks it left [38:52] off so understand that these functions [38:54] should be idempotent and when you code [38:56] them but there is a guaranteed contract [38:58] of at least ones processing feed that [39:00] data into Kinesis firehose roll it up [39:04] into s3 and park' files run athina [39:06] queries on it for real-time analytics a [39:08] lot of use cases where people do exactly [39:10] that it's a wonderful solution there get [39:12] your data off the table doesn't impact [39:14] and that's an asynchronous process you [39:16] have to worry about impacting your [39:17] DynamoDB table with any of this it's [39:20] like the world's best stored procedure [39:21] engine you're not going to knock the [39:23] server sideways by running a bad stored [39:24] procedure so you just get paid more I'm [39:27] okay with that [39:29] no no I'm not a lot a lot of my job is [39:31] spent trying to get people to pay less [39:33] trust me and I do I'm pretty good at it [39:35] I think I'm pretty good at it all right [39:38] so let's talk a little bit about [39:40] composite keys composite keys are [39:42] important part data modeling when we get [39:43] into no SQL it's about creating sort [39:45] keys that matter in this particular [39:47] example we have a a game sessions table [39:50] users have games sessions they have [39:53] different state we're looking for [39:54] sessions that are pending for giving [39:55] users sorted by date we might have an [39:57] index on the sessions table it's [39:59] partitioned on the user it's sorted on [40:01] the session date and it's and what we're [40:04] doing here is we're using order by [40:06] condition as the sort key condition [40:07] which is basically no sort key and then [40:09] we're using a filter condition filter [40:11] conditions and DynamoDB apply to [40:12] extended attributes any attribute that's [40:15] of the items you can say you know items [40:17] that have a particular attribute value [40:19] in this case we're saying where the [40:20] status is pending and so what happens [40:22] here is that we're reading we're reading [40:24] three items and we're we're returning - [40:27] I'm paying to read three in this case [40:29] reading three is the same cost as [40:30] reading two so that filter condition is [40:32] great but if I had 10,000 items in this [40:34] partition and only wanted to read those [40:35] two that could be pretty inefficient [40:37] read/write I'd have to get a whole bunch [40:38] of items and and pay for that and then [40:40] only return two so the way to fix that [40:42] is to use composite keys in this case [40:44] every time we insert the item we'll take [40:46] the status the date will create a new [40:47] sort key called status date now when I [40:49] query the table I can say begins with [40:51] pending and I get only the pending items [40:54] right so we're gonna use a lot of tricks [40:55] like this to create selective reads out [40:57] of very dense partitions right we've got [40:59] a lot of items in our partitions but the [41:00] reads are very thin so it's okay and I [41:03] have a lot of people say oh we're not [41:04] supposed to put lots of items in the [41:05] partitions no we can put all the items [41:07] we want in the partitions it's about the [41:09] velocity how fast am I writing how fast [41:11] am i reading and DynamoDB into a single [41:13] logical partition I can write 1,000 [41:15] WCU's that's kilobytes per second I can [41:18] read 3,000 WCU's at WC I read 3,000 RC [41:21] use and RC is 4 kilobytes a second so I [41:24] can effectively write at those rates and [41:27] read at those rates just be aware of [41:29] that is we're starting to lay the data [41:30] out now what we're going to talk about [41:33] next is how do we actually model complex [41:35] relational data right and I hear this a [41:37] lot right where's where you know how do [41:38] I do this without creating normalized [41:40] data models I need look-up tables I need [41:42] you know all these these constructs that [41:44] we're so used to using and and the [41:45] reality is as I showed you the time [41:47] complexity of those queries is too high [41:49] right we don't want to use those types [41:51] of data models because I mean if it's [41:53] it's that hard for a relational database [41:54] to do it how hard is it going to be for [41:56] your app server right I mean think about [41:59] atom traversing the table I'm gonna be [42:01] iterating result sets at the a player [42:03] it's going to be ugly it's going to be [42:04] grossly inefficient this is exactly what [42:06] we found out at Amazon so learn from our [42:09] mistakes and start to look at how we [42:10] actually model that relational data in a [42:12] way that makes sense so in this [42:13] particular application we're going to be [42:15] talking about employee portal last year [42:17] my service had delivery service had does [42:20] an application acts a dozen access [42:22] patterns I'm going to show you one now [42:23] that has 23 different access patterns [42:25] just to shame and the other question I [42:27] get a lot is how you evolve the [42:29] application right isn't it hard to add [42:30] new access patterns well this model [42:33] started with this employee table and it [42:36] was that document model that we saw [42:37] earlier right and I said you know let [42:39] let's add different things I added [42:41] tickets I added projects I added you [42:43] know a whole bunch of different entities [42:44] right I added that we're going to talk [42:48] about when we get into looking at this [42:50] data 23 different access patterns we're [42:53] accessing this data on multiple [42:54] dimensions in multiple ways and you know [42:58] let's take a look at what some of those [42:59] access patterns look like so on the [43:01] table we created many partitions right [43:03] may types of partitions then we loaded [43:04] these partitions with different types of [43:05] items right so the first partition we [43:07] created I was a building partition [43:09] buildings have meeting reservations they [43:12] have room definitions we have employee [43:15] partitions employee partitions have [43:16] meeting objects right they get invited [43:19] to meetings they have some employee [43:20] metadata in associate with them then we [43:24] have project partitions and project [43:25] partitions have time cards that are [43:28] submitted by employees employees have [43:29] roles on those projects and different [43:31] things that they did and whatnot they're [43:32] submitting projects also have project [43:34] metadata and then we have tickets write [43:37] tickets have and you know the you know [43:39] the subject the the item that's created [43:42] when we open the ticket they have [43:43] messages they're associated the tickets [43:44] they have owners they have assignees and [43:46] we're gonna try and access this data in [43:48] multiple ways tables gonna support nine [43:50] access patterns the first access [43:52] patterns gonna say get me the employee [43:54] metadata or get me the employees [43:56] meetings by employee ID so I'm going to [43:58] query that table by employee ID I'm [44:00] going to give it a date range condition [44:02] and I'm going to give it a filter [44:03] condition that says greater than zero [44:04] because you know I might have [44:06] you know items in there that end up [44:07] being date ranged and they're not [44:08] meetings but that filter condition is [44:11] going to knock out anything that's not a [44:12] meeting and most employee partitions are [44:14] not going to have a lot of items so [44:15] we're gonna be okay with you know [44:16] filtering out those items and making [44:18] that less than selective read the next [44:21] access patter we have is getting me all [44:22] the given meetings for a given building [44:24] during given time range so we can go [44:26] ahead and select by building ID in our [44:28] building partition again with a date [44:29] range condition and you know we're going [44:31] to use a contains condition on an [44:33] attribute that's a copy of the sort key [44:35] because in forward I don't know why but [44:37] in dynamodb API you can't use contains [44:39] on a sort key so if you actually need [44:40] that condition you have to create a copy [44:42] of the sort key and use contains on that [44:44] as a filter condition which is what [44:46] we'll do and that that copy of the sort [44:48] key contains the building the floor or [44:50] the room we're looking for to be able to [44:53] support that access pattern next one is [44:56] going to be get the employees metadata [44:58] by employee ID every employee has a [45:00] metadata item starts with E that's our [45:02] query condition so we query by employee [45:04] ID starts with e gives me that access [45:07] pattern the next pattern is going to be [45:10] get me the ticket history right so I [45:12] want to get the ticket history select [45:13] from the table from the ticket partition [45:15] and gives me the entire ticket history [45:17] for a given access pattern now we're [45:20] gonna be getting project information [45:22] right select my project name with a sort [45:24] key equals project name because that [45:25] item there's only a selected condition [45:28] inside of my partition that returns only [45:29] my project metadata I can query the [45:31] project partition with a date range [45:33] condition to bring back all the time [45:34] cards that were filed on that particular [45:37] project by different employees we can [45:40] also add the filter condition by role [45:42] and bring execute the same query and say [45:45] everybody was a technical product [45:46] manager on this particular project and [45:48] given time range so on and so forth [45:51] the next ya access patterns interesting [45:54] and a lot of customers use this we use [45:55] this internally for reservation systems [45:57] when we have these use cases at Amazon [46:00] oftentimes we know some things about you [46:02] know what reservations have been made [46:04] what things are available but we don't [46:06] necessarily know what things are [46:07] available until we make that calculation [46:08] right get the things that are [46:09] reservations that have been made and get [46:11] the things that are available and then [46:12] figure out you know so on so and so [46:13] forth so in this reservation workflow [46:15] what happens is the client comes in the [46:17] first query goes to the building it's [46:19] going [46:19] get the rooms that are available in the [46:21] building as soon as I open the portal to [46:22] reserve the room I'm gonna go and select [46:25] the time and say get me the rooms that [46:27] are available I'm not gonna get the [46:28] rooms are available and get the [46:29] reservations on that building and floor [46:31] and whatnot that they've selected during [46:33] that time window and I'm gonna let the [46:35] client figure out what rooms are [46:36] available because guess what I already [46:37] sent them down the room metadata those [46:39] are both very small queries but what [46:41] ends up happening at scale and we have [46:42] several customers who have used this [46:44] they've come back and said they've [46:45] actually been able to decrease the size [46:46] their application fleet server fleet [46:49] because they're not triaging that data [46:50] anymore the application server there's [46:52] shoving it all down to the client let [46:53] the client figure it out [46:54] alright the clients have an abundance of [46:56] CPU I'll show you another use case where [46:58] it's is a version control or versioning [47:01] workflow where we need to see versions [47:03] and just send the deltas to the client [47:05] instead of round tripping back to the [47:06] server to get the you know older [47:08] versions but anyways the idea here is [47:10] get the logic down to the client don't [47:12] necessarily push this logic into the [47:14] application server because that's going [47:16] to be expensive right so the other use [47:18] case they are getting to support that is [47:19] we get the room metadata and then of [47:21] course get me the the reservations for a [47:24] given building room whatnot [47:26] alright so next thing we're gonna do is [47:28] index these items and what you can start [47:30] to notice is these tables get smaller [47:31] and smaller because on the primary table [47:33] some of those items will only get [47:34] accessed on one dimension which is the [47:36] primary access pattern that they're [47:37] stored in on the table other items on [47:39] that table need to be indexed and this [47:41] is what we're gonna start to see as we [47:42] start to extend this out so this index [47:44] supports an additional 9 access patterns [47:46] the first access pattern that we're [47:48] gonna get here is get me all of the [47:49] employees meetings by email right one [47:51] access pattern was get my Eve get me by [47:52] employee ID the other ones could get by [47:54] email so I'll index all those meeting [47:56] objects on GSI one using the email the [47:59] next is a primary use case this is [48:01] saying I log in to my portal my portal [48:03] gives me everything for the last 30 days [48:05] which is all my meetings all my time [48:07] cards all my tickets all my messages on [48:09] those tickets everything so I would [48:11] query my employee partition by email and [48:13] say give me everything greater than 30 [48:14] days ago it brings back everything the [48:16] employee metadata all my tickets one [48:19] query five different access patterns [48:21] solved with one single query and this is [48:23] what we're going to try and do with no [48:24] SQL databases we're gonna try to create [48:26] queries that support the retrieval of [48:28] multiple types of objects when this is [48:30] required group these objects into [48:32] partitions on the [48:33] able to support these joints right these [48:35] are joints this is really what this is [48:37] this is this is flat-out a joint [48:39] okay the next access patterns going to [48:42] get me the employees metadata by [48:44] employee ID or by employee email as well [48:46] and then we're going to say this is get [48:56] the ticket history by employee email [48:57] right so we've got all our ticket [48:59] objects are going to be stored in those [49:01] employee partitions using a employee [49:03] email as well so that we can query on [49:06] that dimension we're going to want to [49:09] then go ahead and query our projects by [49:13] star oh this an interesting use that use [49:15] case I have one customer that needed to [49:16] have a start case of get me everything [49:20] that started within the last three [49:21] months it's going to finish in the next [49:22] three months right so it sounds like to [49:24] sort conditions and it kind of is two [49:26] conditions but you can satisfy this [49:27] using a sort condition of filter [49:28] condition all right if I query the table [49:30] and say give me everything with this [49:32] project name or everything across the [49:35] the the active projects in my in my [49:38] company that was started three months [49:40] ago with the greater than three months [49:41] ago and use a filter condition that says [49:43] an target date target delivery is less [49:46] than three months from now then really [49:48] what I've done is I've done kind of a a [49:49] none selective read from the partition [49:51] gotten every project that's currently [49:53] active and the filter condition knocks [49:55] out the ones they're you know not going [49:57] to deliver in the next three months this [49:59] was kind of a budgeting use case where [50:00] like I need to know what projects I'm [50:01] going to cut so I don't wanna cut [50:03] anything is gonna deliver in the next [50:04] three months but anything after that's [50:05] on the table for talking about right I [50:07] don't like those but they sometimes you [50:08] got to do it right alright next access [50:11] pattern is going to be against GSI - [50:13] we're starting to deliver a smaller [50:15] number of of access patterns here [50:19] because we have a fewer number of [50:22] objects that actually need to be indexed [50:24] on these partitions right so the first [50:26] thing we're gonna do here is going to be [50:28] getting the tickets by assignee email [50:30] right so we had to get the ticket by the [50:32] owner email who opened the ticket I want [50:35] to get all the ticket history and when I [50:37] get ticket history by assignee email so [50:39] every one of those ticket objects I [50:40] insert whether it's the opening of the [50:42] ticket or the messages that are [50:43] associated ticket I'm going to add [50:44] additional decorate those objects [50:46] additional attributes so I can pick them [50:48] up by the owner and by the assignee I'm [50:50] going to want to get all the employees [50:52] for a given manager right so in this [50:54] particular case we got all those [50:56] employee metadata items they're going to [50:57] pick up another indexed dimension so [51:00] that I can you know index them by their [51:02] employees manager a query GSI - with the [51:04] manager's ID then I'm going to get that [51:06] we get into the last use case here this [51:09] is the last index we're going to talk [51:10] about there's just a couple of use cases [51:11] on this one we need to on this first [51:16] query we're going to get all the [51:18] employees in a given ceiling but city [51:20] building or floor of a building this is [51:23] that subtree aggregation use case that [51:26] we talked about earlier I see this a lot [51:28] is you know people need to get all the [51:30] products in a given category give me all [51:31] the employees of given place you know we [51:33] can you know create a path just create a [51:35] path of the hierarchy and use begins [51:37] with on that path to get those subtree [51:39] aggregations we want to get all the [51:41] tickets that are languishing so you know [51:43] as I know there's a lot of tickets in [51:45] the organizational I message is going [51:46] down on those tickets I want to get [51:47] everything that hasn't been touched in [51:48] the last 24 hours for escalation so I'm [51:51] going to create a right sharted set of [51:53] partitions here you can see this ticket [51:55] lives in partition 7 there might be you [51:56] know 20 partitions here to increase the [51:58] throughput depending on number of [51:59] tickets my systems are processing but [52:02] again I need a thousand WCU's per [52:04] partition so if I'm aggravated you know [52:07] it cost ten partitions I can write you [52:09] know ten thousand you know tickets per [52:11] second would be a significant amount of [52:13] workload there but let's say the tickets [52:15] might be bigger than one kilobyte in the [52:17] end what we end up with is a set of [52:19] tables a table that describes the access [52:21] patterns that we were tasked with the [52:23] query conditions that I need the table [52:25] of the GSI that I'm going to query the [52:27] filter conditions to use and this is [52:29] exactly the exercise to go through with [52:31] every team I talked to you create that [52:33] entity relationship diagram what kind of [52:35] data are we working with define your [52:36] access patterns how am I actually [52:38] accessing this data how do I need to [52:40] store this data [52:41] okay let's define the data model and [52:42] let's come back with a set of like you [52:45] know clearly defined conditions that I'm [52:47] going to use to produce the results that [52:49] I'm looking for and again I built this [52:50] entire model in iterative fashion right [52:52] I started with one or two access [52:54] patterns and started adding more [52:55] more and more you know with real data [52:58] sometimes you might have to you know do [53:00] an ETL on the existing table and [53:02] decorate the table with the distant [53:03] existing items with new attributes you [53:06] might need to change the values and [53:07] existing attributes but the reality is [53:09] you're not going to have to throw the [53:10] baby out with the bathwater right this [53:12] is an evolution and it's not dissimilar [53:14] to what you go through with the [53:15] relational database it's just we're much [53:17] more familiar with that ok so let's get [53:19] into designing for common patterns this [53:21] is about understanding what type of when [53:24] to use big items why big items might not [53:27] necessarily be the thing for you in this [53:29] particular use case let's talk about a [53:31] customer I was talking to you recently [53:33] it's assurance insurance quoting service [53:34] fairly successful business they're doing [53:37] about 800 quotes per minute peak their [53:39] provision a thousand WCU's on their [53:41] dynamo table the data model looks like [53:42] this basically consumers come in they [53:44] open up insurance quote that create a [53:47] new quote they're going to start [53:48] creating versions of that quote by [53:50] changing things you know I Drive more [53:52] than that or you know I live here [53:54] no actually I'm going to be doing this [53:56] here what not basically the every [53:58] customer might change the quote two or [54:00] three times every time they change the [54:02] quote they would create a new copy of [54:04] the quote with a new version so they had [54:05] a customer ID as the partition key their [54:07] star key was along the lines of quote ID [54:09] under bar version when a customer would [54:12] say okay show me version my current [54:13] version they would just you know scan [54:15] for word index false limit one they [54:17] would get the most current version as [54:19] customers started going back and [54:20] paginating through these versions they [54:22] would look at and go back to the [54:23] database to get a new version right so [54:25] it was great for them they loved it that [54:26] came to me started asking me about [54:27] optimizations is there anything we can [54:29] do here I said well sure the first thing [54:31] we can look at is how much data changes [54:33] when I update a quote there's a lot of [54:36] data change does a little data change [54:38] it turns out that really only a couple [54:40] attributes change the matter of fact 90% [54:42] of the quote is immutable data right [54:44] it's like a dress it's not immutable but [54:46] it never changes right a dress name you [54:49] know phone numbers things like that and [54:51] every time they were creating a version [54:52] of the quote they were writing all that [54:54] data back to the database and creating a [54:56] 50 kilobyte you know update so what we [54:58] did was said look okay why don't you [54:59] store the deltas on the table inside the [55:01] partition change your schema slightly [55:03] now what you've got is a quote ID and [55:06] all those version attributes those are [55:08] just very [55:08] of the various sections of your quote [55:10] and we're going when you query the quote [55:12] you're gonna send down all these version [55:14] objects and you can push the logic to [55:15] assemble the most current view into the [55:17] endpoint write that on the browser and [55:20] so now they drop their provision [55:22] capacity on the table from a thousand to [55:25] fifty WCU's okay because most of the [55:28] time they're not creating big blobs I [55:30] mean every now and then they create a [55:31] new quote but most of their traffic was [55:33] just updates to existing quotes and if [55:36] they ever go above fifty WCU's that's [55:37] okay because they'll just go ahead and [55:38] use the burst bucket to satisfy if they [55:40] get four or five simultaneous quotes [55:42] being created they have no problems they [55:44] see spikes regularly over their [55:45] provision wcu they're fine with that so [55:48] this is a really good example of how I [55:50] can optimize the data model by thinking [55:55] about how I'm implementing the system [55:57] and how it affects the entire stack all [55:59] right we have a lot of customers that do [56:00] things like this to come back and say [56:02] you know I was able to decrease my [56:03] application server fleet by 5% because [56:05] I'm no longer doing I'm not handling [56:07] that data in the application server [56:09] anymore I'm just passing it off to the [56:11] front end and letting them do that [56:13] triage as a matter of fact in this [56:14] system imagine they don't even have to [56:16] come back to the server to get the [56:17] previous versions that the customer gets [56:19] of the current versions is back back [56:20] back there's not one request going back [56:22] to the front-end to the API of their [56:24] service all right it's all sitting there [56:26] on the front end and they know what it [56:27] is okay one of the things I got to say [56:30] before we stop no SQL work bench for [56:32] dynamodb I don't know how many folks are [56:34] aware of this we released this tool a [56:35] couple months ago this is the best tool [56:37] out there for modeling no SQL databases [56:38] today it will build the types of views I [56:41] show you in my presentations these [56:43] aggregate views at the table you can [56:45] what's nice about this is you can create [56:47] just some JSON data define the JSON data [56:50] throw it into the into the modeler and [56:51] then you can start defining indexes on [56:53] your various attributes and I'll [56:54] actually show you how the data your [56:56] sample data lays out it also has load [56:58] from a relational database you can [57:01] connect a relational database it has a [57:02] code generator you can use the code [57:04] generator to you know help build your [57:07] own applications it's it's rock-solid [57:09] stuff there's a nice easy to read URL [57:12] there but I couldn't find anything [57:15] easier so that's what you got but with [57:18] those charts will be online if you [57:19] google for no SQL work bench for dynamo [57:21] DB you'll find it [57:22] easily enough it's really a great tool I [57:24] got more and more customers coming to me [57:25] with their models already done and it [57:27] makes it nice because I can just load it [57:29] in that we can make our tweaks and show [57:30] them exactly what needs to happen [57:32] all right conclusions no SQL is not [57:36] about non-relational data please don't [57:38] use that term if you'd notice I didn't [57:40] use that term very often other than to [57:41] say it doesn't exist because it doesn't [57:43] exist it's a marketing term that was [57:45] invented by marketing people who tried [57:46] describe technology they don't [57:47] understand to other people who don't [57:49] understand it and they came up with this [57:50] it's not relational it must be non [57:52] relational don't use that don't fall in [57:54] that trap [57:54] data is all relational or doesn't matter [57:56] the erd still matters yeah a relational [57:59] database is not deprecated by no SQL you [58:02] know use no SQL for OLTP or decisions [58:05] poor systems at scale use that [58:07] relational database for OLAP but I [58:09] always think cloud native first I'll [58:11] tell you the elasticity of the the AWS [58:14] compute cloud is amazing and of cloud [58:16] native services like dynamo DB is is [58:18] unparalleled in scale so I would go [58:21] there first all right we got a lot of [58:23] tools for you guys to look at when you [58:25] want to learn more about AWS database [58:29] services go here AWS training and [58:32] certification lots of online resources [58:35] digital training courses we've got some [58:37] real good stuff coming out from our [58:39] partners as well Linux Academy I think [58:41] just released a new DynamoDB training a [58:43] modeling course and definitely worth [58:45] taking a look at but there's some tools [58:47] out there for you like there's never [58:48] been there's a lot of information for [58:50] you to kind of master this stuff and [58:52] hopefully you got some good information [58:53] from me today so thank you very much for [58:56] your time [59:02] so by the way if you didn't like the [59:04] session then it's that 450 if you like [59:08] it it's 403 please rate me