Google Infrastructure Rules Stall Basic Five-Terabyte Deployment

I just want to serve 5 terabytes. — Benjamin Staffin

Benjamin Staffin

Rigid reliability mandates and readability gatekeeping render trivial storage workloads impossible to launch without multi-zone over-engineering and peer-bonus bribery.

AI-assisted summary of I just want to serve 5 terabytes. — original source. Check the original for full context.

Source published:

Key ideas

  • A routine request to serve a five-terabyte dataset escalates into a complex distributed systems architecture due to Google-scale overengineering and procedural bureaucracy. (0:00)
  • Single-instance deployments are rejected over Planned Change Request (PCR) risks, forcing the engineer to purchase dual-cell Bigtable quotas across distinct PCR zones with manual replication tickets. (0:00)
  • Production readiness mandates internal Corp Google Front End (GFE) DNS failover and Borgmon monitoring, but implementation is blocked by rigid Borg naming conventions and readability gatekeeping. (1:41)
  • Internal peer bonuses become a black-market currency for code reviews, demanding six bonuses just to inspect and approve a 20-line monitoring configuration. (1:41)

Detailed notes

## Background Published on YouTube by Benjamin Staffin on November 2, 2021, this video originates from a circa-2010 piece titled "Broccoli Man (Production Issues)" by Jon Orwant. The video is credited as the origin of the tech catchphrase "I've forgotten how to count that low," having sparked recurring discussions across Hacker News and engineering blogs regarding large-scale infrastructure complexity.

## TL;DR An engineer attempts to deploy what should be a straightforward five-terabyte dataset, but their simple plan is immediately dismissed by an arrogant senior engineer who insists on building a multi-cell, production-grade architecture. The deployment rapidly balloons into an infrastructure nightmare involving Bigtable replication across multiple Planned Change Request (PCR) zones, Corp GFE DNS failover, and Borgmon monitoring setups. Along the way, the engineer is forced to navigate strict readability gatekeeping and bribe the senior colleague with peer bonuses just to secure code review approvals. The task finally collapses into absurdity when the deployment is blocked by an unallocated compute quota for the monitoring system itself, driving the engineer to homicidal frustration.

## Atmosphere & style The interaction is a biting, Kafkaesque satire of corporate engineering culture, defined by smug elitism, bloated bureaucracy, and ever-shifting architectural goalposts. The dialogue crackles with deadpan comedic tension between the pragmatic junior engineer trying to complete a task and the dismissive senior engineer wielding company dogma like a weapon. As trivial engineering steps morph into absurd logistical hurdles and peer bonus shakedowns, the mood steadily escalates from exasperated bewilderment into darkly hilarious, exhausted rage.

## Bulletin - [00:00](https://www.youtube.com/watch?v=3t6L-FlfeaI&t=0s) A routine request to serve a five-terabyte dataset escalates into a complex distributed systems architecture due to Google-scale overengineering and procedural bureaucracy. - [00:00](https://www.youtube.com/watch?v=3t6L-FlfeaI&t=0s) Single-instance deployments are rejected over Planned Change Request (PCR) risks, forcing the engineer to purchase dual-cell Bigtable quotas across distinct PCR zones with manual replication tickets. - [01:41](https://www.youtube.com/watch?v=3t6L-FlfeaI&t=101s) Production readiness mandates internal Corp Google Front End (GFE) DNS failover and Borgmon monitoring, but implementation is blocked by rigid Borg naming conventions and readability gatekeeping. - [01:41](https://www.youtube.com/watch?v=3t6L-FlfeaI&t=101s) Internal peer bonuses become a black-market currency for code reviews, demanding six bonuses just to inspect and approve a 20-line monitoring configuration.

## Key takeaways - [00:00](https://www.youtube.com/watch?v=3t6L-FlfeaI&t=0s) A senior engineer dismisses a five-terabyte storage request with corporate hubris, claiming five terabytes is too small to count. - [00:00](https://www.youtube.com/watch?v=3t6L-FlfeaI&t=0s) Deploying a basic App Engine app requires a labyrinth of testing and review hurdles, including unit, regression, and end-to-end test suites. - [00:00](https://www.youtube.com/watch?v=3t6L-FlfeaI&t=0s) Originating data from an internal cron job disqualifies simple App Engine storage in favor of Bigtable. - [00:00](https://www.youtube.com/watch?v=3t6L-FlfeaI&t=0s) Single-cell Bigtable instances are deemed non-production grade because maintenance via Planned Change Requests (PCRs) will disrupt them. - [00:00](https://www.youtube.com/watch?v=3t6L-FlfeaI&t=0s) High availability requires buying quota in multiple Bigtable cells across different PCR zones and submitting an internal replication ticket. - [01:41](https://www.youtube.com/watch?v=3t6L-FlfeaI&t=101s) Internal services are expected to fail over cleanly between PCR zones using a Corp Google Front End (GFE) DNS entry rather than running in a single cell. - [01:41](https://www.youtube.com/watch?v=3t6L-FlfeaI&t=101s) Deploying without active monitoring is berated as showing contempt for end users. - [01:41](https://www.youtube.com/watch?v=3t6L-FlfeaI&t=101s) Diplomat monitoring is rejected due to its requirement for strict, predefined Borg job naming conventions. - [01:41](https://www.youtube.com/watch?v=3t6L-FlfeaI&t=101s) Adopting Borgmon creates an administrative roadblock because virtually no engineers hold Borgmon readability certification. - [01:41](https://www.youtube.com/watch?v=3t6L-FlfeaI&t=101s) The senior engineer extorts six peer bonuses—three for code review and three for approval—to sign off on a 20-line Borgmon changelist. - [01:41](https://www.youtube.com/watch?v=3t6L-FlfeaI&t=101s) The deployment encounters a final catch-22 when the approved Borgmon monitoring job itself cannot launch without purchasing additional compute quota in its host cell.

## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=3t6L-FlfeaI&t=0s) The "Simple" Request and Google-Scale Escalation

A software engineer initiates what should be a trivial request: they want to serve a modest five terabytes of data. The senior engineer dismisses the size immediately with staggering corporate hubris, remarking, “Five terabytes is so little that I've forgotten how to count that low.” Encouraged, the engineer plans to deploy the workload to Google App Engine. However, the path to a basic App Engine launch quickly balloons into a labyrinth of internal engineering bureaucracy: registering an App Engine account, writing Python, drafting unit tests, regression tests, and end-to-end tests, creating a changelist (CL), passing the submit queue, securing approval, waiting for the build to turn green, and finally submitting the code. When the engineer notes that this setup will take about a week and mentions the dataset originates from an internal cron job, the senior engineer scoffs and insists the data must be stored in Bigtable instead.

The engineer resigns themselves to Bigtable for the sake of building a "production quality service," only to be met with outright ridicule. The senior engineer retorts, “I made a more robust production quality service squatting over the toilet this morning.” When questioned why App Engine and Bigtable do not qualify as production-grade, the senior engineer invokes the specter of a Planned Change Request (PCR) impacting the serving cell. To withstand maintenance disruptions, the engineer is instructed that they must purchase quota across multiple Bigtable cells in entirely separate PCR zones and submit an internal ticket to set up cell replication, completely shattering any illusion that hosting a simple five-terabyte dataset would be straightforward.

```mermaid flowchart TD accTitle: Escalation of 5 Terabyte Architecture accDescr: Flowchart showing the expanding requirements from basic storage to multi-cell failover and monitoring.

subgraph Storage ["Storage & Replication"] A["5 TB Dataset"] --> B["Bigtable Cell A"] A --> C["Bigtable Cell B"] B -.->|"Replication Ticket"| C end

subgraph Routing ["Traffic Management"] B --> D["PCR Zone 1"] C --> E["PCR Zone 2"] D --> F["Corp GFE DNS Entry"] E --> F end

subgraph Ops ["Observability"] F --> G["Borgmon Setup"] G --> H["Readability Review (3 Peer Bonuses)"] H --> I["Approval (3 Peer Bonuses)"] I --> J["Cell Quota Allocation"] end ```

- Initial deployment steps demanded: Account creation, Python code, unit tests, regression tests, end-to-end tests, CL submission, queue clearance, code review approval, and green build. - Data storage mandates: Deprecating pure App Engine storage in favor of Bigtable due to the cron-generated source. - High-availability mandates: Purchasing quota in two distinct Bigtable cells residing in independent PCR (Planned Change Request) zones, accompanied by an explicit cell replication ticket.

[01:41](https://www.youtube.com/watch?v=3t6L-FlfeaI&t=101s) DNS Failover, Borgmon Readability, and the Peer Bonus Economy

Even after provisioning dual-cell quota and filing replication tickets, the service is still deemed inadequate. The senior engineer compares the current architecture to an inferior competitor, declaring, “If you're Yahoo, sure. This is Google. You need to fail-over cleanly between PCR zones.” The engineer is ordered to launch into both cells and configure an internal Corp Google Front End (GFE) DNS record to route traffic between the instances. When the engineer asks again if the project is finally production quality, they are berated for failing to implement monitoring: “Only if you think your users are scum. Do you? Do you think your users are scum? Why do you hate your users?”

``` +------------------------------------+-----------------------------------------------------+ | Proposed Solution | Bureaucratic / Architectural Blocker | +------------------------------------+-----------------------------------------------------+ | App Engine + Cron | Inadequate; must use Bigtable | | Single Bigtable Cell | Vulnerable to maintenance (PCR hits) | | Multi-cell Bigtable | Must sit in separate PCR zones with corp GFE DNS | | Diplomat Monitoring | Requires specialized Borg job naming conventions | | Borgmon Monitoring | Requires language readability for CL approval | | Borgmon Readability Approval | Extorted for 6 total peer bonuses | | Final Deployment | Blocked by missing quota in the Borgmon cell | +------------------------------------+-----------------------------------------------------+ ```

Attempting to resolve the monitoring mandate, the engineer suggests using Diplomat, only to discover it requires strict, predefined Borg job naming conventions. Pivoting in exasperation to Borgmon, the engineer is advised to copy someone else's configuration, only to collide with Google's readability gatekeeping. The senior engineer admits that virtually nobody holds Borgmon readability except themselves, and refuses to review the 20-line CL without extortion: “One peer bonus? I don't get out of bed for less than three.” After the engineer agrees to pay three peer bonuses for a code review and is extorted for an additional three peer bonuses to actually receive the formal approval, the senior engineer casually drops a final administrative roadblock—asking if the engineer remembered to purchase compute quota in the cell designated to run the Borgmon instance itself. Pushed to homicidal rage, the engineer vows to quit and spend their next job organizing the senior engineer's internal organs in parody of the company's core mission.

- Routing requirement: Dual-cell active launch managed via a Corp GFE DNS entry for cross-zone failover. - Monitoring hurdles: Abandoning Diplomat due to naming constraints, switching to Borgmon, and facing the strict requirement of finding an engineer with official Borgmon readability. - The peer bonus economy: The staff software engineer demands a bribe of three peer bonuses to look at 20 lines of Borgmon code, and another three peer bonuses to issue the LGTM/approval. - The final catch-22: Even with code approved, deployment is stalled due to a lack of quota allocated in the specific cell where the Borgmon monitoring job itself must run.

## Mentioned Google, Google App Engine, Bigtable, Borg, Borgmon, Diplomat, Google Front End (GFE), Yahoo, Python, Jon Orwant, Benjamin Staffin

## Video description This video dates back to about 2010, and is the origin of the phrase "I've forgotten how to count that low" that was recently referenced in a blog post and on hacker news. (https://acesounderglass.com/2021/10/20/i-dont-know-how-to-count-that-low/)

Even more related discussion: https://news.ycombinator.com/item?id=29082014 (2021-11-02) https://rachelbythebay.com/w/2012/04/06/5tb/ https://rachelbythebay.com/w/2021/10/30/5tb/

Original title: "Broccoli Man (Production Issues)" Original author: Jon Orwant (https://www.youtube.com/channel/UC0kuyh1g0DOQiaoxb4zIEhg)

[Source](https://www.youtube.com/watch?v=3t6L-FlfeaI)

中文简报

谷歌基础设施规范阻碍基础5TB存储部署 僵化的可靠性要求与可读性把关，使得若不进行多可用区过度工程化并用同行奖金“贿赂”，微不足道的存储工作负载便无法上线。 一名工程师试图部署一个本应十分简单的5TB数据集，但其简易方案随即遭到一名傲慢的高级工程师驳回，后者坚持要求构建多Cell的生产级架构。部署工作迅速演变成一场基础设施噩梦，涉及跨多个计划变更请求（PCR）可用区的Bigtable复制、企业GFE DNS故障转移以及Borgmon监控配置。在此过程中，该工程师不得不应对严格的代码可读性把关，甚至只能用同行奖金贿赂高级同事以获取代码审查批准。最终，部署因监控系统本身的计算配额未获分配而受阻，整个任务陷入荒谬境地，令该工程师陷入极度崩溃和抓狂。 由于谷歌规模的过度工程化和流程官僚主义，一个提供5TB数据集服务的常规请求演变成了一套复杂的分布式系统架构。 (0:00) 单实例部署因计划变更请求（PCR）风险遭到拒绝，迫使该工程师在不同的PCR可用区购买双Cell Bigtable配额，并提交手动复制工单。 (0:00) 生产就绪规范强制要求内部企业谷歌前端（GFE）DNS故障转移及Borgmon监控，但其实现却因僵化的Borg命名规范与可读性把关而受阻。 (1:41) 内部同行奖金沦为了代码审查的“黑市货币”，仅为了审查并批准一份20行的监控配置就需要支付六笔同行奖金。 (1:41)

Reading summary

Benjamin Staffin's narrative illustrates how an ostensibly trivial task—serving five terabytes of data—spirals into an unmanageable labyrinth of infrastructure redundancy, code-review gatekeeping, and bureaucratic roadblocks. Architectural Escalation to Multi-Cell Redundancy (0:00) What begins as a straightforward proposal to host five terabytes via Google App Engine immediately encounters dismissive corporate hubris and escalating infrastructure requirements. Because the data originates from an internal cron job, App Engine is rejected in favor of Bigtable. However, a single Bigtable deployment is declared inadequate to handle maintenance disruptions from Planned Change Requests (PCRs). The engineer is subsequently forced to purchase storage quota across multiple Bigtable cells in separate PCR zones, file an internal replication ticket, and configure a Corp Google Front End (GFE) DNS record to ensure clean cross-zone failovers. Observability Constraints and Readability Gatekeeping (1:41) Achieving acceptable production standards further requires dedicated monitoring, triggering a series of internal tooling hurdles. A proposal to use Diplomat is ruled out due to strict Borg job naming conventions, leaving Borgmon as the primary alternative. Deploying even a copied 20-line Borgmon configuration stalls because Google's readability system requires approval from certified engineers. With Borgmon readability heavily concentrated among senior staff, the review process becomes an institutional bottleneck that blocks code submission. Transactional Peer Bonuses and Quota Catch-22s (1:41) The governance framework degenerates into an informal transactional economy when the senior engineer leverages readability gatekeeping for personal gain. The senior engineer demands three peer bonuses simply to review the 20-line configuration and an additional three peer bonuses to provide the final approval. Even after the engineer concedes to paying six peer bonuses, deployment is halted by a recursive administrative barrier: failing to allocate compute quota in the specific cell designated to run the Borgmon monitoring job itself. The account highlights how hyper-scaled infrastructure requirements, extreme reliability mandates, and bureaucratic gatekeeping can paralyze basic technical tasks, reducing simple deployments to absurd procedural gridlock.

精炼阅读摘要

Benjamin Staffin 呈现了一则针对超大规模技术组织工程官僚主义的讽刺寓言：一名工程师仅需托管 5 TB 的适度数据，却在资深工程师对“Google 级别生产标准”的严苛执念下，被迫陷入架构过度设计、监控可读性特权寻租与配额死锁的循环困境。 生产标准驱动下的存储架构级联膨胀 (0:00) 一名软件工程师最初仅提出托管 5 TB 数据的常规需求，但资深工程师对此极度轻视，宣称该数据量小到已无需计算。当工程师计划使用 Google App Engine 配合 Python 部署，并准备花费约一周时间完成创建账号、编写单元测试、回归测试、端到端测试、代码变更列表（CL）排队与审批等一整套内部合规流程时，资深工程师却因数据源自内部 cron 定时任务而全盘否定该方案，强制要求转用 Bigtable 存储。 然而，单实例 Bigtable 仍被斥为不具备真正的“生产级质量”。为应对计划变更请求（PCR）对服务单元（Cell）带来的维护冲击，资深工程师强行推高容灾标准，要求必须在两个完全独立的 PCR 区域分别购买 Bigtable Cell 配额、提交底层数据复制工单，并配置 Corp GFE（Google Front End）内部 DNS 记录以实现跨区自动故障转移。原本简单的 5 TB 托管任务，迅速演变为高成本的双活跨区容灾架构。 监控合规门槛与代码可读性特权寻租 (1:41) 在双 Cell 架构部署就绪后，资深工程师以“缺乏监控等于仇视用户”为由，要求必须配备生产级可观测性。工程师起初提议使用轻量级工具 Diplomat，却因无法满足严苛的 Borg 任务命名规范而被迫放弃；转而配置 Borgmon 时，又撞上了内部语言“可读性”（Readability）认证的审批壁垒。由于全组织内极少有人持有 Borgmon 可读性资质，掌握该特权的资深工程师借此进行公开寻租。 面对仅有 20 行的 Borgmon 配置代码，资深工程师声称“少于三个同行奖金（Peer Bonus）绝不干活”，索要 3 个同行奖金作为查看代码的门槛，随后又要求追加 3 个同行奖金才肯给出正式批准（LGTM）。原本用于保障代码质量的内部认证机制，在此异化为特权群体索取个人绩效利益的行政路障。 资源配额死锁与工程官僚体制的反噬 (1:41) 即便工程师妥协并承担了总计 6 个同行奖金的索贿代价，整个上线流程依然无法收尾。在代码审查获批的关键节点，资深工程师提出了致命的行政阻碍：质问工程师是否在专门运行该 Borgmon 监控实例的特定 Cell 中预先采购了计算配额。由于缺乏部署监控工具本身的计算资源配额，即便业务服务准备完毕，整体方案依旧处于被底层流程卡死的停滞状态。 层层加码的容灾指标、内部权限垄断与相互依赖的资源配额，彻底摧毁了开发者的工作积极性。极端的流程合规与过度工程化未能提升交付质量，反而将原本微小的业务诉求逼入无法落地的死循环，使工程师在极度愤怒中产生辞职冲动，折射出大厂技术制度本末倒置的荒谬性。 该叙事揭示了大型科技企业在追求极限高可用与严苛合规过程中出现的制度异化：当生产级标准完全脱离实际业务规模、工程治理演变为特权寻租、基础设施陷入循环配额依赖时，工具与流程的内部复杂性最终反噬了软件开发最基础的交付初衷。

Detailed notes — zh

## Background 该视频由 Benjamin Staffin 于 2021 年 11 月 2 日发布在 YouTube 平台上，其原作《Broccoli Man (Production Issues)》出自 Jon Orwant，可追溯至 2010 年左右。视频简介指出，本片是知名名言“我已经不会数这么小的数了”（I've forgotten how to count that low）的出处，曾被多篇技术博客及 Hacker News 广泛讨论与引用。 ## TL;DR 故事围绕一名普通工程师想要在 Google 内部对外提供区区 5TB 数据服务展开。原本极为朴素的需求，在一位高高在上的主任软件工程师（Staff Software Engineer）指点下，被层层追加了极其冗杂的合规与架构要求：从 App Engine 强制迁移至 Bigtable，到跨 PCR 区域的多 Cell 容灾复制、内部 GFE DNS 解析，再到 Borgmon 监控配置。最终，这位资深工程师甚至利用稀缺的 Borgmon Readability 审批权限索要大量 Peer Bonus 作为审查筹码，并在通过审批的最后一秒追加了新的配额要求，彻底将工程师逼入崩溃并引发辞职威胁。 ## Atmosphere & style 全片充斥着荒诞冷酷的黑色幽默与尖锐的职场讽刺，精准还原了大厂工程师面对过度设计（Over-engineering）与官僚主义枷锁时的窒息感。对话双方的反差极其强烈——一边是自命不凡、教条主义且坐地起价的资深“老法师”，另一边则是从耐心配合、困惑抗议，最终彻底理智蒸发、口吐致命威胁的倒霉基层。观看过程如同一场不断向下坠落的工程噩梦，让经历过类似大厂内耗的读者在会心爆笑之余感到脊背发凉。 ## Bulletin - [00:00](https://www.youtube.com/watch?v=3t6L-FlfeaI&t=0s) 大厂严苛的基础设施与规范极易导致过度工程化，哪怕仅是提供 5TB 的微小数据服务，也会被迫演变为复杂的分布式多 Cell 容灾系统。 - [00:57](https://www.youtube.com/watch?v=3t6L-FlfeaI&t=57s) 高可用架构伴随着极高的内部协作摩擦，从 PCR 隔离、跨 Cell 复制到 GFE DNS 故障切换，每一步都需要层层提交流程和配置审批。 - [02:02](https://www.youtube.com/watch?v=3t6L-FlfeaI&t=122s) 大厂内部的代码审查权限（如 Readability）与有限资源容易滋生寻租现象，甚至催生以 Peer Bonus 换取代码审批的内部政治与灰色交易。 - [02:30](https://www.youtube.com/watch?v=3t6L-FlfeaI&t=150s) 无休止的官僚流程与运维门槛严重消耗一线开发者的心智，最终导致工程师对组织文化产生强烈反弹与离职冲动。 ## Key takeaways - [00:00](https://www.youtube.com/watch?v=3t6L-FlfeaI&t=0s) 工程师带着对外提供 5TB 数据的朴素需求寻求建议，但因数据量过小被对方嘲弄为“小到不会数这个数”。 - [00:15](https://www.youtube.com/watch?v=3t6L-FlfeaI&t=15s) 即便使用 App Engine，也必须严格遵循编写单元测试、回归测试、E2E 测试以及走完代码提交通道的繁琐工程标准。 - [00:35](https://www.youtube.com/watch?v=3t6L-FlfeaI&t=35s) 因为数据源自内部定时任务，方案被强制推翻并要求将数据持久化迁移至分布式数据库 Bigtable。 - [00:57](https://www.youtube.com/watch?v=3t6L-FlfeaI&t=57s) 为应对计划内变更请求（PCR），系统必须跨多个 Bigtable Cell 进行数据复制以满足生产级鲁棒性。 - [01:15](https://www.youtube.com/watch?v=3t6L-FlfeaI&t=75s) 仅仅申请双 Cell 配额并不够，还必须确保它们位于不同的 PCR 隔离区域并提交工单以打通复制。 - [01:30](https://www.youtube.com/watch?v=3t6L-FlfeaI&t=90s) 为实现真正的平滑故障切换，架构进一步蔓延到多实例部署与内部 Corp GFE 上的动态 DNS 流量解析。 - [02:02](https://www.youtube.com/watch?v=3t6L-FlfeaI&t=122s) 上线服务被强加上纲上线的监控考核，而在内部工具 Diplomat 命名不符后只能退回复杂的 Borgmon。 - [02:20](https://www.youtube.com/watch?v=3t6L-FlfeaI&t=140s) Borgmon 提交配置必须具备稀缺的 Readability 权限，促使审查者将审查与批准拆分并索要高达 6 个 Peer Bonus。 - [02:50](https://www.youtube.com/watch?v=3t6L-FlfeaI&t=170s) 当工程师耗尽心力打通所有流程后，对方又抛出监控进程自身所在 Cell 配额未申请的致命阻碍。 - [03:05](https://www.youtube.com/watch?v=3t6L-FlfeaI&t=185s) 工程师在无休止的过度工程与官僚勒索下彻底崩溃，借用 Google 著名使命宣言发出辞职与暴力宣泄的讽刺终结。 ## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=3t6L-FlfeaI&t=0s) “我已经不会数这么小的数了”：从 App Engine 到 Bigtable 的噩梦 一位工程师带着极度朴素的需求前来寻求建议：“I would like to serve five terabytes of data.”（我想对外提供 5TB 的数据服务）。对于规模庞大的 Google 而言，5TB 简直微不足道。对方极其傲慢地回应：“No problem, we are Google. Five terabytes is so little that I've forgotten how to count that low.”（没问题，我们可是 Google。5TB 实在太少了，少到我已经忘记怎么数这么小的数字了）。满心以为事情很轻松的工程师提议直接使用 App Engine，对方表示这确实“非常容易”，只要走完一套完整的 Google 内部研发与提交流程即可：注册 App Engine 账号、写一点 Python 代码、编写单元测试（Unit Test）、编写回归测试（Regression Test）、编写端到端测试（End-to-End Test）、创建变更列表（CL，Changelog）、通过自动化提交队列（Submit Queue）、获取代码审查批准（LGTM / Approval）、等待构建变绿通过，最后完成提交。 工程师无可奈何地接受了这个“只要花上一周”的繁琐流程，随后补充了一个小细节：他的底层数据是由一个内部定时任务（cron job）生成的，询问这是否会有影响。对方立即以标准的 Google 式工程教条否定了 App Engine 的方案，直截了当地宣称：“This is Google. You'll need to store the data in a bigtable.”（这里是 Google。你必须把数据存进 Bigtable 里）。当工程师感叹这虽然有点折腾、但为了做成“生产级质量的服务”（Production quality service）或许还算值得时，对方立刻冷嘲热讽起来：“You call that production quality? I made a more robust production quality service squatting over the toilet this morning.”（你把那叫生产级质量？我今天早上蹲马桶时拉出来的玩意都比它更具生产级鲁棒性）。 ```mermaid flowchart TD accTitle: 从 5TB 极简需求到 Bigtable 架构蔓延的示意图 accDescr: 展示工程师原本只需提供 5TB 数据，却被 Google 繁重的内部标准一步步逼上架构膨胀之路的过程。 subgraph UserGoal["原始需求"] A["提供 5TB 数据服务"] end subgraph AppEngineFlow["初版方案：App Engine"] B["Python 代码 + 三重测试<br/>(Unit, Regression, E2E)"] C["创建 CL 并通过 Submit Queue"] D["获取 Approval 并等待 Build 变绿"] end subgraph BigtableRequirement["架构膨胀：Bigtable 介入"] E["底层数据来自 Cron Job"] F["必须迁移至 Bigtable 存储"] end A --> B B --> C C --> D D --> E E --> F ``` ### [00:57](https://www.youtube.com/watch?v=3t6L-FlfeaI&t=57s) 架构蔓延：PCR 区域、跨 Cell 复制与 GFE DNS 故障切换 困惑的工程师反问为什么 App Engine 配合 Bigtable 还算不上生产级别，对方惊呼：“Oh sweet lord in heaven. What happens when a PCR hits your serving cell?”（老天爷啊。当计划内变更请求 PCR 砸向你的服务单元 cell 时，你该怎么办？）。工程师甚至连 PCR 是什么都不知道，对方解释那是“计划变更请求”（Planned Change Request），并再次搬出大厂口吻：“This is Google. You need to replicate your data across multiple bigtable cells.”（这里是 Google。你需要把数据跨多个 Bigtable cell 进行复制）。工程师按照要求在两个 Bigtable cell 中购买了 Quota（配额），以为大功告成，却再次遭遇当头棒喝：他根本没有确认这两个 cell 是否处于不同的 PCR 隔离区域（PCR zones）。 面对不断凭空冒出的运维规矩，工程师崩溃地抗议道：“Look, I just want to serve 5 terabytes. This shouldn't be hard.”（听着，我特么只想提供 5TB 数据。这不应该这么难）。但对方坚称这一点都不难，甚至紧接着追加了一条漏掉的流程：必须提交工单（file a ticket）以正式建立 cell 之间的数据复制。 当工程师办完工单后再次询问是否达标，对方不屑一顾地嘲讽：“If you're Yahoo, sure. This is Google. You need to fail-over cleanly between PCR zones.”（如果你是雅虎，那确实够了。但这里是 Google。你必须在 PCR 区域之间做到平滑的故障切换）。为了实现这一目标，工程师被要求将服务同时部署发布至两个 cell，并在内部公司级前端网络（Corp GFE）上配置 DNS 解析条目，以便流量在两个实例之间进行动态路由与无缝容灾。 ### [02:02](https://www.youtube.com/watch?v=3t6L-FlfeaI&t=122s) 监控勒索：Borgmon Readability 与 Peer Bonus 交易 面对越来越冗长的清单，工程师直言“我开始恨你了”，但还是硬着头皮完成了 GFE DNS 的配置。然而对方的挑剔变本加厉，甚至上纲上线进行道德拷问：“Only if you think your users are scum. Do you? Do you think your users are scum? Why do you hate your users? How will you know when your service goes down? Have you set up monitoring?”（除非你觉得你的用户全都是人渣。你是这么想的吗？你觉得你的用户是人渣吗？你为什么这么恨你的用户？当服务挂掉时你怎么可能知道？你配监控了吗？）。近乎抓狂的工程师只能妥协去配置监控。他提议使用听说口碑不错的内部工具 Diplomat，结果对方问他 Borg 任务的命名是否符合 Diplomat 的严格规范；一听到命名不对，工程师彻底放弃，转而退回传统的 Borgmon。 然而在 Google 内部，提交 Borgmon 监控配置不仅需要写代码，还需要“Readability”（代码可读性认证/代码规范审批权限）。对方幸灾乐祸地表示：“No one has borgmon readability. Just me, but I don't have time to review your code.”（没人有 Borgmon readability。除了我，但我可没空审查你的代码）。为了换取代码审查（Code Review），工程师不得不动用内部员工激励手段： - 工程师起初提议给对方 1 个 Peer Bonus（同事互评奖励奖金）。 - 对方极其势利地坐地起价：“One peer bonus? I don't get out of bed for less than three.”（1 个 Peer Bonus？少于 3 个我都懒得下床）。 - 工程师斥责这种索贿行为“一点都不符合谷歌精神（That's not very googly）”，但为了推进工作只能照付 3 个 Peer Bonus。 - 对方扫了一眼表示“Looks good to me”（我看行），却拒绝直接批准，声称：“Approval costs three more peer bonuses.”（想要通过代码审批还得再交 3 个 Peer Bonus）。 被层层官僚主义逼入绝境的工程师彻底爆发了杀意：“If you weren't a staff software engineer I would cut you open like a wet bag of groceries and play hopscotch with your entrails.”（如果你不是个主任软件工程师 Staff Software Engineer，我早就把你像切湿杂货袋一样活活剖开，拿你的内脏跳房子了）。然而在被迫又支付了 3 个 Peer Bonus、好不容易拿到 CL 审批之后，对方在最后一秒悠悠地丢出一记绝命追问：“对了，你记得为你打算运行 Borgmon 的那个 cell 购买配额（Quota）了吗？” 这句话成了压垮骆驼的最后一根稻草。工程师心如死灰地做出了职业规划的终极决定：“I am going to quit my job. My next job will be making your vital organs universally accessible and useful.”（我要辞职。我的下一份工作，就是把你的重要内脏器官变得整合全球信息、人人皆可访问并从中受益——借用 Google 著名的使命宣言发起致命威胁）。 ## Mentioned Google, App Engine, Bigtable, Python, PCR (Planned Change Request), GFE (Google Front End), DNS, Yahoo, Diplomat, Borg, Borgmon, Peer Bonus, Jon Orwant, Benjamin Staffin ## Video description This video dates back to about 2010, and is the origin of the phrase "I've forgotten how to count that low" that was recently referenced in a blog post and on hacker news. (https://acesounderglass.com/2021/10/20/i-dont-know-how-to-count-that-low/) Even more related discussion: https://news.ycombinator.com/item?id=29082014 (2021-11-02) https://rachelbythebay.com/w/2012/04/06/5tb/ https://rachelbythebay.com/w/2021/10/30/5tb/ Original title: "Broccoli Man (Production Issues)" Original author: Jon Orwant (https://www.youtube.com/channel/UC0kuyh1g0DOQiaoxb4zIEhg) [Source](https://www.youtube.com/watch?v=3t6L-FlfeaI)

Original transcript

[0:00] I would like to serve five terabytes of data. [0:02] No problem, we are Google. Five terabytes is so  little that I've forgotten how to count that low. [0:09] Great! I'll use appengine, that will be easy. [0:12] Very easy. Just register an appengine account,  write a little Python, write unit tests, [0:19] write a regression test, write an end-to-end  test, create a CL, pass the submit queue, [0:25] get the CL approved, wait for the  build to go green, and submit it. [0:29] Great, that will only take a week. [0:32] By the way my, data is generated by an  internal cron job- will that be a problem? [0:37] This is Google. You'll need to  store the data in a bigtable. [0:40] That's kind of a pain but I suppose it's  worthwhile for a production quality service. [0:45] You call that production quality? I  made a more robust production quality [0:49] service squatting over the toilet this morning. [0:52] Why is appengine and bigtable not production quality? [0:57] Oh sweet lord in heaven. What happens  when a PCR hits your serving cell? [1:02] A PCR? [1:03] Planed change request. [1:05] This is Google. You need to replicate  your data across multiple bigtable cells. [1:10] How do I do that? [1:11] Just purchase quota in two bigtable cells. [1:14] Done. Now is my service production quality? [1:17] You are so funny. Did you make sure that  both cells are in different PCR zones? [1:23] I didn't realize I had to worry  about that. Why didn't you tell me? [1:27] Look, I just want to serve 5  terabytes. This shouldn't be hard. [1:31] It's not hard. This is Google. By the way, make  sure you file a ticket to replicate your cells. [1:38] Done. Now is my service production quality? [1:41] If you're Yahoo, sure. This is Google. You need  to fail-over cleanly between PCR zones. [1:49] Just launch into both cells and create a corp GFE  DNS entry to point between the two instances. [1:55] I am beginning to hate you. But fine, I will  do that. Now is my service production quality? [2:02] Only if you think your users are scum. Do you?  Do you think your users are scum? Why do you [2:08] hate your users? How will you know when your  service goes down? Have you set up monitoring? [2:14] I just want to serve five terabytes.  Why is this so difficult? Fine, [2:20] I will write something in Diplomat;  I've heard good things about Diplomat. [2:23] Did you give your borg jobs the  appropriate names for diplomat? [2:27] Sweet jesus, you have got to be kidding me.  Never mind then, I'll just use borgmon. [2:35] Good. Just modify someone else's  borgmon code and you're good to go. [2:40] You just need to find someone  with borgmon readability. [2:43] Who has borgmon readability? [2:45] No one has borgmon readability. Just me,  but I don't have time to review your code. [2:51] I will give you a peer bonus if you review my code. [2:53] One peer bonus? I don't get out of bed for less than three. [2:58] That's not very googly, but fine. Three peer bonuses. Now review my 20 lines of borgmon code. [3:05] Looks good to me. [3:06] I need approval. [3:07] Approval costs three more peer bonuses. [3:09] If you weren't a staff software engineer I would [3:11] cut you open like a wet bag of groceries  and play hopscotch with your entrails. [3:16] Fine, now approve my CL. [3:18] Done. By the way, did you remember to purchase  quota for the cell where borgmon will run? [3:24] I am going to quit my job. My  next job will be making your [3:27] vital organs universally accessible and useful.