Sun Engineer Shows Shouting at Storage Disks Spikes Latency Past 500ms

Bryan Cantrill

Acoustic vibrations from shouting disrupt hard drive mechanics, triggering write delays that push storage latency above 500 milliseconds.

AI-assisted summary of Shouting in the Datacenter — original source. Check the original for full context.

Source published: 2008-12-31

Key ideas

Brendan Gregg demonstrates inside Sun Microsystems' Fishworks Lab that shouting directly into JBOD storage arrays induces mechanical vibration that dramatically spikes disk write latency. (0:00)
Using DTrace-powered Fishworks analytics, the team isolates and displays per-disk I/O operations exceeding 500 milliseconds immediately following acoustic disturbances. (1:09)
The experiment highlights physical acoustic interference and vibration as tangible operational hazards capable of disrupting sensitive disk head mechanics during active workloads. (1:09)

Detailed notes

## Background Published on YouTube by Bryan Cantrill on December 31, 2008, this video showcases Brendan Gregg from Sun Microsystems' Fishworks team exploring an unexpected phenomenon in storage systems. According to the video description, Gregg demonstrates how acoustic vibration can induce disk latency, accompanied by a link to a circa-2020 retrospective on the experiment.

## TL;DR Inside Sun's Fishworks Lab, Brendan Gregg runs an active write workload across two JBOD storage arrays while monitoring real-time I/O latency with DTrace analytics. After issuing a deadpan warning not to try the experiment at home, he bends down and shouts directly into the drive enclosures. The live monitoring graphs immediately display severe write latency spikes exceeding 500 milliseconds, demonstrating that acoustic vibrations physically disrupt mechanical disk heads and highlighting the power of granular storage observability.

## Atmosphere & style The video carries an authentic, spontaneous 'mad science' energy, underscored by the loud, relentless ambient roar of datacenter cooling equipment. Brendan Gregg presents the experiment with dry, deadpan humor, creating an amusing contrast between his technical rigor and the absurd act of screaming into a server rack. The handheld camera perspective and casual banter between Gregg and the operator make the viewer feel like an accomplice witnessing a startling, unscripted hardware revelation.

## Bulletin - [00:00](https://www.youtube.com/watch?v=tDacjrSCeq4&t=0s) Brendan Gregg demonstrates inside Sun Microsystems' Fishworks Lab that shouting directly into JBOD storage arrays induces mechanical vibration that dramatically spikes disk write latency. - [01:09](https://www.youtube.com/watch?v=tDacjrSCeq4&t=69s) Using DTrace-powered Fishworks analytics, the team isolates and displays per-disk I/O operations exceeding 500 milliseconds immediately following acoustic disturbances. - [01:09](https://www.youtube.com/watch?v=tDacjrSCeq4&t=69s) The experiment highlights physical acoustic interference and vibration as tangible operational hazards capable of disrupting sensitive disk head mechanics during active workloads.

## Key takeaways - [00:00](https://www.youtube.com/watch?v=tDacjrSCeq4&t=0s) The experiment takes place in Sun Microsystems' Fishworks Lab under normal datacenter ambient noise conditions. - [00:00](https://www.youtube.com/watch?v=tDacjrSCeq4&t=0s) A continuous write workload is executed across two active JBOD ('Just a Bunch of Disks') storage arrays. - [00:00](https://www.youtube.com/watch?v=tDacjrSCeq4&t=0s) Fishworks telemetry utilizes DTrace to track disk I/O latency distributions in real time. - [00:00](https://www.youtube.com/watch?v=tDacjrSCeq4&t=0s) Monitoring is specifically filtered to isolate I/O transactions exceeding 520 milliseconds segmented by individual drives. - [00:00](https://www.youtube.com/watch?v=tDacjrSCeq4&t=0s) Brendan issues a humorous disclaimer before shouting directly into the open bays of the active storage arrays. - [01:09](https://www.youtube.com/watch?v=tDacjrSCeq4&t=69s) Real-time telemetry displays immediate, massive spikes in write latency following the acoustic outburst. - [01:09](https://www.youtube.com/watch?v=tDacjrSCeq4&t=69s) Brendan confirms the phenomenon is an authentic physical interaction rather than a visual or software trick. - [01:09](https://www.youtube.com/watch?v=tDacjrSCeq4&t=69s) The latency spikes correlate specifically to the individual JBOD enclosures that were shouted at. - [01:09](https://www.youtube.com/watch?v=tDacjrSCeq4&t=69s) Mechanical vibration from loud sound waves physically perturbs disk heads, causing latency to exceed 500 milliseconds. - [01:09](https://www.youtube.com/watch?v=tDacjrSCeq4&t=69s) Advanced real-time analytics provided by Sun Fishworks reveal granular latency patterns that were previously unobservable. - [01:09](https://www.youtube.com/watch?v=tDacjrSCeq4&t=69s) The video concludes with the lighthearted operational rule: 'Don't shout at your JBODs. They don't like it.'

## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=tDacjrSCeq4&t=0s) Setting Up the Experiment in the Fishworks Lab

Standing amid the loud ambient roar of cooling equipment and servers inside Sun Microsystems' Fishworks Lab, Brendan Gregg introduces an unexpected finding he and the team have just uncovered. To explain the baseline setup before demonstrating the anomaly, Brendan points to a live monitoring screen where he is tracking disk I/O operations broken down by latency using DTrace to perform real-time storage performance analysis. He has configured the tooling to drill down specifically into disk I/O operations taking at least 520 milliseconds, segmented by individual disk drives.

```mermaid flowchart TD accTitle: Acoustic Vibration Experiment Flow accDescr: Diagram illustrating how acoustic shouting induces disk vibration and increases I/O latency observed in DTrace analytics.

subgraph Workload ["Active Workload"] JBOD["Two JBOD Storage Arrays<br/>(Handling active write operations)"] end

subgraph Disturbance ["Acoustic Disturbance"] Shout["Brendan Gregg Shouting Directly at JBOD Enclosures"] end

subgraph Telemetry ["Fishworks Analytics (DTrace)"] Sensors["Disk Vibration Induced"] Latency["I/O Latency Spikes Exceeding 500ms"] end

JBOD --> Sensors Shout --> Sensors Sensors --> Latency ```

Brendan explains that he is actively applying a continuous write workload across two JBOD ("Just a Bunch of Disks") storage chassis located on the rack next to him, pointing out that the activity lights across the arrays are flashing because they are processing transactions. Before executing the demonstration, Brendan issues a humorous deadpan safety disclaimer to the camera: “What I'm going to do is not recommended. This is not supported. Do not try this at home.” He then bends down right in front of the active drives and lets out several loud, forceful shouts directly into the drive bays.

- Location: Fishworks Lab, featuring heavy background noise from air conditioning units and running server hardware. - Workload: Concurrent write operations targeted across two JBOD storage arrays with visible activity lights. - Telemetry: Real-time DTrace instrumentation monitoring I/O latency distributions and isolating operations exceeding 520 milliseconds by drive.

[01:09](https://www.youtube.com/watch?v=tDacjrSCeq4&t=69s) Acoustic Vibration and Disk Latency Spikes

Immediately following the screaming, Brendan and the camera operator turn back to the DTrace analytics dashboard to examine the real-time telemetry graphs, revealing sudden, massive spikes in write latency. Addressing the viewer directly, Brendan emphasizes that this is an authentic physical phenomenon rather than a camera trick: “This is not special effects. This is real.” He notes that mechanical vibration is a severe operational hazard for storage systems, showing that the latency for the disks handling the active write stream immediately escalated. By inspecting the filtered view of transactions taking longer than 500 milliseconds, Brendan observes that the impacted disks correlate directly to the JBOD enclosures he was shouting at.

Brendan credits the Sun Fishworks analytics instrumentation for revealing disk latency patterns at a granular level that previously could not be diagnosed or observed. The demonstration concludes with banter between the camera operator and Brendan about protecting datacenter hardware from acoustic interference. The camera operator warns viewers, “Just don't let Brendan shout at your disks,” prompting Brendan to agree and close out the demonstration with practical operational advice: “That's right. Don't shout at your JBODs. They don't like it.”

- Key finding: High acoustic vibrations directly disrupt disk head mechanics, causing write retries and pushing I/O latency above 500 milliseconds. - System observability: Fishworks analytics provide granular visibility into previously invisible per-disk vibration and latency anomalies.

## Notable quotes > [01:09](https://www.youtube.com/watch?v=tDacjrSCeq4&t=69s) Don't shout at your JBODs. They don't like it. — Brendan Gregg

## Mentioned Brendan Gregg, Bryan Cantrill, Sun Microsystems, Fishworks, DTrace, JBOD

## Video description Brendan Gregg from Sun's Fishworks team makes an interesting discovery about inducing disk latency. For a ca. 2020 retrospective on this 2008 video: https://www.youtube.com/watch?v=_IYzD_NR0W4#t=28m47s

[Source](https://www.youtube.com/watch?v=tDacjrSCeq4)

中文简报

Sun工程师展示对着存储硬盘大吼会导致延迟飙升超过500毫秒大吼产生的声学振动会扰乱机械硬盘的内部结构，引发写入延迟并使存储延迟突破500毫秒。在Sun的Fishworks实验室中，Brendan Gregg在两个JBOD存储阵列上运行活跃的写入负载，同时使用DTrace分析工具监控实时I/O延迟。在面无表情地警告切勿在家中尝试该实验后，他俯下身直接对着硬盘机箱大声呼喊。实时监控图表随即显示出超过500毫秒的严重写入延迟尖峰，这不仅表明声学振动会对机械磁头产生物理层面的扰乱，也突显了细粒度存储可观测性的强大能力。 Brendan Gregg在Sun Microsystems的Fishworks实验室中演示，直接对着JBOD存储阵列大吼会引发机械振动，从而导致硬盘写入延迟急剧飙升。 (0:00) 利用基于DTrace的Fishworks分析工具，团队在受到声学干扰后立即隔离并显示了单盘I/O操作耗时超过500毫秒的情况。 (1:09) 该实验强调了物理声学干扰和振动是切实的运行风险，能够在活跃工作负载期间扰乱敏感的磁头机械结构。 (1:09)

Detailed notes — zh

## Background 该视频由 Bryan Cantrill 于 2008 年 12 月 31 日发布在 YouTube 平台上。视频简介指出，来自 Sun Microsystems Fishworks 团队的 Brendan Gregg 演示了一项关于声波引发磁盘延迟的有趣发现。 ## TL;DR 在 Sun Microsystems 的 Fishworks 实验室中，Brendan Gregg 针对两台正在承受持续写入负载的 JBOD 机械磁盘柜进行了一项即兴物理测试。通过基于 DTrace 技术的 Fishworks Analytics 监控系统，团队重点观测延迟超过 520 毫秒的极端 I/O 请求。当 Brendan 对着磁盘柜正面大声咆哮时，声波引发的空气振动使高速运转的硬盘磁头发生偏离并触发重试机制，导致监控屏幕上对应的磁盘延迟瞬间产生剧烈峰值，直观证明了环境声学振动对机械存储设备性能的直接冲击。 ## Atmosphere & style 整段视频充满极客狂热与实验室恶作剧般的轻松幽默感。伴随着机房风扇震耳欲聋的白噪音，Brendan 顶着嘈杂背景对着镜头严肃警告“切勿模仿”，随后俯身对着机柜发出滑稽而卖力的大吼，前后反差极具喜剧色彩。掌镜的 Bryan Cantrill 与 Brendan 在看到延迟图表瞬间炸裂时的兴奋互动真实自然，让读者仿佛置身于那个充满硬核探索精神与技术突破的工程现场。 ## Bulletin - [00:01](https://www.youtube.com/watch?v=tDacjrSCeq4&t=1s) Sun Microsystems Fishworks 团队演示了声学振动对机械硬盘性能的物理影响，近距离咆哮即可直接诱发 I/O 延迟飙升至 500 毫秒以上。 - [00:01](https://www.youtube.com/watch?v=tDacjrSCeq4&t=1s) 基于 DTrace 的 Fishworks Analytics 工具提供了高精度实时延迟热图，使传统监控难以察觉的微观物理扰动完全可视化。 - [01:09](https://www.youtube.com/watch?v=tDacjrSCeq4&t=69s) 声波冲击造成的延迟剧增严格局限于被吼叫的目标 JBOD 磁盘阵列，清晰验证了声学振动引发磁头寻道重试的局部性机理。 ## Key takeaways - [00:01](https://www.youtube.com/watch?v=tDacjrSCeq4&t=1s) 实验在 Sun Microsystems 的 Fishworks 实验室进行，该团队负责构建软硬件一体化的高性能存储系统。 - [00:01](https://www.youtube.com/watch?v=tDacjrSCeq4&t=1s) 演示中采用基于 DTrace 技术的 Fishworks Analytics 分析工具，能实时将磁盘 I/O 延迟按热图和物理磁盘进行多维下钻细分。 - [00:01](https://www.youtube.com/watch?v=tDacjrSCeq4&t=1s) 为捕捉严重异常，监控视图专门过滤并聚焦于响应耗时超过 520 毫秒（0.52 秒）的极端 I/O 操作。 - [00:01](https://www.youtube.com/watch?v=tDacjrSCeq4&t=1s) 测试对象为两台处于高负载持续写入状态下的 JBOD 机械硬盘阵列机柜。 - [00:01](https://www.youtube.com/watch?v=tDacjrSCeq4&t=1s) Brendan Gregg 弯身对着运转中的 JBOD 磁盘正面大声咆哮，以此制造高分贝空气声学振动。 - [01:09](https://www.youtube.com/watch?v=tDacjrSCeq4&t=69s) 掌镜的 Bryan Cantrill 明确澄清该现象并非特效，而是物理现实在监控系统中的真实呈现。 - [01:09](https://www.youtube.com/watch?v=tDacjrSCeq4&t=69s) 受声波直接震动冲击后，目标磁盘的延迟图表瞬间被超过 500 毫秒的请求大面积填满。 - [01:09](https://www.youtube.com/watch?v=tDacjrSCeq4&t=69s) 延迟尖峰精确指向被吼叫的两台 JBOD 磁盘柜，而机房内其他存储单元性能保持平稳，证实了声学影响的精确空间局部性。 - [01:09](https://www.youtube.com/watch?v=tDacjrSCeq4&t=69s) 机械硬盘磁头悬浮间隙极小，声学振动导致磁头偏离预定磁道并触发校验与旋转等待重试，使 I/O 响应时间拉长至平常的数百倍。 - [01:09](https://www.youtube.com/watch?v=tDacjrSCeq4&t=69s) 实验凸显了细粒度可观测性系统的价值，过去粗颗粒度的平均值采样工具极易掩盖此类瞬态物理性能骤降。 ## Detailed walkthrough ### [00:01](https://www.youtube.com/watch?v=tDacjrSCeq4&t=1s) Fishworks 实验室的实验设置与警告在嘈杂的 Sun Microsystems Fishworks 实验室中，服务器风扇与空调系统的轰鸣声震耳欲聋，Brendan Gregg 不得不对着镜头大声喊话，并为自己的喊叫声向观众致歉。他兴奋地表示团队刚刚有了一项非常有趣的发现，想要第一时间展示出来。监控屏幕上正通过基于 DTrace 技术的 Fishworks 分析工具（Analytics）实时观测磁盘 I/O 指标，界面上展示了按延迟细分的磁盘 I/O 操作分布图。为了捕捉极端异常，Brendan 专门进行了下钻过滤，只展示耗时在 520 毫秒以上的磁盘 I/O 操作，并按具体的物理磁盘进行分组细分。此时系统正对放置在旁边的两台 JBOD 磁盘柜（Just a Bunch of Disks）施加持续的写入负载，机柜前部指示灯规律闪烁，表明磁盘正在高负荷运转。在正式演示动作之前，Brendan 带着实验人员特有的幽默与严肃向观众郑重警告，Brendan Gregg 直言：“What I'm going to do is not recommended.”随后他又补充道：“This is not supported. Do not try this at home.”说完，他快步走向机柜，弯下身子对着两台正在运转的 JBOD 托盘正面发出极其刺耳的持续大声咆哮。 - 实验环境位于 Sun Microsystems 的 Fishworks 实验室，背景噪音由机房空调与高密度服务器产生。 - 观测手段采用基于 DTrace 的 Fishworks Analytics，可实时将 I/O 延迟进行热图分布与磁盘级聚合。 - 重点监控的延迟过滤阈值设定为 520 毫秒（0.52 秒）以上的磁盘 I/O 请求。 - 负载形式为持续写入负载，施加于两台装满机械硬盘的 JBOD 磁盘阵列柜。 ```mermaid flowchart TD accTitle: 呐喊引发磁盘延迟的因果链 accDescr: 展示从对着JBOD咆哮到DTrace分析图表显现延迟尖峰的完整机理过程 subgraph SG1["物理声波扰动"] A["对着 JBOD 磁盘柜大声咆哮"] --> B["产生高分贝声学空气振动"] B --> C["振动传递至机械硬盘外壳与悬臂"] end subgraph SG2["硬盘硬件反应"] C --> D["磁头偏离磁道 / 寻道校验失败"] D --> E["硬盘触发重试机制等待旋转恢复"] end subgraph SG3["软件与观测呈现"] E --> F["I/O 响应时间飙升（> 500 ms）"] F --> G["DTrace / Fishworks Analytics 出现明显延迟峰值"] end ``` ### [01:09](https://www.youtube.com/watch?v=tDacjrSCeq4&t=69s) 呐喊引发的振动与磁盘延迟分析在 Brendan Gregg 结束咆哮后，镜头切回监控屏幕，掌镜的 Bryan Cantrill 忍不住赞叹并立即向观众澄清道：“This is not special effects. This is real.”屏幕上原本平静、极少出现极端超时的监控图表瞬间发生了剧烈变化：在 Brendan 呐喊的那一刻，受负载影响的磁盘延迟曲线陡然飙升，屏幕下方代表延迟超过 500 毫秒的统计柱状图与明细列表瞬间被大面积填满。更具说服力的是，通过系统细分定位，这些发生严重延迟的磁盘精确指向刚才 Brendan 对着大喊的那两台 JBOD，而机房内其他未被直接吼叫的存储单元则未受波及。这一现象揭示了振动对精密机械硬盘性能的致命影响。高速旋转的硬盘磁头悬浮在微米级的高度上，人体近距离大喊所产生的声波与空气压力足以引发物理振动，导致磁头偏离预定磁道并触发内部重试机制，从而导致 I/O 延迟瞬间拉长到平常的数百倍。Brendan 指出，这种现象的存在必须依靠 Fishworks Analytics 这类革命性的可观测性工具才能被直观捕获，过去在传统性能监控中由于采样粒度粗糙，工程师几乎不可能直观“看到”振动造成的微观延迟冲击。视频结尾，Bryan Cantrill 调侃道：“Just don't let Brendan shout at your disks.”对此 Brendan Gregg 笑着表示认同：“That's right. Don't shout at your JBODs. They don't like it.” - 振动对机械硬盘的影响不仅是理论问题，通过人体发声产生的声学振动即可直接诱发磁头定位异常与性能骤降。 - Fishworks Analytics 让底层 I/O 延迟以微秒/毫秒级粒度可视，揭示了以往被忽视的环境物理扰动。 - 只有被近距离吼叫的 JBOD 内部磁盘出现了大于 500 毫秒的请求积压，精准验证了声波震动的局部指向性。 ## Notable quotes > [01:09](https://www.youtube.com/watch?v=tDacjrSCeq4&t=69s) Don't shout at your JBODs. They don't like it. — Brendan Gregg ## Mentioned Brendan Gregg, Bryan Cantrill, Sun Microsystems, Fishworks, Fishworks Analytics, DTrace, JBOD ## Video description Brendan Gregg from Sun's Fishworks team makes an interesting discovery about inducing disk latency. For a ca. 2020 retrospective on this 2008 video: https://www.youtube.com/watch?v=_IYzD_NR0W4#t=28m47s [Source](https://www.youtube.com/watch?v=tDacjrSCeq4)

Source quotes

[1:09] Brendan Gregg: Don't shout at your JBODs. They don't like it.

Source quotes

[1:09] Brendan Gregg: Don't shout at your JBODs. They don't like it.

Original transcript

[0:01] Hi, I'm Brendan. We're here in the [0:02] Fishworks Lab. Sorry for screaming. It's [0:05] very loud in here with the air [0:06] conditioners and service all around me. [0:08] We just made an interesting discovery [0:11] and we thought we'd show you straight [0:12] away. So over [0:15] here, here I'm measuring dis IO [0:18] operations broken down by latency. Um [0:21] I've also drilled down to disk IO [0:23] operations taking at least 520 [0:26] milliseconds broken down by disk. This [0:29] is using Drace so I can do performance [0:31] analysis of discs. I'm applying a right [0:33] workload to two Jbods which are over [0:39] here. The lights are blinking on these [0:41] Jbs because they're doing work. What I'm [0:44] going to do is not recommended. This is [0:47] not supported. Do not try this at home. [1:02] [Applause] [1:09] This is not special effects. This is [1:11] real. What we're looking at here is the [1:13] effect of disc vibration. Vibration is a [1:17] serious issue and we can see the effect [1:19] here that it's caused. So here the uh [1:22] latency for the discs that are under the [1:24] workload has gone up. Here I can see [1:26] which discs have had now had IO [1:29] operations longer than 500 [1:32] milliseconds and I can find out that [1:34] they belong to the Jbods I was just [1:36] shouting at. So amazing stuff. This has [1:40] been made possible by analytics from [1:42] Fishworks which lets us look at things [1:44] we've never been never seen before. disc [1:47] latency. Uh, high latency caused by [1:50] vibration is a real issue. You've seen [1:52] it. It does exist. Just don't let [1:54] Brendon shout at your desks. That's [1:56] right. Don't shout at your Jbugs. They [1:58] don't like it.