Kroah-Hartman Warns Chipmakers to Upstream Early or Lose to Intel

Linaro On Air

Kernel maintainers will aggressively reject substandard corporate code to protect system stability, leaving hardware vendors at risk of multi-month product delays unless they submit drivers during chip simulation.

AI-assisted summary of Keynote: Greg Kroah-Hartman (Linux Foundation) - "I Don't Want Your Code" — original source. Check the original for full context.

Source published: 2013-03-07

Key ideas

Linux kernel development velocity has accelerated to nearly seven changes accepted every hour around the clock, drawing contributions from over 3,000 developers across 420 companies. (0:01)
Hardware manufacturers are urged to adopt an upstream-first development model during chip simulation to avoid 12-to-18-month feedback delays and ensure day-one hardware support. (6:58)
Maintainers act strictly as editors tasked with rejecting substandard code, holding paid corporate developers to uncompromising standards to prevent long-term technical debt. (11:12)
Corporate email servers appending automated legal confidentiality footers prevent maintainers from reviewing or accepting patches due to open-source licensing and legal constraints. (12:45)

Detailed notes

## Background Published on YouTube by Linaro On Air on March 7, 2013, this video captures a keynote by Linux Foundation fellow Greg Kroah-Hartman titled "I Don't Want Your Code." The original upload does not provide any additional show notes or description.

## TL;DR Greg Kroah-Hartman outlines the immense scale and velocity of Linux kernel development to illustrate why subsystem maintainers must enforce uncompromising standards on incoming submissions. Drawing on an audit of inbox blunders as well as real-world patch disasters from Linaro and Intel, he warns against common submission failures like broken builds, monolithic diffs, and corporate legal footers. He champions an "upstream-first" business model to prevent costly out-of-tree technical debt and details the mutual social contract between contributors and maintainers. In closing, he emphasizes that public accountability and rigorous skepticism are essential to protect the long-term stability and health of the operating system.

## Atmosphere & style The presentation carries a frank, humorous, and authoritative tone, blending sharp developer comedy with an uncompromising "tough love" lecture. Kroah-Hartman's delivery is animated and self-deprecating, using dry wit to demystify why maintainers seem perpetually grumpy while holding the room's attention with relatable war stories. The audience frequently chuckles at candid anecdotes and accidental slip-ups, yet the atmosphere remains grounded in the serious professional stakes of maintaining mission-critical infrastructure.

## Bulletin - [00:01](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=1s) Linux kernel development velocity has accelerated to nearly seven changes accepted every hour around the clock, drawing contributions from over 3,000 developers across 420 companies. - [06:58](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=418s) Hardware manufacturers are urged to adopt an upstream-first development model during chip simulation to avoid 12-to-18-month feedback delays and ensure day-one hardware support. - [11:12](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=672s) Maintainers act strictly as editors tasked with rejecting substandard code, holding paid corporate developers to uncompromising standards to prevent long-term technical debt. - [12:45](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=765s) Corporate email servers appending automated legal confidentiality footers prevent maintainers from reviewing or accepting patches due to open-source licensing and legal constraints.

## Key takeaways - [00:01](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=1s) The Linux kernel represents the largest software development effort in human history, accepting nearly seven patches per hour, 24 hours a day, 365 days a year. - [02:05](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=125s) A structured development hierarchy of roughly 700 file maintainers and 86 subsystem maintainers feeds daily into the linux-next branch for automated integration testing. - [04:10](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=250s) Approximately 80% of all kernel contributions are funded by corporate employers, while only 20% come from independent, unpaid developers. - [06:58](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=418s) Submitting hardware support patches only after physical OS bring-up creates a year-and-a-half lag, stranding customers on outdated, broken downstream vendor kernels. - [06:58](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=418s) Upstream-first development during the simulation phase allows companies like Intel to ship silicon with fully functional, mainline kernel support on day one. - [10:36](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=636s) Upstream community collaboration is a proven commercial necessity that reduces technical debt and speeds delivery, rather than an act of engineering charity. - [11:12](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=672s) Maintainers serve as gatekeeping book editors rather than adversaries, protecting the operating system by rejecting flawed ideas and demanding clear code structure. - [11:12](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=672s) Maintainers offer patient mentoring to unpaid hobbyists but enforce zero tolerance for careless workflow mistakes made by paid corporate engineers. - [12:45](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=765s) Automated legal confidentiality disclaimers appended by enterprise mail systems legally forbid maintainers from opening, answering, or accepting patch emails. - [15:09](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=909s) Adherence to canonical kernel coding style is mandatory because human review depends on visual pattern recognition to efficiently spot defects and bugs. - [15:09](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=909s) Compiling kernel trees as the root superuser is an active operational hazard that has previously masked build bugs that silently wiped developer filesystems. - [18:07](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=1087s) Every commit in a patch series must compile cleanly and remain independently bisectable to prevent future regression tracking with git bisect from breaking. - [18:07](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=1087s) Maintainers cannot merge incomplete work based on promises of future cleanups because contributors routinely vanish or change jobs, leaving behind unfixable technical debt. - [20:44](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=1244s) Patch submissions must explicitly articulate why a change is necessary, rather than merely summarizing what lines of code were modified. - [26:24](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=1584s) A catastrophic Linaro USB submission and an Intel TTY patch containing deliberate crash code illustrate why maintainers require senior co-signers after severe quality breaches. - [28:51](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=1731s) The maintainer social contract requires responding to patches within two weeks and keeping submitters clearly informed of whether their code is accepted or rejected. - [31:16](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=1876s) Public development enforces personal and corporate accountability, preventing engineers from hiding fragile hacks behind internal company deadlines. - [32:19](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=1939s) Patch authors bear full responsibility for build-testing across architectures, as maintainers will immediately reject patches that break builds on x86, ARM, or PowerPC.

## Detailed walkthrough ### [00:01](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=1s) Introduction and the Mind-Boggling Scale of Linux Development

Greg Kroah-Hartman introduces himself and outlines his extensive responsibilities within the Linux kernel ecosystem. While Linus Torvalds manages the main development kernels on a three-month cycle, Greg maintains the stable branches, releasing updates roughly once a week. He also maintains the driver core, USB, debugfs, tty, and serial subsystems—noting with a laugh that serial has been his responsibility for 25 years, starting from his very first paid job. Greg jokes that his ultimate goal for the keynote is to make the audience understand why kernel maintainers are so notoriously obnoxious and why no one in their right mind would ever want his job.

To set the stage, Greg presents the staggering development statistics from the previous year, spanning kernel releases 3.4 to 3.8. Over this period, more than 3,000 individual developers representing approximately 420 different companies contributed to the codebase. Greg describes this collective effort as the largest software development process in human history. The sheer velocity of the project is accelerating at an alarming rate; last year, the kernel averaged nearly seven changes accepted every single hour, 24 hours a day, 365 days a year. This is a significant jump from 5.9 changes per hour the year before, and 5.0 the year before that. Greg warns that trying to keep code out-of-tree with the intent of catching up later is a losing battle because the upstream community is moving too fast.

“My goal at the end of this talk is for you to understand why we are as obnoxious as we are and why you would never want to do this job.”

- **Development Velocity:** 3,000 individual developers and ~420 companies contributing. - **Pace of Change:** ~7 changes accepted per hour, 24/7, up from 5.9 and 5.0 in previous years. - **Kernel Versions Covered:** 3.4 to 3.8 over the course of the analyzed year.

[02:05](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=125s) The Kernel Development Pipeline and Key Contributors

Greg explains the highly structured, tiered development model that makes this massive scale possible. The process begins with the 3,000 developers who write patches and submit them to file maintainers, of whom there are about 700 listed in the `MAINTAINERS` file. These maintainers review, sign off on, and accept the patches, routing them up to one of the 86 subsystem maintainers who manage roughly 160 different development trees. Every single business day, Stephen Rothwell merges all of these active trees into the `linux-next` branch for integration testing. When Linus Torvalds opens the two-week merge window, the subsystem maintainers push their accumulated changes to him, while Andrew Morton acts as a safety net, picking up any remaining patches that slipped through the cracks.

To illustrate where the heaviest development activity occurs, Greg highlights some of the top individual contributors by patch volume. He points out H. Hartley Sweeten, who does massive work on the Comedi data acquisition subsystem, and Mark Brown, who dominates embedded sound (ALSA). He praises Al Viro for his relentless work on file systems, and Alexey Dobriyan for his extensive janitorial cleanups. Greg also highlights Daniel Vetter (Intel Graphics) and Johannes Berg (Intel Wireless) to show how much churn occurs within single driver subsystems. He gives a special nod to Sachin Kamat from Linaro, who has been doing excellent cleanup work all over the kernel tree.

“If you want to keep code out of the kernel and think, 'Ah, I will just catch up to them later,' well, they're going faster, and you're going to have to do more work to try and catch up.”

- **Pipeline Infrastructure:** ~700 file maintainers, 86 subsystem maintainers, and ~160 distinct Git trees. - **Integration Testing:** Daily merges of all trees into `linux-next` by Stephen Rothwell. - **Key Subsystems with High Churn:** Intel Graphics (Daniel Vetter), Wireless (Johannes Berg), Embedded Sound (Mark Brown), and Comedi (H. Hartley Sweeten).

[04:10](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=250s) Maintainer Workloads and Corporate Backing

The workload of a top-tier maintainer is incredibly intense. Greg shares that he and David Miller (the networking maintainer) accept the highest volume of patches in the entire project. Over the past year, David Miller interacted with 470 unique developers, while Greg interacted with 450. Greg jokes that when developers approach him at conferences to thank him for accepting their patches, he has to give a generic "you're welcome" because the sheer volume makes it impossible to remember individual submissions. The staging subsystem, which Greg manages, is a major source of this churn, alongside networking, embedded sound, and video (managed by Mauro Carvalho Chehab).

Greg then addresses who actually pays for all of this development. He congratulates Linaro for cracking the top 10 corporate contributors list for the first time, though he notes they are still pulling themselves out of their "own little gravity well" compared to the top contributors. Interestingly, about 20% of the kernel is written by independent individuals, whom Greg jokingly describes as "people that haven't been hired yet." The remaining 80% is funded by corporations. He highlights H. Hartley Sweeten's employer, Vision Engraving Systems, which cracked the top seven corporate contributors solely because of Hartley's prolific output. Meanwhile, companies like Google are rising in the ranks, IBM is sliding down, and Qualcomm is called out for actively hiding their patches behind legal teams.

“Qualcomm's the only company that hides their patches... they're really good at that, they have good lawyers.”

- **Maintainer Outreach:** David Miller accepted patches from 470 unique people; Greg accepted patches from 450 unique people. - **Corporate Contribution Split:** 80% of kernel development is corporate-sponsored; 20% is done by independent developers. - **Corporate Standings:** Google is moving up, IBM is moving down, and Linaro has entered the top 10. Qualcomm is noted for keeping its patches proprietary.

[06:58](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=418s) The Business Case for Upstream-First Development

Greg transitions to a critical critique of how hardware and system-on-chip (SoC) companies design and submit code. He shares a slide detailing a partner company's traditional chip design pipeline: design, emulation, simulation, tape-out, power-on, and finally OS bring-up. Historically, these companies only begin submitting their kernel patches *after* OS bring-up. Because the upstream review process takes three to eight months, their code is accepted long after the hardware team has already moved on to their next six-month design cycle. This creates a sluggish, year-and-a-half feedback loop, forcing customers to run outdated, broken kernels on newly purchased chips.

To solve this, Greg urges companies to adopt the "upstream-first" model, which Linaro successfully utilized for the ARM64 architecture. Under this model, companies submit kernel patches during the simulation phase, months before physical silicon exists. By the time the chip is manufactured and delivered to customers, the support code is already merged into the mainline kernel at kernel.org. Greg points to Intel as the gold standard of this approach; they are so far ahead that they upstream code for entire architectures early, sometimes even ripping out support for experimental chips that never make it to production. He warns the audience that if they do not adopt this proactive approach, Intel will continue to dominate the market. Greg also reveals a humorous anecdote: an anonymous company recently leaked that they had working ARM64 silicon in their labs by accidentally emailing him to ask for ARM64 patch backports to an ancient 3.4 kernel for an active device.

“Intel is that far ahead of you guys... if you don't catch up, they will win, they will beat you.”

```mermaid flowchart TD accTitle: Traditional vs Ideal Kernel Submission Timelines accDescr: A comparison showing how traditional hardware development delays kernel submission until after OS bring-up, whereas the ideal model submits code during simulation so it is ready at launch.

subgraph Traditional ["Traditional Model (Slow & Late)"] A1["Design & Emulation"] --> B1["Simulation & Tape Out"] B1 --> C1["Power On & OS Bring-up"] C1 -->|"Submit Patches"| D1["Accepted 3-8 Months Later"] D1 -->|"Feedback Loop"| E1["Hardware Team Already on Next Design"] end

subgraph Ideal ["Ideal Model (Upstream-First)"] A2["Design & Emulation"] -->|"Submit Patches"| B2["Simulation Phase"] B2 --> C2["Code Accepted in kernel.org"] C2 --> D2["Silicon Power On / Customer Delivery"] D2 --> E2["Working Out-of-the-Box Support"] end ```

- **The Feedback Loop:** Traditional post-silicon submission results in a 12-to-18-month delay for stable upstream support. - **The Intel Example:** Intel upstreams code so early that they occasionally delete entire chip architectures from the kernel before the hardware is even released. - **The Business Reality:** Upstreaming early is not just a community favor; executives at IBM and Intel have publicly stated it is a core business and financial necessity.

[10:36](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=636s) The Business Imperative of Upstream Engagement

Working with the upstream Linux community is not an act of charity or irrational engineering idealism; it is a calculated business practice. Engaging early and often saves companies money, shortens product delivery times, and wins customers by eliminating the crushing technical debt of carrying out-of-tree patches. Industry giants like IBM and Intel have recognized this dynamic for years and openly share their methodologies with the rest of the industry, yet many organizations still fail to heed their example. Greg Kroah-Hartman cautions that ignoring these hard-won lessons places companies in direct jeopardy, warning that engineering organizations must reform their development habits if they wish to survive in the ecosystem.

> “This is how businesses work: work with the community early, often, and you ignore this at your own peril.” — Greg Kroah-Hartman

[11:12](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=672s) Maintainers as Editors: Why Harsh Feedback Protects Linux

Borrowing an analogy originally framed by networking maintainer David S. Miller, Kroah-Hartman explains that kernel maintainers act not as gatekeeping adversaries, but as book editors working directly with authors. In this dynamic, the contributor writes the raw material while the maintainer scrutinizes grammar, suggests structural revisions, pushes back on weak ideas, and provides guidance. Maintainers often yearn to return to writing original code themselves, occasionally tuning out incoming traffic to work on personal side projects, but their editorial discipline is what preserves the integrity of the operating system. If maintainers were simply agreeable and accepted every submission uncritically, Linux would rapidly deteriorate and fail.

This editorial relationship also dictates how strictly maintainers treat different types of contributors. Kroah-Hartman highlights the contrast between independent hobbyists and paid corporate engineers:

- **Unaffiliated and volunteer developers:** These contributors write code purely out of personal passion and a desire to see Linux improve. Maintainers intentionally treat them with patience, offering gentle mentoring and encouragement. - **Corporate, paid developers:** Engineers who contribute on company time are held to an uncompromising standard. Because it is their professional responsibility to master the workflow, maintainers have little patience for careless mistakes or unread documentation.

> “If I am nice and take whatever you send me, Linux will not succeed.” — Greg Kroah-Hartman

```mermaid flowchart TD accTitle: Upstream Maintainer Editorial Filter accDescr: Maintainers evaluate contributors and either guide hobbyists or strictly reject substandard corporate patches to protect kernel health.

subgraph Authors["Patch Contributors"] Vol["Volunteers & Hobbyists (Motivated by passion)"] Corp["Corporate Developers (Paid professional role)"] end

subgraph Editorial["Maintainer Review Engine"] Filter{"Maintainer Assessment"} Patience["Patient mentoring & gentle guidance"] Strict["Strict enforcement of standards & pushback"] end

subgraph Outcome["Kernel Health"] Accept["High-quality upstream code"] Reject["Immediate rejection of low-effort patches"] end

Vol --> Filter Corp --> Filter Filter -->|"Unpaid author"| Patience Filter -->|"Paid professional"| Strict Patience --> Accept Strict --> Accept Strict --> Reject ```

[12:45](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=765s) The 487-Patch Audit: Trivial and Legal Submission Blunders

To demonstrate the sheer volume of substandard contributions received by upstream, Kroah-Hartman conducted an audit of his personal inbox over a quiet two-week merge window six months prior. Because experienced kernel developers know that new features are not accepted during the merge window, they hold their submissions; consequently, the 487 patches Kroah-Hartman received during those two weeks came almost entirely from developers unfamiliar with basic upstream etiquette. The submissions included baffling mistakes, such as patch sequences where the introductory cover email existed but the numbered patches were missing, or instances where patches had no context lines, forcing maintainers to guess where changes belonged.

A particularly severe problem involves corporate email configurations that automatically attach confidentiality disclaimers. When a company's mail server appends a legal footer warning that an email is privileged or confidential and must be deleted if received in error, maintainers face legal constraints. Legal counsel mandates that patches bearing these footers cannot be accepted, reviewed, or even answered, because kernel development is strictly an open, public process. While Microsoft and Huawei resolved these corporate Microsoft Exchange footer issues by documenting proper email-routing procedures, engineers at other firms must actively push their internal IT departments to stop adding legal footers to public mailing list traffic.

> “If your company is putting a footer on emails that you send out, I cannot respond to you because you just told me I couldn't.” — Greg Kroah-Hartman

- **Malformed Sign-offs:** Submissions arrived with the required `Signed-off-by:` tag buried inside a decorative email signature below the patch boundary, rendering automated git ingestion tools incapable of parsing it. - **Missing patches:** Developers sent out multi-part email series containing only partial sets (such as omitting patch 2 of 2 entirely) or emails claiming to have a patch with no code attached. - **Automated mailing list drops:** Several public kernel mailing lists run automated filters configured to discard any email containing corporate legal confidentiality notices immediately upon arrival.

[15:09](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=909s) Build and Formatting Failures: Root Builds, Style, and Non-Compiling Code

The audit revealed widespread disregard for standard development environments and coding practices. Submissions arrived using obsolete, non-unified diff formats that modern tooling cannot read, with leading indentation whitespace stripped by misconfigured email clients, and against entirely unrelated subsystem repositories—such as SCSI tree patches mistakenly emailed to Kroah-Hartman. One patch sent six months prior had been generated against an ancient kernel tree and originated from a file path proving the submitter compiled the kernel as the `root` superuser. Kroah-Hartman reminded the audience that building as root is hazardous, recalling an incident where an unnoticed kernel build bug silently erased `/` on developer machines for nearly three months without maintainers noticing, precisely because none of them ran builds with root privileges.

Formatting compliance remains a mandatory baseline. Human code review relies heavily on visual pattern recognition; when unfamiliar whitespace or inconsistent styles disrupt the visual rhythm of the code, maintainers cannot effectively parse the logic or spot defects. The kernel project provides automated linting scripts (such as `checkpatch.pl`) to fix styling errors before submission. Submitting code that violates the canonical Linux style implies that the author expects thousands of other developers to adapt to their individual preferences, which maintainers reject out of hand. Worse still are patches that do not compile at all—a failure easily caught if developers simply compiled their code prior to sending it out.

> “If I can't read your code, I'm not going to be able to fix your bugs, and I'm not going to take your code—it's as simple as that.” — Greg Kroah-Hartman

[18:07](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=1087s) Broken Trust, Bisectability, and Why Maintainers Must Be Grumpy

Every patch in a multi-part series must compile cleanly and remain bisectable on its own. Authors occasionally introduce changes that break the build in early patches of a series, only to fix the compilation error in the final patch. Maintainers run automated scripts that apply and compile every single commit individually to detect build failures and compiler warnings at each step; breaking intermediate commits ruins `git bisect` for anyone trying to trace future regressions. Similarly unacceptable are submissions delivered with promises that known defects or incomplete implementations will be "fixed later" in follow-up patches. Once code enters the mainline tree, the subsystem maintainer inherits long-term maintenance of that code, and submitters frequently abandon their work the moment their patch merges.

Kroah-Hartman highlights the hazards of unearned trust by recounting a notorious incident in the networking subsystem, where a developer submitted an extensive core networking overhaul with promises to handle follow-up cleanups. The day after the maintainers merged the patch, the contributor's email address bounced, the submitter vanished entirely from the internet, and the networking team spent an entire year untangling and repairing the fallout. Maintainers must also deal with developers who submit monolithic 4,500-line diffs instead of breaking changes down into incremental steps. Unless a change is a completely self-contained new driver or filesystem, authors must show their work incrementally. Kroah-Hartman urges attendees to make use of Linaro's internal training materials and experienced engineers, reminding them that maintainer skepticism is necessary to prevent the kernel from degrading.

> “You want me to be grumpy, you want me to push back on this stuff, you don't want me accepting crap, because that'll hurt Linux.” — Greg Kroah-Hartman

- **The disappearance risk:** Unfinished code cannot be accepted based on trust, because contributors frequently leave projects, change employers, or pass away, leaving maintainers permanently saddled with the technical debt. - **Incremental commits:** Modifying core kernel subsystems requires small, logically discrete patches that clearly demonstrate the progression of the work, rather than massive single rewrites. - **Unverified documentation:** Multiple submissions included broken or unformatted kernel-doc entries that the authors had never attempted to build or validate before mailing.

[20:44](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=1244s) The Anatomy of a Linaro Patch Trainwreck

```mermaid flowchart TD accTitle: Broken Patch Progression accDescr: Sequence showing how an eight-patch USB series was repeatedly mishandled, accepted late, and ultimately reverted.

subgraph Submission ["Patch Submission Issues"] A["8-patch USB series arrives (no rationale, 1-line notes)"] B["4 of 8 patches break the build (ignored by Greg; reviewed by TI)"] C["Resubmissions arrive out of order (labeled 'v2' on fourth attempt)"] A --> B --> C end

subgraph Merge ["Merge Window Violations"] D["Senior dev vouches for 2 patches (accepted past deadline into -rc1)"] E["Typo fix requires a 3rd patch"] C --> D --> E end

subgraph Fallout ["Hardware Breakage & Revert"] F["TI testing reveals hardware breakage across TI platforms"] G["Greg forced to revert all patches and explain delay to Linus"] H["Driver maintainer returns: all 8 patches functionally broken"] E --> F --> G --> H end ```

Illustrating how dangerous bad contributions can be, Greg Kroah-Hartman recounts a specific failure from January, right as the merge window was closing and Linus Torvalds was preparing to release Linux 3.9. Kroah-Hartman sets a strict rule that his subsystem tree closes one week before Torvalds's release so he can stabilize and test everything. Two weeks prior to that cutoff, an eight-patch USB series arrived from a Linaro email address. The submission was fundamentally flawed from the start: each patch contained only a single-line summary stating what code was changed, with zero explanation of *why* the change was needed. Kroah-Hartman remarks that while he can read code to see what it does, he cannot guess the rationale, so he initially ignored the series. To make matters worse, half of the eight patches completely broke the build, proving they had never even been compiled or tested.

Because the actual driver maintainer was on vacation in the Caribbean with no email access, Kroah-Hartman let the submission sit, but external reviewers—including engineers from Texas Instruments—stepped in. When revised patches were finally sent, the submitter neglected to explain what had changed between versions. Kroah-Hartman explains that he receives 200 to 400 patch emails every week and cannot be expected to remember prior feedback or unstated modifications. Subsequent replacement patches arrived out of order without indicating which original patches they were intended to replace. After a senior Linaro kernel team member intervened on the mailing list to publicly demonstrate the correct workflow, the author sent them out yet again, bizarrely labeling their fourth attempt as "version 2."

Past the merge deadline, Linaro developers insisted that two of the patches were urgent and had to make it into -rc1. Because the changes targeted ARM hardware he did not own, Kroah-Hartman relied on the recommendation of a trusted senior developer he had known for years and accepted the two cleaned-up patches. Immediately, configuration typos were discovered, forcing him to take a third patch past his cutoff. Shortly after, Texas Instruments tested the tree and discovered that the patches completely broke their hardware. Furious, TI reported the regression, forcing Kroah-Hartman to revert the changes and draft a changelog for Linus Torvalds explicitly explaining that his tree was delayed past the deadline because he had to rip out broken Linaro code. When the subsystem maintainer finally returned from vacation, he discovered that every single one of the original eight patches was logically flawed and broken. Kroah-Hartman recounts how this typically calm maintainer was pushed to public frustration:

> “This driver maintainer who is very nice, very calm, been around forever, no one would ever accuse him of being grumpy, said this publicly: ‘Uh, you guys got some work to do.’”

- Submissions must clearly explain *why* a patch is needed, not just *what* the code does. - Maintainers process hundreds of patches weekly; resubmissions must always include a changelog of what changed between versions. - Four of the eight original Linaro USB patches failed to compile, and all eight were ultimately found to be logically broken. - Texas Instruments engineers caught the build breakages and later verified that the patches broke all of TI's hardware. - As a consequence of the failure, Kroah-Hartman announced a strict sanction: “I will not take code from this developer again for about a year until so I forget them. I will now require a senior Linaro kernel developer to sign off on any patches from Linaro that I will accept.”

---

[26:24](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=1584s) The Accidental Intel Slip and Testing Devices

To demonstrate that Linaro is not his sole target, Kroah-Hartman shares an equally egregious blunder committed by another major contributor ranked among the top five companies in kernel development. The submission involved a substantial patch series for the TTY layer that was already on its fifth revision. Kroah-Hartman was initially skeptical because the patch appeared to duplicate large amounts of existing code, but multiple reviewers insisted it was solid and urged him to take it. Upon careful inspection, he discovered that embedded inside the cleanup routine—the code executed whenever a device is detached or removed from the system—was debugging code containing an intentional bug that instantly hard-crashed the machine. The developer had left an explicit comment stating the line was meant to "test the code," yet had evidently never unplugged a device during testing.

Had this change merged, detaching any USB serial dongle, wireless modem, or embedded modem would have brought down the entire operating system. Kroah-Hartman was so outraged that he informed the submitter that their future patches would be rejected unless backed by a senior company engineer. In the middle of telling the story to the Linaro audience, he accidentally blurts out the company's identity:

> “I told this guy after that, I said, ‘I will not accept your code. I will require Intel’s—oh, whoops, sorry. I will require their senior developers to sign off on the code before I will accept it.’”

After the audience laughs at the accidental reveal, Kroah-Hartman emphasizes that even massive, highly sophisticated organizations like Intel make severe mistakes. His point is not to single out Linaro, but to hammer home the universal reality of maintenance: Linux already runs reliably today, meaning any incoming code represents an inherent risk to stability. It is entirely within a maintainer's self-interest to reject patches to protect the tree. Contributors must view it as their duty to submit flawless work that makes accepting it an obvious choice, leaving maintainers with zero technical reasons to say no.

- The faulty TTY patch had gone through five revisions and had been reviewed by multiple engineers before reaching Kroah-Hartman. - The debugging line triggered a hard kernel crash on device removal, threatening to break modems, embedded modems, and USB-to-serial adapters across the kernel. - The incident prompted Kroah-Hartman to enforce the same senior-signoff restriction on that developer as he did with the Linaro USB developer. - “You have to convince me that I am doing something extraordinarily foolish by not accepting your code.”

---

[28:51](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=1731s) The Maintainer's Social Contract

Turning from developer responsibilities to maintainer obligations, Kroah-Hartman stresses that the upstream relationship is not a one-way street. If maintainers demand high-quality, properly tested patches, contributors are entitled to clear, reliable commitments in return. First, maintainers should review and respond to patch submissions within one to two weeks. While exceptions occur during the merge window or when maintainers are traveling, contributors have every right to ping a maintainer for an update after two weeks have elapsed—though he playfully pleads that developers should not email him a mere twelve hours after submission while he is asleep or on an airplane.

Second, Kroah-Hartman promises that maintainer feedback should provide semi-constructive criticism focused strictly on technical merits. He addresses recent mainstream press reports expressing shock that kernel hackers occasionally use colorful language in public discussions. While he admits that longtime friends and peers might occasionally curse at each other or call someone stupid, critique must target the code rather than the author. He urges developers to meet maintainers in person at conferences to demystify them, and advises developers to shed their personal egos:

> “I will respond to your code, but detach yourself from your code. My very first driver I submitted, within two minutes it was cut to shreds, and I lost my ego then 15 years ago.”

Third, maintainers have a duty to keep patch authors informed of their submission status. Kroah-Hartman voices his intense frustration with maintainers who let code vanish into a black hole for three months, only for it to silently appear in a release tree. Using public scripts originally authored by Andrew Morton and adapted for his own build systems, Kroah-Hartman sends automated email notifications whenever patches are accepted, rejected, or merged into subsystem branches. He insists that contributors should demand this level of transparency from all their subsystem maintainers, whether updates are delivered via email, automated trackers, or even Twitter bots, ensuring no engineer is left guessing about the fate of their work.

- Contributors should wait one to two weeks before pinging maintainers about unreviewed patches. - Maintainers must criticize the code, not the person, while contributors must cultivate thick skin and separate their identity from their pull requests. - Andrew Morton's publicly available maintenance scripts automate patch state tracking and sender notifications. - Maintainers owe submitters prompt, explicit notifications regarding whether a patch has been accepted or rejected.

[31:16](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=1876s) Personal Responsibility, Doing Work in Public, and Why Maintainers Are Grumpy

Reflecting on a distressing kernel submission scenario, Greg Kroah-Hartman emphasizes how disheartening it is when an engineer is driven to produce subpar code. He observes that it is deeply sad when a developer feels that the work they are writing is so terrible that they know it should never even be submitted in the first place, pointing out that anyone caught in that position is doing something fundamentally wrong by letting someone else force them into compromising their standards. He stresses that the core strength of Linux comes directly from individual accountability: every patch is assigned to the author's name, personally tying their reputation and their company to that code in plain view of the entire world.

Working entirely in public prevents bad practices and prevents developers from burying sloppy work. Because kernel development does not operate under arbitrary corporate deadlines or have sudden mandates thrown at it, the community can afford to hold a high bar. Greg notes that this is precisely why Linux thrives, but it is also why kernel maintainers come across as irritable when poor code is submitted. As Greg Kroah-Hartman puts it: “And the only reason Linux works so well is because personally you're responsible for the code that you send off you are attached to it it's assigned to you and your company better not as those other patches were shown so do it in public and because of that it makes it us work really really well it's high for us to hide crappy hacks we don't have deadlines we don't have things thrown at us like this it works really really good.” Closing his talk, he jokes about delivering his obligatory lecture on the matter and directs attendees to the Linaro website to download his presentation slides.

- Personal accountability and public scrutiny ensure developers take ownership of their patches rather than quietly merging fragile workarounds. - Linus Torvalds and the broader maintainer community resist arbitrary schedule pressure in favor of code quality. - The keynote slides are publicly accessible via the Linaro website.

[32:19](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=1939s) Q&A: Presenting from ARM and Build Testing Across Architectures

Opening the floor to questions, Greg Kroah-Hartman pauses before making a humorous remark about his presentation hardware, joking that while he is presenting from an ARM-based system, it certainly is not a Raspberry Pi, which he playfully disparages. An audience member then asks about cross-architecture expectations, noting that patch submissions often seem to carry an implicit question of whether the code was tested on x86, and asks whether maintainers are primarily testing on x86 machines.

```mermaid flowchart TD accTitle: Maintainer Patch Testing Flow accDescr: Diagram showing how a maintainer handles patch verification across hardware and cross-compilation architectures.

subgraph Receipt["Submission"] P["Incoming Patch"] end

subgraph Verification["Maintainer Verification"] H{"Hardware on hand?"} T["Test on physical hardware"] B["Build test on x86 / native rig"] C["Cross-compile checks (ARM, PowerPC)"] end

subgraph Outcome["Result"] F["Reject: Broke the build on other archs"] A["Accept if build & logic pass"] end

P --> H H -->|"Yes"| T H -->|"No"| B B --> C C -->|"Compile failure"| F C -->|"Clean build"| A ```

Greg clarifies that maintainers do not and cannot test patches on hardware they do not physically own. However, maintainers will unconditionally run build tests on their primary architecture—typically x86—and any patch that breaks a standard build will be rejected immediately. He recounts a recent patch he worked on with John that compiled cleanly on ARM but broke compilation on x86, as well as another submitted patch that broke builds on PowerPC. He reminds developers that cross-compilers exist and should be integrated into everyone's local build systems to catch architectural breakages before code ever reaches maintainers. Emphasizing the limits of his testing responsibilities, Greg Kroah-Hartman explains: “I can't test your code I don't test your code unless it happens to be for the hardware I have I will build it on 86 a better bill because if it didn't you just broke the bill you can't make the bill for other architect.” With that final warning to test builds thoroughly, Greg closes the session to audience applause.

- Maintainers generally do not possess every specialized development board or chip, making real-hardware runtime verification the patch author's responsibility. - A patch must never break the compilation of another supported architecture, regardless of where the author intended it to run. - Build systems should actively leverage cross-compilers to catch multi-architecture build failures across x86, ARM, and PowerPC prior to patch submission.

## Notable quotes > [26:24](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=1584s) You have to convince me that I am doing something extraordinarily foolish by not accepting your code. — Greg Kroah-Hartman

> [11:12](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=672s) If I am nice and take whatever you send me, Linux will not succeed. — Greg Kroah-Hartman

> [10:36](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=636s) This is how businesses work: work with the community early, often, and you ignore this at your own peril. — Greg Kroah-Hartman

> [15:09](https://www.youtube.com/watch?v=fMeH7wqOwXA&t=909s) If I can't read your code, I'm not going to be able to fix your bugs, and I'm not going to take your code—it's as simple as that. — Greg Kroah-Hartman

## Mentioned Greg Kroah-Hartman, Linus Torvalds, Stephen Rothwell, Andrew Morton, David S. Miller, Mauro Carvalho Chehab, H. Hartley Sweeten, Mark Brown, Al Viro, Alexey Dobriyan, Daniel Vetter, Johannes Berg, Sachin Kamat, Linaro, Intel, Google, IBM, Qualcomm, Texas Instruments, Vision Engraving Systems, Huawei, Microsoft, Linux Foundation, Linux kernel, ARM64, PowerPC, x86, Comedi, ALSA, TTY, USB, debugfs, checkpatch.pl, git bisect, linux-next, Raspberry Pi

[Source](https://www.youtube.com/watch?v=fMeH7wqOwXA)

中文简报

Kroah-Hartman警告芯片厂商：尽早向上游提交代码，否则将落后于Intel 内核维护者将坚决拒绝不合规格的企业代码以保护系统稳定性；硬件厂商若不在芯片仿真阶段提交驱动程序，将面临长达数月的产品延期风险。 Greg Kroah-Hartman阐述了Linux内核开发的庞大规模与极高迭代速度，以此说明为何子系统维护者必须对提交的代码执行毫不妥协的高标准。结合对收件箱常见错误的梳理，以及来自Linaro和Intel的实际补丁问题案例，他警告切勿出现构建中断、庞大单一diff补丁以及企业法律免责声明等常见提交缺陷。他倡导“上游优先”的商业模式，以避免代价高昂的树外技术债务，并详细说明了贡献者与维护者之间的双向社会契约。最后他强调，公开的责任机制和严格的审视对于保障操作系统的长期稳定性与健康生态不可或缺。 Linux内核开发速度已加快至全天候平均每小时合入近7项变更，汇聚了来自420家公司、逾3000名开发者的贡献。 (0:01) 敦促硬件制造商在芯片仿真阶段即采用上游优先的开发模式，以避免12至18个月的反馈延迟，并确保硬件上市首日即可获得支持。 (6:58) 维护者的职责严格类似于编辑，负责拒绝不合规格的代码，并对企业付费开发者坚持毫不妥协的标准，以防止积累长期技术债务。 (11:12) 由于开源许可协议和法律限制，企业邮件服务器自动附加的法律保密声明尾缀会导致维护者无法审查或接受相应补丁。 (12:45)

Reading summary

Linux kernel maintainer Greg Kroah-Hartman argues that the operating system's rapid release cycle and stability depend on maintainers aggressively rejecting substandard code, making early upstream engagement and rigorous testing an economic necessity for hardware vendors. Scale and the Upstream-First Imperative (6:58) The Linux kernel processes changes at an unmatched velocity, averaging nearly seven accepted patches every hour around the clock. Across releases 3.4 through 3.8, more than 3,000 developers from approximately 420 companies contributed to the codebase, with corporate entities funding roughly 80% of development. Kroah-Hartman contends that attempting to maintain out-of-tree patches is financially self-defeating because the upstream code moves too fast to catch up later. Historically, hardware vendors wait until physical silicon is powered on and operating systems are brought up before submitting kernel drivers, creating a 12-to-18-month delay before upstream support stabilizes. Kroah-Hartman urges companies to adopt an upstream-first approach by submitting code during the chip simulation phase—a model Linaro demonstrated with ARM64 and Intel routinely executes. Companies that upstream early eliminate out-of-tree technical debt and deliver functional hardware to customers immediately upon launch. Maintainers as Strict Editorial Gatekeepers (11:12) Adopting an analogy from networking maintainer David Miller, Kroah-Hartman describes maintainers not as gatekeeping adversaries, but as editors whose strict skepticism preserves the integrity of the operating system. While maintainers offer patient mentorship to unpaid hobbyists, they enforce uncompromising standards on paid corporate engineers. Uncritical acceptance of incoming contributions would quickly destabilize the kernel. Maintainer scrutiny targets baseline execution and submission hygiene. An audit of Kroah-Hartman's inbox during a merge window revealed that many incoming patches failed basic standards: multi-part patch sets arrived missing crucial patches, formatting scripts were ignored, and automated corporate email disclaimers legally barred maintainers from reviewing submissions. Furthermore, contributors frequently violated bisectability by submitting code that failed to compile on intermediate commits or relied on unfulfilled promises to clean up defects later. Systemic Breakdowns at Linaro and Intel (20:44) To demonstrate the damage caused by poorly vetted code, Kroah-Hartman detailed an eight-patch USB submission from Linaro. The series arrived without explanatory rationale, half of the commits failed to compile, and subsequent revisions were delivered out of order. Accepted late on the recommendation of a senior developer, the changes broke Texas Instruments hardware across multiple platforms, requiring an emergency revert. When the subsystem maintainer reviewed the series, every patch was found to be functionally broken, leading Kroah-Hartman to mandate senior sign-offs for future submissions from that contributor. Even industry leaders make catastrophic oversights. Kroah-Hartman cited an Intel TTY submission on its fifth revision that had already been reviewed by multiple engineers. Inside the cleanup routine triggered on device removal, the author had left an active debugging command designed to intentionally crash the system. Had the maintainer merged the patch, unplugging any USB modem or serial adapter would have brought down the machine. The Reciprocal Social Contract and Testing (28:51) Kroah-Hartman emphasizes that upstream collaboration is a two-way agreement. Maintainers owe contributors technical critiques directed strictly at the code rather than the person, explicit patch status tracking, and a response within one to two weeks. In turn, contributors must shed personal ego during review and take personal responsibility for what they submit under their name. Authors are also expected to handle testing across platforms. Maintainers build incoming code against their primary workstations—predominantly x86—and reject any patch that breaks a standard build, even if the change targets an ARM system they do not physically own. Because maintainers cannot possess every piece of specialized hardware, contributors must use local cross-compilers to verify that their patches build cleanly across x86, ARM, and PowerPC before submission. Because Linux operates without artificial corporate deadlines, the upstream maintainer community can afford to hold code to a high technical bar, leaving maintainers with a duty to reject work unless authors prove that accepting it improves the kernel.

精炼阅读摘要

Linux 基金会内核维护者 Greg Kroah-Hartman 强调，内核维护者严厉驳回劣质代码并非刻意刁难，而是在极高开发速率下维持操作系统稳定性的必要防线；硬件厂商若不将代码提交前置到仿真阶段并遵循社区工程规范，将陷入沉重的技术债务与竞争劣势。开发规模激增与上游优先的商业逻辑 (0:01) 在 Linux 3.4 至 3.8 版本的单一年份内，内核社区汇集了来自约 420 家企业的 3000 多名独立开发者，平均每小时合并近 7 项补丁（高于往年的 5.9 项与 5.0 项）。整个开发过程中，80% 的代码由企业赞助完成，仅有 20% 来自独立开发者。面对这种高强度的演进速率，试图将驱动与修改维持在树外（out-of-tree）并寄望日后跟进的做法在商业上完全不可行，因为跟进的维护成本会呈指数级上升。针对硬件及片上系统（SoC）设计周期，传统芯片厂商通常在硬件流片、上电及操作系统调通后才向上游提交内核代码。由于社区评审需要 3 至 8 个月，当补丁最终合并时，硬件团队早已进入下一个设计循环，导致长达 12 至 18 个月的反馈滞后，迫使最终客户在新硬件上运行陈旧且缺乏维护的内核。相比之下，Intel 与 IBM 将代码提交前置到硅片制造前的硬件仿真阶段，确保物理芯片交付客户时驱动已预置于 kernel.org 主线内核中。这种“上游优先”（Upstream-First）策略已被产业证明是一项关键的财务与商业生存法则。维护者的编辑角色与严苛审核标准 (11:12) Linux 内核维护者并非单纯的技术把关人，其角色更接近对文字极其严苛的图书编辑。维护者的职责是对架构与代码进行推敲、修正和删减，若一味妥协接纳劣质补丁，Linux 系统的架构将迅速劣化。在这种机制下，维护者会耐心地指导出于个人热情参与的无偿业余爱好者，但对企业领薪雇佣的专业工程师则执行毫不妥协的严厉标准，要求其必须熟读规范并对工作流程全权负责。 Kroah-Hartman 对合并窗口期收到的 487 份补丁进行了审计，暴露出大量低级失误。部分企业邮箱系统自动附加法律机密声明页脚，导致维护者因版权与公开合规限制只能将邮件彻底丢弃；此外，提交中频繁出现非标准补丁格式、缺少上下文行、甚至使用 root 超级用户权限构建内核的迹象。由于维护者依赖视觉模式识别来审查逻辑漏洞，不规范的代码缩进和风格会极大阻碍审查效率，而连自身都未尝试编译就对外发送的代码更是会被立即退回。可二分性原则与违规提交流程的惨痛教训 (18:07) 内核补丁序列中的每一个中间提交都必须保证能独立编译通过，以维护 git bisect 自动化追溯缺陷的能力。提交者绝不能寄望于“在后续补丁中修复已知故障”，因为维护者承担着代码的长期维护责任，而外部贡献者在补丁合并后随时可能因离职或项目终止而彻底失联。历史上一项关于核心网络层的巨额重构在合并次日作者即失联，迫使网络子系统团队耗费整整一年才修复所有遗留漏洞。在具体的工程事故中，Linaro 曾提交过一套 8 补丁的 USB 驱动系列，不仅没有任何修改原因说明，其中 4 个补丁更是直接导致构建失败。在强行推动合并进入 -rc1 版本后，Texas Instruments 工程师在实际测试中发现该补丁直接造成全线硬件故障，迫使维护者向 Linus Torvalds 解释延误并紧急回退代码，涉事提交者也被处以一年内不接收其独立提交的严厉惩罚。即便是 Intel 这样顶尖的贡献组织，也曾在一份历经 5 轮评审的 TTY 驱动中遗留了设备拔出时故意触发内核崩溃的调试代码。这些事故证明，维护者必须时刻将拒绝代码作为保障系统安全的第一本能。维护者契约、公开责任与跨架构验证 (28:51) 健康的社区运作建立在双向契约之上：维护者要求补丁具备高水准的同时，也承诺在 1 至 2 周内给出评审反馈，并通过自动化脚本透明公开补丁的接纳与合并状态。技术批评必须严格聚焦于代码质量本身，提交者需要将个人自尊从代码中剥离。在公开透明的环境下工作，每项补丁都会永久署上作者及其雇主的名字，这种强烈的个人声誉绑定机制正是防止工程团队暗中塞入脆弱临时代码的核心屏障。在多架构兼容性方面，维护者不可能拥有市面上所有的开发板与专有芯片，因此实际硬件运行验证必须由代码作者自行承担。维护者会在其本地机器（通常为 x86）以及主要架构上执行构建测试，任何导致 x86、ARM 或 PowerPC 编译中断的补丁都会被一票否决。开发者必须将交叉编译器深度整合到本地构建流程中，确保代码在进入公共邮件列表前已消除了所有跨架构编译缺陷。 Greg Kroah-Hartman 指出，Linux 内核能够在庞大工程规模下保持健壮，核心依托于严苛的守门准则与公开透明的个人声誉绑定机制。然而，随着嵌入式设备和 ARM 芯片在计算领域的迅猛扩张，如何在硬件厂商急促的商业上市周期与开源社区严格的质量审查流程之间达成协同，仍是整个生态系统需要持续克服的工程挑战。

Source quotes

[26:24] Greg Kroah-Hartman: You have to convince me that I am doing something extraordinarily foolish by not accepting your code. [11:12] Greg Kroah-Hartman: If I am nice and take whatever you send me, Linux will not succeed. [10:36] Greg Kroah-Hartman: This is how businesses work: work with the community early, often, and you ignore this at your own peril. [15:09] Greg Kroah-Hartman: If I can't read your code, I'm not going to be able to fix your bugs, and I'm not going to take your code—it's as simple as that.

Original transcript

[0:01] maintainer I maintain the stable Linux [0:03] kernels uh lenus does the development [0:05] kernels every three months I do a stable [0:06] kernel about every week um I also do the [0:10] driver core USB debug FS TTY and serial [0:15] I've been doing serial for like 25 years [0:18] my first paid job and I'm still doing it [0:20] um and I do a lot of colel stuff um I'm [0:22] a maintainer also um so at the end of my [0:25] goal at the end of this talk is to for [0:27] you to understand why we are as [0:30] obnoxious as we [0:31] are and why you would never want to do [0:34] this job okay but first some numbers [0:36] these are always fun this is the Linux [0:38] kernel for last year uh 3.4 to 3.8 years [0:42] worth of development um 3,000 individual [0:46] developers at least 400 I'm guessing [0:49] about 420 companies I haven't done all [0:50] the numbers um this is the largest [0:53] software development process in the [0:54] history of us um this huge it's really [0:59] really really big um it's just amazingly [1:03] big let quote Douglas Adams um and it's [1:06] going really really fast um I run the [1:08] numbers every release and they keep [1:10] scaring me um but last year think about [1:13] this almost seven changes an hour 24 [1:16] hours a day seven days a week uh the [1:18] year before that we were at [1:20] 5.9 year before that 5.0 we keep going [1:24] faster and that's really really [1:26] important for you to know we're going [1:27] faster and so what you thought was the [1:30] development cycle of last year or you [1:32] were used it two years ago so if you [1:33] want to keep code out of the kernel and [1:35] think ah I will just C catch up to them [1:37] later well they're going faster so [1:39] they're going faster and you're gonna [1:41] have to do more work to try and catch up [1:43] that's really really important um so [1:45] much so that was a Year's average last [1:47] Colonel we just [1:49] released even faster um I thought when [1:53] we were at 2 point something that was [1:56] unsustainable um I've been wrong every [1:58] year I thought we were leveling off last [2:01] year we're not that's [2:04] insane so how do we do this so what are [2:07] we doing um the traditional model I give [2:09] this I've given the whole talks about [2:11] this so developers we have 3,000 of them [2:13] they create a patch they send it to the [2:14] maintainer the file about 700 of these [2:17] listed in the maintainers file they look [2:19] at it they sign off on it they review it [2:21] they push it back they accept it they [2:23] send it off to the subsystem maintainers [2:25] we have about I looked last merge [2:27] release we have about 86 of them um [2:30] there's about 160 different trees they [2:32] keep their trees every single business [2:34] day they get merged in Linux next by [2:36] Stephen Rothwell we test them we see [2:38] what's going on when the merge window [2:40] opens all the subsystem maintainers [2:41] shove everything to lenus and then [2:43] Andrew's out there picking up all the [2:44] scraps of everything that we all miss [2:46] that's the development model it's looks [2:49] like a nice pretty tier it looks nice [2:51] and pretty a number of years ago you [2:52] might have seen it ol I graphed this it [2:55] was a 5 meter long by two meter tall [2:58] graph it was insane um it's good [3:01] networking effects we can route around [3:02] anybody we get in the way um but let's [3:06] talk about these subsystem inter oh no [3:08] let's talk about companies people okay [3:10] so this gives you a sense of scale of uh [3:13] quantity not [3:14] quality um I won't judge quality um [3:18] quantity so this gives you a sense of [3:19] tell where things are really happening [3:21] fast in the kernel heartly um is doing [3:23] the comd driver subsystem data [3:25] acquisition Mark around embedded video [3:27] or no embedded sound sorry um Al file [3:29] systems alexel and um Bill are doing [3:33] janitorial work clean up the Kel lots [3:36] and lots of really good cleanup patches [3:37] Alex's awesome I don't I don't even know [3:40] who he works for um me I was doing some [3:42] janitorial work Daniel veter Intel [3:44] Graphics johannisburg Intel Wireless [3:48] think about that for a second one set of [3:49] driver subsystem these guys are doing [3:51] that much work Intel um David Miller [3:54] networking shashan hey lenaro he works [3:57] for you guys he's do what is he doing [3:59] janitorial work he's doing really good [4:01] stuff he's like all over the tree good [4:03] stuff um so this is what we do so this [4:06] is what we did in a year that's a lot a [4:08] lot of work um so what about the [4:10] maintainer so as the patches flow [4:12] through [4:13] something that's all I did last [4:16] year this is all I do so look at that [4:18] number these are number of patches [4:20] accepted these are not the number of [4:21] patches that have been submitted think [4:24] about that for rate of change so me and [4:25] David do a lot a lot of work I looked at [4:28] the numbers actually uh there's 470 [4:31] unique people that David talked to I [4:33] only talked to [4:34] 450 um so if you come up to me and say [4:37] oh thanks for taking my patch I'm like [4:39] you're [4:41] welcome um I really that was unique [4:44] people that we s to Patches Mark again [4:46] embedded video so this shows where the [4:47] churn and the co of the colonel is I do [4:50] a lot of work in staging that's a lot of [4:52] churn networking David and IDE embedded [4:55] sound again Maro video for Linux John [4:56] limbo's Wireless lenus is catch all for [4:59] every thing with Andrew heartley Again [5:01] comedy this is one guy those are the [5:03] patches he wrote and he's in the top 10 [5:05] of what he did think about that Daniel [5:07] better again um Intel but this shows how [5:10] many people behind Daniel are working on [5:13] the Intel Graphics subsystem which is a [5:15] lot it's really good and again Alo file [5:18] systems so these look at this workload [5:23] um that's a lot of work this is what we [5:26] do so let's talk about who pays for this [5:28] for this sense um congratulations you [5:31] guys C cracked the top 10 for the first [5:33] time uh that was good to see but you are [5:35] still as into quote Dave in your own [5:37] little gravity well pulling yourself out [5:39] you are still down the bottom but um the [5:42] interesting numbers is number one and [5:44] number four it's a roughly about 20% of [5:47] the kernel is done by individuals or you [5:49] can f it the other way 80% is done by [5:51] companies these are people that haven't [5:53] been hired [5:55] [Music] [5:57] yet seriously they're very they do say [6:00] but my goal of me as a developer and to [6:03] make Linux succeed is I want to help [6:06] those two groups of people out if you're [6:08] hired to do this stuff you're on your [6:10] own and you better know what you're [6:11] doing remember that in a minute um TI [6:16] good Vision engraving this is um har [6:17] heartley so he works for a company that [6:19] makes engraving systems and they use [6:22] Linux one developer made a company in [6:24] the top seven that should put every [6:26] other company to shame if you think [6:28] about that no Val soua IBM's going down [6:31] Google's moving up which is nice to see [6:34] um lar so then Samsung's there Wilson [6:37] linut Consultants Linux Foundation just [6:39] me and lenus immediate Oracle free scale [6:42] um [6:43] broadcom Qualcomm fell off the list [6:46] qualcomm's the only company that hides [6:47] their [6:48] patches they're really good at that um [6:51] they have good [6:54] lawyers so let's talk about some more so [6:58] I've traveled the past year talking to [7:00] lots of companies um one of your [7:02] partners actually or your member [7:04] companies um gave me this list of how [7:06] they do chip Design This is their what [7:08] they call their development process [7:10] design emulation simulation tape out so [7:12] what the company I was talking to they [7:14] start submitting they work on their [7:16] kernel patches in the middle and they [7:18] submit their kernel patches way over to [7:21] the right after the ow bring up three to [7:24] six to eight months later their kernel [7:26] patches are [7:27] accepted but their hard work guys once [7:30] their Hardware guys get the power on [7:31] they're off on the next design right six [7:33] month Cycles everybody's going on and [7:35] they're making new things so by the time [7:37] the kernel developers and S so [7:40] developers that are doing the Linux [7:42] stuff have the code accepted in the [7:44] kernel the hardware guys have almost [7:45] finished their second design um so any [7:48] feedback loop is hugely slow maybe a [7:52] year and a half a year at the best so [7:54] that the hardware guys can improve the [7:56] chips for the Next Generation for Linux [7:58] this is not what you want to do you will [8:01] always always play ketchup your [8:04] customers are buying these chips they're [8:07] starting to order them at power on [8:08] integration stage right they're starting [8:10] to make their own boards at OS bringup [8:12] time they need code in the kernel they [8:15] need it now they can't work on an older [8:17] kernel they're G you're G to give them [8:19] stuff it doesn't work this is what you [8:22] need to [8:23] do and you did this for arm 64 you're [8:26] getting better colel code submission to [8:28] simulation accepted three to six months [8:31] later then by the time the chip gets to [8:33] the customer they can go download it [8:35] from kernel.org now everybody I talked [8:37] to is like oh yeah right we can't do [8:40] that um your competitors are your [8:44] competitors are doing this your [8:46] competitors are beating you Intel as a [8:49] perfect example two or three three [8:51] Colonel releases ago we ripped a whole [8:54] subsystem for a whole chip architecture [8:58] out of the colonel that never saw the [9:01] light of day Intel is that far ahead of [9:04] you guys they can when the time their [9:07] chip makes it out to production to a [9:10] customer they say go get the col.org [9:12] colel it's all there there's nothing [9:15] that Intel has to do beyond that the [9:17] community supports it Intel's hired the [9:19] right people Intel is beating you if you [9:22] don't catch up they will they'll win [9:26] they will beat you they know how to do [9:28] this and they know how to really really [9:30] well um so you can say no no no we can't [9:34] do this fine that's your your fault but [9:38] and you getting better arm 64 you've [9:40] gotten there but you're on a simulator [9:42] where's the S so's where's the real [9:45] silicon real silicon is in your [9:46] companies I know because some of you [9:48] accidentally emailed me about [9:52] it um yeah one company said hey we want [9:55] the RM 64 patches backported to 3.4 [9:58] because we're shipping this device with [10:00] this I was like oh you have silicon oops [10:02] we weren't supposed to tell you [10:05] that oh anyway there is this silicon is [10:08] out there in your Labs you should have [10:10] the code in the kernel today there's no [10:13] excuse not to and it's not just a oh [10:16] great your customers or your competitors [10:18] are doing it or oh wow the community [10:20] really wants you to it is money it is a [10:24] business decision IBM and Intel have [10:28] publicly stated that these our chief [10:30] Executives Dan Fry is so high up at IBM [10:33] I don't know where he is now I think [10:34] he's in charge of all software now right [10:36] something crazy um it's a business [10:39] decision it saves them money it gets [10:41] them customers and it they're doing and [10:44] they're saves them time this is how [10:46] businesses work work with the community [10:49] early often and you do that you do this [10:51] at your own you ignore this at your own [10:54] Peril IBM and Intel are nice enough to [10:56] tell everybody about this nobody's [10:58] listening [11:00] so if you get anything from this you [11:02] need to change what you're doing getting [11:04] better you still need to change if you [11:06] don't if you want to succeed if you [11:07] don't hey we can all go work for [11:09] in all right so let's talk about what I [11:12] do Dave Miller made this quote a while [11:15] ago and it it's really really good I am [11:19] an editor I work with you I work with [11:22] the people who create something like an [11:24] editor for a Publishing House you are [11:26] the author I correct your grammar I fix [11:28] your responses say no push back do this [11:30] no it wouldn't be great if you do this [11:32] that is what I do I I help you I bring [11:34] you along I console you I want to make [11:38] the end product better the reason I'm [11:40] here and the reason I push back is [11:42] because I want the end result to be [11:44] better I want Linux to succeed you want [11:47] Linux to succeed if I am nice and take [11:49] whatever you send me lunux will not [11:52] succeed we are the editors we also used [11:54] to be writers we dream of being writers [11:57] again and every once in a while we say [11:59] ah forget it I'm gonna ignore you for we [12:01] can do and go do something else um I [12:03] have a new project I'm working on Dave [12:05] picks up things every once in a while [12:06] but we are maintainers are like editors [12:09] view us that way so that list of [12:12] companies um those two groups the [12:14] unaffiliated and the ones we don't know [12:16] those are ones who are doing this out of [12:18] their passion they want Linux to succeed [12:20] better I want them to succeed so I'm [12:22] going to be very nice to them I be very [12:24] helpful for them if you work for a [12:26] company you're getting paid for this [12:28] it's your job know better um don't make [12:31] me mad I'm not I it's not maybe it's not [12:35] fair but I'm not going to be as [12:36] forgiving with you but that's only fair [12:39] because it's your job and it's my job [12:41] that way so to give you a sense of scale [12:45] you saw the numbers of what I accept [12:48] let's talk about what gets sent to [12:51] me um about six months ago I looked at [12:54] during a merge window that was sent to [12:55] me we have a two we merge window when I [12:57] can't take any patches all all the colel [12:59] developers who are part of the colonel [13:01] Community know this and they don't send [13:02] me anything so I took this two week [13:05] window as okay this is stuff sent to me [13:07] by people who don't really know any [13:09] better um I got 487 patches two weeks [13:15] this is a slow period This is my inbox [13:18] so let me look at what was sent to me [13:20] and what was wrong um there was no other [13:24] 47 patches [13:27] [Music] [13:30] that was a good one um this one was nice [13:32] there was oh yeah you can't you're gonna [13:34] make me guess I mean come on um don't do [13:37] this stuff this is all the fun things [13:38] are wrong um P there was no patch [13:42] two that was good um the signed off by [13:46] was in the signature below the file at [13:48] least they got the signed off by in [13:50] there that was a good one but I couldn't [13:51] do it our tools can't handle that um [13:54] this was [13:56] great this happens a lot if you you send [13:59] a patch and your company adds a [14:01] signature saying this might be [14:02] confidential blah blah blah we have [14:04] email lists that it just automatically [14:06] deletes it I am legally required to [14:08] delete that mail and never respond [14:11] seriously that's what my lawyer say if [14:13] your company is putting a footer on a [14:15] patch on emails that you send out I [14:18] cannot respond to you because you just [14:20] told me I [14:21] couldn't you just sent me something that [14:23] was supposed to be public and you're [14:25] basically saying please delete this if [14:26] this is public okay [14:29] external development is done in public [14:32] think about that if your company is [14:33] putting Footers like this you will not [14:36] get code in um if if you have no [14:38] maintainers that are accepting code let [14:40] me know and I'll Point them out our [14:41] lawyers and tell them to ignore it um [14:44] because this is the legal advice I have [14:46] been given um so don't do that push back [14:49] on your on your [14:51] companies um exchange does this um [14:54] Huawei figured out how to fix exchange [14:57] talk to Huawei if you have Exchange they [14:59] actually got it to work they're like oh [15:01] yes we have this magical document and [15:02] here you have to do this this way [15:04] Microsoft also got it to work [15:05] Microsoft's a car devel than car [15:06] developer they got it to work um leading [15:10] spaces removed there's some email client [15:12] that does this this is horrible I have [15:14] to edit Patches by hand I don't know why [15:16] you don't want me to edit your patch by [15:18] hand I didn't know diff still worked in [15:20] the non-unified [15:21] format I think the BSD guys still use it [15:24] I can't read it um I don't know what's [15:27] going on don't do that um created patch [15:30] was created in the wrong place that was [15:32] nice um how many people can see how many [15:35] wrong things were wrong with this one [15:37] this was sent to me six months ago [15:39] remember that uh wrong version really [15:42] old place and you're running this means [15:45] you're running the you're building the [15:46] kernel is root uh we had a bug in the [15:48] colel build process that accidentally [15:50] deleted [15:51] slash none of us noticed it for about [15:54] three months because we didn't do it um [15:56] we're kind of thinking we should put the [15:57] bug back in [15:59] never ever build a Colonel's root never [16:02] build any never run gcc's root um yeah [16:06] that was bad um made against a different [16:08] tree I got Patches sent to me that were [16:10] made against the scuzzy developer tree I [16:12] don't do scuzzy thank god um that was [16:16] funny um wrong coding style this one's [16:18] really important we have tools to check [16:19] this I was talking to somebody else our [16:21] brains work on patterns you need to see [16:24] a patter if once you get used to the [16:26] pattern the metadata goes away and you [16:28] see the content we have a coding style [16:31] the only thing about a coding style is [16:32] you have to follow it it's not might be [16:34] good or bad don't argue that you have to [16:36] have a consistent coding style if I see [16:38] code that is not the Linux kernel coding [16:40] style it hurts I can't see it it doesn't [16:43] I don't know what's going on so the [16:44] first thing you'll get is fix the coding [16:45] style we have tools that automatically [16:47] do this for you um this was [16:50] good somebody thought they were better [16:52] than the 3,000 other cental [16:55] developers uh we have a reason because [16:56] we all work together I want you to fix [16:58] my bugs you want me to fix your bugs if [17:00] I can't read your code I'm not going to [17:02] be able to fix your bugs I'm not going [17:03] to take your code it's as simple as that [17:06] um would not compile that happens a lot [17:10] uh this happened to somebody work he's [17:12] not in this room today [17:14] um yeah come on build your code um [17:18] that's just plain obvious you guys have [17:20] tools internally to handle all this [17:22] stuff for us automatically now I know [17:24] what you really have I'm just not going [17:26] to accept any of this stuff um [17:30] this was good if you send me a patch [17:32] series uh the third patch broke it uh at [17:35] least they fixed it on the last one so [17:37] they knew it was there um but you have [17:39] to we have to be able to not break the [17:41] bill in every single patch um I'm a [17:43] little bit bad about checking this I'll [17:45] apply a 10 patch series at once some sub [17:48] system maintainers have code that [17:50] applies every patch in order builds it [17:53] figures out if it builds figures out the [17:55] warnings figures out other things and [17:57] before it will accept it or not um they [17:59] just sent me that those scripts so I'm [18:01] going to start doing that um that's good [18:03] stuff so you should do that um this one [18:07] was a good one a series because they [18:10] said that oh we'll fix it [18:12] later I don't trust you [18:16] um I don't trust you for a good reason [18:18] um if I accept your code this is [18:21] something important I am now responsible [18:22] for it it has happened many many times [18:25] that people send us large bodies of [18:27] codes saying you're going to fix it up [18:29] once the patch is accepted they [18:30] disappear there's a very well-known [18:32] documented case of somebody accept [18:35] submitted a very large change to [18:37] networking core the networking [18:39] developers accepted it the day later the [18:42] email address disappeared from the net [18:45] it bounced it was gone the person [18:47] disappeared if it was a person the [18:49] networking subsystem spent a year trying [18:52] to make up and fix those problems that [18:54] were caused by that patch we are very [18:57] very reluctant to accepting that are [18:59] broken or what not because it's now my [19:01] responsibility I don't know if you're [19:02] going to be here or not things happen [19:05] people actually pass away we've had [19:06] subst maintainers pass away saying [19:07] they're going to fix stuff up it's a [19:09] fact of life we want stuff to work [19:12] now um and then patch has nothing to do [19:14] with me I get that a lot that's fine um [19:18] this one's good uh 4,500 patch line um [19:20] some actually Laro people started review [19:23] reviewing [19:24] this I was like don't encourage them [19:26] please um patches have have to be like [19:29] your um math professor said show me the [19:32] work I want to see each little [19:33] individual step along the way don't give [19:35] me something big you cannot rewrite the [19:37] scheduler um you can add lots of changes [19:40] along the way one exception is drivers [19:41] we'll take a new driver kach chunk all [19:43] at once new file system all at once [19:45] because that's self-contained but if [19:46] you're modifying existing code show me [19:48] the [19:48] work um colel doc we have documentation [19:51] in the Kel way to do it um they s it [19:54] obviously wrong they try to never even [19:56] check to see if their documentation [19:58] worked not good uh this was calm two [20:01] weeks this is what I do this is what I [20:04] deal with uh I challenge anybody in this [20:07] room to do this stuff and not be [20:10] grumpy but you want me to be grumpy you [20:12] want me to push back on this stuff you [20:13] don't want me accepting crap because [20:15] that'll hurt Ms you guys have the [20:18] resources internal to keep all that [20:20] stuff from happening there was some [20:22] alaro emails and those I'm not going to [20:24] point them out um I've seen deox talk [20:27] it's online you have the resources to [20:28] give to your member companies you guys [20:30] employ people who know how to do this [20:33] you need to listen to them they know [20:35] what they're talking about they're not [20:36] just making you do extra work listen to [20:39] them [20:41] so let's talk about what you guys [20:44] do this was sent to me this is something [20:47] that just happened in January we're [20:50] hitting the end of the merge window or [20:52] end of the lmus was about to release [20:54] three um 3.9 my tree closed I said one [20:58] week before lenus releases I need to [21:00] stop adding stuff to my tree make sure [21:02] everything's stable make sure it's [21:03] working a nine oh it's actually two [21:06] weeks before that eight patch series for [21:08] USB showed up from a l Naro email [21:13] address no description why I needed [21:15] these patches there's one line saying in [21:17] each patch saying what it did I can read [21:20] code I know what it does I don't know [21:22] why I needed it I that was really bad um [21:25] you need to tell me why this is doing [21:27] what it's doing why I need accept it so [21:29] I ignored it um this was a great one [21:33] half of the eight broke the bill [21:35] obviously not ever tested not even built [21:40] um people caught this in the review [21:41] people other I ignored it other your [21:44] member companies I think TI was stepped [21:46] up and reviewed the code it was very [21:48] nice um actually the maintainer of this [21:50] driver at the time was on vacation he [21:52] was in the Caribbean he had no email [21:54] address access so I was going to just [21:56] let things go so they were recent which [22:00] is great but they didn't say what [22:02] changed um I get 400 I get 200 I get 300 [22:06] email patches a week I don't know I [22:09] don't remember what I said you to do [22:11] that patch you need to tell me what you [22:13] changed oh there not going to remember [22:14] you can't assume I remember somebody [22:17] this past week changed a patch by adding [22:19] one line to it to fix something I didn't [22:21] remember that I even told them to add [22:22] that um you got to tell me so no [22:25] description what changed um [22:29] somebody from Laro actually stepped up [22:30] and tried to help out and then they um [22:33] some replacement patches were sent out [22:35] of order not telling me which one they [22:37] had to apply to and just assuming I knew [22:39] which ones to replace them with I mean [22:42] I'm getting really really confused so [22:45] then they resent them [22:47] again um and they didn't even order them [22:49] in what order they had to be in so to [22:52] your credit somebody one of your Colonel [22:53] team members stepped in and said let's [22:55] do this right publicly said this is how [22:57] you do it this is what you need to do [22:59] and how to do it right so they sent him [23:02] again but they said hey fourth time we [23:04] sent it this is version [23:07] two I know that's not right even I'm not [23:10] even paying attention I know that's not [23:12] right so then you're they insisted this [23:15] is after the merge window closed for me [23:17] that two of these had to go in rc1 I [23:20] looked at it it looked all right I [23:22] couldn't really test them it's on arm [23:23] Hardware that I don't have your de [23:27] senior developer said these are okay to [23:28] go in I've known him for years I trusted [23:31] him I said okay let's take [23:33] him he resent them to me cleaned up form [23:36] said just take these two I took them [23:40] turned out they were [23:41] wrong but they just messed up things in [23:43] the configuration so it was just typos [23:45] that was not somebody forgot to clean [23:47] those up that's okay that's it was nice [23:50] but then I had to take a third patch so [23:52] now three patches into on my tree past [23:54] the deadline of when I normally accept [23:56] things to come in so I'm already put [23:58] push the limits of what I'm doing I'm [24:00] like okay [24:04] great TI stepped up again tested these I [24:09] I will give TI credit and said you just [24:11] broke all our [24:12] Hardware um you guys have the [24:14] infrastructure inside to test this stuff [24:17] I know you now I know you do I kind of [24:19] assumed you did now I know um he broke [24:23] half all the ti stuff um that was bad [24:26] they were they were mad so I had to go [24:28] and revert them all I had to add them [24:30] and I had to tell lenus here's the [24:32] reason why patches are in my tree past [24:35] your deadline it's because I had to rip [24:37] out broken lenaro code look at my chain [24:40] slog that I sent to lenus it's in the [24:41] kernel tree now because of what [24:45] happened [24:46] so maintainer returned like great let's [24:50] figure out what's going on here see what [24:55] happens everyone was broken every every [24:58] single one of the eight patches that [24:59] were in there they would build they seem [25:02] like they would run logically they [25:04] didn't even work right and this sub this [25:06] driver maintainer who is very nice very [25:09] calm been around forever no one would [25:11] ever accuse him of being grumpy said [25:13] this [25:18] publicly uh you guys got some work to do [25:23] um that's not going that's not okay read [25:26] the code read what's there read the [25:27] documentation in the code this developer [25:30] obviously hadn't I didn't know that I [25:31] don't know the driver subsystem but if [25:33] you're changing code and you don't even [25:34] read the stuff around it the comments [25:36] around it you shouldn't be doing colel [25:38] Cod this is stuff that now publicly is [25:41] attached to Laro name you have the [25:43] resources to prevent this please please [25:46] do that I will not take code from this [25:50] developer again for about a year until [25:51] so I forget them I will now require a [25:55] senior lenario Colonel developer to sign [25:57] off on any pages from lar that I will [26:00] accept that's my rule now I've told it [26:03] I've told it publicly and that's just [26:05] what it's going to be and I think if you [26:07] were in my position you would do the [26:08] same I hope you would do the same um I [26:11] can't take this I mean you broke lots [26:14] and lots of other Hardware out there [26:15] that's not okay this is why I'm [26:18] grumpy all right that was a lot of the [26:20] narrow bashing um just to be oh I did [26:24] that okay let's talk about somebody else [26:27] that's not just you [26:30] guys half half of you guys I'll wait [26:32] till you get a punchline here um this [26:34] came from a company that is in the top [26:36] five developers um this was the fifth [26:39] revision of a patch set that had been [26:41] reviewed by a number of people for the [26:43] TTY layer I was kind of worried about [26:45] this because it looked like it was [26:46] duplicating a ton of existing code we [26:48] had in there some people said no no no [26:50] it's good please take it it was a big [26:52] patch I started reviewing it finally [26:54] okay five revisions doing good [26:58] um so what this code does is it release [27:02] releas is whenever a device is removed [27:03] from the system somebody decid to put [27:05] some bug debugging code and that would [27:07] crash the machine if you ever pull the [27:09] device out of it they wanted me to take [27:12] this code they even left their helpful [27:14] little comment in there test the code [27:16] how do you test code by putting a bug [27:19] that crashes that stops your machine [27:21] hard this developer was working with the [27:23] driver framework to test the Dynamic [27:25] Addition and removal of devices and they [27:28] never even tested the removal of the [27:30] device and they wanted this in running [27:32] code that meant all modems all wireless [27:36] modems all embedded modems all USB [27:38] serial devices suddenly would not even [27:42] work I told this guy after that I said I [27:46] will not accept your code I will require [27:49] Intel's ohop [27:55] sorry I will require their senior Vel [27:58] opers to sign off on the code before I [28:00] will accept it um so it's not just you [28:03] not so it's not just you getting this [28:05] stuff um getting wrong other people get [28:06] it wrong too um so don't think I'm just [28:09] picking on you um I like inel a lot and [28:12] this happens there I like you guys a lot [28:13] it happens [28:15] so this is what so I have to maintain [28:19] your code in order for Linux to Linux [28:21] works great today it's a risk for me to [28:23] take your code tomorrow it's my [28:25] self-interest not to take your code it's [28:27] that simple [28:29] you have to convince me that I am doing [28:31] something extraordinarily foolish by not [28:34] accepting your code you have to make a [28:35] perfect you have to give me no reason [28:37] for me to want refuse your code you have [28:39] to make it look like I'm the jerk for [28:41] not accepting your code seriously you [28:43] have to make it I have to be like ah of [28:45] course I would take that code that's [28:47] beautiful do that you have the resources [28:49] to do that you do that and if you do [28:51] that here's what I'll give you no excuse [28:55] um as a maintainer I am saying this is [28:59] what I'm going to do for you and this is [29:00] what you need to expect from your [29:01] maintainers this isn't just a one-way [29:02] streak I will respond to your patches [29:05] within one to two weeks if I don't ping [29:07] me uh sometimes in an emerge window okay [29:09] that's busy I'm travel a lot but let me [29:11] know I do not get upset if somebody says [29:13] what's up with my patches that's okay [29:15] but wait two weeks don't send me a patch [29:17] today 12 hours later ask me why I [29:19] haven't you reviewed it it happens I was [29:22] asleep I was actually I don't sleep I [29:25] was on a plane um so do that I will do [29:27] that [29:28] um I will I promise I will offer you [29:31] semic constructive [29:33] criticism I can't IM I will if I I [29:36] should always um criticize the code and [29:39] not the person uh recently a number of [29:42] there's been some comments on Linux Kel [29:44] reporters have discovered hey there's [29:45] these Engineers doing stuff in public [29:47] and they curse imagine that um we don't [29:51] um if I know you I might curse you out [29:54] if lenus knows you he might curse you [29:56] out um like my wife says is just called [29:58] you stupid [30:00] today [30:01] yes um so I will respond to your code if [30:05] you come to Comm come to conferences [30:07] come to things meet us you'll realize [30:09] we're not big scary people we're just [30:11] nice and um okay I might be big and [30:13] scary um but I'm trying to do it for the [30:16] benefit of what you're doing I will [30:18] respond to your code but detach yourself [30:20] from your code my very first driver I [30:21] submitted within two minutes it was cut [30:23] to shreds and I lost my ego then 15 [30:26] years ago um you got to do it it's just [30:28] code um that's it I will let you know [30:31] the status of your patch if I accept it [30:33] you'll get an email if I don't accept it [30:35] you'll get an email um I hate the [30:37] maintainers you throw code at them and [30:38] it just disappears and then three months [30:40] later it shows up in the colonel it's [30:41] like oh I didn't know I forgot about I [30:44] will ask from this from your subsystem [30:46] maintainers to know the status patch [30:47] it's really really important Andrew [30:49] Morton has automatic scripts from that I [30:51] modify them to work with my build system [30:53] his scripts are public my scripts are [30:54] public other maintainers are accepting [30:56] this are doing it as well some actually [30:58] posts things to Twitter for them [31:00] maintainer whatever the maintainer wants [31:01] to do but I will let you know the status [31:02] your patch and then [31:07] finally this came up I posted some code [31:10] was written that in the comments of the [31:11] code it said this never submit this to [31:14] the Linux [31:16] konel and I was like that's really sad [31:19] it's sad that this developer felt that [31:21] the code that he was writing or they [31:23] were writing I don't know if it was he [31:24] sorry um was so bad and it was that they [31:27] knew knew how bad it was and they knew [31:29] they should never even submit it and [31:30] that's wrong that you are doing [31:32] something wrong if you're doing that [31:33] that you're being forced by somebody [31:34] else to do something you know you're [31:36] doing something wrong and the only [31:37] reason Linux works so well is because [31:40] personally you're responsible for the [31:42] code that you send off you are attached [31:44] to it it's assigned to you and your [31:45] company better not as those other [31:47] patches were shown so do it in public [31:50] and because of that it makes it us work [31:53] really really well it's high for us to [31:54] hide crappy hacks we don't have [31:56] deadlines we don't have things thrown at [31:58] us like this it works really really good [32:01] Lena says it best that way so that's why [32:05] Linux is succeeding that's why we're [32:07] grumpy that's why you need to do good [32:10] work and of course my obligatory payment [32:13] lecture and there's my slides they're [32:15] all public they're on the Lara [32:17] website [32:19] questions it's quiet [32:21] yes oh I'm I'm displaying from a [32:24] raspberry piie I hate raspberry piie [32:28] that's a horrible process but no I'm [32:30] presenting from an arm [32:34] system yes there seems to be kind of an [32:37] implicit did you test this on x86 in [32:41] most patch submissions like if you [32:43] really want to be be sure so is that [32:45] something that when you actually test a [32:47] patch with most maintainers they're [32:49] they're typically testing on x86 so [32:51] should it work I can't test your code I [32:54] don't test your code unless it happens [32:55] to be for the hardware I have I will [32:57] build it on 86 a better bill because if [32:59] it didn't you just broke the bill you [33:01] can't make the bill for other architect [33:03] that's a patch that I was talking about [33:04] I was working with John this past week [33:07] wasn't partner partner it would build on [33:09] arm but it wouldn't build on x86 [33:11] somebody else a patch from or that [33:13] wouldn't broke to build on power PC we [33:15] have cross compilers you have them in [33:17] your build [33:21] system so yeah I will build it if it [33:23] breaks my laptop I will even breaks my [33:25] build [33:30] thank you very much [33:33] [Applause]