Google Interview Scores Fail to Predict Job Performance, Veteran Says
Etsy Eng (Etsy Engineering)
Google found zero statistical correlation between technical interview ratings and on-the-job success, proving scalar scoring fails to measure candidates without multi-page narrative assessments and cross-team hiring committees.
AI-assisted summary of Moishe Lettvin - What I Learned Doing 250 Interviews at Google — original source. Check the original for full context.
Source published:
Key ideas
- Internal data at Google showed zero statistical correlation between interview performance scores and subsequent on-the-job performance, underscoring that qualitative narrative feedback is far more predictive than scalar metrics. (9:39)
- In a 2007 experiment, Google's strict Kirkland hiring committee reviewed blinded interview packets of its own sitting members and voted to reject every single one of them, illustrating the systemic bias toward false negatives. (6:01)
- Decoupling interviewing from hiring decisions through cross-functional hiring committees prevents 'island evolution,' a process where individual managers lower standards to meet headcount quotas. (15:05)
- Technical interview loops carry substantial engineering overhead, consuming roughly three engineering hours per one-hour interview once preparation, documentation, and committee reviews are included. (4:31)
Detailed notes
## Background This talk was published on YouTube by the Etsy Eng (Etsy Engineering) channel on May 21, 2014. No video description was provided by the uploader.
## TL;DR Drawing on his experience conducting roughly 250 engineering interviews and serving on Google's hiring committee for six years, Moishe Lettvin provides an insider look into the mechanics, flaws, and design principles of technical hiring. He explains why Google decouples interviewers from final hiring decisions, demonstrating that single scalar scores fail to predict performance and that detailed qualitative narratives are the only reliable signal. Lettvin advocates treating interviews as generous, collaborative conversations built around multi-layered 'onion' problems rather than stress-inducing gauntlets. He contrasts Google's consensus-driven committee approach with Microsoft's manager-centric model, stressing the necessity of strong culture checks to prevent organizational decay while acknowledging that the standard single-day interview remains a deeply flawed local maximum.
## Atmosphere & style The presentation carries a candid, pragmatic, and self-deprecating tone, delivered with the easy confidence of a 25-year industry veteran sharing hard-won wisdom with his new colleagues. Lettvin peppers technical critique with humor and historical anecdotes—from Microsoft war stories to Google's Kirkland committee inadvertently rejecting all of its own members. The vibe feels less like a corporate recruiting seminar and more like an open, philosophical debrief that humanizes candidate evaluation and challenges conventional tech interview dogmas.
## Bulletin - [09:39](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=579s) Internal data at Google showed zero statistical correlation between interview performance scores and subsequent on-the-job performance, underscoring that qualitative narrative feedback is far more predictive than scalar metrics. - [06:01](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=361s) In a 2007 experiment, Google's strict Kirkland hiring committee reviewed blinded interview packets of its own sitting members and voted to reject every single one of them, illustrating the systemic bias toward false negatives. - [15:05](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=905s) Decoupling interviewing from hiring decisions through cross-functional hiring committees prevents 'island evolution,' a process where individual managers lower standards to meet headcount quotas. - [04:31](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=271s) Technical interview loops carry substantial engineering overhead, consuming roughly three engineering hours per one-hour interview once preparation, documentation, and committee reviews are included.
## Key takeaways - [00:00](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=0s) Google separates the interview loop from hiring authority: 4–5 interviewers gather technical evidence, while an independent hiring committee evaluates the blinded packets. - [04:31](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=271s) Conducting interviews costs roughly three hours of engineering time per one-hour slot due to prep, multi-page write-ups, and review processes. - [06:01](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=361s) Google intentionally favored false negatives over false positives, driven by Steve Yegge's thesis that any qualified engineer faces a 'perfect anti-slate' of interviewers who would reject them. - [09:39](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=579s) Quantitative hiring scores fail because projecting multidimensional human capability onto a 1.0–4.0 scalar generates uninformative histograms without real predictive power. - [09:39](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=579s) The rich written narrative explaining how a candidate navigates problems, absorbs hints, and reasons through edge cases is the only reliably predictive component of an interview packet. - [15:05](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=905s) Single hiring managers create 'island evolution' by lowering bars under headcount pressure, whereas centralized cross-functional committees maintain unified company-wide standards. - [18:04](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=1084s) Interviewers must approach sessions like jazz musicians, preparing structured timings while remaining adaptable enough to pivot when questions are compromised or candidates get stuck. - [22:20](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=1340s) A candidate's emotional experience takes precedence over extracting marginal signal, because poor interview impressions severely damage company reputation and recruiting pipelines. - [29:24](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=1764s) Effective technical questions follow an 'onion model,' beginning with an accessible naive baseline before peeling back into deeper algorithmic complexity and optimization trade-offs. - [36:30](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=2190s) The current tech hiring format of five or six one-hour whiteboard interviews in a single day likely represents a local maximum rather than an optimal evaluation method. - [38:02](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=2282s) The practical litmus test for any software hire is whether teammates would feel comfortable holding the on-call pager after that candidate pushes a week's worth of code. - [43:06](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=2586s) Technical brilliance cannot compensate for toxic or disrespectful behavior, and a candidate who creates friction should be rejected regardless of raw domain expertise. - [45:34](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=2734s) Google prohibited interviewers from communicating before submitting their written evaluations, avoiding the groupthink and anchoring bias common in running email threads. - [54:16](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=3256s) Calibrating hints requires avoiding binary or vocabulary-testing questions, allowing interviewers to nudge candidates without simply giving away the algorithmic solution. - [56:33](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=3393s) Asking open-ended technical questions such as 'what is your favorite algorithm?' effectively reveals authentic passion and communication skills without triggering rehearsed behavioral scripts.
## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=0s) Introduction: 25 Years in Tech and the Google Hiring Machine
Moishe Lettvin introduces his talk on the lessons drawn from roughly 250 interviews he conducted during his seven years at Google. Having recently joined Etsy’s Seller Economics team four months prior, Lettvin reflects on realizing while preparing the presentation that he has been in the software industry for 25 years. His career path wound through a small time-and-billing company in Massachusetts, two separate stints at Microsoft, a startup, Google, and now Etsy, where he focuses on math, statistics, and distributed computing alongside domain experts. He notes that throughout this career, he was consistently a non-traditional candidate: he dropped out of college after a single semester with a 2.1 GPA, leaving him perpetually nervous entering interviews with a resume conspicuously missing standard credentials.
```mermaid flowchart TD accTitle: Google Interview and Hiring Committee Pipeline accDescr: Diagram illustrating how candidate feedback flows from interviewers into a packet evaluated by an independent hiring committee. subgraph Slate ["On-Site Slate"] C["Candidate"] --> I1["Interviewer 1"] C --> I2["Interviewer 2"] C --> I3["Interviewer 3"] C --> I4["Interviewer 4 / 5"] end subgraph PacketPrep ["Packet Collation"] I1 --> W1["Written Feedback"] I2 --> W2["Written Feedback"] I3 --> W3["Written Feedback"] I4 --> W4["Written Feedback"] W1 & W2 & W3 & W4 --> P["Candidate Packet"] end subgraph Decision ["Independent Review"] P --> HC["Hiring Committee\n(Disjoint from Slate)"] HC --> D{"Hire / No-Hire Decision"} end ```
Lettvin outlines the mechanics of the Google hiring architecture, in which he served on the hiring committee for six of his seven years and taught interview training for six years. Unlike traditional processes where an interviewing panel makes the ultimate call, Google separates assessment from decision-making. A slate of four or five on-site interviewers individually produces written evaluations. These write-ups are collated into a comprehensive packet delivered to a dedicated hiring committee. The members of that committee evaluate the dossier and determine the hire or no-hire outcome without ever having met or spoken with the candidate in person.
- Lettvin spent seven years at Google, six years on the hiring committee, and six years conducting internal interviewer training. - Entered the tech industry without a college degree following a single semester resulting in a 2.1 GPA. - Transitioned to Etsy's Seller Economics team to study distributed computing, advanced mathematics, and statistics.
---
[02:14](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=134s) The Joys of Interviewing: Intellectual Sparks and Meeting Legends
Interviewing possesses an unexpectedly enjoyable side that Lettvin had never anticipated prior to sitting on the interviewer’s side of the desk. Interacting with new people, exchanging deep domain knowledge, and watching candidates reason through tough computational puzzles creates genuine intellectual connection. He recounts asking his favorite technical interview prompt—finding all valid words on a Boggle board—to a recent college graduate interviewing at Google. The problem features two distinct analytical approaches that eventually converge into a single elegant solution. With just a few gentle prompts, Lettvin watched the candidate experience a breakthrough as both paths snapped together:
> “he could just see in her eyes these two things coming together into this like gleaming beautiful like point and she like literally said oh my god that's order one that's what everybody wants and just like freaked out right and it's like this is so much fun um so so that that's part of it like that you get that personal connection”
Beyond individual breakthrough moments, Lettvin credits his years on the hiring committee with providing him with a complete computer science education. Reviewing hundreds of packets exposed him to a vast catalog of algorithms and evaluation techniques that he had never encountered in formal schooling. Interviewing also provided chances to meet iconic technologists across the broader computing landscape outside standard developer hiring tracks, including Philip Zimmermann, the creator of Pretty Good Privacy (PGP).
- Boggle solver question relies on two parallel algorithmic paths converging into an $O(1)$ lookup structure. - Hiring committee evaluations served as a practical substitute for a traditional academic computer science degree. - Candidates interviewed ranged across various roles, including notable industry figures like Philip Zimmermann.
---
[04:31](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=271s) The Downside: High Stress, Time Sinks, and Whiteboard Friction
The interview process also carries severe operational costs and emotional strain. Conducting a single one-hour interview at Google routinely consumed roughly three hours of an engineer's schedule once preparation, conducting the session, drafting comprehensive feedback, and handling subsequent follow-ups were accounted for. For both candidates and engineers, the social friction of encountering a complete stranger creates anxiety, particularly in an engineering culture where interpersonal interactions are rarely an innate strength.
Interviews can derail due to unexpected communication breakdowns, severe performance anxiety, or personality clashes. Candidates occasionally break down in tears after losing traction on a problem and finding themselves unable to recover. On rare occasions, interviewers face outright hostility; Lettvin recalls an interview with a candidate who carried an intense chip on his shoulder, as well as senior applicants who flatly refused to write code because they felt writing on a whiteboard or computer was beneath their seniority level.
Whiteboard coding itself introduces substantial operational noise into technical assessment:
> “if you've never written code on a whiteboard writing code on a whiteboard is completely it's difficult to do you're used to thinking in Emacs or vim or eclipse or whatever and you're used to autocomplete and you're used to be able to look stuff up in Google when you can't find it and so on and all of a sudden you just got this blank slate you get at code and not only that you're getting quite bored marker all over your sleeves because you can't find the eraser to erase it's just like this kind of cascading problem that can be very hard”
- A single 60-minute interview requires an overhead of approximately three hours of total engineering time. - Whiteboard testing strips away standard software development tools such as autocomplete, documentation lookups, and text editor environments (e.g., Emacs, Vim, Eclipse). - Emotional pressures frequently manifest as tears, acute frustration, or behavioral resistance from senior applicants refusing to code.
---
[06:01](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=361s) Arbitrariness, Steve Yegge's Anti-Slate, and the Kirkland Reality Check
The core structural reality Lettvin observed at Google is that interviewing is inherently noisy, inaccurate, and arbitrary. Noise stems both from artificial candidate environments and from external interviewer variables, such as fatigue, hunger, or domestic arguments before work. Because human error bars are so wide, running the identical candidate through the same interviewers twice can easily yield opposing decisions. To mitigate this volatility, Google allowed borderline applicants to re-interview after 12 to 18 months.
Lettvin shared an office with Steve Yegge, who formulated the concept that every viable candidate possesses both a "perfect slate" and a "perfect anti-slate." Under this dynamic, even for an engineer already successfully hired and thriving at Google, an organizer could easily assemble five company interviewers whose specific biases and pet questions would result in a unanimous rejection. Because the company viewed hiring a bad performer (a false positive) as far worse than passing on a good engineer (a false negative), the hiring bar was intentionally pushed extremely high, amplifying the arbitrary nature of the system.
This systemic randomness was dramatically demonstrated around 2007 at Google’s Kirkland office. The local hiring committee had developed a reputation for being exceptionally strict. One week, after examining a typical batch of eight to ten candidate packets, the committee voted to reject every single one. Immediately afterward, the recruiter delivered an eye-opening revelation:
> “at the end of hiring committee the recruiter was like well guess what guys these were your packets from when you were interviewed here and you just decided not to hire any of yourselves”
- Steve Yegge's "anti-slate" principle posited that five existing Google engineers could be found to reject any successfully hired candidate. - Google's hiring philosophy deliberately favored false negatives over false positives, discarding strong candidates to minimize poor hires. - In 2007, the Kirkland hiring committee evaluated blinded versions of their own historical application packets and rejected every single member of their own committee.
---
[09:39](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=579s) The Illusion of Data: Scoring Clusters and Performance Correlations
Google attempted to manage this uncertainty by amassing large volumes of data on the interviewing process. However, metrics often proved contradictory or uninformative. Lettvin points to public remarks from Google's People Operations chief Laszlo Bock (featured in *The New York Times*) revealing that extensive internal analysis discovered virtually zero correlation between interview performance scores and subsequent on-the-job performance ratings.
This lack of predictive power is rooted in how candidate feedback is scored. Google interviewers graded candidates on a scale from 1.0 to 4.0, where 1.0 represented a definitive rejection ("please don't hire this person") and 4.0 represented an enthusiastic recommendation ("definitely hire this person"). Because extremes were rarely awarded, the score distribution formed an exponential curve heavily clustering around the 2.0 to 3.0 range. This tight compression left hiring committees and analysts with an extremely narrow band of meaningful variation, severely limiting the statistical utility of the data for predicting long-term engineering success.
- Google interviewers scored candidates on a scale ranging from 1.0 (strong no hire) to 4.0 (strong hire). - Scores heavily clustered around the 2.0 and 3.0 thresholds, rarely reaching the extreme ends of the spectrum. - Internal studies highlighted by Laszlo Bock found zero statistical correlation between candidate interview scores and post-hire performance reviews.
[10:28](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=628s) The Flaw in Quantitative Scores and the Power of Written Feedback
Interview scoring systems that rely on simple numeric ratings are fundamentally flawed because isolated numbers mean radically different things depending on who gives them. Moishe Lettvin illustrates this by comparing his own scoring standards to a colleague named Michael on Google’s hiring committee who was notorious for giving extraordinarily difficult interviews: a score of 3.1 from Lettvin represented something entirely different than a 3.1 from Michael. While hiring committees attempted to build histograms of an interviewer's past scoring history to predict hiring outcomes, the histograms actually demonstrated the exact opposite: the colored bars showing whether candidates were hired for any given score were scattered all over the map. Lettvin emphasizes that the only truly predictive and substantive part of the packet was the written feedback—typically three to four pages of rich narrative detailing the candidate's actual thought processes and behaviors.
Attempting to reduce a multifaceted human being to a single number is an exercise in futility, akin to the flawed historical methodologies explored in Stephen Jay Gould's *The Mismeasure of Man*. Lettvin explains that trying to project a complex, high-dimensional cloud of human attributes into a single scalar dimension fails just as badly as relying purely on IQ or SAT scores. As Lettvin sums it up: “I think the upshot of that is it's very hard to put a number on things that matter about about a candidate basically putting a number on a person right is the same as IQ the same as SAT scores it's like the classic Mismeasure of Man conundrum right like you've got this cloud of data that you're looking at and you're trying to project that into one dimension and it basically doesn't work.”
```mermaid flowchart TD accTitle: Distortion of Candidate Evaluation accDescr: Diagram contrasting numeric score reduction with narrative feedback synthesis. subgraph Quant ["Reductionist Numeric Model"] A["Candidate Performance"] --> B["Scalar Score (e.g., 3.1)"] B --> C["Hiring Committee Histogram"] C --> D["Non-Predictive Outcome"] end subgraph Qual ["Holistic Narrative Model"] E["Candidate Behaviors & Nuance"] --> F["Written Narrative (3–4 Pages)"] F --> G["Committee Consensus"] G --> H["Predictive Insight"] end ```
[12:11](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=731s) Testing for Intangibles and Interviewing as Literary Criticism
Beyond the inadequacy of numbers, Lettvin notes that the qualities that make someone truly wonderful to work with cannot be measured by asking direct interview questions. To illustrate, he shares an anecdote about his Google mentor, Bob Day, during Lettvin's second week at the company. Lettvin had been tasked with adding a feature to a task management component for the Windows Google Talk client and was preparing to hack together a kludge. When he sent the code review to Day, Day asked how long it would take to simply rewrite the whole component properly. When Lettvin estimated two weeks, Day immediately told him to just go rewrite it. This interaction showed Lettvin that Day trusted him completely and valued code craftsmanship and maintainability over crude metrics like check-ins per week.
Lettvin points out that Bob Day had also once taken a year off from Microsoft to hack *EverQuest*, rewriting Windows drivers so he could intercept network packets and display real-time client-server communication data on a second screen. You cannot directly ask a candidate in a 45-minute window if they possess that degree of passion or trustworthiness; instead, interviewers must design open-ended problems that provide space for candidates to display their curiosity, creative ability, and enthusiasm. Watching how a person tackles an interesting puzzle reveals far more than whether they happen to know a textbook solution. As Lettvin summarizes: “I think the the key point that I want to say about this is I feel like interviewing is more like literary criticism than it is like statistics right like you you have this kind of big amorphous blob of data and you're trying to distill that but you can't distill it into numbers it only works to distill it into ideas.”
[15:05](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=905s) Preventing "Island Evolution" Through Cross-Team Committees
Interviewing must function as a multi-perspective team effort rather than relying on an isolated hiring manager. Lettvin draws a sharp contrast between the hiring processes at Microsoft and Google. At Microsoft, candidates met a slate of interviewers, but ultimate authority rested in a single hiring manager. This created a dangerous single point of failure driven by perverse organizational incentives: managers had open headcount that they needed to fill urgently for critical projects, and their managerial status was directly linked to headcount size. Fearing they would lose their allotted headcount if they waited, managers would inevitably compromise and hire someone just slightly below the bar. Over time, those newly hired individuals would themselves become interviewers and hiring managers, compounding the decline. Without external checks, this process of “island evolution” caused engineering standards on isolated teams to rapidly decay.
Google avoided this systemic decay by decoupling the hiring decision entirely from individual team managers. Hiring committees were formed across disparate teams and offices (for example, the Kirkland hiring committee), and hiring packets had to achieve genuine consensus across multiple tiers, extending all the way up to executive review where Larry Page personally reviewed and signed off on every single candidate packet. Because candidates were hired for Google as a whole rather than for an individual team's immediate opening, the company preserved consistent engineering quality across the organization and prevented the formation of low-quality team silos.
- **Microsoft's Model:** Single hiring manager held ultimate veto/offer power; headcount was tied to organizational prestige and project deadlines, creating structural pressure to lower the bar. - **Island Evolution:** Sub-bar hires quickly became interviewers and managers, accelerating the drop in quality without any outside pressure dampening the effect. - **Google's Model:** Cross-functional committees reviewed candidate dossiers without team-specific pressure, requiring multi-tiered consensus up to executive sign-off. - **Hiring for the Company:** Candidates entered a general engineering pool rather than a siloed team, maintaining a uniform hiring bar across Google.
[18:04](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=1084s) Interview Preparation, Whiteboard Dry Runs, and Jazz
Preparation is crucial for interviewers, yet companies frequently throw engineers into interview loops with zero guidance. Lettvin recounts the first time he ever conducted an interview at Microsoft: he was informed that very morning that he had been assigned an interview loop with Darrell Richmond, an engineering veteran who had been at Microsoft since roughly 1985 and had authored core platform systems. Having only experienced an interview as a candidate himself a couple of years prior, Lettvin had no idea what he was doing and spent 45 aimless minutes asking Richmond about his current work without gathering any actionable evaluation data. His manager responded in classic Microsoft fashion by forwarding Lettvin's useless feedback to the entire team with an admonition: “don't do this.”
In contrast, Google provided comprehensive four- to five-hour interview training classes that taught engineers how to run structured sessions: 1. Ten minutes for introductions and setting context. 2. Forty minutes focused on solving an in-depth technical problem. 3. Ten minutes reserved for candidate questions and wrap-up.
Beyond formal classes, Lettvin advises engineers to conduct three or four whiteboard dry runs with colleagues to build confidence, pointing out that novice interviewers often feel far more nervous than the candidates they are evaluating due to the pressure to know all the answers. However, preparation must also be balanced with adaptability, because interviews rarely follow a pristine script. As Lettvin explains: “So yeah so be prepared but it's like jazz right like unexpected things are going to happen shits going to go sideways whatever part of being prepared is just knowing that that's going to happen and make it work anyway.” If a candidate reveals they have seen a question before, an interviewer must remain calm and pivot immediately to a backup scenario.
[20:49](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=1249s) Improvising, Making Mistakes, and the Slate Model
Interviewers must be ready to adapt to whatever happens in the room. If a candidate is reluctant to write code on the whiteboard—or hesitant to write code at all for whatever reason—the interviewer needs to improvise rather than get stuck. The best moments often occur when a candidate explores an entirely unexpected path on a favorite question and discovers something new. Moishe Lettvin highlights this openness as essential, declaring: “be prepared to be taught by people you're interviewing it's that's kind of the best-case scenario and make mistakes”. Just like other iterative practices at Etsy, interviewing is a learned skill where beginners inevitably struggle and require practice to improve.
```mermaid flowchart TD accTitle: Google Interview Slate Composition accDescr: Diagram showing how calibrated and new interviewers are paired to protect decision data while training new staff. subgraph Slate["Interview Slate Composition"] A["3 Calibrated Interviewers\n(>25 interviews completed)"] B["2 Inexperienced 'Newbie' Interviewers\n(In training / practice)"] end Slate --> C{"Evaluation Process"} C -->|"If newbie makes a mistake"| D["Discard newbie's data"] C -->|"Solid signal from veterans"| E["Sufficient data remains\nto make hiring decision"] ```
To solve the beginner problem, Google structured interview slates with an intentional balance of experience levels. They would typically staff three "calibrated" interviewers—defined as engineers who had conducted more than 25 interviews—alongside a couple of novices. If the inexperienced interviewers made a mess of their sessions, the hiring committee could simply discard their data and still retain three solid, calibrated data points to make an informed decision. Without continually introducing new people into that pipeline, a company eventually exhausts its pool of experienced interviewers. Despite the leeway given to beginners, Lettvin emphasizes one non-negotiable standard: “the only mistake that I would caution against the only rule is don't be a dick”.
[22:20](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=1340s) Every Interview is a Conversation, Not a Gauntlet
An interview should never be treated as an obstacle course or a stress test to see if a candidate can survive. It is fundamentally a conversation between two human beings, and while conversations can have objectives, intimidation has no place in them. Lettvin stresses that the candidate experience takes precedence over data collection: “the primary goal of this conversation you're having with the with the candidate is make them happy right that overrides everything else”. Risking a candidate's emotional well-being just to extract an elusive technical signal is fundamentally doing the job wrong.
When Lettvin asks the audience to raise their hands if they hold a positive impression of the Google interview process, only about one and a half hands go up. He points out the tragic irony: making candidates happy was Google's official goal as well. Yet a single encounter with an arrogant interviewer leads to a public blog post that circulates across the entire internet, instantly erasing the goodwill of 50 positive experiences. Beyond online reputation, treating people well carries real-world consequences: even candidates who do not get hired have friends from university who might be future hires, or they might be active buyers and sellers on Etsy. Making the experience respectful and pleasant simply makes the world a better place.
[24:35](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=1475s) Core Conversational Goals: Evaluation, Culture, and Generosity
The interview conversation revolves around two core questions. For the interviewer, the question is: "Would I love to work with this person?" Because one's colleagues are the primary indicator of job satisfaction, interviewing is an employee's direct chance to shape their own workplace. Interviewers should look for candidates who raise the bar—people who can make the company qualitatively better, teach others, be taught, and inspire those around them. Lettvin notes, “in my ideal world I want to be surrounded by people who are smarter than me right like that then that's that's why I'm here”. The second question belongs to the candidate: "Would you love to work at this company?" Interviewers must embody and communicate the culture; Lettvin recalls that when interviewing at Etsy, beyond the famous "Etsy school" (hacker school / Crafty and engineering culture) aspects, it was the genuine humanity and enthusiasm of the staff that won him over.
Lettvin describes the best interviews as fundamentally "generous," meaning interviewers show up well-prepared, keep pace with the candidate, ask thoughtful follow-ups, and actively work hard for the candidate rather than expecting the candidate to do all the work:
- **Graph and Trie Traversals (Lettvin as Interviewer):** In a Boggle-style word search question involving graph traversal and a Trie dictionary, a candidate finished the standard problem in roughly 8 minutes. When pushed further, the candidate demonstrated how to traverse the letter graph and the Trie simultaneously in parallel by passing both the Trie node and graph node into a recursive function—a solution Lettvin had never thought of. - **Reservoir Sampling (Lettvin as Candidate at Google):** Tasked with drawing a uniform random sample of size $k$ (e.g., 10 items) from a streaming data source of unknown length, the interviewer skillfully dropped hints that guided Lettvin to the constant-time reservoir sampling algorithm (picking an integer between 0 and total seen, placing the item if the value falls under the threshold). The elegant mathematical convergence made him leave the interview feeling accomplished. - **K-Means Clustering (Lettvin as Candidate at Etsy):** Josh Attenberg used an interview problem to teach Lettvin the mechanics of $k$-means clustering from scratch, turning a technical assessment into a rewarding educational exchange.
[29:24](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=1764s) The Onion Model of Technical Questions
A crucial counterweight to conversational generosity is strict time management. Interviewers frequently encounter candidates who talk excessively or diverge into subjects that are not germane to the evaluation; reigning in tangents requires deliberate practice. When it comes to designing technical interview problems, Lettvin argues that the best questions function like onions: they feature an accessible, naive implementation on the outer surface that candidates can peel back layer by layer into richer, more complex algorithmic challenges.
To illustrate, Lettvin recalls his very first technical interview at Microsoft around 1994. He showed up terrified, wearing a suit for probably the second time in his life, and had to endure five minutes of excruciating small talk about Seattle and his background before the interviewer gave him an engineering problem: write a program to draw a circle. Relieved to escape the small talk and write actual code, Lettvin implemented a naive trigonometric approach, looping from $0$ to $2\pi$ and multiplying the sine and cosine by the radius. Seeing that baseline working solution, the interviewer stripped away the first layer of the onion with the prompt: "Alright, that sucks, speed it up."
[31:08](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=1868s) The Multi-Layered Interview Problem: Bresenham's Algorithm and Conway's Game of Life
Continuing his anecdote about drawing a circle in a Microsoft interview in 1994, Moishe Lettvin walks through the progressive layers of optimization that make an interview question great. The first intuitive step after the basic trigonometry approach is exploiting eight-way symmetry: because circles are symmetrical, you only need to compute one octant (from $0$ to $\frac{\pi}{4}$). While clever, this approach still relied on trigonometric functions, which Moishe notes were notoriously slow in 1994. The interviewer then nudged him to remove sine and cosine entirely. Moishe switched to using square roots to solve for $x$ given $y$ or $y$ given $x$, which still used eight-way symmetry and had the added benefit of eliminating visual gaps between pixels. With fifteen minutes remaining on the clock, the interviewer pushed the boundary one final time: *"No square root."*
As Moishe puts it, that was the moment when “shit gets real.” Guided by the interviewer's carefully calibrated hints, Moishe managed to implement Bresenham’s circle algorithm, an integer-only method that tracks an error accumulator as it steps across the $x$ direction to decide the next pixel. With just three minutes left, the interviewer congratulated him, noting that he had just written the circle-drawing algorithm used in Windows. Moishe highlights this as the gold standard of interviewing: rather than an adversarial shotgun approach firing disconnected trivialities (Question A, Question B, Question C), the entire hour was spent deeply immersed in a single problem, collaborating and peeling back layers. He mentions other problems with that same property, such as Boggle or Conway's Game of Life. A problem like Conway's Game of Life serves as a clean FizzBuzz-style smoke test for basic programming ability, but then opens into forty years of literature and countless paths for optimization, from parallelization to algorithms scaling strictly with the number of living cells ($O(\text{living cells})$).
```mermaid flowchart TD accTitle: Progressive Circle Algorithm Optimizations accDescr: Four stages of the interview problem starting from naive trigonometry to Bresenham's integer algorithm.
subgraph CircleOpt["Progression of Circle Drawing Approaches"] A["Naive Trigonometry<br/>(sin & cos calculations)"] B["8-Way Symmetry<br/>(Compute 1/8th octant: 0 to pi/4)"] C["Algebraic / Square Root<br/>(Solves gaps, avoids trig calls)"] D["Bresenham's Algorithm<br/>(Integer error accumulator, no sqrt)"]
A --> B B --> C C --> D end ```
[33:46](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=2026s) Bandwidth, Artifice, and Eliminating Noise
Moishe advises interviewers to maximize communication bandwidth: prefer in-person interviews over video conferencing, video conferencing over phone calls, and whiteboards or scratchpads over purely verbal dialogue. People have different comfort zones; while some excel at a whiteboard, others need a keyboard and monitor. Recalling his very first Microsoft interview, Moishe shares that he was visibly anxious all day until one interviewer recognized his state, handed him a notebook with a problem, and simply walked out of the office to leave him alone to code. That solitary time provided the exact breathing room he needed to calm down and perform.
``` High Bandwidth ▲ In-person interviews │ Video conferencing (VC) Low Bandwidth ▼ Audio-only phone calls ```
Interviewing carries an inherent, unavoidable artificiality, because acting as an interviewee in an artificial setting is fundamentally detached from performing well in an engineering role day-to-day. As Moishe emphasizes:
> “Interviewing is a different skill than being a great engineer or whatever all you're looking for.”
Because of that gulf, interviewers must focus on underlying indicators that transcend the medium: can this candidate learn, are they creative, and are they enjoyable to work with? To maximize true signal and reduce noise caused by performance anxiety: - Share sample interview questions in advance and demystify the interview format. - Explicitly encourage candidates to practice on a whiteboard if that is how they will be evaluated. - Reject secrecy: Moishe stresses that he would far rather hire a candidate who initially struggled at interviewing but practiced enough to clear the bar than pass on top talent simply because they did not know what to expect. - Remind candidates upfront that the interviewer's role is not to trip them up, but to actively help them showcase their best work.
[36:30](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=2190s) Questioning the Standard Model and Final Advice
Reflecting on the prevailing industry standard—a grueling gauntlet of five or six one-hour sessions in a single day, followed by collating and evaluating the notes—Moishe acknowledges this has become the dominant playbook across major tech companies like Google. Yet he cautions against assuming it is optimal:
> “This general milieu of like five or six interviewers in a day, an hour each, asking some questions, collating that data, that might be the best we can do... it might be the best, but it probably isn't, and I hope it isn't.”
He suggests the tech industry may be trapped at a local maximum. As a promising departure from this rigid structure, he points to Hacker School (now Recurse Center), where observing candidates across two months of intensive, hands-on programming provides far richer signal than any one-day loop ever could. He encourages Etsy to leverage its unique culture and values to experiment with hiring models outside traditional boundaries.
Moishe closes his presentation with two core rules of thumb: - “Have fun; it is very fun, except when it isn't.” - “Always be learning, and don't get stale questions.”
[38:02](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=2282s) Q&A: Classifying Feedback and Balancing False Positives vs. False Negatives
The session opens up to questions from the Etsy audience:
- **Automating Feedback with Machine Learning:** An audience member asks whether Google ever trained natural language classifiers on interviewers' written feedback to extract higher-fidelity signals than numerical scores. Moishe clarifies that he never saw such a system deployed at Google, though he suspects someone explored it. Google intentionally protected human judgment on the hiring committee. Because written reviews vary dramatically based on interviewer personalities—some write harsh critiques, while others are unfailingly polite—committee members rely on their institutional memory to contextualize individual feedback styles, a nuance that machine learning models would struggle to replicate without huge per-interviewer datasets.
- **Risking a Bad Hire vs. Missing Great Talent:** Another attendee asks Moishe for his personal stance on the classic big-tech dilemma: is it better to reject a potentially strong candidate to prevent a bad hire (Google's conservative bias toward false negatives), or should companies take risks on unpolished candidates? Moishe confesses he is torn. Reflecting on his first job in 1988, he recalls how his boss took a gamble on him based on quirky signals:
> “Peter showed me the 6502 assembler game he'd written, and that was awesome, and you could type really fast, so I figured if half the stuff you wrote worked, you were set.”
While that leniency launched his own career, Moishe warns that bad hires trigger severe cascading damage across an engineering organization. He points to the ultimate litmus test for evaluating potential team members: *Would I love to be carrying the on-call pager when this person checks in an entire week's worth of code?*
- **Hiring for the Company vs. Hiring for the Team:** An audience member contrasts Microsoft’s team-specific hiring model with Google's centralized hiring committees. At Microsoft, interviewers had to assess whether they would take the candidate directly onto their own team; you were explicitly forbidden from saying, "I wouldn't want them on my team, but hire them for another group." The attendee raises the concern that Google’s centralized committee model creates a risk of settling for candidates who seem generically "good enough for Google," even if no individual team feels enthusiastic about them.
[41:38](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=2498s) Company-Wide Standards and Domain-Specific Hiring
An audience member asks how to avoid a situation where an interviewer gives a pass to someone who might not meet the bar for their own team. Moishe Lettvin explains that this scenario occasionally surfaced during Google's hiring committee reviews, where each member wrote feedback evaluating the interviewers' write-ups. Whenever someone expressed the sentiment that a candidate was acceptable for another team but not their own, the committee shut it down immediately:
> “It's like if you don't think this person is good enough for your own team then we're not going to hire him and I think like that was just a cultural thing and and and I think like again there's no hard and fast rules but the the culture was such that you knew you were hiring for Google and for somebody who could because you don't even know what team they're going to end up on in hiring committee right like the the allocation process is completely separate so they might very well end up on your team.”
Turning to the question of domain-specific hiring, Lettvin explains that the candidate must first pass the general baseline bar. Once that minimum threshold is cleared, specialized expertise is assessed simply by assigning one person on the interview loop to test that particular technical domain, while the rest conduct standard interviews.
[43:06](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=2586s) Personality Fit vs. Technical Chops and the "No Jerks" Rule
When asked how to weigh personality fit against technical prowess—especially at an organization like Etsy—Lettvin explains that he favors personality when comparing specific technical skillsets against soft skills, as long as the candidate demonstrates high learning agility. If an engineer has proven mastery in JavaScript or Java, the lack of PHP experience is negligible because they can pick it up on the job. However, if the candidate lacks the fundamental capacity to ever reach the required level of technical expertise, technical ability trumps:
> “Like you can't be weak personality wise and strong technically or weak technically in strong personality wise like I think at Etsy it's SuperDuper important to have both.”
An audience member follows up with the widely discussed industry rule: never hire jerks, no matter how brilliant they are. Lettvin wholeheartedly endorses this philosophy, recounting an interview he conducted at Google with a domain expert from ATI who specialized in CUDA (GPU parallel processing). While undeniably brilliant, the candidate treated Lettvin disrespectfully throughout the session. In his written evaluation, Lettvin recorded that the candidate was exceptionally smart, but also a complete jerk. By coincidence, Lettvin sat on the hiring committee when the packet came up, and another committee member who had previously worked with the candidate at ATI chimed in to confirm that the candidate was indeed impossible to work with. Lettvin underlines the destructive cost of keeping toxic individuals:
> “So I it's just it creates a toxic environment when you're working with [jerks] like it makes you not want to go to work and it's I don't care how smart they are it's it drags everybody down.”
[45:34](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=2734s) Real-Time Interview Loop Coordination: Microsoft vs. Google
Addressing a question about whether interviewers should communicate between sessions or maintain a blind slate, Lettvin contrasts the mechanisms at Microsoft and Google:
```mermaid flowchart TD accTitle: Comparison of Interviewer Communication Models accDescr: Contrasts Microsoft running email chain model against Google blind independent writeup model.
subgraph MS["Microsoft: Real-Time Loop"] M1["Interviewer finishes"] --> M2["Sends Hire/No-Hire email"] M2 --> M3["Later interviewers read chain"] M3 -->|"Pro: Targets weak areas"| M4["Can re-test doubts"] M3 -->|"Con: Senior voice anchors"| M5["Amplifies evaluation bias"] end
subgraph GOOG["Google: Independent Evaluations"] G1["Interviewer finishes"] --> G2["Fills in-room paper slip with questions asked"] G2 --> G3["Submits detailed written feedback into system"] G3 --> G4["Permitted to talk only after all feedback is logged"] end ```
At Microsoft, an ongoing email chain tracked the candidate through the loop. As soon as an interview concluded, the interviewer sent an email stating "hire" or "no hire" with their reasoning. While the second interviewer in line might not see it, the one after would. Although this allowed subsequent interviewers to probe areas where the candidate had struggled, it carried a severe drawback: if an early, senior, or highly respected interviewer sent an opinion, downstream interviewers felt pressured not to contradict them, either poisoning or blessing the packet.
Google solved this by strictly prohibiting interviewers from discussing candidates until each had submitted their independent written feedback. To prevent duplicate technical exercises without causing bias, Google used a physical paper slip kept inside the interview room where each interviewer wrote down what problem they had posed and noted any specific area that warranted deeper probing. Lettvin strongly favors this approach, reasoning that it is far better to protect against cognitive bias than to risk contaminating the loop.
[47:39](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=2859s) Shadowing, Paired Interviews, and Etsy's Interview Process
When asked about running two-on-one group interviews, Lettvin notes that Google’s training protocol required trainees to complete two shadow sessions—quietly observing an experienced interviewer—followed by one reverse-shadow interview, where an experienced interviewer silently observed them. Under this "contract," the observer stays quiet and never intervenes. While he worries that having two active interviewers might make a candidate feel ganged up on, Lettvin believes paired interviews can be successful if the interviewers actively manage and dampen that power dynamic.
A recruiter from Etsy asks Lettvin what he would change about Etsy's interview loop based on his own experience going through it. Lettvin admits he was surprised by how little actual programming he was asked to do. However, he praises the overall experience:
> “So so that made me super happy and was actually a big part of the reason I came here yeah.”
He emphasizes that Etsy treated him as a human being, placing genuine value on communication skills and whether people could hold a natural conversation.
[49:31](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=2971s) Packet Calibration and What Makes Strong Feedback Data
Returning to Google's practice of pairing newer interviewers with calibrated ones, Lettvin clarifies that hiring committees do not discard new interviewers' write-ups outright. Instead, they calibrate how much weight they assign:
> “So if you've got three calibrated interviews where they've written really good feedback on the basket of questions that's enough and if two of them they might still give good scores but they're just thin on details and stuff like that it's you can still get that packet through so it's not like you just take them out and don't factor them into the score generally although that can happen for extreme cases it's more like it's just you just don't place as much weight on [them].”
Asked what high-quality feedback data actually contains, Lettvin outlines the components that hiring committees look for in strong notes: - Well-chosen problems that test algorithmic thinking and coding implementation, alongside systems design questions for senior roles. - Detailed step-by-step notes tracking how the candidate attacked the problem, where they required hints, and where they solved steps independently. - Benchmarking the candidate’s performance against how an established Google engineer would handle the identical problem. - A well-argued opinion that flows logically from the documented evidence (e.g., recognizing that an applicant translated an abstract problem into a graph traversal, wrote clean production code, and flagged suboptimal variable names chosen for whiteboard speed).
[52:02](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=3122s) Etsy's Interview Process and Feedback Volume
An audience member asks about the mechanics of the interview debrief at Etsy, mentioning they have only participated once on their very first day and want to know whether people meet in person or submit written evaluations. Another attendee clarifies the current internal process: teams typically hold a pre-huddle before the interview loop begins to coordinate who will cover which technical areas. After the sessions conclude, interviewers submit their independent evaluations into Jobvite first—specifically designed to prevent the kind of information cascades Moishe Lettvin previously warned against—before gathering for a post-huddle to discuss their impressions, after which the hiring manager and approval chain make the decision.
```mermaid flowchart TD accTitle: Etsy Interview Debrief Process accDescr: Flowchart showing the stages from the initial pre-huddle to final candidate approval. subgraph Preparation["Pre-Interview"] A["Pre-Huddle Meeting<br>(Divide technical focus areas)"] --> B["Interview Sessions"] end subgraph Evaluation["Post-Interview"] B --> C["Submit Independent Feedback<br>(Jobvite to prevent information cascades)"] C --> D["Post-Huddle Meeting<br>(Discuss impressions)"] D --> E["Hiring Manager Decision & Approvals"] end ```
The discussion shifts to Lettvin's earlier remark that he wrote two to three pages of feedback per candidate at Google. An engineer notes that Jobvite's feedback box is only about four lines high and overflows after a single paragraph, asking whether the field should be enlarged or if Lettvin was exaggerating. Lettvin confirms he was not exaggerating:
> “I mean but so half of that is probably taken up with code samples but like if you were to print it out double-spaced it would probably I think two pages are pretty typical for a feedback.”
He clarifies that if everyone attending the wrap-up meeting has directly interviewed the candidate, concise bullet points can suffice as personal notes for the meeting. At Google, however, the hiring committee had zero direct exposure to the applicant, requiring exhaustive, self-contained documentation. Lettvin reiterates his broader philosophy:
> “I am an advocate of more data more persistence like writing more up I think it's worthwhile.”
[54:16](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=3256s) Candidate Follow-ups and the Art of Giving Hints
An audience member notes that selling the company to candidates is a key component of the process and asks whether Google or Microsoft conduct retrospective follow-ups with candidates who decline offers or get turned down to learn if something in the interview went awry. Lettvin confirms that Google actively follows up with rejected candidates for two distinct reasons: first, to keep communication lines open with candidates they hope to re-interview in a year, and second, to gather feedback on improving the candidate experience. He notes that the pipeline improved dramatically over his tenure; when he first joined Google, decisions took literal months, whereas it eventually came down to a couple of weeks, though he notes that is still far from ideal.
Another engineer asks for actionable advice on how to guide candidates who stumble, particularly on broad architectural questions (such as walking through what happens during a web request) or during live coding, without handing them the solution outright:
> “Right I mean I think part of it is having questions that lend themselves to hints right like don't have questions with a yes or no answer or a vocab question with the definition.”
Lettvin explains that learning the right dosage of guidance is primarily a matter of repetitions: the more times an interviewer runs a specific problem, the better calibrated they become at nudging candidates along without giving away the explicit answer.
[56:33](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=3393s) Assessing Culture Fit and Personality
A participant describes a tactic used at their previous company to gauge personality and culture fit by asking deliberate curveball questions, such as asking candidates about the most hilarious prank they pulled or their favorite midnight snack. They ask whether Microsoft or Google used equivalent non-technical questions to assess fit. Lettvin responds that neither company really used that approach, but shares his favorite question from his own interview at Etsy, posed to him by Josh:
> “The first question I was asked here by Josh was what's your favorite algorithm which I think is a phenomenal question like in terms of culture fit like do you have one right and then like in terms of technical ability can you describe it right.”
He contrasts this with standard behavioral interview prompts (such as "tell me about a time when X happened"), pointing out that candidates routinely read interview prep guides and rehearse canned answers. A prompt like naming a favorite algorithm catches candidates off-guard while evaluating both genuine passion for the craft and the technical clarity required to explain it on the spot.
[57:58](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=3478s) Interview Prep Books and Offer Turnaround Time
An audience member asks whether any of the commercial interview preparation books on the market are worth reading, or if they are all poor quality. Lettvin recalls reading *Programming Interviews Exposed* and finding that their treatment of the circle-drawing question stopped at an elementary square-root solution rather than exploring the deeper, mathematically elegant algorithmic solutions, which turned him off from the book. However, he strongly vouches for Gayle Laakmann McDowell (author of *Cracking the Coding Interview*):
> “Gale Lachman wrote this book called programming actually I don't even know what's called but anyway I trust her she's super duper smart and she's put a lot of time into this and she was on hiring committee at Google so I suspect that's probably a very good book.”
Lettvin emphasizes that anything helping candidates demystify the interview format reduces unproductive anxiety and benefits both sides of the table.
Finally, an IRC remote viewer asks for Lettvin’s opinion on ideal post-interview turnaround times for extending offers. Lettvin explains the unavoidable organizational tradeoff between speed and consensus: - **Single decision-maker speed:** At Microsoft, Lettvin received an offer the very day he interviewed because his final conversation was directly with the hiring manager, but that model depends entirely on trusting individual managers without checks and balances. - **Committee review latency:** Shifting decisions to an independent hiring committee improves consistency and reduces bias, but inherently injects substantial operational delay.
Lettvin emphasizes how agonizing that lag is from the applicant's side, sharing his own frustrating experience interviewing at Google: he interviewed in April and did not receive an offer until late June, enduring weekly calls from his recruiter reporting that his packet was sitting with yet another committee. He concludes that while balancing thoroughness with velocity is difficult, companies should make their turnaround as fast as their internal safeguards allow.
## Notable quotes > [36:30](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=2190s) This general milieu of like five or six interviewers in a day, an hour each, asking some questions, collating that data, that might be the best we can do... it might be the best, but it probably isn't, and I hope it isn't. — Moishe Lettvin
> [12:11](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=731s) I feel like interviewing is more like literary criticism than it is like statistics — Moishe Lettvin
> Interviewing is a different skill than being a great engineer or whatever all you're looking for. — Moishe Lettvin
> [20:49](https://www.youtube.com/watch?v=r8RxkpUvxK0&t=1249s) the only mistake that I would caution against the only rule is don't be a dick — Moishe Lettvin
## Mentioned ATI, Bob Day, Boggle, Bresenham's circle algorithm, Conway's Game of Life, Cracking the Coding Interview, CUDA, Darrell Richmond, Etsy, EverQuest, Gayle Laakmann McDowell, Google, Google Talk, Hacker School, Jobvite, Josh Attenberg, Larry Page, Laszlo Bock, Microsoft, Moishe Lettvin, Philip Zimmermann, Programming Interviews Exposed, Recurse Center, Stephen Jay Gould, Steve Yegge, The Mismeasure of Man
[Source](https://www.youtube.com/watch?v=r8RxkpUvxK0)