分布式通知架构拆分载荷与状态表防范第三方通道重试与丢包风险

Scott Shi

因外部通道缺乏两阶段提交导致端到端精确一次投递在工程上不可行，高并发通知系统必须在持久化后立即返回异步回执并通过关系型状态机驱动重试。

AI-assisted summary of [Chinese] Mock Interview: design notification system — original source. Check the original for full context.

Source published: 2020-11-29

Key ideas

For a platform scaling to 200 million daily active users averaging 5 alerts per day, notification ingestion reaches an average of 11,600 write QPS, necessitating horizontal scaling and asynchronous decoupling from third-party vendor networks. (33:23)
Strict end-to-end exactly-once delivery across external gateways like SendGrid or Twilio is practically infeasible due to the absence of distributed two-phase commit protocols, making durable at-least-once delivery with receiver deduplication the standard architectural pattern. (24:26)
Storage is bifurcated into high-throughput NoSQL (DynamoDB) for immutable raw message bodies and relational SQL for transactional delivery lifecycle state tracking and stuck-message detection. (72:40)
To prevent HTTP socket timeouts and message loss during massive broadcasts, the ingestion API immediately commits payloads to durable storage and returns an asynchronous tracking receipt ID before worker pools handle downstream dispatch. (67:56)

Detailed notes

## Background Published on YouTube by Scott Shi on November 29, 2020, this video records a comprehensive mock system design interview. The description notes that the session covers designing a notification system and provides a link to a collaborative working document used during the exercise.

## TL;DR The interview begins with a deep dive into the candidate's past payment storage migration project at Uber Money, followed by coaching on framing technical challenges around business impact. The main exercise entails designing an enterprise-scale notification platform supporting email, SMS, mobile push, and phone calls for 200 million daily active users. Together, the interviewer and candidate establish an asynchronous architecture that decouples immutable message payloads in DynamoDB from transactional lifecycle state tracking in a relational database. The discussion thoroughly analyzes delivery semantics, explaining why third-party exactly-once delivery is impractical and demonstrating how adapter layers normalize external provider errors and handle retries.

## Atmosphere & style The session has an authentic, collegial, and rigorous engineering tone that balances high-stakes interview realism with practical senior-level mentorship. Scott seamlessly transitions between probing interviewer and encouraging coach, interjecting with good humor to correct arithmetic slips and steer the candidate away from unnecessary edge over-engineering. It feels like an illuminating behind-the-scenes peer review, highlighting how architectural trade-offs, state machines, and distributed failure modes are actually evaluated in top-tier tech interviews.

## Bulletin - [33:23](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=2003s) For a platform scaling to 200 million daily active users averaging 5 alerts per day, notification ingestion reaches an average of 11,600 write QPS, necessitating horizontal scaling and asynchronous decoupling from third-party vendor networks. - [24:26](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=1466s) Strict end-to-end exactly-once delivery across external gateways like SendGrid or Twilio is practically infeasible due to the absence of distributed two-phase commit protocols, making durable at-least-once delivery with receiver deduplication the standard architectural pattern. - [01:12:40](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=4360s) Storage is bifurcated into high-throughput NoSQL (DynamoDB) for immutable raw message bodies and relational SQL for transactional delivery lifecycle state tracking and stuck-message detection. - [01:07:56](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=4076s) To prevent HTTP socket timeouts and message loss during massive broadcasts, the ingestion API immediately commits payloads to durable storage and returns an asynchronous tracking receipt ID before worker pools handle downstream dispatch.

## Key takeaways - [16:35](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=995s) Senior candidates should frame past engineering projects around architectural ownership, explicit trade-offs, and measurable business impact rather than low-level implementation details or technical jargon. - [14:01](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=841s) Leveraging inherent business TTLs allows zero-downtime database migrations without heavy batch backfill pipelines by letting old records naturally expire after cutting off writes. - [24:26](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=1466s) True end-to-end exactly-once delivery across autonomous third-party APIs is an unsolved distributed dilemma; external network timeouts prevent definitive state verification without unsupported 2PC/XA protocols. - [29:58](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=1798s) A hybrid push-pull architecture is essential: connected clients receive real-time push events, whereas backgrounded or disconnected clients pull accumulated unread notifications via RPC/gRPC upon reconnecting. - [29:58](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=1798s) Global notification ordering can be relaxed, allowing traffic to be segmented into urgent transactional alerts and delay-tolerant low-priority engagement updates. - [37:09](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=2229s) Payload pipelines should never transmit heavy binary multimedia directly; messages must carry lightweight identifiers or HTML references that resolve assets lazily through CDNs. - [51:20](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=3080s) Internal infrastructure services operating within a private corporate network do not require public edge security overhead such as external SSL termination or public client authentication tokens. - [56:48](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=3408s) Asynchronous worker pools are critical to decouple the ingestion layer from erratic downstream provider latency and rate-limiting bottlenecks. - [01:07:56](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=4076s) Issuing an asynchronous tracking receipt token must occur strictly after database persistence; acknowledging requests beforehand creates data-loss vulnerabilities if nodes crash mid-flight. - [01:12:40](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=4360s) Immutable notification message bodies should be separated from mutable delivery states to keep large payload blobs out of transactional relational tables. - [01:18:16](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=4696s) A relational metadata schema tracking recipient delivery states with an updated timestamp enables reconciliation jobs to detect and retry messages stranded in non-terminal states. - [01:32:00](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=5520s) Downstream provider adapters must translate vendor-specific errors into standardized internal status codes to prevent coupling core retry engines to provider-specific nuances. - [01:32:00](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=5520s) Multi-recipient batch broadcasts must record granular delivery states per recipient to avoid re-dispatching alerts to users who already received them when partial batch failures occur.

## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=0s) Pre-Interview Warm-Up and Distributed Systems Concepts

Before the interview formally gets underway, the candidate spends several minutes reviewing distributed systems theory and setting up technical logistics. Muttering through fundamental database and concurrency concepts, the candidate mentally revisits linearizability, master-slave replication, consensus protocols, snapshot isolation, and compare-and-set operations. The discussion touches upon whether serializability or snapshot isolation strictly guarantees linearizability, reinforcing the rigor needed for high-stakes distributed architecture. After troubleshooting screen sharing and access permissions for the interview document, the interviewer and candidate confirm audio and visual feeds, ready to begin the session.

- Key theoretical topics reviewed: linearizability, compare-and-set (CAS), consensus requirements, read/write snapshot isolation, and serializability guarantees. - Interview logistics: granting Google Doc permissions and sharing the screen for the collaborative whiteboarding exercise.

---

[10:20](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=620s) Past Project Deep Dive: Uber Money Payment Storage Migration

The interviewer kicks off the session by asking the candidate to spend roughly five minutes detailing the most technically challenging and impactful project from their professional career, specifically excluding school assignments. The candidate introduces their background on the Uber Money team, handling payments infrastructure. Their most technically demanding project over the past year was migrating the payments "grant storage" away from a legacy standalone DynamoDB cluster into Uber's centralized LedgerStore Gateway. This engineering initiative took about three months to execute and required strict SLA negotiations with downstream platform teams alongside a zero-downtime, lossless data migration pipeline.

When pressed on what constituted the hardest engineering challenge of the migration, the candidate explains the delicate fallback and ordering strategies needed during dual writing and reading. To guarantee zero data loss, the team first routed writes to the new LedgerStore Gateway before falling back to the legacy grant store, subsequently reversing the operational precedence once consistency was verified. Extensive online validation logic had to be authored to continuously reconcile records and confirm that data was identical across both storage engines before retiring the old database.

```mermaid flowchart LR accTitle: Dual-Write Migration Architecture accDescr: Migration pipeline showing shadow writes, validation checks, and cutover between DynamoDB and LedgerStore Gateway.

subgraph ClientLayer["Write Traffic"] A["Payment Ingestion Service"] end

subgraph StorageLayer["Dual-Write & Fallback"] B["LedgerStore Gateway (New Primary)"] C["Legacy DynamoDB (Grant Store)"] D{"Validation Engine & Shadow Reconciler"} end

A -->|"1. Primary Write"| B A -->|"2. Fallback / Shadow Write"| C B -.-> D C -.-> D ```

- System components migrated: standalone DynamoDB to internal LedgerStore Gateway. - Project timeline: approximately 3 months. - Core technical hurdle: lossless dual-write sequencing, dynamic fallback routing, and real-time shadow validation checks.

---

[14:01](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=841s) Project Ownership and Handling Historical Data Expiration

The interviewer probes further into the candidate’s personal autonomy and architectural decision-making, asking whether they drove the effort independently or worked under heavy mentorship, and how historical records were migrated. The candidate clarifies that while they referenced prior partial migrations and consulted internal staff engineers on tricky edge cases, the day-to-day execution and design were their own responsibility.

Regarding historical data backfills, the candidate reveals an elegant operational shortcut: they did not need to run heavy batch backfill pipelines such as Apache Airflow or Piper jobs. The underlying payment grant records possessed a natural business expiration window of six days. By setting a hard timestamp cutoff, freezing writes to legacy storage, and letting the existing six-day TTL elapse, the historical data in the old DynamoDB instances naturally expired without requiring complex offline extract-transform-load (ETL) backfills. The interviewer acknowledges this approach as a practical, low-risk way to sidestep cumbersome backfill scripts.

- Role and ownership: authored core migration logic with guidance from staff engineers; leveraged architectural patterns from previous partial projects. - Historical data strategy: avoided custom batch/Airflow jobs because grant records carried a strict 6-day TTL. - Transition execution: writes to the legacy store were halted before the deadline, allowing old records to age out cleanly.

---

[16:35](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=995s) Interviewer Coaching: Articulating Scope and Business Impact

At this juncture, the interviewer steps out of character to deliver constructive feedback on how senior candidates should pitch technical war stories in interviews. The interviewer emphasizes that interviewers listen closely for independent problem-solving skills, engineering maturity, and crisp articulation. Rather than framing work as merely implementing a task or building an Airflow pipeline that someone else specified, candidates must claim ownership of the end-to-end design, explaining *why* the architectural choices were made and contrasting them with rejected alternatives.

Furthermore, the interviewer cautions against getting bogged down in low-level technical jargon—humorously labeled as "stop words" or compiler-level minutiae like garbage collection details—unless they are directly tied to real business value. Candidates need to connect their technical feats to organizational scale: replacing legacy dispatch infrastructure, deprecating fragile microservices, improving uptime, or safeguarding multi-million dollar payment flows. Framing technical challenges around measurable business impact elevates the candidate from a routine implementer to a high-caliber system architect.

- Core pitch guidance: shift framing from "implementing someone else's flow" to owning system-level problem solving and architectural decisions. - Articulation tips: cut out irrelevant technical jargon ("stop words") and highlight high-level engineering tradeoffs. - Business value alignment: explicitly quantify system impact (e.g., replacing monolithic legacy systems, mitigating financial risk, slashing latency).

---

[20:29](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=1229s) System Design Prompt: Enterprise Notification Platform

The interview transitions into the primary system design exercise: designing a multi-tenant, enterprise-scale notification platform capable of handling diverse notification types across the company. The interviewer outlines the functional scope: the system must support mobile in-app push notifications (Apple APNs and Google FCM), email alerts via vendors like SendGrid, and SMS or voice messaging via providers like Twilio. Because downstream vendors can fail or rate-limit requests, robust third-party integrations with fallback mechanisms are mandatory.

Additionally, the interviewer establishes strict delivery semantics and operational visibility requirements. The platform must offer unified access—either via RESTful HTTP endpoints or Kafka message queues—for any internal upstream team. Upstream callers must be able to specify delivery guarantees, choosing between at-least-once, at-most-once, and potentially exactly-once delivery. The interviewer points out that delivery guarantees carry serious legal and monetary weight, noting that "LinkedIn was fined millions of dollars because it sent promotional emails repeatedly" to users in 2014 and 2015. Finally, the system must feature business-centric observability: not just raw CPU and memory metrics, but live operational dashboards tracking in-flight notifications, delivery progress, and vendor-specific queue backlogs (such as flagging 100 email notifications stalled on SendGrid).

```mermaid flowchart TD accTitle: Notification Platform Scope Overview accDescr: Upstream clients communicating through a unified interface into delivery engines and vendor integrations.

subgraph Upstream["Internal Services"] U1["Order Service"] U2["Marketing Campaign"] U3["Payment Service"] end

subgraph Platform["Notification System"] API["Unified Interface (REST API / Kafka Topics)"] Engine["Notification Router & Delivery State Machine"] Dash["Business Dashboard (Backlog & Vendor Health)"] end

subgraph Vendors["Downstream Delivery Channels"] V1["Apple APNs / Google FCM (In-App Push)"] V2["SendGrid (Email)"] V3["Twilio (SMS / Voice)"] end

U1 --> API U2 --> API U3 --> API API --> Engine Engine -.-> Dash Engine --> V1 Engine --> V2 Engine --> V3 ```

- Supported channels: iOS/Android native push, email (SendGrid), SMS/phone calls (Twilio). - Ingestion interfaces: unified REST API endpoints and asynchronous Kafka message topics. - Delivery guarantees: configurable modes including at-least-once and at-most-once delivery. - Observability mandate: business dashboards displaying real-time message state, pipeline progress, and per-vendor bottleneck counts.

---

[24:26](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=1466s) Scoping Requirements and Clarifying Exactly-Once Semantics

The candidate assumes control of the whiteboard, structuring the problem into business use cases, operational constraints, API contracts, architectural topologies, and data models. Under the MVP scope, the candidate defines multi-channel delivery across mobile apps, email, and SMS, incorporating configurable delivery modes. When discussing delivery semantics, the candidate associates at-least-once with persistent retry queues and durable storage, whereas at-most-once represents a fire-and-forget mechanism where dropped packets bypass retries. For exactly-once, the candidate suggests upstream idempotency filtering at the load balancer layer to deduplicate identical client submissions before processing.

The interviewer steps in to clarify the subtle theoretical distinction between distributed streaming deduplication (such as Kafka’s internal exactly-once semantics) and end-to-end exactly-once delivery across uncoordinated third-party networks. In a notification platform, an internal worker may call an external vendor API like SendGrid or Twilio; if that third-party request experiences a network timeout or drops the response packet, the internal system cannot definitively know whether the vendor dispatched the message. True end-to-end exactly-once delivery across autonomous external systems is an unsolved two-phase commit (XA distributed transaction) dilemma. Because third-party vendors do not support two-phase commits, the interviewer advises treating strict end-to-end exactly-once guarantees as practically infeasible and suggests keeping it out of the core MVP requirements.

- Candidate's structural roadmap: use cases & MVP features $\rightarrow$ constraints $\rightarrow$ API contracts $\rightarrow$ architectural design $\rightarrow$ data storage schema. - Delivery semantics analysis: - At-least-once: backed by durable message stores and backoff retry queues. - At-most-once: lightweight fire-and-forget without persistence or retry overhead. - Exactly-once: constrained by third-party black-box API behavior, where network timeouts make distributed state verification impossible without 2PC/XA protocols. - Architectural decision: scope out rigid third-party exactly-once guarantees from the MVP, prioritizing robust deduplication and at-least-once delivery instead.

[29:58](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=1798s) Push vs. Pull Models, Recurring Alerts, and Latency Expectations

The candidate outlines core MVP and bonus capabilities, focusing on how notifications reach users depending on client state. A distinction is drawn between active and idle subscribers. When a user has their application open and active, notifications can be pushed directly in real time. However, when an app is closed, suspended on iOS, or otherwise disconnected, pushing may fail or get delayed. For these scenarios, the candidate proposes a pull model: once the user reopens the app or background app refresh triggers, the client polls the notification service via an RPC or gRPC endpoint to fetch accumulated unread alerts directly from the data store.

The discussion also touches on scheduling and recurring notifications, which are critical for marketing emails, calendar reminders, and SMS alerts. Moving into non-functional constraints and non-goals, the candidate emphasizes that strict real-time latency is not universally required. Traffic splits into priority tiers: while critical operational alerts must arrive promptly, low-priority engagement updates (such as routine Twitter activity feeds) frequently experience delays of several hours without degrading the core user experience. Furthermore, strict chronological ordering across distinct notifications does not need to be guaranteed by the messaging pipeline.

- **Push Mechanism:** Direct delivery to active, foregrounded client sessions. - **Pull Mechanism:** On-demand RPC/gRPC polling upon client relaunch or background task execution. - **Latency Tiers:** High-priority (urgent/transactional) delivered quickly; low-priority (feed updates, promotions) tolerated with multi-hour lag. - **Ordering:** Out-of-order delivery is acceptable across independent notifications.

---

[33:23](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=2003s) Scale Estimation: Sizing for 200 Million Daily Active Users

Noticing that throughput constraints have not yet been framed, the interviewer intervenes with concrete sizing requirements: "Let's assume that we are designing the notification system for Facebook... you have 200 million active users per day, and each user has five notifications per day on average." A lighthearted moment ensues as the candidate begins writing out the math and the interviewer gently points out a clerical error: "You're missing a zero."

$$\text{Total Daily Volume} = 200{,}000{,}000 \text{ users} \times 5 \frac{\text{notifications}}{\text{user}\cdot\text{day}} = 1{,}000{,}000{,}000 \text{ notifications/day}$$

$$\text{Average QPS} = \frac{1{,}000{,}000{,}000}{24 \times 3600 \text{ seconds}} \approx 11{,}574 \text{ requests/sec}$$

With average write traffic hovering near 11,600 QPS and peak traffic predictably surging much higher, the candidate deduces that notification ingestion and delivery cannot reside on a single host. The system demands horizontal scalability across the ingestion tier and storage sharding to handle write-heavy fan-outs without hitting host-level I/O bottlenecks. In addition to high availability and horizontal scalability, the design must prioritize flexibility to decouple internal callers from end-user channel protocols.

---

[37:09](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=2229s) Message Formats, Multimedia Payloads, and Channel Topics

The conversation pivots to how messages are structured and which communication channels the notification platform must accommodate. The candidate initially considers restricting the scope to pure plain-text payloads to simplify data transfer. The interviewer pushes back slightly, noting that modern notification systems—particularly for emails—frequently handle rich media: "If you're going to send emails, you're going to have images, right?... and you may have possibly video, but most likely you're going to be an HTML file like multimedia."

The candidate agrees to treat rich media as an embedded reference model. Rather than transmitting raw binary blobs across the notification ingestion pipeline, the message payload remains a clean text or HTML document containing reference identifiers (such as image or video IDs). Downstream client renderers resolve and hydrate these media assets lazily via CDNs. The interviewer and candidate agree on four core delivery channels that the system must support:

- **SMS:** Short, text-focused, latency-sensitive mobile carrier messages. - **Email:** Rich HTML formatting referencing externally hosted multimedia assets. - **Phone Calls:** Automated voice notifications delivering pre-recorded audio clips to keep voice-gateway integration straightforward. - **In-App:** Direct feed updates and real-time pop-ups targeting connected native applications.

---

[40:44](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=2444s) API Design: Topics, Templates, and Delivery Semantics

Moving into the API contract, the candidate drafts a set of endpoints centered on a pub/sub paradigm:

``` createTopic(topicId, topicName, serviceType, metadata) send(topicId, sendMode) subscribe(topicId, subMode) ```

The interviewer seeks clarification on terminology, asking whether "topic" implies a raw Apache Kafka topic. The candidate clarifies that here, a topic acts as a business-level notification campaign or reusable template. For instance, an internal marketing service running an ad campaign registers a template with a specific name, service type (`in-app`), priority flags, and security tokens. The interviewer synthesizes the concept: "Oh, I see. So you're referring to more like a notification template, right? Like an email template."

When pressed on what `sendMode` and `subMode` signify, the candidate explains that `sendMode` defines delivery semantics—specifically choosing between at-least-once, at-most-once, or exactly-once delivery guarantees depending on business criticality. The candidate concedes that `subMode` is still an early, exploratory concept intended to allow subscribers to specify delivery priority or filter constraints, but agrees that the foundational three APIs provide a workable base for topic registration, dispatch, and channel listening.

---

[45:33](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=2733s) Pub/Sub Modeling Debate and High-Level Architecture Walkthrough

The interviewer raises a structural design concern regarding how the candidate's pub/sub topic model handles targeted, multi-service routing: "I'm a little confused... could you please explain how your system is going to model the following scenario? Like Service A sends SMS to User 1, Service B sends email to User 2, and Service C sends phone calls to User 2?" The interviewer wants to know how individual user IDs and channel destinations map to generalized topics.

The candidate explains that internal producer services do not communicate directly with external providers or maintain user channel states themselves. Instead, internal services call the notification platform, which evaluates caller intent, retrieves message templates, and routes requests to appropriate sender adapters. The candidate sketches the end-to-end data flow: incoming traffic hits an external Load Balancer, reaches service workers that fetch metadata from a Metadata Store, places dispatches into intermediate temporary message queues, and hands jobs off to sender workers configured with specialized adapter layers for SMS, email, telephony, and mobile push.

```mermaid flowchart LR accTitle: Notification System High-Level Architecture accDescr: Flow diagram showing request routing from producer clients through a load balancer into metadata and temporary queue storage, finishing at downstream sender adapters.

subgraph Ingestion ["Ingestion Layer"] C["Producer Services"] --> LB["Load Balancer"] LB --> GW["Notification Gateway Service"] end

subgraph Storage ["Storage & Queueing"] GW --> MS[("Metadata Store\nTemplates & Configs")] GW --> QS[("Temp Storage / Queue\nPending Messages")] end

subgraph Dispatch ["Dispatch & Adapters"] QS --> SW["Sender Worker Engine"] SW --> AD_SMS["SMS Adapter"] SW --> AD_EM["Email Adapter"] SW --> AD_PH["Phone/Voice Adapter"] SW --> AD_AP["In-App Push Adapter"] end ```

[49:07](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=2947s) Clarifying the Adapter Role and Client Lifecycle

The discussion opens with the candidate clarifying the role of the adapter layer, noting that it acts as the interface bridging the internal pipeline to external delivery channels such as SMS, email, and mobile push notifications. The interviewer agrees with this framing, observing that the adapter essentially serves as a third-party integration interface. Tracing the overall pipeline, the interviewer summarizes the baseline architecture: the system ingests a notification request, queries metadata, writes the payload to an interim message store for durability, and dispatches it through a sender layer to adapters that reach end users.

```mermaid flowchart LR accTitle: Notification Dispatch Pipeline accDescr: High-level overview of message dispatch through temp storage, senders, and adapters to third-party endpoints. subgraph Ingestion["Ingestion & Durability"] direction TB C["Client Service"] --> AP["API Gateway"] AP --> MD["Metadata Service"] AP --> TS["Temp Storage (DynamoDB)"] end subgraph Processing["Dispatch & Delivery"] direction TB TS --> SW["Sender Workers (Async)"] SW --> AD["Adapters"] AD --> EP["Push / SMS / Email"] end subgraph Observability["Event Stream"] SW -.-> KF["Kafka + Flink"] end ```

The interviewer then raises an operational question regarding client communication: should the caller be kept waiting on an open connection throughout this multi-step broadcast, or how will the system respond? To frame the perspective, the candidate conceptualizes the client using a high-fanout Twitter analogy: "for example, Trump send a tweet here, I think Trump is the client, and the sender go all the way to different SMS, it means that all the followers of Trump would receive the SMS or the tweets." The interviewer refocuses the prompt on API design and protocol mechanics, asking what the end-to-end request lifecycle looks like from that client's point of view.

[51:20](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=3080s) Edge Architecture vs. Internal Service Clarification

Addressing the lifecycle, the candidate initially describes a full-featured perimeter ingress setup: a reverse proxy decrypting HTTPS payloads, a normalization layer converting requests into an internal format, local caching to resolve topic identifiers against a Metadata Service, and local disk logging to emit metrics for a real-time progress dashboard. The interviewer pushes back on coupling disk metrics and logs directly to the dashboard path, prompting a pause while the candidate adjusts the data path around storage durability, Flink stream processing, and replication.

The candidate then attempts to place authentication and authorization checks after the load balancer, which prompts the interviewer to intervene and correct the baseline assumptions of the system scope. The interviewer explains that this notification system is strictly an internal infrastructure platform: "I think all the request is actually internal service... like you build a tool for all your company's service." If designing for a company like Facebook, callers such as "Service A" or "Service B" are internal microservices already authenticated on the private network. The candidate immediately acknowledges the pivot, realizing that public edge authentication and perimeter reverse proxy decryption are unnecessary overhead for trusted internal services.

- **Client Nature**: Internal services within the enterprise infrastructure (e.g., Service A notifying User 1) rather than unauthenticated public mobile/web clients. - **Security Boundary**: Mutual trust inside the private internal network eliminates the need for edge auth tokens and SSL offloading at this layer. - **Monitoring Architecture**: Ingestion telemetry is funneled through Kafka and Apache Flink to calculate aggregation metrics for the operational dashboard rather than scraping raw application disk logs.

[56:48](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=3408s) Storage Selection and Asynchronous Worker Design

With the internal boundary established, the candidate restructures the ingest fan-out into three parallel responsibilities: querying the Metadata Service for the target topic definition, persisting the message in a Temporary Storage Service so that inactive or idle subscribers can retrieve updates upon reconnecting, and streaming telemetry events into Kafka and Flink for real-time monitoring.

For the Temporary Storage Service, the candidate chooses a scalable key-value database, specifically selecting Amazon DynamoDB to sustain high read and write throughput during high-traffic notification spikes. The sender component is designed as an elastic worker pool managing WebSocket connections and dispatch tasks, decoupling message generation from delivery. The adapter acts as a dynamic routing layer that delegates each payload to carrier-specific endpoints like APNs, FCM, or SMS gateways.

The interviewer praises the decision to use an asynchronous worker pool, advising the candidate to explicitly highlight core architectural buzzwords like "asynchronous" execution and "separation of concerns" to demonstrate seniority. The candidate agrees, noting down asynchronous worker pools as the linchpin for keeping the ingest layer responsive.

[59:08](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=3548s) Defining the Message and Request Lifecycle

Before detailing the DynamoDB schema, the candidate remembers that the functional requirements demand an "at-least-once" delivery guarantee, necessitating a dedicated Retry Service to handle failed outbound attempts. The interviewer steps in to guide the conversation toward concrete API mechanics: "Once you finish out the design, I do want to spend five minutes explain the life cycle for a message, a life cycle for notification, and life cycle for a client request."

To ground the explanation, the interviewer provides a concrete scenario: Service A wants to send a notification to User 1. The interviewer asks for a numbered step-by-step breakdown: Step 1 is calling the API with metadata and message content, but what exactly does Step 2 do, and how does the conversation proceed?

The candidate walks through the chronological path: 1. The client invokes the ingestion API endpoint with `topic_id`, payload, and delivery policy. 2. The service pre-processes the request through internal routing and retrieves schema definitions from the Metadata Service. 3. The notification payload is written durably into the Temporary Storage Service. 4. Sender workers poll or receive the persisted messages to initiate dispatch. 5. If a delivery attempt fails at the sender or adapter level, the message is routed to the Retry Service for exponential backoff and redelivery.

[01:03:04](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=3784s) Asynchronous Acknowledgement and Status Transitions

The interviewer presses on a fundamental distributed systems dilemma: "When do I know my message is sent, and when do I know my message is received by you?" These represent distinct milestones that must never be conflated. The candidate notes that a message is officially "recorded" once written to the Temporary Storage Service, whereas it is only truly "sent" after the adapter layer receives a confirmed delivery acknowledgement from the end subscriber device or upstream third-party gateway.

```mermaid flowchart TD accTitle: Message Delivery State Transitions accDescr: State machine of an internal notification request from pending through completed or failed. P["Pending (API Request Ingested)"] --> S["Sending (Persisted to Temp Storage)"] S --> C["Completed (All Adapters Acknowledge)"] S --> PC["Partially Completed (Partial Adapter Success)"] S --> F["Failed (Exhausted Retries / Storage Error)"] ```

When the interviewer asks whether a caller should block for minutes while millions of fan-out messages are pushed across the network, the candidate confirms an asynchronous status model with three primary states: - **Pending**: The API has received the request, but it has not yet been validated or committed. - **Sending**: The message is safely persisted to DynamoDB, and asynchronous sender workers are actively dispatching notifications. - **Completed**: All intended recipients have received their notifications. If a subset fails or downstream systems error out, status flags transition to **Partially Completed** or **Failed**.

The interviewer validates this state machine and emphasizes that the synchronous API call must return immediately at the point of persistence: once Step 3 commits the record to Temporary Storage, the API returns an HTTP acknowledgement receipt containing a job or message ID with a status of `Sending`. The client is thus freed immediately, while the sender workers and retry pipelines asynchronously fulfill delivery in the background.

[01:07:56](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=4076s) Asynchronous Polling, Receipt IDs, and Failure Handling

The interviewer, Scott, steers the discussion toward request-response mechanics and connection lifecycle management. He points out that an API cannot simply leave an HTTP request hanging indefinitely while it waits for downstream third-party delivery: "You don't want to hold this request forever." Instead, once the ingestion workflow reaches Step 5, the service must immediately acknowledge the caller by returning a distinct receipt ID. The client application can then asynchronously poll this receipt to inspect the current state of delivery. When the candidate asks why the system cannot immediately respond during the initial "pending" stage, Scott explains the harsh realities of distributed reliability: until the message payload is safely persisted to storage, acknowledging it introduces the risk of data loss. If the worker fails, the server crashes, or the client unexpectedly drops the socket connection before persistence finishes, the message vanishes into thin air while the client remains stuck indefinitely assuming the request is still pending.

```mermaid flowchart TD accTitle: Ingestion and Receipt Polling Flow accDescr: Flow showing asynchronous ingestion from client to storage and subsequent status polling. subgraph Ingestion ["Ingestion Phase"] A["Client Request"] --> B["Ingestion Worker"] B --> C["Topic & Payload Storage"] C --> D["Return Receipt ID"] end subgraph Polling ["Async Verification"] D -.-> E["Client Polls Receipt ID"] E --> F["Metadata Service"] F --> G["Current Delivery Status"] end ```

The discussion solidifies the rule that receipts can only be dispensed after state transitions are committed to topic storage and the database. This boundary ensures durability before handing execution off to background workers that handle fan-out, template rendering, and downstream vendor transport.

- **Receipt Handshake**: Rather than blocking synchronously, the server persists the notification intent and immediately returns an asynchronous tracking token (`receipt_id`). - **Failure Mode Defense**: Premature acknowledgments before database persistence cause orphan states if a node crashes or a TCP socket drops mid-flight. - **Worker Hand-off**: Committing the record to topic storage unblocks background consumers to handle delivery pipelines independently of caller latency.

---

[01:12:40](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=4360s) Data Schema: Temporary Message Storage with DynamoDB

Turning to the data schema, Scott and the candidate determine the division of labor between raw message storage and operational metadata. Drawing direct parallels to an earlier architectural review of Slack's backend (specifically a design discussed around May 26), Scott highlights that chat and notification backends routinely decouple raw content storage from message delivery state machines. For the temporary message storage service, a high-throughput NoSQL key-value store such as Amazon DynamoDB fits best. The discussion debates whether delivery status belongs in this raw payload table; they conclude it should not, because a single broadcast topic ID might fan out across thousands of recipients with distinct operational statuses, while the message payload itself remains immutable.

```mermaid flowchart LR accTitle: DynamoDB Temporary Storage Schema accDescr: Diagram illustrating the partition key, sort key, and attributes for message storage. subgraph KeyStructure ["Key Schema"] PK["Partition Key: message_id (UUID)"] SK["Sort Key: timestamp"] end subgraph Attributes ["Attributes"] T["topic_id"] S["sender_id"] P["payload / body"] end KeyStructure --> Attributes ```

The candidate constructs the primary key layout for DynamoDB, setting `message_id` as the partition key and a `timestamp` as the sort key to order events efficiently. The record retains the core communication payload: sender identity, topic classification, and message content. By treating this store as a fast, durable key-value cache, the system avoids burdening relational databases with large, short-lived payload blobs.

- **Storage Engine**: Amazon DynamoDB acts as the transient, high-throughput key-value engine for message payloads. - **Key Design**: Partition Key is defined as `message_id`, paired with `timestamp` as the Sort Key for chronological ordering. - **Separation of Concerns**: Immutable message contents (body, sender ID, topic ID) are strictly separated from mutable delivery states.

---

[01:18:16](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=4696s) Data Schema: Relational Metadata Store for Lifecycle Tracking

While the message payload resides in NoSQL, the operational metadata requires strict consistency. Scott and the candidate agree that the metadata service should be backed by a relational SQL database because tracking the life cycle of notifications demands atomic transactions and rollback capabilities: "You need transaction, right?" The metadata table manages the progression of notifications from submission to recipient delivery, tracking the specific parameters that govern routing.

The relational schema stores `message_id`, `receiver_id`, `status`, `send_mode`, and `service_type` (e.g., SMS, APNS push, email). Crucially, Scott emphasizes including an updated timestamp field (`updated_at`): "So you know which message has been stuck at what time for how long." By maintaining relational guarantees on this metadata, downstream monitoring jobs can run reconciliation queries, detect messages stranded in non-terminal states, and trigger automatic retries without duplicating the message body.

---

[01:20:55](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=4855s) Tracing the End-to-End Notification Lifecycle

Scott prompts the candidate to spend three minutes detailing the notification's internal lifecycle from its own perspective, distinguishing it clearly from the client's external request-response cycle: "Let's try to describe what's the life cycle for notification looks like." The process begins at notification creation, where a client application requests a message dispatch. The pipeline moves into experiment evaluation and template rendering—for example, resolving localized email templates, user preferences, and variant flags.

```mermaid flowchart TD accTitle: Notification State Lifecycle accDescr: State diagram showing transitions from creation to sending and completion. A["Notification Created"] --> B["Template & Experiment Resolution"] B --> C["Status: PENDING in Metadata"] C --> D["Dispatched to Provider / Adapter"] D --> E["Status: SENDING"] E --> F{"Vendor Response"} F -->|"200 OK"| G["Status: COMPLETED"] F -->|"Error / Timeout"| H["Status: FAILED / RETRY"] ```

Once the payload is constructed, the metadata service records the notification with an initial state of `PENDING`. When background notification dispatch workers pick up the task and invoke downstream carrier adapters, the metadata status transitions to `SENDING`. Once the external gateway (such as an email provider or APNs gateway) acknowledges delivery or responds with a definitive webhook, the worker finalizes the state transition to `COMPLETED` or `FAILED`. This finite state machine guarantees visibility into every phase of delivery.

- **Creation & Hydration**: Client inputs are merged with user preference profiles and dynamic email/push templates. - **Pending Phase**: The generated notification is persisted in SQL metadata with status `PENDING` before network transmission. - **Dispatch to Complete**: Status switches to `SENDING` during adapter execution and shifts to `COMPLETED` once provider delivery callbacks arrive.

---

[01:25:40](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=5140s) Interview Debrief: Delivery Guarantees, Worker Pools, and Distributed Failure Modes

As the mock interview moves into its retrospective feedback stage, Scott asks the candidate to evaluate their performance: "Which part you did good and which part you didn't do well?" The review immediately centers on delivery guarantees: the classic trade-off between at-most-once and at-least-once delivery. In any real-world distributed system, "every single step can fail"—from network timeouts and crashed worker nodes to unacknowledged broker receipts. Designing for at-most-once delivery requires committing state markers early so an interrupted task never repeats, but this risks dropping notifications. Conversely, at-least-once guarantees require idempotent message handling at the receiver side, paired with robust retry loops.

```mermaid flowchart TD accTitle: Distributed System Philosophy accDescr: Subgraph breakdown of core distributed systems patterns discussed during debrief. subgraph IngestionCore ["Separation of Concerns"] A["Sender-Side Ingestion"] --> B["Queue Buffering"] end subgraph ExecutionLayer ["Async Execution"] B --> C["Worker Pool Processing"] C --> D["Event-Driven Notification Router"] end ```

Scott praises the candidate's use of foundational distributed systems patterns: - **Separation of Concerns**: Dividing fast request ingestion from slow, unpredictable downstream network calls. - **Asynchronous Worker Pools**: Decoupling the front-facing API from third-party vendor latencies via dedicated queue consumers. - **Event-Driven Architecture**: Moving state updates across the system using discrete lifecycle events rather than tight, synchronous service coupling. - **Tiered Durability**: Maintaining transient payloads in elastic NoSQL storage while recording authoritative state transitions in ACID-compliant SQL metadata.

---

[01:32:00](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=5520s) Deep-Dive Feedback: Router-Adapter Architecture, Retries, and Status Transitions

In the final minutes of the session, Scott deep-dives into edge-case handling, routing mechanics, and interface boundaries. A key design element reviewed is the relationship between the notification router and provider adapters. Scott advises that the adapter layer must strictly abstract low-level communication protocols and translate vendor-specific error codes into standardized internal status codes. Downstream service providers frequently return partial failures—for example, rate-limiting HTTP 429s, invalid device tokens, or transient connection resets. If an adapter exposes provider-specific nuances directly to the core router, the retry logic becomes brittle and difficult to maintain.

```mermaid flowchart LR accTitle: Router-Adapter Architecture accDescr: Separation of notification router from carrier-specific adapters and error translation. R["Notification Router"] --> A1["Email Adapter (SendGrid)"] R --> A2["Push Adapter (APNs/FCM)"] R --> A3["SMS Adapter (Twilio)"] A1 --> E["Normalized Error Codes & Retries"] A2 --> E A3 --> E ```

Scott notes that notifications often travel in batches, requiring the system to handle chunked grouping and partial completions: a batch of 500 emails might succeed for 490 recipients while 10 fail due to expired addresses. The state machine must record these partial outcomes cleanly rather than marking the entire batch as an ambiguous error. The session concludes with Scott emphasizing how clear categorization of error statuses, centralized retry configuration, and clean adapter abstractions transform a naive notification script into a resilient, production-grade enterprise notification platform.

- **Normalized Status Codes**: Low-level vendor errors (e.g., Twilio 21211, SendGrid 400-series) must be translated by adapters into uniform internal statuses (`RETRYABLE_ERROR`, `FATAL_INVALID_RECIPIENT`). - **Partial Completion States**: Broadcast messages and chunked batches must track individual recipient delivery states to prevent infinite re-dispatching of already-delivered items. - **Configurable Retry Policies**: Backoff intervals, jitter algorithms, and dead-letter queue handoffs should be controlled centrally at the router layer rather than scattered across individual worker adapters.

## Notable quotes > [20:29](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=1229s) LinkedIn was fined millions of dollars because it sent promotional emails repeatedly — Scott Shi

> [33:23](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=2003s) Let's assume that we are designing the notification system for Facebook... you have 200 million active users per day, and each user has five notifications per day on average. — Scott Shi

> [33:23](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=2003s) You're missing a zero. — Scott Shi

> [37:09](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=2229s) If you're going to send emails, you're going to have images, right?... and you may have possibly video, but most likely you're going to be an HTML file like multimedia. — Scott Shi

> [40:44](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=2444s) Oh, I see. So you're referring to more like a notification template, right? Like an email template. — Scott Shi

> [45:33](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=2733s) I'm a little confused... could you please explain how your system is going to model the following scenario? Like Service A sends SMS to User 1, Service B sends email to User 2, and Service C sends phone calls to User 2? — Scott Shi

> [49:07](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=2947s) for example, Trump send a tweet here, I think Trump is the client, and the sender go all the way to different SMS, it means that all the followers of Trump would receive the SMS or the tweets. — Candidate

> [51:20](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=3080s) I think all the request is actually internal service... like you build a tool for all your company's service. — Scott Shi

> [59:08](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=3548s) Once you finish out the design, I do want to spend five minutes explain the life cycle for a message, a life cycle for notification, and life cycle for a client request. — Scott Shi

> [01:03:04](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=3784s) When do I know my message is sent, and when do I know my message is received by you? — Scott Shi

> [01:07:56](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=4076s) You don't want to hold this request forever. — Scott Shi

> [01:18:16](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=4696s) You need transaction, right? — Scott Shi

> [01:18:16](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=4696s) So you know which message has been stuck at what time for how long. — Scott Shi

> [01:20:55](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=4855s) Let's try to describe what's the life cycle for notification looks like. — Scott Shi

> [01:25:40](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=5140s) Which part you did good and which part you didn't do well? — Scott Shi

> [01:25:40](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=5140s) every single step can fail — Scott Shi

## Mentioned Scott Shi, Uber, Uber Money, DynamoDB, LedgerStore Gateway, Apache Airflow, Piper, SendGrid, Twilio, Apple Push Notification service (APNs), Firebase Cloud Messaging (FCM), Apache Kafka, LinkedIn, Facebook, Twitter, Donald Trump, Apache Flink, Slack, Amazon DynamoDB

## Video description in this session, we mocked a system design interview

doc could be found here: https://docs.google.com/document/d/140n69Pw-xqoD3eeGzcv2WH_CDRbgYmYzV6OPHl6NzA8/edit?usp=sharing

[Source](https://www.youtube.com/watch?v=D0bBMzxE5O0)

中文简报

分布式通知架构拆分载荷与状态表防范第三方通道重试与丢包风险因外部通道缺乏两阶段提交导致端到端精确一次投递在工程上不可行，高并发通知系统必须在持久化后立即返回异步回执并通过关系型状态机驱动重试。面试首先深入探讨了候选人在 Uber Money 以往的支付存储迁移项目，随后指导了如何围绕业务影响来阐述技术挑战。核心考核环节要求为 2 亿日活跃用户设计一个支持电子邮件、短信、移动端推送和语音电话的企业级通知平台。面试官与候选人共同确立了一套异步架构，将 DynamoDB 中不可变的消息载荷与关系型数据库中的事务性生命周期状态跟踪进行解耦。讨论深入分析了投递语义，解释了为何依赖第三方实现精确一次投递并不现实，并展示了适配器层如何规范化外部服务商错误及处理重试。对于一个扩展至 2 亿日活跃用户、平均每人每天 5 条告警的平台，通知摄取量平均达到 11,600 写入 QPS，因此必须进行水平扩展并与第三方服务商网络异步解耦。 (33:23) 由于缺乏分布式两阶段提交协议，跨 SendGrid 或 Twilio 等外部网关严格实现端到端精确一次投递在工程上不可行，这使得持久化至少一次投递结合接收端去重成为标准架构模式。 (24:26) 存储被拆分为用于存储不可变原始消息体的高吞吐 NoSQL（DynamoDB），以及用于事务性投递生命周期状态跟踪和卡顿消息检测的关系型 SQL。 (72:40) 为防止大规模广播期间发生 HTTP 套接字超时和消息丢失，摄取 API 会立即将载荷提交至持久化存储并返回异步跟踪回执 ID，随后再由工作池处理下游分发。 (67:56)

Reading summary

In a mock system design interview conducted by Scott Shi, a candidate designs an internal, multi-tenant enterprise notification platform capable of processing one billion messages daily, alongside a review of zero-downtime payment migration strategies. The session highlights that building resilient notification systems requires asynchronous worker pools, dual-tier storage separation, and normalized vendor adapters rather than chasing unattainable end-to-end exactly-once delivery guarantees across third-party networks. Technical Ownership in Zero-Downtime Data Migrations (10:20) The candidate details migrating Uber Money's payment grant storage from a standalone DynamoDB cluster to Uber's centralized LedgerStore Gateway over a three-month timeframe. To achieve zero downtime and zero data loss, the engineering pipeline utilized dynamic dual-writing, priority fallback routing, and an online shadow reconciliation engine to continuously verify data parity. Rather than executing complex Apache Airflow batch ETL pipelines, the team capitalized on the records' inherent six-day business time-to-live (TTL), halting writes to the legacy database so historical data naturally expired. Interviewer Scott steps out of character to coach the candidate on interview communication, emphasizing that senior engineers must claim end-to-end architectural ownership and articulate trade-offs rather than simply describing assigned tasks. He advises eliminating low-level compiler jargon and instead directly tying technical accomplishments to organizational scale, financial risk mitigation, and infrastructure deprecation. Sizing, Trust Boundaries, and Delivery Semantics (20:29) Framing the enterprise notification problem around Facebook-scale requirements, the interviewer sets traffic at 200 million daily active users generating an average of five notifications each, resulting in one billion daily notifications and an average ingestion throughput of approximately 11,574 requests per second. Scott clarifies the security perimeter: because the platform serves internal microservices within a private corporate network, edge reverse proxies, public SSL offloading, and client authentication layers represent unnecessary overhead. The scope requires supporting multi-channel delivery across SMS, phone calls, mobile push, and email, with multimedia payloads handled as lightweight references resolved lazily by CDNs. When clarifying delivery semantics, Scott emphasizes that strict end-to-end exactly-once delivery across uncoordinated third-party vendors like SendGrid or Twilio is practically unfeasible without distributed two-phase commit (XA) protocols, which vendors do not support. Consequently, the architecture prioritizes robust at-least-once delivery backed by durable retry queues while leaving strict exactly-once processing out of the core MVP. Tiered Durability and Asynchronous Ingestion Handshakes (67:56) To decouple client latency from unpredictable downstream fan-out, the architecture establishes an asynchronous ingestion handshake. Internal clients submit notifications and receive an immediate tracking token (`receipt_id`) once the payload is committed to durable storage, allowing callers to poll status later. Scott stresses that premature acknowledgments prior to database persistence risk silent message loss if an ingestion worker crashes or a socket drops, while holding client connections open until vendor delivery causes connection exhaustion. The system adopts a dual-tier storage strategy that strictly isolates immutable content from mutable state. Raw message payloads are written to Amazon DynamoDB—keyed by `message_id` as the partition key and a `timestamp` sort key—functioning as a high-throughput, transient key-value store. Concurrently, notification delivery metadata is assigned to an ACID-compliant relational SQL database storing foreign keys, recipient identifiers, service channels, delivery modes, statuses, and update timestamps. Maintaining transactionality in SQL enables background reconciliation jobs to query stranded records and orchestrate retries without reloading heavy payload bodies. Lifecycle State Machines and Adapter Abstraction (80:55) Notification delivery is governed by an explicit finite state machine advancing from `PENDING` upon metadata creation, to `SENDING` when picked up by asynchronous worker pools, and concluding in `COMPLETED`, `PARTIALLY COMPLETED`, or `FAILED` upon external confirmation. To prevent cross-system coupling, an adapter layer abstracts provider-specific protocols (such as APNs, Google FCM, SendGrid, and Twilio) and normalizes low-level vendor errors (such as HTTP 429 rate limits or invalid recipient codes) into standardized internal error codes. Scott highlights that this normalization allows retry policies—such as exponential backoff, jitter, and dead-letter queue handoffs—to be configured centrally at the router level rather than duplicated across worker adapters. Furthermore, the state machine tracks individual recipient outcomes within chunked batch dispatches, ensuring that partial carrier failures do not trigger duplicate redeliveries to recipients who already received their notifications. The mock interview demonstrates that high-throughput notification architectures depend on asynchronous worker pools, clean lifecycle state tracking, and standardized vendor abstraction layers rather than synchronous delivery pipelines. While the discussion successfully resolved ingestion throughput and dual-tier data modeling, downstream provider rate-limiting algorithms and recipient-level frequency capping mechanisms were not detailed.

精炼阅读摘要

本期模拟面试由 Scott Shi 主持，探讨了企业级分布式通知平台的核心架构设计。方案主张采用关注点分离原则，将快速同步摄取与异步多渠道分发解耦，并结合双层存储与标准化适配器以应对海量写入与第三方网络不确定性。业务规模估算与交付语义边界澄清 (20:29) 面试首先基于类 Facebook 规模确立吞吐指标：系统面向 2 亿日活跃用户，平均每人每日 5 条通知，即每日总量达 10 亿条，折算平均写入负载约 11,574 QPS。在高并发峰值冲击下，摄取层与数据层必须通过水平扩展与存储分片防止单机 I/O 成为瓶颈。在内容形态上，系统采用引用模型传输富媒体，通知载荷仅保留纯文本或嵌入资源 ID 的 HTML，实际媒体数据由客户端惰性拉取解析，从而减轻通道带宽压力。在交付语义上，面试官与候选人划清了理论与工程实现的边界。虽然系统支持至少一次（At-least-once）与至多一次（At-most-once），但面试官指出严格的端到端“精确一次”（Exactly-once）在跨第三方供应商（如 SendGrid、Twilio）场景下存在不可解的两阶段提交（XA/2PC）困境。因外部服务缺乏分布式事务支持，网络超时极易导致状态不可知；加之 2014 至 2015 年间 LinkedIn 曾因重复推送促销邮件而面临巨额罚款，生产环境应通过上游幂等去重配合补偿重试，而非在 MVP 中强求端到端精确一次。非阻塞摄取流与客户端收据交互机制 (51:20) 针对系统边界，面试官纠正了候选人最初设计的公网边缘鉴权与反向代理思路，澄清该通知系统本质上是企业内部微服务基础设施。服务间通信默认处于受信内网，因而无需重复进行边缘 SSL 卸载与公网 Token 校验，Ingress 重点在于保证非阻塞的高并发摄取。为避免下游渠道延迟拖垮上游调用方，API 层必须执行严格的异步交互契约。客户端发起包含主题 ID 与交付策略的请求后，网关在将消息持久化至临时存储层（确保无节点宕机丢包风险）的瞬间，必须立即返回一个跟踪凭证（receipt_id），并将状态置为发送中（Sending）。系统坚决避免同步挂起 HTTP 连接等待下游确认，调用方后续仅需凭此凭证通过专用端点轮询交付进度，实现真正的高吞吐解耦。双层存储分层：NoSQL 负载与 SQL 元数据状态机 (72:40) 数据层采用基于关注点分离的双层存储模型。针对大体量、短生命周期的通知载荷，系统选用高吞吐 NoSQL 键值数据库 Amazon DynamoDB，以 message_id 作为分区键、时间戳为排序键，专门存储不可变的消息主体与主题元数据，避免关系型数据库被大体积瞬态数据拖垮。与此相对，通知的生命周期流转与运维状态则剥离至具备 ACID 事务能力的关系型 SQL 数据库。元数据表维护 message_id、receiver_id、渠道类型、发送模式、当前状态（Pending、Sending、Completed、Failed）以及精确的 updated_at 时间戳。这一结构使得监控引擎能够执行对账查询，实时检测滞留于中间态的异常消息，并在不复制正文载荷的前提下，准确触发批量或局部的故障重试。路由适配器分层与部分失败重试治理 (92:00) 分发端采用通知路由引擎与专有适配器（Adapter）分层架构。由弹性 Worker 线程池异步消费待发送队列，并由适配器封装 APNs、FCM、SendGrid 及 Twilio 等不同厂商的底层通信协议与专有返回码。适配器的核心职责在于屏蔽异构接口差异，将各类厂商特定的错误代码（如限流 HTTP 429 或失效设备 Token）规范化映射为统一的内部状态码（如可重试错误或永久失效），防止下游细节污染核心路由逻辑。针对批量广播场景（如单条广播拆解为 500 个接收者），系统必须维护细粒度的局部交付状态。若出现部分成功与部分失败，状态机精准标记 Partially Completed，并在路由层集中控制重试退避算法与死信队列转移，避免对已交付用户进行全量二次重推。该模拟面试确立了企业级通知系统的核心范式：内网环境应精简外网安全重载，依靠双层存储与轻量 receipt 机制平衡耐久性与吞吐，并通过规范化的适配器解耦第三方依赖风险。不过，讨论将端到端精确一次排除于系统范围之外，跨服务大规模扇出时的多租户配额限制与流量整形策略仍有待更深入的实践细化。

Detailed notes — zh

## Background 该视频由 YouTube 频道主 Scott Shi 于 2020 年 11 月 29 日发布。视频内容为一场时长约 1 小时 39 分钟的模拟系统设计面试（Mock Interview），配套提供了 Google 文档记录，核心课题为设计一个企业级通知系统（Notification System）。 ## TL;DR 视频首先通过分布式一致性理论探讨与候选人过去在 Uber 的支付存储迁移项目进行热身，随后正式切入设计企业级通用通知系统的核心环节。面试官与候选人深入探讨了跨第三方调用时 Exactly-once 的落地局限性，并明确了支持短信、邮件、语音和应用内推送等多渠道的混合架构。系统通过将临时消息存储（DynamoDB）与生命周期元数据存储（SQL 关系型数据库）解耦，配合异步 Worker 池与渠道适配器（Adapter Pattern）实现高吞吐分发。整场面试还详细推演了基于 Receipt ID 的异步轮询状态机、重试退避机制以及在大规模流量下的分布式容错设计原则。 ## Atmosphere & style 整场模拟面试呈现出浓厚的硬核技术交流氛围，节奏严密且层层递进。面试官风格犀利专业、一针见血，在候选人出现概念模糊（如将模板误作 Kafka Topic、混淆公网网关与内网服务）或算错数字时会当场打断指正，同时又极具启发性地引导其纠偏。候选人则展现出扎实的工程功底，在面对高难度分布式边界与状态机推演时能够迅速吸收反馈并调整架构。两人的互动既有技术争鸣的紧张感，又有资深工程师之间复盘探讨的务实与默契。 ## Bulletin - [24:24](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=1464s) 企业级通知系统设计需明确投递语义边界：在涉及第三方渠道（如 SendGrid、Twilio）的跨系统黑盒交互中，强行追求端到端 Exactly-once 是不切实际的，工业界主流选择 At-least-once 配合消费端幂等去重。 - [34:06](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=2046s) 基于 2 亿日活跃用户（DAU）和人均 5 条通知测算，系统日消息量达 10 亿条，平均处理 QPS 约为 11,574 req/s，单机连接与 I/O 无法承载，必须采用水平分片与无状态架构。 - [01:12:40](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=4360s) 通知系统架构核心在于关注点分离与异步解耦：将高吞吐的消息体存储（NoSQL / DynamoDB）与强事务的生命周期元数据表（SQL）分离，避免宽表状态频繁更新带来的写入热点。 - [01:07:56](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=4076s) 内部客户端请求绝不可同步阻塞等待：系统应在完成消息落盘与初始状态落库后立即返回 Receipt ID 释放连接，后续通过后台无状态 Worker Pool 和适配器异步分发，客户端通过轮询获取最终状态。 ## Key takeaways - [24:24](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=1464s) 在第三方 API 无法参与 2PC/XA 分布式事务的现实下，网络超时会导致无法确认外部真实投递结果，因此企业通知系统通常退化为 At-least-once 保证，而不是苛求绝对的 Exactly-once。 - [37:08](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=2228s) 富媒体通知不应直接在核心消息管道中传输大体积二进制文件，而应通过内嵌 ID 或 CDN 链接，将图片与视频渲染解耦给终端客户端异步拉取。 - [40:49](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=2449s) 设计通知系统的 Topic API 时需理清业务概念，它通常指代通知模板（Template）与渠道配置，而非底层的 Kafka 物理主题。 - [51:20](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=3080s) 作为企业内部基础设施，通知服务的调用方均为已认证的受信任微服务，设计时应精简外网反向代理、公网身份鉴权及 HTTPS 解密等冗余复杂度。 - [56:15](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=3375s) 系统设计必须遵循“异步处理（Asynchronous Processing）”与“关注点分离（Separation of Concerns）”，以缓冲削峰并隔离各渠道的异构实现。 - [01:07:56](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=4076s) 客户端回执（Receipt ID）必须在消息可靠写入临时持久化存储后再行返回；过早返回会导致在服务节点崩溃时产生消息丢失与不可自愈的幽灵状态。 - [01:12:40](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=4360s) 数据模型应严格区分数据面与控制面：轻量高吞吐的 Key-Value 存储用于读写消息正文，关系型 SQL 数据库则负责支持状态转移的 ACID 事务与生命周期追踪。 - [01:18:18](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=4698s) 生命周期元数据表中的更新时间戳（updated_at）是核心运维指标，监控组件借此可直接发现并在可视化大盘上告警哪些通知在特定状态超时停滞。 - [01:20:55](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=4855s) 单条通知端到端生命周期清晰划分为：创建与模板渲染（Creation）-> 持久化并返回 Receipt（Pending）-> Worker 派发外部渠道（Sending）-> 终态判定（Completed / Failed / 重试排队）。 - [01:25:37](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=5137s) 分布式系统的核心法则是“每一步都可能发生故障”，因此系统韧性依赖于专门的重试服务（Retry Service）、指数退避策略以及死信队列（DLQ）。 - [01:31:53](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=5513s) 适配器模式（Adapter Pattern）是统一异构推送网关（如 APNs、FCM、Twilio、SendGrid）的标准手段，负责将各渠道错误码归一化为内部状态。 - [01:31:53](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=5513s) 在面向大规模用户推送时，系统应具备分块（Chunking）与批量聚合能力，以平衡数据库 I/O 吞吐与单条通知的时延要求。 ## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=0s) 面试热身与分布式一致性概念探讨在模拟面试正式启动前，双方进行了简短的设备与录制确认。随后，面试官与候选人就分布式系统底层的一致性理论展开了碎片化的概念梳理与探讨。讨论涉及线性一致性（Linearizability）、强一致性（Consistency）、多副本同步机制以及主从架构（Master-Replica Replication）。两人特别分析了 Compare-and-Set（CAS）操作在分布式共识（Consensus）中的要求，以及快照隔离（Snapshot Isolation）和可串行化（Serializability）与线性一致性之间的区别。经过这些分布式事务与阻塞读写机制的理论铺垫后，面试官指导候选人完成屏幕共享与权限设置，正式转入系统设计面试环节。 --- ### [10:20](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=620s) 履历深挖：支付网关与存储迁移项目面试官首先要求候选人用大约 5 分钟时间，聚焦技术复杂度而非纯业务指标，介绍一段过去在工作中负责过的最具技术挑战性的项目（排除校内项目）。候选人介绍自己目前就职于 Uber Money 团队（支付业务线），近期的工作主要围绕支付系统展开。 ```mermaid flowchart LR accTitle: Payment Storage Migration Architecture accDescr: Dual-write and shadow validation migration from standalone DynamoDB to LedgerStore Gateway. subgraph Clients["调用方"] API["Payment Service"] end subgraph Storage["存储层双写与降级"] LSG["LedgerStore Gateway (新存储网关)"] DDB["Standalone DynamoDB (老存储 / 6天TTL)"] end API -->|"1. 写入"| LSG LSG -->|"2. 降级/回退"| DDB ``` - **项目背景**：去年候选人主导了支付凭证存储（Grant Storage）的底层迁移，将原有的独立 DynamoDB 实例无缝切换并集成至公司内部统一的账本存储网关（LedgerStore Gateway）。整个项目历时约 3 个月。 - **技术难点与读写回退**：最大挑战在于高可用金融场景下的读写回退策略（Fallback Strategy）和零数据丢失保障。候选人设计了严密的双写（Dual-write）与影子写入（Shadow write）流程，在写入 LedgerStore Gateway 遇到问题时无缝回退至老存储，并通向反向调序与数据校验机制确保平滑过渡。 - **历史数据处理技巧**：当面试官追问是否编写了大批量 Airflow 或 Piper 离线迁移任务时，候选人解释由于该业务凭证具有 6 天的过期淘汰时间（TTL），团队通过设置时间戳硬截止点，提前几天逐步停止向旧库写入，等待老数据自然过期淘汰，从而省去了复杂繁重的全量历史数据批处理搬迁。 --- ### [16:35](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=995s) 面试官复盘与项目陈述建议面试官对候选人的技术迁移逻辑表示肯定，并给出了关于系统设计面试中“行为与项目背景介绍”的辅导建议： - **展现独立性与主导力**：说明自己是独自承接核心链路还是依靠 Staff 工程师的指导，明确沟通自己负责的系统边界。 - **避免过度陷入底层琐碎细节**：在阐述时不要单纯堆砌技术行话（如编译器、GC 垃圾回收、停用词等技术细节），而是要把技术的架构难度与“为什么这么做”、“解决了什么业务稳定性瓶颈”结合起来。 - **突出全局影响**：以 Uber 替换原有派单调度系统（如淘汰 RTAPI / 现有调度层）为例，高级别项目应该清晰体现系统重构背后的可用性收益和业务影响（Business Impact），而非仅仅是实现流程的平铺直叙。 --- ### [22:14](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=1334s) 通知系统（Notification System）题目定义与需求澄清面试官正式给出本次系统设计的目标题目：**设计一个企业级通知系统（Notification System）**。该系统作为通用的基础服务，面向公司内部所有业务线提供统一接入能力。面试官详细列出了系统的功能与非功能期望： - **多渠道推送集成**：支持主流渠道，包括 App 应用内/系统级推送（Apple APNs、Android FCM）、电子邮件（Email）、短信/电话通知（SMS/Phone）；要求能灵活对接第三方供应商（如用于短信的 Twilio、用于邮件的 SendGrid 等）。 - **统一接入接口**：对外暴露统一规范的 API 端点或 Kafka Topic 供各业务方调用，屏蔽底层供应商的对接差异。 - **递送语义保证（Delivery Guarantees）**：支持不同级别的投递策略，包括至少一次（At-least-once）、至多一次（At-most-once）以及精确一次（Exactly-once）。面试官特别提到 LinkedIn 在 2014-2015 年间因重复发送营销推广邮件被巨额罚款的案例，强调防重复发送的重要性。 - **业务级监控与可视化**：非单纯的机器指标（CPU、内存、磁盘等），而是要监控业务流转健康度。例如各渠道/供应商积压了多少通知、正在投递与发送失败的数量、邮件队列发生超时卡死时的告警与可视化大盘。 --- ### [24:24](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=1464s) 递送语义争鸣：分布式投递中的“Exactly-Once”边界候选人梳理了整体答题框架，提议按照“业务用例与 MVP 功能 $\rightarrow$ 约束条件 $\rightarrow$ API 接口设计 $\rightarrow$ 系统架构图 $\rightarrow$ 数据模型（Data Schema）”的节奏逐步展开，并将核心渠道（App、Email、SMS）与投递模式划分至 MVP 与拓展需求中。随后，面试官针对投递语义（At-least-once、At-most-once 与 Exactly-once）的本质区别发问，双方展开了深入探讨： ```mermaid flowchart TD accTitle: Third-party Notification Timeout Problem accDescr: Sequence illustrating uncertainty during third-party API timeout. subgraph Service["企业通知服务"] Worker["Notification Worker"] end subgraph Vendor["第三方渠道商 (如 Twilio / SendGrid)"] Provider["External API Server"] EndUser["最终用户终端"] end Worker -->|"1. POST /send_email"| Provider Provider -.->|"2. 投递真实邮件"| EndUser Provider x--x|"3. 网络超时 / 无响应"| Worker ``` - **候选人的理解**： - *At-least-once*：依赖重试队列（Retry Queue）和消息持久化（Durability）来保障，哪怕网络抖动也能多次重发直至成功。 - *At-most-once*：采用“即发即弃”（Fire-and-forget）模式，失败则丢弃，无需重试链路。 - *Exactly-once*：不仅需要前后端解耦与消息中间件（如 Kafka）的 Exactly-once 模式支持，还需要在网关与负载均衡阶段通过幂等键（Idempotency Key）过滤重复提交。 - **面试官的纠正与跨系统现实**：面试官指出，业务层面的 Exactly-once 与 Kafka 内部的事务性 Exactly-once 并非同一概念。通知系统本质是一个涉及外部第三方的跨系统分布式事务。当我们调用第三方服务（如 SendGrid）发送邮件且发生 HTTP 请求超时时，由于没有返回，内部服务**根本无法确认外部供应商到底有没有把邮件实际投递到用户终端**。 - **技术妥协与收敛**：真正的端到端绝对 Exactly-once 需要两阶段提交（2PC）或 XA 分布式事务规范，但在第三方黑盒 API 场景下根本无法落地。因此面试官建议在系统设计中跳过这一极端严苛的假设，明确指出在不受控的外部集成场景下追求绝对的 Exactly-once 是不切实际的。 ### [29:58](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=1798s) 核心机制与扩展功能：推拉模型、计划通知与非功能性指标在讨论完基础用例后，候选人进一步梳理最小可行产品（MVP）与加分项（Bonus）的边界，重点提出了针对不同活跃状态订阅者的“推（Push）与拉（Pull）混合模型”。候选人解释道，对于在线的活跃用户（Active subscribers），系统应当主动将通知推送给客户端；但对于离线或闲置用户（Idle subscribers）——比如 iOS 应用已经被彻底划掉关闭、无法实时接收通知的场景——当用户重新启动应用或系统后台任务被唤醒时，客户端可以主动向通知服务发起拉取请求（Pull），或者通过 gRPC / HTTP 直接从数据存储中批量捞取未读消息。面试官随即打断并要求澄清细节：“你这里说的 Push 和 Pull 模型，具体指的是什么？”候选人澄清，除了系统在消息产生时直接推出去之外，发送端或请求方还可以通过 RPC 接口直接拉取数据。随后，候选人补充了另外两项重要的扩展能力：支持定期通知（Recurring notifications）与定时调度通知（Scheduled notifications），这两类能力在电子邮件和短信触达业务中尤为常见。谈及非功能性目标（Non-goals）与性能考量时，候选人做出了几项关键权衡： - **延迟容忍度（Latency）**：并不是所有通知都需要毫秒级即时送达。系统应对通知进行优先级划分（High vs. Low Priority）。以 Twitter 等社交平台为例，低优先级的应用内通知即便延迟几小时推送到终端，也是完全可以接受的。 - **顺序保证（Order of delivery）**：明确将“严格保序”划定为 Non-goal，系统不需要耗费昂贵代价去保证所有通知全局按严格的时间戳顺序投递。 - **高可用性（High Availability）与可扩展性（Scalability）**：通知系统必须随时处于就绪状态，绝不能出现因系统故障而无法向订阅者发出重要紧急通知的情况。 ### [34:06](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=2046s) 业务规模估算与容量测算：单机极限与水平扩展就在候选人准备泛泛而谈可用性时，面试官笑着提醒他漏掉了一个最核心的设计前提：“你好像还没问我业务规模是多少呢。让我们假设这是为 Facebook 设计的通知系统。”面试官随即给出了基准数据：拥有 2 亿（200M）日活跃用户（DAU），平均每位用户每天接收 5 条通知。候选人在草稿纸上记录数字时漏写了一个零，被面试官当场抓包指正：“你少写了一个零。” 根据面试官给出的数据，候选人迅速进行了吞吐量估算。假设所有通知均采用推送方式： $$\text{每日消息总量} = 200,000,000 \times 5 = 1,000,000,000 \text{ 条/天（10亿条）}$$ 折算到每秒平均请求量（QPS）： $$\text{平均 QPS} \approx \frac{1,000,000,000}{24 \times 3600} \approx \frac{1,000,000,000}{86,400} \approx 11,574 \text{ req/s}$$ 候选人指出，即便不算尖峰流量（Peak Traffic），仅平均写入/推送请求就已达到每秒 1.15 万次以上。在高峰期，流量通常还会翻数倍。这个量级的吞吐量和并发读取无论如何都无法在一台单机服务器上承担，必然会轻松打爆单台机器的连接数和 I/O 极限。因此，系统必须在架构层面上引入数据分片（Sharding）以及无状态服务的水平横向扩展能力（Horizontal Scalability）。 ### [37:08](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=2228s) 灵活性与消息类型考量：多媒体渲染与渠道分类除了高可用与高伸缩性，系统的“灵活性（Flexibility）”也是一大重点，必须兼容形形色色的消息发送方与接收终端。针对发送端与通知分类，候选人询问系统一共需要支持多少种不同类型的“主题（Topic）”。面试官反问：“你打算根据什么维度来创建 Topic 呢？”候选人回答，可以根据 Topic 名称、服务类型、Topic 元数据以及消息正文来定义。此时候选人试探性地提出一个简化假设：“在非目标（Non-goals）中，我们是否可以约定系统只负责发送纯文本消息，不处理图片？”面试官立刻提出了异议，表示发邮件通常不可避免会带有图片甚至视频。面试官指出，富文本邮件本质上是一段多媒体 HTML 文件，完全不支持多媒体并不现实，但可以将其作为 Bonus 处理。候选人顺势调整思路，阐述了一种解耦多媒体传输的设计方案： - 通知系统内部的消息体保持纯文本/HTML 结构，不直接在消息管道中传输笨重的大体积二进制文件。 - 正文中如果涉及图片或视频，只内嵌对应的 `image_id`、`video_id` 或 CDN URL 链接。 - 最终的富媒体渲染交由客户端（如浏览器、App 或邮件客户端）异步拉取加载，从而极大减轻核心通知管道的带宽与存储压力。关于下游通知渠道，面试官指出系统至少需要覆盖四类核心 Topic/渠道： - **短信（SMS）**：纯文本、高时效性。 - **电子邮件（Email）**：支持富文本/HTML。 - **电话语音（Phone call）**：支持自动拨号并播放预录音频（Pre-recorded audio），业务逻辑尽量保持简单。 - **应用内推送（In-App）**：高频、轻量级、富交互。 ### [40:49](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=2449s) API 接口定义碰撞：是 Kafka Topic 还是模板（Template）？进入 API 设计环节，候选人首先写下了创建主题的接口签名：`createTopic(topicID, topicName, serviceType, metadata)`。面试官立即敏锐地追问底层概念：“你这里的 Topic 到底指代什么？你是打算在这里创建一个 Kafka Topic 吗？能不能举一个具体的使用例子？” ``` createTopic(topicID, topicName, serviceType, metadata) sendTopic(topicID, sendMode) subscribeTopic(topicID, subMode) ``` 候选人解释道，这指的是通知业务层面的主题。例如 Twitter 想要推送一种新的广告营销通知（Ads Campaign），而非普通的信息流推文，它就会调用该接口向通知系统注册一个新的主题：名称设为 `ads_campaign_1234`，服务类型为 `in-app`，元数据里包含投递优先级以及安全鉴权 Token 等。听到这个解释，面试官恍然大悟，指出候选人实际是在表达“通知模板”的概念：“哦，我明白了，你说的其实更像是一个通知模板（Notification Template），比如邮件模板或者短信模板吧？”候选人赞同了这一修正，确认这些模板与渠道配置信息会持久化存储在专用的元数据存储（Metadata Store）中。接着，候选人又写出了 `sendTopic` 与 `subscribeTopic`，并将发送模式（sendMode）细化为分布式系统中常见的保证语义： - 最少一次（At-least-once） - 最多一次（At-most-once） - 精确一次（Exactly-once）至于订阅模式（subMode），候选人坦言自己还没完全想透彻，初步设想是定义订阅的投递优先级（如 SMS 要求极速直达，而低优先级通道则可以批量攒批）。 ### [45:33](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=2733s) 交互模型理清与高层架构设计面试官对当前的 API 抽象提出了进一步的质疑：“我听得有点糊涂了。在你的模型里，一个 Topic 下面挂载着若干 Subscribers（订阅者）。这意味着每次发送一个 Topic，实际上就是广播给该 Topic 绑定的所有订阅者？如果我要精准控制发送，就必须通过 Subscribe API 动态维护这些订阅者吗？” 面对这个困惑，候选人试图解释终端设备通过调用 Subscribe 接口向通知系统注册接收能力，而发送方则通过通知系统扮演适配器（Adapter）角色分发消息。为了将抽象的讨论拉回具体的业务场景，面试官抛出了一个极为典型的多服务、多用户混合投递考题： > “你能解释一下你的系统如何支撑这样一个真实场景吗：服务 A 需要给用户 1 发送短信（SMS），服务 B 需要给用户 2 发送邮件（Email），服务 C 需要给用户 2 拨打电话（Phone call）？” 候选人表示，这正契合接下来要展开的系统架构。随后，候选人勾勒出了高层架构的核心数据流向：客户端发送请求，首先穿过负载均衡器（Load Balancer），接着分流到两个关键组件——一个是用于读取与校验模板配置的**元数据服务与存储（Metadata Service & Store）**；另一个是用作缓冲削峰的**临时消息存储（Temp Storage / 消息队列）**。缓冲池后端连接着派发服务（Sender），Sender 内部集成各类渠道适配器（Adapter），最终将短信、邮件与电话分流投递给对应的下游服务商。 ```mermaid flowchart TD accTitle: High-Level Notification System Architecture accDescr: Overall message ingestion, metadata resolution, queue buffering, and multi-channel delivery via adapters. subgraph Ingestion ["请求接入层"] Client["业务方客户端\n(Service A / B / C)"] --> LB["负载均衡器 (LB)"] end subgraph Storage ["路由与存储层"] LB --> MetaSvc["元数据服务\n(Metadata Service)"] LB --> TempStore["临时消息存储 / 缓冲队列\n(Temp Storage / Queue)"] MetaSvc -.-> MetaDB[("元数据存储\n(Templates / Topics)")] end subgraph Delivery ["分发与适配层"] TempStore --> Sender["发送调度引擎\n(Sender Engine)"] Sender --> AdaptSMS["短信适配器\n(SMS Adapter)"] Sender --> AdaptEmail["邮件适配器\n(Email Adapter)"] Sender --> AdaptPhone["电话语音适配器\n(Phone Adapter)"] end subgraph Channels ["外部服务通道"] AdaptSMS --> ExtSMS["短信网关 (User 1)"] AdaptEmail --> ExtEmail["邮件网关 (User 2)"] AdaptPhone --> ExtPhone["语音网关 (User 2)"] end ``` ### [49:07](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=2947s) 架构梳理：第三方适配器与客户端生命周期疑问面试者继续解释系统架构中的 Adapter（适配器）层。适配器充当系统与外部多样化设备（如 Email、电话等）对接的中间层，统一适配不同终端的协议。面试官对此表示认可，并指出适配器本质上就是一个第三方集成的统一抽象接口。面试官随后概括了目前的整套数据流：调用方发起请求携带元数据与消息，消息持久化到临时存储（Temp Storage Service）中以保证可靠性，接着投递给 Sender，Sender 经过 Adapter 调用外部第三方 API，最终触达并通知终端用户。 ```mermaid flowchart LR accTitle: Notification Pipeline accDescr: 消息从内部客户端出发，经元数据查询与临时持久化，再由Sender和Adapter异步推送到外部终端 subgraph Ingestion["接入与持久化"] A["Client / Internal Service"] --> B["Metadata Service"] A --> C["Temp Storage Service"] end subgraph Dispatch["调度与分发"] C --> D["Sender Workers"] D --> E["Adapters"] D -.-> F["Retry Service"] end subgraph Egress["终端触达"] E --> G["SMS / Email / Push"] end ``` 然而，面试官随即提出了一个核心的交互问题：在这个整体流程中，系统是否需要让 Client 一直挂起等待？系统究竟打算如何向发起请求的 Client 作出响应？面对面试官追问“如果我让你发个电话通知，你具体怎么处理请求”，面试者举了一个例子：比如把 Client 理解为发布推文的特朗普（Trump），当他发推时，底下的所有粉丝都应该通过短信或应用内通知收到提醒，因此这里的 Client 就是发布者。面试官纠正了这一视角，把问题拉回工程本质：从调用通知服务的客户端视角来看，完整的请求生命周期究竟是什么样的？ ### [51:20](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=3080s) 认知纠偏：内网服务边界与鉴权简化面试者开始从网络传输切入分析生命周期，提出 Client 端会经过反向代理（Reverse Proxy）来解密 HTTPS 请求、进行格式规范化以便内网使用，还会通过本地缓存向 Metadata Service 查询对应的 Topic 与元数据 JSON。此外，面试者还提到在 Client 端或进入负载均衡（Load Balancer）前，将指标（Metrics）与日志打到本地磁盘，以满足前面提到的“实时监控仪表盘”需求。此时面试官介入并纠正了设计方向。面试官指出，把 Dashboard 强行挂在日志和本地 Metrics 之上并不合适。更关键的是，面试官澄清了上下文背景：这个通知系统是公司内部的基础设施工具（如在 Facebook 或类似规模的企业内），调用方（Client）全部都是已认证的内部服务（如 Service A、Service B），而不是外网不可信的终端用户。内部服务互相调用时，根本不需要复杂的外部网络身份认证（Authentication）与授权（Authorization）流程，也不需要在入口处搞繁复的反向代理。 - **调用场景定位**：内部微服务通信（Internal Service to Service），客户端本身即为受信任组件。 - **职责精简**：移除冗余的公网授权认证与复杂的 HTTPS 解密代理层。 ### [56:15](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=3375s) 异步解耦设计：Temp Storage、无状态 Sender 与重试机制理清客户端属于内部服务后，面试者调整了请求分发与存储流向。一个进入系统的内部通知请求主要触发三个动作： 1. 请求 Metadata Service 获取 Topic 详细定义； 2. 写入临时存储服务（Temp Storage Service），保障消息持久性（Durability），即便订阅者离线，重新上线后仍能拉取消费； 3. 将事件推送至 Kafka 与 Apache Flink 进行流式监控统计。针对 Temp Storage，面试者提出采用高吞吐读写且水平可扩展的键值（Key-Value）数据库，并提议选用 DynamoDB。对于消息发送端（Sender），面试者建议使用基于 WebSocket 或工作线程池（Worker Pool）的可伸缩架构，通过 Adapter 路由至不同终端设备。如果下游发送积压，直接扩容无状态的 Sender Workers 即可。面试官敏锐地抓住了架构精髓，提醒面试者在系统设计中一定要明确强调关键词——**"asynchronous"（异步处理）**与 **"separation of concerns"（关注点分离）**。面试者进一步补全了可靠性设计，引入 Retry Service（重试服务）来兑现需求中规定的“At-least-once”（至少交付一次）语义。当 Sender 发送失败时，任务会转入重试队列中由重试服务接管调度。 ### [01:00:00](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=3600s) 消息端到端状态机与异步 Receipt 回执面试官随后将讨论聚焦在生命周期的具体推演上，给出一个具体场景：“假设我是 Service A，想给 User 1 发送一条消息，我调用你的 API 并传入元数据与消息体，之后的一步步状态是什么？我什么时候知道消息已经被你们系统接单记录？我又在什么时候能确切知道消息已经真正送达？” 针对确认时机，面试者给出了清晰界定： - **已记录（Recorded）**：当消息被成功持久化写入 Temp Storage Service 的瞬间； - **已送达（Sent/Delivered）**：当消息经 Adapter 真正推送到终端并收到订阅端的确认响应（ACK）时。在此基础上，面试者将通知在客户端视角的流转划分为三个关键状态： - **Pending（等待中）**：客户端发出调用，但通知系统尚未完成持久化受理； - **Sending（发送中）**：消息已安全落库至 Temp Storage，Sender 正在分发调度，但尚未完全触达所有目标接收者； - **Completed（已完成）** / **Partial Completed（部分成功）**：Sender 已完成全部发送，若遇到个别不可恢复异常则标记为部分完成或失败（Failed）。面试官对这个状态机表示赞同，并指出最核心的交互模式：客户端绝不会同步挂起两分钟去干等通知发出。当请求经过校验并落入持久化存储（即处于 Sending 阶段）后，系统就应当立刻向内部客户端返回一个回执（Receipt / 任务 ID），释放连接，后续的所有推送与重试完全在后台异步流转。 ### [01:07:56](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=4076s) 客户端回执（Receipt ID）与异步轮询机制面试官针对通知系统的请求处理闭环提出了关键考量：系统绝不能永远挂起（hold）客户端的 HTTP 请求。正确的做法是在服务端完成核心持久化（如步骤五）后，立即向客户端返回一个回执 ID（Receipt ID / Request ID）。此后，客户端可以通过主动轮询（polling）该回执来查询当前通知的实际处理状态和最终完成情况。候选人询问为何不能在系统处于 Pending 状态的极早期阶段就直接返回回执。面试官详细解释了系统崩溃与状态一致性的风险：如果在消息尚未真正落盘存储时就返回响应，一旦后端 Worker 或服务器突然宕机，由于没有留下任何持久化记录，后续请求就会彻底丢失，或者导致系统长时间卡在 Pending 状态而无法自愈。更现实的情况是，客户端自身可能随时关闭 Socket 连接，服务器进程也可能崩溃。因此，必须在消息被可靠写入存储和状态表后，再将回执交给客户端，以此保证系统具备确定性的跟踪能力。 ```mermaid flowchart LR accTitle: Client Receipt and Polling Flow accDescr: 客户端发起请求后，系统完成持久化并返回 Receipt ID，客户端随后轮询状态。 subgraph ClientSide["客户端"] C["客户端 Client"] end subgraph Ingestion["接入与持久化"] S["API 服务 / Worker"] DB[("消息持久化存储 & 状态库")] end C -->|"1. 提交通知请求"| S S -->|"2. 消息落盘与初始化状态"| DB DB -->|"3. 确认写入完成"| S S -->|"4. 立即返回 Receipt ID"| C C -.->|"5. 异步轮询当前状态"| S ``` --- ### [01:12:40](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=4360s) 数据模型设计：临时存储与元数据分离进入系统的 Schema 设计环节。面试官引导候选人回顾类似 Slack 架构的成熟实践：将系统拆分为用于存储消息体本身的“消息/临时存储服务”（Message Storage Service），以及专门负责生命周期与交付追踪的“元数据服务”（Metadata Service）。在设计临时存储表时，候选人与面试官讨论了 DynamoDB 等 NoSQL 键值存储的适用性： - **Partition Key**：`message_id` - **Sort Key**：`timestamp`（时间戳） - **属性字段**：`topic_id`、`sender_id`、`content` 等消息体数据。针对通知状态（Status）到底该放在哪张表，两人展开了探讨。如果同一个 Topic 下会产生多条不同 Message ID 的消息，或者多条通知有着各自独立的递送进度，那么状态就属于运行时的生命周期数据，不应与静态的消息内容混在一起。临时存储本质上是轻量、高吞吐的 Key-Value 查找（“给你一个 message ID，返回具体消息内容”）；而生命周期的流转与状态追踪，则由元数据服务全权承担。 - 消息存储服务侧重于高并发写入、读取以及消息持久性（Durability）。 - 消息元数据与递送状态解耦，避免因频繁更新状态而导致宽表产生写入热点或放大开销。 --- ### [01:18:18](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=4698s) 元数据表的事务控制与生命周期管理在确定元数据服务（Metadata Service）的技术选型时，双方达成一致：元数据更适合直接存放在关系型数据库（SQL Database）中。因为通知的生命周期管理需要严格的事务保障（Transactions），例如保证状态变更的原子性，确保消息最终必然被送达或标记失败。面试官建议将发送模式（`send_mode`）和目标服务类型（`service_type`）从消息体抽离出来，统一放入元数据表中： - **核心主键**：`message_id` - **接收者信息**：`receiver_id` - **控制属性**：`status`（如 Pending、Sending、Completed、Failed）、`send_mode`、`service_type` - **运维与审计字段**：`updated_at` / `timestamp` 时间戳字段。该时间戳至关重要，系统监控可以通过它精准排查出“究竟哪条消息在哪种状态下卡了多久”，便于实现超时重试或报警。通过 SQL 数据库的事务机制，系统在推进通知生命周期各阶段时能够维持严格的一致性，防止出现状态漂移或幽灵消息。 --- ### [01:20:55](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=4855s) 端到端单条通知生命周期的梳理面试官要求候选人花三分钟时间，从系统内部视角详细串联一条通知完整的 Life Cycle，理清它与外部客户端视角的差异。以一条具体的邮件通知为例，其生命周期演进过程如下： 1. **Creation（生成阶段）**：上游业务触发请求，通过邮件模板渲染生成具体的邮件内容与配置。 2. **Pending（待发送）**：消息写入持久化临时存储，元数据服务在 SQL 数据库中落库，并将状态初始置为 `pending`。此时服务端向客户端返回 Receipt ID，释放连接。 3. **Sending（投递中）**：调度系统或 Worker Pool 从队列中获取任务，向实际的发送渠道（如外部邮件网关服务）发起调用，同时将元数据状态更新为 `sending`。 4. **Completed / Failed（终态）**：收到下游渠道明确的成功回调或到达重试上限后，系统将状态流转为 `completed` 或记录错误码并标记为失败。 ```mermaid flowchart TD accTitle: Notification Life Cycle accDescr: 展示通知从创建、Pending、Sending 到最终状态的生命周期流转。 Created["1. 模板渲染与创建 Creation"] --> Pending["2. 写入存储并标记 Pending (返回 Receipt ID)"] Pending --> Sending["3. Worker 消费并标记 Sending (调用发送渠道)"] Sending --> Success{"4. 投递结果判定"} Success -->|"成功"| Completed["Completed (成功终态)"] Success -->|"不可恢复错误 / 耗尽重试"| Failed["Failed (失败终态)"] Success -->|"临时故障"| Retry["重试逻辑 (重新排队)"] Retry --> Sending ``` --- ### [01:25:37](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=5137s) 面试复盘：分布式投递语义与容错原则在面试进入总结反馈（Debrief）阶段时，双方重点复盘了分布式系统中的投递语义保证（Delivery Guarantees）。候选人提出了在系统某些步骤中实现 At-most-once（至多一次）或 At-least-once（至少一次）的权衡。面试官强调了分布式系统架构的根本现实： - **无处不在的故障**："In a distributed system, every single step can fail."（分布式系统中每一个单点步骤都有可能失败）。网络分区、Worker 宕机、第三方通道超时都是常态。 - **投递语义的选择**：工业界绝大多数通知系统默认选择 **At-least-once**（至少一次），配合消费端的幂等性（Idempotency）以及去重逻辑来兜底；试图强行做到绝对的 Exactly-once 或依赖脆弱的 At-most-once 往往会导致严重的丢消息风险或极高的架构复杂度。随后，面试官系统梳理了设计这类高并发分布式通知系统的核心架构哲学（Philosophy）： - **关注点分离（Separation of Concerns）**：将消息内容的持久化存储与运行时的生命周期状态解耦；将接入路由与具体渠道适配隔离。 - **异步处理与线程池（Asynchronous Processing & Worker Pool）**：核心主链路非阻塞，依赖可靠的消息队列与 Worker 资源池削峰填谷。 - **事件驱动（Event-Driven）**：基于事件通知触发各阶段状态流转。 --- ### [01:31:53](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=5513s) 复盘深入：适配器模式、重试机制与状态转换面试官针对候选人在设计各功能模块时的技术细节给出了进一步的复盘指导： - **渠道适配器（Adapter Pattern）**：由于上游通知类型多样（Email、SMS、APNs、Webhook 等），必须抽象出适配器层（Adapter）。适配器屏蔽底层各个第三方推送网关（Router / Vendor）的异构 API 规范，向上提供统一的发送接口；适配器内部可处理协议转换与低级状态码映射。 - **重试机制与退避策略（Retry Logic）**：针对第三方服务不可用或网络抖动，必须配置单服务专用的重试逻辑（如指数退避、死信队列 Dead Letter Queue），防止下游抖动打垮上游调度器。 - **细粒度状态转换与错误码（Status Transition & Error Handling）**：在真实生产环境中，通知状态不应只有粗暴的成功与失败，还需要支持部分成功（Partially Completed）、携带内部错误码（Internal Error Code）的失败状态，以便上游对超时、限流、地址无效等情况做出分类应对。 - **批处理与分组通知（Chunking / Group Notification）**：对于大规模群发或批量场景，系统需要具备分块（Chunking）与批量聚合处理能力，以平衡数据库吞吐量与投递延迟。整场面试在对分布式稳定性与工程落地细节的深入交流中圆满结束，候选人对整个系统设计链路中原本含混模糊的状态机和解耦逻辑有了清晰的认识。 ## Notable quotes > [01:25:37](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=5137s) In a distributed system, every single step can fail. — 面试官 > [34:06](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=2046s) 你好像还没问我业务规模是多少呢。让我们假设这是为 Facebook 设计的通知系统。 — 面试官 > [34:06](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=2046s) 你少写了一个零。 — 面试官 > [29:58](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=1798s) 你这里说的 Push 和 Pull 模型，具体指的是什么？ — 面试官 > [40:49](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=2449s) 你这里的 Topic 到底指代什么？你是打算在这里创建一个 Kafka Topic 吗？能不能举一个具体的使用例子？ — 面试官 > [40:49](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=2449s) 哦，我明白了，你说的其实更像是一个通知模板（Notification Template），比如邮件模板或者短信模板吧？ — 面试官 > [45:33](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=2733s) 你能解释一下你的系统如何支撑这样一个真实场景吗：服务 A 需要给用户 1 发送短信（SMS），服务 B 需要给用户 2 发送邮件（Email），服务 C 需要给用户 2 拨打电话（Phone call）？ — 面试官 > [01:00:00](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=3600s) 假设我是 Service A，想给 User 1 发送一条消息，我调用你的 API 并传入元数据与消息体，之后的一步步状态是什么？我什么时候知道消息已经被你们系统接单记录？我又在什么时候能确切知道消息已经真正送达？ — 面试官 ## Mentioned Scott Shi, Uber Money, DynamoDB, LedgerStore Gateway, Airflow, Piper, RTAPI, APNs, FCM, Twilio, SendGrid, Kafka, LinkedIn, Facebook, Twitter, Apache Flink, Slack, Donald Trump ## Video description in this session, we mocked a system design interview doc could be found here: https://docs.google.com/document/d/140n69Pw-xqoD3eeGzcv2WH_CDRbgYmYzV6OPHl6NzA8/edit?usp=sharing [Source](https://www.youtube.com/watch?v=D0bBMzxE5O0)

Source quotes

[85:37] 面试官: In a distributed system, every single step can fail. [34:06] 面试官: 你好像还没问我业务规模是多少呢。让我们假设这是为 Facebook 设计的通知系统。 [34:06] 面试官: 你少写了一个零。 [29:58] 面试官: 你这里说的 Push 和 Pull 模型，具体指的是什么？ [40:49] 面试官: 你这里的 Topic 到底指代什么？你是打算在这里创建一个 Kafka Topic 吗？能不能举一个具体的使用例子？ [40:49] 面试官: 哦，我明白了，你说的其实更像是一个通知模板（Notification Template），比如邮件模板或者短信模板吧？ [45:33] 面试官: 你能解释一下你的系统如何支撑这样一个真实场景吗：服务 A 需要给用户 1 发送短信（SMS），服务 B 需要给用户 2 发送邮件（Email），服务 C 需要给用户 2 拨打电话（Phone call）？ [60:00] 面试官: 假设我是 Service A，想给 User 1 发送一条消息，我调用你的 API 并传入元数据与消息体，之后的一步步状态是什么？我什么时候知道消息已经被你们系统接单记录？我又在什么时候能确切知道消息已经真正送达？

Source quotes

[20:29] Scott Shi: LinkedIn was fined millions of dollars because it sent promotional emails repeatedly [33:23] Scott Shi: Let's assume that we are designing the notification system for Facebook... you have 200 million active users per day, and each user has five notifications per day on average. [33:23] Scott Shi: You're missing a zero. [37:09] Scott Shi: If you're going to send emails, you're going to have images, right?... and you may have possibly video, but most likely you're going to be an HTML file like multimedia. [40:44] Scott Shi: Oh, I see. So you're referring to more like a notification template, right? Like an email template. [45:33] Scott Shi: I'm a little confused... could you please explain how your system is going to model the following scenario? Like Service A sends SMS to User 1, Service B sends email to User 2, and Service C sends phone calls to User 2? [49:07] Candidate: for example, Trump send a tweet here, I think Trump is the client, and the sender go all the way to different SMS, it means that all the followers of Trump would receive the SMS or the tweets. [51:20] Scott Shi: I think all the request is actually internal service... like you build a tool for all your company's service. [59:08] Scott Shi: Once you finish out the design, I do want to spend five minutes explain the life cycle for a message, a life cycle for notification, and life cycle for a client request. [63:04] Scott Shi: When do I know my message is sent, and when do I know my message is received by you? [67:56] Scott Shi: You don't want to hold this request forever. [78:16] Scott Shi: You need transaction, right? [78:16] Scott Shi: So you know which message has been stuck at what time for how long. [80:55] Scott Shi: Let's try to describe what's the life cycle for notification looks like. [85:40] Scott Shi: Which part you did good and which part you didn't do well? [85:40] Scott Shi: every single step can fail

Original transcript

[0:00] Rec [0:05] record [0:13] [Music] [0:24] uh [0:28] okay top [0:37] consistency [0:52] modil always read the latest right um [1:28] replication compe [1:30] master [2:07] see I seeability [2:10] [Music] [2:31] lililililil [3:24] oh see I see [3:32] linear ability [4:06] relability transaction [4:38] compare and [4:40] set [4:42] compare compare [4:53] sentar [4:56] linear linear compar consensus [5:06] lineability [5:16] read linear right linear [5:20] comparar comp [5:22] consensus requirement [5:34] right compar set I [5:48] [Music] [5:53] see conser [6:22] guarante operation [6:38] uh block rate block rate or block [6:54] right snapshot [6:57] isolation guarantee lineability [7:02] snation guility [7:44] laril guarante calization guarante [7:47] lineability [8:26] system design interval [8:34] system P [8:50] project [8:58] problem datab [9:05] problem solving [9:35] request [9:37] [Music] [9:44] access send [10:00] share [10:14] screen or sh again oh request oh F [10:20] request um okay let's get started so [10:23] um okay so um um so first let's talk [10:28] about um some project you did in the [10:33] past most interesting project technical [10:36] challenging [10:39] project technical challenge business [10:43] business impact project [11:05] challenging challenging sorry technical [11:08] most [11:10] challeng let's please spend like maybe [11:13] five minutes to describe technical most [11:16] challenging project you did in the past [11:18] um um during his work um maybe let's [11:22] let's not include school project also [11:25] here [11:33] reference point oh okay I'm currently in [11:36] over Money Team so the payments dealing [11:38] with the payments so my most project um [11:41] most recent projects are mostly about [11:44] payments so the most you asking about [11:47] the most difficult in Tech aspect or [11:50] business aspect Tech perspective let's [11:53] let's actually answer some tech related [11:55] project so Tech perspective maybe um [11:57] yeah so so Tech perspective I think it's [12:01] last year I did some migration of the uh [12:06] payments Grant storage uh from the uh [12:10] from the Dynamo uh stand along Dynamo DV [12:14] into the leder St Gateway uh integration [12:17] so I think this one is relatively tax [12:20] challenging um so it includes the in the [12:25] connection with the [12:27] demarketing uh about talking about the [12:30] negotiation of SLA for the for rer Star [12:35] Gateway and also how to um Sim simly [12:41] similarly migrate from the grand store [12:44] Lexi Dynamo into the new Ledger store [12:48] Gateway um so this project takes around [12:52] three months to finish yeah so what's [12:55] the main difficulty in the project the [12:58] main difficulty I think uh is the [13:02] fallback strategy for read and write uh [13:05] because you have to uh read uh read in [13:08] order from the grand store to uh to make [13:11] sure it's lossless uh it's most [13:14] difficult point you have to uh first uh [13:17] uh ride to the uh Ledger store gateway [13:21] then to the Grand store and then refers [13:24] the order to make sure that there's [13:26] nothing uh lost during this process and [13:30] you have make uh write uh lot of [13:33] validations uh to make sure that during [13:36] this time uh during this process records [13:38] are checkable so that's a most most [13:42] difficult part I think yeah okay so you [13:45] did like d du right and then disable one [13:48] right and double check like Shadow right [13:50] and then reverse the shadow right [13:51] something like that right yeah okay okay [13:54] that's great so [14:01] dat [14:02] migration [14:22] migration flag flag flag [14:42] did you actually do it yourself all by [14:44] yourself or actually you got Mentor from [14:46] other engineer or you know what's your [14:48] role in this project U we have uh some [14:51] previous similar projects that's half [14:54] completed so I could check some [14:56] materials from them most done by myself [15:00] and to and got help from the staff [15:02] Engineers also yeah okay that's great [15:05] and also how did you handle historical [15:08] data did you actually write a script or [15:10] did you write a job or how did you [15:12] migrate existing historical data not not [15:16] you know dite only handles the new [15:18] coming data but not a historical old [15:20] data historical data we set a time stamp [15:24] so some some hardcore temp stamp to set [15:27] before that we stop read stop writing to [15:30] it because the gr has a a limitation of [15:34] um 6 days expiration time so before the [15:38] six days expiration time we don't need [15:40] to care about the writing anyal or [15:43] modification of it so we kill the [15:46] writing to it um briefly and making it [15:51] like separately it makes us easier to [15:54] handle oh okay so basically you don't [15:57] need to do the historic don't have [15:59] migration yeah you don't need to have [16:00] like Piper job airflow job to to migrate [16:04] like B okay okay that's good [16:35] expect [16:42] Implement skill [16:55] problem Talent skill [17:06] Independence [17:07] Mig [17:11] Right add some color to [17:16] it articul articulation [17:38] most challenging [17:50] project [17:53] project Lev [18:07] Implement [18:12] flow Implement [18:16] Flow Design Implement new air [18:20] flow project [18:51] new Uber dispatch system get rid of this [18:55] rtpi or dispatch existing dispatch [19:01] business [19:05] impact [19:23] migration [19:25] engineer engineer [19:49] stop stop word compiler [19:59] technal focus on [20:03] business compiler stop word garbage [20:18] collection Focus buiness bus impact [20:29] uh okay let's start interview or do you [20:31] mind share your screen so we can start [20:46] okay [20:57] system forem [21:33] [Music] [21:43] [Music] [21:46] system conversation knowledge [22:14] conversation okay uh so today let's try [22:17] to design a notification system assume [22:20] that you want to design a system for [22:22] your company and support all kinds of [22:24] notification the notification including [22:27] like in app notification like Apple or [22:29] Android building notification email [22:33] notification phone notification message [22:37] modification and because that you may [22:39] want to use some other third party [22:41] services like tro or sent grade C I [22:45] think it's for email um so uh so you [22:48] want to make sure you support [22:50] integration this integration as well and [22:53] also uh uh let's actually try to support [22:57] more than one delivery message means [22:59] that we can have at least once delivery [23:02] maybe at most once delivery and exactly [23:04] once delivery that's actually quite [23:05] important um because link was actually [23:08] fund for a lot of millions of dollars [23:10] because it send a promotion email for [23:13] more than months in 2015 and 2014 okay [23:16] and also unified unified interface for [23:19] other service to use your system so [23:22] basically you may other hand end point [23:25] or you could be a Kafka topic and other [23:27] service kind call the notification [23:29] service call your notification Service [23:32] uh with um parameters you [23:34] specify and last thing that you want to [23:37] make sure your system is monitored uh [23:40] the health is monitored like uh I'm not [23:43] talking about like Docker like not [23:45] talking about Docker CPU memory disk [23:48] monitors here it's more like business [23:50] oriented it's more like a onitor like [23:53] how many notification are sent and [23:56] what's the progress like how many [23:58] notific [23:59] and for each like each vendor and each [24:02] uh like email let's let's say we have [24:06] 100 email notification uh stuck there so [24:10] if it's stuck for a long time you know [24:13] you you want to be able to visualize it [24:15] and so create a visualization for the [24:19] how many notification are sent in [24:21] progress and C basically that's that's [24:22] the [24:24] requirement um let's first begin uh [24:27] start with is the business case business [24:30] use case [24:33] um and then I will go to the [24:38] constraints um then I will go to [24:42] the um API [24:45] design uh and and [24:48] uh and the [24:51] architecture um [24:54] uh and finally some data schema if we [24:57] have time [24:59] um so for business use [25:02] cases um I want [25:05] to uh first line out the [25:08] MVP [25:10] um features that we have um [25:21] so um so the MVP could [25:24] be um [25:28] uh [25:29] deliver delivery of [25:33] notifications uh for uh [25:36] for uh um for variables uh various of uh [25:42] farious of [25:45] um [25:47] receivers uh the receivers could be uh [25:51] apps uh email or phone or SMS [25:58] and uh the [26:01] delivery uh delivery [26:04] support uh different uh [26:08] different of um mode uh [26:14] like uh at least [26:17] once at most [26:20] once um and uh I [26:23] think um there also [26:27] bonus uh that we could good to have and [26:32] uh is that delivery support [26:36] for uh for uh exact [26:41] ones and there are non goals yeah so I'm [26:46] curious what do you think is difference [26:48] between at least once at most once and [26:50] exactly once um at least once I think uh [26:55] you at least once could be achieved by [26:58] uh retri cues and it uh it requires [27:02] durability um and at most ones means [27:05] that you could fair and forget under [27:08] this case it doesn't need to go through [27:11] a retry que exactly on means that you [27:15] have to um and and the delivery also [27:19] requires differ for example the [27:21] decoupling of the sender and the the [27:25] receiver um for if you use kfka uh you [27:30] should Al also use a corresponding at Le [27:32] once at most once or exactly once model [27:36] uh if the previous day things uh like [27:40] this um and in the requestor side name [27:44] the client side um at most ons or [27:48] exactly ones also means that uh you need [27:52] to um make uh make the make a request be [27:57] item potent uh you need to fill out [28:00] early maybe on the on the load balancer [28:04] stage or earlier make a filter to uh [28:08] filter out um duplicate submissions [28:13] early uh and make extra processings [28:16] later [28:17] maybe is that right oh so actually the [28:21] concept here exactly ones is different [28:23] from kka exective ones the concept here [28:26] is exective ones means that if I send a [28:29] request I expect you receive email [28:32] exactly on but that's difficult because [28:34] it's actually two system two system [28:36] transaction it's like transaction across [28:38] two system right so for example because [28:42] it's a it's a um we most likely going to [28:44] be build this as a web service and uh if [28:48] I say uh if I request your service send [28:51] email to send email to me right and your [28:53] email you're going to call third party [28:55] API third party API and then if thir [28:59] about API time out for whatever reason [29:03] no response you don't really know um if [29:06] the email is sent or [29:08] not so yeah so uh if they want to have [29:12] very strong control exective once uh it [29:15] requires like uh it requires like a [29:18] two-phase uh uh two-phase uh commit [29:21] something like [29:23] that there is a [29:25] uh uh there is a I think there [29:28] architecture very old architecture for [29:30] it U uh transaction x x transaction or [29:35] something but it's it's a it's a [29:38] difficult so let's skip that part it's [29:41] it's fairly difficult and if depends on [29:43] if you have some system you cannot [29:46] control then like if the integration [29:48] third party system you cannot control [29:51] then it's basically impossible okay um [29:56] and that's fair that's fair the divid as [29:58] bonus yes and bonus also includes um MVP [30:04] also [30:05] includes uh push and P [30:08] model [30:11] um for active um for active active [30:18] subscribers and uh idle [30:24] subscribers uh for example when when the [30:28] subscribers are active you need to push [30:31] them out um but some subscribers for [30:35] example you you have um you have apps or [30:40] iOS that's closed or didn't receive any [30:44] notifications when you open your iOS or [30:47] reopen the app uh maybe you should uh [30:52] receive some background uh or some apps [30:56] that's running some background Str as [30:58] you should they they would pull the [31:01] notification surface and you will [31:04] receive multiple notifications when you [31:07] open your iPhone things like that so [31:10] that's one of MVP also and bonus include [31:16] also so by push and P model here I [31:20] understand what's active and ID [31:22] subscriber by push and P model here what [31:25] are you referring to um it means that [31:28] beside the pings uh beside pushing um [31:33] every time after every time as uh sender [31:36] send out uh the requestor could also do [31:40] some kind of Yap or JPC call directly to [31:44] retrieve the message uh message from the [31:46] data store maybe oh okay I [31:50] see um and the the bonus could have [31:54] could also be re recuring [31:59] notification [32:01] or it's especially useful for the emails [32:06] and SMS and the scheduled [32:12] notification um and for n goals I think [32:16] the first one is about [32:19] latency uh for delivering the [32:24] magey notifications [32:28] uh we don't uh require uh delivering of [32:31] notifications every message to be [32:34] immediate uh some of the notification [32:36] are high [32:38] priority um some of them are not uh for [32:41] low priority messages for example [32:44] Twitter messages we often see they are [32:47] delivered [32:49] several Twitter [32:51] notifications in the app we often see it [32:54] deliver several hours later uh so and uh [32:59] we don't need to maintain the [33:05] order order of [33:09] delivery or maintain the order of [33:13] notifications um and uh for [33:23] for for the performance [33:29] uh for the conings um conings I [33:35] think uh we [33:37] need High [33:41] availability U because um we want uh the [33:44] notification we could send a request to [33:48] request notifying our subscribers [33:50] anytime we we want we don't want to [33:55] sometimes we find it impossible to send [33:57] out notifications or especially very [34:00] important [34:01] notifications and high [34:05] scalability actually you haven't asked [34:07] me the skill so let's assume that we we [34:10] are designed the notification for [34:11] Facebook so you have um oh okay so um [34:15] 200 million active user and maybe five [34:18] five notification per day um let's get [34:23] some [34:25] um would you mind to repeat yeah yeah [34:27] let's let's assume that's a notification [34:30] for system for Facebook so you have 200 [34:32] million active user per day and each [34:35] user has file notification in [34:38] average 200 million you're missing Zer [34:42] yeah 200 million active user and then [34:45] per day and then you have file [34:47] notification per day per [34:51] user um five minute five file [34:54] notification per user [34:57] per [34:59] day [35:01] okay um it means that if these are all [35:06] push uh methods that means that um 200 [35:11] million um multipli by five [35:15] notifications um and divide [35:18] by [35:20] um 24 [35:23] hours 6 minutes 6 seconds [35:28] um [36:12] pick [36:18] request [36:26] okay okay okay this looks good to [36:29] me um and we want let me go with this um [36:36] so it means that the the reading uh the [36:39] reading cannot be uh within one machine [36:43] uh it could easily exceed the limitation [36:46] of the one host and we need either [36:50] shading or you need um horizontal [36:56] scalability um so with that we have uh [37:03] also we need High availability has [37:07] scalability and [37:09] flexibility uh means that we need to [37:12] support different senters and different [37:15] receivers uh senders could uh varies [37:19] from uh some like um different topics um [37:26] I wonder how many topics uh in total do [37:29] we have different kinds of topics in [37:31] total do we have uh what topic you're [37:34] going to create a [37:37] Bas [37:39] um oh would you mind would you mind to [37:41] repeat so how you how are you going to [37:44] create topic what topics are you going [37:46] to create based on um I'm going to [37:49] create topic uh based [37:51] [Music] [37:53] on topic name um and the surface [38:00] type and [38:02] uh with um topic and some other topic [38:07] meta metadata [38:10] information and the message itself uh it [38:13] want to send and the message itself want [38:16] to send may [38:17] include um and and one thing uh not non [38:22] goal is that we will just send message [38:25] no no images is that [38:29] allowable yeah so I think in here if [38:31] you're going to send emails you're going [38:33] to have image right um so image is a [38:37] bonus image we can yeah we can add as a [38:40] bonus because if you're going to send [38:42] email you have image here and you may [38:45] have Poss video but most most likely [38:48] you're going to be a HTML file like [38:50] multi media oh okay yeah um so maybe [38:56] some message IDE these uh to refer from [39:00] um mag messages um for example if you [39:03] invent some um some kind of videos or [39:07] some kind of um images uh we could send [39:11] a message like HTML object uh body [39:15] directly invented it with um the video [39:19] ID or the image ID and the render could [39:24] basic load this kind of thing later in [39:26] the CL side [39:28] um so the message itself is a pure text [39:32] version so this is a format of a topic [39:36] itself um and of course topic ID um so [39:42] um I wonder how many topics namely [39:44] different kinds of [39:46] notifications that's the notification [39:49] system could support um do do you [39:52] think yeah I think we need at least one [39:55] topic for SMS like message uh we need [39:58] one topic maybe for [40:01] um um email and another message another [40:05] topic [40:06] for um phone maybe I think phone is [40:10] another one then the last one is in app [40:13] for phone one you probably once you once [40:15] you deal with phone you probably want to [40:17] um PPE in pre-recorded U pre-recorded U [40:22] audio let's keep it simple just a [40:25] pre-recorded audio something like that [40:28] oh [40:29] okay um and uh High availability High [40:34] scalability [40:36] flexibility [40:39] and what other [40:41] things um let's keep it here yeah I [40:44] think this is good let's actually go to [40:45] the API design architecture or database [40:48] I think looks good so far so [40:53] okay API design [40:58] API design is [41:01] like um we have [41:04] um let's [41:06] design [41:08] um create [41:11] topic um with a topic uh [41:15] topic um topic ID with a topic [41:22] name um and as we have the topic here [41:28] um we need [41:30] to surface [41:34] type metad data so what's the topic here [41:37] you are referring to is this a kka topic [41:39] you're going to um I think it's a [41:42] notification topic that we are trying to [41:45] create um so you give an example um for [41:50] example [41:51] um for example [41:54] um notification we [41:58] um for example app [42:02] like um like Twitter wants to send its [42:07] notification and it wants to send a new [42:11] version of the topic uh not feeds topic [42:14] but some kind of adds campaign topic [42:17] into [42:19] the uh into the notification service [42:22] then under this case it will add a topic [42:26] name like [42:27] um as campaign uh ID um one two 3 4 and [42:35] the surface type is in [42:37] app and some metadata de priority uh [42:43] like [42:45] um this kind of this kind of topic uh [42:49] should be [42:50] sent um should be sent uh like uh [42:55] security um token [42:58] security metadata um seems like that um [43:03] with this kind of notification [43:06] sent um oh I see so you're referring to [43:10] more like a notification template right [43:12] it's like email templat okay some some [43:15] kind of this so you could store this [43:18] kind of things in one um either metadata [43:22] store or the topic [43:24] store [43:25] um and the beside the create [43:29] topic I think there's [43:34] a [43:39] send send topic [43:46] mode and [43:54] this subscribe topic I [44:03] mod so can you can you give me some [44:05] example like sand ion sand what's sand [44:09] but what's the value for sand modde um [44:14] so um the topic ID is the corresponding [44:19] topic you want to send the template and [44:22] the send mode is um what we have [44:25] mentioned is um [44:27] is [44:29] um some kind of at at least [44:34] ones at most ones I see uh ex secondary [44:44] ones and the sub [44:48] mode uh is some kind of different I [44:51] think the sube is kind of we want [44:55] different prioritize uh priority uh for [44:59] the subscription U for example some kind [45:03] of SMS we want it to be delivered [45:06] directly and some of them we want it we [45:09] don't care about the latency of it so [45:13] maybe I haven't thought too much about [45:16] the subscription mode that we could have [45:19] um let's put it here but basically what [45:22] I think what I could think is the uh [45:26] this all the three apis that we could [45:30] use for notification system okay so I'm [45:33] little I'm little confused if I'm [45:36] understanding right so for your part you [45:38] have topic and topic have [45:42] subscribers subscribers yeah and so [45:45] basically when you send topic you just [45:47] need to send subscribers and you can use [45:50] subscriber you can have subscribe API to [45:53] manipulate subscribers is that what you [45:55] mean uh [45:58] yes um so the different it um so [46:02] different devices will always listen to [46:05] the [46:05] apis um so it it doesn't mean that we [46:10] the [46:11] subscribers U the different apis could [46:13] call a subscribe API directly to Subs to [46:17] make a [46:18] registration uh in the notification [46:21] system so a notification will send to [46:23] them and uh for example you have making [46:27] a phone call um the phone call the the [46:31] sender uh will send to the notification [46:35] system instead of the phone call um the [46:40] the phone service will send send a [46:42] request directly the sender as a will [46:46] serve as an adapter uh to adapt to all [46:50] different kind of [46:54] services yeah so uh could you please [46:57] explain how your how your system going [46:58] to model the following scenario like [47:00] service a send SMS to user one service B [47:03] send email to user to and services and [47:06] phone calls to user [47:08] to um so this kind of thing is like um [47:13] so it's in fact included in the [47:16] architecture below um I [47:19] would I would include I would um explain [47:24] about this okay sounds good um so so [47:29] the this is a [47:31] client and the client uh goes through a [47:35] load [47:37] balancer and briefly speaking client go [47:41] to a road balancer and uh also um [47:46] there's kind of Meta Meta data [47:49] store metadata storage um for storing [47:54] the [47:55] topic and other metad datas and uh [48:00] meanwhile after load balancer there are [48:04] um kind [48:05] of [48:07] um Temp [48:11] Storage to store the messages to send [48:16] and [48:20] uh with with this [48:28] metadata surface and the Temp Storage [48:34] surface and with [48:37] this [48:43] um with this Temp Storage surface we go [48:48] to [48:49] the cender [48:58] send the with the adapter to different [49:01] um [49:05] SMS uh [49:07] email forone uh things like that so in [49:12] fact it's adapter that goes to all the [49:17] all different uh different uh kinds of [49:21] devices at in the [49:25] last does that s good good yeah so yeah [49:28] I see I see adapter here is a third [49:30] party integration uh basically just a [49:33] interface yeah okay makes sense um so [49:37] I'll move it [49:43] here um [49:46] so so yeah let's let's actually I think [49:49] this looks good uh I understand so [49:51] basically you send uh basically you're [49:53] going to have some metadata information [49:55] and uh record some metadata and then [49:58] message going to start in another [50:00] storage like a temporary story Message [50:02] series or message storage series [50:04] something like that then go to the [50:06] center and then um here uh you may call [50:10] third party API so that's why you have [50:12] adapter here and then at at the end like [50:14] notify user and also uh do you want to [50:17] keep client waiting during this process [50:20] or how do you're going to response to [50:22] the [50:23] client um response to the [50:27] the Cent request yeah so you have a [50:29] client request how do you handle it like [50:32] I ask you to send a phone call yeah okay [50:35] so the client is in fact the one who [50:39] send out the request for example Trump [50:41] send a Twitter uh to send a tweet here I [50:45] think Trump is the client and the sender [50:49] go all the way to different SMS it means [50:52] that all the followers of Trump would [50:56] receive the SMS or the Tweets in app [51:00] tweets so this Cent is in fact the S [51:05] um for example Trump and [51:11] followers so so my so my question is uh [51:15] what's from from client perspective [51:17] what's the life cycle looks like [51:20] um the clients uh respective um the [51:26] client has some kind [51:28] of [51:31] um okay the kind has some kind of uh [51:35] reverse [51:36] proxy uh it means that it send request [51:40] to the server and the request is encrypt [51:44] and with the reverse proxy and they [51:47] could decrypt this kind of https request [51:51] and also uh also to uh uh normalize [51:57] it as a server request to send this for [51:59] internal server use and also the local [52:03] uh local cache uh because um it it just [52:07] send the topic and doesn't know uh [52:11] what's a corresponding topic name and [52:13] metadata other metadata could be so it [52:16] will query the uh metadata [52:19] surface [52:24] um um and uh to to make it as a food [52:29] metadata Json uh topic Json format and [52:32] also it would you would call local disk [52:37] uh to uh [52:39] emit on The [52:41] Matrix as we have mentioned in the [52:45] requirement above that we need a [52:48] realtime system dashboard to show the [52:50] process about how many notification are [52:53] sent and pro progress uh so this is the [52:57] enter gate of this so we Adit the magx [53:01] for the kind side and also some logs uh [53:07] to store it um and in the load balancer [53:12] sorry I I don't think it's a good idea [53:14] to actually uh display dashboard on logs [53:17] and [53:21] Matrix [53:24] FL Matrix [53:52] [Music] [53:55] dashboard structure [54:00] mer guarante [54:03] durability [54:07] track [54:11] okay extra [54:23] database Flink okay [54:31] replication [54:33] durability durability [54:42] guaranteed [54:48] durability window [54:57] okay [55:02] um so [55:04] the the I'll not push the logs and the [55:10] Magics here for monitoring uh instead [55:14] after the load balancer the load [55:15] balancer is for the uh [55:20] authorization authentication and [55:22] authorization yeah so I think I'm lost [55:24] here I think all my request is that all [55:27] all the request is actually internal [55:29] service it's like you build a client [55:33] tool you build a tool for all your [55:35] companies uh [55:37] service so all your client is actually [55:40] validated service for example or if if [55:43] you're look at the example here I give [55:46] is service a send message to user what [55:49] service B and all this is service like [55:52] like let let's say you design this for [55:55] Facebook so all other ABC here is [55:58] Facebook internal [56:00] service okay so the client is here is in [56:03] fact all internal surfaces so you don't [56:06] need [56:07] authorization right you don't need [56:09] authorization authentication okay um so [56:15] so this request go to [56:19] uh three different ways it's first go to [56:23] the metadata surface to retrieve a topic [56:26] then go to Temp Storage surface to store [56:29] data for durability so that idle uh [56:34] subscribers uh could get this dater when [56:37] they are active and the third thing is [56:40] that for monitoring it also goes also [56:42] goes to the Kafka and the Frink for the [56:45] monitoring [56:48] um I'm not so familiar with the Flink [56:51] though um for the for the Temp Storage [56:55] surface um um we could make it uh make [56:59] it uh as because it should be scalable [57:03] so the Temp Storage [57:05] surface uh we could make it as a uh key [57:10] um key value database to support the [57:13] high uh high high read and right [57:22] so we could choose a Dynamo DB and and [57:26] uh for [57:30] senders senders we could um easily um [57:34] make it as [57:36] a different [57:38] kind scaled web socket surface and the [57:43] adapter as our [57:49] router uh two different kinds of devices [57:53] um so [57:56] if there's not enough senters we could [57:59] add [58:02] scalable number of [58:06] workers for the senders and [58:10] uh uh because the sender is uh some kind [58:13] of stateless we don't need to worry [58:16] about this part and same goes to the [58:20] adapters um so you want to do it [58:22] asynchronized like worker P [58:24] asynchronized mode right uh yes okay you [58:27] want to emphasize this keyword like [58:29] asynchronized for separate concern like [58:31] separate concern [58:33] asynchronized yeah some keywords like [58:35] that [58:37] um if so [58:42] the let synchronize [58:45] a [58:47] um so [58:54] and synr nicely okay yeah just iyn is [58:58] fine I think everyone understand [59:00] asnn um could I go to a data schema to [59:05] draw out the schema of Temp Storage [59:08] service sounds good uh I also I also [59:11] have some life cycle question about you [59:13] know what's a life cycle for client [59:15] request and what's a life cycle design [59:18] for notification oh oh that's one more [59:22] about the retry C [59:32] retry surface or Reet or re [59:37] service because we have um at Mo uh at [59:41] least once uh this kind of uh [59:45] requirements and what's the question [59:48] sorry yeah so uh I want to you know once [59:51] you finish out the design I do want to [59:53] you know spend five minutes explain the [59:55] life cycle for a message and life cycle [59:58] a life cycle for notification and life [60:00] cycle for a client [60:02] request um life cycle for [60:06] uh um client request is yeah so let me [60:11] write example here for example I'm a [60:13] cent right I'm a service [60:15] a service a uh I send uh I want to send [60:20] uh uh message to user one user one and [60:26] then what's you know what's the [60:28] different step for for example first [60:29] step I going to to call your [60:32] API with [60:37] metadata and [60:39] message then service two step two what [60:43] you going to respon um okay um so comma [60:48] API is kind of like [60:50] cend and topic ID and with at Le [60:58] once under this case with the model [61:02] under this case on the first we will go [61:05] through the client uh the Cent will [61:10] decrypt um Cent will do some [61:16] pre-processing and uh the load [61:22] balancer and call um call metadata [61:28] surface you get uh [61:33] [Music] [61:37] topic maybe [61:39] Json and the third one is uh go to the [61:43] Temp Storage [61:49] Service uh store uh Store the store the [61:53] message [62:08] and the when the [62:10] message and go to the [62:13] sender send the [62:17] message um [62:20] if if uh [62:23] sender um [62:33] to retry [62:35] surface and the well [62:47] the message [62:58] um this is the life cycle and [63:04] the so so here I have a question when do [63:07] I know my message is sent and when do I [63:09] know my message is received by [63:12] you um so that's a two different [63:15] question right when do I know my message [63:17] is sent when do I know my message is [63:20] recorded by [63:22] you um I I know my message is [63:26] recorded when I store it into Temp [63:29] Storage Service uh I know the message is [63:32] sent once step seven got response from [63:37] the subscriber directly uh saying um [63:41] that's hey I have received your [63:43] notification [63:45] message um so that's that's the thing so [63:49] uh in the in the sender here we also [63:53] send uh have [63:57] topic uh to CF car [64:00] surface for [64:04] monitary um C link [64:09] surface okay so that you want to even [64:12] stream the you want to even stream the [64:14] life cycle notification that's fine U my [64:16] question is that uh as a client are we [64:19] going to to am I am I going to to wait [64:21] for the like two minutes for you to send [64:23] notification and waiting the request [64:25] that they have a request H as a [64:29] client as a client you just send [64:33] okay as a client there are three uh [64:36] three stages so the first stage I think [64:39] it's [64:41] um it's [64:45] pending uh and [64:47] uh name name the the request has not be [64:52] accepted by the notification system or [64:55] notific [64:56] service um the other thing is [65:01] um sending name that [65:06] uh namely um the the message has been [65:10] stored into the Temp Storage [65:14] surface uh and but the sender has not [65:18] yet sent out all the notifications to [65:20] all the [65:22] subscribers um so the third one is [65:25] completed [65:26] means that the sender has sent out all [65:30] the messages that should be sent out and [65:32] it's completed um and if there's some [65:37] exceptions then it could be also kind of [65:40] partial [65:45] completed um that's what I could think [65:48] of so there could be some status code in [65:51] the response I think to indicate that [65:55] where which Step are we stuck in or have [65:59] we gone through the task [66:03] completed okay yeah that makes sense uh [66:05] so you're going to notify your client [66:08] right so for example when when you're [66:09] going to notify your client is pending [66:12] when it's sending and when it's complete [66:14] our field let's say let's actually add a [66:16] field here just to for [66:18] compl uhuh okay um so [66:25] call your API under this case this is [66:28] pending is [66:32] it yeah that's a that's in more like in [66:35] pending State [66:36] reasonable and the Temp Storage surface [66:41] if a storage uh goes successfully then [66:44] it's become sending or filed or the [66:49] storage file to be [66:51] store and uh [66:55] here sender goes to [66:58] the uh [67:07] sending this is [67:15] sending adapter and [67:21] uh SMS Emil [67:39] yeah so uh in this case you actually [67:41] want to uh I think my question is in [67:43] this case you actually want to return a [67:46] receip at step five right after you [67:48] start the message you want to return to [67:51] your client say the message is uh [67:53] sending and then uh you know you you you [67:56] give a you give your client a receip ID [67:58] and your client can ask you uh later [68:01] like what's the current status or the [68:03] notification like so that's my question [68:07] you don't you don't want to hold this [68:08] request forever oh you you mean [68:12] that's you that's but but uh un C is [68:18] like oh okay [68:21] okay so yeah so basically you uh after [68:25] step five we can actually start a new [68:27] page so let's actually start new page [68:29] for this [68:33] F okay so after step five here you [68:37] should return to return to client say [68:41] you turn to client with a [68:45] receipt and client can pull the receipt [68:48] and to [68:50] see what's the completed status to to to [68:54] check the status [68:56] okay right um May I know why what's the [69:01] reason that we couldn't give immediate [69:03] back in the pending stage oh you cannot [69:07] give in pending State because you [69:08] haven't store the [69:10] message and if if the if the worker [69:13] actually failed if your if your server [69:16] actually failed you don't start anything [69:18] it just it just you know feel to [69:21] response yeah R should stuck the pending [69:24] stage for a long time for example next [69:27] time you have a request ID [69:33] andest [69:46] stess this client could close the soet [69:50] this kind of thing your server can [69:53] crash okay okay [69:56] right [70:04] sotif working on [70:18] it topic [70:20] including topic storage MH [70:44] oh [70:49] okay [70:51] database database status [71:19] transation notification [71:43] okay [72:04] step [72:07] five close [72:13] the notification status okay [72:36] work okay okay [72:39] okay [72:40] um um should we go to the data scheme [72:44] yeah let's go to data scheme I think it [72:46] looks fine so far let's actually finish [72:48] the data scheme that's probably last [72:50] session for this but um the data scheme [72:53] should be [72:55] um the Temp [72:59] Storage [73:01] surface um so what should I um Temp [73:07] Storage surface includes [73:11] the [73:13] [Music] [73:15] um message [73:18] id [73:21] id and uh [73:32] Temp Storage [73:34] surface [73:40] status shouldn't status the metadata not [73:43] part of the [73:47] message [73:48] [Music] [73:49] yeah um the status could should be [73:56] you have a metadata table right it [73:58] shouldn't be part of message because for [74:02] example several [74:03] tweets um you have a different messages [74:07] different message ID but the same topic [74:09] ID is [74:11] it different message ID same topic ID [74:14] yeah but I'm seeing the status shouldn't [74:16] be part of this table um under this case [74:19] different message ID means different [74:21] status if I store it in the metadata [74:25] table that [74:27] means um if you have different message [74:30] ID it will be different Ro in metadata [74:32] table [74:35] right [74:37] um could [74:41] be yeah let's put that as a a separate [74:44] part let's let's look at the rest [74:47] part id toic id status this kind of [74:53] table so you have you have two service [74:56] right U this is a similar to slack uh if [74:59] you if you remember the slack discussion [75:02] we have uh uh May 26 I think May 26 in [75:07] slack you you actually have message so [75:09] you have message story service to stall [75:11] the message but you also have a [75:14] different uh service to stall the [75:16] message delivery part so you have [75:19] metadata service to start the message [75:21] delivery part okay [75:25] and for this [75:32] one um message ID topic ID what others [75:37] should I [75:41] do [75:50] um on mode [75:58] sender [76:13] [Music] [76:23] mode let let me copy example so we can [76:26] feel the example here maybe sure yeah so [76:30] this is a this is a [76:33] u this is a denimal DB for message story [76:37] service in in uh slack and we can we can [76:41] fill in here [76:46] maybe um [76:50] so I think the key is message ID [76:55] on the okay [76:58] the partion key is a message [77:04] ID [77:07] yeah and the sort key could [77:10] be S [77:14] key um s key could be temp [77:20] stamp and [77:23] uh this message [77:26] ID and sender [77:35] id [77:37] id send [77:44] mod Sur [77:50] type um [77:55] [Music] [78:04] is a sender ID um does that make sense [78:07] yeah I think this makes [78:09] sense um so this is basic cre [78:14] schema yeah I I think this looks good [78:16] let's actually look at [78:18] metadata um metadata yeah you do have [78:21] metadata right you want to control life [78:23] cycle notification and also metadata is [78:27] like because the metadata is not [78:31] so um [78:34] large the metadata service has [78:38] a message [78:44] ID I I think metadata could be stored in [78:47] SQL directly is it yeah so that's what [78:50] I'm thinking as well you need [78:51] transaction right [78:53] yeah um so under this case that's no um [78:58] that's I think that's not too much to [79:02] store [79:09] um message ID [79:17] status [79:19] status what [79:23] others um [79:33] metadata I think I would say send mode [79:36] send mode and service type should be [79:38] metadata here oh okay right it's a it's [79:42] more like part of the life cycle [79:44] management for the message so think [79:46] about metadata as a life cycle [79:49] management for the message of for the [79:51] notification of for client service right [79:54] in that case message story service or as [79:57] you mention it's temporary story service [79:59] it's more like a a k value lookup so [80:02] it's just give me a message ID give me a [80:07] message right and here you may want to [80:10] also have some receiver ID [80:14] right who a receiver [80:17] [Music] [80:18] okay and then what's yeah Status and [80:22] maybe some temp stamp field sorry [80:25] some time stamp field for for update or [80:29] something like that so you know which [80:33] message has been stuck at what time for [80:35] how [80:39] long yes right so because it's also a [80:43] SQL database so you can enforce [80:45] transaction here so you can make sure [80:47] all the message is sent at end yes [80:55] okay okay so one last question U Can we [80:58] actually spend three minutes to describe [80:59] a life cycle for the [81:02] notification from a notification [81:04] perspective let's let's try to describe [81:06] what's the life cycle for notification [81:08] looks like the life cycle for [81:11] notification um what's the difference [81:13] between between the life here's a life [81:17] cycle for the [81:19] client U because it's different right [81:22] life cycle notification start with a [81:24] notific ation [81:33] creation okay okay so [81:51] so a particular client request [82:01] notification [82:10] notification s finish [82:24] fin [82:29] machine request fin [82:48] machine um a life cycle for notification [82:53] a particular [83:09] notification okay [83:23] notification time [83:37] notific email creation email [83:43] creation email [83:45] template email [83:48] template oh okay okay [83:51] okay different methods [83:55] [Music] [84:08] okay f inata [84:30] expence [84:40] experiment set email set email [84:45] status to [84:52] pending notifications [84:56] service set set sent email and then [85:02] set email [85:04] status to [85:08] sending put po [85:27] sending [85:34] status okay [85:37] sure I think that's [85:41] it do you have further question do do [85:44] you think which part you did good and [85:45] which part you didn't do well um [86:02] at most [86:15] on [86:18] most [86:21] guarantee guarante finish step file [86:32] finish okay okay got [86:38] ITB system every single step can [86:45] fail most step finish [86:55] at [87:08] [Music] [87:12] least at [87:21] least okay at most step file [87:27] finish [87:37] okay [87:41] um message [87:44] [Music] [87:46] system subing [87:57] philosophy separate concern [87:59] asynchronized processing worker [88:13] poolen driven [88:38] [Music] [88:43] Center side send topic [89:21] adap message [89:30] service status [89:35] updated [89:38] status [89:42] [Music] [89:47] okay pending [89:49] State message only notification sending [89:52] State message [89:58] notification [90:08] message d [90:34] stage permanent [90:37] storage [90:43] um system [90:48] message storage service for durability [91:17] ID message St service [91:35] it forever [91:47] almost confused okay [91:53] [Music] [92:00] r [92:01] [Music] [92:34] service [93:22] config [93:35] single service active message [93:54] ret [94:02] logic surface [94:15] [Music] [94:43] requirement okay [95:01] adap different adapter to different [95:04] router could be ended in the standard as [95:10] well adap low level conception [95:26] [Music] [95:47] okay message [95:52] CU adapter [96:09] status transition parti [96:21] completed internal s error code [96:34] part [96:51] chunk um group [96:54] notification [97:46] okay complet confusing [98:22] how have [98:30] Cates [98:56] Scot

分布式通知架构拆分载荷与状态表 防范第三方通道重试与丢包风险

Key ideas

Detailed notes

[10:20](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=620s) Past Project Deep Dive: Uber Money Payment Storage Migration

[14:01](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=841s) Project Ownership and Handling Historical Data Expiration

[16:35](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=995s) Interviewer Coaching: Articulating Scope and Business Impact

[20:29](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=1229s) System Design Prompt: Enterprise Notification Platform

[24:26](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=1466s) Scoping Requirements and Clarifying Exactly-Once Semantics

[29:58](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=1798s) Push vs. Pull Models, Recurring Alerts, and Latency Expectations

[33:23](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=2003s) Scale Estimation: Sizing for 200 Million Daily Active Users

[37:09](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=2229s) Message Formats, Multimedia Payloads, and Channel Topics

[40:44](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=2444s) API Design: Topics, Templates, and Delivery Semantics

[45:33](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=2733s) Pub/Sub Modeling Debate and High-Level Architecture Walkthrough

[49:07](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=2947s) Clarifying the Adapter Role and Client Lifecycle

[51:20](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=3080s) Edge Architecture vs. Internal Service Clarification

[56:48](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=3408s) Storage Selection and Asynchronous Worker Design

[59:08](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=3548s) Defining the Message and Request Lifecycle

[01:03:04](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=3784s) Asynchronous Acknowledgement and Status Transitions

[01:07:56](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=4076s) Asynchronous Polling, Receipt IDs, and Failure Handling

[01:12:40](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=4360s) Data Schema: Temporary Message Storage with DynamoDB

[01:18:16](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=4696s) Data Schema: Relational Metadata Store for Lifecycle Tracking

[01:20:55](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=4855s) Tracing the End-to-End Notification Lifecycle

[01:25:40](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=5140s) Interview Debrief: Delivery Guarantees, Worker Pools, and Distributed Failure Modes

[01:32:00](https://www.youtube.com/watch?v=D0bBMzxE5O0&t=5520s) Deep-Dive Feedback: Router-Adapter Architecture, Retries, and Status Transitions

中文简报

Reading summary

精炼阅读摘要

Detailed notes — zh

Source quotes

Source quotes

Original transcript

分布式通知架构拆分载荷与状态表防范第三方通道重试与丢包风险