Craigslist靠定时静态化推CDN 彻底隔离50万读QPS源站压力

Craigslist靠定时静态化推CDN 彻底隔离50万读QPS源站压力 Craigslist通过每小时批量预渲染静态HTML并全量推送到CDN，以容忍一小时内容更新延迟为代价将全部读流量阻隔在源站之外，从而彻底消除了数据库读分流架构的扩展瓶颈。 本期内容由资深工程师 Scott 对应届毕业生候选人 Leo 展开了一场硬核的 Craigslist 系统设计实战模拟面试。双方针对免注册发帖、邮件 Token 鉴权、日均近亿级新增帖子的 QPS 与存储容量预估展开推演，并重点探讨了媒体与文本分离存储及无单点故障架构。面试官纠正了候选人依赖客户端时间戳与生硬套用账号体系的误区，揭秘了 Craigslist 定时预渲染全量静态 HTML 并推送到 CDN 扛住 500K 读请求的“泥石流”工程解法。最后，Scott 复盘了 New Grad 在系统设计中所需的连贯叙事逻辑，深入剖析了从数据容量与 QPS 两个维度综合评估数据库分片扩容的工业界准则。 Craigslist 采用无注册系统的极简设计，用户免登录发帖后仅凭邮件中下发的高熵随机 Token 链接即可进行管理与修改，系统职责聚焦于发帖与列表展示。 (25:00) 基于全美 4.6 万个邮编大区估算，系统写入 QPS 约为 1,000，读写比高达 100:1；在无缓存极端情况下纯靠数据库抗读需要多达 50 台实例。 (33:45) 面对悬殊读写比，Craigslist 采取批处理流水线定时将各区域页面预渲染成纯静态 HTML 并全量推至 CDN，读流量 100% 绕开源站与数据库，源站仅需消化极轻量的写入请求。 (75:00) 系统设计必须遵守防御性原则，绝不信任客户端时间戳；生产环境服务器利用率通常必须控制在 50% 以下，预留冗余以防半数节点故障引发雪崩。 (60:00)

Scott 认为系统设计面试的核心在于立足具体业务本质展开环环相扣的连贯推导，针对 Craigslist 这类分类广告平台，其关键在于通过免注册的邮件 Token 简化鉴权，并借助定时生成静态 HTML 推送 CDN 的极端读写分离策略化解高并发读瓶颈。 业务驱动的免注册机制与防御式 API (25:00) Craigslist 具备极强的纯粹工具属性，业务上摒弃了常见的用户注册登录系统，仅依赖发帖人填写的邮箱和系统自动生成的高熵随机 Token 链接进行内容修改与删除，依托基于能力的访问控制降低了鉴权复杂度。在接口设计上，Scott 强调服务端必须坚决贯彻防御性原则，绝不信任客户端输入的时钟参数：创建时间与过期时间必须统一由服务端权威计算生成，防止恶意客户端伪造时间或因早于 1970 年的负 Unix 时间戳引发系统越界崩溃。同时，网络通信依托标准 HTTPS/TLS 协议即可保障端到端加密，候选人无需在业务层过早消耗精力实现自定义加密。 容量预估准则与双维度数据库扩容决策 (33:45) 基于全美 46,000 个邮编大区、10 个大类、每类每天新增 200 条帖子的假设，系统日增发帖量约为 9,200 万条，平均写入 QPS 处于 1,000 左右，峰值按业界 5 至 10 倍经验法则估算。针对存储架构，文本（1KB~2KB）写入结构化关系型数据库，而庞大的多媒体图片则剥离存入 AWS S3 等对象存储，数据库仅保存媒体指针。 在容量规划上，系统扩容必须综合“数据大小（Data Size）”与“吞吐量（QPS）”两个瓶颈阈值并取最大值：若按 1.4 TB 年持久化数据量（单实例承载 500 GB 并配置 3 副本）计算，纯存储维度仅需 9 个实例；但若在无缓存场景下面对 50 万读 QPS，按单机处理 1 万 QPS 的硬件上限推导，吞吐维度则强行要求配置 50 个数据库实例以满足并发负载。 极度读写分离：静态 HTML 预渲染与 CDN 卸载 (75:00) 面对高达 100:1 的读写比（写 QPS 约 5K，读 QPS 约 500K），除使用 Redis 或 Memcached 吸收 90% 以上热点读流量的通用方案外，Craigslist 采用了一套极具特色的架构解法：后台 Pipeline 每小时按区域和时间批量生成全量纯静态 HTML 页面并直接推送到边缘 CDN。高达 500K 的读请求在 CDN 边缘节点即被完全阻隔，源站服务器仅需承担 5K 的写流量。 该方案的代价是数据一致性的妥协，用户修改或发帖最差需等待一小时才能在前台生效。但对于二手交易或租房分类场景，这种级别的延迟在业务上完全被允许，从而以极低廉的机器与流量成本支撑了庞大流量。 工业界容灾法则与系统设计叙事逻辑 (85:00) 从求职评估角度，系统设计面试综合考量领域知识、沟通、解题与项目落地能力（各占 25%）。Scott 指出，优秀的回答必须具备清晰的叙事连贯性（Storytelling），严格遵循“业务用例 -> 约束与容量估算 -> 实例规模推导 -> 高层架构 -> 接口与数据模型 -> 瓶颈针对性优化”的主线推进，切忌盲目套用包含用户表、密码登录的通用模板。 在生产工程实践中，单台关系型数据库承载 5K 至 10K QPS 即达到必须扩容的高危阈值，且工业级集群的硬件资源利用率通常必须压制在 50% 以下，预留一半的安全冗余以防止部分节点宕机时剩余节点发生雪崩式崩溃。 该方案论证了如何根据业务特征与延迟容忍度，以“静态化+CDN”将复杂的分布式读压力化繁为简；但其架构红利建立在页面数十年高度固定、缺乏实时交互的特定场景之上，无法直接迁移至需要高强一致性与千人千面个性化推荐的动态现代应用。

[15:00] Scott: 假设现在这个邮件通知系统要支撑 10 亿用户（1 Billion Users），或者要作为底层公共平台服务全公司几十个业务团队，你该如何设计它的高并发架构？ [29:30] Scott: 比如你登录 Facebook 或任何现代网站，Facebook 怎么知道是你？就是因为给你分配了 JWT。 [29:30] Scott: 我们先把摄像头打开。因为在远程面试中，面部表情和肢体语言是非常重要的。 [45:00] Scott: This is not the same. This is the database. This image is the S3. You can put two places together... because you're in different places. [47:30] Scott: There is a protocol called HTTPS. It has a TLS, so it's end-to-end encryption. Let's not worry about encryption... HTTPS/TLS is provide end-to-end encryption. [55:00] Scott: 先问完 Business Use Case，然后算了下 Constraints，然后就出了结果：我们需要 50 个 Database 去做 Read Request，1 个 Database 做 Write Request [60:00] Scott: And also, I don't think you want to provide creation time from client because client can give you any time. You want to generate the creation time from server. [60:00] Scott: Expire time, you can trick the expire time. You don't want to get expired time from your client. Yes. So anything from client, you need to treat it very suspiciously. So you don't trust anything from client, especially around time. [65:00] Scott: 你没有 user ID 在这里，你压根就不需要登录（You never login） [75:00] Scott: 如果你用 CDN 扛下所有静态 HTML，你完全不需要考虑任何 Read 端的架构负担，所有的后端服务都是专门为 Write 定义的 [80:00] Scott: 后端的架构讨论，几乎大部分精力都耗费在数据库选型上；一旦数据库选型出现偏差，引发的系统灾难是毁灭性的（disastrous） [85:00] Scott: 面试一定要顺着面试官给的 Context 走，同一个系统题目因面试官切入点不同，推导方向完全不一样。 [93:40] Scott: 虽然我们说了很多 Red Flag……但是他确实干了很久

[0:00] 说几件事啊今天就是一件事的话就是我想就是尽快的就是在一个月左右吧就是这个这个这个design discussion可能就结束了就是在八月底之前然后把这个Design Discussion结束了,然后多方面原因嘛,最主要的原因还是我时间真的是不够啊,就是实在有点太累了点,就是准备这个的话可能一周要花5到7个小时吧,我觉得这个时间确实是不够,还是有点太累了点。然后另一方面的话,我觉得作为这个System Design的这个这个方面的话,我觉得这个其实cover的东西已经足够多了,就从system design 的这个角度来讲的话cover 东西已经足够多了,就是面试的角度来讲的话然后其次的话,我觉得大部分我知道的大部分就是受这个layoff,Uber受layoff影响的这个同学呢,就是这次已经早间工作了是吧?然后我所以多方面原因考虑,我觉得这个我觉得可能就一个月之内就完结了。然后的话,我们之后的话,我知道那个群里面有很多人挺强的,就是如果有人感兴趣的话,我们可以有一些ad hoc的这种,比如说我知道就是我们比如说Spark这个大神的话,我们可以请过来讲下Spark是吧?然后有这个比较ad hoc的题的话,我们可以讨论一下,但我觉得这个差不多的话到现在的话一个月的话差不多有30个Session左右,我觉得是从System Design的角度来讲是足够了,就是大概30个Session左右。其次的话我感觉这个可能大家积极性确实比之前要稍微低一点是吧,我感觉这个观看人数就是参加的人数也比之前的少了一点是吧,之前的话差不多有60来人,现在的话就基本上有20来人,但是我觉得这个不是一个很主要的原因啊不是很重要的原因,主要原因还是时间不够,并且我觉得编辑递减效应就是你在这个我们这个讨论中学到这个系统设计呢在这个就我们这几个session中渠道系统设计呢应该是绝对够用这个系统设计Design System Interview了我觉得并且我们尽量把这个Session做得完整一点尽量把 session 做得完整一点所以我就想在一个月左右,就是8月底之前的 session 应该就结束了我尽量在这个8月底 session 结束之前把 cover 完整一点 是吧然后是这样子我想听一下大家有什么想法,有什么 feedback 什么的,这块可以讨论一下好的或谢谢大家好的好的那那我感觉是大家都觉得是还是比较支持的是吧就觉得没有什么问题好的,对,我觉得这个其实从System Design的角度来讲的话谢谢大家嗯,从System Design的角度来讲的话,嗯,我觉得这个其实Cover的角度真的是足够多了,但是这个System Design, again,是一种长期积累的过程,所以这个如果大家想学的话,可能还是要多读paper,然后多读书,多读paper我觉得这个只是一个入门吧我从我的角度来讲的话,我觉得是一个入门者,就是其中有很多很细的东西我自己也没有实践过,然后有一些也需要去学习我觉得这个东西还是很多的,并且每次看一下大家写的其他案件上写的这个post也能学到很多东西对我觉得还是一个入门,也是一个长期学习的过程,我觉得这个群的话可以大家一直保留着,然后有新的问题的话可以post 一下,可以聊一下,可以解决一下问题是吧?就是有什么新的System Design的问题的话。好,我们今天interview 的同学是 Leo我把那个 doc 发到那个Dog发到这个Chat里面了Lio你在吗?我在啊,好,Liu,你先简单介绍一下自己,然后我简单,然后我说一下这个面试的这个,对,你先简单介绍一下自己。我是Qual毕业的Master,然后NewRed,然后这次主要是想通过System Design之前自己面了两次吧,然后之前也有在学历每周会听Scott的讲,然后这段时间也确实收益很多,然后这次就是主要是通过这个实例然后进一步学习一下System Design,然后为秋招面试做一些基础吧,然后之前是有一点,实习的经历就是这样,然后也感谢Scott今天来 host 这个 meeting, thank you.好,谢谢,然后瑞铭说,陶妹也是 [5:00] 就是今年是第一年,第二年,明年毕业,明年下年毕业。就是也是要这个System Design的问题,可能会有System Design问题。我想说一下我自己的经验,就是我自己作为一个New Grad,变System Design的经验。我自己New Grad的时候,其实我刚毕业的时候,面试的时候很糟糕,我那天还统计了一下,我那天还在那边看了一下,我从毕业找第一份工作,我面了大概就是连发那个letter 发那个resume到申请resume 我大概申请了58架 我申请了58架并且早工作从第一家发出去以后等早工作等了大概半年多才有第一次 onsite 是吧?就是我其实我申请的很早,我刚入学我就想申请毕业,就是申请早工作了,但是我走到第一个 onsite等了半年多申请了58点,并且我前面有大概有26家都被拒了,然后才拿到第一个第一个onsite,就是前面Foree2有和之前被拘留26家拿到第一个4.2 on-site我其实之前背剧的是蛮多的背剧的是蛮多的然后New Grads找工作是非常困难的因为我自己真的知道我自己你看我申请了58家才拿到我最后是拿到六个offer,就是最后拿了十个onset最后经过了10个昂赛的但是前3个昂赛的全部挂了后面7个昂赛的过了6个我前三个onside全挂了,我开始挂onside是差不多我申请了一年以后了就是我申请我从我从上研究章第一年开始申请然后申请了快到第二年,就是快过了一年之后,我才开始拿到offer然后并且才开始就是拿到第一个昂赛的过了以后后面的昂赛的过的率就极高,差不多百分之七八十吧然后一直就是过的率极高,就是我最后的时候对,最后onsite的10个过了6个嘛,但是前三个是先挂的。差不多前三四十个全部都是挂了的。就是多个原因吧,一方面原因我觉得可能之前还是不够憨葬吧,就是我面试的那会儿其实Leadcode也没多少道题,Leadcode是不到150道题所以那会没有什么刷棋的概念所以我开始面试的时候基本没有刷过题,好多题出来以后就现想,然后觉得确实有点难没做出来 另一方面的话谢谢大家那会儿的话,确实稍微弱一点,就是确实,System Design的话,问一个Design 10.0 URL,基本上是不会答的。但是我从那个onset来讲的话,我面了那么多onset的phone interview,只有大概三到四家公司。面过这个System Design问题这个是System Design总体面的比较少,并且面的System Design问题也比较简单,就是Design TinyURL然后这种的对,那面的比较简单。现在的话,System Design可能是相对来说问的更多一点了。但是我觉得它跟那个刷题一样,就是相对以后的话会比较就是就变成那种八股文那种格式是吧,更容易准备,然后更容易准备的更好一点这种。所以这个以后可能System Design 会更频繁一点。所以我觉得如果是New Grads的话,今年面试会挺难的,非常难。但是我觉得我也走过这个非常难的时候,我只能说我觉得这个练了这么多以后,经历了这么困难的一个面试以后,我觉得其实是比较宝贵的一个经验,就是你知道面试的时候应该准备些什么?然后你知道面试的时候应该准备些什么,然后你就知道恶魔力自己吗?所以我第二次换工作的时候,我就申请了6架,然后拿了4架offer就是相对来说也就比较好难了就是因为知道第一次说怎么那个就是怎么样面试啊,这个套路都熟一点是吧?所以我觉得这个就是对New Grads 来说今年尤其难,我看今年的失业率之前是从这个百分之三年几啊,19年是百分之三年几,然后今年涨到百分之十三年几了。所以New Grads 面试尤其难。所以说这个,像NewGrad System Design的话,不会太深,但是有点准备总是比没有准备的好的。然后一定要Focus 上一些一定要focus 上一些就是就是说 system design 最怕的不是说最怕的是什么呢?最怕的System Design不是说就是,你不知道说什么是正常的,因为System Design本来就是一个非常难的一个东西。最怕的是没有准备,是吧?你完全没有准备,问一个 Design 提前它内容,你是完全没有概念的。然后是完全没有概念的,完全不知道该从哪里那个什么,但是如果熟悉一点的话,知道我们先花点时间聊一下architecture,是吧?然后我们再花点时间了解一下需求,然后再简单计算一下,然后再聊点architecture,再design一下database 然后再design一下dataschema [10:00] 然后再看一下Data Model那你多少能写出一点只要是如果是New Graph来这点其实就足够了就知道你有实践过这个工程上一个完整的经验我觉得这个足够了所以这个是这样子的然后再其次的话,就是最后一点的话,System Design可能是任何一个问题能作为System Design的问题。不只是常见什么Design Uber,Design YouTube,Design Facebook这个是比较常见的,也是Solution比较多的。但是任何一个问题都可以变成System Design问题。有时候我问System Design问题的话,有两种问题是常见的问。比如说一个常见问题,比如说是你要去灭一家公司你去面这家公司,我记得我当时面过一家非常小的公司,这个公司是给非洲提供能源的,就是他是给那个非洲提供照明装备的,他为什么有前途呢?为什么呢?因为这个非洲人啊,就是他给的那个背景调查,就是背景context 我读了一下非洲的居民呢,他能把20%多的钱去买煤油,他是没有造明的,你知道吗?他是没有基础设计建设,所以他没有造明的。所以他把月收入的26%还是20几去买煤油。然后烧煤油但是煤油的效率是非常高的污染非常高然后效率非常高,所以它烧没有以后呢,变成它的cost 非常高,但是它的overhide 很小,就是你给他一块钱他给你一块钱都没有但是如果你要普及电灯的话普及太阳能电灯的话让他一下掏出200块钱他是没有那个钱的所以他每次都花一块钱每次花一块钱买一块钱没有但是他总的cost是远远大于电灯的钱然后这是第一点,然后第二点是第二点是这个就是污染跟肯定更多嘛然后这个照明效果肯定更差所以这个公司是在San Francisco的一个小公司我不知道现在在不在了我就不提公司名了他是做什么呢?他就是说,我现在是把电灯直接卖给你,就直接给你电灯,不让你花任何钱,但是你每使用一天你要付5毛钱或者是7毛5分钱,是吧?你的cost是比没有高的,但是你时间久了以后你的利润是比这个没有好的,其实是有revenue和profit。它这个就target什么印度北部这种贫困地区啊,然后非洲的那个什么埃塞俄比亚就特别穷的地方,它这个市场非常非常大,你知道吧。然后他就是我当时就面这家公司那如果我面这家公司的话其实这个公司的这个Data Model很像什么呢很像什么呢?很像这个小黄车或者是共享单车是吧?因为你需要一个password嘛你输入password以后,你就可以把这个装备使用了,就可以使用这个电灯了。你如果没有password的话,就不能使用这个电灯,是吧?就是跟这个小黄车的这个从Technical solution 是完全一致的是吧?所以我就想了一下把那个小房车的那个大概设计了一下,不管是Architecture,对吧,然后如果你要装蓝牙的话是额外贵点吗?你是要用蓝牙还是WiFi,你是额外贵点吗?可能是贵个几十美金吗?然后如果没有蓝牙的话,那你是不是这块就相当有个那个有个转动的password 是吧,我就设计了一下,然后去准备onset的。但是它onset的时候考的不是这个题,这块point就是,你去onset之前,你对这个公司有所理解,知道它的business model是什么?它onside如果面system design,甚至是behavior question,都有可能问跟它onside的business model相关的一个system design就跟他 business mode 相关的 system design所以像刚才提的给非洲提供能源的这个很像就是很像小黄车这种共享单车他们是相对来说是比较low-tech,但是它这个很像是吧,你就可以考虑一下思考一下能不能解决它。既然你有六个小时时间去onsign的话,包括你的通行的话,可能十个小时的话,那你应该有两个小时时间去思考一下他的business model,然后按他的business model准备一下system design的问题。就是一点就是你准备System Design应该和他的Business Model相关和他日常的问题相关,很有可能过来面试你的Engineer,就把他日常问题中的System Design就抽出来了。所以可能是任何一个问题,你知道吧,就是不能说是简单说,就是如果你简单说啊,我学过Design Uber, Design Lyft,是吧,Design Facebook,那你去面Facebook的时候可能去面Uber的时候可能有点用,但是你去一家小公司的话,他可能不会问这个问题,因为这个问题被考了很多次了,有点像那个什么,revert double link list,或者是这个,list recent use cache,这种coding的问题,会不会就是说这个问题呢,首先to general,大家会容易准备到,其次,和最后做的东西不相关,现在面试的话,如果我准备面试,我讲就是实战性,是吧,你面试, [15:00] 是过来以后你通过面试你是直接能过来以后工作的人你不是说通过面试以后就会发现你只是准备面试很强,但实际工作能力很弱所以这个是就是现在的趋势吧,我觉得也是趋势之一。所以面试的问题,System Design 可能是任何一个问题。可能是任何一个问题,特别是他们在工作中遇见的问题所以这个是很讲究实在性的所以说这个对,实际讲究实战性的,就是所以这个System Design是比较难准备的,这是一点,就是可能是问跟你面试的公司,Business Model相关的问题,或者他日常工作中预见的问题。还有一点,我面New Grads System Design的时候,我面过大概可能二三十个New Grads吧,就是我很喜欢面System Design,我面System Design可能面了十个吧,我喜欢问他Project的问题,比如说,比如说我记得我们组一个同学,当时过来的时候,他在project里面做了一个系统,有点像一个BBS,有点像一亩三分地这样的BBS,然后他做了其中 notification然后说那你帮我设计一个email notification 系统这个email notification 系统是给你们公司用的然后这个QPS 特别高那你怎么设计是吧就直接把你在project中的一个问题把你在你的简历里面列的project的一个问题变成一个skill非常大的问题比如说你的skill直接之前你本来做的一个pet project的就五个user 六个user但我现在就是说假设你有一个billion user那你怎么设计这个或者是你这个feature要好几十个组用,你就是做这个feature的owner的,你怎么样实践这个feature是吧?那你整个这个问题就是直接把你这个简历中的问题或者过去的一个问题变成一个System Design问题,这也是一种常见的做法这个的话,我觉得对New Guides,我个人觉得对New Guides还是比较合适的,因为就是看你有没有思考过,有没有这个思维的时候,有没有这个mindset,怎么样把它做一个production的一个系统。相对来说的话,并且问了一个你project做过的东西的话,你多少是理解更深一点。所以这个我觉得也是比较我常见问的一个System Design的问题。然后其次的话,New Guides还是要准备一些常见的System Design问题,比如Tiny URL这种。基本上只要想到System Design都会想到Tiny URL的话,简单准备一下这个问题是吧。我觉得Tending Y'all已经有很多很好的教程了可以就是东看看西看看,然后是吧,然后把它拼一下,然后看一下你自己overall system design的理解,然后把它设计一下我觉得在System Design这块,如果大家仔细地去把这个问题想通的话,把过去视频看一下的话,我觉得过System Design会是里面最简单的一轮。加上适当的练习,SenseM Design会是你面试中最简单的一轮。因为我觉得这个即使是一个有二三十年的Engineer,有二三十年的工作经验的Engineer,如果你没有专门练习过System Design的话,它的Performance不一定会比你好。这个是我觉得是不一定比较,它可能knowledge 比你多很多,但是我们说过 system design 是好几方面有四分之一的knowledge 有四分之一的communication有四分之一的 problem solving 最后加四分之一过去的 project 是吧你即使有个很有经验的 engineer 如果你没有做一个准备过的话 它的 performance 不一定会比你好好,这是我想聊的所有的关于这个New Grad System Design,还有这个New Grad今年面试的这个事情。嗯好我想看下大家有什么想说的额外说的额外聊的分享的你可以直接unmute然后讨论一下谢谢大家谢谢大家好,如果没有的话,我们就开始好了。好,Leo,你share一下screen吧,然后我这块的话就问了一个非常其实也不是很简单,但是我觉得这个算是System Design的几个入门题之一了就是 cracklist,然后我上周跟Leo聊的时候Leo说网上没有找见这个solution很好我就喜欢问一些网上没有找见solution的这样的更考考就是考验你的常见的思维现在能看见吗我share你好我能看见OK OK OK so let me describe the problem so first thing first because I knew guys so actually going to skip the question was a pasta production you know experience past experience our past project. Let's go to the problem directly. So here I want to ask you how to design correct least.so in this case so user can post the record in particular region特别的 category, so for example I just saw that video game2 weeks ago [20:00] I actually saw the video game two weeks ago on Craggly's. So I can go to San Francisco, I live in South Beach, so I can go to that region and sell a video game. And then so Um, So when I post the post, When I post the post, they actually gave a link to me. So they basically gave a link to me, and then--actually, give me a second. I can demo you this product in case you haven't used it before. Do you mind, let me share my screen. okay oh sure Okay, let me share my screen. Thank you. Okay, I'll stop sharing. Okay, just give me one second. Thank you. Thank you. Yeah, in case you haven't used before. So basically, I saw the video game a few weeks ago, just last last. So this is a post that I posted. So basically, oh, it's already expired. Anyway, so basically, I can go to the San Francisco Bay Area and say I can say for sale and say I can say go to the video, you know, video game sell by owner. then I can posting some additional information, basically give some text description. use some image and I can pin my location into somewhere. So once I post something, once it says, hey, I'm ready to post, What it gives me is basically, it will send me an email link. if you'll send me an email. Within this email, it contains a link, and only whoever has access to the link can modify the post. Whoever have access to the link can modify the post. So then at that time, the post will not show up immediately. It will take some time. It will take maybe 20 minutes, hour, you know, it takes some time to show up. And once it's so up-- so for example, if I want to buy a video game, I can go here and say, hey, I want to buy something, right? I want to, no matter if it's a bike or something, then I can click it and then I can email the owner. and say, "I want to buy this bike," and then you can find a place to buy the bike, right? So that's how we use Quicklix. And do you mind sharing screen again? - Oh, okay. - So in this particular system design, I would like you to basically implement the Mayflow cracklist. So you can post a record, you can post a selling record, no matter it's a video game or something, and you can then, the email will send to the user who publish the record. And after a few hours, the post will be released. And if you show us a list of the posts, right, under a certain category in certain area, and then a user can scan through a list. And that's the main feature I would like you to design. There is a bonus point if you can think of how to make sure a user can search a list. But that's a bonus. It really depends on if I have extra time or not. Okay, so I'd like to lead the task in five parts. So suppose now you're the customer and So basically, the first part I'd like to discuss is the business use case. And the second part is capacity estimation. and constraints. The third part is the architecture of the system and this is some API design. And the last one is the database schema. Thank you. Thank you. Thank you. Thank you. - Yeah, sounds good. This looks very organized. - And so for the business users, things you mentioned, like, Uh, We have Many uh, So in the... In the UI, first the user need to create register console. We have some basic functions. basic functions, I will hear you need to user need to be able to registration so And then we can know the users email address. [25:00] and user ID and password. Thank you. Uh. Just correct me if you think some unnecessary, then we can discuss. Yeah. We don't need to use the registration. So basically, you're just filling an email address. It will send an email with a token, and whoever has that email address, whoever has that token can modify the post. so yeah let's let me help you with flow here so basically Sorry. Here, so what There is no email registration. There is no user registration. You can just post something. you know, pull something, you know, write something with image and then, fill your email address, fill in your email address. Thank you. Thank you. Email will send to you. Thank you. Thank you. and it has a url or the post whoever has this url has a full control to the post Thank you. So it requires no registration. Okay. Thank you. uh so for This the post itself, I think, contains some parts of how we search the relevant ads. So basically in Craigslist, we have some ads that we need. We are interested and we sort it as based on like as locations, different categories. So Well, so how do we define how to sort I think we need to have a search bar here and also you have some filters. Are you agree? - Yeah, so search bar and filter, you can treat it as bonus. Let's actually design the basic flow on my functionality of this. Search bar and filter can be bonus here. - Oh, okay. - Yeah. - So, So for this one, basically, I think it's a web service and this URL since anyone click can have access to it if it's sent to the correct email address. So at this time, I think we don't need to worry about the privacy issue also security security also that's considered availability and probably the scalability so the user must be able to uh you know create, delete, or update the post he's already written and also Oh. Since you don't have the user account, you only need to keep a record of previous postings. So, the Yeah, the user must Hmm. And also. into upload the image. Thank you. And this must be done in only several clicks. which would be easier, easier to access. So, Uh, Able to Uh, able to create Thank you. update or delete the host. and also Hmm. When a new post created, as you mentioned, we need to have some time waiting for the response. So, let's call it wait so wait time. Mm. Thank you. So why do you need to fit for response? Oh, from business perspective, yes. But from product perspective, you know, from practice perspective, you only provide a list of posts and the post functionality, right? You don't need to consider anything else, right? Thank you. So basically, for crack-lista, you only need to provide a posting functionality and a list-of-post functionality. And that's it. And user can view the post. What action user do has nothing to do with cut-list, right? Oh, OK. [30:00] Thank you. Yeah, I think maybe we can discuss it later in in the following parts. So, Okay. uh So an email. Thank you. So a customized user can log in. the Thank you. Craigslist website using a customized email and random email address and as long as it's valid and the Uh. after create a post, maybe after waiting for some time, uh receive this uh email address. uh - Yeah, I can help you a little bit here. I think you have a question about authentication, authorization here, right? You say yes. Okay, yes. So when they have post when you're feeling imposed and also, you know, generate general speaking, your step finish step by the stamp tool. you can generate a token. token here you can use various to generate token but it's most likely to be a token on this bit and you can use some, there are some even bearer token generation algorithm. called JWT token. JWT token, but that's you don't really need that you just need to generate a this piece was a random string. And then when you send email, you basically can see URL plus post. post, you know, post ID, You know, it's REST API, so you have post ID here, and then plus the token. which is a token above. And whoever has this token, is the owner of the post. The reason is that it's, it's cryptic, very difficult. It's very difficult, close to impossible to generate this token. from scratch. Thank you. Okay. So the JWTOT token of VueR token, I actually happened to do something similar. I actually just Implement something like this a few days ago. So this beer token is useful or loss authentication and authorization, but the token here before you can use to identify the user identify something Thank you. It's computational hard to broke into this token. Okay, so the JWT token uh what's his what's that for A JWToken is like, for example, when you're logging into Facebook or you're logging any other modern website. how do Facebook know it's you? Because you have been assigned GWT token. It's JSON Web Token, yeah. uh okay. Thank you. Oh. okay so um Thank you. So for this service, Do you think I'll go to the next step or... Are there some other-- - Yeah, sure. I think this looks fine. Let's go to the next step. Okay, add some maybe some extended function. Maybe consider some like scalability when it's scaled up or when consider massive data or. like some other issues, but okay. I think so move on to this part. So capacity. So I need to know like how many requests per day or per month or - Can you give me some? - So, okay, so here, first, I forgot. Let's open the camera. Because when we were in the interview, the expression, the body language is very important. And now, the remote is also remote. We can open the camera. Okay, so let's assume that you have, you know, in US, we have, let's, in US, we have, Yung-Tongong Huang: 40,000 zip code in us. We have 40,000 46 on the code and assume that for this 46 son zip code, we have 10 category and assume for each category, you will have 200 posts per day. 46,000 lip code. [35:00] and 200 posts per day and 10 category. and time all this together. 46,000 for what? zip code. - And you have 10 category per zip code. You know, it basically just flatten, you know, just times it. You have 10 category, big category, and 200 posts per day. Thank you. Okay, 200 is pretty. - Per category, yeah, per zip code category. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. So for 46,000 zip code, uh So can you explain what was the zip code to me? - Oh, I know Canada use different zip code. I live in Canada before, so zip code is Uh, Postcode. Kind of like postcode. Yeah. Thank you. All right, pause, pause. Oh. Okay, so since we have 40 seats, 46K, also close. uh, Each user has a unique post-of-code. And, uh, There are 200 posts per category per day plus 10 categories. Uh... Thanks. So 46k postcode is the total number of post calls every day or is it for each category? You know, 46-cent postcode is for our U.S. It's like it's a fact. It's actually a fact. Okay, it's a Python. Okay. This is a real real thing. The US has 46,000 postcards, this is a real data. Mm-hmm. you can see how each postcode is a region so you're going to post your we're going to post your ads based on the postcode. Okay. Thank you. Thank you. Thank you. So 46 K postal code. Thank you. for each day 46k, 46k in total, right? Yeah. OK, so Oh. I don't think the capacity, so I need to calculate the QPS in order to know the traffic volume. So the... I think for each postcode, since each user has a unique host of code So that means we have 46K requests per day. I don't think for the categories and the 200 posts. relevant to the QPS at this time, so we can Uh So every day there are 26 hours on this. Uh, some work. Second, so this will be the total QPIs. Uh, I'm not on the right track. Thank you. uh Right. You need to-- so the postcode is like a region, right? You need to times-- so 46 times 10 times 200 and divides by the number of seconds. That's a QPS, right? Thank you. Thank you. So here, so basically-- - Okay, so these are in one region, right? Like one region. - One region. So each of the region, you have 10 category, right? And assume that the average is 200 posts per category per region per day. Okay. So, So this number, I want to formula here. Yeah, let's drop a 500. Thank you. [40:00] So it's about - Yeah, just modified, yeah. So I just calculated roughly. okay okay and so for the Hmm. We need to consider a storage limit since, Post. new post, created post, maybe valid for... maybe sometime maybe one month I know that's for the customer you may have some requirements that can only be valid so for a short period of time and then Maybe. Thank you. We can-- Yeah, so we can see if there's a poster only be valid for seven days. Okay. So it will be ready for seven days of And Thank you. Thank you. Thank you. All right. So I'm going on to calculate the storage limit since I think for seven days. How about do you think we need to know how many How many objects? the bias how many bias in each poll each pose let's do a rough estimation use your best knowledge - Okay, so that's what you only has 500 bytes for each. - Okay. So 500 applies for Uh, Thank you. or 500 objects for each post. So that would be 500 times 11. But times 1K QTS. So, uh, this one will get our orange linux so this one will be - 500 times 1K. Oh. Oh, sorry. It's not... So error... Every day we have this one, the total number. This one's not covered. Yes, this one. So, uh so every day we will have times seven seven days so every week I So you know, valid. Period. We will have suppose each Both have. the 500 files will be uh this this is the Uh... Thank you. And then I need to cover it this one. which will be the memory, no, which will be the storage limit for. Yeah, that's the minimum storage limit. You probably want to type two times two just in case. the calculation-- size calculation here only includes text, right? Thank you. Thank you. So the calculations here only include text. You're not including image. Yes. This one has asked how to pick QPS. Usually, the sum of room is x5 or x10. Average QPS x5 x10 is a big QPS. This is the most simple way of the calculation. Thank you. Okay. About So for the image, Suppose each pose have a each post have at least one image So you normally image the size is larger. Mm. Thank you. about we suppose each each post for the number of bytes, which include both steps and emails I'm not pretty sure if this Oh. these numbers is [45:00] Correct or? Yeah, I think so. So you think it's reasonable? This question is very tricky. You can send text to database. For example, this is a 1K, it's not a 500. 1K is very small. I think this is a 2K. You can send text to database. But you can use image to use the third-party image service, like Cloudinary. I use Cloudinary image service to manage image. Or you can use AWS S3 to write a media. This is not the same. This is the database. This image is the S3. You can put two places together. because you're in different places You skilled also is to use skilled Okay. Okay. Thank you. So for the image, Thank you. Maybe this one called 10K bytes per... Yeah, let's assume it's 100k. 100k is more likely to be image size, maybe even 1 megabytes. Okay. So use number Oh. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. We need to figure out how fast we can calculate. We need to calculate the same time. 46,000 x 10 x 200 That's 100 million 100 million, and 100 million from the 2K of the year, 200GB, so the x7 is 1.4TB. This is still very familiar, if you want to spend a lot of time. This is a thousand times, it's just like 100 x 200, it's just 2K, 2K x 46K is 100 million, right? And then the x7 is 700 million, and then x2K, this can be very quickly. as well as the effect of the effect Okay, this looks good. This looks good. So for the architecture, so The user The user has. clients in the client side. So, or create a post And we send HTTP request to the server and the server once it received So, I think here are some information like So we don't need to consider the password. like the user's email. So the user's email actually when in such a create list should be encrypt things Oh because of security issue. So we need to have a from the server side to server to the database, But before the server... Uh sent requests to the database. Actually, maybe You can consider at encoding encoding parts here to encode the Maybe the email address. And you just... to encrypt email address using new hash functions. So there is a protocol called HTTPS. It has a TLS, so it's end-to-end encryption. Let's not worry about encryption, you know, but for your knowledge, it's called TLS, you know, HTTPS, TLS is provide end-to-end encryption. I think it's also called SSL with a gap about HTTPS provide and when the encryption has go ahead. Okay. And from the... from a server side, wanting to retrieve the data from database, like data each data When we consider data schema like we have the for each post it has some creation time and the location and also the email address. Uh, the phone number. And also [50:00] attached to a user ID. So these are stored in the... database. So once the server refueled the data from database, then extent. Yeah. HTTPS request back to the Client. And then we can have a successful successful response. Otherwise, we may just have a have an error message for the response. Oh, Here, do you think, do I need to draw that paper or... So just draw the high level modules. So we have the Thank you. - One, two, three, two, three. uh yes because Google server and the server. Thank you. Thank you. Thank you. Okay. Then we can send a request from database. So in database, once is We Thank you. So once we shoot the data successfully, we database, we need to send response to the data. Silver Oh. Oh. This is too. This is... This is a request and photo response. We have the Thank you. database to the, to the, um, to send a response to the question. Thank you. and the server will set HTTPS response. This is not true. HPA responses. You have to write request.client. called "htpsession" You actually have a session, I forgot to call session. You can send the response, you can send the end, you can send the end. So the response is not written, you can't waste time. You can think from the server to the database, and then the web request can be response. Socket, socket, it's actually a session. There is a slide I forgot. The number of different sets is not the same. Okay, so... You just think that you just write the server, you write the node.js, you have a request, you have a response. and you can send, set header, and end. And the request will be closed. As we say, the request is actually a self-content request. So, the event response is not a problem. I think that server is from database to read it after, just to return to client. Thank you. Okay, so and also, I think you might have thought of it, but you haven't written it. Usually, this one has a load balancer. load balancer, and this one is a service, and database is using database, which means you have no single point failure. Okay. This is the case. If you have a server and a database, the server is down. The database is down. That's the main reason. Load balancer. Load balancer is a server. This server can be called app server. App server, and then the server can be called database server. It can be called database server, but it's not a problem. Okay. So, the patch and load balancer because the data is also big. meaningful QPS is 5-bit. And then, the general value can be load balancer or [55:00] 这个catch是吧对 如果像这种情况下的话 我们看一下这个数据这个1.4个TB这个数据呢,如果是按这个估计的话,你大概需要三个SQL server一个SQL Server大概是Handle个500GP是没有问题的看你这个需要三个SQL Server这个QPS的话你其实一个Server就够了 这个一个Server就够了因为这个景象大山也过来分享过了,现在一个SQL Server handle 10k 是没问题的。是KQPS是没问题的所以上面这个一个server就够了下面这个需要三个SQL server这个因为你直接靠的是Amazon S3,所以你不管它多少个,它自然帮你scale了是吧所以总体来看你可能需要三个database series然后再加上乘以3的replica所以三个replica training可能需要9个database instance 其中三个primary三个 primary 然后就是说六个 backup是吧这个的话就是在就是就是当你计算完这个这个stretch后的时候你就应该直观的想到我们对应一下是吧对应一下这个share hold,然后这块的话,这个只算了read QPS,read应该够高,对这个read应该更高,所以说但是这个read的好处是你如果用现在大部分的MySQL,你如果用那个step out isolation的话,read和read是分开skill的,所以你有三个replica的话,你的read还是足够去sign然后其次我自己看的话,这个数据的话,QPS其实是不是主要的,Rate QPS即使Rate是一百倍的话,瑞德是它100倍的话,那是如果是一锐的是100倍的话,那可能这个确实需要更多的server,这可能是需要注意这个多于九个 database 的谢谢谢谢我们可以算一下我们就这块算一下如果你是RAID的QPS是5000的话,RAID是100倍是吧?我们可以算一下RAID的话是这个level的QPS那如果是这样子的话如果假设,因为我后面说为什么说可能不需要这么多server,是因为有原因的,就是这个design是有原因的,但假设你说的没有cache你所有的读是从Database读的话,那一个Database是Sell 10k 的话,所以你需要50个Database去Sell read request拜拜Reader Request需要50个database然后 write 只需要一个 database谢谢大家所以这块呢 你这块得像50个database去serve这个read request如果是所有的read from database的话所以你在答完这个 Constraint 以后,就是自然而然的说,我们需要多少个 Data Base 是吧? 然后这里面的话这个App Server的数量这个是可以非常弹性的因为说的App Server是StateList的所以你可以说它是stateless所以它可以非常elastic的去调整所以你不管是一个还是几百个都非常好整非常好搞,但是这个database是相对比较复杂的地方,所以我看的话如果你不做cash,这个database需要50个database的read request和一个Database的Rate Request那这样的话我们这个conversation就更脸惯了是吧你先问完Business Sales Case然后算了下Constraints然后就出了结果就是我们需要50个Database去做Rate Request一个Database to write a request然后我们就可以说我们这个architecture是Client.loadBalancer.AppServer然后再到database 其中app server是stiteless 然后所有的这里面所有的module就是从load balancer开始说的model是至少一份的所以没有single portfolio谢谢OK 然后 这个 API 这个嗯如果我可以API called create host就是RESWordApi create and we also need to update and deletepost and some input parameter we have so here we have aUserID,Location,the created time and also the嗯谢谢大家可以使用力量and the user ID会不会有的 这会会不会有的 [60:00] Thank you. Yeah, let's not worry about type. Let's not worry about data type. Just focus on high-level things. I know you're going to handle the data type, right? Let's just save some time and don't worry about data type. Okay. and the quarter We have updates since we need to modify the post. So We need to consider so have some Uh, once we post an ad and then we We'll wait for some time until we receive the email. So there's a um, Thank you. So do you think we need to have a, Thank you. So the data We need to consider data latency here. Yeah, we definitely need to verify data latency. And also, I don't think you want to provide creation time from client because client can give you any time. Thank you. So you want to generate the creation time from server. Yes. Okay. So that's why you shouldn't be part of the API call, right? Because let's say you get creation time from client. I can say my creation time is tomorrow. My creation time is tomorrow. It was a crazy time years ago. Thank you. Thanks. so okay so it's I mean the time, create time for each post, it will be instant so we can't I have a Okay. Okay. That's... That's... that makes sense I think so don't consider create a post that will be posted after one day or two days. What? You don't need to include create time. So you don't let client specify the create time. Like, that's because client can do some, you know, - Okay. - Malicious thing. - So it's not customized. - Yeah, they can do some malicious thing. And for example, creation time is like before 1970. It may cause your system to crash, right? Because if it's before 1970, Unix time is actually minus value. - Okay. - They can give you some crazy value and your system may crash. Okay, so for the update Oh, yeah, so. for the user ID, the location, created time, and also the it's our time so unfortunately So, remove post. So, user ID location, and it's start time. Here at the uh Remove. So it's already expired. We could have automatically, I think we can You can add a feature here, automatically deleted. So, Right, screw. So you think you don't need to have a and it's time for the create tools. Yeah, same thing. Expire time, you can trick the expire time. You don't want to get expired time from your client. Oh. Yes. So anything from client, you need to treat it very suspiciously. So you don't trust anything from client, especially around time. Oh, okay. okay yeah So Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. So, Yeah, so maybe input variables only need to He was already... [65:00] And this location. Since the expired time and the create time, the user cannot, define it and we don't trust the user to actually define a week, store the data in the database. So this for the API design part. Do you think we have some other Yeah, I think. Yeah, I think missing title content image and Oh, Alex, Right. And also email you want to provide an email right especially create a post Oh, yes. So, You know how posting for me. So we So once we created a post that as so it has the user content information email address phone number also the as itself it belongs to each category and and for So can we assume the post details like the user's email address, phone number, also the images, also the other posting details are attached to a unique user ID? You don have user ID here You never logging Okay You never logging right You don have your ID here over so okay so let let assume let actually skip this part let's assume that we have something so basically in this part assume that we have some title let's assume we have some title starting to delete this format is strong Okay, assume you have a title, you have post information, you have image URLs, you are else, maybe you have more than my image. Assume that you have additionally and email address. Okay, that's probably the essential for this API. Let's focus on the database schema. And then I want to make sure you know at the end. sure you know at the end at the end here is there is a step six called which is photo night assume that you can have some you know you're gonna have something like and how to optimize the spotlight okay let's actually more let's actually go up it you know today by the schema so yeah first thing we consider is sequel versus line SQL So the data itself has some features So Traffic volume every day is, I think is, it's large and for different API requests and different records. So create, update, or delete the post. Simplify doesn't have relationship between these records. And considering the scalability, if we need to add new features or if we need to modify the website it must be easier to scale and also I think we would like we would prefer flexible rather than big schema for the database right here. So do you have like the add into on all your requirements? So practically, they haven't been changed [70:00] for more than 10 years, mainly in the flexible schema. Because if we prefer to Uh - Yeah, the website itself didn't update it, but how about if we wanna I can see there are maybe the current uh user login service. Maybe we need to There is no user login service. So cracklists haven't been changed for 20 years, actually, more than 20 years. It's always cracklist. Okay. Thank you. So since it speaks schema and also So between these Thank you. Oh, the data schema is fixed. So, How about use a seahoe? Pause. Thank you. Thank you. Hmm. We have some other features for the beta except the points I mentioned. Yeah, so, I think this part, I still have time to run up. I'll give you some ideas, a little bit of a sense. Let's wrap up this database. I think this database schema is no more difficult. This is mySQL. You can use mySQL. You can use mySQL. You can use mySQL. It has been a few years since it has changed. Haven't changed for 15 years. And then you can use Amazon S3. You can use this CDN to cache, and then the effect will be very high. The speed of the CDN cache is also very high. This CDN cache is also available to the data close to user, and to read faster, read latency lower. If you use CDN, it will be about 10 times. Ten times faster. 10x faster. So this is enough. This is not a complicated place. Actually, the question is that you answered this question. I was wondering if the bottleneck is what we're doing and how to use the bottleneck. But I think that the time is not enough. I'll answer this question. This bottleneck is what is called? This rate of QPS is greater than write QPS. This is the right QPS. If we look at the right QPS, if we look at the previous calculation, it is 5K. Red QPSC is 5k, Red is 500k 5K 和 500K 的级别 So you have a few ways to do it. But the most simple way is what? The most simple way is to use cache. You don't read this database, you use cache. This cache can be memcache, and readis. It's in memory cache. This is the way we use cache. Memcache or readis. Thank you. This will increase performance by the two-fold number. It will increase 100倍. And if you have memcast and redis, the RAID QPS will increase 95%. So the RAID QPS will not be 50,000, will be 25K. 25k 的话,基本上还是被 data size 所限制,所以你还是只要三个 database,三个 primary,九个 secondary 就够了,三个 primary,六个 secondary 就够了。 所以一般来说,加了 Redis Cache 以后,瑞的 QPS 是会减少 90% 到 95% 的,就是从历史的 Twitter 发过几个文章,历史经验来讲是这样子的。 所以这个就足够了,但是这个 Crack List 有一个非常泥石流的做法, I've read a lot of the correct list. It's called "每小时 every hour" will put posts in the post. POST generates on a static HTML. and then just put on CDN on. Then down. This is the dynamic dynamic. We need to server update. If we're in a dynamic dynamic, like HDN ML, the general CBN, CBN is a real user closer to obsess. So generally, we all [75:00] 静态数据我们主要每次去server side去跟她 update所以这个crack list 之前有一个做法就是就是说白了是你post的东西是吧 我一小时我这块我server端是一个pipeline,我server端一小时generate一个对于每个region对于每一个 region,我送你单一小时 generate 一个Static HTML,然后放到CGA上就可以了不然所以你看到的所有的HTML都是上一小时generate层的一个static HTML那你这样的话,其实你的 read requestRAID request 就很低了,你只要serve write就可以了,RAID就是你不直接RAID database,你RAID都是CDN嘛,只有internal RAID database嗯所以这样的话,这是cracklist非常泥石流的做法,但是我觉得还是非常有效的。反正你这个post lentency,如果一小时大家都能接受的话,它现在就是lentency 半个几十分钟一小时。你这个相当于是你这个read的这个page呢,是static page,html page。大家可以HTN on page,但是它是每小时有个pipeline生存,然后放到CDN上就可以了。所有的read request,所有大的request都不访问你的server,你server可以很小。然后你只要大部分钱可能就配 CDN 就可以了其实在UI端我觉得除了缓存还可以加一些比如说有些重复的 我们就没有必要再重复重复就 upload微信的头像或者是其他一些就很不太可能去update的一些information,就没有必要去重复的去点击的次数,就是减少点击次数,也就是说在UI端,在3rd端一个是提到LogBalancer,一个Catch,哦如果是数据特别大的话可以其实可以考虑分布式就是这里是用三个server来做,三个server嗯server application client at all到这个三个server application server就可以加然后从 Application Server到 AWS Server 也可以再叫 Load Balancer然后因为是它主要起到就是分流的作用这样的话哦可是是森盒的还有也可以加cache然后database端的话其实可以做预数理嘛,比如说database嗯preprocessing,那么我们如果数据量很大的话,其实可以做两个data table,不是都放在一个data table里然后所以要有一部分预先处理之后然后再拿到啊他拿到另外一个table里做这样的话就是可以加快他的处理效率这块的话我看了好多问题,大家问了很多问题,说如果有update的话,它会再次重新生成的CN的Static HTML吗?它会重新生成,它会每小时按照它之前那个Job Post的那个时间,Post的时间和更改的时间,然后重新生成一遍HTML。Static HTML那如果你刚更改的,你的新的updata是不会不会更新的,你只有在下一次peplight run了以后才会更新,所以你看到的可能是几小时前的,就是一小时前的post,可能会是几分钟前的post,所以Worst case是一个update要一小时才能消效这块的话, again,我觉得 Leo 这块你要想清楚一点,就是如果你用 CDN 的话,你其实只 serve write request,你 read request 直接 redrack 到 CDN 了所以如果你一上来就说,我们可以把 CDN 成这个 HTML 的话,你不需要考虑任何 read,你只要把 write 考虑就可以了,所以你的所有的 service 都是根据 write 来定义的,你的 read 就说白了是一个 pipeline,然后 rollout cdn 上就可以了嗯 OK 是吧,那这样的话,它其实这里面如果你不用CDN或不用那个什么的话,这里面最大的traffic是ray的traffic嗯read的话是100:1 相当于read每100个reader,我们可能才会有一个user去read这个东西所以考虑最大的量,比较大的量对你是这个意思就是我们考虑率的时候有包还是外的考虑RAID的时候这个RAID就根本不走你server了,绝去CDN了谢谢大家你这是 [80:00] 今天是content delivery network,说白了是就别人帮你server的,别人帮你hold一个object,或者artifacts,然后他直接从那读了,就不走你server了好,这块的话有几个问题,说cracklist没倒闭,它优势在哪里?每年都硅谷有一大堆人,就是商业分析cracklist为什么没有倒闭,但是他存回来挺好的反正我从09年用cracklist到现在我还是用挺多的也不是很多,但是一年可能会用一两次卖一些不要的东西然后可以对用户习惯嘛因为毕竟有用户在上面能卖出东西你在上面能卖出东西你就喜欢如果有些我想卖的话,我首先我其实不用Facebook了,我比较讨厌Facebook我Facebook也是几年没更新了然后这边说可以普及下各种DB的Storage以及QPSLimit吗?其实这几个组的比较好,比如说这个MySQL的话,我上次请过来分析过一次MySQL的话是那个10k,10k的话是相对来说,你是有比较复杂的那个QPS是10k,如果你有比较简单的RAID和RAID的话,拼着RAID和RAID之间相互不block的话,可能比10K还要更高一点。就是这个,然后如果你是用DynamoDB的话,Dynamo理论上来说它是Infinite skill的,只要舍得花钱它就能帮你skill,因为它这个设计模式就是可以skill的很好,Dynamo这个skill是很好。所以这个MySQL这我有个问题,MySQL属于Seql的Database,DynamoDue属于NoSQL,Amazon的一种NoSQL database这两个我不可能说我又用MySQL又用DynamoDue,我只能选一种对吧?不是,你要看的Use Case,您可以看看之前我们讲,有很多在讲最早的那个就是slack的时候slack design的话就是MySQL和DenmoDB的结合5月27号然后后面的话讲那个YouTube的时候也是MySQL和DenmoDB的结合database使用的时候还是要选择非常小心的你基本上所有的engineer的design discussion都会发生在这个database上然后database实现错的话你的result是disastrous就是说所有的engineer你会发现所有的backend engineer不论都是在database上,基本都在database上,会花很多时间考虑databaseCassandra和Denema是一样的,一致的基本上也是配InfiniteSkill,但Cassandra的话它这个partition不是proportional to nodes所以你可能加nodes的时候,可能会比较复杂一点所以这款的话,有些额外的,就是,quack list 有些额外的feature,但是都是一些小feature,这这款主要的flow是这样子,主要的flow是这样设计,它这个设计其实仔细想一下是蛮简洁的,就是你一小时生成一个static html,其实不管是什么cost还是蛮简洁的,对,好,这块的话,我也是简单的就是,就是给Liu给你提供一点feedback就是我觉得这个表达是不错的表达还是可以的还是比较清晰的然后communication 没有很大的问题然后我没有因为你是牛怪子我也不会问你就是 background除非你有很多 internship 我可能会问一下然后这个 problem 上我觉得这个结构是可以的就是结构还是可以的但是这个故事就是这个要讲究连贯性嘛你比如说你讲的 business case 很好然后我们就计算一下这个strich hold 是吧计算一下这个故事要讲究连贯性比如说你讲的business case很好然后我们就计算一下这个threshold计算一下这个limitation计算一下这个constraint然后计算完constraint之后自然而然的结合出它需要多少个database然后有多少个database然后有多少个replication然后特别是read和read的throughput需要多少database然后你就自然而然的到architecture是FAR architecture,然后是API,然后最后是Database design,Database schema然后最后的话,你可能需要分析一下Bottleneck,然后怎么样optimize the bottleneck就是这个flow呢,我觉得还可以再接着训练一下然后就是前二点然后第三点的话,我觉得从NewGraph的角度来讲的话,我觉得我从NewGraph来讲的话,差不多就是average这个水平就是你可以认为我面试只有一半比你表达的好,有一半没你表达的好就是大概是这种感觉,就是我过去见,但是它这个样本量比较少,大概就十来个左右。样本量总体来说比较少的,十来个左右。能improve的点呢,我觉得还是对于这个database的更深的理解。我觉得这个,我觉得你对于整个系统设计的概念的流程是很好的。 [85:00] 还是比较缺乏经验嘛,其实牛怪是比较正常的。比如说这个用多少个database啊,然后那我们怎么样optimize啊,这个是学习键,这个是没办法的,这个你需要很常年常年的去积累经验,我个人觉得你练习系统计算肯定是比我NewGrid毕业的时候好得多的,我觉得这个其实非常非常难。非常非常难,我觉得这个确实需要积累经验一点然后最后一点就是,面System Design的时候,面System Design的时候,不要focus太多的细节,比如说这个User Login, Authentication,除非你很了解,比如说这个Token,然后怎么样保障Security,你如果不了解的话,我们解释一下。都有 Antoine encryption,比如说这块 HTTPS是 Antoine encryption 的话,你就说这个是有 Antoine 的这个 encryption,所以就不需要花太多人就在这个上面,这个是前任人挺多的时间纠结的对的然后好,我觉得就这些吧,我觉得还是,从今天来说,我觉得Newgres来说还是可以的,就是我觉得你不会,如果你面试的话,你不会因为这轮就挂掉你。就这样吧 就不会是一个 Strong Note可能不会是一个 strong yes 不会是一个 strong no你知道吧 就不会因为自己能挂奖你的 我觉得作为 new grads 的话嗯待会儿我想看有没有额外的feedback或者额外的question谢谢大家嗯然后六女有什么想说的吗?有什么感觉?有什么?对。在这个API 里当时我有点卡住了,就是想,比如你说这个title,这个就是post details,我是想它 attach user id 里。然后因为我其实面试前准备了一点,然后我想的是是用那种带UserID 那么UserID他自己要create password的话 我自己需要一个table 那我在post的话 那我create的这个post 本身我还需要一个database table然后不过我觉得可能这地方就是硬变能力比较差,确实我觉得,因为一旦你说就是,不需要用的table,对,所以我觉得这点就是其实大家面对的,因为你要顺着你面试官提供的语境来,你要顺着你面试官context来,即使面试同样个crack list,比如说我现在同样一个人面试同样个crack list,我因为我的面试点不一样我问的问题不一样,导致你回答的方向是完全不一样的就是你要顺着面试官的语境来,这个是临床准备的话,就是之前准备的话反而有可能会拖你后腿,这个确实的,因为这个你有一些自己有想预想的一些答案,结果发现这个预想答案不能套用到这里,然后就反而拖你后腿了,这确实是这样子另外这几个其实我这几个之前我都看过这几个方法都知道就是说这其实就是做高病发的一些优化的问题那这些就是比如说具体多大的数据量我们需要考虑啊就是有没有比如说是还是要超过多少我之前看的好像要超过达到PEB数量级就是PEB再往上一个级级他说在考虑这阶段PP已经很多了,PP就要几千个机器了PP上千个甚至上万个十万个机器了这块,整体来说,你可能优化嘛,你其实是能优化就优化,直观感觉能优化就优化,你像机块的话,你比如说用write来写的话,你只需要一个database。用锐的话你需要50个database就明显一个优化能优化的劲儿OK反正就跟你写算法一样的,你这块是能用ON的算法也有法的话你现在写个N方的Conditional优化的比如说读写分离的时候,那个read writer,你在这个数据这个就是挺长见识的然后再有一个就是说根据average QPS 我不算一点可能5到10倍然后每个 database 我的本身的 database 我的做三个配分 每个选一个 backup然后需要跟这个流量到底需要多少个database?你提到的是说10k是吗?可以考虑用一个database从RAID和RAID来说 5K到10K可以抢用一个database10K已经很高了一般来说你的utilization rate要在50%以下那这样的话当你有一半机器fill掉的时候request到另一半的时候是不会crush in the server所以你的utilization一般50%以下 [90:00] 这块的话也有个问题说为什么1.4TB的data要用三个SQL这块的话是我看我们,我看Uber内部的话是一个SQL Server是ser 500个GB 的 data,500个GB 的 data 是一个SQL Server能server的。所以这个可能至少要三个,可能要比三个还要多我记得另一位同学问过我一个问题,然后我帮他看了一下,Database SQL的这个 limitation,大概就是几百个GB吧?它这个 limitation 我记得当时看了一下它是被 system限制的,被 operating system限制的,所以500个Giabets用一个SQL的话,我感觉好像没有问题谢谢差不多这500个Giabets可能用一个SQL的现在的话Skill的相对比较好一点之前的话可能50个就得用一个SQL现在skills 确实比之前好一点不好意思打断一下,我问一个问题,以前我好像学过一个关于数据库方面的,是主从数据库的话可以往主数据库里存,然后备份到从数据库,然后从数据库里读,有这种操作吗?还是我记错了。然后读的时候可以从backup和secondary读,是这样的藏件操作。如果这样的话是不是我们把这个也考虑进去那么读的时候其实就不需要读就是我的意思是读的时候从那个backup里边读,我们总需要的数据库的数量就不需要。我也没算需要多少。如果你没有开始的话,我们看一下这里面其实是有两个stretch hold,一个是datasize,一个是qps。这也是skill的常见两个方面,一个是datasize,一个是QPS。从datasize的角度来讲的话,你只要有三个SQL Server就够了,你可能有replica是保证你这个data durability 的嘛。所以3*3就是9个database就够了,就是9个database,你这个从datasets的角度就够了但是从这个QPS的角度来讲的话你是50万个QPS,你至少需要50个Database才够假设你一个database serve 1万个2ps的话谢谢大家所以这两个是起最大值吗谢谢大家嗯哦对是,明白了明白了。你scale 是一般scale 都是两方面,一个是从datasize,一个从qps嘛。那我们现在qps所需要database,假设没有开始的话,是远远大于datasize,所以取格最大值嘛。那肯定是KPS的50比较重要好的,谢谢。好的好的那如果没有额外问题的话今天就到这好了谢谢大家这块有个问题说API Design的话是要Consider REST API吗?还是REST API,Strift,Protobuf,随便选一个?或者你写SudoCode也可以,这个不是重点,你写SudoCode,大概不会有问题的就直接写个function是remote procedure called pseudocode这种像今天我们就写的 pseudocode是没有问题的是可以这样写的,这个不是重点,对任何一个API format都可以。谢谢大家感谢今天感谢Scott大牛组织Mark Interview觉得受益匪浅想问一下下次Mark是什么时候吗下周二,如果你有兴趣做Candidate,可以跟我说一声,聘请我做Candidate?我都不知道怎么联系您,我可能不在大家这个群里能不能给加一下我把我的微信号写下,如果不在那群里的话,你可以加上我微信,把你拉到这群里。好,好。那个HTTPS就是这个,它相当于HTTPS,传统HTTPS就是说它加了security部分,然后你说那个,当时我记得说那个security部分你说可以也是TRSSF,这个是直接可以做end to end increase,如果不是我你只要用HTPS,你是直接有Enterment Encryption的,所以你就不需要担心这个对别人的middleman attack 这种,不用担心。哦 OK 我想的那个email还有那个比如有有password的肯定是需要increption用什么hash function去做一下其实都是通过 HTTPS 传过来后,然后做 hash function 的。好,OK一般Hash的话,Hash在后耳端Hash的可能性比较大 [95:00] 并且我这话说一下,虽然我们说了很多red flag,但是Facebook在很长一段时间超过好几年都是名骂存password包括很长一段时间都是名骂存 password 的所以经常被丞线拿出来取笑,但是他确实干了很久所以我们这边讨论的很多这个明显的red flag在十几年前Facebook是可能每件事都干过谢谢大家可以想一下如果没有额外问题 之前到这吧 再次感谢Liu 我觉得希望你能从这次学到一些东西然后其次的话 System Design Interview 有时候是挺hush的pressure 是挺大,但是我觉得表现是挺冷静的,我感觉好 谢谢大家 今天到这儿了