技术面试官拆解推荐系统设计 警示消息队列滥用与定量缺失

技术面试官拆解推荐系统设计 警示消息队列滥用与定量缺失 推荐系统面试的成败取决于能否用端到端架构清晰划分离线训练与在线Serving，候选人过早陷入局部算法细节或忽视数据定量估算将面临严重失分。 视频完整还原了一场类今日头条新闻推荐系统的系统设计面试全过程。面试官首先指出了候选人在过往项目表达中偏“产品经理视角”、缺乏工程深度与定量权衡的问题，随后引导候选人拆解新闻摄取预处理、内容读取与推荐召回排序三大核心子系统。在推进过程中，双方就大体量正文丢进消息队列的隐患、词袋模型与主题分类、基于内容的过滤与协同过滤的优劣、冷启动及离线与在线架构切分展开了多轮攻防。最后，面试官结合自己的工程经验进行复盘，强调了端到端全景架构图、需求定量估算的重要性，并系统总结了通用后端（SWE）与算法工程（MLE）在系统设计考察中的侧重点差异。 系统设计面试中解决具体问题仅占评价的四分之一，其余核心维度包含过往项目深度、沟通表达、系统硬知识与业务感知力，描述项目时切忌产品经理式的泛化概述，需主动提供量化指标与选型权衡。 (0:00) 新闻等变长非结构化数据不宜直接作为完整负载塞入消息队列，应将不可变正文持久化至分布式 KV 存储或 S3，仅将轻量元数据与唯一标识交由消息队列异步流转以防集群吞吐崩溃。 (15:00) 基于内容的推荐在工业界维护成本极高且难泛化，协同过滤能避免繁重的文本打标与人工偏见，但系统冷启动通常仍需借助发布商信誉、热度与粗粒度分类等启发式规则兜底。 (50:00) 区分通用软件工程师（SWE）与机器学习系统设计（MLE）的分野：SWE 侧重微服务解耦、端到端数据流与高并发拓扑，而 MLE 则深度考察特征工程、离线训练与在线推理的边界及模型选型契合度。 (68:00)

在系统设计面试中，评估重点不仅在于给出可行方案，更在于工程深度、技术权衡以及清晰的架构边界；面对新闻推荐等复杂系统，候选人应根据岗位定位明确端到端拓扑，而非陷入宽泛的产品描述或不可控的算法细节。 避免产品经理视角，突出工程选型与量化权衡 (0:00) 面试官指出，面试中“解决具体问题”仅占考量的四分之一，其余权重分布于过往项目深度、沟通表达、分布式知识储备及业务感知力。候选人在描述过往项目时，若仅以产品经理视角谈论用户留存或功能愿景，却对 DAU、QPS 等具体量化指标语焉不详，会导致面试官无法判断其真实工程贡献与职级信号。 技术阐述的核心在于讲清架构决策的推导过程（Trade-offs）。例如在新闻数据接入链路上，将整篇未经处理的文章直接塞进消息队列会引发 Payload 膨胀并拖垮队列吞吐，而使用自建普通文件系统则缺乏磁盘损坏时的数据完整性校验。合理的工程设计应是将不可变文章正文持久化至分布式 KV 或对象存储，仅将轻量的文章元数据和唯一标识传入消息队列进行异步解耦。 数据清洗与离线在线管道划分 (20:00) 针对新闻预处理，系统需建立文本特征工程，包括停用词过滤、分词、N-Gram 以及词干提取，形成干净的词袋向量并映射至预设的主题分类。针对时效性要求，系统必须清晰划分离线（Offline）与近线/在线（Online/Nearline）的职责边界：分类模型的重训属于低频离线批量任务（如每隔一至两周更新一次），而新抓取文章的打标推断则应由近线消费服务实时调用模型完成。 在数据标注层面，早期冷启动难以依赖无监督学习，通常需结合人工标注头部热门数据以维持模型监督学习迭代。同时，新闻系统必须在接入层引入基于文本相似度（如 Jaccard 相似度）的去重机制，防止多源抓取的同质化突发新闻破坏用户体验。 推荐范式对比：基于内容与协同过滤 (40:00) 在推荐引擎实现上，候选人提出的基于标签倒排索引（Content-Based Filtering）虽然直观，但在工业界存在极高的人工标注与标签维护成本，且容易引入人工分类偏见。相比之下，主流采用的协同过滤（Collaborative Filtering）直接依赖用户行为矩阵（如点击、阅读与评分日志），无需对文章内容逐一打标，具备更优的泛化能力与转化效果。 协同过滤的局限在于难以应对新用户与新文章的冷启动问题。工业实践通常需要退回至启发式规则兜底，例如为新发布内容配置发布商信誉度基础权重，或结合粗粒度分类与严格的时间衰减（Time Decay）因子。为缓解离线模型更新延迟导致的刷新无感知，可在客户端轻量保存短期点击上下文并随请求上报，辅助服务端完成即时推荐调整。 面试策略：架构闭环与岗位定位匹配 (65:00) 从0到1的系统设计面试核心是搭起可运行的端到端框架，展示快速迭代能力，而非一开始陷入局部算法细节。候选人应遵循防御性面试原则，避免主动引入宗教合规、冷门边界条件或自身不熟悉的底层存储组件，以免偏离主干链路。 此外，候选人需根据岗位类型动态调整阐述侧重点：软件工程师（SWE）面试应侧重微服务拓扑、高可用吞吐、去重解耦及全景架构图的呈现，将算法模型视为黑盒；机器学习工程师（MLE）面试则须深入特征工程、离在线推断闭环及召回排序损失函数的数学推导与选型匹配。 本场模拟面试展现了系统设计中自底向上的拆解思路，但也暴露了未主动进行容量估算（DAU/QPS）和缺乏全局立体架构图的通病；对于推荐系统设计，平衡工程架构可行性与算法演进路径是定级的关键。

[0:00] 面试官: describe one of the most challenging past projects [5:00] 面试官: 太偏产品经理（Product Manager）视角，缺少工程深度 [20:00] 面试官: screw up 你的 bag of words [25:00] 候选人: Trump says something about the coronavirus [27:48] 面试官: Which part is offline, which part is online? [32:55] 面试官: How do you get the training data? [36:20] 候选人: recommend the most popular news for each single label [45:00] 面试官: 你觉得这个架构里哪一部分还可以被改进？ [50:00] 面试官: 比如我看了特朗普的白宫发布会（White House Briefing），又看了疫情更新（Coronavirus Update）。Jonas 也看了疫情更新，那系统就可能推荐他去看白宫发布会。如果我喜欢这两篇，但我不看环法自行车赛，系统就不会向 Jonas 推荐环法。 [60:00] 面试官: Above average [65:00] 面试官: Data quality 就决定了你的 Service quality [65:00] 面试官: 不要给自己挖坑！

[0:00] 毕竟我中途还换过组之前的话我做到一些recommendation的一些问题所以说这个而且拿钱啊所以说这个是还是比较主席的 现在的话 因为新的一个新主又做了新的T-shirt好的,好,我们今天呢就是面试,还是按之前的Mocking2用来啊,就是我觉得这样的话,就是每个人的话背景不一样,然后见也不一样,然后理解也不一样,我觉得是比较,比较。就是这个是比较wide的,然后大家可以有一些reference,并且我觉得能主动站出来说我想来参加MockInTV的大家肯定都是有两把刷子的,还是有一地方做的很不错的地方,所以我觉得大家有个reference,然后几次的话也从事探讨一个问题。然后 again,我们再说一遍,就是我们再说一遍就是System Design Interview还是参考主要参考的是四个方面第一个方面就是我会问一下你的Path Project describe one of the most challenging past projects 是吧?然后看一下你过去负责工程的复杂度这是一个方面,然后其次的话,我要问的第二个问题就是,就是我会观察,我会问你个问题,但问题的解决不是所有的,这个问题解决只占四分之一。有 problem solving in transatlantic frequency另外四分之三,一方面是 past project,一方面是你怎么样组织你的语言,怎么样articulate problem,怎么样communicate,是吧?然后起字的话,还有是knowledge,就是你对于系统的理解,system design系统理解,knowledge,然后有一些business sense,以及business sense的这个基本上属于communication那块了,当然有的也属于knowledge吧,就是你要是有knowledge,对,然后好的,能share一下screen吗?你能上一下子归吗分享OK 哪一个 不太熟悉这个怎么怎么 share screen OK 行好,然后你如果不接近的话,开下摄像头,这样的话我们就是真正的mark一下这个interview的过程,因为你其实mark interview的时候,你现在interview的话,system design interview 肯定都是online的,就是都会看到的。Share screen 要是share,哪一个就是你那个document,你上面写吗?还是?对,就是share你的那个document,因为是你写嘛,所以我要看着就是怎么样组织这个问题。其实你还没有给我那个...在那个chat里面?对,没有给我那个requiz你把讲 我让所有人都可以Edit好了谢谢大家或者你给我email,你发个email给我来给你access。其实我已经request了。好的好的好的,我们可以先get started。所以我叫你解释的,我会想解释的一些最多的挑战的事件,你做的在前面。可你可以解释一下哪些挑战的事件你做的事件在前面。OK,so in the past I've worked on a system design project on the back end side. So this is a recommendation system.这就是一个新推出的系统每天我们有很多人的公司和很多媒体的公司, 公司的公司, 公司的公司。所以,媒体的公司是每个小时的公司, 每个小时的公司。 news articles that一些人会觉得有趣的, and hopefully they will improve their user engagement, their interaction with our websites.OK,so I think you mentioned a functionality, business functionality of the service. I have two additional questions. 这一块是你自己要描述的。第一个就是skill和business impact,就是skill是什么,business impact是什么,就是你做的functionality一方面,然后其次的话是很重要的一部分是你做了其中哪一部分,我知道就是很多时候他会很看你做哪部分原因是为什么呢,就是很多人很多人做的时候呢,他就是喜欢把一个很大的project揽到自己手上,但是他其实是做了其中很小的一部分。这个是非常忌讳的所以你这块需要去说不仅要描述是做什么方面的然后你再描述一下其中哪一部分是你做的你做了哪一部分然后最最最最最最后的就是他的business impact 什么就是就是最好是你一次描述出来如果没有做到的话会问你 [5:00] 对OKto the user at real time and how and each time the user maybe clicks onon the article we recommend and after that is going next time the user is going to see maybe new list of recommend recommended news.所以,这个影响是我们的,在这个项目中,我们有增加的用户的用户,很多人都在我认为了学习的 habit of reading news during the during the like in the morning way of evening to check the news on the website is to去 keep themselves updated. OK, I still don't get a concrete number. So how many users are using our service and. How many. It's like that is about so in that is in two phase now when I when I left the team that.他们会把OK,so you have millions of users. How many NILs are you going to classify and recommend it? Like the skills and NILs like per day, like per rolling 24 hours?for rolling 24 hours that was about quite a lot so in in a global scale it was quite a lot but but for this service we initially launched for for the US that was like嗯...好...1000我可以给你一个级续级续所以,在某些部分的事件中,你能够寻找到一个公司的事件吗?Framework, DatabaseOK,so we basically我们在做的the serving part to read these we're going to like do a lot of rankingand to decide which kind of news we should recommend to the users.yeahOK, I see. OK, so today's interview problem is very similar to what you described.这块我先暂停一分钟, 稍微给一点feedback, 这块的话我个人感觉就是你说的还是不够细, 不够细的原因是,这个就是你现在describe的还感觉很像一个产品经理describe给我的就是像一个 Product Manager 给我的就是你比如说 如果从一个 Engineer 的角度的话我可能跟 Kai 说 你这个你是用什么 language 啊 用什么 database 啊然后你的那个flow是怎么样的,你data从哪里来了,进行简单的怎么样处理,你中间有什么,比如说你用了一个message queue,然后什么的,然后你machine learning用的是什么framework,这个我可能,你比如用mxnet,还是google的那个,对这个更感兴趣一点,我感觉现在描述的话,就是fitting to product manager 描述的,可能OK,就是说可能OK,depends on interviewer,但是我个人的感觉就是,不够 in-depths但是我一定会timebox我问的这个时间,我不会超过五分钟。我不会超过五分钟这个时间,因为这个更重要的是后面的interview,但是我直观的感觉就是不够indepth。这个indepth的描述是非常重要的,因为index描述的话,他能看到你对这个project理解有多么深入其次的话你在这过程中真正扮演什么角色,这个interviewer是能感觉到的。大概知道的,比如说你从头到尾的设计的话,你就说我当时为什么选了这个database 是吧?我为什么中间加了一个Message Queue,为什么选了MxNet,或者是Google Machine Learning Platform,为什么选它是吧你这个描述不够详细的,这个需要先练习,但是我直观的感觉就是太general,对,现在的感觉就是太general。其实我之前的练习就是怎么样把一个复杂的系统把它说的先从overview的说的简单,然后之后慢慢的去细化。对,这个也是make sense 的,也是make sense 的,因为很多时候,很多时候你要看那个面试课, [10:00] 有的人的话就喜欢就是你描述一下你的问题他可能这块都不计分都有可能但我的话我是会比较计分的其次的话当然每个面试官都不一样每个人的风格都不一样其次的话again, past projects 会决定你新工作的高度就是如果你上来就跟我上一次说的就是你上来后你是你做了一个什么end to end design recommendation system那我照你过来就有可能做machine learning infrastructure 的那个 head 是吧是吧,或者是co-end但如果你只做了recommendation system其中最小的一部分什么tuning啊或者是notification啊那我过来的话可能就是照你做来做一些做一些这个feature development是吧,这个本质上是不同的所以这个past project是要花时间的要贴切的描述出你到底在那个里面贡献了多久就是贡献了什么,然后做了什么功能这个是要花时间的不能太genal,太genal的话我拿不到任何 signal我只能给一个非常 general的 pass or not pass the signal我不能给你 level of justification signalOK 好 谢谢OK 好OK so let's look at the problemso today we're going to createa recommendation system similar to total so you're going to recommend personal news to individual user so here is some high-level requirement but we can discuss details here the most important thing is that basically you have two data right you have you have news and you have users and you want to make sure you can support internal news publishing so for example the ultimate you know user generated news from your from your selecting users it could be MVP user mostly important user or it could be other users who actually work on the media part so you also want to make sure you ingest news from other news website like other news website like like you know sing-lars you know New New York Times, et cetera. You want to make sure, maybe you want to classify the news and you want to analyze the user behavior. And the most important thing and it's a cool feature is the recommended related news to the right user based on user behavior. Okay. Yeah, so, so for a recommendation system, that is, that the whole, that the whole system itself is like, you can separate into two parts. One is like getting an article for news. From the user perspective, so one is how to get for, get a read a news article from your system. And the second is how to get a recommendation article list those are two different features different different what perspectives and from the like from the like publishers perspective this is on another new system to another another separate functionality that is actually another system to to like to define the entire process to so that so that a public so that the article will get to into the production into serving so i should write down somehow but can i edit yeah i think you if you refresh page you should be how i did try okay okay this is a few sections this is an uh system to publish news to serve to pass over to the serving. Part two is for like get the content of an article article three yes I get news recommendation those are two separate out you can say different different separate systems but work as a whole to make sure that whole the whole flow will actually is successful okay okay yeah So let's take a look at part three. So yes, since you have listed in such an order, part three, first, how to get the news into, no, no, part one, sorry, part one, how to get the news into serving. Get the news ready for serving. So basically, two sources, as you know that down, uh some news websites [15:00] Gracias. Y el otro es, ok, mvp users. ¡Ap, a poco! Ok, pero, ya, so for this kind of, I would say for the news websites, usually We have separate calling systems. We just call the o los famosos webes, webesites, regular basis, para que la gente se hagan las nuevas. Gracias. Gracias. So for this part, We have a qualifier. Gracias. No hay problema. Gracias. la search is yeah so let's for this for the sake of simplicity let's assume you have a the website has rss theater and you actually listen to this rss theater and you you already got it you know you don't need to worry about crawling or you know you get the information yeah i get the information you're getting yours when it's published yes okay okay so let's say after after crawling This is DB for Niels. Gracias. Gracias. Y así, para la contención, para la contención. Entonces, después de obtener esto, So we have to like... y también también y poner algunos identifios para la news. Ok, este es el primer paso. y en el segundo punto y ¿Qué es lo que se llama? Carseifier. Gracias. classify the article according to the contents Y finalmente y también sí ya y también finalmente hasta hasta el caza de la tienda de some sort of like DB ready for serving. Gracias. Gracias. Ok, so this is part one. So after up so. Gracias. So, yeah, let's make it more specific. So the raw content, it doesn't have to be like, it doesn't have to be really, really systematic or really formatted. I can say it. Entonces, una vez que el artículo, el contenido de la página es publicado, el artículo no es publicado. no es modificado al más de la vez no es modificado al el único que lo que hacemos es leer es y no write, read heavy, no write You can call it immutable, right? Yeah, immutable. Gracias. The immutable articles. Gracias. um con immutable articles, we can just save it as, possibly we don't need a DB or something, we just save it as logs. Each article is represented as we put the log into the message queue. So since analyzing the entire flow to classification and leap factorize and etc. The entire flow can take a considerable amount of time. So put that in the message queue, we can make sure that we can El artículo puede ser procesado y consumido, pero no tiene que ser en 100 o 200 miliseconds o algo. Es bien para que se sienta o... Segundo o 10, 20 segundos para terminar el proceso. Es un poco no recomendable. Porque la mensaje queue se puede poner metadata, y el artículo puede ser muy grande. A lo general, el artículo puede ser muy小, es de unos cuantos de palabras. Pero hay un artículo puede ser muy muy muy muy muy muy No quieres poner un mensaje que se pone un tipo de datos más grande. Es decir, que es el sistema de la máquina 1K, 5K, estos articles de size variable [20:00] 非常高一般来说,我是不建议把这个article 放到这个message queue 里面,这个是design 的时候会被question其次的话,Article不放了DB的话,它有一些Data Integration Test,Data Integration就是Integrity,data integrity,会被质疑,就是比如说你这个放了db的话,如果你这个disk坏掉一部分怎么办是吧放DB的话他就给你 throw error就是你保证你从DB读的data一定是正确的要么就有DB error但你放到filesystem里面不一定的啊这个是不一定的,就是你可能读的一个 data 可能和你之前的完全不一样了,这种大部分时候的不是问题,毕竟是 article 但如果是 finance data会有很大的问题所以,OK,所以所以,我们先说,我们先把这个内容放入一个大型的大型。所以,我们先说,在大型的大型,我们不需要…那些大型,我们可以做到…we only have like key value a value db is good enough and it does it doesn't have to be它不必要是快速的所以, as long as we support所以我相信是有利的方法。是的。嗯so after that we just sent a message to the message queue saying hey这个特朗志的记忆是这个记忆是这个记忆是这个记忆这是1OK, then do some more updates.So first we put some identifier, but I think that this step is actually not needed since we already haveso if we put in so we already so since we use a key value key value store we already know what the what the identifier will be所以这会很不重要的事情这次是随着的东西是随着的东西所以在做的东西我们可以用一些的你也要想要要去做一些文字的文字, 如何要做它, 要做一些文字OK,所以,所以,所以,所以, first we're going to like put it in the lab, but back of words,的 words like so instead of less在整个 article里会说Hey, let's say this is articles called Good Morning. So the bag of words will be good.and this is the positionthe position of the word zeroand another word mourning and position maybe one something like that so ofcourse and all the words will be like我们的解释和解释和解释和其他人的解释。我们也有很多的解释。by some kind of like five grams or some other trigrams to to make sure that if two words together make make a like make a make a like for this call一句话,这块是把一个article变成一个word或几个word相连的,一般是指一个word,然后其次的话,它要把那个词变成词根,比如说想不到一个词,比如说capitalism,就变成capital什么的,然后这种,然后把词变成词关以后还要remove一些stop words,比如说the a,这种stop words,就是这种stop words remove,因为它会screw up你的这个backer words,好Yes, I agree. But yeah, sometimes removing the stop words when it depends what kind of quality if the user really want to say, hey, I really need to stop what they're there or something or like [25:00] There. The apple. then your conscious is good, bizarre. from the article. Okay. Yeah, most likely you still want to remove stop words. It's going to be scrub your backwards vector. And then I'll add two words. We talked about how to use this. Raw Page, when I was in the design of the raw page, I was going to use the raw page to be in the Mongolia. and then asynchronous to fix the details. So this part, Jonas, I would recommend you to just make this process simple. For example, I put this immutable to the DB on the file, but in classifying this article, you can say asynchronous to fix the details. So to put some keywords, it means that One is more than a system understanding. The other is that the asynchronous is more than a system. In system design, there are some keywords like immutable, idempotent, asynchronous, One is the key to the understanding of the system, and the other is the key to the system. Bye. Okay, let's continue. I think this is good. So now you have backwards. How are you going to classify the article? Okay, so now that we have the back of words, we just-- run some kind of models to see how likely that so, and also we need some kind of categories or labels. Labels that is actually the same as like topic, news topic. And there, and it's represented as labels. And then we're going to like to run through some models to see, to examine for each article, examine each single word for this article. this word a good for this the word a b morning how likely they are So how likely they are to this article is actually related to a specific topic or not Thank you. - So here, you mentioned labels. You're going to, are you going to use a new topic? So the new topic gonna be defined by human beings, right? - Yes, topics are predefined. Thank you. predefined So yeah, after that we're going to like one through some kind of like classifier. classifiers. to examine the bag of words and see how likely an article is should this article belong to like a specific label. Yeah, an article can be, yeah, it is possible an article may have multiple topics inside. Yeah, let's say for example like Trump say something about the coronavirus. So it's an article about maybe about Trump itself and also coronavirus. So is it possible? Thank you. yeah yeah so here uh which part is offline which part is online so which part is offline the entire flow the classification is offline also It's offline is run in the async pipeline to do the classification, which is actually here. in an asynchronous way. So after the classification, we are going to like label, put all the related labels to that for this article and store it in like in the production in the database for serving. So if your part is run offline, so here you mentioned something like asynchronous and therefore run offline, right? Yes, they are offline. OK, so if it's offline, then how frequently [30:00] you're going to write and then How frequently is it going to run? Okay, how do you define offline or online? - So in the machine learning part, right? Most likely you need to train a model, which is You need to train the model, and then you need to deploy the model to do the work, right? Yeah. That's why you have offline parts and online parts. Yeah. Yeah. So In your system, which part is offline, which part is online? Like online is serving requests, and offline you do some... Okay, for this model, this entire flow is actually online. So, but for the offline, we can... yeah. So, So for the offline, this is not all part of the offline. So Yeah, so we train, we can retrain the model once in a while, but we don't have to do it as often. uh, not less frequently. maybe once in a week or two weeks. Okay, so I still have a yellow flag. Thank you. This model is classified from Nils dot labeled And the other thing, we have a news website, right? So it's a news website. So it's time-lieness. Right? And on the other hand, the label is "human defined". - Yeah. - That you can only get weekly old news. weekly old news classified. with new label? Oh no! My meaning is that this model is trained well. When deployed to this system, every time there is a new one crawled in, it will be from the message queue to make the message. Yes, because it's a news website, so you have a lot of topics, like the latest topic, Today, we have two blackheads. They have It's a vaccine, right? It's a vaccine. So, in the end, there are two vaccines. So, in the end, you can change the mode. This is a new topic. Thank you. Topic, we usually do not do a final step in the process. But I found that - Okay, okay, okay, I see. So the topic here, The top here is actually quite--it's not real-time. Basically, it's a It's more like politics, something like that. Yeah, something like that. Okay. Okay, so it's not like something very, very new, like a coronavirus trending or something. It's not like a Twitter topic. It's not a Twitter top icon. Okay. It's random buzzer. Yeah, that's me remove this. Okay. Okay, makes sense. Okay, so let's examine the flow again. So you first crowd of news and you put the news into immutable database, immutable raw news into a database. Then you pause the news and you break down to backwards. And once you have break down to backwards, you use the machine learning model to classify the news to a selected range of topics. And then are you going to assign weight to this topic? Like this news is like belong to this topic or weight like something? Thank you. for that particular topic, yes, maybe we should assign some topics for some some topics that are too generic way. So yeah, let's say it is too generic. So the weight will be should be a little low. So that's is basically like like T F ID after like inverted let's go Let's look to the IDF part. Okay, so additionally, I do have a question here. How do you get the training data? How did I get the training data? From Nils to topic, how do you get your training data? from the [35:00] news topic to how to get the training data so so you have machine learning model classified from news to label right from news to label a topic then how do you get the training data for this how to get the training data for this one. Yeah, I just... so each day we can um So each day we can just put for the news, since we have already stored up, we just maybe put some kind of like, Hmm. so we only have to only we have to like let's say we train this model once once a week we only need to examine the the news for the past maybe a week to two And that's it. just retrain the model with the existing data. That's good enough. - Right, but you don't have the labeling data, right? You don't know-- - I don't have the labeling data yet. but So that is in, yeah, we can label the data in two ways. One is like having some human operators to do that for and and of course it's actually better to like only label. only said yeah human operators are expensive so we we can limit the the news the article news to the very popular ones. But for a lot of other articles that are not popular, we just ignore them. Okay, it makes sense. This one I see there is a comment. So bootstrapping impossible to get training data. So this training data you have to have supervised data. Supervised training data. You definitely need to classify. And this is a certain thing. And it may be an ongoing process. It's classification. You can change label after you change. For example, you have a big topic before. break some of the few topics, maybe before the data chain will not reclassify. This is also a lot of human involved involved. This is a certain thing. So if you just say some of the system, you can't do unsupervised learning. At least from now, it's hard to do unsupervised learning. And then the other thing, there is a lot of training should be done in batch rate. Prediction can be done by offline or online. This is reasonable, training is usually offline. And the prediction is the prediction. For example, you have already written a lot of data, you can use a new model to classify it. This is the offline and online version. So I think these comments are good. Okay, so here, so basically if they review the flow, let's assume that you have some human operator help you to classify data. you have from news. to labels and what you're going to do next. Oh, OK, so I have the news for what with neighbors. Your basic how you run your strike and then you have a server to process the news and map the news to the labels. Yes. And what's the next? So the next is we're going to, well, so in serving for the recommendation, so we do need to like update the record. So yeah I should just break it in the into two parts so so recommendation is I yeah the next step is you is how to get the content of the article. So yeah, I already answered that. So we since we put in the key value store So we already able to like get an article, get a news article from from the db that's already there but the problem is how to get how to get the recommendation Thank you. the news list for users. Thank you. So, So there are many ways to like to do the kind of ranking. So if the system is starting from very fresh, and we would like to just go ahead for some some simple implementation like like we only recommend the very For each label, we only recommend the most popular news. for each single label to the user. and that is actually yeah it's very actually very so for some in initial design that's maybe good enough So here, we don't have news popularity. We only have news backwards and news map into label. We don't have news popularity here defined. Okay, so in that sense, like yeah recommendation is actually what what i'm saying is recommendation is actually a incremental [40:00] improving process. So there's no specific company that can just make one specific recommendation. engine and then works everywhere. Actually, that is a very huge cause and highly risky. So just to make it To decrease the risk, we start from something simple, like, according to popularity, that is one thing. and maybe like the other publishers publishers popularity or maybe some population or whatever. None of the popularity. or credibility. you Okay, so you can define some publisher credibility, but popularity you only get later, right? really brand new is published. You don't really know what's the popularity of that news. Yeah. Yeah. Let's say we already done that part. The next thing is how to recommend Yeah, have to make it like more robust. Thank you. So we can do it with a like as, in a solution similar to the search engine. So basically we put that for each specific label. We do have like a posting list for each specific label. What other article? Thank you. What are the news? IDs that are relevant to that specific label. Thank you. And this is actually a sorted list. is sorted by... maybe like relevance sorted by relevance Thank you. in such a list. So after, let's say create a very simple example here, like for each label like politics. Thank you. We have a news ID, maybe like A, maybe 1. three five six so each of them is sorted right based on some of it like more okay based on the publisher credibility what feels it what other fields you're going to look at here relevance. relevance relevance okay all relevance and you'll see Relevance, this is maybe about sports and let's say three, six, ten. So yeah, so each time, So, each time we do grab some kind of, for some labels, we just grab multiple lists like this. And then, do some kind of like rank, take some of the ranking. Let's say if the user needs only about 20, 20 articles on the website. We just grab multiple multiple popular labels and then we do the do the ranking and pick the pick the top 20 to from them Okay. So, yeah. So here, my question is, again, you want to do a personalized Nielsen recommendation system. How do you know user preference here? Thank you. Yes, so the user preference is we can, so there are two ways. One is user history. Yeah, actually the user history. So with the user history, user history comes with logging, comes from logging. to log yeah we put all the like the user browsing history and article browsing history inside the login so once in a while we have some some other some other like model training more trained models to to update their existing users users preference and then store it in somewhere. Let's say for the user one, the model looks like a prefer labels Prefer labels. For this politics, the user is actually visit how many. Score is like 0.9. This is very, maybe, like politics very much, like very much. [45:00] and maybe for sports, that the user only clicks maybe once in a while, maybe 0.1, that he doesn't like it that much, and etc. We once in a while we generate such a model. And then we just pick the related categories for the user. But the prop, yes, and it also makes sense. If the user group is not that that large so for the active user if we say we did so for let's say we have like one million user let's assume that the daily after the daily after user and yeah the users that is active active users in maybe maybe an hour. It's actually only a quest more. It's actually way below 1 million. We only regenerate the model for the users that are actually active in I mean, maybe a past hour or something. That would be like... I don't know, that would be like hundreds or maybe thousands. Thank you. Thank you. - Thank you. Okay, I think it makes sense. Yes. Yeah, so here. So I don't have additional question. Do you think which part can be improved for this part? Where can be improved? Think about which part you can improve for this one? Yes. So which part can be improved? So to regenerate the user model, it actually comes with latency and comes with logging. If the user each time clicks an article and send the logging directly and regenerate the user model, the user model that would be very time consuming. It takes at least minutes, I guess. yeah usually so to to make sure that each time the user gets new recommendation list um to make sure that that happens. So we can't only rely on the logging itself. So one more approach is after a user clicks, an article that the user actually stores some kind of the user history locally and then just whenever he asks for a new recommendation list, he sends that particular the particular black, the use of history. serialized use of history to the to the server and from that user history and user information is going to like grab some other some other articles or maybe some other labels. so that the next time the user can see different, yeah, different news from different categories and different articles. Okay, sounds good. I'll do some other things. I'll put the question in the bottom. Thank you. Let me talk about communication. I think Archiculate and communication is very good. I think the question is very clear. I think it's not a problem. And from knowledge perspective, I think it's a yellow flag. The fact is that you can put the news in message queue and then use database to send it. I think it will affect you. If you do an interview, you will be able to say that you are going to be a little bit. And communication and knowledge. And the knowledge of this approach is the content filtering process. This is what you do with recommendation system. Recommendation system has a different way. Content filtering is a different way. You can put content mapping on label, and then user mapping on label. Then we have two two mapping. This is content filtering. Actually, the best thing in the technology industry is Collaborative filtering. Collabative Ratio Filtering, I was talking about YouTube. I have mentioned that I don't care what you're doing. I don't want to write an article. Map to label on, because if we do content filtering, from article to label mapping, It's very hard to use, you need to do label data, you need to do label data, you need to do label data, you need to do label data, you need to do label data, and then you need to do label. And then you need to do label, and then you need to do label. Content filtering. This is very hard to use. But from the topic to the topic, or from the label to the label, because the article is good to the article, it's pretty simple. You can see which article, you have to wait for the time, then you will be more interested in the label. [50:00] 这部分是不需要做这个Machine Learning或者说不太需要做这个但是你还是有很大的Machine Learning Cost其次的话这个Label 划分其实是非常human biased 是吧你怎么样比如说这个你划分成Politics但是Politics可能还有一些Chronavirus相关是吧有这样的相关那个相关这个Label 划分是非常human biased 的这个的话是所以Collaborative filtering 相对来说是就是不把article和人划分到这个label上不把article和人划分到label上他怎么做呢就是说比如说我看了article比如说我看了article Trump的那个什么比如说white house briefing然后我又看了article比如说我又看了articlechronovirus update然后Jonas看了chronovirus update那他可能就更改进去看white house briefing然后我看了这两个我喜欢这两个但是我不喜欢看article比如说不喜欢看article比如说这个还法自行车赛啊是吧因为我看了这两个我不喜欢看还法自行车赛那可能很有可能又不像Jonas推荐还法自行车赛是吧他是根据这个就是根据跟你相似的用户看了什么和不喜欢什么然后决定向你推荐什么不向你推荐什么叫 Collaborative filtering你其实就免去了把article从news到label这个classification的一部分就检查了这一部分我觉得相对来说recommendation system这个cost更低点并且很多经验表明它这个accuracy好像更好一点但是content filtering也work的就是它这个首先有更多Human Cost其次的话可能就是Pre-Session可能没有Collaborative filtering更好一点这是一点然后其次的话这个Timeliness因为你是个News Recommendation System所以你可能更多需要就是Focus就是你上来就说Timeliness应该是其中的一个Goal那我在这个System里怎么样解决了Timeliness对吧这是对这个的话是你就是分析这个 business requirementrequirement是比较重要的就是你分析 business requirement你是个 news recommendation你要推荐昨天的 news别人都知道了你这个系统就没人用了所以这个 timeliness是很重要的一个 goal所以你一定要是就是思考的就是怎么样我的设计能能满足这个 timeliness的的goal所以这是很重要的一部分是吧OK对然后其次的话因为它是个machine learning system所有的machine learning system一定要思考的问题是online和offlineoffline是modal training和modal deployment的部分online是怎么样serving就是ongoing request一般都要花单独的半分钟时间说明哪一部分是offline哪一部分是online这个是所有的machine learning system我觉得都是要比较提一下的但是我没有follow up最新的Machina我不知道有没有这种continuous的什么deployment到Machina model有个continuous deployment至少我上次follow的时候还没有这种什么Machina model能边deploy边更新边serve边更新然后这块的话还是从business的角度有一个question是完全略过了duplicates你从不同的news website去crowd的时候你很多爆炸性的news都是重复同一条news是吧你如果像一个user recommend 同一条news来自于不同的website user会毛掉是吧在搞什么这个也是和business相关就是duplicates和timeliness是和你的business相关的就是这个你的business use code 相关的我当时面试之前我大概想了一下最后一点的话你往下 scroll 一点的话往下 scroll 一点的话words processing我感觉这个knowledge不管是stop wordsback wordstf idea你还是不欠缺的我觉得有个不太明确不太明确是你说的是collaborative filtering但是我觉得作为user-basedcollaborative filtering就是那个index没办法做很实时的一个更新其实我当初的想法就是说做这样的如果是说我们针对于keyword like collaborative的话我们就可以做成像search engine一样做一个一个的这样的一个try一样的东西然后就做成做比这个word这个word,这个news里面的word对应的是哪一个相关的哪些news ID是相关的然后我们就有时候他们要更新这个东西的时候我们就把这个这个词条这个整个路径给大家给更新掉然后就可以说大概就这样 [55:00] 对,我看到这块留言有问如果是collaborative filtering的话怎么样解决co-start problem,这个是比较难解决的,这个确实是比较难解决的,可能之前的这几个news,之前前几个user可能是有小白数的,就是帮你做label,就是user帮你做label这部分,这个确实是比较co-start的,可能之前的话你只能用一些heuristic logic,或者用content filtering,是吧,你可以把这两个combine起来吗?这个确实ColdStar的这个problem不管是 content filtering 和 collaborative filtering 都有这个问题,因为 content 的话,如果你是一个新的 topic 的话,新的一个 label 的话,那确实也是有 cold start 的 problem 。嗯或者是郝林你有什么想法吗?就是Cold Start Problem你有什么Recommendation吗?我也没有,所以我就想问问来着,谢谢。你说对,就是就比如说把这两个结合起来也许可以解决一部分问题。其次它是一个news website,所以这个肯定是有time decay的,所以这个timeline是一定有 decay的,就是你时间越久的news你可能优先级越低。时间越新的News可以分的更糙一点,但是你的频分越高然后这块人们做news duplication的啊你接着说我可以讲话吗就是刚才呃关于那个costar问题我觉得你说其实挺对的costar问题基本上是没有办法用clap的filter来做基本上也是用非常简单的书背似的或者是很简单直接推荐给就像你认识的人的看过的news给你就好了然后另外有几个问题就是一个是刚刚跟你讲那个Continuous Deployment,好像你说这叫Online Training,我有几年没有follow了,所以我不太清楚了,你可以解释一下online training吗?Online training就是说你有这样的model,然后你收到新的一个data之后,你直接在在线直接training 但是通常这是很难的然后基本上我觉得业界做onlinetraining也很少就是比较细的,我觉得是,感觉直观上是可做的,但是好像,反正我基本上之前没有听过啊,如果你有article的话,你可以发给大家,然后你可以放到这个,就是你可以comment到这个这个上面就是Online Training的一个link,有什么新的paper的话可以看一下分享一下谢谢大家好 然后我看到谢谢大家谢谢大家然后设计推荐系统是专注在model本身是根据面试者的背景的?对的,这个是根据面试者背景的。这个我之前和Jonas聊了一下,他说他做recommendation system,他问所以这个的话如果你没有做Machine Learning相关的,很有可能不论这个Machine Learning相关的这个系统。但是如果你面试的是Machine-人力相关的系统,它有可能会问你这个问题。要不和你职位相关,要不和你过去的背景相关。最后我再说额外两句,就是Jonas我觉得这块的话从始至终没有谈到两个问题,一个问题是quantity,就是你没有问我多少个user和多少个newsOK,这个的话还是要问一下,就是因为你不同的Quantity,你可能Solution会影响你Solution,其次的话,我觉得这是一个套路吧,就是你上来一定要问Quantity的Analysis,就把它定性,定性下来以后你要定量问一下,就是多少个News,多少个User是吧,然后,然后你才能大概就是具体分析嘛。这个是要问的然后其次的话,另一点我觉得没有问的谢谢大家So duplicate not good at all?沉浸到这个语境中,这个scenarios中,这个context中才能想到有duplicates的问题。那一点没有问我的,就是没有做,就是没有没有画出来是整体的Architecture我觉得是可能还是一上栏的话先花几分钟解释architecture然后我还是建议architecture先画一个图然后把每个component的定义一下我觉得这块定义成三个part是挺不错的但是我作为一个面试官来说,我没有看到一个立体的一个architecture的照,你知道吧?我觉得在线画读是难了一点,但是不管是用Google JAR还是什么,你得画一个大概的图。好,然后我最后,对,我最后,你让我试一下screen。我觉得总体来说的话,这个communication非常好然后 knowledge 我觉得大部分还是有的有一两个inlow flag不成问题,然后problem基本上solve了 [60:00] 我觉得Background 的话也不错,就是整体来说这个表现还是我觉得会above average的,我觉得还是很不错的谢谢好 让我下一下我的 screenOK 先停止了我最后想说一点就是我记得我之前跟大家分享过一个就是我做的这个Personal Recommendation System,这是我七年前做的,就是13年上研究生所做的然后这块的话就是就是做的时候就是当时有点像其实是有点像头条了,但是头条是12年开始做的嘛,我是13年做的,所以另外是个研究生论文了,所以很多时候都是研究生写的这个,就是学术界比较偏学术的,也算比较偏学术吧,但是就是Crawl的时候,当时Crawl了,如果你看这个mojo的话,Craw News大概是招了50Gb的data不是个Gear by Theta 我忘了可爱的上买OK反正是curl一些data,然后这块的话是有一些duplication,然后把这个data,我觉得就是缺少一个flow吧,但是我这块就收下这个当时设计的设计的 flow 然后就是就是complete 一下这个 problem 啊就是先 crowd data 然后然后classified data 然后classifier 然后有个duplication这个duplication是有一个专门的duplication algorithm 你们问这个duplication algorithm 是专门有个duplication algorithm我们当时用的是我们当时用的Duplication algorithm是谢谢大家这是我当时的接屏幕,这是我们当时的Crow的News,所以它有一个News,我Crow下来以后呢是一个HTML,然后把它我只留了第一个image,就是说这是一个原来的news,我拨了以后有一个image,剩下的这个text然后这块有一个algorithm叫Jacoat similarity,你可以用这个来dedoop这个Dtube效率还是挺高的,我记得当时的Dtube效率还是挺高的,它其实算法也比较简单。然后Dtube效率挺高的所以这个Cloud的Data是放到Mongo里面的然后Process的Data我记得好像也是放到Mongo里面的这是两个data然后这个data完了以后呢然后我当时做的也是把它做Tax,就是把那个News变成一个Tax,然后有一些UserData然后呢用这块是想用collaborative filtering的但我记得好像当时没有完成这个collaborative filtering感觉没有完成然后就有一些其他的东西,但是总体来看我觉得就是在这次面试之前 Jonas的话我推荐你画一个大概的high level 是吧这个high level 可能crawler 这块变成一个是吧Color变成一个,然后Ddupe这一块,后面发现可能需要Ddupe,然后加一个Ddupe Mojo,然后这个format的话就要写一个,比如说这个这个data format是什么样子,然后这块的话应该有个text processing是吧有一个 text processing 这一块然后 text processing,然后有个 tag,然后有个 personal category database,然后 user database,然后之后有自己的 collaborative filtering是像 rating, email, click, user history还有其他都是Secondary但是我觉得就是这几个还要就是Crawler, Data Model, Tag, 你要用 Collaborative Filtering 的 Recommendations in System, 我觉得很少有的好好对,就是这样子的。好,我们看一下大家有没有额外的问题,如果有的话我们回答一下。谢谢大家谢谢大家谢谢大家谢谢大家没有的话就到这好了非常感谢大家非常感谢 Jonas 您有问题就是你就是如果说这种推荐系统的话一般你是觉得是应该是model driven的就是business model driven的还是说是system driven的好问题,model的话是一个,就是这块的话你是从0到1,从5到1,所以你这块的话你从最后结果来看可以是model driven的,data和model,准确来说是data driven,data quality就决定了你的service quality但是我们现在是从零到一设计一个框架所以你其实计划的话要把所有的mojo和框架打出来然后你这个data和modal优化是你拿到data以后才额外优化的所以从工作的角度来讲data和modal是可能是最重要的但是你从0到1的时候你这个框架怎么化,你这个service怎么划分你都有哪些mojo是这个考察 [65:00] 重点那就拿这个例子来说其实你的数据就是大的分为三块一个是用户数据一个是新闻数据还有个是用户跟新闻的interaction的数据就只有这三类然后你的模型的区分就比如说你用不用tag 然后其实tag可能是一个就是超级参数在别的模型里面对吧或者就是呃就是就因为我的意思就是说你是不是应该从这个角度去讲因为而不是说我要搞一个tag然后因为tag只是持有了一种方法而且这个东西achieve了某一种就是business的问题对对这个也是我觉得这个也是make sense的就是这个tag其实是中间的一个一个不facing user 的一个变量所以你有没有这个变量其实都可以我觉得这个还是要看就是Jonas是怎么样怎么样把这个问题阐释的我总体来说我感觉阐释的过程中是非常清晰的这个是因人而异的就是如果你觉得不需要tag 你可以用其他方法来把这个问题解决也是可以的你挑一个你可以做到的就可以,而不是说你要去做一个体现你在这方面有经验就是一个比较generalized solution你不应该去tuckle一个比较generalized solution而是你是先提出一种可行性方案对你要你从零到有设计一个东西你肯定是先把所有的module和service 换了表示这个这个方案可行的然后再在优化最好的是你把框架搭好以后说这其中哪一部分可以不停的迭代不停的优化就是fast iteration and fast optimization 是吧反复的optimization 然后这样的话可以达到它的service比如说你说用 content filter in加上的collaboration, review, filtering,这两个可以反复的优化,然后接待过去,然后使user experience 更好。那个学院制造所以最重要的最后最重要的还是business case和user experience但为了满足business case和user experience必须有非常quality data然后你设计的system必须能满足这个use case和这个这个Quality Data Collecting的过程是吧然后你才能满足所以是从0到1的过程谢谢好这个问题的话我之前在做的时候其实还有一些可能不太知道了吧就是recommendation content这种东西的话就容易会出一种legal的问题比如说Christmas当时我也不知道Christmas这种article也许Christmas的时候很火但是对于他们Muslim或者别的宗教的人来说他们其实很反感这种东西所以说在在他的ranking的时候你要做很多business logic嗯对,这个是你在工作中有的经验,你在面试中不用提这个什么legal啊,什么concern啊,完全不用提的,这个tech corner可以浪费很多时间。你可以提一下可能有legal concern,然后跳过了追问很细了,这个也是合你工作经验相反的,我觉得这是比较好的一点好就是 again,就是说我觉得反复强调的一点就是面试中反复强调的一点就是不要给自己挖坑,是吧?不要给自己挖坑,不要说就是比如说你对Cassandra和DynamoDB不熟,你不要说这块应该用Cassandra,然后别人问Cassandra,然后突然忘了Cassandra,中间是什么东西了,这个不要给自己挖口,你只谈自己熟的东西,用自己熟的东西能解释出一个end to end system,work就可以了,然后不熟的东西尽量避开,不要给自己挖口。好,我们还有额外的问题。我有一点点补充,因为对推荐系统来说,因为是好像是一个非常经典的对测试这个Machinoline Engineer的这样的题目。所以他有专门的测试那个ML Design的这样的一个cycle吧。然后他在这个cycle里面,他一般不太关心这种System Architecture怎么去搭建,而他会比较focus上怎么去做这个modeling。所以我想说如果我们这次的这个是更focus on system design,可能就需要一些是不是简单去提一下,就是说我under the hood我有这样一个machinology model has been trained and has been built,就不需要踏实太多的 detail了,因为对这样的一个NEOs的推荐系统来说可能需要一些NLP的基础所以我觉得不知道是不是这样对,我比较赞同啊,就是,但是我这边的话,因为我,就是我们assume 这个是面的software engineer,然后我个人的这个 machine learning 的这个基础也就,也就非常非常浅显,所以这个我没法面试就是,不管你是,比如说你说一个简单的什么,Ecident Trick,然后,然后这个我还是Logistic Regress,SVM可能还知道一点,但Deep Learning的话,很细的话,我肯定是一点都不懂的。这个还是跟你面试的职位相关,如果面试是, [70:00] 或 machine learning engineer 的话你可能从系统搭建更多 如果你面试DataSenters的或者是就是直接train这个model,优化这个model的话那你可能就是有不同的面试角度了这个跟面试角度也相关但我觉得这块说的非常好,我觉得确实如果是面Machine Learning Model的话,可能要简单提一下嗯几部分就是怎么样选这个 feature怎么样选 feature 然后怎么 offline 和 online 部分 然后可能用什么model更契合这个问题啊这些可能还刷减问题一下我自己的病人谢谢大家大家还有什么额外的补充吗?我觉得补充的非常好好那今天就到这里为止吧非常谢谢Jonas嗯是福祉有个人追手吗?我感觉要按著故障好的,谢谢。非常感谢Jonas,非常感谢大家。拜拜 谢谢你 拜拜