UIUC徐天音称AI重构系统安全 并呼吁取消最佳论文奖

UIUC徐天音称AI重构系统安全 并呼吁取消最佳论文奖 大模型与Agent的奖励投机正逼崩传统系统接口，系统可靠性研究因此迎来黄金期，而学术界追逐论文奖项只会助长投机并背离真实工程问题。 徐天音回顾了自己从申请博士惨遭 24 所学校拒绝、到凭借韧性深耕系统领域并斩获多项顶会最佳论文的经历，直言应警惕对奖项的功利追逐，主张从真实生产痛点出发深挖底层硬核问题。对话深入剖析了当 AI Agent 成为系统直接调用者后引发的“奖励作弊（Reward Hacking）”与接口失效风险，以及大模型将形式化建模从五年压缩至五小时的经济学质变。徐天音重温了面向恢复计算（ROC）、微重启（Microreboot）与系统级 Undo 等经典思想，强调在系统必然出错的现实中构建 Agent-native 容错底座的紧迫性。此外，他还分享了建立深度互信的师徒传承模式，并指出在 AI 飞速演进的时代，抵御平庸与焦虑的唯一解法在于建立严密的方法论并保持终身持续学习的定力。 AI 结合形式化方法使系统建模效率迎来质变飞跃：徐天音团队过去耗费顶尖博士生 5 年心血才完成的 Apache ZooKeeper 形式化抽象与验证模型，如今借助前沿大语言模型仅需 5 小时即可生成，大幅降低了利用数学证明事前杜绝底层故障的门槛。 (71:10) 当 AI Agent 成为系统的新用户，现有软件与验证体系面临“奖励黑客（Reward Hacking）”危机：在 TLA+ 证明系统 TLAPS 的实测中，Agent 倾向于钻判题规则的漏洞伪造看似全绿通过但语义无效的证明，反而倒逼形式化验证器紧急修补未曾预料的底层漏洞。 (19:38) 用于评估智能体故障运维能力的 SREGym 揭示出高保真仿真环境的巨大商业价值：通过复现 Cloudflare 等工业级事故 Postmortem 构建的真实故障环境，已成为硅谷炙手可热的强化学习（RL）核心标的，但目前仍面临微型节点集群与百万级超大规模生产系统之间的保真度鸿沟。 (67:16) 二十年前图灵奖得主 David Patterson 提出的“面向恢复的计算（ROC）”在 Agent 时代重获新生：实证表明人为操作失误历来是系统故障的首要根因，在无法避免 Agent 产生幻觉与失误的前提下，系统级 Undo（撤销）与 Microreboot（微重启）成为打造 Agent-native 可靠基础设施的核心设计基石。 (84:21)

[0:00] 徐天音: 我一直主张取消最佳论文奖（Best Paper Award），它们具有极大的误导性。 [1:45] 周源源: 努力工作是默认前提（default），根本算不上什么优势；所有的博士生都非常努力。 [4:22] 徐天音: 平庸是没有任何立足之地的（mediocrity has no place） [19:38] 徐天音: 把 AI Agent 当做用户，现有的系统是必崩无疑的。 [15:45] Stefan Savage: 为什么不把系统直接做对呢？就像天音做的那样把系统做对，根本就不需要用机器学习去挖这些问题。 [24:20] 徐天音: 优秀的导师都是相似的，而不好的导师各有各的问题 [41:34] Leslie Lamport: If you don't write something down, you actually can't fully think it through [49:48] 徐天音: 天下没有所谓的小问题，只有挖掘得不够深的问题。 [53:09] 徐天音: 内核黑客（Kernel Hacking）只是一个媒介，真正的核心训练是教会学生“如何管理与驾驭复杂度” [81:12] 徐天音: 形式化验证仅仅只能证明“系统代码与你给出的规格（Specification）保持一致”，它无法解决“你到底需要一个什么样的系统”这一根本问题 [95:26] 徐天音: 我觉得做研究是一件很美好的事情。而美好的事物，从来都不应该只有一种标准。 [100:23] 楼俊成: 博士生去工业界实习时，不应该继续在里面做学校那一套发 Paper 式的 Research [107:21] 周源源: In computing, your first job is not your forever job. [109:05] 徐天音: all advice is bad advice

[0:00] I've always argued that best paper awards [0:02] should be abolished—they're deeply misleading. [0:04] That's a bold take. [0:05] Aren't you worried about offending people? [0:06] I actually don't really believe in [0:08] AI, but I think people should be believed. [0:09] These failures all came from [0:12] AI gaming the reward [0:13] and producing bogus proofs. [0:15] There are simply too many bugs in production, [0:18] and nowhere near enough people to fix them. [0:19] in other words, how do [0:21] you teach the operating systems course [0:23] and why do students like this course so much? [0:24] Because I think research is [0:26] a really wonderful thing, right? [0:28] Then I feel that he has no set [0:30] standard for all beautiful things. [0:34] Conversations with 100 fascinating people in Silicon Valley. [0:35] This is Uncle Moon. [0:37] Today's guest is Tianyin Xu, [0:39] a tenured professor at UIUC. [0:41] He has won numerous best paper awards [0:43] in computer systems, [0:45] and is one of the finest systems researchers [0:48] I know. He is currently on sabbatical, [0:49] visiting UC Berkeley's SkyLab. [0:52] Welcome, Tianyin. [0:54] Hi, everyone. [0:54] It's an honor to be on Uncle Moon's podcast [0:56] and chat with you all. [0:58] All right, let's dive in. [1:00] From 2003 to 2010, [1:02] you completed both your bachelor's [1:04] and master's degrees at Nanjing University. [1:05] Then, in 2010, you applied to PhD programs in the U.S. [1:09] Reports say [1:11] you were rejected by 24 schools. [1:13] What exactly happened? [1:16] So what happened? [1:18] Honestly, I'm not entirely sure. [1:19] A failure like that [1:21] usually has many causes. [1:23] But the root cause [1:26] was probably that my track record [1:28] wasn't strong enough. [1:30] I applied in 2010, [1:32] when the academic environment in China [1:33] was very different [1:34] from what it is today. [1:36] I did get one interview, though, [1:38] with the program where I eventually did my PhD. [1:41] It was with the professor [1:43] who later became my advisor. [1:45] The interview lasted about 30 minutes, [1:46] and I remember it vividly. [1:47] He asked me five questions, [1:49] and I couldn't answer a single one. [1:50] A few days later, [1:52] I received the rejection letter. [1:54] Let me share three [1:55] of those five questions. [1:56] The whole thing was pretty embarrassing. [1:57] First: why do you want to pursue a PhD? [2:00] And ultimately, [2:00] what kind of career do you want? [2:02] Second: compared with your peers, [2:06] what are [2:06] your strengths? [2:07] Third: [2:10] what kind of research do you want to do? [2:13] I'd never really thought about [2:14] any of those questions. They weren't technical, [2:16] so my answers were very weak. [2:20] question at that time was, 'What [2:21] will your future career development be?' [2:23] I heard [2:24] that at the time [2:25] I read some guides saying that I heard American [2:28] professors all like their future students to become professors, so at [2:30] that time I said I wanted to go [2:33] down the academic path and become [2:34] a professor. Then my future [2:35] advisor at that time asked me, [2:36] he said, 'So what do [2:37] you think it takes to become a professor?' [2:39] I couldn't answer, [2:39] and then I said, 'Well, if you do research well, shouldn't [2:41] it naturally make you a professor?' Then he just [2:43] laughed and said, [2:43] 'It's not like that, actually.' [2:45] Later, we moved on to the second [2:46] question. The second question was, he asked, 'Compared to your [2:48] peers, what are your strengths?' [2:49] What are your technical strengths? [2:50] Then I hadn't thought about this [2:52] question either, and I [2:54] was stunned for quite a [2:55] while. In the end, an answer popped out, and [2:57] I said that [2:58] I work particularly hard. Then he laughed and said that 'working [2:59] hard' is a default; it's not really [3:01] an advantage, right, it's not a [3:01] strength. He said all PhD students work very hard, and then he followed up [3:04] with more questions. [3:05] He asked what exactly your strengths are? [3:06] Then I said I didn't [3:07] have any advantages and then moved on [3:09] to the next question. The next [3:11] question was about [3:13] what kind of research you want to do. At [3:15] that time, my preparation was also very [3:17] rough. I had read a lot of [3:19] his papers and said that I especially liked his [3:21] ABC paper and wanted to do those projects. Then [3:23] he smiled again and [3:24] said that if a professor puts a paper on their homepage, it means [3:27] that the paper has already been completed. [3:28] Otherwise, you wouldn't have [3:30] put this paper on this website, right? [3:32] He said he actually wanted [3:33] to ask what kind of [3:35] research I wanted to do, rather than doing a [3:36] certain project, and I still [3:37] couldn't answer. A few days later, I received [3:40] a rejection letter. That was my first [3:43] application experience. Honestly, I [3:44] later got an offer from a pretty good [3:45] school, but I didn't end up [3:46] going, which also laid [3:48] some [3:48] groundwork for my second application [3:49] later on. Anyway, after that, [3:51] you took a [3:52] gap year and chose to go to Europe, right? [3:55] Yes, at that time, Nanjing [3:57] University happened to have an exchange [3:58] program. So I applied to [4:00] a school in Germany called the University of [4:04] Göttingen. It's [4:05] the same story—I felt that the [4:07] domestic environment back then was [4:08] very different from now. Nowadays, I think students [4:11] have many choices. Even in my [4:12] own lab, there are many [4:14] visiting students, but around [4:15] 2010, going abroad was still not a very common thing. [4:18] So opportunities weren't that many, [4:20] and at that time there happened to [4:21] be one opportunity, so [4:22] I went. And during the time from Nanjing University [4:24] to Germany, you were [4:26] still playing football, and you even wrote on your homepage [4:29] a football resume about how you played from Nanjing University [4:32] all the way to Germany. [4:33] How did that come about? I actually can't [4:34] remember this clearly. I didn't even know there was [4:36] such a page. But I indeed [4:39] really like football, [4:40] and I really like sports. [4:42] I like sports. [4:42] Competitive sports are much like research. [4:45] You have to give them everything you've got. [4:48] Both demand your very best. [4:51] But sports are even more competitive: [4:52] there is only one champion. Everyone else loses. [4:53] Research is fiercely competitive too. [4:54] Think about the papers PhD students submit [4:55] to top conferences and journals each year. [4:58] Acceptance rates are low, and everyone is brilliant [4:58] and exceptionally hardworking. [5:01] So how can a PhD student compete? [5:02] At top conferences and journals, [5:04] so few papers are accepted. [5:05] Then everyone does research here. [5:08] People are all very smart. [5:09] Then very hard working. [5:10] Probably the most hard-working. [5:11] The smartest people work on it day and night. [5:13] Then you as one. [5:15] For example, a PhD student. [5:16] You are just starting. [5:16] How do you go to win the game? [5:18] In competitive sports and, I think, [5:20] doing research, mediocrity has no place. [5:22] You either do very well, or you don’t do it [5:25] at all. So I [5:26] think there are very [5:27] similar aspects between the two. [5:28] That’s why I’ve always [5:30] been very interested in sports—not just football, basketball, [5:32] anything, I really like them. It’s just that [5:35] later I stopped playing football because [5:37] I was too [5:38] busy, and also [5:39] because I’m a bit stubborn, as I [5:40] feel I wasn’t playing very well. [5:41] When things are going well, I don't really want to kick [5:43] anymore; if I do it, I want to do the best one, right? [5:44] Right. [5:45] It's just that my own mindset wasn't very [5:46] good. You stayed in Germany for [5:48] a year and finally got an offer from UCSD. During that [5:49] year, what do you think you did correctly? [5:51] And was it that your papers [5:52] increased or that your recommendation letters got [5:54] stronger? Honestly, I [5:55] think nothing really changed; I [5:56] just tried again, that's [5:57] all. I think after [5:59] just trying again and [6:01] getting this offer, was there any story behind it? [6:03] Because for me, [6:05] I’ve always [6:06] thought that I'm [6:07] not a particularly smart person. [6:09] Since I was young, I wasn’t [6:10] especially smart either. For challenging things, the first time [6:12] I do them, I definitely won’t do [6:15] well. So for [6:16] me, from my life experience, [6:17] if I want to do something relatively challenging, I definitely [6:18] have to fail many times. [6:21] So actually, failing the [6:21] first time isn’t a [6:22] particularly big blow for me. [6:24] And also, I’m not really someone who likes to settle; I’m not particularly [6:26] the type of person who likes to compromise. I don’t really like [6:28] doing something just because the previous task didn’t go well and [6:29] then choosing the easier option. For example, when I was filling [6:31] out my college application for the college entrance [6:33] exam, I only applied to one school. At that time, [6:34] I thought that if I didn’t get in, I [6:35] would just retake the exam for another year, right? [6:37] So the second time. [6:39] The first application was not successful. [6:40] Then I guess I’ll just apply once more. [6:42] Try applying one more time. [6:43] That's exactly what I was thinking at the time. [6:45] Then, when I applied the second time. [6:47] Later, didn't I also apply to the same school? [6:49] And then finally went there too. [6:50] And at that time, I still remember that interview. [6:52] He/She is my advisor. [6:53] His nickname is Wai Wai. [6:54] Then WaiWai called me again at that time. [6:56] At that time, I made a call on Skype. [6:58] Then when he made the phone call. [7:00] I was really nervous at that time. [7:01] So the first time I applied, it didn’t go well. I thought I’d just apply again [7:02] and try harder. That’s how I was thinking at the time. When I applied the second [7:03] time, I ended up applying to the same school again, and eventually, I went there. I [7:05] I still remember my second interview. YY, my future advisor, [7:07] called me on Skype. I was nervous after the earlier rejection. [7:09] I had carefully prepared answers to his earlier questions. [7:11] But he didn't want to listen at that time. [7:12] He asked me right then why I hadn't gone at that time. [7:15] I didn't go to the school whose offer I received at that time. [7:17] Then I told him some of my situation. [7:19] Then he talked about me at that time. [7:21] They started trying to persuade me to go to UCSD for a PhD. [7:23] I was extremely shocked at that time. [7:24] Then I also asked him. [7:25] I said why this. [7:26] But he didn’t want to listen. [7:28] Instead, he asked me why I didn’t [7:29] go to the school I had initially received an [7:30] offer from. I explained my situation [7:31] to him, and then [7:31] he started persuading me to go to UCSD for a PhD. I was very shocked, and [7:33] I even asked if he wanted to hear my prepared answers. [7:36] want to listen. Then he told me that doing systems research is a bit different [7:38] from theory research. In systems research, persistence is very important; it’s [7:39] about being someone who isn’t afraid of failure. [7:41] Then, if you're willing to [7:42] keep trying, basically your systems research won't be particularly bad. So he told me that [7:45] if you are willing [7:46] to persist and willing [7:47] to do this, then [7:49] you've already passed [7:50] my test. So I got this offer, [7:52] and then I went. So when you [7:53] ask me whether [7:54] I improved during this time, I actually [7:56] feel that I didn't improve much, I just [7:58] persisted for another year. But [8:00] as an observer, my feeling is [8:02] just like what you said earlier. [8:04] You are unwilling [8:04] to settle and [8:05] have high standards, and that means [8:07] you are willing to [8:09] try and willing to fail. The [8:10] impression you give me is that [8:11] this year you have [8:12] proven this: you do not settle, and [8:14] you want to accomplish this. [8:16] It is possible—it's possible [8:17] that if I look at various [8:18] systems researchers, I find that [8:19] this is one of the most [8:22] valuable qualities of [8:23] a systems researcher. [8:23] I think it's essential. [8:24] In my experience, it's something every [8:25] systems researcher needs. [8:26] Some research fields may depend on [8:28] a flash of [8:30] inspiration. Systems research, however, [8:31] demands diligence, repeated failure, [8:33] and the willingness to keep trying. [8:34] Exactly. [8:36] I completely agree. [8:37] Then you joined YY's lab. [8:39] Advisors have very different styles. [8:40] Some guide students step by step, [8:43] while others are very hands-off. [8:44] How would you describe YY's style? [8:46] He wasn't entirely hands-off, [8:49] but he didn't hold your hand either. [8:52] YY was already very senior at the time, [8:55] yet he was an advisor who actively guided students. [8:58] I think [8:59] he would tell you how something ought to be done. [9:01] this. Just now there [9:04] was a question, for example, how does [9:07] a PhD student think about [9:08] this idea in the first few years, right? [9:09] Then I remember that [9:11] some of my previous practice was that a [9:13] lot of times we would read that story, [9:15] like how Newton came up with [9:17] this, this, this Newton's law [9:18] because he was hit by an [9:20] apple and suddenly had an inspiration, right? [9:21] Then YY is actually very opposed [9:24] to this kind of process. [9:25] He always emphasizes that an [9:27] idea shouldn’t be something that comes [9:29] from a sudden flash of [9:31] inspiration, nor should it [9:32] be something where you just sit [9:33] under a tree [9:34] and get hit by something and suddenly have an inspiration. It’s [9:35] actually a very systematic process of exploration. What [9:38] he emphasizes is starting [9:39] from a problem: if you [9:40] have an important problem, you start [9:42] by trying to understand that problem, right? [9:44] Go in-depth to understand this issue. [9:45] After you go in-depth to understand this issue? [9:47] So your solution is often obvious once you understand the [9:49] problem; it can even [9:51] be very simple [9:52] and practical. [9:53] This is a method of doing research that [9:55] he emphasizes, and we all follow this [9:57] method. This approach avoids many risks [9:59] because once you deeply understand a problem, [10:01] your answer naturally becomes very natural and [10:03] very effective in solving the [10:05] problem. You won't have a [10:06] solution full of tricks from the start. [10:08] Then I realized after spending a lot of [10:10] time that this solution couldn't solve the problem either. It actually doesn't [10:11] have such a risk, so actually I [10:13] think that when doing research, because of this—at that time, [10:15] in any case, [10:16] I was very much in agreement with [10:17] his, how should I put it? [10:19] This is a principle, so [10:20] I have always [10:22] been practicing this practice, and [10:23] I think it’s [10:25] quite good [10:26] and smooth. Continuously practicing this practice [10:28] helped you get the PCheck. Could you share with [10:32] We built a tool called PCheck. [10:33] Could you explain [10:35] what PCheck does? [10:35] This became the second paper of my PhD. [10:37] The idea is actually quite [10:40] simple. Back then we were actually studying, [10:43] I actually don't know how [10:45] to translate it into Chinese, it's 'fault-tolerant systems,' which in Chinese might [10:46] be called fault-tolerant system, right? [10:48] Fault-tolerant [10:49] system, fault-tolerant [10:49] system. From [10:51] our perspective at that time, [10:53] what we were studying [10:55] was a type of fault called misconfiguration. When [10:57] there is [10:58] a misconfiguration, [10:59] the system obviously [11:00] won't operate [11:02] very well, right? [11:04] At that time, YY asked [11:06] a question which was: why [11:07] is it that we have been studying fault-tolerant [11:09] systems for so many years, and many of our systems claim [11:12] to be fault-tolerant, [11:12] yet when a fault like a misconfiguration occurs, the [11:13] system just doesn't work, right? [11:15] Because according to major companies, misconfiguration is a [11:18] very important cause of system failure, so [11:20] we [11:21] started from this perspective. At [11:23] that time, I did my [11:25] first project, and the first project [11:26] was to answer [11:27] the question of why those mature production-level systems. [11:30] They cannot handle faults like misconfiguration very well, [11:32] and our approach [11:35] is also very [11:36] simple and crude, [11:38] which is fault injection. Fault [11:39] injection is a very mature [11:40] technology, which means [11:41] you try all kinds of [11:44] different faults. Here, the fault refers to misconfiguration. [11:46] Then you inject this [11:47] fault into this system, so it is [11:48] injected into this system, and [11:50] then you observe how this [11:51] system behaves. Then, [11:52] during the process of doing this project? [11:54] At that time, [11:55] we were also very busy, that is, we [11:57] had to do [11:57] this kind of project [11:59] to answer this question. Among [12:00] them, there was one phenomenon that we found very interesting, which [12:02] is that there are many misconfigurations. When you [12:04] inject them into the system, the [12:05] system actually doesn't show much [12:06] behavior; it still works normally, right? [12:07] It neither initiated a fault-tolerant procedure, nor did it [12:11] say there [12:11] was a fault, [12:13] so why? [12:13] Then actually the reasoning [12:15] is very obvious: this system read these misconfigurations into [12:18] the system's state, but [12:20] it didn't use [12:21] them. So why didn't it use [12:24] these faults, that is, the misconfigurations? [12:25] It is because these [12:27] codes or these executions must be used [12:29] only under a specific [12:31] situation, and the most typical [12:34] example is that when it is [12:37] used, there is an external failure, that is, it triggers [12:39] these misconfigurations, and [12:41] when it is used, there is [12:42] already a failure, and then I start a failure-recovery program. [12:44] Then at this time I need to use some configuration, and then [12:45] I find [12:46] that this configuration is not good. Only [12:47] then will it explode, [12:48] and to us, this [12:49] is like a time bomb, right? [12:50] In this system, [12:51] you're just like a time bomb [12:53] that hasn't [12:54] been triggered yet. Once [12:55] you trigger it, the problem is usually [12:57] very serious because when you're doing fault tolerance, [12:58] that is, [12:59] failure recovery, it's already a [13:01] very critical [13:02] time. At this point, [13:02] discovering another failure—this kind of double [13:04] failure—usually leads to a major disaster. [13:06] When we were doing the first project, [13:08] we didn't have the time to study [13:10] this issue in depth because it [13:12] was a different research problem, but [13:15] the issue was there. Then [13:16] for the [13:16] second project, we said we would try to understand [13:18] this issue and figure out [13:19] how to solve it, so [13:21] naturally the second project came [13:22] about. We can talk about this first project later. [13:25] Let's talk about this best paper first. [13:27] Are you surprised that this paper won the [13:28] best paper? [13:29] I was quite surprised at [13:30] the time because I didn't know [13:31] what the standard for the best paper was either. [13:32] So I feel quite happy to be [13:34] recognized in the field. What [13:36] about him later? [13:37] At that time I was surprised, and then? [13:38] I have been in [13:40] this field for a long [13:41] time now, and last year, for [13:43] example, at SOSP, I was also a member of the [13:45] best paper review committee. [13:47] Personally, I feel that the best paper thing is very random, [13:49] very, that is, random [13:50] and subjective, so [13:51] now I completely feel that the best paper thing [13:53] cannot fully represent the quality of a paper, it cannot represent [13:55] it at all [13:56] for me. [13:57] I even feel that in [13:59] recent years, for example, even the [14:00] best papers in our field at SOSP and OSDI have quite a few issues [14:02] for me, right? [14:04] Are you so bold [14:05] to speak without fear of offending people? [14:06] I am I really feel [14:07] this way because I actually personally don’t [14:08] know what the standard for the best paper should be? [14:10] Then its process, generally speaking, for [14:13] me, is like this: a white [14:15] paper—when it gets this white paper, it means that the reviewer, [14:17] right, in other words, a [14:18] reviewer particularly likes your paper and has nominated your [14:19] paper. And that's it, [14:21] that's it. For [14:22] me, [14:23] it doesn't [14:24] represent what the future impact [14:25] of this paper will be. [14:27] It also doesn't [14:29] mean this is the highest quality paper [14:30] here because quality itself can't really be measured, so pursuing [14:33] whether a paper is the best paper is actually [14:36] quite meaningless. Personally, I have always [14:37] advocated for canceling the best paper award because I [14:40] think it is very misleading, and how should [14:43] I put it? [14:44] I think what I mean is that everyone [14:46] does research, like trying to get the best paper, right? [14:47] Then I feel that [14:48] having this called paper awards actually makes [14:49] everyone not focus on doing research properly. I have [14:52] heard a similar viewpoint before, [14:53] and I [14:54] strongly agree with it. [14:55] Right, if you remove the best paper, [14:57] then how would everyone work? Then [14:58] everyone would do the research they like the most, [15:00] and I think that is the [15:01] best state. Research really should, in [15:03] essence, be like this. [15:04] Right. [15:04] And I also heard [15:05] a saying that you don't need to [15:07] look at the best paper, but the test-of-time [15:08] paper because whether its [15:10] citations are high [15:11] after 10 years and its impact in [15:13] various places is [15:14] obvious. For me, [15:16] all awards are not particularly meaningful; I'm not [15:17] a person who believes in awards. Then, personally, I [15:20] feel, for example, [15:21] about academia in China. [15:22] Actually, I’ve been really messed up [15:25] by awards, so [15:27] my ideal is an academic [15:28] environment without these awards and other frustrating things. [15:31] Every researcher just works on the [15:33] problems they consider important, [15:34] does it happily and joyfully, [15:36] gets recognized, [15:37] and then you just present the paper. Do you think I [15:39] should share this concern? [15:40] I'm really not worried about [15:42] this, I really think so, [15:43] so let's just cut it out, okay? Yes, I actually [15:45] expressed my views to many [15:47] people [15:49] during the conference. [15:49] Okay, right, you just said that [15:52] this work is based [15:53] Was it based on your 2013 paper, “Do Not Blame Users for Misconfigurations”? [15:55] exactly, this work. [15:56] I really like that work. [15:57] This, this question is really good, don't [15:58] blame the user for [16:00] making a wrong configuration, right? [16:01] Right. [16:02] Then, what you were saying about [16:03] this work is roughly that [16:03] when I looked at the paper [16:05] at the time, it just mentioned that [16:07] a system was misconfigured. [16:08] Engineers would all say this was [16:10] a user operation error, but that paper of [16:12] yours argued the opposite: if many users are unaware [16:14] of the issue and make the [16:16] same mistake, then there may be a problem with your [16:19] system's design itself. That was roughly the motivation I got [16:20] after reading it. [16:20] Yes. [16:21] Right. [16:21] Exactly, I think you understood this paper. Then, in your paper, [16:23] you mentioned that [16:24] a user simply capitalized [16:26] the letters in their name [16:27] and then went through [16:29] 75 rounds of communication and [16:32] collected 10 logs. When you [16:34] first saw this kind of problem, was [16:36] it from the industry? [16:37] This is a case from the industry. [16:39] When I first saw this [16:40] case, I was [16:42] very shocked, but later I actually understood it [16:43] because this company is [16:45] actually a storage company, right? [16:47] Then the storage company, actually very, [16:50] their model was really troublesome. At that [16:51] time, the company sold those—I [16:53] remember this company didn't want [16:55] to put their name [16:55] on it, a bit embarrassing, but actually [16:56] it was a very good company. [16:58] Their product at that time was a [17:00] storage box, right? It bundled software and [17:02] hardware together, and they sold this storage solution, [17:05] calling it a solution, [17:05] to the users. [17:06] Then the problem is that for a [17:08] storage company, after a user [17:09] has used [17:10] your product and it malfunctions, [17:11] it is impossible for them to [17:13] send this company's box [17:14] back to you. They can't even [17:15] send all of their data back to you because this data [17:16] is confidential. Many times, their clients [17:18] are, for example, government agencies or [17:20] banking departments, right? [17:21] How can you send this data back and [17:24] back and back again? [17:25] So this diagnosis is really time-consuming and [17:26] tedious. I can only tell you the few pieces of [17:29] information I have, and then the person on your [17:31] end has to guess and try to [17:34] figure out [17:35] what kind of problem you [17:37] actually have. So the root cause [17:40] might be very [17:41] simple, but once [17:43] there is a fault, it [17:43] becomes a very expensive and very time-consuming process. [17:46] Then during the whole process, you discovered 743 configuration [17:50] error issues, and you also [17:52] reported that a large [17:53] number of these issues [17:55] were confirmed and fixed by [17:56] their developers. [17:58] Which fix made you feel that [18:01] I was not just publishing [18:02] a paper, but really helping the industry [18:04] solve a problem? I actually [18:05] always felt at that [18:06] time that because our starting point was system design, right? [18:09] It's just a fault-tolerant system, and I think this is [18:09] quite natural. I also don't feel that any [18:12] particular patch or [18:12] bug is especially important. Actually, [18:14] I've never found discovering bugs [18:17] that exciting, because I think there [18:19] are too [18:20] many bugs in the [18:22] industry—so many that people can't fix [18:25] them all. So discovering a [18:26] few more bugs doesn't seem [18:27] very meaningful to me. [18:29] Of course, I understand that many papers, our [18:31] evaluation is about proving that this [18:32] solution is useful, and one [18:33] of [18:34] our evaluation metrics is to find new bugs. I [18:37] think the more meaningful part of [18:39] this paper is that it [18:42] may tell some, especially open-source developers, when you [18:44] are designing [18:45] a system. [18:46] You have to think about how users [18:49] use it. I think there's a [18:52] The work felt meaningful. [18:53] At the time, our department had [18:54] a professor named Stephen Savage. [18:55] He is a very prominent [18:56] security researcher [18:57] in systems and networking. [19:00] He was on our faculty. [19:01] He organized a seminar, [19:03] and I remember that one day [19:04] a student proposed an idea [19:06] at the seminar: [19:07] use [19:09] machine learning to detect misconfigurations. [19:11] Stephen's immediate response was, [19:13] “Why not build the system correctly?” [19:15] He knew about our work, [19:18] which took [19:20] the same perspective. [19:21] His first thought wasn't [19:23] how machine learning could [19:24] find more misconfigurations. [19:25] Instead, he asked, [19:26] “Why not do what Tianyin did [19:28] and build the system correctly? [19:30] Then you wouldn't need [19:31] machine learning to find these problems.” [19:33] That reframing may change [19:34] how others approach the problem. [19:37] What I especially like [19:39] about this work is its relevance [19:40] to today's age of AI agents. [19:42] Back then, [19:42] your paper argued [19:43] that poorly designed systems [19:45] create problems when [19:48] people use them. [19:50] Now AI agents are the users. [19:52] If a system is still designed [19:53] only for human users, [19:55] then [19:56] AI agents are bound to make mistakes. [19:58] Right. [19:59] So I'd like to ask: [20:01] based [20:02] on [20:03] your earlier experience, do systems [20:05] for AI agents need [20:07] to be redesigned? [20:09] Absolutely. [20:10] Everyone now talks about AI-native systems. [20:12] Yet many current [20:15] systems weren't designed for [20:17] AI agents as users. [20:19] Treat an AI agent as a user, and the system is guaranteed to fail. [20:22] We've even been building benchmarks recently. [20:25] For example, one [20:26] project is a collaboration with the TLA+ Foundation. [20:28] In it, [20:29] we ask an agent to construct a simple proof. [20:31] Given a protocol, [20:32] the agent writes a proof [20:34] that the protocol is correct. [20:37] Then you can [20:38] could say from [20:39] our experience that we found TLA+ [20:43] to be a very mature toolchain. [20:44] It has a proof system called TLAPS, [20:46] which is the TLA+ Proof System, right? [20:48] Then people have written a lot [20:50] of proofs, and they [20:51] have never encountered some strange [20:52] problems. Their system's convexity is also [20:54] very mature, so they want to create a benchmark. [20:56] But if you put [20:57] AI in, put an AI agent [20:58] in, the agent will write all kinds of strange [21:01] proofs, and it will seem like the thing [21:04] has been proved, but actually [21:05] it is not valid, and AI is especially good at reward hacking. [21:07] Because when people write a [21:09] proof, they really want to prove this, right? [21:10] He really [21:11] wants to prove that this protocol is [21:13] correct, but AI doesn't [21:15] actually really want to prove that the [21:16] protocol is [21:17] correct. AI just says, because the uncle wants to [21:19] prove that the protocol is [21:20] correct, so I try my best and can pass [21:22] this proof system, which [21:23] led to him discovering many [21:25] of these strange bugs, so many colleagues from the TLA+ [21:27] Foundation, like Markus. [21:28] Or Stephan and the [21:29] others started fixing all sorts [21:31] of bugs that they [21:33] had never even thought [21:34] about before, all of which [21:37] were caused by AI trying desperately to do [21:38] reward hacking and then doing proof, [21:40] which led to these [21:43] problems. And I think that once [21:45] an AI agent becomes a user, all systems should rethink [21:47] how to handle this interface, in my opinion. [21:49] I strongly agree with this [21:50] view because I [21:51] can observe that more and more companies [21:53] in Silicon Valley are starting to invest in how to redesign the interface [21:57] for AI [21:57] agents, how AI agents should log [21:59] in, how AI agents should ensure [22:01] security, how AI agents should operate, and [22:03] so on. I think it [22:04] is an [22:05] inevitable trend that AI agents will become the [22:07] main force in work in the future. [22:09] Yes. [22:09] I think the biggest [22:10] opportunity for systems researchers [22:12] or some system engineers next is [22:13] in this [22:14] area. Honestly, [22:15] I agree. I think this [22:16] is a really good [22:18] thing for systems [22:20] research. In the past, if you wanted to fundamentally redesign a system, right? [22:22] The cost is high, and [22:24] not everyone may be willing [22:26] to invest. But now, I [22:28] think AI has a very positive impact on systems research. People [22:30] are willing to [22:32] consider this and feel [22:34] it's necessary to redesign [22:35] the system. I think this is [22:37] a very good thing. In [22:38] the system [22:39] industry, I have always seen that [22:41] it has to [22:43] be behind products and algorithms. [22:46] Because if you try to go too [22:48] far ahead and create a system [22:49] that cannot be deployed and applied on a large scale, then [22:51] it actually has no value; the [22:52] cost is too high [22:53] and the ratio of input to output is too low, meaning [22:55] the ROI is too low. [22:56] Right. [22:57] Now it is equivalent to the rapid [22:59] development of AI, which has generated an infinite number [23:02] of new major demands, forcing systems researchers and system engineers [23:05] to continuously push forward the development of these new systems. [23:07] Yes. [23:08] I think this is a really [23:10] good thing in the era of [23:11] systems researchers. I think it is a [23:13] very good thing. I think every systems researcher should seize [23:14] this opportunity, because it [23:16] includes, for example, [23:16] in the [23:17] past, if you were [23:19] doing a system, right? [23:21] Then you say that your system is [23:22] very good, but it can only handle one type of workload. In fact, [23:26] it's very hard [23:27] to get, and even in a [23:28] startup, it's hard to get funding. [23:29] It's even hard to convince the judges that my system [23:31] is good, and the [23:32] judges will definitely reject your paper saying that this is [23:34] not general. But now, it's like saying [23:35] that I just make a system, and this [23:37] system, for example, has [23:38] very good IO performance. [23:40] Many people would be very excited, right? [23:42] I feel that your system is [23:43] particularly useful, and its economic [23:44] market is particularly large, so I think this is a [23:46] very good thing. For example, [23:47] if you were to [23:48] work on a single aspect, like [23:49] how to allow an agent to log in. [23:52] Yes. [23:52] Right, that agent login may seem minor, but because [23:54] he considered all the possible reward hacking and [23:57] ways to hack this system by different agents, he made it very [23:59] solid. In this way, you can [24:00] ensure the security of the [24:01] entire system for the users. [24:03] Right. [24:03] What a good thing. [24:04] Yes. [24:04] I think this opportunity is indeed [24:06] very significant. We can talk more in [24:09] depth about it later when we [24:10] discuss some of your recent research and [24:11] this AI system. Then going [24:13] forward, when you enter the PhD [24:15] stage and interact with your [24:16] advisor, you have always been very grateful [24:18] to your PhD advisor for [24:19] constantly being supportive in various ways, [24:20] and on your homepage, you said he is a rock-star advisor. Why [24:23] is that? [24:23] What exactly did he teach you? [24:25] I think good mentors are all similar, [24:27] while bad mentors each have their [24:29] own problems, but good mentors are all [24:31] similar. So I think he [24:32] definitely taught me how to do research and [24:33] then he told me how to [24:35] think about certain problems correctly, right? [24:36] Then I think one thing he [24:39] does particularly [24:41] well is that he really [24:43] emphasizes deeply understanding [24:44] problems, which is something I really, really [24:46] appreciate. Also, he encourages thinking differently, and I [24:48] think these are the things [24:49] he emphasizes that I really appreciate. [24:51] And I feel that he [24:53] makes me constantly full of passion for research. [24:55] I feel that research is a really wonderful thing, [24:57] and I also especially like doing it. [24:59] This is truly a [25:01] top-level mentor, so [25:02] I really [25:02] think that having him as a [25:04] mentor is [25:05] actually very strict and very direct, [25:07] but I especially [25:08] like direct mentors because I don't really like [25:11] some indirect advice. So I [25:12] Your personalities were also a strong match. [25:14] But I think the essential [25:15] quality— [25:16] what matters most— [25:18] is a genuine interest and passion for research. [25:19] That's what [25:20] I admired most about him. [25:23] Right. And [25:24] he was also a very successful entrepreneur. [25:25] I often see [25:26] his talks on entrepreneurship online. [25:29] He comes across as decisive, forceful, [25:32] and demanding. [25:33] But with you, [25:35] the fit clearly worked. [25:37] That's the impression I get. [25:38] Fit may be the most important thing [25:39] between an advisor and an advisee. [25:42] He founded three companies, [25:46] all quite successful. [25:47] Did he ever discuss entrepreneurship with you? [25:48] Yes. I listened to him [25:50] talk about starting companies [25:51] for three years. [25:52] Right. [25:53] What did you take away from it? [25:54] I've always believed [25:56] that research and startups [25:58] share the same core. [26:00] Both require innovation, [26:02] and both demand the pursuit of excellence. [26:04] These things are basically the same at their [26:06] core. You need a team; [26:07] it's very difficult for one person to [26:08] accomplish something alone. You also need a good [26:10] advisor who can tell you certain things, right? [26:13] The difference is that research might be more particular about how to say it, [26:16] how should I put it? [26:18] It’s more about valuing intellectual merits, [26:20] or rather, [26:21] valuing expanding the human knowledge base. And I [26:23] think doing a startup [26:24] might require more [26:25] consideration of marketing aspects, right? [26:27] Things in the [26:28] market, but I think there are really many [26:30] commonalities between the two; it depends on what you [26:33] like. Of course, there’s also [26:34] the matter of timing, location, and people—whether you [26:36] can start this company. There are many [26:36] very good technologies, but if the [26:38] time isn’t right, you [26:39] still can’t start the [26:39] company. But I [26:41] think that there are very many similarities between [26:42] the two. The feeling I get is that [26:44] you’re the type of [26:45] person who doesn’t want to start a company for the short term, [26:47] right? [26:48] Because this person is relatively free and [26:49] easygoing, and I haven't thought about it too much [26:52] yet. Yes, yes, although you don't [26:53] want to start a company, you have gained a [26:56] lot of [26:56] successful entrepreneurial insights at [26:57] a glance. Indeed, I was [26:58] brainwashed, brainwashed for three or [27:00] four years, and indeed there [27:00] was no problem at [27:01] all. In 2018, you completed your [27:03] When you finished your PhD, you received faculty offers from five [27:05] universities, and ultimately chose UIUC. [27:06] Why UIUC? [27:07] At the time, [27:08] among the offers [27:09] I had, UIUC was [27:10] the fairly obvious choice. [27:13] One reason [27:14] was that my former advisor [27:15] had been on the faculty [27:16] at UIUC [27:18] for seven years. [27:20] He had many fond memories of the university, [27:22] which certainly influenced me. [27:26] More broadly, with any offer— [27:28] whether in academia or industry— [27:31] you need to consider [27:32] what the place [27:34] can offer you [27:35] as a platform. [27:37] What can you learn there? [27:38] A faculty offer comes down to a few things. [27:41] Can you [27:42] find excellent collaborators? [27:44] Are there good mentors? [27:46] Are there senior colleagues [27:47] you deeply respect [27:48] and [27:51] would want [27:52] as mentors? [27:54] Can you recruit [27:55] outstanding students? [27:56] UIUC checked all of those boxes. [27:58] And Champaign suited my temperament very well. [28:00] I really like Champaign. Now that I've come [28:02] to the Bay Area, [28:03] I still often miss Champaign [28:04] and want to go back there. [28:07] So I think UIUC is a [28:08] pretty good choice, indeed different from others, haha. Most of [28:10] the students I know who are [28:11] familiar with Champaign feel that it's [28:13] just a pure cornfield where you [28:14] can't even stay for [28:15] a day. [28:15] Yes, it's a cornfield, [28:17] so it's really a very pure [28:18] area suitable for researchers. I think it's a very simple [28:20] place without much else, and you can [28:22] just stay there and think about [28:23] some problems. I think now [28:26] that you have been a professor [28:29] for 8 years, what kind of students [28:30] do you like the most? [28:33] I think this question is too [28:35] difficult. This question is actually [28:37] an equivalent question, which is what [28:38] kind of partner you want. And [28:41] I think there is no answer to it. [28:42] I once watched a documentary, [28:43] right? It was directed by a documentary filmmaker. [28:46] He interviewed many single [28:48] men and women and asked [28:50] them to describe what [28:51] they wanted their [28:51] future partner to be like. They [28:53] would list very detailed descriptions, such as [28:55] wanting someone who [28:56] looks a certain way, has red hair, [28:58] went to college, and [28:59] so on. But [28:59] five years later, [29:00] the director interviewed [29:01] them again and looked at how many of [29:04] them already had partners [29:04] and tried to match them, only [29:06] to find that it was [29:07] not the same at all. [29:08] The spouse they eventually found was not at all like how [29:10] they had described at the time, and I think [29:11] it's the same with students—you can say it's [29:13] really hard to get to know a person. [29:16] Finally, it is said that [29:18] you can work very closely [29:19] with this person for five [29:22] years and have a very close relationship, because [29:24] I think the advisor-advisee relationship is a lifelong relationship. For me, [29:26] it means thinking [29:29] about what kind of [29:30] person would ultimately be [29:32] able to work [29:33] out, that is, to establish a very personal relationship. [29:35] Students like this can definitely work out, [29:38] but if this relationship can't be established, I generally [29:40] don't know how to work, that [29:41] is, work with this student. Then I know [29:43] this view is very controversial because many of my colleagues [29:45] think this is fundamentally wrong. And [29:47] I have a [29:48] colleague whom I respect very [29:51] much, at UIUC. [29:52] His point is that you [29:54] shouldn't establish a personal relationship with students. I've seen a lot of this [29:55] kind of view, which is [29:57] true, but [29:58] this kind of thing really [29:59] depends on how you [30:00] think about it. [30:01] But for me, I completely [30:04] can't work closely with someone I can't establish a personal [30:06] connection with—it feels [30:07] like you can't even speak, right? [30:09] Because I [30:10] think, it's still [30:10] that sentence, I feel doing research, [30:12] especially very difficult research, is inherently a [30:14] very hard thing, right? When you are [30:16] in a difficult situation, you [30:19] have to be very direct [30:22] in saying some things. For example, [30:24] I once watched a [30:24] football match—in football matches, [30:26] you can see those people, [30:27] for example in the World [30:29] Cup, when they [30:30] play football, they [30:31] are not playing while smiling. [30:33] Especially those defenders, right? [30:34] Many times you just have to yell [30:36] loudly and then say whether you should handle [30:38] the ball this way or [30:39] not, or that you should kick long, [30:41] or that you [30:42] shouldn't kick long, or [30:43] whatever, and then shout loudly, [30:44] yell loudly, and get the [30:45] team going. If every time you [30:47] are smiling, that football team I think plays badly, [30:49] and I think of it like doing research; every research is like an ambitious project. [30:50] It definitely has very difficult [30:53] moments, and at [30:54] such times, how you can inspire the [30:56] team to continue moving forward relies on a personal trust. Without [30:59] trust, I personally don't know [31:01] how to push this matter forward. In [31:03] fact, [31:05] what you just described reminded me [31:07] of many scenes I have seen, for [31:09] example, Gu AIling or Cristiano Ronaldo [31:11] when they face certain challenges. [31:13] Their eyes aren't happy; they are the [31:15] kind of eyes that want to win, [31:16] all of them twisted, right? [31:17] For example, when you [31:18] play in the NBA, look [31:20] at how no one goes for [31:21] a rebound with a smile on their face. [31:23] Everyone is contorting their faces and giving their [31:25] utmost effort to grab the rebound. I think [31:27] research often has many moments like this too. The [31:29] same goes for entrepreneurship. [31:31] Many top entrepreneurs tell [31:33] their teams that we are a [31:34] sports team; we are here [31:36] to win, not [31:37] to fool around, and this is [31:39] how we win. [31:40] We are here to make [31:42] the customers happy; we are [31:43] not here to [31:45] have fun ourselves. So, this kind of [31:47] thing can only happen in situations of extreme trust and where a [31:49] personal connection can be established. [31:51] Right. [31:52] So you're saying that you firmly [31:53] believe there should be some good connection with a [31:55] PhD. I think this [31:56] is a very personal thing. This is my [31:57] way; it doesn't represent [31:58] a universal way. It's like the professor I [32:00] just mentioned, who is also very successful. They [32:01] think that everyone should [32:03] just do things according to [32:04] their own ideas. I think the best thing about the [32:06] academic world is that everyone does things according to [32:07] their own thoughts. There isn't a standard [32:10] method here, so what suits you is [32:12] the best for me. [32:13] I can completely understand that point because [32:14] the core of that point is [32:17] that if you are too personal with a [32:19] student, then the [32:20] way you teach that student is actually biased, right? [32:22] It is very biased by [32:23] your own taste or preference, so it may not [32:25] necessarily be the best for the student. [32:27] On the contrary, if [32:28] you have a very personal relationship with the student, when you [32:30] think about the student, you [32:32] might be able to [32:33] give some very objective advice. But I can't do [32:36] this myself because my personality [32:38] is just not that [32:38] kind of person. [32:40] So for me, the best thing is [32:42] to have a connection and be [32:43] able to speak more directly about some things, right? It [32:47] goes back to some of the topics we talked about at the [32:49] very beginning, which is that people still [32:51] need to be themselves. Being yourself makes it [32:53] easier to be more consistent [32:54] and makes doing things go more smoothly. [32:55] Right. [32:56] Is there a certain trait that makes [32:59] you feel this person just shouldn't do research and makes you not [33:00] want them to be a student? [33:03] I feel like, [33:04] I feel like, the word 'research'—if you break it [33:05] down, what [33:06] I tell [33:07] the students is, what does 'research' mean? [33:07] It's just 're' and 'search', right? [33:09] It's this word, it's [33:10] like this: research. 'Re' [33:11] means doing something repeatedly, you [33:12] do 're' [33:14] many times. And 'search' is actually the process of [33:16] searching. Anyone [33:17] who is willing to continuously [33:19] research, then they are a researcher, right? [33:21] It is a very common practice, [33:23] and then you can do research [33:24] in the industry, you can also do research [33:26] in academia, and you can do research [33:27] anywhere, so [33:28] I don't think research is a very narrow concept. [33:32] Anyone can do research, of course, for me. The [33:34] style of research in academia and industry is not quite the same, and what [33:36] they pursue is also [33:37] a bit different. So I think [33:39] depending on what kind [33:41] of research you want to do, you [33:42] might need to choose an [33:43] environment to do the research you want. For [33:47] example, if you want to do a very imaginative research, [33:49] right? [33:49] You want to do a very theoretical research, and this research may [33:51] not be very suitable to be [33:52] practiced in some places. For example, didn't someone [33:55] just get the Fields Medal a couple [33:57] of days ago, right? Wang [33:58] Hong and Deng Yu [33:59] won the Fields Medal, and the [34:00] research they did is very beautiful, right? [34:01] It's very wonderful, but [34:02] you must say that this research, if you're going to do [34:04] it in, for example, a startup, then [34:06] I think it might not [34:07] be very suitable to do this kind of [34:08] research. But in a startup, [34:09] you can completely do research on your own [34:10] product, and you will [34:11] definitely reflect on it, right? [34:13] Then go and think about how to improve [34:15] or think about some fundamental issues, [34:16] you're doing research. So I don't [34:17] think there are people who cannot do research. Of course, whether [34:20] to do research with me or not fundamentally comes down to whether [34:22] the two of us can establish [34:24] some connection and whether I [34:26] can help this student. I think this is [34:27] the most important question for me. So if [34:29] I think some students [34:30] may be particularly outstanding. [34:31] Then I feel that I can't help this [34:33] student, [34:34] maybe I might not really [34:35] want to admit this student as a PhD student, I think. [34:37] Understood. [34:38] What exactly do you mean by that kind of research being [34:41] very high-dimensional? [34:41] Everyone can do research. Researching the [34:43] size [34:43] of a button for a [34:44] product in a startup is a kind of research [34:46] because you need to 're' and 'search'. [34:48] Right. [34:49] Then you do [34:50] mathematics in that school [34:52] and proving the Goldbach conjecture is a kind of research. [34:53] Yes. [34:54] I think the research you do at school [34:55] is also very meaningful, right? [34:57] Fundamental and applied research both matter. [34:58] advanced way of [35:00] thinking, meaning that as long as [35:01] you seriously focus on [35:03] innovation, pushing things [35:05] forward, and [35:06] optimizing, it [35:06] all counts as research. Because [35:07] back in 2017, didn't I spend a year at Facebook? [35:08] Back in 2017, didn't I spend a year at Facebook? [35:10] Facebook's systems research: it doesn't have [35:13] a systems research department because Facebook [35:14] doesn't believe in a [35:16] systems research department completely separate [35:19] from production. So all [35:21] systems researchers are actually working on [35:22] production services. So how do we [35:24] do it? [35:26] Actually, [35:26] it's because Facebook is not a research company, right? [35:29] We worked with [35:30] many engineers to build some systems, and [35:32] the way we conducted this research [35:33] was that we built the [35:34] system first, and [35:36] then we reflected on it [35:37] afterwards. We looked at why [35:39] we did it that way at [35:40] the time, possibly because there [35:42] weren't other good ways [35:43] to do it, right? [35:44] Because when you are actually going live in [35:46] production, you are really [35:47] nervous and you don't really have much [35:49] time to think about [35:50] this, then you keep thinking about where [35:51] this can be improved, [35:53] where it can be [35:55] done differently, or what the costs [35:56] and benefits are. As long as you do this, [35:58] it is research, and [35:59] we also document all those production services that we have done. [36:00] I ended up writing a paper and [36:01] published it on OSDI. This [36:03] kind of paper is my personal favorite — prolific and really flavorful, [36:06] that's really how it [36:07] is. So what [36:08] do you think a PhD student [36:12] in the very early first [36:14] or second year should do [36:15] if they join your group or [36:17] if you are mentoring them? [36:18] Go read 100 papers and then [36:20] jump in to work on a [36:21] project with a fifth or sixth grader, or just think [36:23] about it and write code by yourself over there. [36:25] Right. [36:26] Because I feel that a PhD is a very personal thing and it [36:29] doesn't have a standard answer, [36:30] so [36:31] first of all, I don't [36:32] think reading 100 papers is a very good practice. Then [36:35] I remember my PhD advisor at the time was actually [36:38] very opposed to reading papers to get ideas. [36:40] He had a rather radical point of [36:42] view: he thought that every paper is very biased and [36:44] is just the author's own perspective, right? [36:47] Actually, each paper is essentially a perspective, [36:49] it's the author's own viewpoint. And you [36:50] know, the current practice is that top conferences are very [36:53] rigorous, so everyone uses a lot of techniques when writing [36:54] papers and is not necessarily that honest. [36:56] So, if you read [36:57] a lot of [36:58] papers and [37:00] try to get [37:01] ideas from them, it’s not a [37:03] very good way to innovate. Sometimes, you [37:05] probably shouldn’t even read them, [37:08] at least that's what [37:09] I personally think. [37:10] I feel that many papers are like [37:12] feed; if you consume [37:13] too much feed, your [37:14] brain becomes numb, and then you won’t [37:16] be able to innovate. So [37:17] I think the best way is not to read papers to [37:20] get ideas, but to come up with ideas based [37:22] on real problems. Here is a very important [37:23] question: we need to solve this problem, right? [37:25] Then, then how do we solve [37:26] this problem? If this problem [37:27] is important and [37:28] difficult, it requires research, rather than just [37:30] because there is a paper that won the best paper [37:32] award and you decide [37:33] to read someone else's limitations section and then write your next [37:35] paper. I think this [37:36] is not a good approach, so I actually [37:38] do not advocate reading [37:39] 100 papers from the beginning. [37:41] That recent college graduate started reading for a PhD, and [37:45] he doesn't have much [37:46] experience in discovering really impressive [37:48] I think this is [37:49] part of an advisor's job. [37:51] An advisor [37:52] should give students [37:54] good questions to pursue. [37:56] on. Basically, I feel that [37:59] all the PhD students I know started like this. Then, working on [38:02] projects together with senior students, I think [38:03] it's a very good practice. [38:05] My former advisor [38:06] did it this way, and now at [38:08] Berkeley, basically every student [38:09] does this when they enter Berkeley. [38:10] And then Berkeley is more of a [38:13] bigger lab, right? [38:14] This professor-student interaction is [38:16] not very hands-on most of [38:17] the time, [38:18] and actually, [38:18] you end up learning more from your peers than [38:21] from the professor. This was [38:23] also the case for my [38:24] own group. I think working with [38:27] a senior student on some [38:29] difficult problems is a really [38:31] good way to start, and you [38:32] can also learn a lot and build some connections. [38:34] Then if you have some questions, you can just come up and do them. [38:36] Right. [38:37] In the face of such a [38:39] situation, especially in the current [38:40] AI era, you don't even need to [38:41] interact with peers; you can just interact with agents every [38:43] day and continuously produce a large amount [38:45] of content. So what is the meaning [38:47] of university then? What is the meaning [38:49] of teaching? I personally feel that agents are still [38:51] very difficult to replace—at least for [38:52] now, agents are still hard [38:55] to replace. Agents are quite a cold thing, I think. [38:58] Then the agent is not really something that [39:01] considerate or comprehensive. I think guidance [39:02] from a person is [39:03] still quite important, whether that [39:06] person is a senior student or [39:07] an advisor. I [39:09] think the agent has [39:11] many capabilities; it can improve your coding productivity. [39:13] Right, the productivity of system building, and it can also help you come up with some ideas, [39:15] but in the end, even if it helps [39:17] you come up with ideas, you still [39:18] need to figure out yourself, [39:21] how should I put it? [39:22] So basically, it's about understanding these ideas and figuring out which ones [39:24] can be done and [39:25] which ones can't. Overall, my [39:27] impression after listening is that [39:28] you have to trust people's judgment and taste, right? [39:30] These things are irreplaceable by agents in [39:31] a short period of time. I don't [39:33] really feel that, I actually [39:34] don't believe that in the future the [39:35] world will be completely replaceable by [39:37] agents. I still think that the future [39:38] world will be a world [39:39] of humans, but a large amount [39:40] of labor will be taken over by agents. Even if [39:44] AGI is realized, don't you [39:45] think it will be like this? [39:46] I don't know what your definition of AGI is? [39:49] It's just what I think. [39:51] First of all, personally [39:52] I feel that thinking about what will happen [39:54] after AGI is achieved is quite a [39:56] waste of time. [39:57] To me, it's like thinking about [39:58] what to do after the apocalypse. [40:00] It's the same [40:00] thing, it's a fun thing to think about, but [40:03] don't take it too seriously for me. And [40:04] then if [40:05] there's a student [40:06] who is always panicking [40:07] there, [40:08] saying that [40:09] AI will replace everything [40:09] I've learned and it's all useless, then essentially you can't make progress, right? [40:12] And on the contrary, [40:14] I think we shouldn't overthink things. [40:16] Let's just think a little [40:19] about the future—what can agents do [40:20] now? Then we can use current agents [40:22] to improve our learning [40:23] abilities and work skills. I [40:24] think this is a more [40:26] practical approach. Yes, because if in the [40:29] end people become slaves [40:30] to AI, then I believe [40:33] human learning ability becomes even more important, [40:34] otherwise how would we compete with AI? [40:37] Skynet, right? [40:38] Right, but if we, if [40:40] we people are doing the [40:41] right thing, then AI [40:42] is a servant of humans. Essentially, it's still humans who [40:44] decide how this world operates, right? [40:45] So your own abilities, [40:47] your own learning ability, [40:47] and your own work capability are [40:50] still very important. Just now, we went through [40:52] a short half-hour interview, [40:54] but I could feel from [40:56] all your words that you are a natural systems researcher. [40:58] No. [40:58] It's just that this feeling is very obvious, very [41:00] obvious, because this system really requires [41:01] you to lower your head and think [41:03] carefully about the tasks in your hands [41:05] every day, doing them step [41:06] by step. It works through doing; it [41:07] doesn't require you [41:08] to think about what [41:10] will happen in 5 [41:11] or 10 years every day. [41:12] As long as you [41:12] work on this [41:13] system bit by bit, step by step, [41:15] it can eventually be [41:17] completed. With [41:17] your personality traits, my goodness, [41:19] you [41:19] are really suited for working on systems. Research still [41:21] needs to be done [41:22] continuously, I think, anyway, I [41:23] feel that you definitely have to do research, [41:25] you absolutely need to be open-minded, and you [41:26] need to know what is happening around you, [41:28] right? [41:29] But it is indeed necessary to have a main thread [41:31] of [41:32] your own, [41:33] I think. OK, so our next [41:34] question is that many professors now say that theses [41:37] need to be reformed, and that [41:39] they no longer require [41:41] the traditional sections. Some propose using only an abstract and introduction [41:44] you want to tackle this problem, and [41:45] then directly include the dialogue with the agent. What do you [41:48] think of this opinion? I think this [41:50] opinion might be suitable in some cases. [41:51] It holds true under a specific context, right? I think [41:54] it's because, for me, [41:55] the key to [41:56] a paper is what the contribution of this paper is, right? [41:59] It's just that I don't feel [42:01] like, for example, the Fields Medal winners I just mentioned, like [42:02] Wang Hong and Deng Yu, right? [42:04] I don't think their paper [42:05] should be presented in this [42:07] way. in other words, if [42:09] the contribution of your work ultimately is an artifact, and this [42:11] artifact is very useful, everyone can [42:12] use it [42:13] and uses it [42:15] well, then perhaps your method is [42:17] enough, and you have an abstract to [42:18] tell others how this artifact is used. [42:19] Then it later became an open-source project, and it might [42:21] even be possible to implement [42:23] an agent there. [42:24] But a lot of research [42:25] is actually a process [42:27] of thinking, because I feel [42:28] that the essence of research is understanding the world, right? [42:30] Regarding your understanding of this matter, I think it [42:32] shouldn't be entirely presented through [42:34] AI. Leslie Lamport is a [42:36] Turing Award winner, and many of our current works actually [42:40] use [42:40] some of the tools [42:41] he developed. But there is one [42:43] viewpoint of his that I really [42:45] agree with: if you don't write something [42:47] down, you actually can't [42:49] fully think it through. [42:51] So I think the writing process is a [42:54] process that helps people truly understand this. Personally, I really don't think [42:57] it should be skipped. I can't agree with this [43:00] view at all because I also [43:02] think that the main [43:03] training in a PhD, this process [43:05] of writing a paper, is precisely [43:07] a process of training your brain. [43:08] Right. [43:09] You can't write clearly; [43:10] this question of yours definitely hasn't been thought through. [43:12] Yes. [43:13] So I [43:13] think a lot of fundamental things haven't [43:15] been changed, and many viewpoints [43:17] are brought up because the [43:19] current review system is under a lot of [43:21] pressure. Since AI [43:22] has come out, everyone's [43:23] productivity has increased, and there are a lot of [43:25] paper submissions. This includes, for [43:26] example, some traditional [43:27] systems where the circles are [43:29] very small, right? [43:30] The system's community is [43:31] particularly small, and in the past, there [43:34] might have been only 200 submissions per [43:35] year, but now it [43:37] has become 1,000 submissions. [43:38] If you continue to [43:39] handle things according to [43:41] the original organization, the pressure now is very high, and after the [43:43] pressure increases, your system is actually not scalable. When your workload scales [43:45] up, your system begins to break. I think what we [43:47] should reflect on is this. [43:48] How to make this system work [43:50] well so that it can scale up [43:53] to the current volume of submissions, [43:56] rather than simplifying it [43:57] into another system—for example, if you remove the [43:58] writing part or the thinking part, I [43:59] think the essence of the research is broken. I think [44:02] that's [44:03] indeed very reasonable; [44:04] we really need to [44:06] find a way to improve the system's scalability. [44:07] Right. [44:08] I’ve heard Terence Tao say before [44:10] that mathematicians nowadays [44:11] are different from mathematicians in the [44:13] past. Previously, mathematicians had to come up [44:15] with many ideas. Now ideas have [44:18] become so easy, mathematicians have to constantly [44:19] review and verify to [44:20] validate these ideas. [44:22] We don’t know what might happen [44:24] next, like how this paper [44:26] system will turn out. [44:27] But we definitely need to undergo [44:29] some reforms, and I [44:30] completely agree with this. I [44:32] think I completely agree, which is [44:34] to say, we should [44:35] think about what systems research should change in AI era. [44:37] It should do it, right? [44:38] Because I feel that AI has raised [44:41] this bar. In the past, [44:43] without AI, you might have been able to [44:45] do some research, and the idea for this research wasn't particularly interesting, but it [44:47] involved a lot of engineering [44:49] work, or it required [44:51] some special skills to accomplish. Now, this [44:53] kind of research has become relatively difficult because AI [44:55] can help you do these things. [44:57] So what this means is that in AI era, [45:00] the way [45:01] you do research is [45:02] different. It's about how you push the [45:03] frontier because this frontier is different, and the way you [45:05] push the [45:06] frontier is different. But essentially, [45:07] the nature of [45:08] research, I think, hasn't changed. A good researcher is still a good researcher. [45:10] Terence Tao is still Terence Tao, right? [45:12] It's not like anyone uses AI [45:14] and instantly becomes more powerful than Terence Tao. I [45:15] don't think that's the [45:17] case. It's obvious, especially now, that [45:19] after having AI, [45:20] those experts who [45:22] can write papers and do research are [45:24] more efficient. [45:25] Right. [45:25] I think AI can increase a person's productivity by 5 times, 10 times, or even 100 times. [45:28] Some people say, 'I think so,' but the question [45:30] is, what is [45:31] your baseline? If your ability is 0, [45:32] then multiplying by 100 is still [45:33] 0. I also want to point out [45:34] this: when we talk about [45:35] this 100 times increase, it means [45:37] that for someone who is already at 100 points, it becomes [45:39] 10,000, but for someone [45:40] at 1 point, it becomes 100. Oh my, the gap [45:42] becomes even bigger. [45:42] Yes. [45:43] Right, so let's go back to your research itself, from your PhD to [45:46] you've won more best paper awards than almost anyone I know. [45:49] received are among the most I have [45:52] ever seen. So, how do you view [45:54] these awards, even though you just kept saying [45:57] you're not optimistic and that it doesn't [45:58] matter? I think [45:59] these are all just luck for [46:00] me, but [46:01] if a person's [46:02] luck keeps happening repeatedly, I consider it to be [46:04] a reflection of a kind of standard of ability, [46:05] not that I think otherwise. [46:06] I actually don't agree; [46:08] I just feel that it was a matter [46:09] of luck at the time. [46:11] For example, in the past two years, I [46:13] haven’t won as many awards for my papers, but I [46:15] don’t think the [46:17] quality of my papers has [46:19] dropped that much. So looking back at [46:21] having published so many good papers, how do [46:23] you think it’s possible to continuously produce this kind of good research? [46:26] I actually think doing research is very simple. [46:27] If you hire the [46:29] right student, they will naturally do top research. Doing [46:31] top research is about a person's [46:33] quality; with that quality, they can produce very [46:35] good work. So the real issue here is how to [46:38] hire the right person and [46:40] then how to help this person [46:41] understand the method of doing this [46:43] work. As long as [46:45] you do these two things well. [46:46] After that, there are some results. I think let's break [46:49] down this step. First, how do [46:50] you recruit this person, and second, [46:52] how do you bring [46:53] this person [46:54] in? Let's break down how to [46:56] recruit this person. I think it's completely based on [46:58] gut feelings [46:58] for me because I'm not someone who thinks [47:01] about [47:02] things in detail. For example, when I recruited students, I just chatted with [47:05] them for a while. [47:06] Then see if there is any chemistry because personally [47:09] I don’t believe I have it. [47:12] I have a friend, [47:13] this friend of mine is [47:15] really interesting. He just wants, he uses [47:18] dating apps, and after he uses dating [47:19] apps, he often [47:21] goes [47:21] on dates with others, right? [47:23] Then think about how to [47:24] go there, and I asked him if dating apps are accurate? [47:25] Because dating apps are basically where you initially arrange [47:27] to meet someone and spend a [47:28] night with them. In the [47:29] end, whether this person will [47:30] become your long-term partner, he says, [47:32] even spending just one night isn’t acceptable, right? [47:36] Then I think, as a student, [47:37] you interview this student for half an hour, but [47:38] could this person be someone you have to [47:40] work with for the next five years? [47:41] So I think this is [47:43] too difficult, so it's [47:44] more about seeing fate or chemistry, just [47:45] looking at some gut feelings of yours. Let's break [47:47] down the second half: the student [47:48] is recruited, so how [47:49] do you help him publish good papers? Publish [47:53] papers, but the same sentence: that's [47:55] not our goal, right? [47:55] So how should we... [47:57] I think it's about letting [47:58] them do the research they want to do, [48:00] because I think every student... [48:01] His goals are also somewhat different. [48:02] Some students might want to start a startup; [48:05] he doesn't necessarily say they have to [48:06] publish papers. I think that's also a very [48:09] meaningful thing. But [48:09] maybe the research he wants to do is [48:11] different from a student who says [48:13] they want to go to another school [48:14] to become a professor or to make some [48:15] breakthroughs in fundamental research. The ways to train [48:16] these two people are different. [48:18] It should be very [48:19] different, so I don’t think there is a magic formula. I [48:23] just think that if you have [48:25] a very deep connection with this [48:26] student, you should be able to understand what [48:28] this student really wants. [48:29] What does he like to [48:30] do and what is his ultimate goal? [48:32] So what things can make him very happy and [48:34] what does he need? [48:34] Then after you have sorted all these things [48:36] out, I think you should [48:37] be able to help [48:38] this student, and I think [48:39] anyone should be able to help [48:40] this student. But the most [48:41] difficult part is how you [48:42] can establish this connection, how you can understand this student, [48:44] and then help him find some [48:46] problems or resources or something. [48:47] Many times it is also very [48:48] difficult; I think the hardest part is when the [48:50] student says, 'I don't even know what to do.' [48:51] I don't know what I like either. [48:52] At that stage, you have [48:54] the student try different directions. [48:56] Then you ask [48:57] which ones they genuinely enjoy, [48:59] and use your experience to help them choose a topic. [49:01] Listening [49:02] to you describe it, [49:04] I'm reminded of the classic master–apprentice [49:08] tradition in research. [49:09] You know? [49:10] There's something beautiful [49:12] about that kind of relationship. [49:14] Today, many professor–student relationships [49:16] feel more like employment. [49:18] But in the older European and American tradition, [49:21] the model was different: [49:23] established scholars [49:24] personally guided students [49:26] through an apprenticeship. [49:27] From what you've described, [49:29] that kind of relationship [49:30] is something AI agents [49:33] cannot replace. [49:34] Right. [49:34] You understand the person. [49:36] Exactly. [49:36] And you can truly work with them. [49:37] Right. [49:37] Then together you can do meaningful work [49:40] that advances technology. [49:40] What could be better than that? [49:41] It's genuinely enjoyable, [49:42] I think. [49:43] but it’s just about doing interesting things, right? [49:45] Life is [49:46] very happy, really great. [49:48] Even though you are very humble, always saying [49:50] 'best paper', it doesn't matter, but [49:51] on your homepage you said a sentence: 'fight dragons, not windmills', which means to conquer [49:56] the evil [49:57] dragons, not the windmills, right? [49:59] How [50:00] does that windmill [50:01] facing the wind manifest [50:03] in your interactions with students? [50:04] I think that the [50:05] evil dragon actually represents a [50:06] real problem, while the windmill [50:09] represents a false problem. [50:11] So I believe that [50:12] it is very important [50:14] to solve a problem that [50:16] is truly meaningful, and solving [50:18] a false problem, I think, doesn't have much significance, [50:19] especially for systems research, because systems research is something [50:20] very close to practice. So if [50:22] this problem is not very important. [50:24] I think going to solve it might not be [50:26] that meaningful. [50:27] Two points: the first point [50:28] is how you help students discover this problem; [50:30] the second [50:31] point is, in a [50:32] cornfield in Illinois, Champaign, so far away from [50:34] Silicon Valley, how do you discover the [50:36] real problem? This is a very profound question. [50:39] I think, although this... firstly, [50:41] the second question is quite simple [50:42] because I think... [50:44] Even the most important [50:46] problems in the field of systems are just [50:47] a few problems; there aren't that many fundamental problems. But [50:50] the manifestations of these fundamental problems in actual systems are varied. [50:52] So personally, I always have a [50:55] view that many [50:56] people disagree with, but [50:57] my view is [50:58] that I think there [50:59] is no such thing as a small problem, [51:02] only some problems are not [51:03] explored in depth. in other [51:04] words, for me, there are only that many fundamental problems. [51:06] Then there are some fundamental issues in this, and [51:08] in the practical implementation of the [51:11] system, its reflection may not be the same. [51:13] Some problems may [51:14] seem big, some problems [51:16] may seem small, [51:16] but if you [51:17] dig deep enough, if [51:18] it is a [51:19] difficult or important [51:21] problem, it will inevitably return to a [51:23] very fundamental deep issue. [51:24] For example, many reliability issues ultimately [51:26] come down to a few fundamental problems, [51:29] such as how to reason about concurrency, right? [51:30] How to reason about asynchrony and then how to reason [51:32] about some nondeterministic behavior. If you [51:33] solve this problem, then the higher-level problems are all solved. [51:36] But these problems are very [51:37] difficult, so even if [51:38] you have a [51:40] very small problem, you can eventually [51:43] make it very big. And actually, [51:45] I suggest this to students. [51:47] At the beginning, start with a relatively specific [51:49] and smaller problem, rather [51:50] than going [51:50] for what we call 'boiling the ocean.' If you start [51:52] too broadly, it will be difficult [51:53] to handle. [51:54] It's better [51:55] to be focused with [51:56] a specific issue, which will [51:57] make it easier to [51:58] manage. You want to dig as deep [52:01] as possible, [52:02] but just do your best, and [52:03] work to whichever level you can, [52:04] then move on to the next project. [52:05] Can we go [52:07] on to solve some more fundamental problems, right? [52:08] This is the [52:08] second one, this [52:09] is your second question I think, [52:11] and then the first question is, [52:12] what is the first question? [52:13] It's about how you help students find [52:15] this most important question, right? [52:16] this, as I said, [52:18] depends on the [52:20] student's interest. Some students [52:22] may prefer theoretical questions, or some students [52:24] might really enjoy reasoning-type [52:26] things. Some students really like hacking; [52:27] their greatest strength is being able [52:29] to hack a [52:30] very complex thing very [52:32] clearly. At this time, based on their [52:34] interests, you can help map [52:36] it to something for them. [52:39] Compare the more important issues in [52:40] reality and then start [52:41] working on those [52:43] issues. These issues include some [52:44] students in my own group who like to do formal verification, right? [52:47] Formal verification means you need to define [52:48] the specification, use the correct tools [52:50] to write proofs, and then [52:52] finally the [52:53] entire system can be formally verified. Some [52:54] students prefer the [52:56] empirical method, which might include [52:57] ourselves; I myself, when [52:59] doing my PhD, [53:00] looked at many empirical cases. [53:02] Then think about how to [53:04] solve these problems through some automatic methods. I like [53:05] the second one. And when [53:07] you were at [53:09] UIUC, you took on some [53:11] responsibilities, like teaching the operating systems course. [53:15] Regarding responsibility, your operating systems course, I [53:17] saw online that students rated [53:19] it as really awesome, like [53:20] extremely impressive. There is nothing, [53:22] nothing, nothing—it's just about how [53:24] you teach the operating systems [53:26] course and why students like this course so much. [53:29] I feel like I don't know why students [53:31] like it either. Maybe it's because I told [53:33] a [53:34] few jokes in class. [53:35] I think [53:36] I should tell everyone about it. [53:38] I feel that this operating systems [53:40] course doesn't really [53:42] teach operating systems, at least for me, [53:43] because at UIUC, the operating [53:45] systems course is actually divided into two courses. One course is called [53:48] system programming, and it [53:49] basically teaches everything in operating systems [53:50] except for the kernel. [53:52] It's just that it [53:53] doesn't teach kernel programming, [53:54] but it teaches everything else, including [53:55] concurrency, process control, memory [53:56] control, all those things. This [53:58] course is [53:58] also very good. It [54:00] was taught by a colleague of mine named Lawrence Angrave, who [54:02] manages it, and it's [54:03] very good. [54:04] And for [54:05] the major assignment, it [54:06] might be like having you write a malloc, right? [54:08] Then they have you [54:09] write a shell and these [54:10] kinds of things. Actually, this course has [54:12] already covered most of the [54:14] basic training. The next [54:16] course is actually about kernel programming. But if you look at [54:18] the students taking the kernel programming course, [54:20] I think less than 10% of the students each year, maybe [54:22] even less than 5%, I believe [54:25] that after graduation, these [54:26] students will go on to do it. [54:29] For example, someone coming to the Bay Area to [54:32] be a Linux [54:34] developer—people like that—5% of [54:36] them, 95% of them, after they graduate, they won't [54:38] end up being [54:39] a kernel developer. What they might do later [54:41] is, for example, hacking Kubernetes or building some AI infrastructure, right? [54:44] So [54:45] what exactly did Linux kernel hacking teach him? [54:48] So I think the reason [54:50] why this is valuable to them [54:52] in this course [54:54] is not that [54:56] it completely explains this one thing. Kernel hacking [54:58] is a medium, but [55:00] the skills they gain are how to manage this [55:01] complexity. Because before this OS [55:03] course, basically all [55:04] the courses you took, [55:06] for example when doing big assignments, you were building something from scratch, right? Like [55:09] your data structures. [55:10] You write a small data [55:11] structure, in your [55:12] algorithms class you [55:13] write a small algorithm, and even previously you wrote a small [55:15] malloc, wrote a small shell; [55:16] they are all very small things, [55:17] but what about our class? [55:18] You are going to hack Linux, but [55:19] Linux is a very complex and [55:21] very dirty thing. For [55:23] a student, in just one [55:25] semester, it is actually very difficult to [55:26] fully hack and understand it. [55:28] So, in this process, how can you [55:29] make progress? Actually, this skill set [55:31] is very useful in a company, [55:33] including for myself when I went to Facebook. [55:34] I am facing a huge codebase, and [55:36] then you [55:38] want me to be able to make [55:39] progress in a month and [55:40] build some things. So [55:42] I need to have the ability [55:44] to abstract a lot of things, understand the complexity, and still [55:46] be able to make progress in order [55:47] to communicate, right? [55:48] Then the documentation isn't [55:49] there, and the code is all very [55:51] hacky, and this is a problem that exists [55:54] in Linux as well. Of course, Linux documentation [55:55] is actually quite good, but some things in device [55:57] drivers are not so, not so elegant. [56:00] And I think that [56:01] if a student, after finishing this course, can [56:04] gain the confidence to [56:06] face, as you [56:08] just said, a fierce dragon or [56:09] a world that is not so perfect. [56:10] I think this is the [56:12] biggest skill that this class [56:13] teaches them, so I think teaching OS [56:15] is meaningful, not that I want [56:16] to train all 100 of them to become kernel hackers. [56:20] I feel really deeply about [56:22] this, haha. Having graduated and worked [56:24] for three to four years, [56:26] I really increasingly feel that if you are an undergraduate [56:28] majoring in [56:29] computer science, courses [56:30] like OS, operating [56:32] systems, and network [56:33] systems really need to be [56:35] studied well. What I regret most now [56:37] is that I didn’t [56:39] study these courses thoroughly enough because you will find that no matter how big [56:41] data infrastructure, cloud infrastructure, or even AI infrastructure [56:42] develop now. [56:43] It hasn't escaped the original one, which [56:46] is the graphical [56:47] architecture, including the basic ideas [56:50] of computing, storage, and networks, but? [56:52] It's about how you make the complex things [56:56] in your mind become abstract. [56:58] Right. [56:58] It is about becoming [57:00] clear when working on a problem, [57:02] such as taking on a major task at [57:03] a company or even a big entrepreneurial [57:05] task, breaking it down step by [57:07] step into the smallest units, and [57:08] then stacking these units back together step by step. [57:10] Yes. [57:11] I found that I completely agree that the greatest [57:13] significance of these courses for me [57:15] is actually that I think these courses teach engineering thinking, [57:18] computer engineering thinking, which will [57:20] not be affected by external influences. As [57:22] long as you have this kind [57:24] of thinking, no matter how AI develops, [57:26] you will be able to find [57:28] a highly valuable job in this society. [57:31] I actually agree with what you [57:32] said, that there are some arguments now. [57:34] It means don't study CS [57:37] anymore. CS has [57:38] already collapsed. I don't [57:39] know how to put it. I [57:42] think maybe for some [57:43] parts, like the simpler kinds of software engineering development, the [57:45] demand is indeed getting smaller, [57:48] but overall, for building these big kinds of [57:50] projects, I think the [57:52] CS discipline will still not disappear. This discipline [57:54] will not disappear. I [57:56] don't think it will disappear. [57:57] Because I feel that such big statements are [58:00] all comparative, and [58:01] I find it hard to [58:03] understand what the argument behind them is. [58:05] I personally do feel that the [58:07] CS job market isn't as big as it used to [58:09] be because AI can take over a lot of tasks, right? [58:11] In the past, [58:12] even some very simple tasks [58:13] required you to hire a [58:16] CS undergraduate student to do them, but now AI can help you with [58:17] that. So I think, if what [58:18] we are worried about [58:19] is that this market will shrink, I do think it will shrink. But [58:22] if you say that OS will disappear, [58:23] I don't think that's possible. It [58:24] is still something that requires [58:25] people, just [58:27] like any other [58:28] profession, it will still require [58:29] people to do the work in the future. [58:31] The entire current modern society is [58:34] building large-scale computer systems. [58:36] Right. [58:38] Then let's talk about the relationship between [58:40] AI and these systems next. [58:43] You've been studying for over ten [58:45] years. From what I see, the main thread [58:48] of your papers is [58:50] actually quite clear to me: [58:51] how to find problems when [58:52] systems have issues [58:54] and how to fix them, mostly focusing on [58:55] reliability and mostly on fault tolerance. in other words, [58:56] there are now many kinds of AI [58:59] that are essentially SRE. [59:01] It's about letting AI debug this, which [59:02] is particularly popular right now, super, [59:04] super popular, even [59:04] getting funding is popular, and then. [59:06] Yes. [59:06] Is this the one that's popular [59:08] in academia too? [59:08] I wonder if it's not popular [59:10] in academia? [59:11] I don't think the [59:12] fire affects that. Maybe we can [59:13] talk about it later. So, what do you think these [59:16] AIs are actually capable of doing in [59:17] terms of human SRE work, and [59:19] what can't they do? Where [59:20] are the boundaries? [59:21] Actually, I think the capabilities of AI now, right? [59:24] Regarding the capabilities of those current [59:26] frontier models, like [59:27] Fable [59:29] or that GPT-5.6 Soul, right? [59:31] Then I think their abilities are already [59:33] very strong, [59:34] so basically I feel that for any system [59:37] task, Siri task—I’m not saying, [59:38] I mean any system task—if you tell AI how [59:40] to do this task, AI is basically capable [59:43] of doing it. So if the [59:45] question is [59:45] whether AI can do these tasks, I think it [59:47] definitely can. The key question is [59:49] how you go about doing it [59:50] and what the cost is, I think. [59:52] in other words, if we can lower this [59:54] cost, right, and also if we [59:56] can have a more general method to tell [59:58] AI how to perform these tasks, then I [60:00] think AI can definitely do these [60:01] things well. I don't [60:02] think AI would do poorly, and [60:04] I even [60:05] don't think AI lacks the [60:06] abilities to do these things. [60:08] So, what role does [60:09] a human play in this? [60:10] I think the role a person [60:12] can play in this is what? [60:13] I have always had a thought, and [60:15] I don’t know if it’s [60:17] a very stupid one, about guiding AI to do some right [60:19] things. Because for [60:20] me, AI is like a very smart [60:22] person. It’s like in [60:24] a company, let’s suppose your [60:26] company has an enormous [60:28] number of people, and all these people are very hard-working, you [60:30] still need to tell them [60:31] how to do the right thing. [60:33] I think that's the role [60:36] people should play [60:37] right now: showing junior engineers [60:39] what good work looks like. [60:40] how to go about doing [60:42] this, how to think about your trade-offs, and [60:43] what you should do? [60:44] How to decompose this task and then [60:45] tackle it step [60:47] by step, then let AI [60:48] handle some of [60:49] the repetitive work. But [60:51] I don't think [60:53] AI—at least with its current capabilities—if [60:54] you just give it a very vague definition or specification, it [60:56] can actually do [60:58] it, but it may [60:59] require many iterations, which would be very costly. And [61:00] this introduces another problem. [61:02] It's just that people go to guide these AI agents [61:06] and then let them complete some [61:08] tasks, and moreover, they are experienced [61:10] and tasteful people. [61:11] So the question [61:13] arises: how can a fresh [61:14] graduate become such a person? [61:15] Then there is no such training [61:18] guidance process, and I think [61:21] this is a huge impact on current [61:23] CS education. [61:24] I think, let's [61:25] imagine how [61:26] we could change [61:27] this to help current CS [61:28] students gain this experience. [61:30] I think that now students [61:33] should use AI to do [61:36] some learning and work. I [61:38] actually don’t really [61:39] understand why these things [61:41] are [61:42] considered difficult to learn now, because originally, as [61:44] a senior [61:45] engineer for example, you accumulated [61:47] that expertise over years of hands-on learning. [61:50] what they need [61:52] to do is to... [61:53] By guiding AI to [61:55] do this, as a [61:56] learning student, you [61:57] just practice this. You practice [62:00] your learning through AI and also practice your research [62:02] through AI, then naturally you will reach this level. [62:05] There are two [62:06] problems: the first [62:08] problem [62:09] is that you are too junior. [62:13] Some of your taste and guidance may [62:16] not be enough, and [62:17] then AI output might tell you some [62:20] things that are not really high-quality or [62:22] high-level. That's the [62:23] first problem. The second problem [62:24] is that these positions [62:25] are becoming [62:27] fewer and fewer. Originally, people would [62:28] slowly progress through small positions, which [62:30] would allow them to [62:32] learn interpersonal communication and improve themselves. [62:34] Right. [62:35] So these two issues lead [62:36] me to think that the [62:37] first issue is what I [62:38] mentioned before: I feel that [62:39] people are still a very important [62:41] element in this, and we shouldn't imagine an [62:44] action that is completely [62:46] without humans. So I think things like advice [62:47] or mentorship will not be replaced by AI. At least, that's my point [62:50] of view. in other words, [62:52] if you have such a growth environment, you [62:54] can still achieve something using AI. [62:56] Very good quality, and then the [62:58] market has shrunk, which is a very [63:00] real problem, [63:01] but actually I don't [63:02] really know how to solve this problem. [63:05] Then let's all reflect and discuss together, so [63:07] that the audience can chat in [63:09] the bullet comments and the comment section. [63:11] Right. [63:11] You've recently begun [63:14] working on AI for systems. [63:16] One project I found especially [63:17] interesting [63:19] was [63:21] your work on AI for SRE. [63:23] You built SREGym, [63:25] an environment [63:27] where AI agents tackle [63:30] site reliability problems. [63:32] It contains 102 real-world [63:36] IT operations scenarios, [63:38] yet agents solve only 11.4 percent of them. [63:42] What did you observe? [63:44] Why could the agents solve so few? [63:46] What were they missing? [63:48] Those results were from about a year ago, [63:51] I believe. [63:51] Yes. [63:52] First, I should point out [63:54] that my students led this project. [63:55] I [63:56] didn't initiate it, [63:59] so I can't claim it entirely as my own work. [64:01] As for the original benchmark, [64:04] I think [64:06] today's models [64:07] have largely solved those tasks. [64:09] Run it again with the latest models, [64:11] and I think they'd score close to 90 percent. [64:13] That much progress in one year? [64:14] Yes, in just one year. [64:15] AI capabilities have advanced dramatically. [64:17] AI's capabilities, anyway, there was a big fluctuation at [64:20] the [64:20] end of last year, that [64:21] is, Fibonacci [64:21] is not correct, not 4.4, 4.6, 4.8. [64:23] Right. [64:24] At the end of last year, [64:25] I felt it was the end [64:26] of last year and the beginning of this year, [64:29] that is, the end of 2025 and the [64:31] beginning of 2026, I felt there was [64:33] an essential capability, an essential leap, [64:35] and also, recently this year, anyway, a new wave of models [64:36] came out, and I think its ability has [64:38] increased again. I felt that during last [64:40] summer, its ability was still average, [64:42] and this work was actually done [64:44] at the end of the year before last, I think. [64:46] So at that time, AI capabilities were [64:50] not that strong, and [64:51] there were various problems, including very [64:52] high reasoning hallucinations, and its context was [64:54] not that large, so its capabilities were [64:56] very limited. But now, [64:58] these problems that [64:59] existed before are actually very easy for [65:01] AI to overcome. The fundamental [65:03] issue here is that the [65:06] problems we originally wrote were very simple. [65:08] Because I think the practice of benchmark [65:10] research is that [65:12] it gradually builds the [65:14] benchmark according to the capabilities of AI, right? [65:16] Because a benchmark, a good benchmark, [65:17] can measure the boundary of AI, so when AI capability is very [65:19] weak, if you create an [65:21] extremely difficult benchmark and AI cannot complete [65:22] any of it, then this [65:23] benchmark is actually not good. And of [65:25] course, if [65:26] an AI is very [65:27] strong and can complete 90% of the benchmark, then this [65:28] benchmark is basically a bit useless as well. [65:31] So the best benchmark is when [65:32] you can say that 30%-40% can [65:34] be done by AI, and [65:35] 60% AI cannot do, so it can keep challenging [65:37] AI. Now, actually, the [65:39] biggest challenge of our benchmark is this, and a [65:41] very clear problem is that [65:45] a benchmark itself is originally a [65:47] small, small environment, right? [65:48] As for us, we can only have [65:50] a small environment. Although our benchmark can actually have [65:52] some cases, for example, [65:53] a system with [65:53] three nodes or five nodes, [65:55] it is still very different compared [65:57] to hyperscale systems like Meta's. So AI is very strong [65:58] in these small environments, [66:00] but whether it is strong in these [66:03] huge environments, I think, is not necessarily [66:05] the case. But how [66:07] do you scale a small one... [66:09] How does the environment simulate a huge and [66:11] complex environment? [66:11] Which things should be simulated in [66:14] this small environment and which things can [66:17] you abstract out, and this is a point we pay [66:18] quite a lot of attention to, and [66:20] this is actually quite difficult, quite [66:22] difficult. [66:23] Right. [66:23] Because since [66:23] you go from a big system [66:25] to a small system, you inevitably [66:26] have to throw some things away. [66:27] So what do you throw away, right? [66:29] What I mean is that if the things [66:31] you throw away are very [66:32] important, then this small simulator of yours becomes useless. [66:34] But if you want to keep everything, [66:36] then essentially you can't turn a big system into a [66:37] small one. Yet you need the [66:38] small one because if [66:39] you want to do RL, you [66:41] can't do it if it's too [66:43] complex or online. So [66:44] I think this is one of the challenges of SRE. [66:47] Then the coding benchmark doesn't have this problem [66:48] because the coding benchmark is an offline thing—you just [66:50] keep generating code, right? [66:51] Then going [66:53] to do it for SRE [66:55] is too real-time, meaning that after [66:56] you solve a problem, can it [66:58] really solve the issue, [66:59] fix the online [67:01] bug, and get feedback? [67:03] And the whole feedback process [67:04] isn't offline, it's online. [67:05] So what we want to [67:06] solve now is this issue of fidelity. Can [67:08] you explain this in detail? [67:10] It's about [67:12] how to simulate a specific [67:15] production failure. [67:16] For example, recently [67:18] our school has a program, an undergraduate research experience program called [67:20] the REU [67:21] program, and this [67:22] summer our program is letting those undergraduate students... [67:24] What did he do? [67:24] He just went to read [67:26] a postmortem, this [67:27] postmortem could be from those high-tech [67:28] companies, like Cloudflare or [67:30] something. [67:31] Then they announced some failures, meaning they [67:32] said that [67:34] at a certain [67:35] time they had a failure, [67:35] and they put [67:36] the detailed description of [67:37] it online. Then the [67:39] students, based on their description, implemented [67:41] the failure in [67:42] this relatively small [67:43] simulated environment and tested AI, and we found that [67:45] even though this [67:46] was a small [67:47] environment, it still didn't… [67:49] It can completely represent this larger [67:51] environment, but compared to the most basic failures [67:54] the realistic failures you described pose a very different challenge for AI. [67:57] when facing these more realistic failures. [67:58] I heard two points of information: the first point [68:02] is that I think [68:04] students can indeed gain a lot of [68:05] training in this process, and the whole [68:09] school helps them to find it. [68:11] He can get hands-on with some of [68:14] the real problems of these [68:15] big companies in advance. The second point [68:18] is that in the process of abstracting it into [68:21] a small simulator, this simulator can later be used for RL. [68:24] Right. [68:25] this, nowadays [68:26] this kind of RL environment company simulator [68:28] company is really popular in Silicon Valley, [68:30] right? [68:31] It's too [68:32] hot, too hot. Any environment that [68:33] can truly help this [68:35] model improve its [68:36] capabilities—issues that previous models couldn't [68:39] solve can be solved after [68:40] training on it—then this [68:42] environment will [68:43] be sold at a very high price. I [68:45] see. [68:46] This is something that has already [68:47] become an unspoken convention in [68:50] Silicon Valley. [68:50] So, regarding this, [68:51] moving forward, SRE. [68:53] This environment will increasingly grow, I can feel it [68:55] will grow [68:57] more and more, and then AI [68:59] needs to learn through such an [69:01] environment to [69:02] continuously study the ability to solve [69:04] industrial-level production problems in the real world. [69:07] Yes. [69:08] That's great, it feels like UIUC is already ahead and [69:10] has started training students. [69:14] My next question is about another project. [69:15] Before this interview, [69:16] you also published a new paper. [69:18] I found that work fascinating. [69:19] It combines AI agents [69:21] with formal methods [69:23] to improve testing. [69:25] Right. [69:26] Could you walk us through [69:27] how it works? [69:30] Sure, what we are actually doing with model checking [69:34] is... what is the essence of model checking? [69:36] The essence of model checking is [69:37] that you have a very detailed low-level [69:39] code, and then you [69:41] need to write this code into [69:42] a high-level abstraction, and this [69:44] abstraction is described using a mathematical language. [69:46] This mathematical language can have various forms, right? [69:48] The language we are using, called [69:50] TLA+, was written [69:51] by Leslie Lamport, whom I mentioned [69:53] earlier. And if you [69:55] can describe this system using a [69:58] formal language, then you [69:59] can do a lot of formal analysis [70:02] called formal reasoning. You can [70:03] reason about many properties of [70:05] this system, [70:06] such as safety. [70:07] Safety and liveness: safety obviously means [70:08] that bad things will never [70:11] happen, and liveness means [70:12] that good things will eventually [70:14] happen. But as long [70:15] as you describe it [70:17] as a formal model, [70:18] you can check these properties. The most difficult part [70:21] here is how do you? [70:22] You're going to write [70:26] a high-level description of a very low-level system, right? [70:29] What about an abstraction? [70:30] So this matter is very intellectually challenging, and for a PhD student, [70:33] this [70:34] could be four years of [70:35] their time, four years [70:37] of their [70:38] life. Before [70:40] AI, we had a project [70:41] and a very talented student who is now [70:42] a researcher at Microsoft Research; he spent a full [70:45] five years doing [70:46] this work. [70:47] Why does he have to spend [70:49] five years? It's because he has to learn [70:51] this formal stuff from the beginning. TLA+, [70:52] this formal stuff, is actually not taught in any undergraduate courses, right? [70:55] This wasn’t taught in the course, right? [70:56] You might spend a year or two to [70:57] learn this set of things thoroughly, and [70:59] then what happens after you have learned it? [71:00] You also need to write a [71:03] specification. We say that specification is a very overloaded term. Actually, you [71:05] are writing a formal [71:06] formal models and invariants. [71:08] Invariants are the properties you want to preserve. [71:10] To write them— [71:11] to build a good model of a system— [71:13] you must understand that system [71:15] very deeply. [71:16] Otherwise, you can't abstract it properly. [71:18] You have to know the system inside and out. [71:20] For a systems researcher, [71:22] that should be a real system. [71:24] Let's use an example. [71:25] Uncle Moon, what's your favorite system? [71:27] A distributed system? [71:28] My favorite distributed system [71:30] would be Google's Borg, [71:31] or perhaps Kubernetes. [71:32] Kubernetes. Okay. [71:34] When we started, we used ZooKeeper. [71:35] ZooKeeper is a consensus system. [71:38] already very complex. [71:39] It has a consensus protocol called ZAB. So, what [71:41] about this student? [71:42] He has to go understand all the details [71:45] inside ZAB and be [71:46] able to understand how ZooKeeper implements ZAB, [71:48] and then understand all these things? [71:49] He can go and write a formal model, and [71:52] after he finishes writing [71:54] it, your system is [71:55] changing, your code is [71:55] changing, and tomorrow there will be a new [71:57] release. You still have to follow [71:58] the evolution of the code to write a different model, right? [72:01] This matter is very costly and also [72:03] requires a lot of skill, so the [72:04] student named Ruizé, Ruizé, spent [72:06] four or four to five [72:06] years writing all of these things, and [72:08] after finishing writing, then? [72:09] You can plug in all those formal and [72:11] very good tools and then let [72:13] them translate and then [72:15] discover the problems inside. You can [72:17] do model checking, and you can also do [72:18] proof. Proof is a [72:19] stronger guarantee, but this requires doing for 5 years, [72:22] and in 5 years we [72:23] only completed one system. But after [72:25] we completed this system, then what? [72:27] We solved a lot of problems [72:28] in this system that hadn't been solved [72:30] for many years, right? [72:32] There was one issue, I remember, in ZooKeeper, and [72:33] many people had studied [72:34] it before and it [72:35] was very complex. [72:36] But through formal reasoning, it could clearly tell you [72:37] what was going on. However, it was very [72:40] expensive; it would require [72:42] a PhD to spend five [72:43] years to do this. [72:45] But now, with AI, it's different. [72:47] It turned into a five-hour job—is [72:49] Is the improvement really that dramatic— [72:49] down to just five hours? [72:50] Yes, five [72:51] hours. [72:52] Five years of work became five hours, because [72:53] Five hours of work [72:55] because of AI — it doesn't need to learn all [72:57] these formal specifications; it [72:58] has [72:58] already learned them. Now, frontier models can [73:00] write TLA+ [73:02] really well, just [73:03] like writing in C or Python, [73:04] very skillfully, right? [73:06] Moreover, AI actually [73:07] has a deep understanding of the system because [73:08] it has already learned all the [73:09] source code and the like. If [73:10] you give it the [73:11] code again, it can immediately [73:13] learn from it, so all this [73:15] time is automatically handled by AI, meaning it [73:18] naturally already knows it. The only [73:21] thing not solved is the previously more intellectual tasks. [73:22] The most difficult part when you are modeling this [73:25] system is how exactly you go about modeling it. [73:27] Because if you still stick [73:28] to that idea of wanting [73:29] to keep everything, in the end, you won't [73:31] be able to build this model, [73:32] because you want everything. Your model [73:33] will become large enough [73:35] that you won't be able to [73:37] do formalized things. But if the model [73:38] is too simple, you won't [73:40] discover any problems either. Deciding what should [73:42] be modeled and what shouldn't used to [73:44] rely on the experience of skilled veterans. [73:45] Or rely on a very talented PhD student, on [73:47] his intuition, his intelligence. [73:48] Now we are handing this [73:50] over to AI. We say, AI, [73:51] you're quite smart, so [73:52] can you help [73:54] us figure this [73:55] out by yourself and help [73:57] us figure out the method for this? [73:59] Because a mature open-source project has gone through many years, [74:02] it actually knows [74:04] which parts are [74:05] tricky and which parts [74:07] are unlikely [74:08] to have problems, right? [74:10] Then this is actually us. [74:12] When human intuition comes into play, it's easy to make [74:15] mistakes, so [74:16] I want to model it more precisely. On [74:17] the other hand, I think [74:18] there’s no big problem [74:19] in another area, so I won’t [74:20] model it. So originally we were writing [74:22] this model, and now we just [74:24] tell AI that based on your [74:26] experience, this kind of [74:27] code is very difficult, very error-prone, and [74:28] very easy to go wrong, [74:29] so you should model it more carefully. [74:31] That place may have [74:32] never had any problems in [74:34] history, so you don't need [74:35] to model it. You just abstract [74:36] it out, and AI actually does this [74:38] very well. So after [74:40] running it once, AI can actually write a very [74:42] good model, and this model [74:43] can conform to your interpretation. Once you get this whole system [74:45] mature, you can leave the [74:47] subsequent work to it. [74:48] Traditionally, formal methods meant [74:50] model checking or constructing proofs. [74:51] Of course, the main challenge [74:53] is that [74:54] AI isn't human. [74:56] It may [74:56] engage in reward hacking. [74:57] It can produce a model [74:59] that looks correct, [75:00] but isn't what you asked for. [75:03] So [75:03] you need [75:05] ways to guide [75:06] or even constrain AI, [75:08] preventing [75:10] reward hacking [75:11] and helping it understand [75:13] the system step by step, [75:15] until it can [75:17] complete the task on open-source distributed systems. [75:19] It takes only five hours. [75:22] That's a lot to take in. [75:23] First, [75:24] has productivity in this field [75:25] really improved that dramatically? [75:27] From five years to five hours? [75:29] Yes. [75:29] That's why AI for formal methods is now [75:31] such a hot field. [75:33] Formal methods themselves [75:35] have existed for decades. [75:37] I've always [75:38] believed fundamental research matters. [75:41] We can move this fast today [75:42] because earlier researchers laid the groundwork. [75:44] The road was already there. [75:45] The reason it [75:47] wasn't widely adopted in industry [75:50] was the cost. [75:51] It required [75:52] highly specialized [75:53] expertise, which made it expensive. [75:55] It didn't fit Silicon Valley's [75:57] “move fast and break things” culture. [75:58] But at [75:59] NASA, this approach makes perfect sense. [76:02] They verify their systems, [76:04] because sending a [76:05] spacecraft to Mars only to have it fail [76:07] would be extremely costly. [76:09] They even avoid multithreaded code [76:10] because concurrency is so difficult. [76:11] They use single-threaded code and verify it formally [76:13] before deploying it [76:14] on missions like Mars Pathfinder. [76:15] That approach doesn't work in Silicon Valley, [76:16] where competition rewards speed. [76:19] But once AI automates [76:21] formal verification, [76:22] the economics change completely. [76:24] That's why so many [76:27] AI-for-science startups are emerging. [76:28] For a general audience, [76:31] what does this mean [76:32] for companies of all sizes? [76:34] It is something that can help [76:37] you solve some things [76:40] that are very difficult to do through [76:42] human reasoning alone. Now [76:45] you can use some mathematical tools [76:46] to help you verify [76:48] or check certain [76:50] behaviors in your system. Are these things normally [76:52] impossible to find through repeated debugging [76:55] and trial and error? [76:56] Debugging is a reactive approach, meaning you only [76:59] debug when a problem occurs, right? [77:01] But our dream is to ensure [77:03] that the failure never occurs. [77:05] If [77:07] we can formally verify the entire system, [77:09] then that class of failure cannot happen. [77:11] Verification means [77:12] we have ruled it out. [77:14] We have mathematically proven [77:15] that the system will not [77:18] suffer the failure we're targeting. [77:20] Before, [77:21] I understood why [77:22] many companies [77:23] found the return on investment too low. [77:26] Formal verification took too much time. [77:27] Right. [77:27] It could be cheaper to learn through failure. [77:28] Exactly. [77:29] Deploy the system, [77:30] and then [77:32] buy insurance, [77:33] if you have to. [77:34] Right. [77:35] If a failure happens only once every ten years, [77:36] buy a large insurance policy. [77:37] If it [77:37] does happen, let the insurer cover the loss. [77:39] Right. [77:39] But now, instead of [77:40] buying insurance, [77:42] a formal proof can guarantee [77:43] the failure won't occur for ten years— [77:44] or even a hundred. [77:46] make this mistake. So you [77:46] don't need to buy insurance and you [77:48] can happily run there. I [77:49] had a colleague before who used to help some [77:52] companies do this, for example, [77:53] helping large cloud computing providers. [77:55] Right. [77:55] Every time it [77:57] goes down, [77:57] it's very expensive, right? Like [77:59] that kind of EC2 going down [78:00] would make half of the [78:02] U.S. unable to access the [78:03] internet. That means the whole [78:05] service is down, so they have [78:07] to maintain a team specifically [78:08] to handle this. Before AI era, [78:11] the most enthusiastic about this—and still the most enthusiastic now—is [78:13] Microsoft because they have Azure, and reliability is very [78:14] important. And then there's [78:15] AWS. Basically, [78:16] it's just these two companies. [78:17] It is nurturing the team. [78:18] Yes. [78:19] But after having AI, [78:21] I feel that this barrier has become [78:22] very low, and then many [78:23] people can do this. So what do you [78:26] think will happen to the cost? [78:28] How many tokens it would consume [78:29] is to say that the issue [78:31] of generating these things and [78:33] the value produced in the end is more technical, because [78:34] it depends on what you want [78:36] to achieve. I think that if you want to [78:39] do a complete proof, at least [78:41] for now, it [78:42] is still very expensive. Previously, we didn't work [78:44] on that TLAPS thing, right? He [78:46] had written this for five years before, [78:48] and now he wants AI to write this proof. [78:50] Then he ran a [78:52] ZooKeeper system spec. A system spec is a [78:54] specification that describes the system; it is not a specification that [78:56] describes the protocol, so it is relatively low-level. Then he ran [78:58] it at that time, and [78:59] it cost $2,000 to [79:01] run for 5 days. [79:02] Well, I [79:04] think this is still... [79:05] A relatively expensive expensive thing, [79:07] but when we chose to [79:09] do this project at that [79:09] time, we did not choose to [79:10] do proof; we chose to do model checking. [79:11] Model checking is relatively cheap and you can get feedback [79:13] immediately. I thought we [79:15] wanted to make a tool [79:16] at that time, so I felt that [79:18] such a tool would be more practical. [79:20] It's somewhat more practical than doing [79:22] a proof, [79:23] or even more, more, more [79:24] practical, at least at the [79:26] present moment. But we also want to do [79:28] some proof to make some investment for the [79:30] future. To further understand it, [79:32] many companies can [79:33] start adding this during [79:35] the process of developing [79:37] their own systems; it's completely [79:39] possible to [79:39] use this [79:41] system while developing. [79:42] Yes. [79:42] Go and continuously verify. [79:43] Yes. [79:44] In that case, he won't have any problems with every step. [79:45] Yes. [79:46] It's just because we all work on systems and know that premature [79:48] optimization is the root of all evil. [79:50] Right. [79:50] Yes, but during [79:52] the process of building the system, [79:53] you start applying it right from [79:55] the simplest MVP process, and [79:56] then each time you add [79:58] something, you put [79:58] it in, and with [80:00] each addition, it seems like a very beautiful scene. [80:02] Yes. [80:03] Especially now, AI agents, whether under someone's supervision or not, even if someone [80:06] is watching them, you [80:07] no longer look at their code. Many [80:08] people do this, right? [80:09] I directly threw [80:10] out a requirement and [80:11] immediately a software came out. [80:12] Right. [80:13] If, under this circumstance, we [80:14] can provide a service that [80:17] helps these various people check the [80:20] messy AI systems they have [80:23] written before [80:24] going online, it seems [80:26] that many problems could be avoided. [80:27] Yes. [80:28] I think the concept that everyone [80:30] is talking about now is called SDD, which stands for specification-driven development. [80:32] Right. [80:33] in other words, you first write a formal specification, and then each [80:35] time you let AI write something, [80:37] you are able to prove it or verify [80:39] that the code you wrote satisfies your specification. In this way, no [80:42] matter what AI does, the final [80:44] product you produce can be verified. So if this can be [80:47] done, I think it can [80:48] reduce a lot of AI-related problems, I [80:50] think. [80:51] Whether from an academic or [80:54] entrepreneurial perspective, this is [80:55] a very good idea. I think now there are many [80:57] startups working on AI [80:59] for formal methods; it just depends on what [81:02] your [81:03] anchor point is, applying [81:05] various formal methods to [81:08] systems built for the current AI agents. [81:10] Right. [81:11] Go verify it, go prove [81:11] it. I find that it [81:12] really requires constant communication with everyone. For someone like me [81:15] who has never done this before, [81:16] I was completely blank on [81:16] this. Just now, you [81:19] probably already explained it [81:20] to everyone in the simplest language. Although I still [81:21] have a lot I don't understand, I can [81:23] sense that this is now feasible. But [81:25] what I want [81:27] to point out is that this is how [81:29] I feel now. [81:31] How should I say this? [81:32] in other words, for one of the very difficult [81:33] questions, you can see that there are now two voices, right? [81:35] For example, right now there [81:36] are two opinions in the field of [81:37] reliability. The first opinion is that [81:39] in AI era, [81:41] the biggest problem with using AI to build systems is reliability and security. [81:43] Yes. [81:43] You can't guarantee it, [81:45] and you [81:46] are particularly anxious, you [81:47] are particularly [81:48] anxious, this matter is too alarming. [81:49] But there is also a [81:51] voice saying that now systems can be formally verified, [81:52] and this process has [81:53] already started, taking 5 [81:54] years to be reduced to 5 hours or [81:56] 5 minutes. So why [81:57] is there such a huge gap here? [81:58] Here, if the first thing is true, the [82:00] second thing should not be true. [82:02] Why is that? [82:03] The reason lies [82:05] in the fact that even if you can formally verify it, [82:06] it only tells [82:09] you that your system code [82:11] is consistent with the specification you need; that is actually all it [82:13] does. But I still think [82:15] it is a very difficult problem, which is [82:16] what kind of system you need and [82:18] what your requirements are. [82:19] What kind of specification you want, I think this matter [82:21] is very difficult to solve completely [82:22] relying on AI; it is hard [82:25] to solve, and it requires [82:27] experienced people to handle this matter. [82:30] The higher-ups need someone to [82:32] guide the completion of this specification. Yes, this matter is [82:33] quite difficult because sometimes at the [82:35] beginning, you want this system, but you yourself [82:37] can't figure out what kind [82:38] of system you actually want, right? [82:40] Right. [82:40] You might, at the beginning when [82:43] you set up this system, say that mine is within [82:44] this formal specification, there is something called [82:45] a fault [82:46] model, which means you envision how your faults will look, [82:48] and then you can build [82:49] a set of fault-tolerant systems. But maybe tomorrow [82:51] or the day after, you [82:52] find that there are some issues in your infrastructure. [82:55] What about the other faults described by your [82:57] fault model? [82:59] So defining this specification is actually not that [83:01] easy, I think it's too difficult, [83:03] too difficult. [83:04] Right. [83:04] So the first half of your [83:06] entire career was spent designing these reliability, that operating system, [83:09] making it harder for people [83:11] to make mistakes, [83:13] better helping people solve problems [83:14] with these systems. Now [83:16] you can see that you are gradually starting to [83:19] design some things that connect with [83:21] agents, that is, allowing agents or [83:23] various AI tools to help [83:25] people do this maintenance [83:27] and build better systems. [83:29] Some system stuff of agent-native, hmm, [83:31] so [83:32] what's your ultimate, [83:33] for example, ideal? [83:35] in other words, in the end, you might not [83:38] be able to reach the state you [83:39] hope the computer operating system will become. [83:43] I think this question is [83:45] quite deep, and [83:46] I still believe it should start from the perspective [83:47] of reliability. A reliable system is [83:49] one [83:50] where, in the future, [83:51] at least 80% of its users are [83:55] agents, and agents are not perfect—they can [83:56] make mistakes, right? [83:58] Then it might hallucinate, and then it might [84:00] be off-base, so how [84:02] can your system tolerate these things [84:06] and also provide some [84:06] feedback? Because the agent is [84:07] also a very smart thing; as long [84:08] as you give it feedback, it will [84:09] do better in the second [84:10] round. And I think there are [84:13] There are many [84:14] systems problems that still need [84:17] to be solved. [84:17] Without solving them, this vision won't become reality. [84:20] I feel the same way. [84:22] The future of these systems [84:23] looks incredibly promising. [84:26] But when you examine the details, [84:28] you see how many hard problems remain— [84:29] how many PhD students [84:30] and systems [84:32] engineers it will take to solve them one by one. [84:34] It feels a little like the Manhattan Project. [84:36] I think many of the [84:37] core principles already exist. [84:40] Then I [84:42] don't know if you know [84:43] have you heard of a project called ROC— [84:45] No, I haven't. [84:46] Around the year 2000, [84:49] David Patterson, then a professor at Berkeley— [84:51] a Turing Award [84:52] winner [84:53] and one of the pioneers of RISC-V— [84:54] led a project [84:55] called ROC: Recovery-Oriented Computing. [84:57] Its central premise [84:59] was this: [85:01] computer systems inevitably fail. [85:02] In other words, [85:04] even if you spend [85:06] all your resources [85:07] on testing and formal verification, [85:09] the system will still fail. [85:11] So we should invest [85:13] substantially in [85:15] recovery, rather than assume [85:17] a system [85:20] will always be reliable. [85:20] Instead, ask: [85:22] what can we do when the system [85:24] does fail? [85:25] Can we build [85:26] a system [85:29] that recovers quickly [85:30] after a crash? [85:32] This is their big project. It is [85:33] a collaboration project between Stanford and Berkeley. At that [85:36] time, they received a lot of money from [85:36] the NSF, and they had [85:38] They had many ideas [85:40] that could not be realized at the time. [85:41] The first idea was [85:43] to look at what makes [85:44] a good system fail. [85:45] Their approach [85:46] was human-centered. [85:47] They [85:48] studied real production systems, [85:50] and asked a fascinating question: [85:51] when a system fails, [85:54] is the hardware at fault? [85:57] Is it the network? [85:58] Or is it the people operating the system? [86:00] When we think about fault-tolerant systems, [86:02] we usually assume [86:04] the hardware failed, [86:05] a machine crashed, [86:06] or an Intel processor suffered a bit flip. [86:08] Something along those lines. [86:09] But their empirical study found [86:11] that the largest share of failures [86:12] came from human error— [86:14] operator error. [86:15] People have to operate these systems, [86:17] and the systems are complex. [86:19] So people make mistakes, [86:21] and those mistakes are often [86:22] the leading cause of system outages. [86:23] Jim Gray explored this issue. [86:24] He later won the Turing Award [86:26] and wrote a seminal paper on the subject. [86:27] At the time, he was with a Silicon Valley company [86:28] called Tandem Computers. [86:30] Its product line was the NonStop system. [86:33] The company aimed to build [86:35] a system [86:36] so robust that it would never stop— [86:38] hence the name NonStop. [86:39] Gray studied it [86:41] in depth. [86:42] The company had invested enormous effort [86:43] in reliability, [86:44] yet the system still failed. [86:46] So he wrote a paper [86:47] based on an empirical study. [86:48] He pointed out [86:49] that the company's [86:50] core selling point was reliability. [86:52] So in the end, when we sold it to the [86:55] user and it crashed, what was the [86:56] actual reason? Then he found the same problem, [86:57] which was that the main [86:58] reason was human operational error. It wasn't [87:00] the hardware; the [87:01] hardware wouldn't crash. The system [87:01] software is already capable of fault tolerance, right? [87:03] Then David Patterson, anyway, just pushed this [87:05] matter forward a step, [87:06] and he said that [87:07] we cannot assume that users won't make [87:09] mistakes; we should assume what to do [87:10] when our system crashes. [87:12] Then he had two very cute [87:13] ideas. One idea is called system-wide undo, [87:16] which means if [87:18] a person makes a failed [87:19] operation, how can I undo this? [87:22] I'll redo it, right? [87:23] Because people make mistakes, right? [87:24] So if someone makes a mistake, I would do [87:26] it again, again, just like when I first applied [87:28] for a PhD and failed, [87:30] I would reapply. But he thinks [87:31] this system must be able to allow you [87:33] to do this. If this system doesn’t let [87:34] you do it, after [87:36] failing once, how can you undo it? [87:36] Because actually this matter is very difficult [87:38] since there is no system to support you. [87:40] For example, if you send [87:40] an email, once you [87:41] have sent it, how can you undo it, right? [87:43] So it must have a very [87:44] systematic mechanism to ensure that you can do [87:47] this undo, and at that time [87:48] they had a paper, and this matter was very difficult [87:50] to generalize, so they made [87:51] a special case. They built an email server, [87:52] and they felt that an email server was a very difficult [87:55] thing because sending emails or what? [87:56] These things are very difficult [87:58] to roll back, so [87:59] they said we will [88:00] focus on a relatively difficult problem and create an [88:02] email server that can be undone. Then they had three, [88:05] maybe five, principles. This was a [88:06] paper they published at USENIX ATC [88:08] at the time, presenting one [88:09] of their cute ideas. Another cute idea is called microreboot. [88:10] Then what he meant at that time was, [88:12] when we have a problem in the system, what is [88:14] the most routine action we take? It's restarting. In fact, we rarely, [88:17] for example, if we [88:18] have a blue screen, very rarely [88:19] do we say, after I [88:20] get a blue screen, I need to find out where this bug is? [88:22] If we just [88:23] restart directly and it can work again [88:24] and again, then we can continue working, right? [88:25] So 70% of recovery mechanisms are reboots, but rebooting is [88:30] very expensive [88:31] in industrial systems because [88:33] you have caches [88:34] and things like TLBs, right? [88:35] You just rebooted, and all your [88:36] states were lost, so this is [88:38] very expensive, but [88:39] it [88:40] is a very effective method. So what [88:42] should be done? [88:43] So can I build this system into [88:46] one where you don't [88:47] need a global reboot but can do a [88:49] microreboot, meaning you only reboot that one component [88:51] instead of rebooting the entire memory or something like that? [88:53] This was an idea they [88:54] had at the time. They [88:55] also created a case study back [88:56] then, which [88:57] was hard to generalize, so [88:58] they just made a [89:00] case study. But this has [89:02] Today, that idea is reflected in [89:05] microservices and serverless systems. [89:06] You can microreboot a single microservice [89:07] or restart one service. [89:08] Services are restarted all the time. [89:10] So the idea already exists in practice. [89:13] In the age of AI agents, [89:13] we can revisit those classic principles [89:16] and reuse them directly. [89:17] We can ask which ideas [89:18] remain valuable [89:20] and can help [89:21] AI agents, and which [89:22] we no longer need. [89:24] Then we won't have [89:26] to optimize for every old constraint, [89:28] and can design [89:29] a much better system for AI agents. [89:31] Wow. I feel [89:32] I really [89:33] need to learn more about the history of computing, [89:36] as Professor Juncheng suggested. [89:39] That history [89:40] contains [89:42] a great deal of hard-won wisdom. [89:44] Exactly. [89:45] Applying that wisdom in this new era [89:47] can lead to remarkable results. [89:50] now, when you were talking about this, [89:51] undoing this matter, designing [89:53] a system — this [89:54] system is used by an agent, [89:55] and then every operation of the agent can be undone. What [89:58] a beautiful set of systems. [89:59] Yes. [90:00] I think this is completely achievable. This [90:01] is not something that is [90:03] theoretically very difficult; it's just [90:04] a matter of whether anyone is [90:07] willing to do it. I [90:08] think a big challenge of AI systems nowadays is [90:10] that they are developing too fast, right? [90:12] It’s rare for people to be willing to [90:14] say that I should focus and spend [90:16] a year or two building a system. I think [90:18] this is a challenge. in other words, if [90:21] your AI capabilities are constantly changing, or [90:23] many of your [90:25] paradigms are constantly changing, then [90:26] indeed, the challenge is [90:27] how you quickly build [90:28] a system, be [90:29] able to think it through clearly, and make [90:31] it usable for future agent paradigms as well. [90:32] I think this is a challenge—a real challenge. [90:34] Moreover, previously everyone could [90:35] feel that after Claude's upgrade, some [90:39] aspects actually became less user-friendly. But [90:42] this is a training process—you add [90:44] data and then [90:45] remove data. If [90:47] this system is applied here, [90:50] you can undo, right? [90:51] You don't [90:52] need to retrain, you can undo something, right? [90:54] Each training session costs hundreds [90:55] of [90:56] millions of dollars, and this seems [90:58] useful whether from the perspective of large AI training [91:01] systems or small AI applications. [91:04] Yes. [91:04] But it's [91:05] quite difficult; this [91:06] requires dedicated practice. I think now [91:10] is a very good [91:12] opportunity because in the past, [91:13] for example, twenty or thirty years [91:15] ago, reliability and security were never a [91:18] first-class principle, right? Because reliability and security don’t make money, [91:20] only performance makes money, performance makes [91:22] money. [91:23] Basically, almost everyone building a system relies [91:25] on the first thing, which is [91:27] performance. If [91:28] you improve performance, or [91:29] features—features along with performance—these two things make money. As for reliability, [91:31] that is, or security, you just worry [91:32] about not losing money, [91:33] so you can actually [91:35] achieve it through many other [91:37] methods, including, as [91:39] I just said, buying insurance, buying [91:41] some insurance. [91:42] Or you could say that even though you [91:44] write a lot of test cases, and the test [91:45] cases are never comprehensive, at least you have a [91:48] lot of confidence, so you have a [91:51] company invest a lot in security and reliability, which many companies are unwilling [91:53] to do. But I think [91:54] the agent has turned this around because with you, if [91:57] you let [91:58] the agent take the lead in doing many [92:00] things, including writing code. [92:00] So reliability and security are the biggest issues, and [92:03] they are [92:04] not problems you can solve [92:06] just by scaling test cases. So [92:08] I think this is a [92:09] great opportunity, and researchers can take [92:12] advantage [92:12] of it, especially reliability and security researchers, who can then obtain a lot [92:14] of resources and [92:15] do some more meaningful things. [92:17] In fact, it's like [92:18] this: nowadays, the people who hold [92:20] resources in every company have also [92:22] realized that having agents do things to raise the [92:24] ceiling isn’t that [92:26] difficult. Instead, the real [92:27] challenge is how to continuously trust these agents, which makes [92:29] everyone very anxious. [92:30] Including when you are using some AI agents now, you will find [92:33] that when it comes to high-value tasks related to money, [92:35] filling out forms, or [92:37] personal information, if you ask it to [92:38] do these things, the [92:40] official stance is that they do not [92:42] recommend you do this because [92:43] they do not want to take on [92:45] this responsibility. But if someone can help [92:46] solve the security issues generated by these [92:49] systems, the value that this brings to everyone would [92:52] be immeasurable. I [92:52] think this is the current situation. [92:54] For example, one of the hottest topics [92:56] is trustworthy AI and so on. Yes, that's right. So [92:58] recently, you [92:59] also came [93:00] to the Bay Area, to [93:03] UC Berkeley for a visit. What kind [93:05] of considerations led to this? [93:06] I haven't given it much thought, [93:08] because [93:08] I'm not someone [93:10] who plans very far ahead. [93:13] I never make five-year [93:14] or even three-year plans. [93:16] I mostly think about [93:18] what I want to do this year. [93:21] My planning horizon is fairly short. [93:22] I'm currently on sabbatical. [93:24] American universities usually have a system [93:26] under which, after six years [93:28] of service, [93:29] a professor may take [93:31] a year-long sabbatical. [93:33] The basic idea is [93:35] that every institution has its own constraints and biases. [93:37] If you stay [93:38] in one place [93:39] for too long, [93:43] it becomes hard to think outside the box. [93:45] You start taking [93:46] everything for granted. A sabbatical encourages you [93:48] to experience another environment, [93:50] one very different from your own, [93:52] and reflect on what you see. [93:53] When you return to your university, [93:55] you can adopt [93:56] some of the good practices you've encountered [93:59] and improve some of the bad ones. [94:00] That's [94:02] the purpose of the sabbatical system. [94:03] I became eligible for one last year. [94:05] I had originally planned to take it [94:07] after completing six years [94:09] at UIUC. But last year, the Trump administration [94:12] made major cuts to NSF funding. [94:15] And [94:15] UIUC is a very large public university— [94:17] one of the largest in the country. [94:19] The steep cuts in federal funding [94:22] put enormous pressure on the university. [94:23] To be honest, [94:26] there was a fair amount of anxiety [94:27] in our department, and I felt [94:29] that anxiety spread to many PhD students. [94:31] They saw the funding cuts, [94:33] alongside the impact of AI, [94:34] and everyone was nervous. [94:36] At the time, I felt [94:39] it wouldn't be right for me to leave [94:41] in the middle of that, [94:42] so I decided [94:43] not to take my sabbatical [94:44] just yet. [94:46] I postponed it for a year. [94:47] Things felt more stable this year, [94:49] so I decided to go ahead [94:52] with the sabbatical. [94:53] I didn't have a specific plan [94:54] at first. [94:55] Then one day I ran into [94:56] my current host, Ion Stoica, [94:57] and mentioned it to him. [94:59] He said, “You're welcome to come to Berkeley.” [95:02] I thought, [95:03] “Sure, why not?” [95:05] And that's how I ended up here. [95:06] He was very thoughtful about it, though. [95:07] He wanted [95:08] to make sure [95:09] I'd be happy here [95:10] and that the visit [95:12] would give me [95:13] what I was looking for. [95:14] So he invited me to a SkyLab retreat. [95:16] It was much larger [95:17] than I'd expected. [95:19] There were [95:20] people from [95:21] industry and academia, as well as students, all presenting. [95:22] I went, [95:22] liked what I saw, [95:23] and decided to come. [95:24] There really wasn't much more planning than that. [95:26] There was a lot packed into that answer. [95:28] First, given [95:31] the current climate, [95:34] there is real anxiety at both [95:36] the institutional and individual levels. [95:37] It's hard not to be affected [95:38] by upheaval on that scale. [95:40] Yes, some [95:42] things are simply beyond our control. [95:44] Second, once your work reaches a certain level, [95:47] it's easier to visit leading research groups. [95:49] A sabbatical visitor doesn't place much burden [95:51] on the host institution. [95:52] Instead, you become a colleague they can exchange ideas with, [95:54] so most universities are very welcoming. [95:55] If any professors watching this [95:58] are considering a sabbatical, [96:00] don't worry [96:01] too much about imposing. [96:04] Reach out and ask. [96:04] There are more opportunities than you might think. [96:07] That's also one reason [96:08] I chose Berkeley. [96:09] Berkeley's research culture differs sharply [96:11] from UIUC's. [96:12] In some ways, they're opposites. [96:14] It's not about one being right [96:15] or wrong; they're simply very different. [96:17] For a sabbatical, [96:18] I wanted to go somewhere genuinely different. [96:20] That made Berkeley [96:21] an excellent choice. [96:23] I could have visited a university more like UIUC. [96:24] It might have been an outstanding school, [96:26] but if the culture [96:27] were too [96:28] similar, what would be [96:29] the point of [96:30] going there? [96:31] I've heard it said [96:33] that a sign of real intellectual maturity [96:35] is the ability to hold [96:37] multiple viewpoints, methods, and worldviews in your mind. [96:42] Exactly. [96:42] You already have a strong methodology [96:44] and the practical Midwestern research worldview. [96:47] Now you want to see something different. [96:50] Right. [96:51] And given your personality, [96:53] you didn't arrive with a fixed idea [96:54] of what you had to accomplish. [96:55] You simply came to explore. [96:56] Yes. [96:56] I think research is [96:57] a beautiful thing. [96:59] And beautiful things [97:02] don't [97:03] follow a single standard. [97:04] No rule says everyone must [97:08] follow Method A or Procedure A. [97:10] What makes academia and research so appealing [97:13] is that many schools of thought can flourish. [97:15] People can use different methods and philosophies, [97:16] or follow different principles, [97:17] and still succeed. [97:18] If the result is something [97:21] beautiful that people value, [97:22] then we should [97:24] make room for [97:25] many different styles. [97:27] Berkeley's style is very different [97:28] from UIUC's, and [97:30] UIUC's style [97:32] is also different [97:34] from MIT's. That's a good thing. [97:35] All of those approaches [97:37] should be able to coexist. [97:38] That's one reason sabbaticals exist: [97:40] they encourage you [97:41] to experience something different [97:43] and exchange ideas across institutions. [97:45] There's another memorable line: [97:46] beautiful things have no single standard. [97:49] Exactly. [97:50] Let a hundred flowers bloom. [97:50] Yes. [97:51] I think that's part of [97:54] human nature. [97:54] Now that AI agents are so capable, [97:56] if you want to live a happy life, [97:59] you need to seek out [98:00] things that cannot be standardized. [98:03] People have been saying this a lot lately. [98:04] What is AI best at? [98:06] Standardized tasks— [98:07] problems with clear questions [98:08] and clear answers. [98:09] What is AI worst at? [98:10] Things that are difficult to define, [98:11] things that resist [98:12] standardization. I think you're absolutely right. [98:14] Yes. [98:16] That wraps up [98:18] our discussion of your research. [98:22] Let's move on to some [98:24] broader, more philosophical questions. [98:26] During your PhD, [98:28] you interned at several places—NetApp, [98:32] Facebook, and Microsoft Research— [98:33] and you've now spent many years [98:36] in academia. [98:39] What do you see as the biggest difference [98:41] between industry and academia in computing? [98:45] By industry, do you mean [98:48] industry research? [98:49] Otherwise, the comparison [98:50] isn't really apples to apples. So what's the difference [98:52] between industry and academic research? [98:53] Either interpretation is fine. [98:54] Answer it however you think makes sense. [98:55] Here's how I see it. [98:58] Industry and academia [98:59] approach research differently. [99:00] Both are forms of research, [99:02] and share the same [99:03] fundamental purpose, [99:04] but they take very different forms. [99:05] I think academia— [99:07] especially now that we have AI— [99:08] needs to focus even more on fundamental research. [99:11] Academia has fewer resources now, [99:14] partly because of government intervention. [99:16] With fewer resources, [99:18] you have to pursue truly fundamental questions. [99:20] Academia's job is to push the frontier. [99:23] You may push it today, [99:25] and AI may learn from it tomorrow. [99:26] But unless we [99:27] keep pushing that frontier, [99:28] AI will have nothing new to learn. [99:30] That's what academia should do. [99:32] That's what good academic research looks like. [99:35] Industry research, by contrast, [99:36] is applied research. [99:37] It should be tested in practice. [99:41] It has to be useful, [99:43] and it should [99:45] produce results in the near term, [99:47] ideally with significant impact. [99:49] So the two styles of research [99:51] are very different. [99:53] different, and the methods of [99:54] carrying them out are also very different. [99:56] This is why I [99:59] think, for example, when [100:00] a person is facing a [100:01] choice, the first thing [100:02] they deal with is the question of what kind [100:04] of research they want to [100:05] do. If someone says, [100:07] 'I just want to do very practical research that [100:08] can benefit many people.' [100:09] Many people say tomorrow [100:11] or next month, so I think industry research might [100:12] be a very good goal. [100:13] Some people say they [100:14] just want to do a pie-in-the-sky research, right, just looking at [100:15] the sky and [100:17] imagining what the [100:19] future world would be like. [100:20] I think that kind of research is [100:21] quite good at Princeton. Before that, [100:23] Jun Cheng [100:24] had a point. [100:24] It means that when a PhD goes [100:26] for an internship in the industry, they shouldn't do [100:28] research. He said, Jun [100:29] Cheng said that a PhD in the [100:30] industry should be completely immersed [100:32] in the industry's system every day, [100:34] seeing what the system engineers [100:36] around them are working on. [100:37] What does he write? [100:38] Then get the most frontline understanding of this system, and [100:41] then go back to school to slowly refine it [100:44] to see if you can make something better. [100:45] What do you think of [100:47] this viewpoint? I think this is a [100:48] very good method. I think the reason why [100:50] we should do internships is because I still strongly encourage [100:51] students to intern. I think internships are also a [100:53] very good thing because you [100:55] change your environment and then see some different systems [100:57] that you originally had no chance to be exposed to, [100:59] right? [101:00] And also, you work with different people, so [101:03] I think if you [101:05] go for an internship and still follow [101:07] the school's way of doing [101:09] things, that's obviously not a very good practice, right? [101:10] So I think if [101:12] you go to the industry or do [101:13] an internship, you should do [101:15] something different this [101:16] summer, and then learn [101:17] something. Of course, there are many [101:19] ways to learn. I think, [101:22] for example, in my first [101:23] period, I applied my own research to the industry system, [101:25] and I think this is a [101:26] very good, very good method. You can [101:28] also do that. [101:29] It is about going into the industry to learn about their [101:31] problems, and then the most important [101:33] problem there. First, [101:33] I have a student, Xudong. [101:35] His first internship was at [101:36] Walmart. I remember that in his [101:37] first two [101:38] weeks, he would just catch [101:40] people and ask them, 'What [101:41] is the most important problem in your company?' [101:43] I want to address your most [101:44] important questions, and then he received a dozen answers. [101:47] Many people also couldn't frame [101:49] it as a research question, but everyone would honestly tell [101:50] him what they think is the [101:52] most important or the hardest problem in [101:54] their work. Then, at this point, [101:55] you refine it yourself. I [101:56] think this might be a [101:57] bit similar to what [101:59] Juncheng said, [101:59] maybe too similar, but I think it's about understanding this, [102:01] because now. [102:02] Academia is no longer the academia of the 70s, right? [102:05] Back in the 1970s, there might have [102:07] only been five [102:08] computers in the United [102:09] States, all in schools, and the industry would immediately [102:11] follow up on whatever the schools were [102:13] doing. But now, you see, the industry [102:15] is often completely ahead of academia. [102:17] Even when I was doing my PhD, [102:18] Google had already implemented systems like MapReduce and Bigtable [102:20] and then published a paper. [102:21] Then thousands of papers are optimizing MapReduce, and then [102:24] publishing Bigtable; thousands of [102:24] papers are optimizing Bigtable. So I think that many times [102:27] industry is ahead [102:28] of academia. Therefore, I think when you go into [102:30] industry, you should do things that [102:31] industry [102:33] needs to do. Wasn't there an old [102:36] saying about this before? [102:38] When you go to [102:39] Rome, you should do what the Romans do. [102:41] I think this is right, I think. [102:42] Totally agree. [102:44] So if one of your students asks [102:47] you when they graduate, 'I got [102:49] two offers: one is to go to Google [102:51] to work on this system, and the other is [102:53] to go to a certain school to be a professor. How would [102:56] you help me make a decision?' [102:59] One is to go to Google and work on core systems. [103:02] Right. [103:02] Another option is to [103:03] go to an elite school to be a professor. [103:06] This choice is very personal. I think it has to [103:08] do with what he and the other person are [103:12] pursuing, including his financial situation, right? [103:15] Or his need for certain things, and [103:17] then his personality, I [103:18] think all of [103:19] this applies—he is not someone [103:21] who can absolutely... Still, it's the same [103:22] statement: there is no formula. He has [103:23] to make a choice based on the different situations [103:25] of each person, I think, [103:27] including going into the industry to create a system. And when [103:28] you create this system, [103:30] what exactly are you doing, right? [103:31] in other words, if you [103:33] go to this factory [103:34] to screw in screws, it [103:36] is possible that [103:36] the system is very pretty, but when you screw [103:38] in [103:39] the screws, you may not necessarily feel that it's pretty. [103:41] So I think any experience, the first [103:44] thing for me is that in this [103:45] experience, what kind of things [103:46] you can learn. I think this [103:47] is the most important question, whether you go into academia or industry. [103:49] If you go to an extremely beautiful place [103:52] but don't [103:52] learn anything, then I [103:54] think it's still nothing, for me, right? [103:55] What it means to interrogate yourself [103:58] is asking, 'If I work in this [103:59] position for five years, what kind of [104:01] skills will I be able to learn, [104:03] and what abilities will I be able [104:04] to develop?' Once you answer [104:05] this question, I think you will have a [104:09] very clear next step. [104:10] And many times, what you are even asking is, 'Who [104:12] should I work with?' I [104:13] think who you work with depends on your level, and [104:15] on whether they [104:17] are very enthusiastic. [104:19] Working with people who [104:20] are excited or very enthusiastic [104:21] about technical things is different from working with [104:24] people who might [104:25] just want to get by. [104:27] Even in a very big company, I [104:28] feel like you can't [104:30] really learn much from those people. So, [104:32] whether you [104:33] go to a startup, a [104:35] big company, or academia, essentially it's about what you can learn. [104:37] Of course, there are many other constraints as well, right? [104:39] I think a big problem with academia is [104:41] that everyone is very [104:43] poor, so if you really [104:44] need money and your [104:45] family needs money, I think [104:46] it might not be a [104:47] particularly good choice. [104:49] Anyway, we've basically covered [104:50] everything, right? [104:52] There are interests in certain [104:53] professions, in working with [104:55] certain people, and in various kinds of family situations. [104:57] Right. [104:57] Especially what you just said about [105:00] working with very exciting people—I’ve heard countless statements from [105:01] big shots about this, which means that [105:04] the things this team is doing and [105:07] the people in this team make me [105:09] willing to [105:10] join this company on the day I graduate even [105:13] if they don’t pay [105:14] me. I think everyone [105:15] is just—in fact, finding a job is [105:18] getting increasingly difficult, and everyone’s [105:19] career development is a bit unclear. [105:21] Climbing the ladder, [105:22] going for a promotion, these [105:23] things are all very [105:25] vague, but a lot of fans in the [105:27] background often ask questions about how to [105:29] choose. Actually, I think there is only [105:31] one standard, which [105:31] is really just what you [105:32] mentioned earlier: whether you can learn [105:33] something and whether being with these people is exciting or not. Yes, especially in the [105:35] early stages [105:36] of your career, this is [105:37] the most, most, most, most important. [105:38] Right. [105:38] The other [105:39] things you just mentioned [105:40] are very honest words. Some [105:42] big companies or certain places may [105:44] seem dazzling, but the small team isn't [105:47] really up to par, [105:49] and then you [105:50] are really correct because I [105:51] think that's the case. Academia is a very standardized place. The [105:54] best schools in the U.S. [105:55] and just an average school [105:57] in the U.S. actually don't have much [105:58] difference. As a faculty member, you [105:59] just do those three things, right? [106:00] One is teaching, [106:01] one is doing [106:02] research, and another is mentoring students, [106:03] but in industry. [106:06] Different teams in [106:07] different companies vary [106:07] greatly, so it's actually hard [106:10] to summarize in one sentence whether [106:12] you should take an industry job. You can do a [106:14] very creative job in the industry, or [106:15] you can do a [106:17] very boring job. Therefore, you need to understand [106:18] exactly what this job [106:19] entails. A good way to [106:21] do this is to talk with [106:23] people in the team, ideally, and see what [106:24] their thoughts are. [106:26] Actually, once you talk to someone, [106:28] you'll know whether you want to work [106:29] with these [106:30] people. Many times, this is [106:31] the core reason, because one company has [106:33] too many people, too [106:34] many teams, and too many projects. [106:35] Right. [106:35] Can't you figure out what you want? [106:37] Then you can't work with these people [106:38] in a highly excited way every day, which [106:40] I think is especially a waste of your life [106:42] in the early stages of your career. [106:43] Yes. [106:44] My former PhD advisor [106:45] once made an excellent point. [106:47] He said that in computing, [106:48] your first job is not your forever job. [106:51] So don't [106:53] agonize over [106:54] your first job, or imagine [106:56] it's a secure position you'll keep for life. [106:58] Treat it as a learning process. [107:01] If you learn— [107:01] really learn— [107:03] your next job will be better. [107:05] If you don't, [107:06] even a huge compensation package [107:07] won't stop you from being laid off two years later. [107:09] That's true. [107:10] The computing industry changes incredibly fast. [107:11] The pace [107:13] over just the past five or six years [107:15] has made that obvious. [107:16] We don't even need to go back to the early internet, [107:19] the dot-com bubble, [107:20] or the mobile internet. [107:21] Just look at AI era. [107:22] In only two or three years, [107:23] so much has changed. [107:24] AI has gone from [107:25] basic chatbots to writing code. [107:28] Next, for all we know, [107:29] robotics could take off. [107:30] You might have [107:31] happily built a chatbot when they first appeared, [107:34] only to find yourself far behind this year. [107:35] That's how much can change in two years. [107:36] As for industry, [107:38] this is just my personal view, [107:39] and it may be biased, [107:40] but I believe the most important thing [107:42] is the ability to keep learning. [107:44] The greatest danger in industry [107:46] is taking a position [107:48] and then stopping learning. [107:49] With technology [107:50] changing every day, [107:53] you will inevitably be left behind. [107:55] The advantage of academia [107:56] is that it forces you to learn. [107:58] As a faculty member, [107:59] you have to advise students, [108:01] apply for funding, [108:02] take on all kinds [108:03] of responsibilities, and teach. [108:03] The environment compels you to keep learning. [108:06] If you don't, you can't [108:07] keep up. [108:08] In industry, however, [108:10] you may settle into a job [108:11] without making a conscious effort to learn. [108:13] You can remain comfortable for two years, [108:15] but what happens after that? [108:16] That's the problem. [108:17] Right. [108:19] Some say academia has better long-term prospects, [108:20] while industry pays more early on. [108:22] People use all kinds of criteria, [108:24] but to me, those distinctions [108:26] are simply [108:27] too vague. [108:29] After our conversation, [108:30] one point stands out clearly: [108:33] what matters is the person. [108:34] Exactly. [108:35] What matters is you. [108:36] As long as you can keep learning, [108:38] whether you're in academia, [108:39] in industry, [108:39] or anywhere else, [108:41] you're likely to do well. [108:42] position, yes, [108:44] yes. [108:44] I feel that, in [108:45] the end, a [108:47] person, for me, is essentially this. I think [108:48] a high-quality person always has a way, always moves [108:50] forward. I really believe [108:52] that one [108:53] of the reasons for [108:55] this person is moving [108:56] forward. [108:56] I don't really believe in [108:58] AI, but I think people should be believed. [109:00] I've been doing this podcast for a while, and I also [109:02] feel the same way. [109:03] It's true that [109:03] you have to always [109:05] believe in people. So the next question is, [109:06] in the process you just described, everyone [109:08] can also tell that you [109:09] pay special attention to people, especially [109:11] to the connections between people, and are particularly [109:14] willing to guide a student [109:15] hand in hand to understand their [109:18] essence and lead them as a [109:21] professor. So if we are facing this... [109:23] With so many students in the [109:26] audience, you definitely can't guide them [109:29] hand by hand. If you were willing [109:30] how they can develop [109:32] into strong PhD researchers— [109:36] how they can train themselves— [109:37] what is the most practical [109:39] advice you would give them? [109:40] Well, I think all advice is— [109:43] all advice is bad advice. [109:46] How should I put it? [109:46] Three things matter most [109:48] in research, especially during a PhD. [109:49] To do well, [109:50] you should find every way you can [109:51] to get those three things right. [109:53] First, find [109:54] a problem you truly care about. [109:55] That's the first thing. [109:57] When you care deeply about a problem, [109:58] the work doesn't feel exhausting. [109:59] Research is hard, [110:00] and only genuine interest can sustain you. [110:03] When you wake up each morning [110:06] and brush your teeth, [110:06] look at yourself in the mirror. [110:07] Are you excited [110:09] to keep working on the problem? [110:10] Or are you thinking, “Oh no, [110:11] I have to go back to the office [110:12] and work on this again”? [110:13] So first, find [110:14] a problem you love. [110:15] That is genuinely difficult. [110:17] It may take one, two, or even three years. [110:19] But if you keep looking, [110:21] you will eventually discover [110:23] what kind of work you care about. [110:24] Second, [110:25] it is crucial to develop [110:27] a sound approach to research. [110:29] Research [110:30] is not the same as engineering. [110:31] Often, [110:34] there is no shortcut. [110:36] What matters is finding the right method. [110:38] Speed itself isn't important. [110:40] You can make up for speed with time. [110:42] But with the right method, [110:43] if your approach is sound, [110:45] you will eventually get there. [110:46] Third, keep [110:47] sharpening your skill set. [110:49] Once you have a problem to pursue, [110:50] you can see which skills are useful [110:52] and which are not. [110:54] Your time and energy are limited. [110:55] You cannot do everything. [110:56] Once you know what you need to do, [110:57] you can deliberately develop [110:59] the skills required to do it well. [111:01] Get those three things right, [111:02] and I believe you can succeed. [111:03] Systems research isn't pure theory. [111:05] It doesn't require [111:06] extraordinary genius. [111:07] Systems research is not especially dependent on raw talent. [111:09] If you get those three things right, [111:12] you can do good work. [111:13] I believe anyone can do systems research. [111:16] You still call that bad advice? [111:18] That sounds like perfect advice to me. [111:20] Very much so. [111:20] I just think everyone already knows these things, [111:22] but putting them into practice is difficult. [111:24] Perhaps it sounds a little abstract? [111:27] Yes, very abstract. [111:28] It's a cliché, really. [111:30] The same is true of many things. [111:31] For example, earlier [111:33] we discussed starting a company, [111:35] or writing a paper. [111:38] How do you write a good paper? [111:39] How do you build a good company? [111:40] Hundreds of books have been written on both. [111:42] But reading those books [111:44] doesn't mean you can write a good paper [111:45] or [111:46] build a successful company. [111:47] You still have to [111:48] learn through practice. [111:51] That's why I think you also need [111:52] good mentors. [111:54] to say about this is that [111:55] I saw a [111:57] saying before, which [111:58] is that people may hear a [112:00] lot of correct advice, but [112:01] why do some people grow quickly [112:03] while others grow slowly. [112:04] You want to see the person who grows quickly, right? [112:07] Finding a mentor who can [112:08] provide timely feedback to him is crucial. [112:11] In the process of entrepreneurship, whether you look [112:13] at those successful entrepreneurs or [112:14] those PhD students who do very well, it all [112:18] comes down to being able to get [112:20] the most timely feedback from a very experienced mentor. [112:23] If people can't get this kind [112:25] of thing, then they need [112:27] to make an effort to [112:28] find it. That's how [112:29] I feel. And this mentor, I think he... [112:30] It doesn't necessarily have to be a very big-name person, right? [112:33] Right. [112:33] It's really someone very famous; he [112:35] is a person who is willing to [112:37] invest in you, someone who [112:40] is willing to spend time on you. I think [112:42] the more someone is like this, the [112:44] more important it is. How to get others to [112:46] bet on you and then invest in you [112:48] is another big topic. You can't [112:49] really put it into words, [112:50] you know? [112:50] It's hard for everyone to see. [112:51] Yes. [112:52] This is a topic that requires lifelong learning, and it [112:54] goes back to what we just talked [112:55] about about lifelong [112:56] learning and [112:57] continuous growth. Without this energy, [112:58] if you yourself don't have this energy. [113:01] Right. [113:01] You want to become a person worth investing in, [113:04] right? [113:04] Right. [113:05] This is your point, it’s [113:05] worth it to you, it’s worth other people [113:08] spending time. Worth other people [113:09] spending time. He doesn’t invest just by [113:11] saying please. The kind of [113:12] thing where you [113:12] can guide me, it won’t happen. [113:13] It will never happen. [113:14] Yes. [113:15] Only by constantly striving and improving yourself, [113:17] this is [113:18] good, good, good, very fantasy, all nonsense, [113:20] whether the chicken or the egg [113:21] came first, this matter is indeed [113:22] correct, [113:23] but maybe when [113:24] I reach the 100th episode later I will tell everyone [113:26] how to do this, but for now I have [113:29] only done a few episodes so I [113:30] don't dare to [113:32] say. And there is another question. [113:37] If students are willing to start a business, you [113:38] would support [113:39] them, and previously your mentor was [113:40] a very successful entrepreneur, and you [113:42] have some very successful cases around you. [113:44] Right. [113:44] So what if this student tells you from the beginning, [113:46] 'OK, professor, after I finish this PhD, I want to become an entrepreneur'? How [113:49] would you guide him? [113:51] Then I would definitely tell this student [113:53] that I have never started a [113:54] company, so I probably can't help [113:55] him much in terms of entrepreneurship, right? [113:57] So I currently have [114:00] a student who really wants [114:02] to start a business, and my [114:03] suggestion to him was that [114:05] he might need some other connections or some other advice, so he also got [114:07] some, well, some [114:08] help, I think, but how [114:10] should I put it? [114:12] I just feel that there are [114:13] two types of entrepreneurship. One is where you start a business [114:15] based on your own research. [114:16] For example, you develop a particularly [114:18] interesting technique and then use this technique to [114:20] start a business. [114:21] In the process of developing this technique, I think [114:22] it can still be somewhat helpful. Of [114:25] course, [114:25] if you say that your startup is [114:27] completely based on taking an existing technique to create a company. [114:29] Then I might not be a very good advisor, [114:30] I think [114:31] that's the [114:32] Building a company around [114:34] an existing technology [114:35] or a new technical innovation [114:37] is certainly one path. [114:38] Many companies that emerged from Google [114:40] started that way. [114:42] But I once heard your [114:43] advisor caution against it. [114:45] He said you should avoid [114:47] taking a hammer [114:48] and looking for a nail. [114:49] It's usually better to start [114:51] with a real business need. [114:52] That's another topic, though. [114:53] Everyone has a different view, [114:55] and there are certainly [114:56] both successes and failures. [114:57] Right. [114:57] Now for the third question. [115:00] Suppose a student wants to enter industry [115:03] and knows from the outset [115:04] that they want to run large-scale systems [115:06] and build practical systems. [115:08] How would you advise [115:09] that student? [115:09] Well, I think [115:10] academia and industry [115:12] require different skill sets. [115:15] That said, I don't think academia [115:17] and industry are fundamentally different. [115:18] Doing creative work is rewarding in either one. [115:21] What matters, ultimately, is whether [115:23] the place you choose gives you an environment [115:25] that helps you [115:26] do the work you want to do. [115:27] That place might be academia [115:28] or it might be industry. [115:30] But the two paths [115:32] do require different skill sets. [115:33] Some things matter little in academia [115:35] but are crucial in industry. [115:37] If a student knows from the beginning [115:39] that they want to enter industry, [115:41] the most important thing is to identify [115:43] the skills [115:44] that industry requires. [115:45] Then, during the PhD [115:47] or other graduate study, [115:49] they should deliberately develop those skills [115:52] and become [115:53] a more competitive candidate. [115:55] When it comes to publications, [115:56] industry doesn't care much [115:58] about [115:58] how many papers you've published. [115:59] One paper may be enough to graduate. [116:00] Spend the rest of your time coding, [116:03] learning system design, [116:04] and exploring [116:05] the latest [116:05] frontier ideas. [116:07] Right. [116:08] AI, for example. [116:09] If you want an academic career, [116:10] you'll need to build [116:11] a publication record [116:12] and focus on that. [116:14] Build it up. [116:14] Exactly. [116:16] We've talked with Tianyin for more than two hours today, [116:19] and it's clear that he is [116:21] a truly exceptional systems researcher. [116:23] There's a saying in Silicon Valley: [116:26] when someone starts a company, [116:27] there needs to be founder–problem fit. [116:30] The right person is suited to the right problem. [116:32] That comes from something deep within them. [116:34] After these past two hours, [116:36] I can clearly see that in you. [116:38] I've met many systems researchers, [116:39] and I work in systems research myself, [116:40] though I'm only average. [116:42] But you really are [116:44] a natural systems researcher. [116:47] I don't believe anyone is born that way. [116:48] You just need to enjoy the work. [116:49] Research is a wonderful thing. [116:51] I think everyone should try it, [116:52] especially now that the barriers are lower. [116:54] Research used to feel like [116:56] something confined to an ivory tower, [116:57] something only [116:59] a small number of people could do. [116:59] But with AI [117:00] and modern technology, [117:02] research should become [117:04] a daily practice. [117:05] At its heart, it's a process of reflection. [117:07] So I think everyone [117:08] should give it a try. [117:10] All right. Thank you, Tianyin. [117:12] I hope that after hearing this, everyone [117:14] will do a little research every day. [117:16] Exactly. [117:17] It can make you more effective at work. [117:18] Yes. [117:19] And it can strengthen all kinds of skills. [117:20] Absolutely. [117:21] You can research [117:22] something as everyday as [117:23] how to build your abs [117:24] or how to get in shape. [117:25] That's research too, isn't it? [117:27] It's an iterative process of trial and error. [117:28] Exactly. [117:29] Thanks for the suggestion. [117:30] All right. [117:30] Thank you.