AI 的演进也完美契合了这一过程。早期的 AI 只会机械地模仿人类给出的标准答案；随后的机器学习让机器能够从海量数据中自动发现模式（例如通过数千张猫的照片学会分辨猫和狗）；而真正让机器学会独立思考和决策的，正是强化学习。强化学习为机器构建了“行动与反馈”的闭环，机器不再是被动地被告知答案，而是通过自主行动获得结果，并根据奖励或惩罚调整策略——“做对了就得到奖励，做错了下次换种方法”。这一机制触及了人类学习的本质，标志着 AI 从“被教会”走向了“自己领悟”的飞跃。
OpenAI 的 Zong Wang 则认为，可验证奖励（RVR）可以被视为一种更通用的“先生成、后验证”范式，因为验证一个答案往往比生成一个答案要容易得多。他强调，这种方法不仅能用于微调，还能用于培养更强大的推理能力，甚至可以应用于基于对抗辩论（Debate）或自我博弈（Self-play）的强化学习框架中。
LinkedIn 的 Al Wurzke-Armifar 补充道，强化学习的魅力在于它能让智能体超越导师的水平，而不仅是重复导师教过的东西。在实际应用中，任务往往需要可验证与不可验证奖励的结合。例如，在写代码时，代码能否成功编译是“可验证”的；但如果要写一段总结，或者提出一个创新性的想法，如何定义“好”与“创新”就变得非常模糊。此时，业界开始转向利用大语言模型（LLM）作为裁判（LLM-as-a-judge），在一定程度上替代人类来评估这些主观性极强的任务。
[24:59](https://www.bilibili.com/video/BV1EtypBBEPF?t=1499s) Process-Based Rewards and the Challenge of Dense Signals
The panel shifts to a spontaneous question regarding hierarchical reinforcement learning (RL) and the training of Large Language Models (LLMs). Specifically, they address the difficulty of deriving and implementing process-based rewards (or intermediate checkpoints) rather than relying solely on a single outcome-based reward at the end of a long sequence of, say, 10,000 tokens.
Li-Hang Liu expresses optimism about this direction, noting that while outcome-based rewards have proven highly effective so far, complex problems and long-sequence agentic workflows benefit greatly from intermediate signals. He suggests that process rewards do not even need to be permanent; they can serve as a curriculum training technique. "We use intermediate reward at the beginning to guide a solution, to guide exploration, and then eventually you take away the intermediate reward and only let the model learn from the final reward."
Adding to this, another panelist shares insights from a recent paper they published on arXiv. They explain that the necessity of process-based rewards depends heavily on the size of the starting model:
* **Larger Models:** Possess more inherent power and can find a workable, verifiable solution at the end of a sequence on their own. "If your model cannot see that good cookie at the end at all, then you can't learn anything."
* **Smaller Models:** Struggle to reach the end successfully without guidance. Providing smaller models with an enriched, dense reward function helps them navigate to a successful outcome.
* **Compute Efficiency:** Properly shaping the reward function for smaller models can yield a three-to-four-fold saving in compute to achieve the same outcome, which is a massive advantage in a resource-constrained environment.
[27:45](https://www.bilibili.com/video/BV1EtypBBEPF?t=1665s) Can Reinforcement Learning Generate New Knowledge?
The discussion transitions to the future of RL and its potential to push AI capabilities from simple chatbots to innovators. The host asks whether RL can eventually generate entirely new knowledge and act as an innovator through pure reasoning, or if it will always require human-annotated data.
Li-Hang points out that RL has already successfully discovered new mathematical facts by exploring mathematical spaces. However, he cautions that the ceiling of RL is fundamentally bound by the quality of the simulator or the environment model. If the simulator is built strictly on existing human knowledge and rules, the agent can only explore within that framework. "It won't be able to tell you to create new physics theory, right, or new discipline." To go beyond that, something fundamentally different is required.
Zheng agrees, stating that the paradigm's success depends entirely on the accuracy of the dynamics and reward models. In fields where accurate simulation is too costly or impossible, RL may not be the best path forward, and researchers will have to rely on available datasets or alternative paradigms.
Offering a counter-perspective, another panelist brings up AlphaGo's famous "Move 37" as historical proof that AI can discover things beyond human understanding. In the 3,000-year history of Go, humans initially flagged the move as a mistake, but it turned out to be the winning play. They argue that if we can build highly accurate simulations of physics, dynamics, or materials, and pair them with rich reward functions, RL agents can absolutely go beyond human capabilities and discover new scientific frontiers.
The host poses an open, creative question: Is it possible to train RL systems without relying on binary or scalar numerical rewards? For instance, could probability distributions of words at the token level be leveraged as guidance instead of a final scalar reward?
One panelist invokes the philosophy of reinforcement learning pioneer Rich Sutton. "From Rich's perspective, everything has to come down to a singular numerical reward function." They recount a podcast where Sutton was asked how complex human lives—filled with competing desires for family, career, hobbies, and fun—can be reduced to a single reward. Sutton's belief is that there is indeed a single maximizing signal, and all complex behaviors naturally emerge from trying to optimize it. However, the panelist suggests that we can use human preferences (e.g., pairwise comparisons of which summary is better) to walk backward and mathematically extract these scalar rewards, as humans are much better at comparing two options than assigning an absolute numerical score like "0.7".
Zheng adds that even if we design systems where rewards are vectors or probability distributions, the current paradigm still requires converting those complex signals back into a scalar to perform optimization. He notes one major exception:
* **Multi-Agent Reinforcement Learning:** In multi-agent settings, each agent has its own distinct reward, making the overall reward a vector. This introduces concepts like Nash equilibrium, which differ from standard single-objective optimization.
[34:20](https://www.bilibili.com/video/BV1EtypBBEPF?t=2060s) Near-Term Advancements in RL
As the panel wraps up, the host asks each guest to share where they believe RL will deliver the most significant near-term advancements over the next year—whether in personalization, scaling, or general AI system capabilities.
One panelist concludes by emphasizing their passion for abstraction. "Since I spoke a lot, I think the abstraction is the way to go. I'm very passionate about it. I think it's going to give us a lot of goodies coming."
> [24:59](https://www.bilibili.com/video/BV1EtypBBEPF?t=1499s) We use intermediate reward at the beginning to guide a solution, to guide exploration, and then eventually you take away the intermediate reward and only let the model learn from the final reward. — Li-Hang Liu
> [24:59](https://www.bilibili.com/video/BV1EtypBBEPF?t=1499s) If your model cannot see that good cookie at the end at all, then you can't learn anything. — Another panelist
> [27:45](https://www.bilibili.com/video/BV1EtypBBEPF?t=1665s) It won't be able to tell you to create new physics theory, right, or new discipline. — Li-Hang
> [31:45](https://www.bilibili.com/video/BV1EtypBBEPF?t=1905s) From Rich's perspective, everything has to come down to a singular numerical reward function. — One panelist
> [34:20](https://www.bilibili.com/video/BV1EtypBBEPF?t=2060s) Since I spoke a lot, I think the abstraction is the way to go. I'm very passionate about it. I think it's going to give us a lot of goodies coming. — One panelist
## Mentioned
AlphaGo, DeepMind, Andrej Karpathy, OpenAI, 硅谷101, Alignment 2025 峰会, 朱哲清, Pokee.ai, Meta, 李航, 亚马逊, Alborz Geramifard, LinkedIn, Zheng Wen, 斯坦福大学, Zong Wang, Al Wurzke-Armifar, Scale AI, Surge AI, Richard Sutton, Oak, Li-Hang Liu
[0:00] 做对了就得到奖励，做错了下次换种方法
[4:59] 李航: 当选 A、B、C、D 即可验证；但当我们步入社会开始工作，面对的往往是复杂的商业决策和高难度的判断，没有一个现成的对勾可以去勾选
[9:59] Al Wurzke-Armifar: 把钱花在刀刃上
[9:59] Zong Wang: 应该在什么时候、向哪位专家、提问什么样的问题
[14:59] 李航: 跳出框框思考
[14:59] 先打车去机场，再坐飞机，最后入住酒店
[24:59] Li-Hang Liu: We use intermediate reward at the beginning to guide a solution, to guide exploration, and then eventually you take away the intermediate reward and only let the model learn from the final reward.
[24:59] Another panelist: If your model cannot see that good cookie at the end at all, then you can't learn anything.
[27:45] Li-Hang: It won't be able to tell you to create new physics theory, right, or new discipline.
[31:45] One panelist: From Rich's perspective, everything has to come down to a singular numerical reward function.
[34:20] One panelist: Since I spoke a lot, I think the abstraction is the way to go. I'm very passionate about it. I think it's going to give us a lot of goodies coming.
Original transcript
[0:00] 大家或许还记得2016年的那场震惊世界的对异DeepMind的AlphaGo成为了第一个击败为其世界冠军的人工智能它的成长过程堪称AI史上一个转折点它先学习人类的棋谱掌握基本下法接着开始自我对异每天和自己下上百万盘棋不断根据输赢信号优化策略最终它发现了人类从未想过的走法成为了第一个击败世界冠军的AI如今从推荐算法到自动驾驶从工业控制到智能体训练强化学习都在悄悄成为AI进步背后的第二引擎当大力初起技时的AI发展范围强化学习成为了各大AI巨头接下来的救命稻草被认为是接下来人工智能到达AGI的关键技术然而最近在AI业界也是不少人开始质疑强化学习的路径就包括了OpenAI的联合城市成员Andre Kampathy在最近接受播客采访的时候公开表示他认为我们距离AGI的实现仍然有十年之遥强化学习虽然优于此前的技术但是本身效率低下且充满缺陷所以研究强化学习路线的AI科学家们是怎么认为的他们正在研究什么样的技术强化学习领域目前最艰难的挑战又是什么呢在今年的10月5号硅谷101举办的Alignment 2025峰会上我们就非常荣幸的邀请到了四位分别在OpenAI、亚马逊、Chemeta以及LinkedIn的强化学习领域的重量级嘉宾与我们一同扑析这一前沿领域的最新进展与未来的发展方向那因为这场在硅谷的大会是全英文的,我们将内容翻译成中文分享给大家。那这场panel呢,可以说是真的非常难得,是强化学习领域最顶级的panel之一了,希望对大家有所帮助和启发。Enjoy!That's alright.Alright, hope you guys have a great morning so far.So I'm on today to have a wonderful panel with me today.in fact all of them are my friends to some extent like for the last like many yearsI've known each of them for quite a few years so farso it would be great to have this opportunity to also reunion with all of my friends hereso we're going to focus on RL todayand we all know that reinforcement is core to today's AI advancementsfrom earlier this year of DeepSeek's release of R1 where they have reinforcement fine-tuning to really have some sort of self-improvement on coding and math to later this year where we see a lot of applications on math, coding, chemistry, physics, in-body AI type of applications on reinforced learning that enable like really strong intelligence. And we've seen a lot of are our companies that raise a lot of money as well. So we're going to dive into all of that today. But before that, I'd love to have the panel introduce themselves. Let me start from Lee Hong. Yeah. Sure. Yeah. Hi, everyone. My name's Lee Hong from Amazon. I work on reinforcement learning and language models for a while. And it's a pleasure to join the panel. And thanks to the organizer for inviting me here. Chen. Hi. My name is Zong Wang. So I'm currently working on OpenAI, and before that, I was at DeepMind. So I also work on reinforcement learning, and actually, I'm a big fan of Li Hang's paper, and also working on large language models and AGI in general. I'm very happy to join the panel today. Good morning, everyone. I'm Al Wurzke-Armifar. I'm a distinguished scientist at LinkedIn, mostly focusing on HNT RL. Before coming to LinkedIn, I was at Meta, where I was actually working with Bill closely, and before that I was in Amazon Alexa.那如果我们从头看,人类的学习其实经历了三个阶段,那最早的我们通过模仿来学习,看到别人怎么做就照着做。那后来呢我们开始总结规律知道什么样的做法更加容易成功而真正的智慧来自于第三个阶段也就是我们开始自己去试人和智能的学习之路呢其实也是这样的最早呢AI只会模仿它靠大量粒子去复现人类的答案之后呢机器学习让它能够从数据里面自己发现模式比如说呢看到几千张只猫的照片就能够分辨出猫和狗但是那还不算真正的会学真正让机器学会思考学会决策的是强化学习强化学习让机器有了行动与反馈的回路它不是被告知答案而是通过行动获得结果再根据此来调整做对了就得到奖励那么做错了下次换种方法那么听起来很简单但是这就是人类学习的本质所以我们可以说强化学习是AI从来
[4:59] 被教会到自己领悟的那一步。 到那一步之后呢,机器不只是零线实际,而是开始在这个世界里学习如何行动。 That's great. Cool. So let's dive right in. I think people, if you guys know the recent advancement in reinforced learning, one thing that has been like triggering a lot of interest so far is how can we train RO agents in language models without actual supervised data? That is that there's no human labeling, there's no human expert annotated data, and how do you train these models? And one of the trends is ROVR, reinforcement from verifiable rewards. The idea is that you can literally use rules and sort of like verifier systems from math to really verify whether or not the outputs of certain language models are correct. So the first question is, there's a lot of papers coming out recently on RRVR. And then this is sort of the almost like a consensus on how these things are going. So where do you guys think RRVR can go before beyond just current reinforcement fine tuning on math and coding Yeah I think it definitely has a lot of great impacts in FARTS But on the other hand, we also see that many problems are not verifiable. They don't have verifiable outcomes. So I think that at some point there is a need to go beyond that to handle non-verifiable problems. An example is that when we are children, we are learning in school, we have tests that are verifiable. They have verifiable results, right? A, B, C, D, choose one of them. But then when we actually go out to work, and it's hard to say that you have to do this in order to check the mark, right? There's a lot of high judgment call and non-verifiable outcomes that we have to explore, and so that's a much fuzzier place. So I think that's where we need to do more beyond verifiable rewards. Yeah, I think for reinforcement learning is a verified reward. My understanding is that this approach can be used in a very general paradigm, which is you generate first and then verify. And the reason is that usually it's much easier to verify than generate. I mean, if you think about whether or not it can go beyond fan tuning I think it can And actually there are already some work on that For example you can use this approach to develop stronger reasoning skills And you can also use it in somewhat different framework compared to fine tuning like I mean reinforcement learning based on debate or reinforcement learning based on self I think all of these go beyond fine tuning I agree with all the points raised. I think reinforcement learning kind of gives you the capability to get us to go beyond of what a tutor or a mentor basically shows you and says, hey, this is what I want you to maximize. And I think we are there on the crusp of how do we define this goodness or the cookie for the agent. And verifiable rewards are definitely like a way to go because it's very crystal clear how to define them. But many of the tasks, I think, requires a combination of both. There are things that we know that has to happen. If you're compiling a code, we know that the code should be built successfully. But there are things that we actually don't know exactly how to define them. Like if you're writing a summary, or if you're providing an innovative idea, how innovative it is, or how good is your summary? And I think that where we need to get inspiration from humans essentially to guide us to how to shape this reward function And there is a lot of litany of work of starting from comparison and alignment DPO all of those algorithms all the way to now getting richer richer, using LLM to become essentially a judge and fill the shoes of humans for those cases. Yeah, I think we're alluding to the next question, basically, where reward design itself might not just be completely verifiable. It might be designing a mixture of verifiable reward with human feedback and alignment type of rewards. But one of the things that is puzzling a lot of researchers and also production people is that, like, what is the golden standard here? Because if you have verifiable reward, basically you don't need data. But then if you have RHF type of information where you have human feedback as one of the judgment, then you actually would require a lot of data for human feedback. And then for very complex tasks like trying to solve a level like sort of chemistry Olympics or physics Olympics type of questions, if you guys don't know, a single data annotation on a response from a language model takes about $20 from a PhD. So where do we cut? How much data do we need? Is there a way that we can?
[9:59] 那到这里我先介入一下那其实Bill提到的这个问题为何还要隐藏着一个更现实的挑战就是我们到底在用谁的数据来训练这些系统那过去几年我们看到像Skill AI Search AI Merkur这样的公司都在为这些人类反馈提供标注数据但是行业正在发生明显的转变人们开始意识到AI不再需要廉价重复的人工标注而是需要专家集的数据来自博士研究员或者真正懂业务的专业人士那比如说Marker的做法就很典型他们不再雇佣大规模的外包团队而是邀请澳塞的选手博士生工程师来给模型打分那么这样做的成本确实是更高的平均报酬是大约每小时95美元而一些传统标注公司的报酬约是每小时30美元但是呢 这样带来的信号质量模型稳定性却远超传统外包式的数据那么这也解释了Bill所说的黄金标准难题在未来的强化学习当中啊我们也许不再追求更多的数据而是追求更好的判断而判断的价值往往呢 是来自于动行的人那么接下来呢,我们就再来听一听几位嘉宾怎么看从可验证奖励到人类反馈这条路该怎么走I'm sure, I think we need bothIn reality, we want to be cost awareSo humans judgments are expensiveAs Bill mentioned as an example20 dollars per sampleand you need like gobs of data that would be like you know making your bank account breakso we want to make sure that you are using it judiciously but we can't just also ignore thatbecause for many of the tasks we actually don't know how to create this reward function soit's more like a think of it as an active learning situation so you can create essentially amechanism to gauge your llm judges so you can get like let's say four of us here decide about like是小组的大组的组合,如何我们其实去处理和设计和设计如果我们都应该设计,如果我们都应该设计,我们就要去看LM Judges看看他们的组合,我们应该是否我们的设计。如果他们应该,我们应该是一个宜负,现在的性格,不过的性格,我们应该能够用的数据,然后我们可以接近了远远的。如果有些域,我们可以去看,我们应该can occasionally go and see, are you doing the right thing? And if it is some holes over there, we have to go back and say, hey, let's collect some data in this area, make it better, and then go back and apply it. Yeah, I fully agree. So I think there might be two ideas to handle this. Firstly, you said we might try to decompose the problem a little bit. Because even for a whole problem, maybe it's not verifiable. but maybe we can decompose it into some subtasks. And for some of the subtasks of some of the steps, it's verifiable. So that basically can already partially solve the problem and make the problem easy. And another solution is obviously is to keep human in the loop. And also that's why we do reinforcement learning from human feedback. I mean, I think that if in this case, exploration is very critical. The reason is that because it's so costly, so you want to identify the right expert to ask the right question. So this clearly requires a very effective exploration algorithm. Yeah, these are the great ideas, and I just wanted to add that there is another, maybe another possibility of looking at the stream and experience of the model, right? Where the model can rely on its own capabilities or external model's capabilities to extract signals from this experience, stream of experience and dual learning, right? And then at the end of the day, I think there's something that's maybe too obvious to say is that the capability of the power of RL is limited by how good a simulator is, how good environment you use to train the system so I think depends on how all these are great ideas but I think at the end of the day we need to make trade-off on how to make sure the environment is of high quality那么接下来呢我们会聊到强化学习中的探索这个主题那这里解释一下在强化学习RIL当中呢Agent通过与环境交互来学习 and the goal of the goal is to make the most important and long-term reward. This process is a common goal of two main goals. The first is Exploitation, which means using the knowledge of the knowledge, to be able to get the most highest reward. The second is Exploration, which means explore new and not fully understand the action.
[14:59] 关于环境的知识而这两者之间的平衡被称为是Exploration Exploitation Trade-off而刚才OpenAI的正文提到的探索Exploration在强化学习当中其实指的就是智能体主动的去尝试新的行动或者策略以发现潜在更优其果的过程而不仅仅是利用已有的经验那么在嘉宾提到的场景当中因为与专家交互的成本非常高那么探索实际上意味着要高效地决定问谁,问什么问题因此呢,一个有效的探索算法并不是盲目视错而是在代价与信息增益之间找到最优的平衡 I might want to actually dive into it a little bit because we mentioned a couple of really good ideas one around exploration the other amount like rewards that's like a mixture of different types of reward and subproblem like breakdowns there's a bunch of ideas that surface I want to like touch on exploration first and elaborate on that a little bit因为 I think everyone on this panel actually worked on exploration at some point in your career. This is one of the topics that a lot of RO people are today not focusing on enough. Because you see the PPO algorithm, GRPO algorithm, DPO algorithm, they're fine-tuning an existing model that has already been trained in a collapsing modality. And then exploration is really needed when you actually have little information about the environment and trying to explore and understand what's the most important part of the equation. So the question for Jen is, where do you think is the balance between pre-training and post-training where you actually really want to induce strong exploration behavior? I think we want to make sure that when you're doing the pre-training, you have a generally good model across a wide set of tasks. If you start seeing that you're getting better in some of these tasks but worse in some other tasks, you're doing too far. And I think that's the part that you have to delegate that to the post-training, essentially, stage. The main, from my perspective, the main role of the pre-training is to have a model that has a good understanding of fundamentals of the basics and the key ingredients of how to solve these problems in a higher level, and then have the post-training you can fine-tune on a specific task that you actually care about it. And I think that's the breaking point to me. It's like you can look at your evaluation across the board and see if you are start regressing on some of these models in these domains compared to the other ones. Yeah, I fully agree. I think that usually pre-training is a step where you acquire the general knowledge. So you want your model to be expressive and roughly understand the world. And post-training is you want to adapt it so that you can become an expert in some field. I also want to mention that I think the exploration is crucial in cases where the data acquisition is very costly. Because here you cannot really acquire a lot of data. So you need to want to do an exploration to make sure that you acquire informative data. It's also important in cases that not every human or every reader can give you some useful feedback. So in that case, you also need to explore to identify an expert from which you can get the useful and informative feedback. For exploration, I think that today's exploration is very different from what we used to talk about exploration. And when pre-training is done, I think the space of exploration is already very limited. I think the goal there is just to make sure the model has a good starting point and to get a good trade-off between how good you chain and how fast you can chain a model in post-chaining. At least that's my understanding. But I think for real more aggressive exploration, it may be interesting to look at beyond the token level or something that has more high level and do exploration there and then combine that with token level exploration. I think that may allow the model to think out of the box or maybe more aggressively in the post-training process. yeah that's great i think that alludes to both questions of like higher level abstraction and also hierarchical planning um so if you guys haven't heard rick sutton's latest talk about the oak architecture it emphasizes a lot about an hierarchical structure of options and then also sort of like lower level planning on top of that so the idea is that you can actually break down very complex tasks into like sub-problems, and then each sub-problem can be solved by some level planning inside a sort of work model, and then outside you actually can sort of learn by experience. That to be a simple example,
[19:59] 又提到的Oak架构就来自于Richard Shadden他被誉为强化学习之父Shadden奠定了现代强化学习的核心理论比如说时间差分学习TD Learning和价值函数Value Function2024年他还获得了图灵奖那么今年他是提出了Oak架构全称是Options and Knowledge正如Bill所说这个架构非常强调选项的分层结构以及在此基础上的低层次规划那么通俗的来说就是把复杂任务拆解成为子任务每个子任务内部呢都有自己的规划机制而整个系统呢又能够通过不断的积累经验来整体变得更聪明那举个例子啊就像人类规划旅行你先设定目标去巴黎再西化为订机票打包订酒店等小计划每一步呢都是一个子任务那么完成之后呢又能总结经验让下次的旅行更顺利那这就是加弹所设想的智能体学习方式通过在计划中学习计划,让智能从经验中自然成长。 So here, I want to pose two questions, and then any one of you guys can answer any one of them, if you wish. So one is, on the abstraction and representation level, how much should we go beyond To a whole level if we want to really generate longer kind of context intelligence? And the second, how much is it important to have an hierarchical structure in reasoning on both questions? Yeah. Yeah, I do think that it's very important that eventually we look at hierarchical reinforcement learning in AI. The reason is currently, I mean, most of the signals are just a token level. And it's so low level. And this actually causes a lot of problems. I mean, for example, this causes a lot of problems of credit assignment and also data efficiency. So I do think this is very crucial that eventually we go to a higher level, go to a higher abstraction to look at something like subtasks, plans, modules, and functions, and do planning there. I think that if we eventually want to look at a very long horizon, I do think this is very crucial. And I also believe that if we do it correctly, it can significantly improve data efficiency and also better solve the credit assignment problem. Yeah, so I also agree that we need something like that, a high level and combined with the token level. There may be a distinction, though, in the question, that when we talk about the inference time execution, at the end of the day, when the model produced, the output is still at the token level one by one, at least in the current framework. But I guess what I was trying to say to express is that this high level thinking or high level training may be needed in the post training stage where we set up the reward model, we set up the problem. And that's how we get a signal to the model to learn at the high level how to jump from local minimum to another better landscape of the policy space, as opposed to doing this narrow or local exploration at token level. But at the end of the day, in the inference time, I think it's still, probably we're still looking at token by token level. So I'm going to...I actually like the direction that this is going, but I want to take it even further. So we as humans, when we want to plan to go to Japan, for example, from here, we don't think about the little muscle twitches that we have to make in order to make that happen, which is like, go through the door and then take a cab and then go to the, you know, take a plane. We actually think on a higher level of abstraction. We think about, okay, I need to get a cab from here, then take the plane, and then go to the hotel. And then you start bringing those back into a lower level decision. It's like, okay, if I want to take a cab, how do I do it? I call an Uber or a Lyft or something like that. And I think this is what I'm thinking currently LLMs are missing. You're thinking in the token level still, which is the muscle twitches of humans. And I think as you start going into bigger and bigger problem you want to solve, you need that level of abstraction in order to let you plan that far down the road. And the one way of doing it, which what Li Hang just mentioned, is like when you want to do the backups, you actually think about further away decision outcomes and then bring them back in rather than waiting for this to trickle through all the tokens. But one level higher than that is you actually have a policy actually thinks in a higher level because we think that way. So we can have a policy to think, it's like what are the chain of high level things that I have to do in order to get to that outcome? Like I have to do, like I have to call this tool and then get the results filtered by that thing and then do something else in order to get to the reasoning that I want. And I think that's the way, currently there is not much work that I have seen out there but I think that would be the direction to actually unlock the whole new
[24:59] of reasoning available to these LLMs. That's great. I want to add one question I didn't send you guys before. It's exactly on this line where... So when we're talking about hierarchical RL and then also especially when you're training LLMs, one of the things that people have tried a lot in training LLMs is to add process-based reward. And then if you want to do hierarchical RL to some extent, you have to have process-based reward. After you finish the whole 10,000 tokens, you have long sequence, before you end, there will be checkpoints in the middle. Like, OK, after I finish this task, here's the reward. Next task, another reward, and so on and so forth. What do you guys think, like, it is really hard for us to really sort of derive some sort of process reward when we're training LLMs in the next, like, few years? Yeah, I do think this is a very promising direction. I mean, definitely there are some challenges, but overall, I'm optimistic about that. Yeah, I think that's an interesting, promising idea to try. I think at some point, although right now, I think outcome-based reward seems to go a long way, but I can imagine that there some complex problems We need some intermediate signals Even for now I think when you call a long sequence of agentic calls I think some recent work is finding that it beneficial to have some intermediate signals And also another point I want to make is that this process reward or intermediate reward, they don't have to find a reward. It can be used as a solution or a technique as part of a curriculum training. We use intermediate reward at the beginning to guide a solution, to guide exploration, and then eventually you take away the intermediate reward and only let the model learn from the final reward. So I think that's another possibility. We actually recently looked at that problem very specifically. It's actually on archive if you want to read that paper, but to some extent it also depends on the model that you're starting with. bigger model often have more power. So if you give them a problem to solve, they probably would find easier to a workable solution. Not the perfect solution, but they can get to somewhere that I actually see this verifiable reward at the end. Because if your model cannot see that good cookie at the end at all, then you can't learn anything. But for the smaller models they might not actually be able to do that So the thing that Li Hang just mentioned it is much more important for a smaller model to actually have access to a more enriched dense reward so they can find their way out all the way to the end, a good end, versus as you go through a bigger model, they might not need as much as massaging in the beginning for the reward model to make it work. But the interesting finding, in fact, is one domain, but the smaller models, if you actually shape them well, you can actually get about three to four times compute saving and you get to the same outcome. So it's a balance of how hard you can actually work on your work function to actually make it informative enough for the models to be able to tackle the problem. But you rip the benefits and GPU costs, which now everyone has limited resources of. Yeah, and that also brings importance in exploration, right? If you have very sparse reward at the end, instead of just doing exploration, another way you could potentially do is probably adding more insightful intermediate rewards. All right, let's move on to a different topic. I think we talked enough about how to train RL agents on LLMs but I think we want to do a little bit more forward Where does where does RL really land us So one of the things that OpenAI has brought up many years ago probably not that many years ago, because ChatGPT wasn't out for that many years, is the level of intelligence from, you know, just a chatbot to, you know, a reasoner to an innovator and eventually become an organization. I think to some extent, it's a product play, but to some extent as well, it is sort of showcasing the capabilities of LLMs and large models in terms of their sort of like capabilities scientifically. So the question to the panel is like, can the route of RL eventually give us the possibility of generating new knowledge and being actually an innovator through pure RL reasoning? Or does that require a lot of additional data that's human annotated for this to happen? I think we've tried all orders, so we're going to go back to the beginning. Li-Hu, maybe you can start. LI-HU LIU- Sure. Great question. Yeah, I think it's possible for RL to create new knowledge without human labor. And then as we have seen that I think there is a success story where you let the RL
[29:59] system or model to explore in math, you can find new mathematical facts. I think these are examples. But I just want to go back to an earlier point, making that the ceiling of RL depends on how good the simulator is, or how good the model is. If the model we provide in our system is in the form of the rules or just the knowledge we know about human know, I think at the end of the day, the RL can only explore wrong knowledge or new knowledge in this framework. It won't be able to tell you to create new physics theory, right, or new discipline. I think in order to go beyond that, I think we need something different, something fundamentally different, probably. Yeah, I do agree. I feel like, I mean, as Li-Hang has just mentioned, so I mean, I always clearly depends on the model of the environment. I mean, which most specifically depends on the model of the dynamics, the model of the reward. So I think that for the means of fields where we have relatively accurate environment model, and also suppose we do the compute in a reasonably correct way I do feel that our paradigm can take us very far away But there might be some fields where either it impossible or too costly to do some very accurate simulation. I mean, in such fields, and probably, I mean, I.O. is not the best paradigm to move forward. And we need to either use some available data site of thinking another paradigm to move forward. So first I want to answer, I think AI has shown us already that it can do discovery beyond humans. If you follow AlphaGo, you know Move 37. It's like a move that the AI made that everyone thought with the 3,000 years of Go history, they thought this was a mistake. And then later on it turned out that this was actually the winning move. So it was something counterintuitive, but it worked. And it aligns with what both Li Hang and Zheng mentioned, which is if you have a good simulator, which in terms of Go it was, and also a really good reward function, eventually it can go beyond what every human is capable of doing. And I think for LLM, we are changing the domain. We are going towards understanding new physics and dynamics or material But the problem is the same If you have a really good simulation that the agent can freely explore go beyond what humans are actually providing and also really really good rich reward function I think it is very, very much possible given the past history that we have seen in this space. Yeah. Cool. So, one of the things I do want to mention is like going beyond what we were just talking on innovation is like this goes back to the fundamentals compute where verification is fundamentally easier than generation or creating solutions. So a lot of times that we're talking about verifications on these innovative systems, they are basically the simulator they want. Now, the additional question I want to add on here is, is there a possibility that we can go beyond numerical rewards? Because in a lot of situations, like what Jen said, verification systems or simulators of RO systems, they rely on a binary or numerical reward signal for us to be able to train the RO system going that direction. So is there a possibility that we can go beyond numerical reward system to be able to learn something smarter without actually annotating with numerical rewards This is a complete open question so feel free to skip if you don have an answer for this I have a question for you actually What do you mean by non rewards At the end of the day we're doing all this matrix multiplication at scale. So right so like you can think of like for example the probability of distribution of words like on the token level they're not technically reward, right? They're basically probabilities. Can you leverage those kind of probability distributions as a guidance for you to train our system instead of just using one final reward at the end as a reward signal? So I can start taking a crack at it. I'm Rich as a student. From Rich's perspective, everything has to come down to a singular numerical reward function. And there is a really interesting podcast with Rich. I highly recommend go listen to it. But other people say, hey, we as humans, we have a lot of things that we want to maximize. We have family, we have work, we have career inspiration, we have, I don't know, fun, hobbies. How all of those become one single reward function? And his answer is like, maybe there is a single thing that we want to maximize, and it just shows itself in various ways in our life. And his belief
[34:59] that for the agents is going to be the same thing. If you find that what should drive the agent, all of these amazing behaviors would actually emerge from that maximizing signal. But your question, maybe some of these can come from preference. Maybe we cannot put a point that is like, how good this is, is it's 0.7? We don't know. But if you ask enough from people, it's like, hey, is this one better than that one? We know that humans are much better in that job compared to give you like, how good is this summary? and maybe we can then walk backwards from that and for example extract that do you want to take a quick stab at it yeah i kind of agree i think there do have some settings in which we think the reward should be a vector or reward should be a probability distribution but for the current paradigm eventually i mean if that's a case the same thing you have not finished your reward model and somehow you need to convert the vector or the probability distribution into a scalar so that you can optimize. I do want to mention an exception. It in multi reinforcement learning You can think each agent has its own reward So in this case you can think the reward is a vector And also in such settings you will deal with notions like Nash equilibrium, which is a little bit different than just doing optimization. Cool. Very briefly, I think numerical rewards, I think we need that, something like that. But on top of that, I think maybe multi-dimension of numerical rewards. Cool. That goes to a lot to the generalized file function stuff as well. Yeah, I just want to throw one creative question out there so that there's actually a discussion then, pre-thought of answers. All right, so we're almost out of time. So I want to spend the last few minutes trying to ask each panelist about where do you guys think the most near-term advancement that RL can bring us, either on personalization, scaling, or just any AI system advancements you will see from RL. in the next year or so. Since I spoke a lot, I think the abstraction is the way to go. I'm very passionate about it. I think it's going to give us a lot of goodies coming.
