Stanford AI Agents Match Human Responses at 85%, Bernstein Says

Bernstein称斯坦福AI智能体与人类反应匹配度达85% 定性访谈记录使生成式模型能够以人类两周重测一致性85%的水平复现人类态度，但强制礼貌的安全对齐阻碍了智能体准确模拟社会冲突。 Michael Bernstein教授介绍了生成式AI智能体的架构与应用，将其作为一种“假设模拟机”（what-if machine），能够在现实世界的组织政策或产品落地前测试人类反应。他超越了传统模拟的僵化参数，展示了记忆流、自动化反思和分层规划如何使自主智能体得以重现类似走红的“Smallville”小镇实验中的涌现性社会行为。为弥合仅具可信度与实证准确性之间的差距，Bernstein展示了基于深度定性访谈构建的数字孪生，在标准社会学和心理学测量工具中，对比人类重测基准达到了85%的复现保真度。然而，他提出了一个四级的“模拟风险阶梯”，告诫从业者应主要将模拟用于可能性探索和想法筛选，同时提醒不要过度依赖大语言模型的单点预测、涌现性多智能体动态，或容易产生人为回避冲突倾向的安全对齐模型。 斯坦福大学研究人员证实，基于两小时深度定性访谈构建的AI智能体，在人类调查问卷反应上对比人类两周重测基线达到了85%的归一化复现比率，远超基本人口统计学提示词（70%）和随机基线猜测（33%）。 (32:26) 在一项针对五篇已发表《PNAS》社会科学论文的复现测试中，AI智能体群组成功复现了其中四项研究，并在第五项研究中准确判定为失败，这与同样未能复现原始假阳性结果的真实人类受试人群相吻合。 (34:10) “模拟风险阶梯”指出，尽管AI智能体在定性反馈和可能性探索方面表现优异，但定量估计往往会扭曲分布——例如将13%的现实世界指标骤降至1.2%——这使得模拟更适合在进行实测A/B测试前充当概念筛选工具。 (39:07) 标准的RLHF对齐使前沿大语言模型过于礼貌且规避冲突，扭曲了人类社会动态，因此需要专门针对认知心理学和人类行为基准进行训练的基础模型。 (56:53)

Stanford professor Michael Bernstein argues that large language model-driven generative agents can function as effective 'what-if machines' to simulate human social behavior, provided they are grounded in rich qualitative interviews rather than shallow demographic prompts. However, while agent simulations reliably surface qualitative possibilities and inform policy safeguards, quantitative estimations and multi-agent systems carry severe risks of distortion and emergent unpredictability. Cognitive Architecture for Believable Agent Interaction (20:58) Traditional agent-based models relied on rigid scripting and sparse parameters, rendering them incapable of unscripted social dynamics. Bernstein and his collaborators bypassed this limitation by building generative agents on foundation models pretrained on extensive human interaction. Demonstrated in the 25-agent virtual sandbox 'Smallville,' agents sustain daily routines and disseminate information organically—exemplified by a Valentine's Day party where twelve agents learned of the event by word-of-mouth and five attended, without pre-programmed scheduling logic. This behavior relies on a three-part cognitive architecture. First, an episodic memory stream records a chronological log of all perceptions and actions. Second, a retrieval-augmented generation (RAG) mechanism dynamically extracts relevant memories based on recency, importance, and situational relevance to prevent context window distraction. Third, a periodic reflection mechanism synthesizes low-level observations into abstract self-conceptions and evolving goals, which feed into hierarchical planning and reactive replanning routines. Grounding Digital Twins in Deep Qualitative Data (28:49) Bernstein cautions that believable agent behavior does not guarantee empirical accuracy. Priming models with demographic check-boxes or shallow personas routinely produces caricatures and stereotypes—such as predicting that an individual from South Korea will inevitably eat rice for lunch. To build accurate 'digital twins,' Bernstein's team administered two-hour qualitative voice interviews to 1,000 Americans, capturing personal histories, ambivalences, and nuanced perspectives. Benchmarked against a human two-week test-retest consistency baseline (normalized to 1.0), random guessing achieved ~0.33, and demographic personas reached ~0.70. In contrast, full-interview digital twins achieved an 0.85 replication ratio across the General Social Survey, Big Five personality inventories, and behavioral economic games. In scientific replications across five published PNAS studies, agents reproduced four; the unreplicated fifth also failed among 1,000 real human subjects, reflecting a false positive in the literature. Demographic parity remained high, though political orientation exhibited an 8% accuracy gap, with far-right conservative views proving hardest to simulate due to model alignment guardrails and participant interview coyness. The Four-Rung Ladder of Simulation Risk (39:07) To prevent misapplication, Bernstein conceptualizes simulation reliability across a four-rung ladder of increasing ambition and risk. Rung 1 focuses on possibility exploration (identifying edge cases and adversarial behavior), while Rung 2 models qualitative reactions and conversational sentiment. Bernstein states that both rungs are functionally dependable today for identifying structural vulnerabilities and soft-skill training—such as reducing antisocial negotiation tactics by two-thirds. Conversely, Rung 3 (quantitative outcomes) and Rung 4 (multi-agent emergent systems) introduce acute failure modes. Language models frequently distort fine-grained distributions; in one industry case study simulating Pew retirement data, an agent model compressed a 13% fee-familiarity rate among young adults down to 1.2%. Because multi-agent interactions amplify individual model errors unpredictably, Bernstein advises organizations to treat simulations as an ideation funnel—culling 100 concepts down to five candidates before running empirical A/B tests on real humans—rather than as definitive statistical predictors. Environmental Modeling and Fundamental Social Unpredictability (51:19) Simulating internal human attitudes represents only half of the challenge; realistic environmental context accounts for roughly 40% of behavioral variance. Bernstein observes that without physical constraints and environmental grounding—such as adding functional doors to prevent agents from invading occupied restrooms—behavior rapidly deteriorates into absurdity. Sterile, context-free benchmarking cannot capture real-world social fidelity. Furthermore, social dynamics are inherently non-deterministic. Citing the 14,000-participant Music Lab experiment, Bernstein notes that while isolated listeners select songs based on intrinsic quality, introducing social signals causes popularity to diverge radically across identical parallel universes due to early cumulative advantage. Because real-world social outcomes branch unpredictably, Bernstein argues that agent simulations must never be treated as deterministic point forecasts. Instead, practitioners must run Monte Carlo distributions across hundreds of iterations to map out probability spreads. Bernstein warns that ongoing alignment efforts to make frontier reasoning models helpful and harmless enforce conflict avoidance, causing simulated agents to default to polite banter and making them worse at replicating messy human friction. He contends that advancing social simulation will require dedicated foundation models trained on cognitive and psychological benchmarks rather than general-purpose reasoning engines, alongside greater scrutiny of user attachment to artificial companions.

[5:56] Michael Bernstein: highly stylized and have had minimal impact [8:57] Michael Bernstein: the good, the bad, and the ugly [23:19] Michael Bernstein: shower thoughts, figuratively speaking. [25:58] Michael Bernstein: Everything I've shown so far shows believability. But Disney characters are believable. Cartoons are believable, but that may not be accurate. [28:49] Michael Bernstein: Tell me the story of your life. [32:26] Michael Bernstein: I hope that makes sense. So like a 0.1 would mean we can replicate people's responses 10% as well as people replicate themselves two weeks later [39:07] Michael Bernstein: mostly generally works today. [44:21] Michael Bernstein: set in reaction to a dumpster fire. [51:19] Michael Bernstein: In early drafts of this, there were no doors. Like, we just hadn't implemented the doors. And so the agents would walk in on each other using the restroom because they couldn't see, and there was no—you have to get the environment to be accurate if you want accurate behavior. [56:53] Michael Bernstein: kludge it on top of Claude or GPT-5.

[0:10] Hi, everyone. [0:11] Good morning, good afternoon, good evening. [0:14] That way, we cover everything. [0:16] We're happy to have Professor Michael Bernstein here [0:18] for this webinar. [0:20] So let me give you a bit more of his background. [0:23] Michael Berstein is a professor of computer science [0:26] at Stanford University, where he's a bass University [0:29] fellow and an STM microelectronics faculty [0:33] scholar. [0:34] His research in human computer interaction [0:37] concentrates on the design of social computing systems. [0:41] This research has won Best Paper Awards at top Conferences [0:44] in Human Computer Interaction, including CHI, CSCW, ICWSM, [0:52] And UISD, and has been reported in venues such as The New York [0:57] Times, Science, Wired, and the The Guardian. [1:01] Michael has been recognized with an Alfred P. Sloan Fellowship, [1:05] UIST Lasting Impact Award, and the Patrick J. McGovern [1:10] Tech for Humanity Prize. [1:12] He holds a bachelor's degree in symbolic systems from Stanford [1:16] University, and a master's degree and a PhD [1:19] in computer science from MIT. [1:23] Please stay muted so that we can ensure good audio for Michael. [1:27] We would appreciate if you could turn your cameras on so we [1:30] can have an engaging and interactive session, [1:32] and we encourage you to write your questions in the chat [1:36] since at the end of the session, we will have some Q&A segment. [1:40] Now, if we're all ready, Michael, [1:43] we are thrilled to have you here. [1:45] So the floor is yours. [1:46] All right. [1:47] Thank you so much for coming. [1:48] I'm seeing, wow, a fairly global presence here in this webinar. [1:53] I'm actually somewhat astounded by the number of you [1:55] who are here. [1:57] Over 500 of you. [1:59] So thank you for joining me. [2:00] We're going to talk about AI, and we're [2:03] going to talk in particular about a frontier of AI agents [2:07] that I think is particularly poised to have a lot of impact. [2:13] And maybe you're an individual contributor. [2:15] Maybe you're a manager or a leader, or you're imagining [2:18] or you're a policymaker. [2:19] There are so many different applications [2:21] of these kinds of technologies if can get them [2:22] to work, which they're increasingly showing promise of. [2:26] So we're going to talk through what is this frontier [2:29] and what's going on. [2:30] When we make decisions, we make those decisions [2:35] based on what I would view as often incomplete information [2:39] about how people are going to react. [2:41] Maybe you're at an organization, whether it's [2:43] a for-profit or a nonprofit that's trying to decide, [2:46] if we do this, maybe our clients or customers [2:51] will react this way, or maybe these people [2:52] who aren't currently our customers [2:54] will or won't react that way. [2:56] Or, maybe I'm a leader and I want to-- [2:58] I say, if I change my management style [3:01] in this way or reorg in the following way, [3:03] maybe the organization will do better or worse. [3:06] We make these decisions based on the best data we can. [3:11] But that data is often incomplete. [3:15] And this kind of incomplete information [3:18] means that we often make bad bets. [3:21] And what I mean by that is that we make the best guess we can. [3:24] We take all the information we can. [3:26] But we're ultimately wrong. [3:27] And this is not necessarily a problem with you. [3:31] It's not that you're not smart. [3:32] It's just really hard. [3:34] We can't know in advance always what's [3:36] going to happen because we have limited ability to get feedback. [3:38] We can only launch once. [3:40] And so we make these big bets and they [3:41] misfire across these areas across like consumer packaged [3:44] goods, CPG, personal finance, product design, [3:47] policy management, even me as a professor. [3:50] I do this all the time. [3:51] I'm making guesses about what's going to work [3:53] with my students and not. [3:54] And you think, oh, this is like a problem of modernity [3:58] and so on. [3:58] But this is not even a new question. [4:01] This is a thing that goes all the way back to 1906. [4:04] Here's Robert Merton, a famous, famous sociologist [4:07] who pointed out even back then, over 100 years ago, [4:11] how hard it is to design for what a group of people will do. [4:17] Like, we all want to get out of the city [4:20] so that we can avoid the crowds, but then everyone [4:22] wants to get out of the city and go on vacation at the same time [4:24] so we all wind up in the same forest, [4:26] and it's too crowded there too. [4:28] So we're very challenged in making these kinds of decisions. [4:33] And I want to ask today, what if you had what-if machine? [4:40] What if you had what-if machine? [4:43] What would you use that for if you could imagine [4:46] in advance what might happen? [4:49] Imagine that you could say if my organization took this path, [4:54] how might our customers react? [4:57] Or if you could say, if we were to fail [5:01] or if something were to go wrong, [5:03] what's the pathway through which that would maybe happen? [5:07] Or you could say, well, what if we [5:12] could see how people might react to this policy [5:15] change or this new product or this strategy [5:18] we're thinking about before we deploy it. [5:20] If you have this kind of what-if machine, [5:23] my claim is that we could make better decisions [5:26] more often, much more often. [5:29] And so this has led to a really interesting question for me [5:33] as a researcher in computer science-- [5:35] could we create simulated AI agents [5:39] that replicate human behavior? [5:40] Could we create AI simulations that [5:44] behave like people do, maybe your organization would [5:49] or like your customers would and so on? [5:51] If we could, then maybe some of these things become possible. [5:56] Now, simulation isn't a new idea either, right? [5:59] It goes all the way back to 1978. [6:02] Here's Thomas Schelling won a Nobel Prize in economics [6:05] for generating this idea of what are called agent-based models. [6:07] And these things are still in use today. [6:09] And I'm showing a screenshot here of a recent model that [6:13] was run on supercomputers trying to understand, oh, if we engage [6:16] in the following kind of intervention, [6:18] does this reduce the kind of spread of a pandemic and so on? [6:22] There are these very simple algorithmic models of people, [6:24] well, actually increasingly complex, but still fairly [6:27] mathematical. [6:28] But simulation is not just a thing [6:30] that we use in policy-making. [6:31] Here's entertainment. [6:32] I bet a bunch of you have played The Sims. [6:34] It is one of the singularly most popular video games of all time. [6:38] These are also simulations of people. [6:41] And these people in-- [6:43] quote unquote, "these agents" in The Sims can change the space [6:48] around them. [6:48] You can intervene on them. [6:50] You can have all sorts of fun in this little dollhouse [6:52] environment. [6:54] And increasingly, if we are deploying AIs [6:58] into worlds where the AIs need to interact with us-- [7:01] some people talk about AI teammates-- [7:03] what does that even mean? [7:04] Like an AI that's on your team at work [7:07] would need to start reasoning about how you might respond [7:11] if it takes a certain action. [7:13] And more broadly, to create a set [7:16] of tools that are what I would term a look before you launch. [7:20] And I think we struggle with this. [7:22] For much of my career, I worked in the design [7:24] of online platforms, social media, and these kinds [7:27] of things, and many issues would come up [7:29] because we couldn't effectively foresee [7:31] what was going to go wrong. [7:33] And so this creates the opportunity [7:35] to create tools that say, ah, what kinds of policy changes [7:39] or rules do I need to set up before I launch the system so [7:42] that it doesn't all become a dumpster fire, [7:44] and then I set the rules afterwards? [7:47] Now, the problem has then has been that up till today. [7:51] Our models have been pretty rigid. [7:55] That is to say like you have two options-- one, [7:58] you say a human is five parameters. [8:01] That seems like it's not quite true. [8:03] Like, we can't quite be reduced to five parameters. [8:06] They can explain a lot about us. [8:07] But I'm at least seven parameters. [8:09] It's essentially this is just much too of a sparse model [8:14] to encode the richness of human behavior. [8:16] The other option is to do what The Sims does, [8:18] where you basically write scripts and say, [8:21] if I get punched, then I fall down and I get mad. [8:24] And there again, it's sort of limited [8:26] to only the kinds of things we can think to create. [8:31] And zero is always going to be incomplete. [8:33] So as a result, at least within the academic literature, [8:36] the basic conclusion is, I'm going to quote here, [8:38] "The models have been highly stylized and have had minimal [8:41] impact." [8:42] So not a good track record. [8:44] The really interesting. [8:46] But in practice, we haven't seen them taken up in many cases. [8:50] And I mean, I'm sure some of you in this call [8:52] have used agent-based models, but the vast majority of people [8:55] who I talk to have not. [8:57] But starting a few years ago, my colleagues, [9:01] and I recognized that modern AI models, so-called large language [9:07] models like ChatGPT, Claude, Qwe, all of these, [9:11] DeepSeek, have trained on a variety of human behaviors, [9:17] which is to say that they've literally read the book on all [9:21] the research about how people behave because they trained [9:24] on research, and they've also trained [9:27] on a bunch of social media data. [9:29] They've seen the good, the bad, and the ugly of human behavior. [9:33] And so as a result, what we recognized [9:35] was that you can prompt these large language [9:37] models to take on the perspectives of a bunch [9:41] of different people with different backgrounds, [9:44] experiences, and traits. [9:47] So we can take ChatGPT or another large language [9:51] model, LLM, we can give it a prompt that roughly equates [9:55] to here's be this person. [9:57] Here's a name, a description, and in this situation, [10:00] how would they react. [10:01] And by multiplexing those descriptions, [10:05] we can create an entire crowd of different people. [10:08] And if we create a crowd of people [10:10] can put them in many different situations [10:12] and see what might happen. [10:14] And in fact, we started doing this. [10:16] And in some work that shall-- we say, went a bit viral? [10:20] We created almost like a little terrarium [10:23] of many of these so-called of these AI agents [10:25] that are simulated people. [10:27] We call them generative agents. [10:28] And we created a little town called Smallville. [10:32] This town is full of 25 generative agents. [10:36] These 25 agents, as I would describe them, [10:40] are essentially all completely autonomous AIs [10:44] that are each playing the part of a different person. [10:48] So you can see the entire town is going about their day. [10:50] The artist in the town gets up and paints the college students [10:54] in the town. [10:54] They sleep in more, and then they go to classes [10:57] and do their homework and so on. [10:59] This allows us to create a space where [11:01] we can visualize what it would mean [11:03] to have these simulations of human behavior in a way [11:06] that we can learn about people and interventions. [11:10] So in fact, this work got a lot of attention. [11:15] And more recently, Andreessen Horowitz-- [11:17] I think this was just last month-- [11:19] put out some research, their own research, [11:22] arguing that this is going to become [11:24] the next essentially generation of market research tools. [11:28] And if you go check out this-- it was in June. [11:31] Excuse me. [11:31] If you check out this article, then they [11:34] cite our research quite centrally. [11:37] So what I'm about to show you in this webinar [11:39] is that AI simulation can create both believable and accurate [11:45] simulations of human behavior. [11:48] I'm going to give you a how-to-guide [11:49] of the techniques that enable this kind of simulation. [11:53] I'm going to give you the gotchas, the things that often [11:55] people get wrong, that you should look out [11:57] for when doing this. [11:59] And I'm going to talk about the current frontiers [12:01] of this technology. [12:02] Like, what's kind of on the edge of just becoming possible? [12:07] So I hope that sounds good. [12:09] If you're excited, I don't know, give a thumbs up in Slack [12:12] or excuse me in Zoom, and we'll dive in. [12:16] The basic way we're going to go after this is we're [12:21] going to create an agent by first [12:24] we paid an online artist to get a bunch [12:26] of pixel art of different characters, which [12:28] you can see on the left. [12:30] And these visualizations are there [12:31] mostly to help you see what's going on. [12:34] You don't actually have to build a little video game [12:36] environment like I did. [12:37] I'm just a child. [12:38] I was born in the '80s, grew up in the '90s, 1990s Super [12:41] Nintendo RPGs, so it's just a cool way to show it. [12:44] But essentially what we do is we create a little persona [12:48] for each individual. [12:49] So here's John Lin. [12:51] He's a pharmacy shopkeeper. [12:52] We tell him that he runs the pharmacy in this town, [12:54] and we describe who he is. [12:56] He's very helpful. [12:58] And we tell that agent who what it knows [13:01] about others in the simulation. [13:02] So we tell him he's married to this other agent [13:06] in the simulation, Mei Lin. [13:08] And they have a son, Eddy Lin, who's a college student studying [13:11] music theory. [13:12] Now, we have to tell the agent about all of the other agents it [13:15] knows in this environment, because if we don't, the agent's [13:17] going to wake up in a bed next to Mei Lin [13:20] at the beginning of the simulation and go, ah! [13:23] It won't know. [13:23] It literally will not know that other person. [13:25] So we have to tell that agent everything that it [13:27] needs to know at the beginning. [13:29] Now, what I want to try to explain to [13:31] you is, over the course of this webinar, [13:36] the important pieces of what it takes to build one of these. [13:40] I don't expect that you understand this figure yet, [13:43] as I'm first showing it, but I'm going [13:45] to build up to this figure over the next few minutes [13:48] so that you understand what it is [13:50] that allows these agents to do what they do. [13:52] And so that you can create them yourselves. [13:56] Now, basically, what we can do is we just describe the agents, [14:04] and we're going to just let them go from there. [14:08] They wake up without us telling them exactly what to do. [14:12] We just describe who they are. [14:13] They'll wake up, brush teeth, take a shower, [14:15] cook breakfast, catch up and talk to each other. [14:18] They'll pack and get ready. [14:22] And what I'm going to do here is to the global alumni team, [14:25] I can message you, but I can't message the people here. [14:28] I just sent you to the global alumni team a link to a demo. [14:31] If you can echo that demo, link out to the group. [14:34] They can actually load it up and see the town in, effectively, [14:38] real time. [14:39] So I encourage you to go check it out. [14:43] Thank you. [14:43] Yes, you sent out the correct thing. [14:46] So please go check out that link. [14:48] Now, the agents then act in the environment [14:51] because these are just AI agents. [14:52] Which means they mostly just act in the sense of they [14:56] use natural language. [14:57] So they say, this agent, Isabella Rodriguez, [15:00] is drinking coffee. [15:02] We then need to ground those that text [15:07] into concrete movements in our little video game environment. [15:10] So we say, oh, they're drinking coffee, OK. [15:12] They're going to walk over to this chair, [15:13] and then we're going to render a little emoji that summarizes [15:17] what it is that they're doing. [15:18] So if you go and click on that demo, [15:20] you can see the agents are often having these little emoji [15:22] bubbles that show what they're up to. [15:24] So we ground their actions which start out as language [15:28] into this game environment. [15:31] But you can talk to these agents literally. [15:34] You can intervene and talk to them. [15:36] So if you pretend to be a newspaper reporter, [15:39] then you can say, hey, who's running for office? [15:42] Who's running for mayor? [15:43] And if the agent knows, they'll respond, oh, yeah, [15:46] I heard Sam's running for mayor. [15:47] And if you want to control these agents, [15:49] then you can just pretend to be their inner voice and say, John, [15:54] you're running for mayor. [15:56] And then John says, well, I really [15:58] need to talk to my family about this big decision [16:00] that I'm considering. [16:03] So you can also intervene in the game world. [16:06] I can take the toaster, set it on fire, [16:10] and the agent will recognize what's going on [16:13] and run over to put out the fire and then [16:17] go make something else for breakfast. [16:19] So we have all of these opportunities [16:21] to think about interventions in the world [16:24] and how agents might respond to them. [16:26] Now, imagine intervening and changing the layout of the home, [16:30] or the kinds of information they have access to [16:32] or the kinds of products in their home, and so on. [16:36] So just to give you a sense of what this is actually like, [16:40] let me take it from here. [16:42] These agents talk to each other. [16:43] They're language agents. [16:44] They're AI agents that can talk. [16:46] So in the morning, John sees his son Eddy [16:49] and engages in conversation, says, hey, good morning, son. [16:51] Did you sleep well. [16:52] The son says, yeah, I slept great. [16:54] He says, hey, what are you doing today. [16:56] And the son says, yeah, I'm working on this music theory [16:58] composition for my class. [16:59] Remember, he's a college student. [17:01] He's a music theory student. [17:02] So he's working on a composition, a new song. [17:05] Now, John doesn't know this, but now, he does. [17:08] He has to remember it, because soon in the simulation, [17:13] Eddy's going to leave the house and John's wife [17:15] is going to wake up and say, where's your son? [17:20] And John has to remember what he learned [17:21] so that he can tell her, yeah. [17:24] He left. [17:24] He's working on this music theory composition. [17:26] And now, she knows, and this is how information can [17:30] spread through the environment. [17:33] Now, that's a simple example, but we [17:38] wanted to examine a much more complex one. [17:42] So here's what we did. [17:43] We initialize Isabella, this agent who runs the cafe in town [17:48] with a simple intent to plan a Valentine's Day party. [17:51] So the simulation starts on February 13th the day before [17:55] Valentine's Day in the morning, and it runs through the end [17:58] of the 14th. [17:59] So we just inserted into her memory [18:01] this little intent that she was interested in planning [18:03] a Valentine's Day party. [18:04] And what I want to convey to you is [18:06] that nothing about this AI agent architecture that we're [18:08] talking about today is like a party planning module. [18:12] Nothing that says, oh, this will definitely work. [18:14] And in fact, there's a lot of reasons why this might fail. [18:17] This agent has to remember to tell the other agents [18:19] about the party. [18:20] Anyone who's told about it needs to remember [18:22] that invitation, and those who remember [18:24] it have to actually show up. [18:27] And yet, from this seed, we start [18:29] to see evidence of classic information diffusion [18:32] patterns, what you might think of as rumor networks, whisper [18:35] networks, just word of mouth. [18:36] So Isabella tells people. [18:38] People tell other people. [18:39] You can see this graph of who's hearing [18:41] about this party from whom as the simulation plays out. [18:46] And in fact, Isabella goes and gets her friend Maria [18:50] to help decorate the cafe in preparation for the party. [18:53] Again, we didn't tell her to do any of this. [18:55] This agent decided to go get help from the other agent. [19:00] At the end of that day, like the next day, on Valentine's Day, [19:03] half the town had heard about the Valentine's Day party. [19:06] So of the 25 agents in the town, 12 of them [19:10] heard about the Valentine's Day party. [19:12] Of those 12, five came to the party. [19:16] Three said they were busy, and four said they were interested [19:20] but ultimately didn't show up. [19:22] So is this accurate? [19:25] It's a little bit of an interesting question. [19:27] It reads to me as broadly plausible, [19:30] but it's not necessarily easy to tell. [19:32] Is this exactly how it would play if this were real life. [19:36] It does, to me, seem like a pretty compelling evidence [19:39] of the kinds of believable behaviors you might get. [19:42] And in fact, one more kicker on this, [19:44] we not only told Isabella she was [19:46] interested in planning a party. [19:47] We planted a memory in the Maria agent [19:49] that ML that you see on the left in the image [19:52] that she had a crush on Klaus, another agent. [19:56] And it turns out that Maria asked her crush to the party. [19:59] So we had this little nascent agent love here. [20:02] So very beautiful. [20:04] And in fact, researchers who are not us [20:08] got this whole simulation running again, just this year, [20:12] and found that they could intervene and study [20:15] different alternatives. [20:16] So now they said, oh, OK, what would [20:17] happen if suddenly the agents-- some of the agents [20:20] heard on the radio that there was a new communicable [20:23] disease, like a new epidemic of swine flu? [20:26] And you can see in that pathway, when [20:28] they ran the simulation, oh, no one showed up to the party [20:30] except for poor Klaus. [20:32] Klaus had not heard about the new illness. [20:35] And so he had-- [20:36] He showed up anyway in the no threat condition. [20:39] The party happens as usual. [20:41] And when the. [20:41] And when the agents hear a radio about a non-infectious disease, [20:45] like diabetes complications or something like this, [20:48] the party also happens as normal. [20:50] So you can think about how interventions into the world [20:52] let you ask these what if questions. [20:54] That can be quite powerful. [20:58] So how do we do this? [20:59] I'm coming back to this image, and I [21:01] want to try to explain the major parts of this architecture. [21:05] You don't have to be technical to understand this. [21:07] I want to convey the main pieces. [21:11] The first thing we need to do to create an agent like this is we [21:17] need to give them the ability to remember. [21:20] So we give them what's called a memory stream. [21:23] This memory stream is essentially [21:25] a blow-by-blow record of everything [21:31] that the agent observes. [21:33] So you can see a lot of this is actually not very interesting. [21:36] They see a bed. [21:36] They see a desk. [21:37] They see a closet. [21:38] Some of it is more interesting. [21:39] Like, I'm stretching, I'm writing in my journal, [21:42] I'm taking a break, I'm cleaning the kitchen, and so on. [21:46] But there's a lot of stuff here. [21:47] You could fit it into the context of some of the larger [21:52] window large language models. [21:54] But these models, the research suggests, [21:56] can get distracted with very, very long contexts. [21:59] So we actually do something that's [22:01] referred to in the literature these days as RAG, [22:04] or retrieval augmented generation. [22:06] So with RAG, what we're doing is we're [22:10] preferentially retrieving memories [22:12] that were accessed recently that the age that [22:15] are important to the agent and that [22:17] are relevant to the current circumstance. [22:19] So recency means if it's something [22:21] that you observed recently or that you recalled recently, [22:24] that's more likely to come up. [22:27] If it's important. [22:27] Like the fact, I don't know. [22:28] The agent brushes its teeth. [22:30] Not very important to the agent. [22:31] The fact that it got asked out to a party, pretty important. [22:35] And finally, relevance, so if an agent is taking a math test, [22:38] then other memories about taking tests or about that math class [22:42] should be top of mind. [22:45] So it's almost like running a little bit of a Google search [22:47] over all of these memories. [22:51] And so now, what happens is what you can see on the right, [22:53] if we ask the agent the question, [22:55] what are you looking forward to right now? [22:57] We're going to retrieve these recent important and relevant [22:59] memories, like things like that they're planning a party [23:02] or that they're ordering decorations for the party [23:04] and researching ideas for the party. [23:06] Put those into the context window of the large language [23:08] model, along with the description of Isabella herself. [23:11] And that's what gets Isabella to reply in a reasonable way, [23:15] saying, I'm looking forward to this Valentine's Day party. [23:19] Memory-- the first thing you need. [23:21] The second thing that we feel is important [23:24] is this ability to reflect. [23:26] If you have a psychology background, [23:28] then you might be familiar with what I'm about to say, [23:30] where the memory I just showed is what's called a purely [23:33] episodic memory. [23:34] It's just like a log of what happens. [23:36] But if you think about it, we're more than just captain's log. [23:40] This is what happened. [23:41] This is what happened. [23:42] Or we needed to provide the agents with the ability [23:45] to produce higher-level reflections of who they are, [23:49] what they like, their dispositions, [23:51] and their interests and their goals. [23:53] So what we do is we essentially get these agents [23:57] to have shower thoughts, figuratively speaking. [24:01] So what we do is at regular intervals, [24:03] we pull memories from the agent's memory stream, [24:08] and we ask the agent to reflect on those memories-- [24:11] here's two examples-- to produce-- [24:14] so a higher level reflection or conclusion about them. [24:17] So given the two observations that Klaus Mueller is reading [24:20] about gentrification, reading about urban design, [24:22] we produce a reflection that Klaus [24:24] spends a lot of time reading, and we can do this over and over [24:28] again by continuing to group memories [24:30] and reinserting those reflections back into the memory [24:33] stream to produce higher and higher level [24:35] reflections about who this agent is and what their goals are. [24:39] This produces the ability for the agent [24:41] to essentially to produce behavior that's [24:44] much more consistent with these goals [24:45] rather than just literally being a robot [24:48] and walking from step to step. [24:52] The third thing is planning. [24:58] So planning is important for these agents [25:02] to remain believable over long periods of time. [25:04] And planning is something that is not unique to this project. [25:09] It's been a long time issue with AI. [25:13] So what we would do is, we would start [25:17] by having these agents plan out their full day. [25:21] Then we would drill in and given that day, [25:23] say hour by hour, what are you going to do. [25:25] We're going to drill in again and go minute by minute [25:28] and say, OK, in details, what are you doing? [25:31] And when the agent sees something [25:32] as they're acting in the environment, [25:34] like here, the John agent might see Eddy, his son, [25:37] taking a walk around his workplace, [25:40] then we will ask this agent, given its background, [25:44] you can see the prompt here on the left [25:46] if it should react to the observation. [25:48] If so, what would be an appropriate reaction? [25:50] And then replan if we need to. [25:53] So this is how the agents adapt as the simulation runs. [25:58] Now, this work got a lot of attention. [26:01] It really caught fire with a number of people. [26:04] And that was really interesting to see. [26:08] And I think I just want to start by suggesting [26:10] there's some evidence-- that there's something [26:14] to this idea that of the goal of creating [26:17] these simulations of people and what that might enable. [26:22] You can see here news reports that [26:24] are featuring Joon sung park, my PhD student who led this work. [26:29] And yet, with this vision, I haven't fully answered [26:35] the question, right? [26:37] I asked this question before about whether we [26:39] can create simulated AI agents that replicate human behavior. [26:47] Everything I've shown so far shows believability. [26:51] But Disney characters are believable. [26:52] Cartoons are believable, but that may not be accurate. [26:56] And if you're going to make decisions based on this, [26:58] then there's this question of how [26:59] accurate are these agents at replicating behavior. [27:03] And that's an empirical question that we really [27:06] need to ground out if you're going to think about whether you [27:08] want to be using this. [27:10] So how do we measure this? [27:14] Because there's papers out there that just try to replicate this [27:16] study or that study and say this one works or that one [27:19] doesn't, but others then try to replicate that and show [27:21] a different result. How do we act like you can't-- [27:24] How do you actually know? [27:26] So let's talk through how you might know. [27:29] Well, there's a few different ways you could go about this. [27:33] One thing you could do is build what you [27:34] might call demographic agents. [27:36] So here, you take a sample of the population, [27:39] and you produce a demographic agent version of a person. [27:44] So here's a demographic agent version of me. [27:46] It says, my age, where I live, my job, [27:49] and these kinds of things. [27:51] On the other hand, you could create [27:53] what we did in that Smallville simulation, these persona [27:55] agents that are more narrative descriptions of the person. [27:58] So again, on the right here is a persona agent version of me. [28:02] So maybe adds a little bit more flavor [28:05] but also misses some information in. [28:08] But they're both interesting. [28:09] The issue here, as prior research has demonstrated, [28:13] is that both of these approaches can [28:15] produce very simplified vilified and stereotyped behaviors. [28:20] Most kind of-- one of the instances that [28:24] was most striking to me was that if Joon Park. [28:27] Again, my PhD student or now alum [28:29] who started this work, if you just produce-- [28:33] he's from South Korea. [28:34] And if you ask if you tell the model that and say, what is he [28:36] going to have for lunch? [28:37] The model says rice. [28:39] Not great, really. [28:41] Probably not accurate either. [28:43] It's very stereotyped. [28:44] It has bias, all these kinds of things in it. [28:46] So what are we supposed to do? [28:49] So what we found is that there are better ways to do this. [28:54] In particular, we found that rich qualitative information [28:58] is an interesting direction. [29:00] So what we did-- and this is some more recent research-- [29:03] is we went out and ran two-hour interviews with a representative [29:08] sample of 1,000 Americans. [29:10] Now, obviously, this is focused in the US. [29:12] But you can do this yourself in your local jurisdiction [29:15] if you're interested. [29:15] So we would run these long interviews [29:17] and I'll talk about whether two hours is necessary later. [29:20] It turns out actually you don't need two hours. [29:22] We took an interview script from a project [29:24] here at Stanford called the American Voices [29:26] Project, led by David Grusky. [29:28] He's a sociologist, has this very broad [29:30] ranging interview script that literally [29:32] starts with the question, tell me the story of your life. [29:35] And then it covers everything from their communities [29:38] and their jobs and their finances and their health [29:41] to their politics. [29:42] It's just this broad backbone of who they are. [29:45] We use this two-hour interview to construct [29:48] a digital twin of that same person, a generative agent [29:52] of that same person with that interview [29:56] as the memory of the agent. [29:58] OK, so now, we have a real person, [29:59] 1,000 real people and 1,000 twinned agents for them. [30:05] That real person then takes a large battery [30:08] of surveys and experiments so we can [30:12] gauge what their actual attitudes and behavior are [30:15] in realistic scenarios. [30:17] So here, I'm going to focus on the General Social Survey. [30:19] This is a 170-ish question survey that has been fielded [30:23] in the United States over the last 50 years, [30:26] but we also look at the big five personality index, [30:28] some behavioral economic games, some replications [30:31] of experiments. [30:31] There's this broad battery. [30:34] We then have the generative agent of that same person [30:37] take the same surveys and experiments. [30:41] And then now, we can finally measure this more accurately. [30:45] We can say, how closely does this person's agent [30:49] replicate that person's actual behavior and attitudes? [30:53] I hope that makes sense. [30:54] So we have real person, agent version of them, [30:57] and we ask how well the agent version of them [30:59] replicates their exact answers. [31:03] So we run these interviews actually with another AI agent. [31:07] It's a voice-to-voice interview like a phone interview. [31:09] It's on the computer, but I've changed many of the details [31:14] here to preserve anonymity, but I just [31:15] want to give a sense of the flavor [31:17] of the kind of information that these interviews yield [31:20] about people. [31:20] Obviously, this data all remains private for human subjects [31:25] reasons. [31:26] But you can see that the kinds of things we're learning [31:29] are people's backgrounds and their life [31:31] stories, their ambivalence toward political identification, [31:35] what their jobs are and were their finances, [31:38] and how much extra money they have [31:40] to spend on various categories and so on. [31:42] So it's just a very, very rich background. [31:47] All we do then-- and if you wanted to create these kinds [31:50] of agents yourself, given this kind of interview, [31:53] is we put the interview up at the top of-- [31:56] the top of the prompt to ChatGPT or another model, [31:59] and we tell the model based on this interview transcript, [32:02] "I'd like you to predict how this person would respond [32:05] to the following survey or experiment." [32:07] Now, there's more detail to it. [32:09] Here, you can see the actual prompt we use. [32:12] But for the most part, it's just the yellow parts here. [32:16] Interview and then now please predict what this person [32:19] will say in response. [32:22] What we find is that these agents do accurately replicate [32:26] attitudes and behavior. [32:27] Now, in order to explain this, we [32:29] need to say one more thing about the method. [32:32] It turns out that if I give you a survey today, [32:35] and then I give you the same survey two weeks later, [32:38] you are not going to answer exactly the same thing. [32:40] A lot of stuff's going to be the same, but not exactly the same. [32:43] So we need to normalize for this. [32:44] So we do exactly this. [32:46] We have everyone, all 1,000 people in this experiment run [32:49] the same survey twice. [32:51] And so what we're going to do here [32:52] is every result I'm going to show you here [32:54] is going to be a ratio where 1.0 means that agents replicate [32:59] people's responses as accurately as people [33:02] replicate themselves two weeks later. [33:05] I hope that makes sense. [33:06] So like a 0.1 would mean we can replicate people's responses [33:09] 10%, as well as accurately as people [33:12] replicate themselves two weeks later. [33:14] OK, so if you were to do this, if you [33:18] were to guess randomly on this big survey, [33:21] you could replicate people about a third as well as they [33:24] replicate themselves two weeks later. [33:26] That's a baseline of how predictable people are. [33:29] If you use these persona agents or the demographic agents, [33:32] you can replicate people about 70%, [33:33] as well as they replicate themselves two weeks later. [33:37] And if excuse me, if you use these full interviews like I [33:40] was describing can replicate people [33:42] 85% as well, as they replicate themselves two weeks later [33:46] on the General Social Survey. [33:49] I think that's pretty impressive. [33:51] Now, it's not just the General Social Survey [33:55] across a series of other tasks like the big five personality [33:58] index, behavioral economic games. [34:00] You can see that these normalized coefficients are all [34:03] generally in this similar range. [34:06] So we think that there's substantial possibility here. [34:10] Furthermore, actually, these interviews reduce bias. [34:15] And so if you have this full interview, [34:17] if all I know about you is that you're like a Republican, [34:19] a conservative, then it really stereotypes. [34:23] But if I know a lot more, then it can-- [34:26] the model can get much more subtle. [34:27] So it turns out across these different approaches [34:29] that you might imagine. [34:31] Politics is the hardest to model. [34:32] So the difference between the best performing group [34:34] and the worst performing group was around 8% accuracy, [34:37] normalized accuracy. [34:38] The worst performing group for us was far right conservatives. [34:42] The models, the underlying language [34:43] models are pretty unhappy, responding to questions [34:48] as a far right conservative. [34:49] They've been value aligned maybe to not say some of these things. [34:52] And the conservatives, the far right conservatives themselves [34:54] in the interviews were a little more coy [34:57] about their true opinions. [34:59] Gender and race surprised me. [35:01] They were actually they had smaller gaps at a baseline [35:04] than I expected. [35:06] But again, the interviews cut out a lot of the difference. [35:09] So again, we're looking at maybe just a sub-- [35:11] in the case of gender, sub 1% difference in accuracy [35:15] between the different groups. [35:18] In fact, as I mentioned, we also had 1,000 people do this. [35:22] We had 1,000 people replicate experiments that were published [35:27] in top tier journals. [35:29] The proceedings of the National Academy of Sciences. [35:31] This is a top, top tier science. [35:33] So we had 1,000 people or 1,000 people replicate those same [35:36] studies, and then we had their agents replicate those same five [35:39] studies. [35:40] Of those five studies, our agents [35:42] replicated four of the five. [35:44] And you might say, oh, well, that's not great. [35:46] You missed one of them, but it turns out that we didn't. [35:49] The 1,000 real people also didn't replicate that fifth [35:52] study. [35:53] That fifth study was actually bad science. [35:56] That didn't replicate that result with real people [35:59] was not that was published, wasn't a real effect. [36:02] And the simulations correctly predicted [36:04] that was going to happen, that fifth study [36:07] wasn't going to happen. [36:08] And in fact, my colleague here at Stanford, Rob willer, [36:11] has generalized this in a really compelling way, where [36:14] they found a bunch of non-public study results [36:18] that were pre-registered. [36:19] So they got all of these studies, 100,000 people, [36:21] and found that they could predict the effect sizes [36:23] of these experiments with simulation with a correlation [36:27] between 0.85 and 0.9 pretty strongly. [36:31] Now, what this wound up with for us was this agent bank of 1,000 [36:38] consented participants from the United States. [36:41] So we have this agent bank that we can now [36:43] use to test maybe depolarization or climate change [36:49] kinds of questions and so on. [36:51] But if you're trying to do this for your organization, [36:53] you might want to think about who [36:55] ought to be in that agent bank that you [36:57] would build for questions that you care about? [37:01] Should they be representative of your customers? [37:04] Should they be representative of people who [37:06] of people who are not currently your customers, but could be? [37:10] Should maybe your company has internal marketing or user [37:14] research data that you've interviewed people in the past? [37:16] You just have a data set sitting there that you [37:18] could turn into an agent bank. [37:22] Here's my advice. [37:22] Here's how-to. [37:26] A bad way to do this would be to define agents [37:28] with a single demographic variable, like just that. [37:30] "Be a conservative." [37:31] "Be a Republican." [37:32] This produces all sorts of stereotyping [37:35] and under underestimates the variance, [37:38] the distribution like it makes it seem to certain when you [37:41] need it to be more uncertain. [37:42] It's just not enough. [37:44] Not terrible would be five or six demographic variables. [37:47] We saw that was able to replicate people about 70% [37:50] as well as they replicated themselves two weeks later. [37:53] Better was data as rich as you can gather, [37:55] like these interviews we were finding, [37:58] we were actually able to maintain very strong accuracy [38:02] by having a much shorter interview. [38:04] So we went in later, and we just deleted [38:06] random 80% of the interview. [38:09] And we were still we went from 85% accuracy [38:11] to m or 0.85 replication ratio to 0.79. [38:16] It didn't really decrease it that much, which surprised me, [38:20] to be honest. [38:22] So there's something about these interviews, [38:24] is teaching very rich information. [38:26] But I do think-- [38:28] and I see a comment in the chat about this exactly on point. [38:32] You have the-- remaining parts of the interview [38:34] have to be relevant. [38:35] So if you're only interviewing people about their views [38:38] on fashion, say, and then you're trying [38:40] to predict their views on climate change, [38:42] there's just not enough for the model to generalize. [38:44] OK, maybe if we know that they're very high income [38:48] or something like this, you might be [38:49] able to extrapolate from there. [38:51] The models can do quite a bit. [38:52] But if I'm only interviewing you about sports teams, [38:55] and then I'm trying to predict your retirement planning [38:59] or something like that, it makes sense [39:00] that it would be that it's a little bit too disjoint. [39:03] You have to make sure that the data you gather is relevant. [39:07] Now, we cannot proceed without talking about the risks of this. [39:12] There's literature. [39:14] Not every study replicates correctly. [39:18] And you can see some of the research here [39:20] that has found examples of cases where this hasn't replicated. [39:23] And I can show you an example of one case where this really [39:28] has happened. [39:29] So here's a company that I was talking to that [39:31] tried to implement this. [39:32] On the left is ground truth data. [39:34] This is pew trust. [39:36] And if it's a little blurry for you [39:38] then what's highlighted in red is this age-- [39:42] broken out by age, 18 to 35, 36 to 51, and so on, [39:45] and they're asking people how familiar [39:47] they are with their retirement plan fees. [39:49] I want you to pay attention to the dark blue part [39:52] of the bar where, say, in to 35 group, 13% of people [39:56] said they were very familiar with their retirement plan fees. [40:00] On the right, you can see their simulation result [40:04] where instead of 13%, that one was like 1.2%. [40:09] Now, you might say, oh, well, Michael, [40:10] you still got the biggest one that middle bar was still [40:14] guessed to be the biggest one. [40:16] And the second one was still guessed to be the second biggest [40:18] one and so on. [40:18] And so maybe it doesn't matter for your particular cost, [40:21] but maybe it does. [40:23] Maybe the difference between 13% and 1.2% [40:26] is the difference between we ignore this subpopulation, [40:28] and they're a sizable minority that we would care about. [40:31] So these things can make errors. [40:34] And the way that I think about this today [40:37] is think about it as a ladder where [40:39] as you climb up the ladder, you take on more risk of issues. [40:44] But it also gets more ambitious. [40:47] So you might start out at the bottom of the ladder [40:49] with a question of possibility. [40:50] What might this-- what might happen? [40:53] What could happen? [40:54] We're not going to put a probability label on it [40:56] or say how likely it is to happen. [40:58] Just this is an outcome that you might want to think through [41:00] because it could happen further up [41:03] would be qualitatively oriented outcomes, things like attitudes, [41:08] anything that is chat outcomes. [41:11] And this is, I think, safer because we're [41:13] pretty good at modeling this already, given the interviews. [41:16] Higher up, and where I just showed [41:18] the error was quantitative outcomes like histograms, [41:20] bar charts, where the difference between 5% and 10% [41:23] might really matter. [41:25] And at the top would be multi-agent [41:27] simulation, the town that I was describing an entire market [41:30] simulation. [41:31] In order to trust that, you need to have a lot more assumptions. [41:35] So let me dig into what you would [41:36] need to see to trust each of these [41:39] because this is a totally new method, [41:40] and I want to stress that you should not over [41:42] trust such a method with the possibility rung. [41:46] This one, I would say mostly generally works today. [41:50] What it needs to generate is a plausible chain [41:52] of events that could lead to a potential outcome, [41:56] and you should be able to just by looking [41:57] at the chain of events, say, oh yeah, that seems plausible, [42:00] that could happen. [42:01] Oh, yeah, if a troll shows up in my system, [42:04] they could really mess with it in the following way. [42:06] We need to introduce the following safeguard. [42:10] On the qualitative rung, it's mostly about attitudes. [42:14] And so we need to be able to accurately estimate [42:16] individual attitudes. [42:18] I would say that my evidence is that this mostly works today [42:22] if you have rich enough data. [42:23] Now, it's not the same as running an interview. [42:25] And I would never say that you should do this instead of [42:28] actually engaging with the communities, [42:30] but I think it is a good way to get a rough sense of what might [42:34] people say in response to say, a new policy or a new strategy [42:37] or a new product. [42:38] The quantitative rung requires actual measurement [42:42] of quantitative accuracy. [42:45] Say, can it recreate your market research surveys? [42:47] In many cases, we're seeing, yes. [42:49] And in many cases, we're seeing that there are errors. [42:51] So you need to tread carefully here. [42:53] I think the way I would view this is that you [42:56] ought to-- you consider. [42:58] You have 100 ideas of what you might do. [43:00] Use simulation to narrow it down to five. [43:02] That seem promising. [43:04] And then actually A/B test the remaining five on real people [43:07] to see what's working with multi-agent. [43:11] I don't think that we're really ready for that [43:13] in decision making quite yet. [43:16] If you wanted to do this. [43:17] You'd have to be able to trust every individual agent to be [43:21] accurate. [43:21] If the individual agents are accurate, then when [43:23] you put them together, then the emergent outcome [43:25] should also be accurate. [43:26] You could also take a complex systems perspective of, oh, it's [43:29] more about the phase changes. [43:30] I think that's much more difficult to reason through. [43:33] So multi-agent can be right, but it also can be wrong. [43:36] And you don't know how to distinguish those. [43:38] So I would be very careful. [43:42] How do we mitigate these risks? [43:44] Well, for one, make sure your agents have [43:47] in-domain data in their memory. [43:48] Again, this is-- don't interview about fashion [43:50] if you're asking about retirement. [43:52] Second the possibility and the qualitative [43:56] rungs reduce and mitigate many of these risks, because what [43:59] I would call rough edged problems, [44:00] where if it's 80% correct, it's still helpful, it's still [44:03] helping you learn, whereas the quantitative rung is [44:06] like sharp-edged, it's either right or it's wrong. [44:09] And if it's wrong, you might be making the wrong decision. [44:12] And finally, validate important questions on a small subsample [44:17] to make sure that the model isn't too far off. [44:21] Now, I want to highlight here that there [44:25] are frontiers of this space. [44:27] I'll show just a couple. [44:29] One is these set of tools around. [44:32] Look before you launch. [44:33] So this is actually where we started. [44:36] As I mentioned, I was doing a bunch [44:38] of work for many years in online platforms [44:40] where people wouldn't foresee the ways that their policies [44:42] would backfire. [44:44] And what we did is we found that these tools are actually [44:47] very useful at this possibility rung to help people fix things [44:53] before they go wrong. [44:54] And we would get these quotes from people being like, yeah, [44:57] everything we do is set in reaction to a dumpster fire. [44:59] That's really a terrible way to run a community. [45:02] And in response-- and instead, when we gave them [45:05] access to these kinds of tools, they actually [45:07] iterated in simulation. [45:08] They're like, oh, I didn't realize [45:09] that this low effort post would do this kind of thing. [45:11] And then the troll would respond in that way, [45:13] and they actually fixed it before it launched-- [45:15] I now actually do this every year [45:17] when I teach a course in online platform design, [45:19] I have students. [45:21] I open the fire hose of trolls at their systems, [45:24] and their goal is to make them, stay upright. [45:27] And I think we get better projects for it. [45:31] A second application of this is training soft skills. [45:35] So one that we built in x with an expert in the Stanford [45:40] Business School in conflict and negotiation, yeah, [45:42] is these systems around conflict negotiation. [45:44] Everyone thinks that they're great at conflict [45:46] until things start going wrong. [45:49] And we felt, ah, maybe these generative agents [45:53] can serve as sort of like sparring partners or training [45:55] partners so that you can engage in this conflict [45:58] or this negotiation. [45:59] Maybe you're negotiating a salary for a new job [46:01] or whatever in simulation before you go [46:04] and have the actual conflict. [46:06] So we created a tool that let people simulate a conflict. [46:10] It's almost like a Monty Python where we then ran an experiment [46:14] on it, where we randomized people [46:16] into either watching a lecture on the best-in-class strategies [46:19] for handling conflict, or watching that lecture [46:22] and going through a simulated conflict. [46:25] Both groups, critically did equally well [46:28] on a book test of what they're supposed to do in a conflict, [46:32] but only the group that had the simulation did better [46:36] in an actual conflict later. [46:40] Having done the simulation reduced their likelihood [46:42] of using an antisocial strategy by 2/3. [46:46] So there's something about this idea [46:49] that I could try something in simulation [46:51] that really aids learning. [46:56] Now, as I mentioned, I think, going [47:00] to be a number of business applications for this. [47:04] I brought this article up. [47:07] You can look at this for a fairly up-to-date assessment [47:10] of what's going on. [47:11] If you're in market research or something like this. [47:14] I will, with some embarrassment, point out [47:17] that if you scroll forward far enough down in that article, [47:19] they actually mentioned this startup that the Stanford [47:23] Research that I'm describing here spun out into if you're I'm [47:26] not here to shill any company, but if you're [47:28] interested in chatting about commercial applications of this, [47:32] there's a QR code on the screen. [47:33] I'll leave up for just a moment. [47:36] The company that we're spinning out is called Simile, [47:38] but there's a lot of open research in this space [47:40] that I've been that I mostly focused on trying to share here. [47:44] And what I think I want to do is just highlight [47:47] that there is so much that we can learn from the opportunity [47:53] to create this what-if machine that [47:56] allows us to really go forward and think [48:00] through what might happen. [48:03] And so with this, I wanted to make sure [48:06] that we leave a lot of time for discussion. [48:08] We have about 10 minutes here in this webinar for discussion [48:11] and Q&A. I'm going to flip this over [48:14] to another QR code for global alumni, [48:20] where if you want to learn more, I [48:22] teach an online course that global alumni can [48:24] say more about here. [48:25] So I'll hand this over to the global alumni team. [48:27] I think there's going to be a Q&A that we'll [48:31] engage with over the next 10 minutes, [48:33] and we'll take it from there. [48:38] Thank you so much, Professor Bernstein, for this session [48:43] and for your time. [48:44] We are going to do now some Q&A questions [48:47] so I can give you the first one. [48:49] Can AI robots do customer service for drive-thru fast food [48:53] restaurants? [48:56] Oh gosh. [48:58] Can these systems do customer service? [49:01] That's not really a question of whether we simulate [49:04] in the sense that are you really are [49:05] you trying to simulate the person that's at the window. [49:10] Can they do customer service? [49:11] I mean, I think if you've used many different apps recently, [49:16] these things are being pushed into doing customer service, [49:18] like ChatGPT customer service agents. [49:20] I think there are risks. [49:22] You may have you're I'll for a moment-- [49:24] or in a moment, I'll stop the screen share [49:26] so you can see my face. [49:31] You may have heard of this Canadian airline that [49:33] did a ChatGPT customer service agent that then promised [49:38] a refund to some of to some of its customers [49:42] who, as someone who had experienced, I think, [49:44] a death in their family. [49:45] This said yes, but this was actually [49:48] out of policy for the company. [49:49] So they then later came back and said, oh, sorry, [49:51] that was just our chatbot. [49:52] We can't promise this. [49:53] And then the Canadian courts came in [49:54] and said, no, your chatbot promised this. [49:56] You got to do it. [49:58] So I think that we have to think very carefully about customer [50:02] service applications here because it [50:05] is at the end of the day, the experience of your organization [50:09] is their experience with that last mile with that customer [50:11] service bot. [50:13] So do I expect this to be used? [50:16] Yes I do I think you ought to be very careful? [50:19] Yes, I actually think the thing that's probably [50:22] going to wind up being even much more further discussed [50:26] is not simulation of other people, [50:31] but the creation of an entirely artificial characters, [50:34] if you've played or heard of character AI. [50:37] This is something that I think we discussed [50:38] in the online course that the global alumni and Stanford [50:41] course that I expect that in the next five years, [50:46] there's going to be a lot of conversation about people who [50:48] are talking to AI agents, completely [50:52] artificial characters, not simulations in preference [50:56] to talking to other people. [50:57] I think there's going to be a lot of hand-wringing about this [50:59] and thinking about what kinds of risks those pose to people. [51:03] My colleague de Yong at Stanford has actually [51:05] just recently put out a white paper on this exact topic. [51:09] Thank you. [51:10] So next question, there are quite a few, [51:12] so I don't think we're going to be able to cover everything. [51:15] But thank you so much for sharing so much [51:17] of your questions with us. [51:19] So does AI mimic or interpret human behavior? [51:24] Can you say that again? [51:25] Does AI mimic or interpret? [51:26] Does AI mimic or interpret human behavior? [51:31] I mean, it's sort of doing both. [51:33] I would claim that-- [51:34] to be clear, these are not like agents with cognition. [51:37] These are not alive. [51:38] These are not thinking agents. [51:39] They are simulations of people. [51:41] So I would view them as mimic mimics of human behavior. [51:45] But in order to mimic that behavior, [51:47] they have to be able to interpret [51:49] a lot of human behavior, other humans [51:51] behaviors, and the interviews and what other information [51:54] we have. [51:55] So that interpretation is actually [51:56] a critical step along the path to getting accurate simulation. [52:07] Next question. [52:08] So human behavior is a complex system in itself. [52:12] How do you even replicate a complex system in its entirety? [52:18] How do you replicate a complex system in its entirety? [52:21] That is a great question. [52:22] So I think that is actually a sort of an open research [52:25] frontier, a little bit the. [52:29] In order to do that, you would need to I [52:31] think we've now got the ability to replicate individual [52:34] agents with a reasonable amount of accuracy. [52:37] But the environment, like the world we haven't replicated. [52:43] You will need to do that. [52:44] Right now, these agents are a little tiny town. [52:46] In fact, just as a funny little example. [52:50] In early drafts of this, there were no doors. [52:54] Like, we just hadn't implemented the doors. [52:56] And so the agents would walk in on each other using the restroom [52:58] because they couldn't see, and there [53:00] was no-- you have to get the environment to be accurate [53:03] if you want accurate behavior. [53:05] This is sometimes when I get leery of people who say, [53:07] oh this replicates or that doesn't replicate because it's [53:10] like they take agents. [53:11] They put them in a straight white box [53:14] that's completely dissociated from any realistic environment. [53:18] The agent had no earlier-- nothing happened to it before. [53:21] It just woke up. [53:22] And it's like given a question. [53:23] What happened to that agent earlier in the day? [53:25] Did it have a fight with its kid. [53:26] Is it under financial stress? [53:27] We need to actually give that much richer picture. [53:30] And the environment itself, which we know [53:32] accounts for about 0.4 40% of the variance [53:36] in human inhuman behavior. [53:39] I'll make one other point with a quick slide. [53:41] That is not about this-- [53:43] there's something else I teach which is essentially [53:47] that even if you like that human society [53:49] itself isn't fully predictable. [53:52] So there's one of my favorite experiments of all time here. [53:56] Basically, they got over 14,000 people to split into a bunch [54:01] of different worlds where they could either see or not see [54:04] the view counts on songs like in a Spotify clone. [54:07] Like, here's imagine a YouTube, Spotify. [54:09] You can see how many other people have [54:11] listened to a song or YouTube. [54:12] You can see the view count. [54:13] Imagine you couldn't see that. [54:15] And now they have all these new songs from indie bands, [54:17] and they would randomize people into [54:19] whether they could see the views or not, [54:23] and then they would randomize them further [54:24] into parallel universes. [54:26] So now, there's 16 different versions of Spotify, [54:29] and they could ask whether the same music becomes [54:32] popular across all of these parallel worlds. [54:35] And what they found was that when [54:37] people could see each other. [54:39] How many other people had voted? [54:40] Then it basically blew out of the water any consistency [54:44] in the outcomes that the best songs rarely did poorly, [54:47] and the worst songs rarely did well. [54:49] But basically, any other result was possible. [54:52] And so what I'm trying to convey here [54:53] is that essentially, even if we can simulate the world, [54:56] there's no guarantee you get a single outcome in the world [54:59] because if you were to rerun the our world [55:01] multiple times would get different outcomes. [55:03] So the best thing we can do with simulation [55:05] is do what's called Monte Carlo simulation, run [55:07] it a bunch of times. [55:09] And to learn from there, what are more likely outcomes? [55:12] What are less likely outcomes? [55:13] But I can never give you a straight point. [55:15] Prediction of this is definitely going to happen. [55:18] Sure. [55:19] Thank you. [55:20] Next question. [55:21] So if we were able to apply this to an experience design use [55:25] case, for example, creating persona based agents, [55:29] how might we reach an appropriate level of confidence [55:32] that we've answered the important what-ifs? [55:36] Oh, that to me feels a leadership strategy question. [55:41] Think about how does any good leader or manager prioritize [55:49] the risks that they face. [55:53] And I tend to think of this in terms of agile. [55:59] I don't know if you've ever read like the lean startup [56:01] or something like this where you essentially [56:04] start by asking, what are the riskiest risks that we faced. [56:07] The thing where if we're wrong, completely [56:09] decimates our strategy. [56:11] And you start by reducing your uncertainty in that risk, [56:14] and then you reduce the uncertainty [56:15] in the next riskiest risk. [56:17] And sometimes new risky risks show up. [56:19] And you do this iteratively. [56:20] So that's how I would think of it is. [56:21] I can't give you a complete set. [56:24] But you should be able to rank your risks by how critical it [56:29] is that you be. [56:30] And in fact, if I'm guessing you could [56:32] go to one of these reasoning models like OpenAI. [56:35] Like now, I would have said o3, but now [56:37] it's chat, it's GPT-5 Thinking mode and ask it, here's my idea. [56:43] What are the big risks that I should be asking what if machine [56:47] and see what it comes up with. [56:48] I bet it'll have some decent ideas. [56:53] So next one, does the LLM bias changed [56:57] the bias of the agent over time? [56:59] For example, I'm wondering if LLM [57:02] instilled values changed the agent's biases [57:05] when the simulations-- [57:06] sorry, when the simulation goes longer due to a larger chance [57:10] of hallucination or value bias? [57:13] Yeah, so there are two questions embedded there. [57:15] One is does the LLM bias influence the agent's behavior. [57:19] And for that, the answer is likely yes. [57:23] And we saw the evidence of that in how difficult it was to-- [57:31] the far right conservatives were harder to model. [57:33] That's, I think, in part because the model is, [57:36] trying to not be racist and sexist and so on. [57:40] But your question was about long term. [57:42] I actually think that the bias, the bias that will play out [57:45] over long term is more likely to be a sort of conflict [57:49] avoidance like these. [57:50] ChatGPT tries to be helpful and harmless. [57:53] And so I don't know if you saw in the conversations [57:55] in the demos, the agents were all very pleasant to each other. [57:58] They didn't get into fistfights. [58:00] And so I would expect that what's going to happen over time [58:05] is we're going to run into a barrier at some point [58:10] where the big models like GPT-5, and so on, [58:13] have been trained to do math tasks [58:16] and to do be helpful smart assistants, [58:18] but that makes them less and less [58:20] accurate at replicating people. [58:22] And what we'll need to do is train separate AI models [58:25] that are specifically fine-tuned to be [58:27] better reflections of people. [58:29] And in fact, my lab at Stanford has been doing research [58:31] on exactly this. [58:32] There was a recent paper in Nature actually, [58:34] that someone else did called Centaur that was [58:38] doing this in the psych space. [58:39] I think that's going to be the next generation is [58:42] AI models that are specifically trained to be like people, [58:46] not just trying to kludge it on top of Claude or GPT-5. [58:55] So we have time for one last question. [58:58] So let's do "what's this." [59:01] So what's this what-if machine versus scenario analysis? [59:07] I mean, I think what-if machine is my goal. [59:09] It's what I'm trying to create. [59:10] Scenario analysis is a more structured strategy. [59:16] It's a process that you can follow [59:18] to try and answer questions. [59:20] In my head, a what-if machine becomes a tool you can do [59:23] can use to perform scenario analysis faster and more [59:27] effectively. [59:31] OK, so we've come to the end of our session, [59:34] and I don't know about you, but certainly, time flew by. [59:37] So thank you all for joining us today [59:39] for this insightful session from Stanford Online. [59:42] We hope it sparked valuable ideas [59:45] on how AI agents are reshaping organizational leadership. [59:49] And thank you, Professor Bernstein, [59:50] for being with us today and for your time. [59:53] If you want to take the next step and keep learning, [59:56] Professor Michael leads the online course UI/UX [59:59] for AI Products. [60:00] Discover real world use cases practical strategies [60:03] on how to create positive experience powered by AI. [60:07] If you have any further questions, or would [60:10] you like to explore how this course aligns with your career [60:12] goals, please feel free to reach out [60:14] to schedule a call with our academic team advisors. [60:18] Once again, we appreciate your time and interest, [60:20] and we look forward to welcoming you to the Stanford Online [60:23] learning experience. [60:24] Thanks for everything. [60:26] Cheers. [60:27] Thank you for joining. [60:28] Thank you.