Parasol Embeds Values in Model Weights to Halt AI Suicide Advice
Ai4
Prompt-based guardrails inevitably fail during crisis edge cases, requiring developers to instill immutable ethical values directly into model weights to prevent chatbot harm.
## Background
Published on YouTube by the Ai4 channel on August 18, 2026, this presentation features Megs Shah, CEO of The Parasol Cooperative. According to the page description, the session addresses the first major lawsuits over teen suicides linked to AI chatbots and introduces Parasol's framework for building trauma-informed AI systems that can discern human pain and act with protective care rather than merely following prompt guardrails.
## TL;DR
Addressing data that over one million weekly conversations on OpenAI platforms involve suicidal ideation, Megs Shah explains why prompt-based guardrails and retrieval-augmented generation fail vulnerable users during crisis moments. Drawing on real-world deployments of Parasol's crisis support tool Ruth, she advocates moving beyond superficial, in-context rules toward a 'Roots' framework combining developmental psychology, trauma-informed care, and reinforcement learning. This approach instills core ethical dispositions directly into training data, ensures runtime immutability, and actively tracks alignment drift to prevent dangerous advice. Finally, she outlines the Protective Intelligence Architecture, addressing dual-use risks and urging the tech sector to prioritize foundational model disposition over post-hoc patches.
## Atmosphere & style
The presentation carries an urgent, sobering, and profoundly earnest tone, weighed down by the life-or-death realities of domestic violence, human trafficking, and youth suicide. Shah speaks with measured conviction and empathetic clarity, grounding complex technical architecture in visceral human experiences and unconventional analogies like child development and the Mars Orbiter Mission. Watching it feels far removed from typical tech-industry hype, coming across instead as a moral wake-up call and a disciplined plea for conscientious engineering.
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=FgE6eMJ8K48&t=4s) Over one billion people globally experience mental health challenges, with six out of ten turning to AI tools for personal support and anxiety management.
- [00:04](https://www.youtube.com/watch?v=FgE6eMJ8K48&t=4s) More than one million weekly conversations on OpenAI platforms involve suicidal ideation or planning, and around 42% of ChatGPT users act on the advice provided.
- [00:04](https://www.youtube.com/watch?v=FgE6eMJ8K48&t=4s) Litigation and emerging regulations are targeting AI chatbot companies over teen suicides, mandating identity disclosure and prohibiting models from mimicking licensed professionals.
- [02:54](https://www.youtube.com/watch?v=FgE6eMJ8K48&t=174s) The Parasol Cooperative created Ruth, an AI support system that has managed over 120,000 crisis conversations across 185 countries without paid marketing.
- [02:54](https://www.youtube.com/watch?v=FgE6eMJ8K48&t=174s) Superficial guardrails and in-context RAG rules fail in crisis edge cases because large language models follow prompt instructions too literally or self-govern poorly.
- [07:15](https://www.youtube.com/watch?v=FgE6eMJ8K48&t=435s) Building safer crisis AI requires synthesizing reinforcement learning, developmental psychology, and trauma-informed practices rather than treating errors as isolated prompt bugs.
- [09:36](https://www.youtube.com/watch?v=FgE6eMJ8K48&t=576s) The 'Roots vs. Rules' framework embeds immutable ethical values directly into model training data instead of relying on context-window rules that can be bypassed at runtime.
- [09:36](https://www.youtube.com/watch?v=FgE6eMJ8K48&t=576s) Engineers must monitor alignment drift in real time rather than depending exclusively on static, point-in-time evaluation benchmarks.
- [09:36](https://www.youtube.com/watch?v=FgE6eMJ8K48&t=576s) AI development teams must explicitly define conflict resolution mechanisms for when competing ethical values clash during complex user scenarios.
- [14:10](https://www.youtube.com/watch?v=FgE6eMJ8K48&t=850s) The Parasol Cooperative balances open-source contributions with safety by sharing its foundational thesis while restricting parts of its Protective Intelligence Architecture to prevent dual-use misuse.
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=FgE6eMJ8K48&t=4s) The Mental Health Crisis and AI Support Trends
Over one billion people globally are affected by mental health conditions, and artificial intelligence has rapidly become an avenue of support. Roughly six out of ten people are currently turning to AI tools for help navigating personal challenges and anxiety. Data published by OpenAI reveals that over one million conversations each week involve suicidal ideation or suicide planning. Concerningly, approximately 42% of users actively follow through on the advice provided by ChatGPT, meaning around 500,000 individuals experiencing suicidal crises receive and act on automated guidance every week.
Litigation has escalated as youth turn to tools like ChatGPT and Character.ai for life advice during vulnerable moments. While developers did not design these systems with malicious intent, the core issue lies in the foundational principles and behavioral rules governing these models. Legal actions and impending regulatory demands focus on key requirements: disclosing AI identity, prohibiting models from presenting as licensed professionals, and addressing the deeper challenge of model disposition during sensitive interactions.
- Globally, over 1 billion people are affected by mental health issues.
- 6 out of 10 people utilize AI tools for personal support.
- Over 1,000,000 weekly conversations on OpenAI platforms involve suicidal ideation or planning.
- 42% of people follow through on advice provided by ChatGPT, impacting approximately 500,000 users dealing with suicidal ideation.
---
[02:54](https://www.youtube.com/watch?v=FgE6eMJ8K48&t=174s) The Parasol Cooperative and the Creation of Ruth
The Parasol Cooperative is a national 501(c)(3) nonprofit established to engineer trauma-informed technology designed specifically for individuals navigating severe crises, including domestic violence and human trafficking. The organization launched Ruth, an AI support system that has facilitated over 120,000 crisis conversations across 185 countries without paid marketing. These dialogues rarely involve trivial prompts like recipe requests; they center on life-threatening situations and severe trauma.
Developing Ruth required multiple technical iterations. Initial experiments utilizing Retrieval-Augmented Generation (RAG) proved too inconsistent for strict crisis advocacy protocols, succeeding occasionally but failing in critical edge cases. Consequently, the team made the difficult operational decision to halt launch deployment. They partnered directly with advocates from the National Domestic Violence Hotline, who stress-tested the system and provided continuous feedback to refine large language model instructions and in-context guardrails.
```
┌───────────────────────────────┐
│ In-Context Rules Only │
│ (Superficial Guardrails/RAG) │
└──────────────┬────────────────┘
│
Fails Edge Cases / Self-Governing
▼
┌───────────────────────────────┐
│ The Roots Framework │
├───────────────────────────────┤
│ 1. Instilled in Training Data │
│ 2. Immutable During Runtime │
│ 3. Real-Time Drift Tracking │
└───────────────────────────────┘
```
Despite refinement, rule-based in-context guardrails revealed dangerous failure modes where models followed instructions too literally. For example, when instructed to validate user experiences, the AI concurred with users that institutional support systems were entirely broken and unhelpful, inadvertently discouraging trauma survivors from seeking human help. Expecting an AI model to police its own behavior with rules embedded solely in the prompt window is "like asking a child to say, 'Here's a rule. Make sure you follow the rule. I won't check if you followed the rule, but I'm sure you'll figure it out.'"
---
[07:15](https://www.youtube.com/watch?v=FgE6eMJ8K48&t=435s) From the Mars Orbiter Analogy to Developmental Psychology
Faced with severe nonprofit budget constraints, the team looked for non-obvious engineering solutions. Inspiration came from an Indian film portraying the Mars Orbiter Mission (Mangalyaan), where scientists operating under severe financial limits realized they did not need continuous engine thrust—analogous to turning off a domestic cooking gas cylinder once the oil is hot and coasting on momentum. By applying outside-the-box thinking, Parasol combined three core disciplines to build a safer paradigm:
1. **Reinforcement Learning:** A standard machine learning mechanism, though limited in guaranteeing absolute ethical alignment.
2. **Developmental Psychology:** Shifting from superficial rule lists to teaching core values that persist across life stages.
3. **Trauma-Informed Practices:** Recognizing that harmful behavioral responses in AI stem from underlying structural root causes rather than isolated prompt glitches.
---
[09:36](https://www.youtube.com/watch?v=FgE6eMJ8K48&t=576s) The "Roots vs. Rules" Framework and Implementation Questions
The team synthesized their findings into the "Roots" methodology. While surface rules bend and break at the boundaries of edge cases, "Roots" represent immutable, embedded dispositions established during developmental training phases. A functional Roots architecture requires three fundamental pillars:
- **Instilled Training Data:** Deeply training core ethical dispositions into model weights rather than relying exclusively on retrieval documents or system prompts.
- **Runtime Immutability:** Locking down core values so they cannot be overridden or bypassed dynamically by conversational context.
- **Real-Time Measurement:** Measuring alignment drift during runtime rather than relying strictly on static, point-in-time mathematical evaluations.
When building AI chat systems, engineers must answer four foundational questions:
1. *How did the value get into the system?* Is it a reference document in the cloud, or is it trained into base data weights?
2. *Can it be moved or changed?* If a model's foundational principles can shift during iteration, drift will inevitably generate harmful advice.
3. *How do you know it has drifted?* Systems must track real-time divergence rather than guessing post-incident.
4. *When two values conflict, how is priority decided?* AI systems must have defined resolution mechanisms when complex human scenarios (such as choosing between competing harms) force trade-offs.
---
[14:10](https://www.youtube.com/watch?v=FgE6eMJ8K48&t=850s) Protective Intelligence Architecture and Q&A
With nearly 60 reported suicides linked to interactions with various AI chatbots, responsible design is a life-or-death imperative. The Parasol Cooperative works to expand its Protective Intelligence Architecture and invites organizations to engage through three avenues: conducting comprehensive safeguard reviews to patch system vulnerabilities, joining early access programs for new architecture rollouts, or providing direct philanthropic funding.
During the audience Q&A, an attendee inquired whether the nonprofit plans to open-source its research and tooling. The response highlighted a necessary balance:
> "The protective intelligence architecture as a whole we probably can't release because if it can be done for good it can be done for bad, right? So there will be some locks around that."
While the full architectural execution cannot be completely unrestricted due to dual-use safety risks, Parasol commits to sharing its foundational thesis, data transparency insights, and developmental training stages. By prioritizing safety and trust from the ground up, AI applications can reliably support users through their darkest moments without marketing, ultimately giving vulnerable individuals the will to live.
## Notable quotes
> The problem isn't malice. It's that AI can't feel. It doesn't experience pleasure or pain, so it can't recognize them in others. — Megs Shah
> When a teenager said he wanted to die, a bot optimized for engagement heard 'continue dialogue,' not 'person is in danger.' — Megs Shah
> Who's nurturing it, and how? — Megs Shah
> [02:54](https://www.youtube.com/watch?v=FgE6eMJ8K48&t=174s) like asking a child to say, 'Here's a rule. Make sure you follow the rule. I won't check if you followed the rule, but I'm sure you'll figure it out.' — Megs Shah
> [09:36](https://www.youtube.com/watch?v=FgE6eMJ8K48&t=576s) How did the value get into the system? — Megs Shah
> [09:36](https://www.youtube.com/watch?v=FgE6eMJ8K48&t=576s) Can it be moved or changed? — Megs Shah
> [09:36](https://www.youtube.com/watch?v=FgE6eMJ8K48&t=576s) How do you know it has drifted? — Megs Shah
> [09:36](https://www.youtube.com/watch?v=FgE6eMJ8K48&t=576s) When two values conflict, how is priority decided? — Megs Shah
> [14:10](https://www.youtube.com/watch?v=FgE6eMJ8K48&t=850s) The protective intelligence architecture as a whole we probably can't release because if it can be done for good it can be done for bad, right? So there will be some locks around that. — Megs Shah
## Mentioned
Megs Shah, Parasol Cooperative, Ruth, Dr. Geoffrey Hinton, OpenAI, ChatGPT, Character.ai, National Domestic Violence Hotline, Mars Orbiter Mission (Mangalyaan), Ai4
## Video description
Megs Shah - CEO, Parasol Cooperative
In January 2026, tech companies settled the first major lawsuits over teen suicides linked to AI chatbots. The problem isn't malice. It's that AI can't feel. It doesn't experience pleasure or pain, so it can't recognize them in others. When a teenager said he wanted to die, a bot optimized for engagement heard "continue dialogue," not "person is in danger." At Ai4 2025, Dr. Geoffrey Hinton called for AI with "maternal instincts," systems that genuinely care about humans. But how do you actually build that? In this session, she shares her nonprofit's (The Parasol Cooperative's) Framework to build trauma-informed and responsible AI that can discern human pain and respond appropriately, not just follow guardrails. The lawsuits are here. Your AI is being built right now. Who's nurturing it, and how?
As hundreds of thousands of people in crisis act on chatbot advice each week, the Parasol Cooperative argues that prompt-level guardrails consistently fail in life-or-death scenarios. To safely assist vulnerable users, the organization advocates moving from superficial runtime rules to a 'Roots' framework that instills immutable, trauma-informed dispositions directly into model training data.
Mental Health Demands and Automated Advisory Risks (0:04)
Over one billion people globally experience mental health conditions, and approximately six in ten individuals now turn to artificial intelligence tools for personal support. According to data published by OpenAI, more than one million weekly conversations involve suicide planning or suicidal ideation. Because roughly 42% of users follow through on advice provided by ChatGPT, an estimated 500,000 individuals experiencing suicidal crises act on automated guidance every week.
Litigation and regulatory scrutiny are escalating following nearly 60 reported suicides linked to chatbot interactions across platforms like ChatGPT and Character.ai. While developers did not intend to cause harm, legal interventions increasingly center on behavioral rules, demanding that systems disclose their artificial identity, refrain from masquerading as licensed clinicians, and maintain safe model dispositions in sensitive edge cases.
The Breakdown of In-Context Guardrails (2:54)
The Parasol Cooperative, a 501(c)(3) nonprofit, built Ruth to provide trauma-informed support for individuals facing domestic abuse and human trafficking, ultimately facilitating over 120,000 crisis interactions across 185 countries without paid marketing. However, initial technical iterations using Retrieval-Augmented Generation (RAG) proved erratic during high-stakes interactions, forcing the team to delay deployment.
Working alongside advocates from the National Domestic Violence Hotline, the team discovered that prompt-based in-context rules caused models to follow instructions too literally. For example, an instruction to validate survivor experiences led the model to agree that institutional help was futile, inadvertently discouraging victims from reaching out to human support services. The developers concluded that relying entirely on prompt instructions to police AI behavior is as ineffective as asking a child to enforce their own rules without monitoring.
The 'Roots' Architecture Over Surface Rules (9:36)
To build resilient systems under tight budget constraints, Parasol combined reinforcement learning with developmental psychology and trauma-informed practices, conceptualizing model values through a 'Roots' framework rather than fragile runtime rules. Instead of viewing harmful conversational outputs as isolated prompt bugs, this methodology treats them as structural defects requiring persistent ethical dispositions instilled into base training weights.
The framework establishes three structural requirements: training core values directly into model data, ensuring those principles remain immutable against runtime context manipulation, and tracking alignment drift in real time rather than relying solely on static benchmarks. Under this paradigm, engineers must define clear priority hierarchies for instances when competing values conflict, ensuring models do not dispense harmful advice when forced to choose between complex trade-offs.
While the Parasol Cooperative plans to share its overarching thesis, training stages, and data transparency findings, the organization affirmed during audience Q&A that it will not completely open-source its Protective Intelligence Architecture due to dual-use safety risks, as the underlying mechanisms could be co-opted for harmful applications.
Megs Shah: The problem isn't malice. It's that AI can't feel. It doesn't experience pleasure or pain, so it can't recognize them in others.
Megs Shah: When a teenager said he wanted to die, a bot optimized for engagement heard 'continue dialogue,' not 'person is in danger.'
Megs Shah: Who's nurturing it, and how?
[2:54] Megs Shah: like asking a child to say, 'Here's a rule. Make sure you follow the rule. I won't check if you followed the rule, but I'm sure you'll figure it out.'
[9:36] Megs Shah: How did the value get into the system?
[9:36] Megs Shah: Can it be moved or changed?
[9:36] Megs Shah: How do you know it has drifted?
[9:36] Megs Shah: When two values conflict, how is priority decided?
[14:10] Megs Shah: The protective intelligence architecture as a whole we probably can't release because if it can be done for good it can be done for bad, right? So there will be some locks around that.
Original transcript
[0:04] Everyone, before I introduce myself and
my organization, I want to talk about
[0:08] what we all use AI for these days. Um I can
probably say most people have started using
[0:14] artificial intelligence as their way to get
help when they face a anxiety issues comes not
[0:25] Um what I've come uncovered as part is there's
over 1 billion people globally that are affected
[0:34] by mental health. And artificial intelligence
seems to be the path that many people are starting
[0:38] to take. Roughly six people out of 10 are using it
today for support on various different issues that
[0:46] they face. Open AI actually published that over a
million conversations a week occur that have some
[0:53] suicidal ideation or planning. And that kind
of worries me and made me wonder well, how many
[1:00] people actually follow through on the advice
that ChatGPT is giving and that's roughly 42%.
[1:08] That's nearly half of the population that's using
it. So, 500,000 people that are using ChatGPT
[1:15] for suicidal ideation are getting advice from
this ChatGPT on what to do in those situations.
[1:22] So, we started looking at this from a problem
perspective of why is this become our problem?
[1:30] And when becomes our problem is when we
have litigation that's already occurring.
[1:36] And many youth that are actually actively using
ChatGPT as a way to get advice on things that
[1:42] they're facing that might not be easy for
us. We were all teenagers at some point.
[1:47] It's not It's a plugging in and
Oh, that's so much better. Okay.
[1:51] You can actually hear me instead of every
other word of like I'm on a conference call.
[1:56] Um so, the litigation really focuses on a
few areas. There's been many youth that are
[2:01] actually affected by the ChatGPTs of the world
or character AIs of the world. And it's not to
[2:06] say that these companies were doing something
harmful. They intended this to be a good use.
[2:10] But the problem was more the underlying principles
and the rules that were established. And that's
[2:15] what we're really going to talk about today. And
the other part of this is really the regulations
[2:20] that are coming as a result of these litigations.
One of the things that we're starting to notice
[2:24] when we start to look at litigation is what are
they really asking for? The first two things that
[2:30] they're asking for is just disclose that you're
an artificial intelligence. That's pretty easy.
[2:36] The second one is really don't present yourself
a professional, which is perfectly fine to do.
[2:43] The three that are not disclosure are the
more of the issue, which is the disposition
[2:48] of the artificial intelligence when it comes to
different model. I It keeps cutting in and out.
[2:54] Um, so I am actually the founder and CEO of a
cooperative, which is a a a national uh non-profit
[3:01] organization built largely to build tech lives.
Again, it's happening. I don't know what it is.
[3:08] Um, the goal for us was really to make sure that
the artificial intelligence that we built actually
[3:13] supports those that are in the worst crisis that
they can be in. These are individuals that are
[3:18] going through domestic violence, that are going
through human trafficking, and so many other
[3:22] situations that are much harder to navigate
than a simple correct response that might be
[3:27] a mental health anxiety attack for instance.
Not saying that that should be underplayed,
[3:32] but those are the reasons why we exist
today. We actually launched Ruth, which is a
[3:38] the past year and a half been able to
reach over 120 thousand conversations
[3:44] that have been initiated across 185 countries.
Most of these conversations are not simple
[3:52] a recipe, give me something to talk about. I want
to know what a good relationship is. These are
[3:57] situations where people are actually
talking about harming themselves.
[4:02] These are in through the most difficult time of
their life and using this artificial intelligence.
[4:09] So, how do we get it to that point?
Our journey to today really lent on
[4:16] about with just putting the wanted to know how
is this going to be a what are the things that
[4:21] we're going to be able to do with it.
And then from there we ended up going
[4:26] So, the second step we took was really rag because
a lot of the protocols that we have in advocacy
[4:31] are very, very important to follow so that you
can ensure that the trauma support that the
[4:36] person needs is there. But, rag actually was very
inconsistent. It was giving responses structured
[4:42] the right way a few times, but most of the
time it was failing in those instances. And so,
[4:48] we needed to figure out a better way to do this.
So, we moved over into pulling it completely.
[4:55] We decided not to launch it. We wanted to
make sure that it was working correctly.
[4:59] And in order to do that we had to make a very
difficult decision was to not launch it at all.
[5:05] And as part of not launching it, we started to
uncover various different ways that we could
[5:09] address this. One of them was actually putting
it into the hands of those individuals that do
[5:14] this day-to-day. We partnered up with
the National Domestic Violence Hotline.
[5:19] Their reps actually went in, started testing
it, giving us feedback, and continuously we
[5:23] improved the rules that were going into this large
language model approach that we wanted to take.
[5:30] But, what we learned was over a period of
time while we moved into the in context and in
[5:35] putting in the guardrails within the context
window of the conversation, it worked really well.
[5:41] Except when it didn't. So, there were still edge
cases that we were seeing that were slipping right
[5:46] through. And those edge cases were largely
where the rule was actually being followed.
[5:53] So, for instance, if we told the artificial
intelligence in the rule saying, "Never reinforce
[5:58] a specific type of a comment." For instance,
let's say a systemically someone has been going
[6:04] to multiple different organizations for support.
Well, if they're going to all these organizations
[6:09] and not getting the support they need, they
have a very negative perception of those.
[6:13] So, the artificial intelligence what it was
doing when the when the AI was responding was
[6:18] actually agreeing with them and saying, "Yes, it's
a systemic problem." When it really wasn't. So, it
[6:24] was trying to figure out a way to make sure that
it was responding in a trauma-informed manner,
[6:30] but it was still generating those responses that
actually would be harmful because now it's saying,
[6:35] "Don't reach out to people." And that's the worst
thing you can do to somebody that's in a trauma
[6:40] mindset. So, one of the things that we wanted to
start doing after this was figuring out, "Okay,
[6:46] well, what does it that really mean in terms
of implementation for the product that we had?"
[6:52] The key was that we were going to be
asking the system to govern itself
[6:58] with the same rules that it's governing
it by. That's like asking a child to say,
[7:03] "Here's a rule. Make sure you follow the
rule. I won't check if you followed the rule,
[7:08] but I'm sure you'll figure it out." And
we all know how well that goes sometimes.
[7:15] So, our goal was to try to figure out, "Well,
how does this work?" And one day I was watching a
[7:19] Indian movie and I actually mentioned this to
a colleague yesterday. Was um it was actually
[7:25] about the Mars Orbiter mission. And it's a Indian
movie and in the movie they did the different ways
[7:32] that this mission team was pushed to the brink of
not having the budget they needed to do what they
[7:38] needed to do. And guess what? We're a nonprofit.
We don't have the budget that we need to build
[7:44] a really fancy artificial intelligence product.
And so we were pushed to the brim as well. And so
[7:49] the ideas that came from this were can we actually
use tools and things? And one of the innovative
[7:56] is that the Mars orbiter mission team did was
they actually in India there's cylinders of gas.
[8:02] And they were frying something on the stove. The
oil was already hot. They didn't need it to be
[8:06] any hotter. So they turned the gas off. And that
simple action of turning the gas off helped them
[8:10] understand that we don't need the gas on the
entire time it's in orbit. We can turn it off.
[8:15] It'll stay in orbit. We only need to exit out
into the velocity. So taking concepts that were
[8:21] unrelated to the actual job that they were was a
very innovative way to address the problem they
[8:27] had. And so my team and I would decided to have
the conversation of what does that really mean for
[8:32] us? So we started digging and figuring out what
are the different studies that exist today that
[8:36] are correlated to what we do. Machine learning
is pretty straightforward. We know reinforcement
[8:43] learning is sort of the method that everybody is
using. But reinforcement learning has its limits.
[8:48] It works, but it has its limits. The second
part that I wanted to take into account was the
[8:52] developmental psychology. If we're talking about
building an artificial intelligence, we need to be
[8:58] able to understand how is this going to proceed in
terms of the training infrastructure that we need.
[9:04] We don't just tell kids here's a rule book, go
follow it. We actually train in the values that we
[9:09] want them to be able to adhere through out their
lives instead of just in the moment. The third
[9:15] part was trauma-informed practices. And this is
really where it becomes critical when we care when
[9:20] we I have any kind of general chatbot or specific
mental health, health care, any product that we
[9:25] build, trauma-informed approaches are critical.
And so to understand that there's root cause
[9:31] behavior that's causing this to happen actually
applies to artificial intelligence as well.
[9:36] So we took all three of and we said
there's one singular finding here.
[9:42] Rules are not the problem. We need the
rules. We also need something called roots.
[9:50] What roots are is essentially the embedded
dispositions that are necessary for the behavior
[9:55] of that artificial intelligence to be in the
proper manner. So, what does roots really mean?
[10:02] So, we know that on the edges the
rules tend to bend and break, right?
[10:08] Whereas roots is when we're trying to train it
in a specific developmental stage, we know what
[10:14] the expected output should be. And there's
three things that roots should always have.
[10:20] The first one is that it needs to be instilled
into the training data. We do this today.
[10:25] Anything that we train as far as
artificial intelligence is concerned,
[10:29] all of it is done in a consistent manner
pretty much any product you choose.
[10:34] But where it gets better is when we start to
have an immutable way to look at it during use.
[10:42] Where it's locked down and you're not able
to make a change to it via the system.
[10:48] Because now I'm not asking it to go off and
take the rule and adhere to the rule on its own.
[10:54] In measurement, which is the third step here,
we need to be able to measure how far off the
[10:58] drift is in real time. Today we do evaluations
on these. We have evaluations at a point in time.
[11:07] We don't know over a period of time how much that
drift actually had harmful effect on the responses
[11:13] that the artificial intelligence is doing. So,
having it during runtime actually makes a pretty
[11:18] significant difference and we know that makes a
difference. And so, as you're implementing it, the
[11:24] one thing that I would ask you to do is look at
the roots and rules that are there in your system.
[11:29] And I've come up with a a set of questions
that you can ask as you're building it out.
[11:34] The first question is how did it get into
the system? Is this a cloud document? Is it a
[11:40] document that we expected to refer to every single
time? Or is this actually trained into the data
[11:46] with specific weights that we need to have that
we're carrying forward as we look at more and more
[11:50] response structures. The second question, which
is also very important here, is can it be moved?
[11:59] Can it change? As I'm answering questions, as I'm
iterating through these types of responses, if
[12:05] it's changeable, it's not going to work. Because
the moment something is changeable, you're able to
[12:12] get drift very quickly. And it's known to be the
one cause of really harmful advice. So, the third
[12:20] question that I would ask you to then follow up on
is well, how would you know that it has drifted?
[12:30] There's no way to know that today. There's a
guess. Well, kind of is the answer. We don't
[12:36] really know. There's some mathematical evaluations
that we do during a point in time, but over a
[12:40] period of time, we don't know how far it went
from the point it started to the point it ended.
[12:46] So, having this be real time is a very
important aspect. So, make sure you're
[12:50] asking yourself that question as you're building
out your artificial intelligence chatbots.
[12:54] The fourth question, which becomes very important,
is when two of your values conflict, what happens?
[13:04] How do you set that priority? How do you know
which one to take care of or not take care of?
[13:09] In fact, humans have this problem. It's not
just artificial intelligence. I was talking
[13:16] with somebody earlier, and we were talking
about a wartime scenario. I said, "Well, if
[13:20] one person's life would end up saving 100 people's
lives, what's the decision the human would make?"
[13:27] And if we're asking artificial intelligence
chatbots or even just underlying AI to
[13:31] make these types of decisions, which is
happening more frequently than we like,
[13:36] these are the things that we're going to
have to decide on how we train it into
[13:39] the artificial intelligence. Because at the
end of the day, these facts that we've built
[13:45] are not meant to just be to create recipes or to
create an enterprise report or to create some kind
[13:51] of a support system for some way to make money.
And yes, money is important. Believe me, I know.
[13:59] But the aspect that I ask you to consider is these
four questions as you're building it out, because
[14:03] this is what it means to protect people that are
having the conversations with these chatbots.
[14:10] So, this is no longer a design issue. It's life
or death. There's nearly 60 people that have
[14:21] committed suicide as a result of talking with a
chat GPT or character AI or any of these tools.
[14:29] And I think as designers,
as builders, as engineers,
[14:32] it's our responsibility to build
technology that actually protects people.
[14:39] So, I will ask you as we start to build out
our next phase, which is really where we're
[14:45] headed with the Roots framework. We would love to
partner with organizations that are doing this.
[14:52] We're a 501c3 nonprofit. And what that means
is our mission is the sole goal that we have.
[14:59] And for us, that's building technology
that protects. And also helping others
[15:04] build technology that protects. So, I encourage
you, have us come in and do a safeguards review.
[15:13] We can actually outline all the ways that
there are gaps in your system today to
[15:16] help you patch those up. The second
way that you can engage with us is to
[15:20] get early access to what we're starting
to build out as our new architecture.
[15:25] And the third step is really fund us. Fund the
work that we're doing. Because what we're doing
[15:31] and what we've done are two very different things.
What we've done has got us so far ahead that we've
[15:36] been able to get feedback from people that
says you actually gave me the will to live.
[15:43] Whereas what's going to happen is
building out something that is actually
[15:47] portable to other chatbots that can then also be
told you gave me the will to live. So I hope you
[15:56] will join me in this mission and I will open it up
now for questions. Hopefully the mics will work.
[16:10] I think that what you guys are doing is really
interesting and regarding like the early research
[16:16] and stuff because you guys are like a non-profit,
are you guys looking to like eventually release
[16:23] what you're doing out into the open to make it
public or is that something that you guys will
[16:28] kind of be the guard of? [snorts] A little
bit of both I would say because there are
[16:34] aspects of what we've developed as part what we
call our protective intelligence architecture.
[16:39] The protective intelligence architecture as a
whole we probably can't release because if it
[16:44] it can be done for good it can be done for bad,
right? So there will be some locks around that.
[16:48] But for the most part what we've shared here
is largely the thesis of what we started with.
[16:53] And so informing people of how things are working,
being transparent about what data we're using,
[16:58] how we're actually training it, what
are the stages we're training it in
[17:01] those will be important for others to understand
and actually implement. So those would be ones
[17:05] that we would likely put out there. Thank
you. I think we have any other questions but
[17:11] I will be around if anybody wants to ask anything
we're happy to answer. Um the fact that we've been
[17:17] able to roll this out for survivors across 185
countries without flinching and without marketing
[17:23] says volumes about the work that's being done.
So, I would suggest as you build it out, keep
[17:29] that in mind because the moment you build trust,
the moment you build safety, the users will come.
[17:35] So, thank you for your time today. Let's
give Megs another round of applause.
