PromptQL Enterprise AI Deployments Reveal Reliability Debt Stalls Adoption
Ai4
Enterprise AI projects fail to deliver economic value because models lack explicit confidence signaling and fail to incorporate unwritten organizational tribal knowledge.
## Background
Published on YouTube by the channel Ai4 on August 29, 2025, this video features a presentation by Rajoshi Ghosh, Chief Ecosystem Officer and Founder of PromptQL. The page description explains that the talk addresses why enterprise AI agent projects stall due to accuracy issues and outlines strategies for building reliable systems by leveraging company-specific data, workflows, and tribal knowledge.
## TL;DR
Rajoshi Ghosh explains that accuracy debt and the absence of explicit reliability signals are the primary reasons enterprise AI initiatives fail to transition from pilots to production. Generic off-the-shelf LLMs struggle with unwritten organizational context and domain-specific terminology, making absolute initial accuracy on messy data unrealistic. PromptQL addresses this challenge using an Accuracy Flywheel architecture that combines native confidence signaling with human steering to capture tribal knowledge and refine system logic. Real-world implementations demonstrate how this approach improves sales forecast reliability, reduces healthcare scheduling times, and speeds up enterprise SaaS customer onboarding.
## Atmosphere & style
The presentation carries a professional, authoritative, and practical tone driven by real-world enterprise experience. Ghosh communicates with clear clarity and confidence, framing complex AI integration challenges into structured architectural concepts and actionable tradeoffs. Watching the presentation feels like a focused, insightful executive briefing that bridges technical system design with tangible business metrics and operational realities.
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=tHS5Z3m2Pd4&t=4s) The rapid growth and high demand for enterprise AI drive executives to reallocate funds from non-dedicated budgets to secure AI solutions.
- [01:06](https://www.youtube.com/watch?v=tHS5Z3m2Pd4&t=66s) Accuracy debt and a lack of explicit reliability signals serve as the main barriers holding back widespread enterprise AI adoption.
- [01:06](https://www.youtube.com/watch?v=tHS5Z3m2Pd4&t=66s) Generic off-the-shelf LLMs fail in complex business environments because they lack organizational context and implicit tribal knowledge.
- [02:13](https://www.youtube.com/watch?v=tHS5Z3m2Pd4&t=133s) Ilya Sutskever highlighted that a lack of AI model reliability is the single primary bottleneck to achieving full economic value.
- [04:23](https://www.youtube.com/watch?v=tHS5Z3m2Pd4&t=263s) Building trust in enterprise AI requires models to explicitly signal uncertainty, mirroring how human colleagues communicate confidence levels.
- [04:23](https://www.youtube.com/watch?v=tHS5Z3m2Pd4&t=263s) The Accuracy Flywheel creates a continuous learning loop where domain experts correct low-confidence AI outputs to permanently update system logic.
- [07:24](https://www.youtube.com/watch?v=tHS5Z3m2Pd4&t=444s) Waiting for pristine data cleanup delays deployments by 6 to 9 months; starting with native reliability signaling allows safe deployment on messy data.
- [09:30](https://www.youtube.com/watch?v=tHS5Z3m2Pd4&t=570s) Enterprise AI architecture requires a separation of concerns, letting domain experts focus on evaluation sets while technical platforms manage infrastructure.
- [13:50](https://www.youtube.com/watch?v=tHS5Z3m2Pd4&t=830s) PromptQL demonstrates reliability by warning users with yellow confidence signals when key business definitions (like churn risk or VIP criteria) are missing.
- [16:12](https://www.youtube.com/watch?v=tHS5Z3m2Pd4&t=972s) Repeatability—receiving identical outputs across multiple identical queries—is a fundamental benchmark for measuring AI agent accuracy.
- [17:22](https://www.youtube.com/watch?v=tHS5Z3m2Pd4&t=1042s) Solving data visibility in fragmented Salesforce accounts helped a 25,000-person sales force mitigate a 10% risk to revenue forecasting.
- [17:22](https://www.youtube.com/watch?v=tHS5Z3m2Pd4&t=1042s) Automating complex MRI scheduling and procedure code matching reduced handle times from 12 to 6 minutes, yielding a $50 million economic impact.
- [19:01](https://www.youtube.com/watch?v=tHS5Z3m2Pd4&t=1141s) Automating database schema mapping with AI agents drastically accelerates enterprise customer onboarding timelines.
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=tHS5Z3m2Pd4&t=4s) Introduction and the Momentum of Enterprise AI
Rajoshi Ghosh, Chief Ecosystem Officer and Founder of PromptQL, opens the presentation by introducing herself and reflecting on her decade-long background in building and selling enterprise B2B software. Over the six months following PromptQL's launch, demand for enterprise AI solutions reached a fever pitch. PromptQL rapidly closed seven-figure contracts with Fortune 500 companies, followed by immediate account expansions and rapid growth.
She highlights a prevalent dynamic in the current enterprise landscape: business leaders are eager to drive AI-powered transformation, yet formal budget line items have often not caught up. As a result, executives are reallocating funds from disparate budgets to procure effective AI capabilities. To address this demand, Ghosh frames her talk around sharing key lessons from early enterprise deployments on why projects succeed or stall.
- **Founder background**: 10+ years of experience building and selling enterprise B2B software.
- **Commercial traction**: Closed multiple 7-figure deals with Fortune 500 enterprises within months of launching PromptQL.
- **Enterprise budget dynamics**: High urgency to adopt AI drives business leaders to pull funds from various non-dedicated budgets.
```
ENTERPRISE AI MOMENTUM
┌─────────────────────────────────────────────────────────────┐
│ High Demand for AI Transformation ──> Lack of Fixed Budget │
└──────────────────────────────┬──────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Executive Budget Reallocation ──> PromptQL 7-Figure │
│ Across Departments Enterprise Deals │
└─────────────────────────────────────────────────────────────┘
```
[01:06](https://www.youtube.com/watch?v=tHS5Z3m2Pd4&t=66s) The Two Core Insights: The Accuracy Trap and Custom AI
Before diving into the supporting material, Ghosh distills her experience into two core insights that define enterprise AI adoption:
1. **Accuracy is a Trap**: Enterprise leaders invest heavily in data preparation and infrastructure, expecting high accuracy out of the box. However, when systems lack explicit accuracy signals or generate confidently wrong answers, trust breaks down, halting enterprise-wide adoption.
2. **The Need for Custom AI and Tribal Knowledge**: Standard off-the-shelf LLMs cannot succeed in a complex enterprise environment. Models must be custom-tailored to incorporate the specific data, process workflows, and unwritten "tribal knowledge" unique to each organization.
- **The Accuracy Barrier**: Accuracy debt stands as the single biggest factor holding back AI integration inside enterprises.
- **The Off-the-Shelf Limit**: Generic models lack organizational context and cannot navigate implicit business terminology without customization.
[02:13](https://www.youtube.com/watch?v=tHS5Z3m2Pd4&t=133s) The Accuracy Debt Problem and Citing Ilya Sutskever
Ghosh illustrates the accuracy problem using a concrete healthcare enterprise example. If a user asks an AI system to *"pull up patient readmissions rates for DRG470"*, a human colleague with domain experience easily understands what "DRG470" signifies. Tenured employees rely heavily on implicit knowledge—unwritten terminology, informal rules, and "water cooler conversations"—that is rarely documented in metadata or data catalogs. When generic AI encounters messy enterprise data and ambiguous business terms, it risks becoming "confidently wrong." Business leaders cannot take unverified AI outputs and present them to a board of directors or report them on public earnings calls.
To reinforce the impact of model reliability, Ghosh references a 2023 interview where Dwarkesh Patel asked OpenAI co-founder Ilya Sutskever (now founder of Safe Superintelligence) what might cause AI to fail to deliver its expected economic impact. Sutskever identified **reliability** as the single primary bottleneck. If workers cannot trust model outputs and must manually verify every answer, the AI fails to generate real economic value.
- **Enterprise domain example**: Medical classification codes like "DRG470" require domain-specific context that generic models lack.
- **The danger of hallucinatory confidence**: High statistical confidence coupled with incorrect logic destroys executive trust.
- **Ilya Sutskever quote context**: On Dwarkesh Patel's podcast, Sutskever emphasized that lack of reliability by 2030 would be the main blocker to AI achieving its full economic potential.
[04:23](https://www.youtube.com/watch?v=tHS5Z3m2Pd4&t=263s) Human Trust vs. AI Trust: The Accuracy Flywheel
To understand how to make AI reliable, Ghosh compares AI behavior to human workplace trust. Humans are rarely 100% accurate, yet organizations function effectively because people know their own limits. A trusted human colleague provides best-guess answers while explicitly signaling uncertainty: *"I looked at the customer tables, but the data was messy, so I'm not 100% sure."* Building impactful AI for mission-critical enterprise data requires reproducing this explicit confidence signaling.
Ghosh introduces the concept of **AcmeQL** (a placeholder for an enterprise's custom AI model) operating on an **Accuracy Flywheel**:
```
THE ACCURACY FLYWHEEL
┌─────────────────────────────────────────────────────────────┐
│ 1. AI Execution & Native Reliability Signaling │
│ Model flags low-confidence answers or missing context. │
└──────────────────────────────┬──────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ 2. Human Feedback & Steering │
│ Domain experts correct assumptions & supply missing │
│ tribal knowledge. │
└──────────────────────────────┬──────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ 3. Knowledge Base & System Updating │
│ Feedback permanently updates enterprise semantic logic. │
└─────────────────────────────────────────────────────────────┘
```
When the system signals low confidence or ambiguity, domain experts intervene to teach the model the correct logic. This feedback loop feeds back into the organization's knowledge base, ensuring the AI retains the knowledge for future queries.
- **Human reliability parallel**: Trust depends on the ability to recognize uncertainty and signal doubt, not infallible correctness.
- **Continuous learning loop**: Teaching the model on edge cases updates the central knowledge base, preventing repeated errors.
[07:24](https://www.youtube.com/watch?v=tHS5Z3m2Pd4&t=444s) Overcoming the "Messy Data" Bottleneck
A common reason enterprise AI projects stall is the belief that organizational data must be perfectly cleaned and consolidated before deploying AI. Ghosh emphasizes that data is perpetually evolving and will never reach a flawless state. Waiting for pristine data preparation—whether through knowledge graphs, semantic layers, or data pipelines—delays deployments by 6 to 9 months and risks project abandonment if early tests yield errors.
Ghosh poses a key tradeoff to the audience:
> *"Would you prefer an AI that is 50% accurate but tells you when it's wrong, or an AI that's 90% accurate but you have no idea if it's accurate or not?"*
Choosing an AI system that transparently signals its uncertainty allows organizations to bypass initial data cleanup delays. By starting with existing, imperfect data infrastructure, the system can establish a baseline, flag ambiguous outputs, and incrementally refine its knowledge through real-world usage.
- **Timeline friction**: Traditional data cleanup initiatives take 6 to 9 months before an AI tool can even be deployed.
- **User drop-off risk**: Initial incorrect outputs lead users to abandon new tools, akin to early e-commerce users leaving sites with endless loading spinners.
- **Deployment strategy**: Deploying with native reliability signaling allows teams to launch safely on messy data while continuously improving accuracy over time.
[09:30](https://www.youtube.com/watch?v=tHS5Z3m2Pd4&t=570s) PromptQL Architecture and Separation of Concerns
PromptQL is designed as a learning platform for building custom enterprise AI models that natively signal confidence levels and learn on the job. Achieving production-grade accuracy requires continuous maintenance across multiple layers: building evaluation sets (evals), managing model updates, context engineering, tool orchestration (e.g., text-to-SQL, RAG), and maintaining data pipelines.
To manage this complexity without creating operational bottlenecks, Ghosh advocates for a **separation of concerns** between technical setup and business domain expertise:
```
ENTERPRISE AI ARCHITECTURE
┌─────────────────────────────────────────────────────────────┐
│ Technical Platform / Infrastructure │
│ (Model updates, context engineering, tool integration, RAG) │
└──────────────────────────────┬──────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Business Expertise Team │
│ (Eval creation, metadata review, business logic steering) │
└─────────────────────────────────────────────────────────────┘
```
The business expertise team—those who deeply understand the domain—focuses on writing evaluations, reviewing metadata, and approving system logic updates, rather than managing low-level data preparation pipelines.
- **Most critical AI artifact**: Eval sets (evaluation benchmarks) are the foundational requirement for any enterprise AI project.
- **Onboarding timeline**: PromptQL connects to existing databases, data warehouses, APIs, and catalogs, reaching an operational baseline level of accuracy in approximately two weeks.
- **No data pre-reqs**: Connects directly to legacy infrastructure without requiring upfront data lake house overhauls.
[13:50](https://www.youtube.com/watch?v=tHS5Z3m2Pd4&t=830s) Walkthrough: AI Analyst Demo and Confidence Signaling
Ghosh demonstrates PromptQL's capabilities through an AI Analyst scenario connected to SaaS control-plane, billing, and support systems. A user submits the query: *"Who are my VIP customers, and how many of them are at risk of churn?"*
The model inspects the underlying data sources and generates an answer, accompanied by an explicit **reliability signal**:
1. **Initial Unreliable Output (Yellow/Low Confidence Signal)**: The AI explicitly warns the user that its answer relies on unverified assumptions. It flags that the enterprise data lacks formal definitions for both "churn risk" and "VIP customers." To answer, the model substituted generic definitions, noting: *"I did not find anything telling me how to calculate churn in your data... VIP customers, I did not find this anywhere in your data or metadata... This is my assumption."*
2. **Human Correction**: The user supplies the company's explicit business rules:
- **Churn Risk**: Declining platform usage paired with negative sentiment across the last 10 support tickets.
- **VIP Customers**: Enterprise plan accounts billed at greater than $10,000 per year.
3. **Updated Reliable Output (Green/High Confidence Signal)**: The model accepts the user's definitions, re-executes the query, and displays a green confidence indicator. The system explicitly restates the applied business rules in its execution plan and retains these parameters for future sessions.
Ghosh concludes this portion by pointing out a practical testing tip for evaluating enterprise AI products: running the same query before and after supplying business definitions to verify whether the underlying calculations and target entities adapt correctly.
- **Demo parameters**: VIP criteria ($10,000/year billing on Enterprise plans) and Churn criteria (usage drop + negative sentiment on last 10 tickets).
- **Transparency UI**: Shows clear visual signals (yellow warning vs. green checkmark) along with execution plans detailing assumptions made by the model.
[16:12](https://www.youtube.com/watch?v=tHS5Z3m2Pd4&t=972s) Ensuring Repeatability and Capturing Tribal Knowledge in AI Automations
Repeatability serves as one of the primary benchmarks for measuring accuracy in AI agents. If you ask a question five times and receive the exact same output every single time, the system demonstrates repeatability—a critical signal that the solution is trustworthy and accurate. This principle directly applies when utilizing custom models to act as an AI engineer. Rather than merely querying for data, users can prompt the system to construct complex workflows, such as automatically processing incoming member files and updating system records within a credit union context.
When an AI agent encounters domain-specific concepts, it often flags its own limited reliability due to ambiguous business definitions. For example, an agent might prompt the user to verify its interpretation of a "new member." This interaction creates an opportunity for human experts to supply unwritten tribal knowledge—rules often stored solely in personal notebooks or mental models. Once the user corrects the definition, the AI updates its underlying metadata and internal understanding, returning a high-confidence green signal to safely proceed with execution.
```
+-----------------------------------------------------------------+
| User Automation Request |
| "Process new member files & add system accounts" |
+-----------------------------------------------------------------+
|
v
+-----------------------------------------------------------------+
| AI Agent Evaluates |
| Flags "New Member" definition & reliability |
+-----------------------------------------------------------------+
|
v
+-----------------------------------------------------------------+
| Human Expert Correction |
| Provides tribal knowledge & domain rules |
+-----------------------------------------------------------------+
|
v
+-----------------------------------------------------------------+
| Metadata & Model Update |
| Achieves high confidence -> Executes process |
+-----------------------------------------------------------------+
```
[17:22](https://www.youtube.com/watch?v=tHS5Z3m2Pd4&t=1042s) Real-World Enterprise Impact: Data Visibility in Sales and Radiology Coding
Building reliable and accurate AI unlocks massive operational and financial value, as demonstrated by real-world enterprise deployments. In a Fortune 500 company featuring a 25,000-member sales team, years of corporate acquisitions created fragmented Salesforce accounts where overlapping client records existed across multiple acquired entities. Because every acquired team had historically sold to similar enterprise clients, salespeople lacked clear cross-organization visibility when prospecting. This data blind spot introduced a 10% risk factor into their quarterly revenue forecasting—a significant vulnerability for a publicly traded company reporting regular financial results. By embedding an accurate AI analyst widget directly inside Salesforce, the company successfully solved the visibility issue and dramatically improved forecast reliability.
In the healthcare sector, a radiology network faced major operational inefficiencies surrounding MRI scheduling and procedure coding. Assigning the correct procedure code requires evaluating numerous patient variables, including age, gender, diabetic status, fasting requirements, and the presence of pacemakers. Miscoding results in missed appointments and unused scanning slots, incurring substantial financial losses. Because call center staff had to write custom business logic for every incoming request, errors remained frequent. Deploying an AI engineering solution to automate this rule-matching process reduced scheduling handle time from 12 minutes down to 6 minutes, yielding an estimated $50 million economic impact.
- **Sales Visibility Risk:** A 25,000-person sales force faced a 10% forecast accuracy risk due to duplicate customer records across acquired Salesforce instances.
- **Healthcare Cost Savings:** Automating complex radiology scheduling logic dropped processing time from 12 to 6 minutes, driving a $50M business impact.
- **Key Determinants for MRI Coding:** Age, gender, diabetes status, fasting requirements, and medical devices (e.g., pacemakers).
[19:01](https://www.youtube.com/watch?v=tHS5Z3m2Pd4&t=1141s) Accelerating SaaS Onboarding and Concluding Remarks
A classic operational bottleneck for SaaS platforms involves schema integration across disparate client bases. In financial institutions like credit unions, onboarding a new customer typically takes months because engineers must manually map the client's custom database schema to the platform's internal schema. By deploying AI agents capable of understanding data structures and mapping fields automatically, organizations can eliminate manual engineering overhead, dramatically accelerating customer onboarding and time-to-value.
Wrapping up the presentation, Rajoshi Ghosh from PromptQL emphasizes that overcoming accuracy barriers is the definitive step to unlocking enterprise AI adoption. Attendees interested in exploring AI agents and domain-specific metadata strategies are invited to connect directly after the session or visit PromptQL at Booth 356.
> "If you get the same answer five times, it's repeatable... and then you know that it's also signaling the accuracy."
## Notable quotes
> [07:24](https://www.youtube.com/watch?v=tHS5Z3m2Pd4&t=444s) Would you prefer an AI that is 50% accurate but tells you when it's wrong, or an AI that's 90% accurate but you have no idea if it's accurate or not? — Rajoshi Ghosh
> [04:23](https://www.youtube.com/watch?v=tHS5Z3m2Pd4&t=263s) I looked at the customer tables, but the data was messy, so I'm not 100% sure. — Rajoshi Ghosh
> [13:50](https://www.youtube.com/watch?v=tHS5Z3m2Pd4&t=830s) I did not find anything telling me how to calculate churn in your data... VIP customers, I did not find this anywhere in your data or metadata... This is my assumption. — PromptQL AI Model
> [19:01](https://www.youtube.com/watch?v=tHS5Z3m2Pd4&t=1141s) If you get the same answer five times, it's repeatable... and then you know that it's also signaling the accuracy. — Rajoshi Ghosh
## Video description
Rajoshi Ghosh, Chief Ecosystem Officer/Founder, PromptQL
Abstract
Many AI agent projects stall after promising pilots because they can’t deliver the accuracy businesses need to trust them.
This talk shares lessons from early enterprise deployments on why accuracy debt holds back adoption—and how leading teams are overcoming it by building AI agents that truly understand their company’s unique data, processes, and language.
We’ll discuss why generic solutions often fail, share a practical framework for evaluating AI agents, and show how to design systems that deliver reliable, explainable results your team can count on.
PromptQL founder Rajoshi Ghosh argues that enterprise AI initiatives stall because generic models lack unwritten tribal knowledge and fail to signal uncertainty, leading to broken trust and halted adoption.
The Accuracy Trap and Missing Tribal Knowledge (1:06)
Enterprise leaders often expect high accuracy immediately, but generic off-the-shelf models struggle because they lack organizational context and implicit terminology, such as medical classification codes like 'DRG470'. When systems encounter messy enterprise data and produce confidently incorrect answers, business executives lose trust and cannot rely on outputs for board presentations or earnings reports. Citing a 2023 interview where Ilya Sutskever identified model reliability as the primary bottleneck to AI's economic impact by 2030, Ghosh argues that manually verifying answers eliminates AI's economic value.
Deploying the Accuracy Flywheel on Imperfect Data (4:23)
Ghosh contends that waiting 6 to 9 months for pristine data preparation via semantic layers or pipelines delays deployments needlessly, as enterprise data is constantly evolving. Instead, she advocates using an 'Accuracy Flywheel' where the AI natively flags ambiguous inputs or low-confidence outputs, mirroring human colleagues who state when they are unsure. Domain experts then intervene to correct assumptions, permanently updating the central knowledge base.
Separation of Concerns and Confidence Signaling (9:30)
PromptQL structures enterprise AI architecture by separating technical platform management—such as model updates, retrieval-augmented generation (RAG), and context engineering—from domain business expertise. Under this framework, business teams focus on writing evaluation benchmarks ('evals') and reviewing metadata, enabling the system to connect directly to legacy databases and catalogs within roughly two weeks. In a demonstration analyzing customer churn and VIP accounts, the model initially surfaced a yellow low-confidence warning noting missing definitions for 'churn risk' and 'VIP customers'; once human operators supplied explicit billing and ticket parameters, the system transitioned to a green high-confidence output and retained the logic for subsequent runs.
Enterprise Outcomes in Sales, Healthcare, and SaaS (17:22)
When models incorporate repeatable logic and domain rules, they resolve costly organizational bottlenecks. For a Fortune 500 company with a 25,000-person sales force, fragmented Salesforce instances following multiple acquisitions created account visibility blind spots that posed a 10% risk to quarterly forecasting; an embedded AI analyst resolved the overlap. In healthcare, automating complex MRI scheduling rules involving variables like age, diabetes status, and pacemakers cut call center handle times from 12 minutes to 6 minutes, driving an estimated $50 million economic impact. Similar automation accelerates credit union onboarding by replacing months of manual database schema mapping.
Ghosh emphasizes that overcoming accuracy and reliability hurdles through explicit confidence signaling and human-guided metadata updates remains the critical step for organizations aiming to operationalize AI without overhauling existing legacy data architectures.
[7:24] Rajoshi Ghosh: Would you prefer an AI that is 50% accurate but tells you when it's wrong, or an AI that's 90% accurate but you have no idea if it's accurate or not?
[4:23] Rajoshi Ghosh: I looked at the customer tables, but the data was messy, so I'm not 100% sure.
[13:50] PromptQL AI Model: I did not find anything telling me how to calculate churn in your data... VIP customers, I did not find this anywhere in your data or metadata... This is my assumption.
[19:01] Rajoshi Ghosh: If you get the same answer five times, it's repeatable... and then you know that it's also signaling the accuracy.
Original transcript
[0:04] So just quick introduction, I'm Rajoshi.
[0:06] I'm one of the founders of PromptQL and
[0:09] you know I've been sort of building and
[0:11] sort of selling enterprise B2B software
[0:13] for over
[0:16] 10 years and just the last 6 months
[0:20] since we launched PromQL things have
[0:22] been a little bit crazy. Uh we've sort
[0:24] of closed seven figure deals with
[0:26] Fortune 500 companies in a few months
[0:29] and each of them have like expanded and
[0:31] they're like rapidly growing. We've had
[0:33] conversations where customers didn't
[0:36] have a specific budget but they're kind
[0:37] of trying to pull stuff out of
[0:38] everywhere. Like it's it's a completely
[0:41] crazy time that we're living through.
[0:43] Like people want AI. People are sort of
[0:46] like budget line items haven't caught
[0:48] up. Things haven't caught up. You want
[0:50] you want to drive that change. there's a
[0:52] lot of energy and we are sort of seeing
[0:54] and living that momentum and it's been
[0:56] insane. So what I really want to do
[0:58] today over the next 20 minutes is sort
[1:01] of share some of those learnings with
[1:03] you all as to what have we seen right so
[1:06] before I go into any of my slides I want
[1:08] to kind of start off by disclosing these
[1:11] two big insights that we've gotten one
[1:14] is the title of the slide which is
[1:16] accuracy is a big trap like you put in a
[1:20] lot of investment make a lot of uh build
[1:23] up you know prep your data do all of
[1:25] that and you kind of launch things and
[1:27] you realize that accuracy is a trap and
[1:29] that kind of really becomes a big
[1:31] problem with adoption. So that's a big
[1:33] thing that we've seen that it is one of
[1:35] the biggest problems today that's
[1:37] holding AI in enterprises back. The
[1:39] second big insight is that this is like
[1:42] the cheat sheet. You need custom AI. You
[1:45] need to actually build your models. You
[1:49] can't take off-the-shelf model and
[1:52] expect it to work in your environment.
[1:54] And that's the tribal knowledge piece
[1:56] that we'll be unpacking over the next
[1:58] few minutes. So those are the two
[2:00] insights. There's a big accuracy problem
[2:02] that you need to sort of preempt. And
[2:04] number two, you need custom AI. So what
[2:06] does this really look like?
[2:09] Oops.
[2:13] So what is the accuracy problem? So AI
[2:17] on data, we want AI to work within our
[2:19] enterprise, right? So imagine you have
[2:22] AI on data and then you ask this AI, hey
[2:25] pull up patient readmission rates for
[2:27] DRG470,
[2:29] right?
[2:31] Do you think AI is going to get this
[2:33] right? Your colleague in this business
[2:36] would get it right because they know
[2:37] what DRG470 means. And there are just
[2:39] all of these phrases that you use
[2:41] day-to-day with your colleagues that you
[2:43] take for granted. You've never bothered
[2:45] to write it down. You've never bothered
[2:47] to fill up metadata. You've not Your
[2:49] data is messy. Half the stuff is in your
[2:52] head. That's what makes colleagues so
[2:55] like useful with tenure, right? They're
[2:57] just they know stuff and it's stuff the
[2:59] water cooler conversation. Somebody
[3:00] joins your team, you're just kind of
[3:01] telling them stuff. So, is AI going to
[3:04] get this right? But you know what?
[3:06] Here's the kicker. But will that stop AI
[3:09] from being confidently wrong? Right? You
[3:13] may have the most accurate AI, but will
[3:15] you even know? And are you now going to
[3:17] pull out stuff that this AI tells you,
[3:19] put it on your board deck and your
[3:20] public earnings call, right? And this is
[3:22] kind of the accuracy problem. And it's
[3:24] not just us that's
[3:27] talking about this. This is like from a
[3:29] old podcast from 2023. I mean that's
[3:33] really old in the AI world, right? But
[3:35] um this is one of the Ilia Sutska one of
[3:38] the co-founders of open AI uh currently
[3:41] started a company called super
[3:42] intelligence and he was on this podcast
[3:43] by Daresh and he was asked you know if
[3:46] AI is not all it's all we think it's
[3:48] going to be what do you think will be
[3:50] the reason for its failure like what is
[3:52] going to cause it to not create the
[3:53] economic impact that we think it can
[3:55] create and he gives all the caveats he
[3:58] says I don't think that's going to be
[3:59] the case but if you have if I have to
[4:03] say something it would only be
[4:05] reliability. And he goes on to explain
[4:07] what he means. He says that if you
[4:09] cannot trust the responses AI gives you
[4:11] and you kind of have to keep going back
[4:12] and checking, then it's not going to be
[4:14] able to create the kind of economic
[4:15] impact for it to be useful. And the time
[4:18] frame in this for this conversation was
[4:20] by 2030, right? So this is a problem. So
[4:23] how do you build accurate AI? What do
[4:25] you do? Like presumably a lot of you
[4:28] you're here at this conference because
[4:29] you're learning about the latest and
[4:30] greatest. You want to be those change
[4:32] makers within your companies.
[4:36] Have use cases that are beyond
[4:37] summarizations and writing emails,
[4:39] right? You want to do real things with
[4:40] AI? It sounds great. Like what if it
[4:43] could just work the way you expect it
[4:44] to? So let's what is accuracy? What is
[4:47] reliability? Why do you trust your
[4:49] colleagues? Right? Um do are humans
[4:52] always right? Are humans 100% accurate?
[4:56] No. Right? But we work with them all the
[4:57] time. We trust people. That's how that's
[5:00] how we work. So what makes humans
[5:02] accurate? When you ask your colleague a
[5:04] question,
[5:06] they get they understand the data and
[5:08] the domain. They know your tribal
[5:09] knowledge. They know what to do with
[5:11] this question, where to look, but also
[5:14] they know to signal when they're wrong.
[5:17] They will tell you that, hey, I got you
[5:19] this response, but I'm not 100% sure
[5:22] this is right because I was looking at
[5:23] the customer tables and this data was
[5:25] really messy. There are five different
[5:27] tables. This is my best guess, but you
[5:29] know, I may be wrong. And that's how you
[5:31] build up trust with humans. Can you do
[5:33] that with AI? And but you know what? You
[5:36] have to do that with AI. There is
[5:37] literally no other way to build that
[5:40] trust when you're working with
[5:42] missionritical data and mission critical
[5:44] systems when you want to build sort of
[5:46] impactful or do impactful projects with
[5:48] AI. So you need to build this custom AI,
[5:52] right? AcmeQL. Acme is short for a
[5:55] company that makes everything. It's just
[5:57] a it's like your company, right? Like
[5:59] subacme with the name of your company,
[6:01] but you need your company's
[6:04] like AI that understands your business,
[6:08] your domain, your tribal knowledge, and
[6:10] can build trust by learning on the job,
[6:13] right? That's kind of what you need
[6:14] because off-the-shelf AI is it's going
[6:17] to be hard for it to really be like your
[6:21] veteran colleagues, right? So, what does
[6:23] this look like? And this is the accuracy
[6:25] flywheel, right? A model that can signal
[6:28] to you if its response is accurate or
[6:30] not, right? And then when it's wrong, if
[6:34] it says, hey, I'm not sure if this is
[6:36] right or this is low confidence levels,
[6:38] you tell it, you teach this model how to
[6:41] correct itself, how to be right, right?
[6:44] You kind of steer this model in a
[6:46] particular direction. And this is that
[6:47] tribal knowledge gap. This is that water
[6:49] cooler conversation. This is how you
[6:51] onboard a new intern or a new colleague
[6:53] in your firm. And that knowledge once
[6:56] given or once taught kind of goes back
[6:58] and updates itself in sort of the
[7:02] knowledge base of the company. Right?
[7:04] And this it's constantly learning from
[7:06] the feedback so that the next time you
[7:08] ask the same question or a similar
[7:09] question with the same phrases, it knows
[7:11] how to get it get it right. So that's
[7:14] kind of what a custom AI model, your
[7:17] company's model that understands your
[7:19] business, your domain, your tribal
[7:20] knowledge ought to look like.
[7:24] And so let's play it out, right? So if
[7:26] you're trying to build something within
[7:27] your company, you've probably invested
[7:30] in a semantic layer. So the step back,
[7:33] one of the biggest things I hear from
[7:35] people when they're trying to build AI
[7:36] is that our data is super messy. We've
[7:38] got to clean up our data before we can
[7:40] do any of this AI thing. But imagine
[7:42] like any of you who work with data, how
[7:44] many of you work with like data on the
[7:46] data side of the house? Yeah, you're
[7:48] there is no perfect state. It will never
[7:51] get to a perfect state. It is always
[7:53] that is the nature of our businesses and
[7:55] domains. It's constantly evolving. And
[7:58] so data prep cannot hold your AI
[8:01] investments back. So you invest in a
[8:03] semantic layer or a knowledge graph and
[8:05] you build all of this and now you launch
[8:07] it. Your users are going to use this and
[8:10] first You know, we'll probably anyone
[8:13] using a new tool, you're probably going
[8:14] to ask it a few questions that you know
[8:15] the answer to because you're kind of
[8:17] testing it out. It gets something wrong
[8:19] and that's it. They're not going to use
[8:20] it again. This is like early days of
[8:22] e-commerce when you see that spinning
[8:23] wheel and you know, you don't go back to
[8:25] that website anymore. That's that's what
[8:27] adoption is like. It gives you a wrong
[8:29] answer. You don't trust it anymore. So
[8:30] all of that data prep work that t that
[8:32] has taken six to nine months and you
[8:34] launch this project and that's it. There
[8:36] is no adoption. This is happening at the
[8:38] biggest tech companies not just
[8:40] enterprise but Silicon Valley tech
[8:42] companies tech forward also I have heard
[8:45] this so many times. So one step better
[8:49] would you prefer an AI that is 50%
[8:52] accurate but tells you when it's wrong
[8:54] or would you prefer an AI that's 90%
[8:57] accurate but you have no idea if it's
[9:00] accurate or not. Right?
[9:04] Guessing it's the 50%. So that's sort of
[9:06] the next step. You build this, but you
[9:08] want AI to tell you that it's 50% if
[9:10] it's if it's wrong. And then you kind of
[9:13] keep improving on that. Every time it's
[9:15] wrong, you teach it how to get accurate
[9:17] and then it learns. And that's the
[9:18] continuous line uh continuous learning.
[9:21] And that's kind of where you want to be.
[9:23] That's where you build trust. That's
[9:24] where you start actually making big
[9:27] impact with AI in enterprise.
[9:30] And that's kind of what we've done.
[9:32] That's what we've built. We've built
[9:33] PromptQL. It's a platform that helps you
[9:36] build custom AI. You can staff up a team
[9:38] and build the custom AI. We're a
[9:40] learning platform that kind of helps you
[9:41] do that. And unlike other AI, it
[9:45] natively signals confidence level. I'm
[9:47] going to show you some um uh show you
[9:49] what this looks like. And it's
[9:50] constantly learning so that it can
[9:52] behave like a human colleague and learn
[9:54] on the job. Right? So that's kind of
[9:57] what we've been doing. So what this
[9:59] looks like if you are trying to build
[10:01] accuracy into your AI there's it's like
[10:06] you ask a question you have your AI it
[10:08] gives out an answer and behind for that
[10:11] answer to be accurate there's a lot of
[10:13] work that needs to be done everything
[10:15] from building eval sets which is by the
[10:17] way the most important thing if you want
[10:19] to if you're starting an AI project
[10:20] haven't built your eval sets you have to
[10:22] build your eval sets right evaluation
[10:24] tests of how you how do you know
[10:26] something is right so you build an eval
[10:28] test eval set then you have to keep up
[10:30] with your model updates you're doing the
[10:34] context engineering you're figuring out
[10:36] what kinds of tools you need for the
[10:39] type of things that you want you want
[10:40] your agent to do your text to SQL rag
[10:43] whatever you want you're building your
[10:44] tools and then you're prepping your data
[10:46] layer anything here changes new model
[10:48] update comes out you're going to have to
[10:50] redo all of this or a lot of it setting
[10:53] it up once takes 6 to9 months to achieve
[10:56] high levels of accuracy and then
[10:58] anything changes and you're going to
[11:00] have to keep investing in this system.
[11:02] Right? So this is sort of what the a
[11:04] custom AI team that you would build
[11:06] would have to do and this is very
[11:08] important for you to kind constantly
[11:11] keep working on that accuracy level so
[11:12] your adoption increases
[11:15] with something with promptql in place
[11:18] where the custom model is built or
[11:20] you've built your custom model an ideal
[11:21] world would look like you have your
[11:23] business expertise team this is the team
[11:26] that really understands your data and
[11:28] domain really really well and these
[11:31] people are then working on building
[11:33] evals and constantly reviewing your
[11:36] metadata, constantly reviewing this
[11:38] teaching that everyone is doing, you
[11:40] know, approve these changes don't. So
[11:42] you're constantly keeping track of what
[11:44] the metadata looks like and that's sort
[11:46] of this business expertise team. So if
[11:48] you are building your own custom models,
[11:50] a good way to think about it is to sort
[11:53] of create the separation so that it's
[11:55] not all mixed in. It's not this one team
[11:58] that's also doing the data prep but also
[12:00] trying to do the eval sets. Can we it
[12:02] the separation of concerns becomes a
[12:04] useful mental model to kind of keep
[12:06] innovating.
[12:09] And then how do you onboard AI? You
[12:11] onboard AI like you would onboard a
[12:14] human colleague who's just joined. A
[12:16] super intelligent human colleague who's
[12:18] joined will have to be onboarded.
[12:20] They'll have to be taught what is all
[12:22] the data that you need to connect to,
[12:24] what is where you you kind of teach them
[12:26] everything. So in in the AI world, this
[12:28] looks like anywhere that you have
[12:30] context, all your databases, data
[12:33] warehouses, APIs, cataloges that you've
[12:35] invested in or semantic layers that you
[12:37] whatever you've already invested in. If
[12:39] you've not made a bunch of these
[12:40] investments, great. You you don't need
[12:42] to wait on that or you don't need to
[12:44] make those investments. But any work
[12:46] that you've done, your messy data, you
[12:48] plug in, you just tell connect the
[12:51] system, tell it what your, you know,
[12:54] connect it to your messy data. It infers
[12:56] based on any work that you've previously
[12:58] already done and then you're just
[13:01] teaching it on the job. So you don't
[13:02] have this endless data prep but it's
[13:04] kind of learning on the job and this
[13:06] process takes about 2 weeks when we set
[13:09] this up and then you reach this baseline
[13:13] level of accuracy where you're able the
[13:15] system is able to signal accuracy so you
[13:18] can get started right and then it's just
[13:20] continuous learning after that. So you
[13:22] kind of set it up to a baseline level of
[13:25] accuracy by which is with your messy
[13:27] data because you're not waiting for your
[13:29] messy data to get perfect. You can keep
[13:31] doing that every time you improve
[13:33] something. It'll improve the system. But
[13:34] you don't have to wait. You sort of set
[13:36] it up and then it sort of learns on the
[13:39] job or you have somebody constantly
[13:40] cleaning up the data and it just keeps
[13:42] get making the system better because it
[13:44] can still work because it can tell you
[13:46] when it's wrong. So you're not kind of
[13:47] gated on your data being messy. Now
[13:50] let's look into look at some demos of
[13:52] what this actually looks like, right? Um
[13:54] we're going to look at an AI analyst
[13:56] demo first. Let me um so this is sort of
[13:59] like the date like this is connecting to
[14:02] the connected to the systems that are
[14:04] sort of your SAS platform, your um uh
[14:07] control pane data. So it has billing
[14:09] data, customer information, things like
[14:10] that. And you've connected it and you
[14:11] want sort of this AI analyst on your
[14:14] sort of uh um support and control plane
[14:17] systems, right? And so my question is
[14:19] who are my vi VIP customers um and
[14:23] uh how many of them are at the risk of
[14:25] churn. So then this ACMEQL model
[14:29] understands
[14:30] has connected to your messy data
[14:32] understands a bunch of this goes and
[14:34] tells you how it intends to solve the
[14:36] problem and then
[14:38] gives you this artifact here. It sort of
[14:40] gives you these customers but then
[14:42] there's a little signal over here which
[14:44] is the reliability signal and says hey
[14:46] I've given you an answer but this is not
[14:49] very reliable because this is what I
[14:52] have assumed churn risk is this is how I
[14:55] calculate churn risk. I don't know if
[14:56] this is the same as yours because I did
[14:58] not find anything telling me how to
[14:59] calculate churn in your data or anything
[15:01] that I have access to. Number two, VIP
[15:04] customers. You don't tell you I did not
[15:06] find this anywhere in your data or
[15:07] metadata as to what you mean by VIP
[15:09] customers. So, this is my assumption.
[15:10] I've gone ahead and done something
[15:12] because I want to show up as being good,
[15:14] but I don't know if this is what you
[15:15] mean. So, then you're like, nah, that's
[15:17] not how I work. This is what I mean by
[15:19] churn. I have very specific definitions.
[15:20] This is what I mean by churn. Um,
[15:23] declining usage and negative sentiment
[15:24] in the last 10 tickets. And this is what
[15:26] I define by VIP customers. Enterprise
[15:28] plan over is that 10,000? Yeah, 10,000
[15:31] per year in billing. this is these are
[15:33] my definitions they're pretty standard
[15:34] please don't apply things just use my
[15:37] definition says got it I understand kind
[15:40] of does this now and then you get a
[15:41] green signal which is like now I'm
[15:43] actually very confident in my response
[15:45] right next time you ask the question it
[15:48] goes ahead and you can see the plan now
[15:49] it says okay this is how I have ident
[15:51] this is these are my definitions it's
[15:52] still telling you it's repeating the
[15:54] definitions that you had earlier told it
[15:56] kind of goes back and does this and
[15:58] again has the little green check and for
[16:00] those of you who looked at the names you
[16:02] will see that the names that it has
[16:04] picked up as the top five customers are
[16:06] exactly the same as the previous one.
[16:08] This is a great hack by the way to test
[16:10] your AI systems. Ask it the same
[16:12] question five times. If you get the same
[16:14] answer five times, it's repeatable
[16:15] because one way to test accuracy is
[16:17] repeatability. If there is a right
[16:18] answer, you'll have to give the same
[16:20] answer five times, right? So that's so
[16:23] it gets repeatable and then you know
[16:24] that it it's also signaling the
[16:26] accuracy. Same thing sort of applies
[16:28] when you use something uh use the custom
[16:30] model to build an AI engineer. Right? So
[16:34] I'm not going to go into this in too
[16:35] much detail, but this is a credit union
[16:37] kind of domain. The question is instead
[16:38] of sort of giving you the answers and
[16:40] the data, you're asking it to build an
[16:42] automation. It says create an automation
[16:43] that looks at new member files that have
[16:45] come in and adds their accounts to the
[16:47] system. Now again, new member is a
[16:49] definition. It goes ahead and does it
[16:50] and it says hey limited reliability
[16:52] because this is how I have defined uh
[16:54] new member. Is this okay? And then I say
[16:56] no that's not right. This is not how we
[16:58] do things and detailed explanation of
[17:00] new member which I have never bothered
[17:01] to write down anywhere right or it's in
[17:03] my little notebook somewhere. So I kind
[17:05] of define it and then it goes ahead does
[17:08] it gets the green signal. So right so
[17:10] this is kind of that reliability signal.
[17:12] It signals accuracy. You teach it. It
[17:14] updates its understanding updates the
[17:16] metadata and then goes ahead and does it
[17:18] and gives you the confidence score that
[17:20] you can go ahead. If you have reliable
[17:22] AI, if you have accurate AI, what does
[17:24] that unlock? And these are actual case
[17:27] studies of customers that we're working
[17:29] with. Fortune 500 company, AI analyst,
[17:33] 25,000 member sales team have has
[17:37] acquired a bunch of different um uh
[17:40] companies over the year. Every
[17:42] Salesforce account that they have has
[17:45] the same customer. Everyone sold to
[17:46] everyone. These are enterprise
[17:47] companies. So now when they want to
[17:49] prospect into a new customer, just
[17:51] having visibility of this data is a huge
[17:53] problem. And this actually leads to 10%
[17:56] of risk in their forecast accuracy which
[17:58] in a public company they're reporting
[18:00] this every every um quarter. So solving
[18:04] this visibility in a way as a widget
[18:07] inside of your sales force such that
[18:09] your um sales folks can actually use it
[18:11] has this massive impact and has like the
[18:14] forecast improvement radiology lab
[18:17] across the company. This is an AI
[18:19] engineering sort of use case. You call
[18:22] in and you want an MRI scan. Now you
[18:24] need to be assigned a specific code. Now
[18:26] this code depends on your age, gender,
[18:28] are you diabetic, do you need to fast,
[18:31] do you have a pacemaker, a bunch of
[18:32] things. You get this wrong, you come in
[18:34] for your appointment, you miss that
[18:35] slot, it costs the the company quite a
[18:38] bit. And you can also imagine how
[18:40] specialized these people need to be who
[18:42] are putting in that code. So they
[18:43] actually have to write custom business
[18:44] logic every time a scheduling request
[18:46] comes in to kind of um get the right
[18:49] code. And there's still lots of errors.
[18:51] So they've estimated that bringing this
[18:52] down with the help of AI from 12 to 6
[18:55] minutes is about a $50 million in
[18:57] impact. Um I know I'm almost out of
[18:59] time. The last thing I'll sort of say is
[19:01] um um the this is sort of a very typical
[19:05] SAS use case which is that I have a
[19:08] particular schema and every customer I
[19:10] integrate with has a different schema.
[19:12] There's a credit union company. Um so
[19:14] every so onboarding a customer takes
[19:16] many months because all I'm doing is
[19:18] kind of mapping their schema to my
[19:19] schema. Right? Again, this is something
[19:21] that can this an AI engineering sort of
[19:23] use case where a lot of this can be
[19:25] automated and then it's a huge sort of
[19:27] improvement um in sort of uh integrating
[19:30] with your customers. That's my time
[19:33] here. I'm Rajoshi with Prompql. I'm
[19:35] happy to answer any questions. Um we're
[19:38] at booth 356. I think we're all here for
[19:40] a few more hours. I'll also be hanging
[19:41] around here. But thank you so much for
[19:43] your time. Um hope to chat with you,
[19:45] continue chatting.
