Ramp Scraps Multi-Agent Strategy to Consolidate on Unified Tool Engine

Ramp: Lessons from Building a New AI Product - The Pragmatic Summit — The Pragmatic Engineer

The Pragmatic Engineer

AI productivity gains will expand engineering ambition into previously unviable products and system overhauls rather than reduce developer headcount.

AI-assisted summary of Ramp: Lessons from Building a New AI Product - The Pragmatic Summit — original source. Check the original for full context.

Source published:

Key ideas

  • Ramp shifted its core AI architecture from building dozens of specialized, disconnected agents to a single unified agent framework backed by a catalog of hundreds of deterministic skills and tools. (3:31)
  • Early production dogfooding revealed that agent errors stemmed almost entirely from missing domain context (such as HRIS employee titles) rather than model deficiencies, while human approvals proved too inconsistent to serve as ground truth. (10:40)
  • Ramp Inspect, an internal background coding agent running in isolated Modal sandboxes with access to over 150,000 automated tests, now authors more than 50% of the company's weekly merged pull requests. (27:25)
  • Ramp leadership disputes the narrative that AI will reduce engineering headcount, arguing that software demand is open-ended and productivity gains are instead funneled into tackling previously cost-prohibitive legacy rewrites and complex workflows. (35:54)

Detailed notes

## Background Published on YouTube by The Pragmatic Engineer on March 9, 2026, this session from The Pragmatic Summit features Ramp engineering leaders Nik Koblov, Veeral Patel, Will Koh, and Ian Tracey. According to the description, the panel shares concrete lessons from building and scaling applied AI products at Ramp, covering architectural pivots, policy evaluation, evaluation tooling, infrastructure, and engineering culture.

## TL;DR Ramp's engineering team details their journey building AI-native financial software, highlighting an architectural pivot away from fragmented micro-agents toward a unified foundational agent backed by an extensive catalog of deterministic tools. Using their automated Policy Agent as a primary case study, they illustrate how natural language acts as a programming interface, why context gaps—not model limitations—cause most agent failures, and why relying on historical user actions as ground truth fails without strict, continuous evaluations. The speakers also explain their centralized Applied AI infrastructure for model routing, rate-limiting, and cost tracing, alongside an internal autonomous coding agent called Ramp Inspect that now drives over 50% of merged production pull requests. Ultimately, they argue that AI tools will not eliminate developer roles, but will instead widen the performance gap between process-obsessed coders and context-driven engineers while dramatically raising ambitions for software scope.

## Atmosphere & style The presentation carries the brisk, candid energy of an engineering post-mortem stripped of conventional AI marketing hype and focused strictly on production reality. Moving seamlessly between four technical leaders, the delivery is conversational, lucid, and grounded in relatable operational scars—from parsing blurry restaurant checks to admitting how often approving managers make mistakes on lazy Sunday afternoons. It feels refreshingly pragmatic and technically rigorous, balancing honest confessions about the messiness of autonomous workflows with an optimistic, rallying defense of software engineering's future and the irreplaceable value of human judgment.

## Bulletin - [03:31](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=211s) Ramp shifted its core AI architecture from building dozens of specialized, disconnected agents to a single unified agent framework backed by a catalog of hundreds of deterministic skills and tools. - [10:40](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=640s) Early production dogfooding revealed that agent errors stemmed almost entirely from missing domain context (such as HRIS employee titles) rather than model deficiencies, while human approvals proved too inconsistent to serve as ground truth. - [27:25](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=1645s) Ramp Inspect, an internal background coding agent running in isolated Modal sandboxes with access to over 150,000 automated tests, now authors more than 50% of the company's weekly merged pull requests. - [35:54](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=2154s) Ramp leadership disputes the narrative that AI will reduce engineering headcount, arguing that software demand is open-ended and productivity gains are instead funneled into tackling previously cost-prohibitive legacy rewrites and complex workflows.

## Key takeaways - [03:31](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=211s) Product architectures should converge on a single foundational agent equipped with an extensive tool catalog rather than dozens of siloed, task-specific bots. - [05:11](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=311s) OmniChat unifies conversational entry points across the application and compiles plain-English instructions into deterministic, runnable playbooks. - [07:17](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=437s) Treating natural-language policy documents as the product's codebase eliminates the need for brittle, hardcoded deterministic logic trees. - [10:40](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=640s) AI systems cannot be built in one shot; starting with an ultra-constrained problem like a single coffee receipt reveals critical missing context much faster than theoretical planning. - [12:32](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=752s) Transitioning from sequential chains to agentic loops yields greater autonomy at the expense of inspectability, turning smaller black boxes into much larger ones. - [15:29](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=929s) Historical human approval actions are an unreliable ground truth because managers frequently approve out-of-policy expenses due to fatigue or lack of policy knowledge. - [15:29](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=929s) Teams should avoid perfectionism by starting evaluation suites with as few as five critical test cases and building lightweight, rapidly modifiable internal labeling tools. - [19:15](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=1155s) Running offline evals in CI and monitoring online leading indicators—such as the frequency of 'unsure' agent outputs—catches unexpected regressions when updating prompts or tools. - [20:46](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=1246s) Autonomous enterprise adoption requires incremental trust-building, moving from advisory suggestions to configurable autonomy sliders based on transaction risk thresholds. - [22:46](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=1366s) A centralized Applied AI Service protects downstream product teams from breaking API changes while managing rate limits, heavy batching, and cost tracing across model providers. - [25:52](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=1552s) Preventing hallucinations in sensitive financial workflows depends on restricting models to a curated catalog of deterministic internal business tools. - [27:25](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=1645s) Background coding agents like Ramp Inspect resolve context fragmentation by pulling logs, database read replicas, alerts, and Slack threads into isolated developer sandboxes. - [29:48](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=1788s) Making coding agents multiplayer-first and triggering them directly from Slack allows non-engineers to run bug investigations and iterate on code alongside developers. - [31:17](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=1877s) AI coding agents widen the performance gap between outcome-driven engineers who navigate ambiguity and process-heavy engineers who fixate on syntax and bureaucracy. - [34:16](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=2056s) The hardest part of engineering remains navigating the 'long middle' from prototype to production and applying judgment under information scarcity, not generating syntax. - [35:54](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=2154s) Unlike physical manufacturing, software development faces infinite demand, meaning agent-driven leverage raises the standard of product ambition rather than capping developer employment.

## Detailed walkthrough ### [00:05](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=5s) Ramp's Core Mission and the Simplest Expense Use Case

Nik Koblov opens the session by outlining Ramp's role as the leading finance platform for modern businesses, serving over 50,000 customers with a simple value proposition: saving companies time and money. Glancing out at the audience, he jokes about spotting attendee name tags from existing customer organizations and notes that everyone in the room is clutching coffee. That simple cup of coffee serves as the baseline for demonstrating how legacy expense management breaks down. Traditionally, expensing a single coffee takes roughly fifteen minutes of manual effort spread across cardholders and finance staff—capturing receipts, typing out memos, and assigning general ledger (GL) codes. Multiplied across an entire organization, this friction compounds into an immense administrative sink.

Ramp tackled this problem by collapsing those manual steps into an automated, agentic flow that runs the moment an employee swipes or taps their card. About three years ago, Ramp began introducing one-shot AI automations to perform tasks like normalizing merchant names, drafting expense memos, categorizing transactions against accounting codes, and sourcing or matching receipts. Beyond basic card swipes, nearly every corporate persona—from accounts payable clerks and procurement managers to finance directors and data teams—remains bogged down in repetitive manual requests. Koblov recalls that Ramp previously maintained an internal Slack channel called `#help-data`, where colleagues would ask for raw data and an engineer or analyst would manually write a SQL query to export a CSV. Ramp decommissioned that manual channel about a year and a half ago by replacing it with automation.

- **Customer reach**: 50,000+ business customers using Ramp's finance platform. - **Legacy friction**: A single coffee expense typically consumes 15 minutes across various manual bookkeeping tasks. - **Automation history**: Started three years ago with one-shot tasks (merchant normalization, receipt matching, memo drafting), expanding outward to eliminate manual workflows like data-pull requests.

---

[03:31](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=211s) Paradigm Shift: Moving from a Zillion Agents to a Single Agent with Many Skills

Reflecting on recent product evolution—especially following rapid shifts in early February—Koblov highlights a major architectural lesson learned while attempting to automate every internal and external role. Originally, Ramp leaned into building specialized agents for every distinct job to be done (including "Brian," an agent featured in their Super Bowl advertisement). Last year, Ramp intentionally let individual product teams experiment autonomously. This decentralized approach produced roughly four different, siloed implementations for both synchronous interactions and background jobs. Koblov argues that this proliferation of disconnected bots was the wrong long-term trajectory.

Instead, Ramp discovered that modern AI applications must consolidate toward a single foundational agent endowed with a massive catalog of specialized tools and skills. Koblov contrasts how legacy software operates with the agentic paradigm. Traditional software typically confines itself to data context and API tools (actions 4 and 5 of a standard execution loop). In the new paradigm, software takes responsibility for the entire chain: ingesting events (like an incoming invoice), reading prompt instructions, enforcing organizational guardrails and policies, gathering surrounding data, and executing the appropriate tool calls autonomously.

As Nik Koblov explains the architectural shift:

> “So, what we learned is you don't need to build a thousand agents.”

He emphasizes the imperative to streamline the product architecture into a unified execution engine:

> “And with rethink, simplification of your stack. Uh So, what we learned is you don't need to build a thousand agents. We intentionally last year allowed each individual team to go and experiment. And we ended up maybe with four different ways of doing the same thing both for synchronous agents as well as for background agents. Um but instead you want to drive your framework towards a single agent with a thousand skills.”

```mermaid flowchart TD accTitle: Legacy Software vs Autonomous Agent Loop accDescr: Diagram contrasting traditional software tool calls with the complete five-stage autonomous system of action.

subgraph Legacy["Traditional Software Scope"] D["4. Data Context"] E["5. APIs & Action Tools"] end

subgraph Modern["Modern Autonomous System"] A["1. Event / Trigger (e.g., Invoice, Card Swipe)"] B["2. Prompt Instructions"] C["3. Policies & Guardrails"] end

A --> B B --> C C --> D D --> E ```

---

[05:11](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=311s) OmniChat, Lightweight Orchestration, and Runnable Playbooks

To unify how users interact with agents, Ramp consolidated five separate conversational prototypes developed over the past year into a single, standardized interface dubbed "OmniChat" (short for omnipresent chat). OmniChat sits across every surface of Ramp's web application, working alongside traditional UI elements like data tables and action buttons rather than replacing them entirely. Koblov illustrates this with an employee onboarding scenario: when an admin types "Please onboard a new employee," OmniChat resolves the individual to their corporate employee ID, queries the HRIS system to determine their reporting hierarchy, identifies an existing agentic workflow called the "New Hire Playbook," and prompts the administrator for confirmation to run it.

Powering this is an in-house, lightweight agent orchestration framework designed for rapid tool creation. The abstraction layer is lightweight enough that a product manager recently "vibe-coded" around 20 functional tools without needing dedicated engineering assistance. For complex multi-step processes, admins can write plain-English descriptions of the intended lifecycle—such as issuing a corporate card, enforcing receipt upload requirements on every transaction, sending a welcome message on Slack, and scheduling an automated check-in after two weeks. Ramp compiles these human-readable instructions into a deterministic, runnable workflow that the core agent executes reliably.

When a transaction occurs, the transaction flows seamlessly between agents: the Policy Agent verifies corporate spend limits in real time, passes the context to an accounting agent that maps the expense against the chart of accounts and ERP records, and either auto-approves the charge or flags it for human review if it breaches materiality thresholds or policy rules.

```mermaid flowchart LR accTitle: Real-Time Card Swipe Processing Flow accDescr: Step-by-step transaction handling from card swipe to policy check, accounting classification, and final approval or review.

Swipe["Employee Card Swipe"] --> Policy["Policy Agent Review\n(Real-Time Checks)"] Policy --> Acct["Accounting Agent\n(ERP & GL Mapping)"] Acct --> Decision{"Compliant Spend?"} Decision -->|"Yes"| Approve["Auto-Approve Transaction"] Decision -->|"No / High Value"| Escalate["Human-in-the-Loop Review"] ```

---

[07:17](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=437s) Inside the Policy Agent: English as the New Programming Language

Veeral Patel takes the stage to unpack the technical design and operational philosophy behind the Policy Agent, one of Ramp's most heavily utilized products. He displays a blurry restaurant receipt that finance teams typically squint at hundreds of times a day. Rather than forcing a tired controller to manually parse the check, the Policy Agent analyzes both the raw image and the transaction metadata simultaneously. In this specific receipt, the agent detected eight dining guests—a detail Patel admits he could barely make out himself—confirmed that the spend fell under Ramp's internal $80-per-person dinner cap, verified that the stated business purpose was a team welcome dinner, validated the merchant and dollar figures, and recommended instant approval.

Conversely, Patel highlights instances where the agent caught out-of-policy spending: it approved an OpenAI charge incurred by an employee testing ChatGPT features as a valid business expense, but rejected a small $3 bakery charge because it was not incurred during eligible weekend work or overtime hours. When a Fortune 500 customer approached Ramp requesting custom approval and rejection logic across complex enterprise rules, Ramp avoided the trap of hardcoding brittle, deterministic logic trees into the platform.

Referencing AI researcher Andrej Karpathy's observation, Patel explains how Ramp shifted its core philosophy:

> “And we kind of saw this as an opportunity not to kind of add more incremental deterministic rules that kind of define our product. And I worked on some of the first versions of these, um but actually kind of take out uh a page from Andrej Karpathy saying that English is the new programming language and kind of turn the expense policy into the rules themselves.”

To build and tune this agentic policy evaluator, the engineering team operated like an agile startup within Ramp. They partnered closely with enterprise design partners—including the Fortune 500 company—holding weekly feedback sessions to inspect edge cases, understand nuanced accounting requirements, and refine how the model interprets natural-language policy documents against real-world spend.

[10:40](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=640s) Internal Dogfooding and the Reality of AI Context

Veeral Patel reflects on a fundamental cultural realization across Ramp: AI products simply cannot be built in a single stroke. Teams must resist the urge to chase perfection on day one and instead start with an extremely constrained problem. To put this into practice, Ramp leaned heavily on internal dogfooding. Rather than trying to automate corporate spend across the board, they scoped down to determining whether a simple “coffee with a colleague” transaction ought to be approved or rejected. These single-dollar-amount expenses were categorized by Ramp's internal finance team as low-risk, making them the ideal sandbox for early experimentation.

```mermaid flowchart TD accTitle: Iterative Context Enrichment for Policy Agent accDescr: Flow showing how internal dogfooding revealed context gaps rather than model failures. subgraph Dogfooding["Internal Dogfooding"] A["Low-Risk Expenses<br/>('Coffee with a Colleague')"] B["Policy Agent Evaluates Expense"] end subgraph GapAnalysis["Root Cause Discovery"] C{"Error Analysis"} D["Model Capability Defect<br/>(Rare)"] E["Missing Domain Context<br/>(Primary Cause)"] end subgraph ContextExpansion["Context Enrichment"] F["HRIS Employee Metadata<br/>(Titles, Roles, Org Levels)"] G["Granular Receipt Data<br/>(Itemized Expenses, Flight Class)"] end

A --> B B --> C C -->|Minority| D C -->|Majority| E E --> F E --> G ```

Once this test ran against production data, the team encountered an essential early learning: when Policy Agent produced the wrong outcome, the fault rarely lay with the underlying LLM. Instead, the failure stemmed from missing context provided to the model. While the team could have spent weeks trying to brainstorm every potential piece of context prior to writing code, learning from live internal data proved vastly superior. For instance, they noticed that an employee’s role and organizational title are critical when interpreting expense policy documents. An executive in the C-suite might be allowed to book first-class flights or enjoy elevated dining limits that junior employees cannot. As a consequence, engineering began extracting richer details from receipts and piping in HRIS metadata already resident on Ramp's platform.

- Veeral Patel: “I think one of the main important I guess things that we realized across Ramp is that we really needed to lean into the fact that AI products cannot be one-shotted.”

[12:32](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=752s) The Evolution from Simple Chains to Agentic Workflows

Staff Engineer Will Koh describes the initial impulse when setting out to build Policy Agent internally: going huge and dreaming of automating all of finance reviews in one go. However, reality demanded starting with that solitary question: is this specific cup of coffee compliant with the expense policy? Even though the problem appeared elementary on the surface, expanding and generalizing it to diverse businesses inevitably compounds complexity. Will emphasizes that simpler systems are inherently easier to iterate on, allowing teams to determine what works before layering on advanced mechanics.

```mermaid flowchart LR accTitle: Three Phases of Policy Agent Architecture accDescr: Evolution from sequential LLM calls to conditional tool use and finally full agentic loops. subgraph V1["Phase 1: Deterministic Chain"] A1["Incoming Expense"] --> B1["Context Retrieval"] B1 --> C1["Sequential LLM Pipeline"] C1 --> D1["Rule-Based Output"] end subgraph V2["Phase 2: Conditional & Tools"] A2["Classify Category<br/>(Meal, Travel, Ent.)"] --> B2["Conditional Prompting"] B2 --> C2["Autonomous Lookup Tools<br/>(Flight Info, Seniority)"] end subgraph V3["Phase 3: Agentic Loop"] A3["Shared Company Toolbox"] <--> B3["Autonomous Agent Loop<br/>(Reasoning & Policy Checks)"] B3 --> C3["Write Capabilities<br/>(Auto-Approve, Decision Logs)"] end

V1 --> V2 V2 --> V3 ```

Ramp's architecture progressed through three distinct phases: 1. **Deterministic Sequential Chains:** An expense arrived, surrounding context was retrieved, and the data was routed through a tightly defined series of sequential LLM calls answering specific questions (Is it in policy? Why is it in policy? How do we present this rationale to the user?). 2. **Conditional Categorization & Targeted Tools:** Recognizing that expenses differ fundamentally, the team added a classification step (travel, meal, or entertainment) followed by conditional prompting. They introduced targeted tools so the model could independently retrieve external data, such as flight itineraries or employee seniority levels. 3. **Full Agentic Workflows:** The system transformed into an agentic loop equipped with shared company-wide toolboxes that can read across the entire Ramp platform. The agent gained write permissions, allowing it to log decisions, generate formal audit rationales, and autonomously approve expenses on behalf of human reviewers.

This architectural shift brought a fundamental engineering trade-off between power and inspectability. As autonomy and intelligence increased, traceability decreased. While engineers can review reasoning tokens produced by the LLM, the inner workings become a much larger black box that executes tool calls and arrives at conclusions on its own terms.

- Will Koh: “A smaller black box becomes a bigger black box as the system becomes more complex.”

[15:29](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=929s) Ground Truth, Disillusionment with User Actions, and Custom Evals

Building on the reality of this expanding black box, Will Koh highlights that teams must implement rock-solid auditability from day one. Systems must be architected so that even if one only inspects inputs and outputs, correctness remains verifiable. Initially, the team assumed that human user behavior could serve as the gold standard of truth—if an approving manager marked an expense as approved, the agent should do the same. In practice, users proved to be unreliable ground truth. Managers frequently approve expenses without understanding the policy, trust their employees blindly, act out of fatigue, or clear their queues on a Sunday afternoon without scrutiny. When finance teams subsequently flagged these approved transactions as unacceptable card charges, the team realized they could not blindly imitate user actions and had to create their own strict definition of correctness.

```mermaid flowchart TD accTitle: Weekly Labeling and Evaluation Pipeline accDescr: Cross-functional alignment loop powering dataset curation and lightweight evals. subgraph Alignment["Weekly Human-in-the-Loop"] A["Cross-Functional Team<br/>(Eng, PM, Design)"] --> B["Weekly Labeling Sessions"] B --> C["Curated Ground Truth Dataset"] end subgraph Tooling["Internal Streamlit Workspace"] D["Claude Opus-Generated UI<br/>(Clockwork / Streamlit)"] --> B E["Rapid Vibe/Clock Coding<br/>by Non-Engineers"] --> D end subgraph Verification["Continuous Evals"] C --> F["Targeted Eval Runs<br/>(Started with 5 test cases)"] F --> G["Instant, Interpretable Pass/Fail"] G --> H["Fast Model & Context Iteration"] end ```

To establish this baseline, Ramp organized weekly cross-functional labeling sessions with all stakeholders on the product. This practice yielded two major benefits: it created a hardened, verified ground truth dataset for testing, and it ensured seamless alignment across teams whenever the agent missed context or erred. While evaluating third-party annotation vendors, Ramp found commercial tools either overly rigid or too broad. Instead of wasting weeks evaluating software, the engineers used Claude Opus to vibe-code a custom Streamlit data-labeling interface in an internal workspace named Clockwork. Because the UI lived in an isolated codebase, non-engineers could easily modify it on the fly, with changes deploying in seconds. Armed with this dataset, the team ran evaluations continuously, purposefully starting small with just five mission-critical test cases rather than letting perfectionism stall their momentum.

- Will Koh: “Turns out the users are actually incorrect. They're wrong. They are sometimes, you know, they don't know the expense policy, you know, they trust their employees, they're lazy, it's a Sunday, who knows.” - Will Koh: “I wouldn't say that, you know, don't let perfectionism, you know, get in the way. You don't need a full data set of a thousand data points that you're testing against every iteration. We started with five.”

[19:15](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=1155s) Evals in CI, Online Health Checks, and Seamless Model Upgrades

Building agents requires continuous regression testing, because every time engineers attempt to improve an agent's reasoning—such as injecting additional prompt context, adding new tools, or tweaking instructions—unforeseen side effects frequently emerge. A seemingly helpful docstring update or supplementary guideline can easily conflict with existing prompts or mislead the model. Integrating offline evaluation datasets into continuous integration (CI) ensures that unintended regressions are blocked before code is merged. Alongside offline evaluations, online evaluations serve as critical leading indicators of system health. For Ramp's Policy Agent, tracking the rate of "unsure" decisions—instances where the agent lacked sufficient context to render an expense judgment—provides a straightforward, real-time pulse on production performance.

Moreover, having a robust evaluation benchmark in place decouples system logic from underlying model vendors. Whether Anthropic or OpenAI ships a major new release, teams can evaluate the delta cleanly without guessing. As the speaker explains:

> “Whenever you think you're doing something right for the LLMs or agent, giving more context, giving it tools, more likely than not it's probably going to have some kind of bad, you know, consequence that you didn't see happening.”

- **CI Regression Gates**: Validates prompt revisions, tool signatures, and docstring clarity on historical offline test sets before code hits production. - **Online Leading Metrics**: Monitoring telemetry such as the volume of "unsure" outputs flags runtime anomalies or missing context immediately. - **Confident Model Swaps**: Benchmarking suites allow teams to rapidly adopt next-generation foundation models or catch regressions caused by behavioral shifts in updated model weights.

---

[20:46](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=1246s) The Finance "CLAUDE.md" and Earning Autonomous Trust

Engineers are comfortable controlling generative code tools through context files like `CLAUDE.md`, specifying exact rules such as suppressing inline comments. Ramp discovered an unexpected parallel on the finance side: finance operators care deeply about their company expense policy documents, which effectively act as their own prompt configuration files. Initially, finance teams were nervous about modifying policy documents because corporate compliance policies typically require navigating substantial bureaucratic red tape. However, once Ramp provided immediate feedback loops showing how policy edits instantly altered agent evaluation decisions, finance administrators embraced the process enthusiastically.

Deploying the Policy Agent into large enterprise organizations required deliberate trust-building. Ramp deliberately started with Fortune 500 enterprise customers because their transaction volume made reviewing hundreds of small expenses like coffees a massive operational drain. Rather than taking autonomous actions out of the gate, the system initially provided only non-binding "suggestions." Once administrators saw consistently accurate recommendations, they proactively requested full automation for low-risk spend. In response, Ramp introduced an autonomy slider, empowering customers to configure automatic approvals for transactions under specific thresholds (such as $200) while keeping human oversight on the rest.

> “Don't We didn't do any autonomous action. We're just like, 'Hey, we're going to give you a suggestion.' That's That's how That's kind of how we phrased it. Suggestions. And then eventually they came to us and were like, 'Okay, you know what? I want to go from suggestions to auto approvals. Like anything under $200, you guys are mostly right. I don't care about this. Let me just go auto approve it.'”

- **In-Product Feedback Loops**: Giving users direct mechanisms to refine policy texts allows non-technical operators to iteratively improve agent accuracy on their own. - **Enterprise Adoption Strategy**: Started with high-volume enterprise spenders where small-ticket reviews consumed disproportionate human hours. - **The Autonomy Slider**: Shifted governance from advisory suggestions to automated approvals on a configurable dollar-threshold basis once user confidence matured.

---

[22:46](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=1366s) AI Infrastructure as a Service: Unifying SDKs, Batching, and Cost Tracing

Ian Tracey introduces the operational foundation supporting Ramp's applied AI products, framing the effort as both a challenging infrastructure initiative and an internal cultural transformation designed to provide leverage for engineers and cross-functional partners alike. At the core of this strategy is Ramp's internal Applied AI Service. While conceptually resembling an LLM proxy or routing layer such as LiteLLM, the platform layers on three crucial internal capabilities:

```mermaid flowchart TD accTitle: Ramp Applied AI Service Architecture accDescr: Shows the Applied AI Service abstracting model providers, batch processing, and cost tracing for downstream teams.

subgraph Clients["Internal Product Teams"] P1["Policy Agent"] P2["Reimbursements"] P3["Internal Tooling"] end

subgraph CoreService["Ramp Applied AI Service"] A["Unified SDK & Structured Output"] B["Batch Processing & Rate-Limit Engine"] C["Cost Tracing & Pareto Analysis"] end

subgraph Providers["Model Layer (1-Line Config Swaps)"] M1["OpenAI (GPT)"] M2["Anthropic (Claude)"] M3["Google (Gemini)"] end

Clients --> CoreService A --> Providers B --> Providers C --> Providers ```

The service offers a uniform SDK that guarantees structured outputs across constantly shifting provider APIs, shielding feature teams from provider-specific breaking changes. Secondly, it manages heavy batch processing, workflow orchestration, and rate-limit backpressure for bulk document extraction and asynchronous evaluations across providers like Anthropic. Finally, the service captures fine-grained cost tracing attributed across teams and products, identifying cost-performance Pareto frontiers to ensure new AI services remain economically sustainable. Upgrading downstream systems to frontier models requires only a single centralized configuration change rather than updating dozens of disparate call sites across the codebase.

Ian emphasizes the dual nature of scaling applied AI inside the organization:

> “This section is pretty intentionally named AI infrastructure and culture cuz we think that this is both like a really challenging infrastructure problem, but it's also a really challenging culture problem and changing how you work as well is a big part of the story.”

- **Centralized Abstraction**: Downstream product engineers swap underlying foundation models via single-line configuration updates without rewriting application-level calls. - **Managed Rate Limits and Batching**: Handles offline job queuing and provider quotas centrally for document parsing and eval suites. - **Cost and Unit Economics**: Granular cost-allocation dashboards allow Ramp to identify unsustainably expensive workflows early and optimize model selection per use case.

---

[25:52](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=1552s) Curing Hallucinations with an Internal Tool Catalog

Handling sensitive financial workflows and enterprise transaction data makes safety and hallucination prevention top engineering priorities. Ramp's primary mechanism for keeping models grounded is an expansive internal tool catalog built in collaboration with domain product teams. Instead of letting agents generate outputs unconstrained, agents execute discrete, deterministic operations against curated business logic—such as fetching specific policy snippets, determining per-diem rates, or querying recent transaction histories.

This shared catalog provides immediate visibility into capability gaps while serving both core production flows and rapid internal prototypes. An engineer experimenting with a new reimbursement agent can assemble existing production tools without learning underlying backends or rebuilding integrations from scratch. Ramp's catalog already spans hundreds of standardized tools, with long-term expectations of expanding into multiple thousands as agent capabilities scale across the broader financial infrastructure.

As Ian notes regarding the balance between safety and agent power:

> “And I think often times, uh you know, something that I hear from engineers in the space is this kind of concept of hallucination and safety, and how are you actually going to be able to produce a lot of these things to have benefits to downstream finance teams? And we're pretty big believers that it all comes down to the catalog of tools that teams are building and integrating with on a daily basis.”

- **Shared Building Blocks**: Curates pre-built, domain-specific utilities (e.g., policy queries, per-diem calculations, transaction lookups) accessible across all microservices. - **Prototyping Velocity**: Developers can stand up "vibe-coded" prototypes using existing, vetted financial data connectors without engineering bespoke APIs. - **Scale Trajectory**: The catalog has grown to several hundred tools and is targeted to reach thousands of fine-grained capabilities.

[27:25](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=1645s) Ramp Inspect: Context Fragmentation and the Background Coding Agent

Even when developers leverage modern developer tools like Claude Code or Codex, an enormous amount of daily engineering context remains completely fragmented and unintegrated. A developer's workflow requires pulling logs from Datadog, checking production database read replicas, navigating alerting pipelines, parsing alerts from incident.io, retrieving context from scattered Slack threads, referencing Notion documentation, and tapping into the tacit domain knowledge held by specific product teams. At the end of last year, Ramp tackled this context fragmentation directly by building their own internal background coding agent, named Ramp Inspect (which Ramp has documented and open-sourced in blueprint form). Rather than functioning as a synchronous inline autocomplete tool, Ramp Inspect runs autonomously in the background while engineers are sitting in meetings or when bug reports and incident alerts arise.

The operational adoption across Ramp has been massive: Ramp Inspect is currently responsible for over 50% of the pull requests merged into production on a weekly basis. Ramp tracks adoption metrics across the company on an internal leaderboard to foster subtle, healthy competition and inspire broader team participation. While engineering accounts for the largest share of sessions, non-engineering departments—including product management, design, risk, legal, corporate finance, marketing, and customer experience (CX)—actively initiate sessions. These teams use Inspect to execute simple copy adjustments, modify business logic, investigate bugs, and respond to live incidents without waiting on engineering bottlenecks.

```mermaid flowchart TD accTitle: Ramp Inspect Architecture and Context Ingestion accDescr: Diagram illustrating how Ramp Inspect aggregates multiple context sources into Modal sandboxes to generate GitHub PRs.

subgraph Inputs["Trigger Channels"] A["Kanban UI"] B["API"] C["Slack Thread (Full History)"] end

subgraph Context["Integrated Context Layer"] D["Datadog Logs"] E["DB Read Replicas"] F["incident.io & Alerts"] G["Notion Docs & Team Context"] end

subgraph Execution["Modal Sandbox Environment"] H["Isolated Dev Sandbox"] I["Task Tracking Engine"] J["VS Code via VNC"] K["Chrome DevTools & MCP"] L["150k+ Test Suite & CI"] end

Inputs --> H Context --> H H --> I I --> J J --> K K --> L L --> M["GitHub Branch & Verified PR"] ```

- Ramp Inspect accounts for more than 50% of all pull requests merged to production each week. - Sessions run in isolated Modal sandboxes that mirror Ramp's production development environment and can spin up, pause, resume, and tear down rapidly. - Sandboxes include a full VS Code environment streamed over VNC with Chrome DevTools and Model Context Protocol (MCP) integrations for full-stack engineering. - The agent has direct access to Ramp’s test suite of over 150,000 automated tests; it monitors CI outcomes in GitHub, interprets test failures, and iteratively patches broken code before pinging the engineer that the PR is ready for review. - The blueprint of the system architecture is published at `builders.ramp.com`, and an open-source implementation is maintained in the `open-inspect` GitHub repository.

---

[29:48](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=1788s) Multiplayer-First Design and Collaborative AI Tooling

A crucial design philosophy behind Ramp Inspect is its multiplayer-first architecture. Rather than isolating the coding agent inside an individual engineer's local IDE, sessions can be shared directly with cross-functional partners like product managers and designers. When pairing with a designer or PM, an engineer can watch their prompts execute in real time, help them refine their prompting strategies, or debug when an interaction fails unexpectedly. This collaborative feedback loop transformed the agent from a solitary developer tool into an active driver of cross-functional team alignment.

Inspect sessions can be kicked off through multiple interfaces: a Kanban board web UI, an internal programmatic API, or directly from an ongoing Slack conversation. When triggered inside Slack, the system automatically pulls the full conversational context of the preceding thread, eliminating the need for users to manually copy-paste background details, reproduction steps, or user error descriptions into a fresh prompt window.

- Multiplayer sessions allow non-technical partners to share live links, flag unexpected edge-case behaviors, and quickly level up their prompting skills alongside engineers. - Triggering an agent run from Slack preserves all historical thread context automatically, avoiding redundant back-and-forth prompting.

---

[31:17](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=1877s) Engineering Divergence: Team A vs. Team B

With routine firefighting, copy updates, and minor bug fixes increasingly handled by background agents, Ramp's leadership has reevaluated what high-leverage software engineering looks like in an AI-native environment. The speaker introduces a comparative thought experiment contrasting two archetypes of engineering teams: "Team A" and "Team B." Team A focuses relentlessly on business impact, thrives in ambiguous problem spaces, understands product goals, data, and business drivers, adopts new tools quickly, invents creative workarounds, and obsesses over the end-user experience. Team B, conversely, falls back on debating library choices, introduces bureaucratic process whenever complexity increases, continually lobbies for headcount increases, bike-sheds subjective aesthetic debates (such as functional programming paradigms versus specific TypeScript versions), jumps to write code before understanding the fundamental problem, and fixates on performative code quality or subjective nitpicks.

Reflecting on working across both environments, the speaker argues that the rise of AI agents will dramatically widen the performance and career divergence between these two mindsets. While recent academic analyses—such as a late-2024 Harvard study on junior versus senior hiring trends—attribute shifting engineering employment patterns purely to years of experience, the speaker disputes that framing. Raw coding speed was never the most difficult aspect of engineering; instead, higher-level judgment, domain context, and hard-earned institutional experience are what distinguish truly impactful engineers from those who simply produce syntax.

> “And I think one thing that a lot of the narratives that we see in the media gets get wrong about coding agents is they don't really identify the fact that you could still build the wrong thing just a lot faster, and you can build like bigger messes.”

| Quality Dimension | Team A (Impact-Driven) | Team B (Process- & Tech-Driven) | |:--- |:--- |:--- | | **Primary Focus** | User experience, business outcomes, and impact | Library choices, syntactic paradigms, and headcount complaints | | **Handling Chaos** | Explores ambiguous spaces with creative solutions | Imposes administrative process and organizational overhead | | **Tool Adoption** | Eagerly experiments with and operationalizes AI agents | Relies on performative nitpicks and bike-shedding code reviews | | **Problem Approach**| Digs deep into product, business context, and data first | Starts "vibe coding" before grasping the underlying problem |

---

[34:16](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=2056s) The Long Middle, Infinite Demand, and Software's Future

When assessing senior and Staff-plus engineers, organizations pay primarily for their judgment, their foresight, and their scar tissue—the hard-won perspective needed to recognize when a proposed architecture from an advanced model like Claude 3.7 Sonnet or Claude Opus is fundamentally flawed or unworkable. In an environment saturated with AI hype and discourse around SaaS market valuations, public narratives often overlook how difficult it is to take software from a prototype to a durable production system. While "vibe coding" a proof-of-concept is trivial, the hardest parts of engineering remain distinctly human: uncovering what actually needs to be built, deeply understanding customer workflows, convincing skeptical stakeholders that an ambitious technical bet is worthwhile, making architectural decisions under severe information scarcity, and maintaining engineering momentum through the grinding, unglamorous "long middle" of a project.

Addressing broader industry anxiety and doomerism surrounding software engineering careers, the speaker rejects the idea that AI will cap the demand for developers. Unlike agriculture or manufacturing assembly lines—where output reaches clear physical saturation points—software development is fundamentally open-ended and never truly completed. Internally, Ramp embraces the working mantra "job's not finished." Freeing engineering capacity from low-level debugging and manual implementation tasks does not eliminate engineering jobs; instead, it enables organizations to aggressively pursue high-value, complex opportunities and agentic workflows that were previously deemed too expensive or resource-constrained to attempt.

> “Unlike maybe factory work or farming, software's never done. We have this really kind of like meme internally where we say, you know, jobs not finished.”

[35:54](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=2154s) Expanding Ambition and Raising the Bar for Software

The conversation concludes with a broader perspective on how AI shifts engineering strategy, far beyond simple labor reduction or headcount optimization. Rather than using productivity gains to cut teams in half, organizations are using the newfound capacity to tackle ambitious problems across the financial stack that previously seemed insurmountable. Teams are actively moving into adjacent markets, stitching together significantly more comprehensive value for customers, and fundamentally rethinking what kind of software they can justify building. As the speaker explains:

> “It’s not going to be like because everyone’s 2x more productive, you need two less or half the people.”

Instead of shrinking teams, companies will direct their engineering leverage toward rebuilding legacy or brittle systems that were previously deemed far too expensive or risky to touch. Projects that once seemed completely outside a company's core focus suddenly become viable investments. For instance, developing an internal background coding agent within a company focused primarily on financial operations software might have once seemed like a bizarre distraction, but in this new paradigm, it makes complete sense. Ultimately, the next decade of software development will be defined by an aggressive push to elevate what "good enough" looks like, enabling teams to deliver previously unimaginable user experiences.

- **Expanding footprint:** AI efficiency is channeled into entering adjacent markets and bundling deeper customer value rather than trimming headcount. - **Tackling legacy tech debt:** Systems previously considered too costly or complex to overhaul are being rewritten and modernized. - **Unconventional internal tools:** Building internal autonomous coding agents now represents a high-leverage initiative even for specialized fintech companies. - **A decade of higher standards:** The coming decade will center on drastically raising user expectations and shipping transformative product experiences across the industry.

## Notable quotes > [12:32](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=752s) A smaller black box becomes a bigger black box as the system becomes more complex. — Will Koh

> [34:16](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=2056s) Unlike maybe factory work or farming, software's never done.

> [15:29](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=929s) You don't need a full data set of a thousand data points that you're testing against every iteration. We started with five. — Will Koh

> [19:15](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=1155s) Whenever you think you're doing something right for the LLMs or agent, giving more context, giving it tools, more likely than not it's probably going to have some kind of bad, you know, consequence that you didn't see happening.

> [10:40](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=640s) AI products cannot be one-shotted. — Veeral Patel

## Mentioned Nik Koblov, Veeral Patel, Will Koh, Ian Tracey, Ramp, The Pragmatic Summit, OmniChat, Policy Agent, Brian, Andrej Karpathy, Claude Opus, Claude 3.7 Sonnet, Claude Code, OpenAI, Anthropic, Google, Streamlit, Clockwork, LiteLLM, Applied AI Service, Ramp Inspect, open-inspect, Modal, Datadog, incident.io, Notion, Slack, GitHub, VS Code

## Video description With Nik Koblov (EVP Engineering), Veeral Patel (Director, Applied AI and Spend), Will Koh (Staff Engineer, Applied AI), and Ian Tracey (Staff Software Engineer). Recorded at The Pragmatic Summit www.pragmaticsummit.com

Watch the session with Q&A also included: https://newsletter.pragmaticengineer.com/p/the-pragmatic-summit-recordings

0:00 Intro 3:31 Pivot: from many agents to one agent with many skills 7:17 Policy Agent 12:21 Tooling lessons 22:46 Infrastructure + culture 26:13 Tool catalogs, sandboxing, and reliability at scale

[Source](https://www.youtube.com/watch?v=NMs8C2_3M0w)

Reading summary

Ramp's engineering leaders argue that building successful AI products requires shifting from fragmented bots to unified agents with curated tools, grounded by rigorous domain context, custom evaluations, and shared internal infrastructure. Based on their internal deployments, they contend that increased engineering leverage will expand product scope and ambition rather than shrink development teams. Architectural Consolidation and Policy as Code (3:31) Ramp initially experimented with siloed agents across teams, which produced roughly four separate implementations for synchronous and background tasks. The team concluded that building thousands of isolated agents was the wrong trajectory, instead consolidating their stack around a single foundational agent equipped with an expanding catalog of tools. To streamline interaction, Ramp combined five conversational prototypes into OmniChat, an interface that operates alongside standard UI elements across their platform. An in-house orchestration layer allows administrators and product managers to compile plain-English instructions into deterministic workflows, such as employee onboarding and spend policy checks. Iterative Context Enrichment and Ground Truth (10:40) Early testing of the Policy Agent revealed that model errors usually stemmed from missing domain context rather than underlying LLM defects. By dogfooding on low-risk transactions like coffee expenses, the team discovered that interpreting natural-language compliance rules required feeding in HRIS role hierarchies and itemized receipt details. Architecture evolved from deterministic sequential chains to conditional tooling, and ultimately into full agentic loops that read Ramp platform data and log audit rationales autonomously. Evaluation Infrastructure and Gradual Autonomy (15:29) Ramp abandoned user approval actions as a ground truth baseline because managers often approve expenses out of fatigue, unfamiliarity with policy, or convenience. Instead, the team held weekly cross-functional labeling sessions and built a custom Streamlit labeling tool using Claude Opus, starting evaluations with just five mission-critical test cases. To avoid regressions, offline evaluation benchmarks were embedded into continuous integration pipelines, while production health was tracked via leading indicators such as the rate of 'unsure' agent decisions. Enterprise adoption followed an autonomy slider model: high-volume customers began with non-binding agent suggestions and transitioned to automated approvals only after confidence was established under specific thresholds like $200. Centralized AI Platform and Tool Catalogs (22:46) To manage execution risks and provider instability, Ramp created an Applied AI Service that unifies structured SDK outputs, manages batch processing, and tracks cost-performance Pareto frontiers across model providers with single-line configuration swaps. To mitigate hallucinations in sensitive finance workflows, agents are restricted to executing discrete operations against a shared catalog of hundreds of domain tools. This centralized catalog allows engineers to assemble production-grade workflows without rebuilding backend integrations. Internal Coding Agents and Evolving Engineering Roles (27:25) To counter context fragmentation across logs, database replicas, Slack, and alerts, Ramp deployed an internal background coding agent called Ramp Inspect, which runs in isolated Modal sandboxes equipped with VS Code, Chrome DevTools, and access to a 150,000-test suite. The agent accounts for over 50% of weekly merged pull requests across Ramp, with cross-functional participation from design, product, and legal teams. Leadership claims this shift widens the divergence between business-impact-focused engineers and process-driven engineers, emphasizing that architectural judgment and deep domain understanding remain critical human responsibilities. Ramp's presenters conclude that AI agents will not eliminate software engineering jobs because software development, unlike physical assembly lines, is never finished. Instead, productivity gains are expected to be directed toward rebuilding legacy technical debt, entering adjacent markets, and raising baseline product quality across complex financial operations.

Source quotes

[12:32] Will Koh: A smaller black box becomes a bigger black box as the system becomes more complex. [34:16] Unlike maybe factory work or farming, software's never done. [15:29] Will Koh: You don't need a full data set of a thousand data points that you're testing against every iteration. We started with five. [19:15] Whenever you think you're doing something right for the LLMs or agent, giving more context, giving it tools, more likely than not it's probably going to have some kind of bad, you know, consequence that you didn't see happening. [10:40] Veeral Patel: AI products cannot be one-shotted.

Original transcript

[0:05] Today we're going to talk about AI at [0:07] Ramp and uh [0:10] I'm going to give an intro quick [0:11] introduction into what Ramp is. [0:13] Um [0:14] really briefly, we're going to walk [0:15] through the simplest possible expense [0:17] use case that you guys can all resonate [0:19] cuz I see everybody's drinking coffee. [0:22] And [0:23] then we're going to [0:24] talk quickly about a lesson that we [0:27] learned [0:28] this year while we were building [0:29] Brazilian agents. Um and sort of the [0:32] pivot in the paradigm that's happening [0:34] especially after February 6th. [0:36] And [0:38] then we're going to double click onto [0:39] how we built one of our most popular [0:41] agents, the policy agent. [0:43] Um and then finally, we'll dig in into [0:46] the infrastructure built that this is [0:49] requiring requiring to do on our side. [0:52] And in my mind most importantly, the [0:54] culture shift that needs to happen on [0:57] everyone's teams in order to be able to [0:59] operate in a way that delivers products [1:02] into the hands of your customers in the [1:04] fastest and most impactful way. [1:06] Uh so without further ado, [1:09] quick intro about Ramp. We are number [1:11] one finance platform for modern [1:13] businesses with 50,000 plus customers [1:17] and we're in the business of saving you [1:19] time and money. [1:21] Uh we have uh [1:23] I've seen some of the some of those [1:25] names on the on the name tags here. So [1:26] thank you for being Ramp customers. Uh [1:30] Really exciting. Uh really quickly, so [1:33] cup of coffee [1:34] takes [1:36] usually about 15 minutes of your time [1:40] cuz you got to do these three simple [1:42] things which unfortunately take minutes. [1:45] This compounds through the company. [1:47] And what Ramp does in the simplest [1:50] possible way, we just condense time and [1:53] return money back. [1:55] Uh so a simple story over a transaction [1:58] from tapping the card to writing a memo [2:00] to classifying the transaction according [2:02] to your GL to sourcing the receipt, [2:05] attaching the receipt, um, [2:07] normalizing the merchant to your, um, [2:10] inventory of merchants is all done [2:12] agentically at Ramp. And this was our [2:14] first foray, uh, probably by now, [2:17] uh, you guys still feel here? Yeah. [2:19] Probably by now about 3 years ago, we [2:20] started doing this one-shot things with [2:22] AI. Uh, normalize merchant, write a [2:25] memo, and it's been working really, [2:27] really well as the models get better. [2:29] Uh, what else is going on at the [2:31] company? Well, literally every persona, [2:35] uh, at the company is wasting time on a [2:40] lot of manual work. Uh, so from AP [2:43] clerks to your finance team, from your [2:45] purchasing teams, uh, keep going to more [2:48] finance work, your data teams, uh, at [2:51] Ramp we used to have a channel called [2:53] help data where somebody will ask for a [2:55] CSV and a poor person will go and write [2:57] a SQL query. [2:58] Uh, it we replaced it about, uh, a year [3:01] and a half ago. [3:02] Uh, so a lot of time being spent and the [3:05] complexity has a ramp shape. It only [3:07] increases as you go through different [3:09] jobs to be done. Um, so if you guys [3:12] watch Super Bowl, uh, you might be [3:13] familiar with Brian, um, our agent. Uh, [3:16] so we've been writing a lot of agents [3:18] literally for every job to be done to [3:20] cover the entirety in the end state, the [3:23] entirety of what admins, employees, and [3:26] finance teams are doing that is not [3:29] directly related to making the money. We [3:31] want you all to be making money and [3:33] focus on your customers, not on how to [3:35] close the books. [3:37] Uh, [3:38] but what's been happening for the past [3:40] few weeks is, uh, that we're living [3:43] through the most exciting paradigm shift [3:45] in software, um, and it requires [3:48] complete rethink. And with rethink, [3:51] simplification of your stack. Uh So, [3:54] what we learned is you don't need to [3:56] build a thousand agents. We [3:57] intentionally last year allowed each [3:59] individual team to go and experiment. [4:01] And we ended up maybe with four [4:03] different ways of doing the same thing [4:06] both for synchronous agents as well as [4:07] for background agents. Um but instead [4:11] you want to drive your framework towards [4:15] a single agent with a thousand skills. [4:19] Uh So, let's talk about what the [4:21] software traditionally used to focus on. [4:23] So, every process, um especially in the [4:26] modern modern AI stack, boils down to [4:28] having an event. Um so, a prompt you can [4:31] receive an invoice and you want to pay [4:32] it. Um some prompt instructions of what [4:36] you want to do with it and some [4:37] guardrails like a policy, uh like an [4:39] expense policy or your payables policy. [4:41] Um context, what is the data that the [4:44] agent should consider. And then finally, [4:47] tools. These are APIs and actions that [4:50] you can do. And traditionally software [4:51] would focus on only four and five. [4:54] Uh In the new paradigm [4:56] software is doing everything. So, you [5:00] want to focus on building an autonomous [5:03] system of action that can react, reason, [5:06] and act without a human or with very [5:08] little human supervision. [5:11] Um so, what does it mean in terms of [5:13] what we're building? [5:15] So, first, uh [5:17] we decided we go into consolidate the [5:20] interactions. [5:22] Um verbal interactions uh [5:25] with the agents to a single [5:27] conversational UX. Uh We literally at [5:29] the end of last year we had about five [5:31] different conversational UXs. We now [5:33] have consolidated it into what we call [5:35] an OmniChat. Omni meaning for [5:37] omnipresent. It is now being deployed to [5:39] every surface of the product. And it [5:42] works well with the traditional UX [5:43] because you still need tables and [5:44] buttons. And uh you don't always want to [5:47] be talking uh to your software. [5:50] But this is a good example what Omni [5:52] chat looks like. Please onboard a new [5:54] employee [5:55] Omni chat can resolve an employee to an [5:59] employee ID and look up through an HRIS [6:01] tool [6:02] their corporate structure and it found a [6:05] workflow agentic workflow that we [6:07] created previously called the new hire [6:09] playbook. And the agent is asking, would [6:11] you like me to to onboard the person [6:13] using this playbook? [6:14] How is this possible? We built a [6:16] in-house lightweight agent framework [6:18] that provides orchestration with tools [6:22] that engineers are very quickly building [6:24] and most recently we have one product [6:26] manager Vibe coded about 20 tools so [6:28] engineers are no longer needed to build [6:30] these tools. Um [6:32] and sometimes your workflows are [6:34] involved such as employee onboarding [6:37] consists of four steps. So you can just [6:39] go and ramp and describe what do you [6:41] want to happen when a new employee [6:42] joins, give them a card, make sure they [6:46] get receipts for every transaction, [6:47] congratulate them on on Slack and check [6:49] in with them in two weeks. [6:51] We now are able to compile this into a [6:54] runnable deterministic workflow [6:56] and then give it to the agent to [6:58] execute. Playbooks make use of tools [7:02] and how this all comes together [7:05] this is an example which Viral is going [7:07] to double click next is [7:10] upon swiping the card [7:12] there's a real-time policy review that's [7:14] happening directly in the software [7:17] and policy agent enforces [7:20] your company requirements with regard to [7:23] spend. [7:24] Therefore, it's very safe to give Ramp [7:26] cards to literally every employee in [7:27] your company. And there's a handoff [7:29] happening with an accounting code and [7:31] agent that classifies this transaction, [7:34] applies the rules of your back office [7:37] team of your finance team. As an [7:39] employee I have no idea how certain [7:41] transaction should match to our GL, and [7:43] that's what typical traditional products [7:45] would do. They will expose it to you. Um [7:47] so the agent is much better doing it [7:49] because it has the full context of your [7:51] chart of accounts, it understands your [7:52] ERP, [7:54] and then it can either auto-approve or [7:56] in the worst-case scenario, it will [7:57] involve uh the human in the loop to [7:59] review my materiality or notify that [8:02] there is an out-of-policy spend. [8:04] Um with that, uh please welcome Viral, [8:07] who will [8:08] dive deeper into the policy agent. [8:12] Thanks, Nick. [8:17] Oops. [8:19] Awesome. So, a lot of finance teams are [8:22] looking at receipts like this basically [8:24] every day, and maybe they might have [8:26] hundreds or thousands of these. If you [8:28] told me to look at this and decide if I [8:30] should approve or reject this [8:31] transaction, I'm probably going to make [8:32] a mistake. [8:34] So, policy agent basically reasons on [8:36] this image and all the transaction data [8:38] that we have and told me that there were [8:40] eight guests in the receipt. I could [8:42] barely see that when I was looking at [8:43] it. Uh it was below the $80 a person cap [8:46] that we have internally. [8:48] Uh they were going for team welcome [8:50] dinner. Uh and so because the amount was [8:53] verified as well and the merchant, uh [8:55] policy agent told me to approve this [8:56] transaction. [8:59] Similarly, for this open AI transaction, [9:01] Anan was testing out um some some [9:03] ChatGPT features, and so policy agent [9:06] told me this was a valid uh business [9:08] expense and told me to approve it. And [9:10] then this $3 bakery charge was told uh [9:13] was was uh rejected because uh it wasn't [9:17] uh part of an overtime purchase and it [9:19] didn't happen on the weekend. [9:22] So, really we looked at this as an [9:24] opportunity to rethink how Ramp was set [9:27] up. Um controllers and finance teams are [9:30] looking at transactions like these and [9:32] and making these decisions every day. [9:34] And a Fortune 500 company that is one of [9:36] our customers was coming to us and [9:38] saying, "Hey, can you uh uh make sure [9:40] that you approve these types of expenses [9:42] and reject these types of expenses?" And [9:43] they basically had a list of all the [9:45] rules that uh Ramp uh should should [9:47] follow. And we kind of saw this as an [9:49] opportunity not to kind of add more [9:52] incremental deterministic rules that [9:55] kind of define our product. And I worked [9:56] on some of the first versions of these, [9:58] um but actually kind of take out uh a [10:01] page from Andrej Karpathy saying that [10:03] English is the new programming language [10:05] and kind of turn the expense policy into [10:07] the rules themselves. So, [10:09] um [10:10] you can you can see Ramp's expense [10:11] policy on the left and and this is a [10:13] screenshot from our production [10:14] environment, but we are seeing really [10:16] great uh use out of our policy agent [10:19] product. And it kind of needed to start [10:23] it kind of needed to start really um [10:25] organically. So, we kind of operated [10:27] like a early-stage startup. We're [10:28] already very incremental and and and [10:30] fast at Ramp, but uh we found some [10:32] design partners like that Fortune 500 [10:34] company. We iterated really quickly, and [10:37] we had weekly weekly meetings with all [10:39] of them to kind of understand exactly [10:40] what uh feedback we wanted to hear and [10:43] what what we could improve. [10:46] I think one of the main important um [10:49] I guess things that we realized across [10:51] uh Ramp is that we really needed to lean [10:54] into the fact that AI products cannot be [10:56] one-shotted. You need to start with [10:58] something simple. And so, as long as [11:00] everyone on your team, PMs, designers, [11:03] engineers are aligned that you're not [11:04] going to have perfection on day one, I [11:07] think that was actually one of the main [11:08] like cultural learnings. Um and so, we [11:11] dogfooded a lot of this work internally [11:13] uh and started with an even more [11:14] constrained problem of trying to decide [11:17] whether our coffee with a colleague [11:18] transaction should be approved or [11:20] rejected. These are single uh [11:22] uh [11:23] dollar amount transactions that are low [11:24] risk um [11:26] according to our finance team. And so, [11:28] we started uh with these transactions. [11:30] And uh one of the early learnings, [11:32] especially as we kind of release this [11:35] into production, was that a lot of the [11:37] reason that policy agent would be wrong [11:39] would be less on the models themselves [11:41] and more about the context that we were [11:42] giving [11:43] to to LLM's themselves. So, we we could [11:46] have sat down and thought about all the [11:48] context in the beginning before we even [11:50] kicked off any engineering work, but we [11:52] realized actually the best thing would [11:53] be to learn from some of our live [11:55] internal data. And so, for example, we [11:58] learned that the role in the title of an [12:00] employee is super important when looking [12:02] at expense policy docs or in level [12:04] C-suite, for example, might have higher [12:06] limits. Maybe they can fly on first [12:07] class for for certain flights. And so, [12:09] we started extracting more information [12:11] from receipts, started pulling in [12:13] information from HRS fields that are [12:15] already on ramp. And so, [12:17] Will is going to kind of talk you [12:19] through exactly the iterations that we [12:21] went through to implement policy agent [12:23] and and some of the learnings along the [12:25] way. [12:32] Is this down? [12:33] It's down. Yeah. Okay. [12:37] All right, cool. Um [12:39] awesome. So, when we first started [12:41] building the policy agent internally, [12:43] we dream we went big. We're like, "Hey, [12:45] let's automate all of finance. Let's [12:47] automate all reviews." But when it came [12:49] down to it, we actually have to start [12:50] small. Is that cup of coffee, you know, [12:53] in your expense policy? And the reason [12:55] that we did that was because even though [12:56] the problem sounds simple [12:58] to automate, you know, is this a simple [13:01] question, is this in policy or not? [13:04] It was going to grow to be complex. Kind [13:06] of like Vimal said, we could have gone [13:07] down and we could have figured out what [13:08] context do we have, how can we add it, [13:10] how can we put it all together in a way [13:11] that LLM can understand, and you know, [13:13] put it all together from the get-go. But [13:16] we knew that even if we aimed and got [13:19] everything right the first time, it was [13:20] probably going to be wrong once you [13:21] applied and generalized it and then to [13:23] another business. [13:24] Um [13:25] so, [13:27] the simpler the system, I think the [13:29] easier it is to iterate on top of it. [13:31] And once you iterate, you know what's [13:32] going to work, you know what's not, and [13:33] you can kind of layer complexity on top [13:34] of that. And I think that's pretty [13:35] important to um keep in mind when you're [13:37] building a um [13:39] LM or an agent starter. So, for us, [13:42] we started really simple, very very um [13:45] kind of the classic, you know, we have [13:46] an expense come in, retrieve the context [13:48] around it, we pass it through a series [13:50] of LM calls that are very well defined [13:52] of like, "Hey, is this in policy? Why is [13:54] it in policy? How can we show the user [13:55] that's in policy?" And then give an [13:57] output that uh makes sense in this way [13:59] to the user. [14:00] Eventually, we learned that each expense [14:02] is kind of different. We can classify an [14:04] expense based on is it travel? Is it a [14:05] meal? Is it entertainment? Do [14:07] conditional prompting, and then retrieve [14:09] context based on that, and then pass it [14:11] through a series of LM calls, and give [14:12] it some tools so that it can also [14:14] autonomously decide, "Hey, um I need [14:16] flight information actually, or I need [14:17] this employee's level." Um and kind of [14:19] layer that on top. [14:21] And a few iterations later, we came to a [14:23] full-on agentic workflow. Um we ended up [14:26] with um complex tools to read across all [14:30] of our platform, and these tools are [14:31] shared across our all of our agents. [14:33] It's not just for policy agent. We have [14:35] a company internal toolbox that all of [14:37] our agents are easily can, you know, [14:39] reach into and use. And we gave it the [14:41] um we gave it the um capability to write [14:44] as well. So, it's now writing decisions, [14:46] it's writing uh reasoning, it's writing [14:48] auto-proving expenses on users' behalf. [14:51] Um and it goes in a loop. So, um you [14:53] know, now it's more of a black box, and [14:55] that's kind of the trade-off you get. [14:57] Um [14:58] as you go from simple to complex [15:00] systems, um your capability goes up, [15:03] your uh autonomy goes up, your agents [15:04] are able to do more, your AI can do [15:06] more, your AI seems smarter. But, in [15:08] exchange, you're going to be able to [15:10] you're losing traceability and [15:11] explainability. Uh we look at it now, we [15:13] can kind of look at the reasoning tokens [15:15] that the LM gives us, but in the end, we [15:16] have no control over it. It's going to [15:18] do what it thinks it's right, it's going [15:19] to make the tool calls, it's going to [15:20] tell you it's right or wrong. So, a [15:22] smaller black box becomes a bigger black [15:24] box as the system becomes more complex. [15:29] So, one thing that is really important [15:31] when doing something like this is from [15:32] the beginning, you need really good [15:33] auditability. [15:34] Um assume even if you know how it it [15:37] works, assume that your inputs and [15:39] outputs are all you know, and make sure [15:40] that it's correct. Um [15:42] so [15:44] if it was a black box system and you [15:45] only saw the input output, can you [15:47] verify that it did the right thing? And [15:48] even if that black box changes, you [15:50] should be able to reason about whether [15:51] the output is correct. [15:53] Um [15:54] as with many products that we built at [15:56] Ramp and across, you know, other [15:58] companies, we thought that the users [15:59] would be correct. Uh you know, if the [16:01] user says approve, the agent should [16:02] approve. If the user says reject, the [16:04] agent should reject. But turns out [16:07] the users are actually incorrect. [16:08] They're wrong. They are sometimes, you [16:10] know, they don't know the expense [16:10] policy, you know, they trust their [16:12] employees, they're lazy, it's a Sunday, [16:14] who knows. Um so, turns out we can't [16:17] always do what the users are doing cuz [16:19] sometimes that's where our finance teams [16:21] come back to you and are like, "Hey, [16:22] this is wrong. This shouldn't be on the [16:24] uh company card." [16:25] So, we have to define our own definition [16:28] of correctness. [16:29] Um and to do that, um we had a weekly [16:31] labeling session with across functions [16:33] that are working on this product. Um and [16:35] that had two um kind of really good [16:37] outcomes. One was that we had a ground [16:40] truth data set that we could always test [16:42] against and we knew that this was [16:43] correct. And two was that everyone was [16:45] on the same page. If our agent got [16:47] something wrong, everyone knew that it [16:48] got it wrong. Or you know, our agent is [16:50] missing context, everyone knew that it's [16:52] missing that context. So, there was less [16:54] communication, everyone's on the same [16:55] page, and um they could focus on what's [16:57] really priority and kind of have [16:59] alignment on that. [17:02] Initially, um [17:04] getting all those people together in a [17:05] room every week, giving them homework to [17:07] label 100 data points, it's expensive. [17:09] You know, that everyone everyone has [17:10] things to do and it sometimes they don't [17:12] come back with their homework done. It's [17:14] just a kind of like almost becomes [17:16] tedious even though it's so important. [17:17] So, we wanted to make it as simple as [17:19] possible, and the way we did that was [17:20] that we looked for third-party vendors [17:23] that could provide us the tools to label [17:24] data and collect the data. [17:26] But, turns out some tools are too [17:28] specific to a use case, some tools are [17:29] too general, and we could have spent [17:31] weeks trying out different tools, but we [17:33] decided let's just build our own. Um so, [17:35] we used Clockwork using Streamlit. We [17:38] basically one-shotted all of this, and [17:40] the greatest part of it all is that it's [17:41] low maintenance, um low risk. It's in a [17:44] particle base that it breaks, [17:46] we can fix it right away. Deploy's [17:47] happening like instant seconds. And [17:49] non-engineers can go and personalize it. [17:50] They can they can vibe code it. They can [17:52] clock code it. And this was at Opus 4. [17:53] So, now at Opus 4.6, I expect it's even [17:56] better, and uh with something like that, [17:58] it's definitely easier and cheaper [17:59] sometimes to do something one-off like [18:00] this. [18:04] And [18:06] with that with the ground truth data [18:07] set, we were able to make quick [18:08] iterations. We're able to find out, [18:10] "Hey, we need employee levels. Add that. [18:11] How does that work?" Running it against [18:13] this data set, does it actually catch [18:14] it? And now say accept or approve. [18:17] Um and we're able to make really quick [18:18] iterations, and that was kind of the key [18:20] um [18:21] that was actually kind of a key point in [18:22] developing this. Uh we had really early [18:24] confidence that this could actually [18:26] work, and we were able to actually buy [18:27] get a lot of buy-in, um get a lot of [18:30] customers on board it and that kind of [18:31] try it out as a design partner. [18:33] Um [18:34] and [18:36] as part of like doing that iteration [18:38] with the data set, you had evals, and I [18:40] feel like evals are very you know, [18:41] obviously everyone I think in the zoom [18:43] now knows about evals and what they [18:44] mean, but um it's pretty important to [18:46] have them early on. I wouldn't say that, [18:48] you know, don't let perfectionism, you [18:49] know, get in the way. You don't need a [18:51] full data set of a thousand data points [18:52] that you're testing against every [18:53] iteration. We started with five. You [18:55] know, and we knew that those five we [18:57] were not going to fail. We kept adding [18:58] and adding and adding. And [19:01] you know, make sure it's easy to run. [19:03] Anyone could go and just run that [19:04] command. And then make sure that the [19:05] results are really easy to understand. [19:07] Um they're able to look at it, get [19:09] instant, you know, output like and [19:10] understand like, "Hey, this is what the [19:11] model's doing. This is like good, this [19:13] is bad, and like if you want to do it as [19:15] part of your CI, then everyone now can [19:17] just hopefully merge in code because [19:20] whenever um [19:22] whenever you think you're doing [19:23] something right for the LLMs or agent, [19:25] giving more context, giving it tools, [19:27] more likely than not it's probably going [19:28] to have some kind of bad, you know, [19:29] consequence that you didn't see [19:30] happening. Context was wrong, um whether [19:33] it be the tool instructions were wrong, [19:35] or maybe the docstring was like a little [19:36] confusing and conflicting. Um so it [19:37] might have consequences. You just want [19:39] to You just want to make sure you're [19:40] catching against those. Um and then I'll [19:42] touch on it briefly, but online evals [19:44] are also great. So these are offline. [19:46] You have a data set, it's historical, [19:47] you're testing it, but if you can, [19:49] online evals can be a little more [19:50] confusing and uh harder to kind of [19:52] measure, but if you can measure anything [19:54] that as your users are interacting with [19:55] the system, definitely as a leading [19:57] metric also set them up. And for us, [19:59] part of that was, hey, how many our [20:00] rates of like decisions. We had an [20:02] unsure decision, which is which just [20:04] meant that the agent didn't have enough [20:05] information. So we can measure that [20:07] online. So it's much simpler eval, but [20:09] that also gave us a pretty good health [20:10] check um as our system was running. [20:15] Cool. And another great part about evals [20:17] is that with evals, you can make [20:19] confident model changes uh whenever a [20:21] new model comes out, Opus 46, GPT-53, [20:24] you want to make sure that you can [20:26] leverage those new models because [20:27] sometimes that could be the difference [20:28] between, you know, your system getting [20:30] one part of the problem right to wrong, [20:32] but it could also be the opposite. It [20:33] could have It could actually be [20:35] not good without any prompt changes or [20:36] changing how your system works. So um [20:38] having evals really set it up and being [20:40] able to benchmark really helps um make [20:42] confident model changes. [20:46] Cool. Um so now that policy agent, we've [20:49] been developing this for a while, it's [20:50] available for everyone on the Man [20:52] platform. Some of the things that we [20:53] learned along the way is that [20:55] um cloud code as engineers is very [20:57] exciting. We have full control, we get [20:58] to modify our cloud MD, we get to make [21:00] sure, you know, tell it to not leave [21:01] comments, it won't leave comments, [21:02] hopefully. Um [21:04] turns out it's It's just us, um finance [21:06] people also really like to have you [21:07] know, modify their cloud MD, which is [21:09] their expense policy. So, if something [21:11] went wrong with the decision, then we [21:13] just like tell them, "Hey, go update [21:14] your policy doc." Which to them it's a [21:16] little scary concept to begin. Like this [21:17] is a document. Like you know, you don't [21:19] mess with that. Um, you have to go [21:20] through a lot of hoops if you're going [21:21] to mess with that. Um, but then it turns [21:23] out if you get them really excited about [21:24] the feedback loop, "Hey, change that. [21:26] You'll see it right away." Turns out [21:27] they'll be like really excited to do [21:28] this. [21:29] Um, and then trust builds over time. So, [21:32] some of the earlier customers that we [21:33] had were some of the Fortune 500. We [21:36] actually started with a really big, you [21:37] know, um, [21:38] enterprise customers that we had cuz we [21:39] thought that they would have the most [21:40] value. They have the most expenses [21:42] coming in. They have the most time to [21:43] spend on reviewing [21:45] coffee expenses. [21:47] Um, so, [21:48] you know, roll it out to them. Let them [21:50] have the trust. Don't We didn't do any [21:51] autonomous action. We're just like, [21:52] "Hey, we're going to give you a [21:53] suggestion." That's That's how That's [21:55] kind of how we phrased it. Suggestions. [21:57] And then eventually they came to us and [21:58] were like, "Okay, you know what? I want [22:00] to go from suggestions to auto [22:02] approvals. Like anything under $200, you [22:04] guys are mostly right. I don't care [22:05] about this. Let me just go auto approve [22:07] it." So, we gave them the autonomy [22:08] slider. We gave them a way to just like [22:10] turn it on and then they actually could [22:12] do it themselves. [22:13] And then, [22:14] last but not least, um, similar to LLMs, [22:17] users thrive in, you know, in-product [22:18] feedback loops. Um, so, you know, when [22:20] you're building an AI product and you [22:22] have a full way of like LLMs can test if [22:25] it's code was right and still to go [22:27] iterate, users are the same way. Um, [22:29] give them in-product ways to improve the [22:31] expense policy doc, improve the agent [22:33] and how it operates. And, um, they're [22:35] more than excited to kind of take it [22:36] over themselves and um, kind of improve [22:38] it and personalize it for them. So, [22:41] um, [22:42] from here I'll pass it on to Ian who's [22:43] going to have to kind of talk about the [22:44] infrastructure and the culture that we [22:45] have at Ramp that kind of, you know, led [22:47] us to building the policy agent. [22:54] Hey, everybody. [22:56] So, you've heard a little a little bit [22:58] about like how we're kind of getting [22:59] leverage to all of the different finance [23:01] teams as we operate on top of their [23:03] financial infrastructure and really try [23:05] to get leverage for our customers. Um, [23:07] but I think a big thing that we also [23:08] spend a lot of time thinking about is [23:11] how can we get leverage for Ramp itself, [23:14] the engineers, our XFN works, all the [23:16] people that we work with um, every [23:18] single day. And this slide is this [23:20] section is pretty intentionally named AI [23:22] infrastructure and culture cuz we think [23:24] that this is both like a really [23:26] challenging infrastructure problem, but [23:27] it's also a really challenging culture [23:29] problem and changing how you work as [23:31] well is a big part of the story. [23:35] And so to kind of start on the [23:36] infrastructure side, the core of how [23:38] most of applied AI happens at Ramp is [23:41] our applied AI surface a service. And at [23:44] like a 10,000 ft view, this looks [23:46] something kind of like an LLM proxy [23:49] or something like light LLM, but there's [23:50] really three kind of main extensions [23:52] that we've invested in to make this a [23:54] lot more powerful for a lot of our use [23:55] cases. [23:56] The first is like structured output and [23:58] consistent API and SDKs across different [24:01] model providers. This can be pretty [24:02] tricky to do especially with how quickly [24:04] the APIs are changing, but it's a [24:05] problem that we don't want downstream [24:07] product teams to have to think about. So [24:09] if you have an idea of I want to switch [24:10] from [24:11] GPT 5.3 to Opus or I want to try Gemini [24:16] 3 Pro, you should be able to do that [24:17] with a config change and really quickly [24:19] be able to iterate on semantic [24:21] similarity and trying to do a bunch of [24:23] different, um, you know, code sandboxing [24:25] and structured output calls that way. [24:27] The other thing that we've spent a ton [24:29] of time thinking about is kind of batch [24:30] processing and workflow handling. This [24:32] is really useful for evals or if you're [24:33] doing like bulk for us bulk document or [24:36] data analysis, [24:37] um, and that's something that we also [24:39] don't want teams to have to spend a [24:40] bunch of time on of how do you want to [24:41] batch this and handle it with rate [24:42] limits and do we want to do this on an [24:44] offline or online job with something [24:46] like Anthropic? We just want to handle [24:48] that for downstream consumers so they [24:49] can just focus on providing value for [24:51] downstream customers. [24:53] And then the last which is a pretty big [24:54] deal is the ability to trace different [24:56] costs across teams and against products [24:58] as well. And this allows us to kind of [25:00] identify the Pareto, you know, curve of [25:02] like what is the best kind of model [25:04] performance for cost, how are these [25:05] evolving over time, what teams are [25:07] actually not, you know, building [25:09] something that's going to be sustainable [25:10] long-term for different product [25:11] services. And this can be really, really [25:13] important to just remove all this work [25:15] from internal teams having to think [25:17] about this. [25:18] And the last thing that's kind of, I [25:20] think, funny to think about and we often [25:21] joke about that, you know, our customers [25:24] are actually using the front more of a [25:25] frontier model than they may even know [25:27] even is out yet, is it allows us to stay [25:29] at the frontier when a new model comes [25:30] out, it's a one-line config change that [25:32] impacts every single SDK downstream. And [25:35] so, rather than teams having to learn [25:36] the SDK or go into 12 or dozens of [25:39] different call sites, they can just [25:41] change it in one place for their [25:42] specific team, and they now get the [25:44] benefit of being on the latest and [25:45] greatest models um that we've kind of [25:47] vetted and built into the rest of the [25:48] system. [25:52] Our product, as you've kind of heard [25:54] earlier, earlier, works on a lot of like [25:56] very sensitive data and very sensitive [25:59] workflows. And I think often times, uh [26:01] you know, something that I hear from [26:02] engineers in the space is this kind of [26:04] concept of hallucination and safety, and [26:07] how are you actually going to be able to [26:08] produce a lot of these things to have [26:09] benefits to downstream finance teams? [26:12] And we're pretty big believers that it [26:13] all comes down to the catalog of tools [26:15] that teams are building and integrating [26:16] with on a daily basis. [26:18] And so, what you're seeing here is our [26:20] internal tool catalog. So, an example [26:23] would be like get a policy snippet or [26:25] per diem rate or a recent transactions. [26:28] And these are built alongside of product [26:30] teams to really understand a lot of the [26:31] nuances in the data and the use case. [26:34] And what's really cool about this is not [26:35] only can you see where there's gaps in [26:36] our offering that, oh, we actually don't [26:38] have a tool for this specific use case. [26:40] These can be used both in internal repos [26:42] and our core product. And so, if you [26:44] have an idea of I want to do a cool [26:45] reimbursement agent idea, here are the [26:48] different ways to integrate the tools, [26:49] the different APIs and systems that they [26:50] integrate with, and now you can [26:51] prototype that on a totally new product [26:53] and Vibe coded surface area without [26:56] having to worry about like learning all [26:57] of these things from scratch or building [26:59] the tools on your own. [27:01] We're up to like many hundreds of these [27:02] tools today and we, as Nick mentioned [27:04] earlier, thinks that this could be like [27:06] multiple thousands over time. [27:11] On the topic of context, another big [27:14] thing we think about is context for our [27:15] customers of how do we actually [27:17] integrate the financial stack and allow [27:18] them to be a little more productive. [27:20] What we noticed like a very similar [27:21] problem internally [27:23] on our engineering team. And I think [27:25] something that's like not as always [27:26] obvious is that, you know, even if [27:28] you're using something like Cloud Code [27:30] or Codex, there's all this fragmentation [27:32] of actually what you do on a daily basis [27:34] to get work done in your company that [27:35] that's not integrated, too. [27:37] There's logs in DataDog, there's a [27:39] production, you know, database that has [27:41] a bunch of things going on, there's [27:43] different alerting systems, there's [27:44] incident.io, there's a Slack message you [27:46] have to pull in, there's a Notion doc, [27:48] and then there's a lot of like knowledge [27:50] that those actual specific product teams [27:51] have of how they actually need to get [27:53] work done as well. [27:55] And so, at the end of last year, we [27:58] decided to start out and try to solve [28:00] this problem of how can we actually [28:01] integrate all this context and build our [28:03] own internal background coding agent, [28:05] which we've called Ramp Inspect. You may [28:07] have seen this on LinkedIn or X. We [28:09] actually have open-sourced the blueprint [28:10] of how we built this, and at the end I [28:12] can definitely show you guys a link of [28:13] where to find that. And the the progress [28:16] has been pretty phenomenal of actually [28:18] integrating this into a background agent [28:20] that can run autonomously as people are [28:22] in meetings, if as bug fixes come up, [28:24] and things like that. And currently this [28:27] month, Ramp Inspect is responsible for [28:29] over 50% of PRs that we merge to [28:31] production. I have some interesting, [28:33] we're like really big nerds with stats [28:35] and numbers and things like that, so we [28:37] have this dashboard to kind of create [28:38] this like interesting, one like subtle [28:41] healthy competition, but also inspire [28:44] people that they can actually use this [28:45] as well. And as you can engineering [28:48] has a huge lead of the amount of [28:50] sessions, but you also have product, you [28:51] also have design, there's risk, legal, [28:54] corporate finance, and even marketing [28:55] and CX teams using Ramp Inspect. And [28:57] they're doing things like simple copy [28:59] changes, they're doing logic fixes, [29:02] they're trying to respond to incidents [29:03] or bugs. And what's been really cool to [29:06] see as this has evolved over time, [29:08] whoop, [29:11] is how we've actually designed a couple [29:13] of these things with some core [29:15] principles to be really powerful. So [29:17] what you're seeing here is a Ramp [29:18] Inspect session. I think this is an [29:19] example of like a query that we were [29:21] trying to fix. [29:22] This spins up in the background really [29:24] fast [29:25] modal code sandbox. This allows us to [29:27] like resume, spin up, and spin down [29:28] these containers in an isolated [29:30] environment which has the same [29:31] environment that you would have if [29:32] you're developing a ramp. [29:34] There's a series of tasks to keep it on [29:35] track and it creates a GitHub branch and [29:37] integrates with all of the context [29:39] documents, our data dog, our read [29:41] replica so we can actually write [29:43] queries, and different context documents [29:45] that product teams have [29:47] have put together. [29:48] And what's really I think subtle about [29:49] how we've designed this is we've [29:51] designed it to be multiplayer first. And [29:53] that means that as you integrate or you [29:55] try to pair with like a designer or [29:57] somebody on the PM team, they you can [29:59] actually help them like level up their [30:01] own prompting skills. They can give us [30:03] feedback of hey, click on this link, [30:04] this actually failed in a way that I [30:06] wasn't expecting. And so that can be a [30:08] really great source of like [30:09] cross-functional collaboration. That was [30:11] a very subtle design choice that we made [30:13] that ended up being a really big impact [30:15] for the company. [30:17] And then these can be kicked off either [30:18] via a Kanban UI, we have an API, and [30:21] then also a Slack thread. And we can [30:23] take the full context of the Slack [30:24] thread when it is actually kicked off so [30:26] you don't have to re-prompt it with a [30:27] bunch of conversation that happened [30:28] earlier. [30:32] What you see here is we also have a full [30:34] VS Code environment. We run VNC inside [30:36] of a modal sandbox as well so this [30:38] allows us to have Chrome DevTools and [30:40] MCP so we can actually do full stack [30:42] work, which is pretty cool. And it has [30:44] access to the 150 plus thousand tests [30:46] that we have. So, it also knows if [30:47] things are broken, can respond to the CI [30:50] inside of GitHub, and actually patch [30:52] fixes before it actually pings you that [30:54] the PR is done. [30:56] Um the link for this is [30:58] builders.ramp.com. [31:00] I think it's like one of the first uh [31:01] blog post that we have uh or the most [31:03] recent blog post that we have, and we [31:05] open source like the whole blueprint of [31:06] how to build this and put this together [31:07] as well. I think there's also a GitHub [31:09] repo called open inspect, which is an [31:11] open source implementation of this as [31:13] well. [31:17] So, it's been pretty interesting to see [31:19] the impact that Ramp Inspect has had. [31:20] We're over 50% of PRs that we merge on a [31:23] weekly basis goes through the system. [31:25] And so, with all this time not spent on [31:28] thinking about these really low-level [31:29] firefighting tasks or really low-level [31:32] small fixes or tweaks that can be kind [31:34] of democratized across the company, [31:36] we're really rethinking like how our [31:37] engineering teams operate and think [31:39] about their job and how they can [31:41] actually be really impactful in this new [31:44] kind of AI-native future. [31:47] And so, as a thought experiment, um we [31:49] let's pretend we have two different [31:50] teams. I'm sure everyone in this room [31:52] has worked with like their handful of [31:54] extraordinary teams, maybe teams that [31:55] are finding their footing. [31:57] And you'll notice that there's like a [31:58] couple of different qualities that may [32:00] sound that may resonate. [32:02] So, we have team A on the left here. And [32:04] let's say that they really care about [32:05] impact, they handle ambiguous problems, [32:07] they understand the product, business, [32:08] and data, they adopt new tools, they can [32:11] find creative solutions, and they obsess [32:13] over like the user experience. [32:15] And then team B may also resonate with [32:17] some people. You know, they debate [32:19] libraries, they add process when they [32:21] things start to feel chaotic. They [32:22] constantly complain about head count. [32:25] They bike shed the details instead of [32:26] actually focusing on the user [32:28] experience. Like, hey, should we use you [32:30] know, functional programming paradigm [32:31] here or what version of, you know, [32:33] different TypeScript libraries do we [32:34] want to use? [32:36] And then they build before understanding [32:37] the [32:38] right? They just say, "Hey, we're going [32:39] to just five code this, bro. Don't [32:40] worry." Or they focus on, you know, [32:42] performative code quality or nitpicks [32:44] that may not actually They may be very [32:46] much like a subjective kind of matter of [32:48] fact, as well. I've worked on both of [32:50] these teams, and I think the argument [32:52] that I'm going to make today is that [32:54] there's going to be a divergence, I [32:55] think, depending on what side of the [32:56] aisle you land there. [32:58] This is a study from Harvard that was [32:59] out, uh, I think the end of last year. [33:02] And it was very much geared towards [33:03] juniors and and seniors in terms of [33:05] what's actually happening with hiring [33:06] trends in uh in engineering since AI [33:09] tools have accelerated. And I think what [33:11] this glosses over is that I don't think [33:12] it's just a years of experience problem. [33:14] I actually think it's very much, um, all [33:17] of the different qualities that I said [33:18] in team A versus team B that really make [33:21] it apparent that like coding was never [33:23] really the hardest part of a lot of jobs [33:25] for a lot for a long time. There's all [33:27] these other engineering principles that [33:29] become really important than just raw [33:31] coding speed. [33:32] So, when you think about like a staff or [33:34] a staff plus engineer, you're really [33:36] compensating those people more for a lot [33:39] of the judgment that they bring to the [33:40] table, the context, the ability to see [33:42] around corners, all the learning that [33:43] they have, the actual like scar tissue. [33:46] And so, if, you know, you ask Opus 46 to [33:48] do something, they'll have the knowledge [33:50] to actually know if that is not going to [33:52] work or that's actually a bad idea. And [33:54] I think one thing that a lot of the [33:56] narratives that we see in the media gets [33:57] get wrong about coding agents is they [33:59] don't really identify the fact that you [34:01] could still build the wrong thing just a [34:03] lot faster, and you can build like [34:04] bigger messes. And I think that having a [34:07] lot of these skills of a team A and [34:09] really focusing on like what is the [34:10] context and reason behind this will only [34:12] become more important, um, in AI. [34:16] And so, what does that actually look [34:18] like? We hit on some of these things. [34:20] Figuring out what to build and [34:22] understanding users well enough. [34:24] Selling an idea to skeptical [34:25] stakeholders. This is still something [34:27] when we decided to build a a background [34:29] coding agent, this was not something [34:30] that was obvious that we should be [34:31] spending time on this. [34:33] Having good design design decisions with [34:35] incomplete information and maintaining [34:38] momentum through the long middle of this [34:40] project, which can be really gnarly. And [34:42] I think this last [34:43] bit, you know, everyone in this room I'm [34:44] sure is painfully aware of you know, the [34:47] conversation around SaaS and and the [34:49] stock market and things like that. And I [34:51] think this is like a big element that [34:52] they gloss over, which is that yes, it's [34:54] easy to vibe code something, but [34:56] actually going through that middle [34:57] process is like why you need really good [34:59] engineers to actually get something [35:00] deployed that has product market fit, [35:03] that people are really excited about. Um [35:05] and I think not enough people recognize [35:07] that. [35:10] And so where does that leave us? [35:11] Personally, I think there's a lot of [35:13] kind of doomerism and and scariness [35:14] around a lot of the AI narratives, but I [35:17] think it's also a really exciting time [35:19] to be building. [35:20] Unlike maybe factory work or farming, [35:23] software's never done. We have this uh [35:26] really [35:27] kind of like meme internally where we [35:29] say, you know, jobs not finished. You've [35:30] probably seen in the marketing as well. [35:32] I think software's perpetually not [35:34] finished. And so with all this extra [35:36] capacity, with people focusing less on [35:38] this kind of low-level work and more on [35:40] high-leverage engineering tasks, I think [35:42] four things are going to really happen. [35:45] I think companies are just going to [35:46] chase opportunities they couldn't afford [35:47] to pursue. I don't know if we would be [35:49] chasing these like agentic workflows and [35:52] really thinking about bigger scale [35:54] problems in the financial stack if this [35:56] technology didn't exist. [35:58] People are going to enter adjacent [35:59] markets. They're going to try to stitch [36:00] together more value for customers. It's [36:02] not going to be like because everyone's [36:04] 2x more productive, you need two less or [36:06] half the people. [36:08] You're going to rebuild systems that are [36:09] too expensive to touch. I think building [36:11] an internal background coding agent uh [36:14] for a company that does financial [36:15] operations um software felt like [36:17] probably a pretty crazy idea, but now [36:19] that makes a ton of sense. [36:21] And raise the bar for what good enough [36:22] means. I think, you know, being able to [36:24] kind of build more mind-blowing [36:26] experiences for users, provide a lot [36:28] more value is going to be the narrative [36:30] of the next decade. And I'm super [36:32] excited to be able to build some of [36:34] these things and see what everyone in [36:35] this room is going to build, too. So, [36:37] thank you.