The Pragmatic Engineer
AI productivity gains will expand engineering ambition into previously unviable products and system overhauls rather than reduce developer headcount.
AI-assisted summary of Ramp: Lessons from Building a New AI Product - The Pragmatic Summit — original source. Check the original for full context.
Source published:
Key ideas
- Ramp shifted its core AI architecture from building dozens of specialized, disconnected agents to a single unified agent framework backed by a catalog of hundreds of deterministic skills and tools. (3:31)
- Early production dogfooding revealed that agent errors stemmed almost entirely from missing domain context (such as HRIS employee titles) rather than model deficiencies, while human approvals proved too inconsistent to serve as ground truth. (10:40)
- Ramp Inspect, an internal background coding agent running in isolated Modal sandboxes with access to over 150,000 automated tests, now authors more than 50% of the company's weekly merged pull requests. (27:25)
- Ramp leadership disputes the narrative that AI will reduce engineering headcount, arguing that software demand is open-ended and productivity gains are instead funneled into tackling previously cost-prohibitive legacy rewrites and complex workflows. (35:54)
Detailed notes
## Background Published on YouTube by The Pragmatic Engineer on March 9, 2026, this session from The Pragmatic Summit features Ramp engineering leaders Nik Koblov, Veeral Patel, Will Koh, and Ian Tracey. According to the description, the panel shares concrete lessons from building and scaling applied AI products at Ramp, covering architectural pivots, policy evaluation, evaluation tooling, infrastructure, and engineering culture.
## TL;DR Ramp's engineering team details their journey building AI-native financial software, highlighting an architectural pivot away from fragmented micro-agents toward a unified foundational agent backed by an extensive catalog of deterministic tools. Using their automated Policy Agent as a primary case study, they illustrate how natural language acts as a programming interface, why context gaps—not model limitations—cause most agent failures, and why relying on historical user actions as ground truth fails without strict, continuous evaluations. The speakers also explain their centralized Applied AI infrastructure for model routing, rate-limiting, and cost tracing, alongside an internal autonomous coding agent called Ramp Inspect that now drives over 50% of merged production pull requests. Ultimately, they argue that AI tools will not eliminate developer roles, but will instead widen the performance gap between process-obsessed coders and context-driven engineers while dramatically raising ambitions for software scope.
## Atmosphere & style The presentation carries the brisk, candid energy of an engineering post-mortem stripped of conventional AI marketing hype and focused strictly on production reality. Moving seamlessly between four technical leaders, the delivery is conversational, lucid, and grounded in relatable operational scars—from parsing blurry restaurant checks to admitting how often approving managers make mistakes on lazy Sunday afternoons. It feels refreshingly pragmatic and technically rigorous, balancing honest confessions about the messiness of autonomous workflows with an optimistic, rallying defense of software engineering's future and the irreplaceable value of human judgment.
## Bulletin - [03:31](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=211s) Ramp shifted its core AI architecture from building dozens of specialized, disconnected agents to a single unified agent framework backed by a catalog of hundreds of deterministic skills and tools. - [10:40](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=640s) Early production dogfooding revealed that agent errors stemmed almost entirely from missing domain context (such as HRIS employee titles) rather than model deficiencies, while human approvals proved too inconsistent to serve as ground truth. - [27:25](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=1645s) Ramp Inspect, an internal background coding agent running in isolated Modal sandboxes with access to over 150,000 automated tests, now authors more than 50% of the company's weekly merged pull requests. - [35:54](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=2154s) Ramp leadership disputes the narrative that AI will reduce engineering headcount, arguing that software demand is open-ended and productivity gains are instead funneled into tackling previously cost-prohibitive legacy rewrites and complex workflows.
## Key takeaways - [03:31](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=211s) Product architectures should converge on a single foundational agent equipped with an extensive tool catalog rather than dozens of siloed, task-specific bots. - [05:11](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=311s) OmniChat unifies conversational entry points across the application and compiles plain-English instructions into deterministic, runnable playbooks. - [07:17](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=437s) Treating natural-language policy documents as the product's codebase eliminates the need for brittle, hardcoded deterministic logic trees. - [10:40](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=640s) AI systems cannot be built in one shot; starting with an ultra-constrained problem like a single coffee receipt reveals critical missing context much faster than theoretical planning. - [12:32](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=752s) Transitioning from sequential chains to agentic loops yields greater autonomy at the expense of inspectability, turning smaller black boxes into much larger ones. - [15:29](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=929s) Historical human approval actions are an unreliable ground truth because managers frequently approve out-of-policy expenses due to fatigue or lack of policy knowledge. - [15:29](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=929s) Teams should avoid perfectionism by starting evaluation suites with as few as five critical test cases and building lightweight, rapidly modifiable internal labeling tools. - [19:15](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=1155s) Running offline evals in CI and monitoring online leading indicators—such as the frequency of 'unsure' agent outputs—catches unexpected regressions when updating prompts or tools. - [20:46](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=1246s) Autonomous enterprise adoption requires incremental trust-building, moving from advisory suggestions to configurable autonomy sliders based on transaction risk thresholds. - [22:46](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=1366s) A centralized Applied AI Service protects downstream product teams from breaking API changes while managing rate limits, heavy batching, and cost tracing across model providers. - [25:52](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=1552s) Preventing hallucinations in sensitive financial workflows depends on restricting models to a curated catalog of deterministic internal business tools. - [27:25](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=1645s) Background coding agents like Ramp Inspect resolve context fragmentation by pulling logs, database read replicas, alerts, and Slack threads into isolated developer sandboxes. - [29:48](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=1788s) Making coding agents multiplayer-first and triggering them directly from Slack allows non-engineers to run bug investigations and iterate on code alongside developers. - [31:17](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=1877s) AI coding agents widen the performance gap between outcome-driven engineers who navigate ambiguity and process-heavy engineers who fixate on syntax and bureaucracy. - [34:16](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=2056s) The hardest part of engineering remains navigating the 'long middle' from prototype to production and applying judgment under information scarcity, not generating syntax. - [35:54](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=2154s) Unlike physical manufacturing, software development faces infinite demand, meaning agent-driven leverage raises the standard of product ambition rather than capping developer employment.
## Detailed walkthrough ### [00:05](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=5s) Ramp's Core Mission and the Simplest Expense Use Case
Nik Koblov opens the session by outlining Ramp's role as the leading finance platform for modern businesses, serving over 50,000 customers with a simple value proposition: saving companies time and money. Glancing out at the audience, he jokes about spotting attendee name tags from existing customer organizations and notes that everyone in the room is clutching coffee. That simple cup of coffee serves as the baseline for demonstrating how legacy expense management breaks down. Traditionally, expensing a single coffee takes roughly fifteen minutes of manual effort spread across cardholders and finance staff—capturing receipts, typing out memos, and assigning general ledger (GL) codes. Multiplied across an entire organization, this friction compounds into an immense administrative sink.
Ramp tackled this problem by collapsing those manual steps into an automated, agentic flow that runs the moment an employee swipes or taps their card. About three years ago, Ramp began introducing one-shot AI automations to perform tasks like normalizing merchant names, drafting expense memos, categorizing transactions against accounting codes, and sourcing or matching receipts. Beyond basic card swipes, nearly every corporate persona—from accounts payable clerks and procurement managers to finance directors and data teams—remains bogged down in repetitive manual requests. Koblov recalls that Ramp previously maintained an internal Slack channel called `#help-data`, where colleagues would ask for raw data and an engineer or analyst would manually write a SQL query to export a CSV. Ramp decommissioned that manual channel about a year and a half ago by replacing it with automation.
- **Customer reach**: 50,000+ business customers using Ramp's finance platform. - **Legacy friction**: A single coffee expense typically consumes 15 minutes across various manual bookkeeping tasks. - **Automation history**: Started three years ago with one-shot tasks (merchant normalization, receipt matching, memo drafting), expanding outward to eliminate manual workflows like data-pull requests.
---
[03:31](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=211s) Paradigm Shift: Moving from a Zillion Agents to a Single Agent with Many Skills
Reflecting on recent product evolution—especially following rapid shifts in early February—Koblov highlights a major architectural lesson learned while attempting to automate every internal and external role. Originally, Ramp leaned into building specialized agents for every distinct job to be done (including "Brian," an agent featured in their Super Bowl advertisement). Last year, Ramp intentionally let individual product teams experiment autonomously. This decentralized approach produced roughly four different, siloed implementations for both synchronous interactions and background jobs. Koblov argues that this proliferation of disconnected bots was the wrong long-term trajectory.
Instead, Ramp discovered that modern AI applications must consolidate toward a single foundational agent endowed with a massive catalog of specialized tools and skills. Koblov contrasts how legacy software operates with the agentic paradigm. Traditional software typically confines itself to data context and API tools (actions 4 and 5 of a standard execution loop). In the new paradigm, software takes responsibility for the entire chain: ingesting events (like an incoming invoice), reading prompt instructions, enforcing organizational guardrails and policies, gathering surrounding data, and executing the appropriate tool calls autonomously.
As Nik Koblov explains the architectural shift:
> “So, what we learned is you don't need to build a thousand agents.”
He emphasizes the imperative to streamline the product architecture into a unified execution engine:
> “And with rethink, simplification of your stack. Uh So, what we learned is you don't need to build a thousand agents. We intentionally last year allowed each individual team to go and experiment. And we ended up maybe with four different ways of doing the same thing both for synchronous agents as well as for background agents. Um but instead you want to drive your framework towards a single agent with a thousand skills.”
```mermaid flowchart TD accTitle: Legacy Software vs Autonomous Agent Loop accDescr: Diagram contrasting traditional software tool calls with the complete five-stage autonomous system of action.
subgraph Legacy["Traditional Software Scope"] D["4. Data Context"] E["5. APIs & Action Tools"] end
subgraph Modern["Modern Autonomous System"] A["1. Event / Trigger (e.g., Invoice, Card Swipe)"] B["2. Prompt Instructions"] C["3. Policies & Guardrails"] end
A --> B B --> C C --> D D --> E ```
---
[05:11](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=311s) OmniChat, Lightweight Orchestration, and Runnable Playbooks
To unify how users interact with agents, Ramp consolidated five separate conversational prototypes developed over the past year into a single, standardized interface dubbed "OmniChat" (short for omnipresent chat). OmniChat sits across every surface of Ramp's web application, working alongside traditional UI elements like data tables and action buttons rather than replacing them entirely. Koblov illustrates this with an employee onboarding scenario: when an admin types "Please onboard a new employee," OmniChat resolves the individual to their corporate employee ID, queries the HRIS system to determine their reporting hierarchy, identifies an existing agentic workflow called the "New Hire Playbook," and prompts the administrator for confirmation to run it.
Powering this is an in-house, lightweight agent orchestration framework designed for rapid tool creation. The abstraction layer is lightweight enough that a product manager recently "vibe-coded" around 20 functional tools without needing dedicated engineering assistance. For complex multi-step processes, admins can write plain-English descriptions of the intended lifecycle—such as issuing a corporate card, enforcing receipt upload requirements on every transaction, sending a welcome message on Slack, and scheduling an automated check-in after two weeks. Ramp compiles these human-readable instructions into a deterministic, runnable workflow that the core agent executes reliably.
When a transaction occurs, the transaction flows seamlessly between agents: the Policy Agent verifies corporate spend limits in real time, passes the context to an accounting agent that maps the expense against the chart of accounts and ERP records, and either auto-approves the charge or flags it for human review if it breaches materiality thresholds or policy rules.
```mermaid flowchart LR accTitle: Real-Time Card Swipe Processing Flow accDescr: Step-by-step transaction handling from card swipe to policy check, accounting classification, and final approval or review.
Swipe["Employee Card Swipe"] --> Policy["Policy Agent Review\n(Real-Time Checks)"] Policy --> Acct["Accounting Agent\n(ERP & GL Mapping)"] Acct --> Decision{"Compliant Spend?"} Decision -->|"Yes"| Approve["Auto-Approve Transaction"] Decision -->|"No / High Value"| Escalate["Human-in-the-Loop Review"] ```
---
[07:17](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=437s) Inside the Policy Agent: English as the New Programming Language
Veeral Patel takes the stage to unpack the technical design and operational philosophy behind the Policy Agent, one of Ramp's most heavily utilized products. He displays a blurry restaurant receipt that finance teams typically squint at hundreds of times a day. Rather than forcing a tired controller to manually parse the check, the Policy Agent analyzes both the raw image and the transaction metadata simultaneously. In this specific receipt, the agent detected eight dining guests—a detail Patel admits he could barely make out himself—confirmed that the spend fell under Ramp's internal $80-per-person dinner cap, verified that the stated business purpose was a team welcome dinner, validated the merchant and dollar figures, and recommended instant approval.
Conversely, Patel highlights instances where the agent caught out-of-policy spending: it approved an OpenAI charge incurred by an employee testing ChatGPT features as a valid business expense, but rejected a small $3 bakery charge because it was not incurred during eligible weekend work or overtime hours. When a Fortune 500 customer approached Ramp requesting custom approval and rejection logic across complex enterprise rules, Ramp avoided the trap of hardcoding brittle, deterministic logic trees into the platform.
Referencing AI researcher Andrej Karpathy's observation, Patel explains how Ramp shifted its core philosophy:
> “And we kind of saw this as an opportunity not to kind of add more incremental deterministic rules that kind of define our product. And I worked on some of the first versions of these, um but actually kind of take out uh a page from Andrej Karpathy saying that English is the new programming language and kind of turn the expense policy into the rules themselves.”
To build and tune this agentic policy evaluator, the engineering team operated like an agile startup within Ramp. They partnered closely with enterprise design partners—including the Fortune 500 company—holding weekly feedback sessions to inspect edge cases, understand nuanced accounting requirements, and refine how the model interprets natural-language policy documents against real-world spend.
[10:40](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=640s) Internal Dogfooding and the Reality of AI Context
Veeral Patel reflects on a fundamental cultural realization across Ramp: AI products simply cannot be built in a single stroke. Teams must resist the urge to chase perfection on day one and instead start with an extremely constrained problem. To put this into practice, Ramp leaned heavily on internal dogfooding. Rather than trying to automate corporate spend across the board, they scoped down to determining whether a simple “coffee with a colleague” transaction ought to be approved or rejected. These single-dollar-amount expenses were categorized by Ramp's internal finance team as low-risk, making them the ideal sandbox for early experimentation.
```mermaid flowchart TD accTitle: Iterative Context Enrichment for Policy Agent accDescr: Flow showing how internal dogfooding revealed context gaps rather than model failures. subgraph Dogfooding["Internal Dogfooding"] A["Low-Risk Expenses<br/>('Coffee with a Colleague')"] B["Policy Agent Evaluates Expense"] end subgraph GapAnalysis["Root Cause Discovery"] C{"Error Analysis"} D["Model Capability Defect<br/>(Rare)"] E["Missing Domain Context<br/>(Primary Cause)"] end subgraph ContextExpansion["Context Enrichment"] F["HRIS Employee Metadata<br/>(Titles, Roles, Org Levels)"] G["Granular Receipt Data<br/>(Itemized Expenses, Flight Class)"] end
A --> B B --> C C -->|Minority| D C -->|Majority| E E --> F E --> G ```
Once this test ran against production data, the team encountered an essential early learning: when Policy Agent produced the wrong outcome, the fault rarely lay with the underlying LLM. Instead, the failure stemmed from missing context provided to the model. While the team could have spent weeks trying to brainstorm every potential piece of context prior to writing code, learning from live internal data proved vastly superior. For instance, they noticed that an employee’s role and organizational title are critical when interpreting expense policy documents. An executive in the C-suite might be allowed to book first-class flights or enjoy elevated dining limits that junior employees cannot. As a consequence, engineering began extracting richer details from receipts and piping in HRIS metadata already resident on Ramp's platform.
- Veeral Patel: “I think one of the main important I guess things that we realized across Ramp is that we really needed to lean into the fact that AI products cannot be one-shotted.”
[12:32](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=752s) The Evolution from Simple Chains to Agentic Workflows
Staff Engineer Will Koh describes the initial impulse when setting out to build Policy Agent internally: going huge and dreaming of automating all of finance reviews in one go. However, reality demanded starting with that solitary question: is this specific cup of coffee compliant with the expense policy? Even though the problem appeared elementary on the surface, expanding and generalizing it to diverse businesses inevitably compounds complexity. Will emphasizes that simpler systems are inherently easier to iterate on, allowing teams to determine what works before layering on advanced mechanics.
```mermaid flowchart LR accTitle: Three Phases of Policy Agent Architecture accDescr: Evolution from sequential LLM calls to conditional tool use and finally full agentic loops. subgraph V1["Phase 1: Deterministic Chain"] A1["Incoming Expense"] --> B1["Context Retrieval"] B1 --> C1["Sequential LLM Pipeline"] C1 --> D1["Rule-Based Output"] end subgraph V2["Phase 2: Conditional & Tools"] A2["Classify Category<br/>(Meal, Travel, Ent.)"] --> B2["Conditional Prompting"] B2 --> C2["Autonomous Lookup Tools<br/>(Flight Info, Seniority)"] end subgraph V3["Phase 3: Agentic Loop"] A3["Shared Company Toolbox"] <--> B3["Autonomous Agent Loop<br/>(Reasoning & Policy Checks)"] B3 --> C3["Write Capabilities<br/>(Auto-Approve, Decision Logs)"] end
V1 --> V2 V2 --> V3 ```
Ramp's architecture progressed through three distinct phases: 1. **Deterministic Sequential Chains:** An expense arrived, surrounding context was retrieved, and the data was routed through a tightly defined series of sequential LLM calls answering specific questions (Is it in policy? Why is it in policy? How do we present this rationale to the user?). 2. **Conditional Categorization & Targeted Tools:** Recognizing that expenses differ fundamentally, the team added a classification step (travel, meal, or entertainment) followed by conditional prompting. They introduced targeted tools so the model could independently retrieve external data, such as flight itineraries or employee seniority levels. 3. **Full Agentic Workflows:** The system transformed into an agentic loop equipped with shared company-wide toolboxes that can read across the entire Ramp platform. The agent gained write permissions, allowing it to log decisions, generate formal audit rationales, and autonomously approve expenses on behalf of human reviewers.
This architectural shift brought a fundamental engineering trade-off between power and inspectability. As autonomy and intelligence increased, traceability decreased. While engineers can review reasoning tokens produced by the LLM, the inner workings become a much larger black box that executes tool calls and arrives at conclusions on its own terms.
- Will Koh: “A smaller black box becomes a bigger black box as the system becomes more complex.”
[15:29](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=929s) Ground Truth, Disillusionment with User Actions, and Custom Evals
Building on the reality of this expanding black box, Will Koh highlights that teams must implement rock-solid auditability from day one. Systems must be architected so that even if one only inspects inputs and outputs, correctness remains verifiable. Initially, the team assumed that human user behavior could serve as the gold standard of truth—if an approving manager marked an expense as approved, the agent should do the same. In practice, users proved to be unreliable ground truth. Managers frequently approve expenses without understanding the policy, trust their employees blindly, act out of fatigue, or clear their queues on a Sunday afternoon without scrutiny. When finance teams subsequently flagged these approved transactions as unacceptable card charges, the team realized they could not blindly imitate user actions and had to create their own strict definition of correctness.
```mermaid flowchart TD accTitle: Weekly Labeling and Evaluation Pipeline accDescr: Cross-functional alignment loop powering dataset curation and lightweight evals. subgraph Alignment["Weekly Human-in-the-Loop"] A["Cross-Functional Team<br/>(Eng, PM, Design)"] --> B["Weekly Labeling Sessions"] B --> C["Curated Ground Truth Dataset"] end subgraph Tooling["Internal Streamlit Workspace"] D["Claude Opus-Generated UI<br/>(Clockwork / Streamlit)"] --> B E["Rapid Vibe/Clock Coding<br/>by Non-Engineers"] --> D end subgraph Verification["Continuous Evals"] C --> F["Targeted Eval Runs<br/>(Started with 5 test cases)"] F --> G["Instant, Interpretable Pass/Fail"] G --> H["Fast Model & Context Iteration"] end ```
To establish this baseline, Ramp organized weekly cross-functional labeling sessions with all stakeholders on the product. This practice yielded two major benefits: it created a hardened, verified ground truth dataset for testing, and it ensured seamless alignment across teams whenever the agent missed context or erred. While evaluating third-party annotation vendors, Ramp found commercial tools either overly rigid or too broad. Instead of wasting weeks evaluating software, the engineers used Claude Opus to vibe-code a custom Streamlit data-labeling interface in an internal workspace named Clockwork. Because the UI lived in an isolated codebase, non-engineers could easily modify it on the fly, with changes deploying in seconds. Armed with this dataset, the team ran evaluations continuously, purposefully starting small with just five mission-critical test cases rather than letting perfectionism stall their momentum.
- Will Koh: “Turns out the users are actually incorrect. They're wrong. They are sometimes, you know, they don't know the expense policy, you know, they trust their employees, they're lazy, it's a Sunday, who knows.” - Will Koh: “I wouldn't say that, you know, don't let perfectionism, you know, get in the way. You don't need a full data set of a thousand data points that you're testing against every iteration. We started with five.”
[19:15](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=1155s) Evals in CI, Online Health Checks, and Seamless Model Upgrades
Building agents requires continuous regression testing, because every time engineers attempt to improve an agent's reasoning—such as injecting additional prompt context, adding new tools, or tweaking instructions—unforeseen side effects frequently emerge. A seemingly helpful docstring update or supplementary guideline can easily conflict with existing prompts or mislead the model. Integrating offline evaluation datasets into continuous integration (CI) ensures that unintended regressions are blocked before code is merged. Alongside offline evaluations, online evaluations serve as critical leading indicators of system health. For Ramp's Policy Agent, tracking the rate of "unsure" decisions—instances where the agent lacked sufficient context to render an expense judgment—provides a straightforward, real-time pulse on production performance.
Moreover, having a robust evaluation benchmark in place decouples system logic from underlying model vendors. Whether Anthropic or OpenAI ships a major new release, teams can evaluate the delta cleanly without guessing. As the speaker explains:
> “Whenever you think you're doing something right for the LLMs or agent, giving more context, giving it tools, more likely than not it's probably going to have some kind of bad, you know, consequence that you didn't see happening.”
- **CI Regression Gates**: Validates prompt revisions, tool signatures, and docstring clarity on historical offline test sets before code hits production. - **Online Leading Metrics**: Monitoring telemetry such as the volume of "unsure" outputs flags runtime anomalies or missing context immediately. - **Confident Model Swaps**: Benchmarking suites allow teams to rapidly adopt next-generation foundation models or catch regressions caused by behavioral shifts in updated model weights.
---
[20:46](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=1246s) The Finance "CLAUDE.md" and Earning Autonomous Trust
Engineers are comfortable controlling generative code tools through context files like `CLAUDE.md`, specifying exact rules such as suppressing inline comments. Ramp discovered an unexpected parallel on the finance side: finance operators care deeply about their company expense policy documents, which effectively act as their own prompt configuration files. Initially, finance teams were nervous about modifying policy documents because corporate compliance policies typically require navigating substantial bureaucratic red tape. However, once Ramp provided immediate feedback loops showing how policy edits instantly altered agent evaluation decisions, finance administrators embraced the process enthusiastically.
Deploying the Policy Agent into large enterprise organizations required deliberate trust-building. Ramp deliberately started with Fortune 500 enterprise customers because their transaction volume made reviewing hundreds of small expenses like coffees a massive operational drain. Rather than taking autonomous actions out of the gate, the system initially provided only non-binding "suggestions." Once administrators saw consistently accurate recommendations, they proactively requested full automation for low-risk spend. In response, Ramp introduced an autonomy slider, empowering customers to configure automatic approvals for transactions under specific thresholds (such as $200) while keeping human oversight on the rest.
> “Don't We didn't do any autonomous action. We're just like, 'Hey, we're going to give you a suggestion.' That's That's how That's kind of how we phrased it. Suggestions. And then eventually they came to us and were like, 'Okay, you know what? I want to go from suggestions to auto approvals. Like anything under $200, you guys are mostly right. I don't care about this. Let me just go auto approve it.'”
- **In-Product Feedback Loops**: Giving users direct mechanisms to refine policy texts allows non-technical operators to iteratively improve agent accuracy on their own. - **Enterprise Adoption Strategy**: Started with high-volume enterprise spenders where small-ticket reviews consumed disproportionate human hours. - **The Autonomy Slider**: Shifted governance from advisory suggestions to automated approvals on a configurable dollar-threshold basis once user confidence matured.
---
[22:46](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=1366s) AI Infrastructure as a Service: Unifying SDKs, Batching, and Cost Tracing
Ian Tracey introduces the operational foundation supporting Ramp's applied AI products, framing the effort as both a challenging infrastructure initiative and an internal cultural transformation designed to provide leverage for engineers and cross-functional partners alike. At the core of this strategy is Ramp's internal Applied AI Service. While conceptually resembling an LLM proxy or routing layer such as LiteLLM, the platform layers on three crucial internal capabilities:
```mermaid flowchart TD accTitle: Ramp Applied AI Service Architecture accDescr: Shows the Applied AI Service abstracting model providers, batch processing, and cost tracing for downstream teams.
subgraph Clients["Internal Product Teams"] P1["Policy Agent"] P2["Reimbursements"] P3["Internal Tooling"] end
subgraph CoreService["Ramp Applied AI Service"] A["Unified SDK & Structured Output"] B["Batch Processing & Rate-Limit Engine"] C["Cost Tracing & Pareto Analysis"] end
subgraph Providers["Model Layer (1-Line Config Swaps)"] M1["OpenAI (GPT)"] M2["Anthropic (Claude)"] M3["Google (Gemini)"] end
Clients --> CoreService A --> Providers B --> Providers C --> Providers ```
The service offers a uniform SDK that guarantees structured outputs across constantly shifting provider APIs, shielding feature teams from provider-specific breaking changes. Secondly, it manages heavy batch processing, workflow orchestration, and rate-limit backpressure for bulk document extraction and asynchronous evaluations across providers like Anthropic. Finally, the service captures fine-grained cost tracing attributed across teams and products, identifying cost-performance Pareto frontiers to ensure new AI services remain economically sustainable. Upgrading downstream systems to frontier models requires only a single centralized configuration change rather than updating dozens of disparate call sites across the codebase.
Ian emphasizes the dual nature of scaling applied AI inside the organization:
> “This section is pretty intentionally named AI infrastructure and culture cuz we think that this is both like a really challenging infrastructure problem, but it's also a really challenging culture problem and changing how you work as well is a big part of the story.”
- **Centralized Abstraction**: Downstream product engineers swap underlying foundation models via single-line configuration updates without rewriting application-level calls. - **Managed Rate Limits and Batching**: Handles offline job queuing and provider quotas centrally for document parsing and eval suites. - **Cost and Unit Economics**: Granular cost-allocation dashboards allow Ramp to identify unsustainably expensive workflows early and optimize model selection per use case.
---
[25:52](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=1552s) Curing Hallucinations with an Internal Tool Catalog
Handling sensitive financial workflows and enterprise transaction data makes safety and hallucination prevention top engineering priorities. Ramp's primary mechanism for keeping models grounded is an expansive internal tool catalog built in collaboration with domain product teams. Instead of letting agents generate outputs unconstrained, agents execute discrete, deterministic operations against curated business logic—such as fetching specific policy snippets, determining per-diem rates, or querying recent transaction histories.
This shared catalog provides immediate visibility into capability gaps while serving both core production flows and rapid internal prototypes. An engineer experimenting with a new reimbursement agent can assemble existing production tools without learning underlying backends or rebuilding integrations from scratch. Ramp's catalog already spans hundreds of standardized tools, with long-term expectations of expanding into multiple thousands as agent capabilities scale across the broader financial infrastructure.
As Ian notes regarding the balance between safety and agent power:
> “And I think often times, uh you know, something that I hear from engineers in the space is this kind of concept of hallucination and safety, and how are you actually going to be able to produce a lot of these things to have benefits to downstream finance teams? And we're pretty big believers that it all comes down to the catalog of tools that teams are building and integrating with on a daily basis.”
- **Shared Building Blocks**: Curates pre-built, domain-specific utilities (e.g., policy queries, per-diem calculations, transaction lookups) accessible across all microservices. - **Prototyping Velocity**: Developers can stand up "vibe-coded" prototypes using existing, vetted financial data connectors without engineering bespoke APIs. - **Scale Trajectory**: The catalog has grown to several hundred tools and is targeted to reach thousands of fine-grained capabilities.
[27:25](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=1645s) Ramp Inspect: Context Fragmentation and the Background Coding Agent
Even when developers leverage modern developer tools like Claude Code or Codex, an enormous amount of daily engineering context remains completely fragmented and unintegrated. A developer's workflow requires pulling logs from Datadog, checking production database read replicas, navigating alerting pipelines, parsing alerts from incident.io, retrieving context from scattered Slack threads, referencing Notion documentation, and tapping into the tacit domain knowledge held by specific product teams. At the end of last year, Ramp tackled this context fragmentation directly by building their own internal background coding agent, named Ramp Inspect (which Ramp has documented and open-sourced in blueprint form). Rather than functioning as a synchronous inline autocomplete tool, Ramp Inspect runs autonomously in the background while engineers are sitting in meetings or when bug reports and incident alerts arise.
The operational adoption across Ramp has been massive: Ramp Inspect is currently responsible for over 50% of the pull requests merged into production on a weekly basis. Ramp tracks adoption metrics across the company on an internal leaderboard to foster subtle, healthy competition and inspire broader team participation. While engineering accounts for the largest share of sessions, non-engineering departments—including product management, design, risk, legal, corporate finance, marketing, and customer experience (CX)—actively initiate sessions. These teams use Inspect to execute simple copy adjustments, modify business logic, investigate bugs, and respond to live incidents without waiting on engineering bottlenecks.
```mermaid flowchart TD accTitle: Ramp Inspect Architecture and Context Ingestion accDescr: Diagram illustrating how Ramp Inspect aggregates multiple context sources into Modal sandboxes to generate GitHub PRs.
subgraph Inputs["Trigger Channels"] A["Kanban UI"] B["API"] C["Slack Thread (Full History)"] end
subgraph Context["Integrated Context Layer"] D["Datadog Logs"] E["DB Read Replicas"] F["incident.io & Alerts"] G["Notion Docs & Team Context"] end
subgraph Execution["Modal Sandbox Environment"] H["Isolated Dev Sandbox"] I["Task Tracking Engine"] J["VS Code via VNC"] K["Chrome DevTools & MCP"] L["150k+ Test Suite & CI"] end
Inputs --> H Context --> H H --> I I --> J J --> K K --> L L --> M["GitHub Branch & Verified PR"] ```
- Ramp Inspect accounts for more than 50% of all pull requests merged to production each week. - Sessions run in isolated Modal sandboxes that mirror Ramp's production development environment and can spin up, pause, resume, and tear down rapidly. - Sandboxes include a full VS Code environment streamed over VNC with Chrome DevTools and Model Context Protocol (MCP) integrations for full-stack engineering. - The agent has direct access to Ramp’s test suite of over 150,000 automated tests; it monitors CI outcomes in GitHub, interprets test failures, and iteratively patches broken code before pinging the engineer that the PR is ready for review. - The blueprint of the system architecture is published at `builders.ramp.com`, and an open-source implementation is maintained in the `open-inspect` GitHub repository.
---
[29:48](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=1788s) Multiplayer-First Design and Collaborative AI Tooling
A crucial design philosophy behind Ramp Inspect is its multiplayer-first architecture. Rather than isolating the coding agent inside an individual engineer's local IDE, sessions can be shared directly with cross-functional partners like product managers and designers. When pairing with a designer or PM, an engineer can watch their prompts execute in real time, help them refine their prompting strategies, or debug when an interaction fails unexpectedly. This collaborative feedback loop transformed the agent from a solitary developer tool into an active driver of cross-functional team alignment.
Inspect sessions can be kicked off through multiple interfaces: a Kanban board web UI, an internal programmatic API, or directly from an ongoing Slack conversation. When triggered inside Slack, the system automatically pulls the full conversational context of the preceding thread, eliminating the need for users to manually copy-paste background details, reproduction steps, or user error descriptions into a fresh prompt window.
- Multiplayer sessions allow non-technical partners to share live links, flag unexpected edge-case behaviors, and quickly level up their prompting skills alongside engineers. - Triggering an agent run from Slack preserves all historical thread context automatically, avoiding redundant back-and-forth prompting.
---
[31:17](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=1877s) Engineering Divergence: Team A vs. Team B
With routine firefighting, copy updates, and minor bug fixes increasingly handled by background agents, Ramp's leadership has reevaluated what high-leverage software engineering looks like in an AI-native environment. The speaker introduces a comparative thought experiment contrasting two archetypes of engineering teams: "Team A" and "Team B." Team A focuses relentlessly on business impact, thrives in ambiguous problem spaces, understands product goals, data, and business drivers, adopts new tools quickly, invents creative workarounds, and obsesses over the end-user experience. Team B, conversely, falls back on debating library choices, introduces bureaucratic process whenever complexity increases, continually lobbies for headcount increases, bike-sheds subjective aesthetic debates (such as functional programming paradigms versus specific TypeScript versions), jumps to write code before understanding the fundamental problem, and fixates on performative code quality or subjective nitpicks.
Reflecting on working across both environments, the speaker argues that the rise of AI agents will dramatically widen the performance and career divergence between these two mindsets. While recent academic analyses—such as a late-2024 Harvard study on junior versus senior hiring trends—attribute shifting engineering employment patterns purely to years of experience, the speaker disputes that framing. Raw coding speed was never the most difficult aspect of engineering; instead, higher-level judgment, domain context, and hard-earned institutional experience are what distinguish truly impactful engineers from those who simply produce syntax.
> “And I think one thing that a lot of the narratives that we see in the media gets get wrong about coding agents is they don't really identify the fact that you could still build the wrong thing just a lot faster, and you can build like bigger messes.”
| Quality Dimension | Team A (Impact-Driven) | Team B (Process- & Tech-Driven) | |:--- |:--- |:--- | | **Primary Focus** | User experience, business outcomes, and impact | Library choices, syntactic paradigms, and headcount complaints | | **Handling Chaos** | Explores ambiguous spaces with creative solutions | Imposes administrative process and organizational overhead | | **Tool Adoption** | Eagerly experiments with and operationalizes AI agents | Relies on performative nitpicks and bike-shedding code reviews | | **Problem Approach**| Digs deep into product, business context, and data first | Starts "vibe coding" before grasping the underlying problem |
---
[34:16](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=2056s) The Long Middle, Infinite Demand, and Software's Future
When assessing senior and Staff-plus engineers, organizations pay primarily for their judgment, their foresight, and their scar tissue—the hard-won perspective needed to recognize when a proposed architecture from an advanced model like Claude 3.7 Sonnet or Claude Opus is fundamentally flawed or unworkable. In an environment saturated with AI hype and discourse around SaaS market valuations, public narratives often overlook how difficult it is to take software from a prototype to a durable production system. While "vibe coding" a proof-of-concept is trivial, the hardest parts of engineering remain distinctly human: uncovering what actually needs to be built, deeply understanding customer workflows, convincing skeptical stakeholders that an ambitious technical bet is worthwhile, making architectural decisions under severe information scarcity, and maintaining engineering momentum through the grinding, unglamorous "long middle" of a project.
Addressing broader industry anxiety and doomerism surrounding software engineering careers, the speaker rejects the idea that AI will cap the demand for developers. Unlike agriculture or manufacturing assembly lines—where output reaches clear physical saturation points—software development is fundamentally open-ended and never truly completed. Internally, Ramp embraces the working mantra "job's not finished." Freeing engineering capacity from low-level debugging and manual implementation tasks does not eliminate engineering jobs; instead, it enables organizations to aggressively pursue high-value, complex opportunities and agentic workflows that were previously deemed too expensive or resource-constrained to attempt.
> “Unlike maybe factory work or farming, software's never done. We have this really kind of like meme internally where we say, you know, jobs not finished.”
[35:54](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=2154s) Expanding Ambition and Raising the Bar for Software
The conversation concludes with a broader perspective on how AI shifts engineering strategy, far beyond simple labor reduction or headcount optimization. Rather than using productivity gains to cut teams in half, organizations are using the newfound capacity to tackle ambitious problems across the financial stack that previously seemed insurmountable. Teams are actively moving into adjacent markets, stitching together significantly more comprehensive value for customers, and fundamentally rethinking what kind of software they can justify building. As the speaker explains:
> “It’s not going to be like because everyone’s 2x more productive, you need two less or half the people.”
Instead of shrinking teams, companies will direct their engineering leverage toward rebuilding legacy or brittle systems that were previously deemed far too expensive or risky to touch. Projects that once seemed completely outside a company's core focus suddenly become viable investments. For instance, developing an internal background coding agent within a company focused primarily on financial operations software might have once seemed like a bizarre distraction, but in this new paradigm, it makes complete sense. Ultimately, the next decade of software development will be defined by an aggressive push to elevate what "good enough" looks like, enabling teams to deliver previously unimaginable user experiences.
- **Expanding footprint:** AI efficiency is channeled into entering adjacent markets and bundling deeper customer value rather than trimming headcount. - **Tackling legacy tech debt:** Systems previously considered too costly or complex to overhaul are being rewritten and modernized. - **Unconventional internal tools:** Building internal autonomous coding agents now represents a high-leverage initiative even for specialized fintech companies. - **A decade of higher standards:** The coming decade will center on drastically raising user expectations and shipping transformative product experiences across the industry.
## Notable quotes > [12:32](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=752s) A smaller black box becomes a bigger black box as the system becomes more complex. — Will Koh
> [34:16](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=2056s) Unlike maybe factory work or farming, software's never done.
> [15:29](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=929s) You don't need a full data set of a thousand data points that you're testing against every iteration. We started with five. — Will Koh
> [19:15](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=1155s) Whenever you think you're doing something right for the LLMs or agent, giving more context, giving it tools, more likely than not it's probably going to have some kind of bad, you know, consequence that you didn't see happening.
> [10:40](https://www.youtube.com/watch?v=NMs8C2_3M0w&t=640s) AI products cannot be one-shotted. — Veeral Patel
## Mentioned Nik Koblov, Veeral Patel, Will Koh, Ian Tracey, Ramp, The Pragmatic Summit, OmniChat, Policy Agent, Brian, Andrej Karpathy, Claude Opus, Claude 3.7 Sonnet, Claude Code, OpenAI, Anthropic, Google, Streamlit, Clockwork, LiteLLM, Applied AI Service, Ramp Inspect, open-inspect, Modal, Datadog, incident.io, Notion, Slack, GitHub, VS Code
## Video description With Nik Koblov (EVP Engineering), Veeral Patel (Director, Applied AI and Spend), Will Koh (Staff Engineer, Applied AI), and Ian Tracey (Staff Software Engineer). Recorded at The Pragmatic Summit www.pragmaticsummit.com
Watch the session with Q&A also included: https://newsletter.pragmaticengineer.com/p/the-pragmatic-summit-recordings
0:00 Intro 3:31 Pivot: from many agents to one agent with many skills 7:17 Policy Agent 12:21 Tooling lessons 22:46 Infrastructure + culture 26:13 Tool catalogs, sandboxing, and reliability at scale
[Source](https://www.youtube.com/watch?v=NMs8C2_3M0w)