Booking.com, Google Unveil Enterprise Architecture for Gemini Agents

From prototype to production: 45 minutes to a reliable Gemini Enterprise Agent Platform agent — Google Cloud Tech

Google Cloud Tech

Scaling enterprise AI agents requires replacing manual prompt engineering with automated trajectory evaluations, ephemeral machine identities, and standardized deployment templates.

AI-assisted summary of From prototype to production: 45 minutes to a reliable Gemini Enterprise Agent Platform agent — original source. Check the original for full context.

Source published:

Key ideas

  • Booking.com evolved its generative AI stack from basic 2023 wrappers to an enterprise gateway and production agent platform handling multi-vertical travel workflows at scale. (2:30)
  • Agent evaluation must assess both final responses and internal execution trajectories, using exact, in-order, or any-order matching against golden test datasets. (13:35)
  • Application Design Center (ADC) and Gemini Cloud Assist translate local agent codebases into Terraform templates and cloud architectures in roughly 30 minutes. (23:15)
  • Model Armor acts as an inline threat filter stationed between end users and LLMs to prevent PII leakage, prompt injections, and system prompt extraction without fragile prompt engineering. (30:06)
  • OpenTelemetry has been extended with GenAI-specific signals to capture model metadata, token consumption, and multi-turn tool traces alongside conversational BigQuery analytics. (35:57)

Detailed notes

## Background Published on YouTube by Google Cloud Tech on June 25, 2026, this Google Cloud Next session presents a production-first guide to deploying hardened agents on the Gemini Enterprise Agent Platform. The session description highlights practical architectures for connecting agents via Model Context Protocol (MCP) servers, securing them with enterprise guardrails, monitoring them with real-time observability, and ensuring accuracy through automated evaluations.

## TL;DR Engineers from Booking.com and Google outline the architectural transition required to move autonomous GenAI agents from local prototypes into hardened, enterprise-grade cloud deployments. Drawing on Booking.com's high-throughput travel platform and a multi-agent London concierge sample application, the speakers detail four foundational pillars: multi-turn response and trajectory evaluation, accelerated cloud deployment using Application Design Center (ADC) and Gemini Cloud Assist, runtime security enforced by ephemeral agent identities and Model Armor proxies, and day-zero observability powered by OpenTelemetry and BigQuery analytics. The session demonstrates how platform-level governance, agent gateways, and golden architectural templates allow organizations to safely scale collaborative AI agents without compounding manual engineering overhead.

## Atmosphere & style The session carries the crisp, rapid-fire momentum of an applied engineering masterclass, balancing large-scale corporate credibility with hands-on console demonstrations. Alibek Datbayev and Maria-Irène Călinoiu set a grounded foundation with real production metrics from Booking.com, before Manasa Kandula and Naz Bayrak engage in an energetic tag-team delivery filled with natural onstage rapport and self-deprecating banter about audience clapping and minor AV screen hiccups. The mood remains pragmatic and transparent throughout, openly addressing production realities like inevitable system failures and high token costs while equipping viewers with concrete, actionable architecture patterns.

## Bulletin - [02:30](https://www.youtube.com/watch?v=fkCTifAqVGg&t=150s) Booking.com evolved its generative AI stack from basic 2023 wrappers to an enterprise gateway and production agent platform handling multi-vertical travel workflows at scale. - [13:35](https://www.youtube.com/watch?v=fkCTifAqVGg&t=815s) Agent evaluation must assess both final responses and internal execution trajectories, using exact, in-order, or any-order matching against golden test datasets. - [23:15](https://www.youtube.com/watch?v=fkCTifAqVGg&t=1395s) Application Design Center (ADC) and Gemini Cloud Assist translate local agent codebases into Terraform templates and cloud architectures in roughly 30 minutes. - [30:06](https://www.youtube.com/watch?v=fkCTifAqVGg&t=1806s) Model Armor acts as an inline threat filter stationed between end users and LLMs to prevent PII leakage, prompt injections, and system prompt extraction without fragile prompt engineering. - [35:57](https://www.youtube.com/watch?v=fkCTifAqVGg&t=2157s) OpenTelemetry has been extended with GenAI-specific signals to capture model metadata, token consumption, and multi-turn tool traces alongside conversational BigQuery analytics.

## Key takeaways - [02:30](https://www.youtube.com/watch?v=fkCTifAqVGg&t=150s) Booking.com runs hundreds of production ML models serving 500,000 requests per second with a p999 latency target around 50 milliseconds. - [08:45](https://www.youtube.com/watch?v=fkCTifAqVGg&t=525s) Production conversational agents require tailored serving tiers like provisioned throughput rather than default on-demand endpoints to guarantee latency and tokens per minute. - [08:45](https://www.youtube.com/watch?v=fkCTifAqVGg&t=525s) Building enterprise AI platforms requires investing equally in platform infrastructure and internal developer enablement, documentation, and hackathons. - [13:35](https://www.youtube.com/watch?v=fkCTifAqVGg&t=815s) Evaluating non-deterministic agents requires trajectory evaluation to inspect tool calls, parameters, and sequence fidelity rather than black-box response checks alone. - [16:11](https://www.youtube.com/watch?v=fkCTifAqVGg&t=971s) Trajectory evaluation matching rules range from strict exact matches for high-stakes operational workflows to flexible any-order matches for open-ended recommendations. - [18:32](https://www.youtube.com/watch?v=fkCTifAqVGg&t=1112s) Automated simulation runs benchmark multi-turn conversations against golden JSON datasets and stream granular traces directly into BigQuery. - [20:05](https://www.youtube.com/watch?v=fkCTifAqVGg&t=1205s) Gemini Cloud Assist and the Application Design Center bridge the prototype-to-cloud gap by generating Terraform configurations and visual architecture canvases from source code. - [25:52](https://www.youtube.com/watch?v=fkCTifAqVGg&t=1552s) An enterprise Agent Registry acts as a centralized catalog to prevent duplicate tooling, while an Agent Gateway enforces platform-level communication boundaries. - [27:58](https://www.youtube.com/watch?v=fkCTifAqVGg&t=1678s) Agent Identity dynamically provisions ephemeral, least-privilege IAM credentials at runtime, producing granular audit trails tied to exact agent versions. - [30:06](https://www.youtube.com/watch?v=fkCTifAqVGg&t=1806s) Model Armor secures agents against PII exfiltration, prompt injection, and jailbreak attempts at the network layer rather than relying on endless prompt engineering. - [33:38](https://www.youtube.com/watch?v=fkCTifAqVGg&t=2018s) Embedding native observability into SDKs shifts telemetry from an expensive day-two afterthought to a day-zero operational standard. - [35:57](https://www.youtube.com/watch?v=fkCTifAqVGg&t=2157s) OpenTelemetry extensions capture GenAI-specific metrics, including model checkpoint versions, inference latency, temperature settings, and tool invocations. - [38:00](https://www.youtube.com/watch?v=fkCTifAqVGg&t=2280s) The Agent Development Kit BigQuery plugin streams conversational turns and metadata into SQL tables for business sentiment and token cost analysis. - [42:02](https://www.youtube.com/watch?v=fkCTifAqVGg&t=2522s) ADC golden templates allow platform teams to encode security guardrails, IAM standards, and observability pipelines into pre-approved architectures that scale across development teams.

## Detailed walkthrough ### [00:13](https://www.youtube.com/watch?v=fkCTifAqVGg&t=13s) Introduction and the Booking.com & Google Partnership

Alibek Datbayev, an engineering manager at Booking.com leading the AI Platform team, opens the session on taking reliable agents from prototype to production. He introduces his colleague Maria-Irène Călinoiu alongside Googlers Manasa Kandula and Naz Bayrak, setting the stage for a practical masterclass on hardened enterprise AI. Alibek highlights that Booking.com has collaborated with Google for years beyond traditional performance marketing: “It’s a battle tested partnership that is built on deep trust and technology that actually delivers.” Booking.com actively leverages Gemini and Vertex AI to unlock automated, smarter travel experiences centered around its foundational mission—making it easier for everyone to experience the world.

To emphasize that mission, Alibek asks the audience for a show of hands on who traveled internationally to attend the conference, observing that almost the entire room raised their hands. He follows up by asking how many used the Booking.com app to plan their trip, expressing excitement at the strong response. He underscores that travel fosters personal growth and cultural understanding precisely because people rediscover what they love and explore unfamiliar territory, directly embodying the core purpose behind the products his team supports.

[02:30](https://www.youtube.com/watch?v=fkCTifAqVGg&t=150s) Scale of the Booking.com Platform and Evolution of Gen AI

Alibek provides concrete numbers demonstrating the sheer magnitude of Booking.com's operations. Rather than functioning simply as a hotel reservation website, it operates as a fully integrated multi-vertical travel ecosystem covering accommodations, flights, rental cars, attractions, and ground transport. The platform's scale is backed by massive production metrics:

- **6.8+ billion** guest arrivals facilitated since 2010. - **175,000** global destinations covered across 70 countries. - **140** international offices, with roughly 7,000 employees based at the global headquarters in Amsterdam, Netherlands. - **10+ years** of running predictive machine learning models in production. - **Hundreds of production ML models** handling approximately 500,000 requests per second with a p999 latency target around 50 milliseconds.

```mermaid flowchart LR accTitle: Booking.com Generative AI Evolution accDescr: Timeline of Booking.com AI maturity from Phase 0 experimentation to an enterprise gateway and production agent platform. subgraph S1["Phase 0: 2023"] A["ChatGPT Moment<br/>Simple wrapper apps"] end subgraph S2["Phase 1: Early 2024"] B["Stack revamp<br/>In-house models<br/>Multi-provider setup"] end subgraph S3["Phase 2: Gateway"] C["Single API gateway<br/>Built-in governance<br/>Usage monitoring"] end subgraph S4["Phase 3: Production"] D["Enterprise Ecosystem<br/>Evals & safety guardrails<br/>Agentic capabilities"] end A --> B B --> C C --> D ```

Turning to generative AI, Alibek traces the architectural progression since the "ChatGPT moment" in 2023. Phase 0 began with basic API wrappers to get a foot in the door. By early 2024, Booking.com revamped its core stack to incorporate in-house models across a multi-provider landscape. The pivotal milestone occurred when the platform unified all access under a single internal gateway featuring built-in governance, cost attribution, and usage monitoring. Today, Booking.com manages the complete generative lifecycle, spanning advanced evaluations, enterprise safety guardrails, and production-ready agentic capabilities to scale multiple travel use cases rapidly.

[05:47](https://www.youtube.com/watch?v=fkCTifAqVGg&t=347s) Gemini Use Cases at Booking.com

Maria-Irène Călinoiu takes over to detail four distinct applications powered by Gemini models across customer-facing, internal engineering, and partner-facing workflows. The flagship customer-facing project, built as a proof of concept alongside Google engineers, is titled **Know Before You Go**. It tackles the stressful gap between booking an itinerary and arriving at the destination. Travelers can dynamically generate customized itineraries based on group dynamics or time constraints and figure out airport-to-hotel transit. Under the hood, the system relies on Gemini grounded with Google Maps and Google Search to merge real-time external knowledge with complex reasoning.

```mermaid flowchart TD accTitle: Gemini Use Cases at Booking.com accDescr: Four production and POC generative AI use cases across customer, internal, partner, and multimodal touchpoints. subgraph External["Customer & Partner Touchpoints"] A["Know Before You Go<br/>(Grounding via Maps & Search)"] C["Partner Analytics<br/>(Review signal extraction)"] D["Reels to Reality<br/>(Video URLs to itineraries)"] end subgraph Internal["Developer Productivity"] B["BigBot Registry Chatbot<br/>(Data discovery & workflows)"] end ```

Beyond customer planning, Maria-Irène outlines three other deployments: - **BigBot (Internal Data Registry):** A developer-facing chatbot that helps engineers locate data assets across a vast corporate repository, ask architectural questions, and build natural language automated workflows. - **Partner Analytics (Host Facing):** Instead of using large language models solely for text generation, this application performs structured signal extraction on raw customer comments and reviews. Property partners can immediately spot recurring operational pain points, such as complaints regarding pool maintenance or breakfast variety, and prioritize repairs. - **Reels to Reality (Multimodal Travel Planning):** Recognizing that travelers draw heavy inspiration from social video, this project accepts direct YouTube URLs into Gemini to parse video contexts and automatically assemble concrete, bookable itineraries within the Booking.com app.

[08:45](https://www.youtube.com/watch?v=fkCTifAqVGg&t=525s) Production Learnings: Serving Tiers and Platform Enablement

Maria-Irène emphasizes that finding a suitable model checkpoint is minor compared to solving how models are served and consumed internally. Booking.com experimented extensively across Vertex AI serving tiers, learning early on that standard on-demand endpoints were neither fast nor predictable enough for conversational applications like Know Before You Go. The team migrated to provisioned throughput to guarantee dedicated tokens per minute, and adopted priority processing to handle demanding multimodal workloads like Reels to Reality. As Maria-Irène observes, “The real takeaway has been that we are trying to match the serving strategy with the production need, rather than just assume that a default is going to do the trick.”

The second operational challenge centers on knowledge enablement. Because the AI platform team provides self-service infrastructure rather than writing application code for every product vertical, building the platform constitutes only half the battle. The other half involves running internal hackathons, compiling up-to-date documentation on model providers, and guiding product teams on which architecture fits their latency and cost profile. She encapsulates the company's trajectory: “Booking.com has been moving from isolated experiments to a governed multi-provider production ready AI ecosystem.”

[11:13](https://www.youtube.com/watch?v=fkCTifAqVGg&t=673s) The Four Pillars of Production Agent Architecture

Manasa Kandula steps up to transition the presentation from Booking.com's user journey into the core engineering curriculum for deploying Gemini Enterprise agents. She identifies the central anxieties development teams encounter once they push past initial toy prototypes: guaranteeing agent quality against strict corporate metrics, maintaining control as the sheer count of tools and downstream agents scales up, and keeping production systems locked down against novel security vulnerabilities.

To resolve these enterprise bottlenecks, Manasa outlines the four core pillars that she and Naz Bayrak will unpack across the remainder of the session: 1. **Pre-Production Agent Evaluation:** Implementing automated evaluation techniques, benchmarks, and multi-turn scoring rubrics before code ever touches real users. 2. **Accelerated Code-to-Cloud Deployment:** Best practices for packaging code and moving from local development into managed cloud infrastructure cleanly and quickly. 3. **Runtime Agent Identity and Security:** Provisioning fine-grained agent identities, implementing model guardrails, and enforcing least-privilege tool security during live execution. 4. **End-to-End Observability and Auditability:** Establishing complete tracing, inspection, and debugging capabilities to isolate failures and monitor agent reasoning paths in real time.

[12:25](https://www.youtube.com/watch?v=fkCTifAqVGg&t=745s) Introducing the Speakers and the London Concierge Sample Demo

Naz Bayrak introduces her colleague Manasa Kandula, an application modernization specialist, and introduces herself as an AI specialist, noting that both are based out of Google’s Amsterdam office. To ground the discussion in practical enterprise architecture, they walk through a multi-agent travel application inspired by their colleagues at Booking.com. At the core of the system is the **London travel concierge**, the primary front-facing agent designed to interact directly with users and provide personalized itinerary recommendations.

The London concierge orchestrates tasks by communicating directly with two downstream agents: a ticket agent and an activities agent. To handle dynamic conditions, the system connects to external environments through Model Context Protocol (MCP) servers. Because London weather is notoriously unpredictable, the concierge relies on a weather MCP server for live updates. In parallel, it queries a custom London data MCP server that bridges the agent to proprietary enterprise information stored in Cloud SQL, ensuring that itinerary and venue suggestions reflect authoritative business data rather than generic web knowledge. The entire infrastructure is backed by Vertex AI Agent Engine and hardened using Model Armor to maintain enterprise guardrails.

```mermaid flowchart TD accTitle: London Concierge Multi-Agent Architecture accDescr: Architecture showing the London travel concierge routing requests to sub-agents and external MCP servers backed by Cloud SQL, Agent Engine, and Model Armor.

User(["End User"]) --> MainAgent["London Travel Concierge\n(Main Orchestrator)"]

subgraph SubAgents ["Collaborating Sub-Agents"] TicketAgent["Ticket Agent"] ActivitiesAgent["Activities Agent"] end

subgraph DataIntegrations ["MCP Server Integrations"] WeatherMCP["Weather MCP Server\n(Live Weather)"] DataMCP["London Data MCP Server"] CloudSQL[("Cloud SQL\nPostgreSQL + Vector")] DataMCP --> CloudSQL end

MainAgent --> TicketAgent MainAgent --> ActivitiesAgent MainAgent --> WeatherMCP MainAgent --> DataMCP

subgraph EnterpriseInfra ["Platform Foundation"] AgentEngine["Vertex AI Agent Engine"] ModelArmor["Model Armor Security"] end ```

[13:35](https://www.youtube.com/watch?v=fkCTifAqVGg&t=815s) Evaluation Paradigms: Response vs. Trajectory

Evaluating agents is fundamentally more complex than evaluating traditional machine learning models because agents operate non-deterministically. Naz highlights this inherent unpredictability: “Agents are non-deterministic and this brings some challenges to our evaluation systems.” She divides agent evaluation into two major components: - **Response Evaluation:** Treating the agent as a black box, this method disregards everything occurring internally during execution and focuses strictly on the final output. This mirrors standard LLM evaluation methodologies. - **Trajectory Evaluation:** Unique to autonomous multi-agent systems, this method peers into the internal reasoning path. It inspects which specific tools were invoked, which parameters were supplied, which sub-agents were engaged, and the precise sequence of collaboration. Evaluators match this execution path against a predefined "golden goal" dataset to verify operational fidelity.

[14:30](https://www.youtube.com/watch?v=fkCTifAqVGg&t=870s) Evolution from Human-in-the-Loop to LLM-as-a-Judge and Automated Metrics

Four years ago, evaluation relied almost entirely on human review: an evaluator inspected an output and manually determined whether it was good enough based on business criteria. However, manual human evaluation is inherently subjective, expensive, and difficult to scale across thousands of production requests. The industry adapted by moving toward **LLM-as-a-Judge**, where a secondary language model is tasked with grading the output of the primary model. Naz explains: “Now the advantage of using LLM as Judge is very scalable. It's efficient and consistent.” Despite these benefits, LLM-as-a-Judge has clear constraints: it is fundamentally bounded by the evaluator model's intelligence and struggles with complex deterministic reasoning, such as verifying code modernization.

To build rigorous evaluation pipelines, teams incorporate automated metrics. While conventional AI practitioners rely on classical statistical metrics like RMSE or mean absolute error, LLM-based agent evaluation uses automated metrics such as hallucination, groundedness, and coherence scores (often computed via an evaluator LLM). For agent systems specifically, SDKs combine these response scores with automated trajectory metrics to measure tool-calling accuracy alongside text quality.

[16:11](https://www.youtube.com/watch?v=fkCTifAqVGg&t=971s) Trajectory Metrics: Exact Match, In-Order Match, and Any-Order Match

When configuring trajectory evaluation within an agent SDK, architects must define how strictly an agent's execution path must match the golden benchmark dataset. Naz breaks these down into three primary matching strategies:

- **Exact Match:** The agent must replicate the benchmark path identically. Every tool called, every input argument provided, and the absolute sequence of executions must match the golden test set flawlessly. The final prose output is secondary to whether the operational mechanics were executed with absolute precision. This is essential for operations like the ticket agent, where applying the wrong promotional code or charging the wrong fee has real financial repercussions. - **In-Order Match:** The agent must hit the required milestones in the correct sequence, but deviations or side detours are permitted. If an agent spontaneously triggers a Google Search tool that was not in the golden test set, latency may increase slightly, but the run still passes as long as the essential tools execute in the required order. Naz notes: “In order match so you can detour, but as long as you get to the right destination, meaning you call the right tools, that's OK.” - **Any-Order Match:** The agent must call all required tools, but the execution order is irrelevant. This is common in exploratory or recommendation tasks; for example, whether the London concierge gathers recommendations for Soho or Westminster first does not matter, as long as both relevant data sets are retrieved for the user.

While automated trajectory metrics offer objectivity and massive scale, they have limitations: they struggle to quantify creativity, and systems can appear to "cheat" or pass artificially if an engineer selects an overly lenient metric for a strict operational workflow.

[18:32](https://www.youtube.com/watch?v=fkCTifAqVGg&t=1112s) Simulation and Granular Metric Logging Demo

In a live walkthrough of their sample configuration, Naz demonstrates how metrics and test cases are structured using the agent SDK. The setup loads a configuration containing both response metrics (hallucination, safety) and trajectory metrics (tool invocation fidelity). Test datasets are created directly within the SDK user interface by engaging in multi-turn dialogues with the agent and saving high-quality runs as a golden test set in JSON format.

During automated simulation runs: - The system plays back the input prompts against the agent. - The agent’s outputs and tool invocations are executed and compared turn-by-turn against the golden baseline. - Individual scores are recorded for every conversation turn (prompt, expected response, actual output, and metric results). - The granular execution trace and aggregated conversation-level metrics are streamed and persisted into BigQuery, providing an audit trail to decide if an agent is ready for production.

[20:05](https://www.youtube.com/watch?v=fkCTifAqVGg&t=1205s) Accelerating Developer Velocity: Gemini Cloud Assist and Application Design Center

Naz yields the floor to Manasa Kandula, prompting lighthearted applause from the audience. Manasa quips: “We're going to be switching back and forth often, so don't need to clap every time, but we appreciate it.” Manasa focuses on developer velocity, noting that while modern code-assist tools make hacking together a local prototype fast and easy, the transition to production often stalls when moving code into a secure, hardened enterprise cloud architecture. The shift to AI agents introduces unfamiliar runtimes, evolving security perimeters, and new architectural paradigms that significantly increase cognitive load on developers.

To bridge this gap, teams can use **Gemini Cloud Assist** alongside the **Application Design Center (ADC)**: - **Gemini Cloud Assist:** Functions as an AI cloud architect. It parses codebases, interprets developer intent, and maps required agent primitives to idiomatic Google Cloud infrastructure. In the London concierge app, which depends on a PostgreSQL database with the `pgvector` extension, Gemini Cloud Assist automatically recommends provisioning a Cloud SQL for PostgreSQL instance configured as a private instance to satisfy enterprise isolation requirements. - **Application Design Center (ADC):** Acts as a centralized repository to store, manage, and deploy approved architectural templates. It ensures governance and delivers out-of-the-box observability metrics from day one. - **Developer Workflow via MCP:** Developers can interact with their codebase through Gemini CLI in their IDE, backed by Gemini Cloud Assist and ADC MCP servers. These MCP tools analyze local code, generate appropriate deployment architectures, deploy resources straight to Google Cloud, and troubleshoot deployment discrepancies inline.

The demonstration shows an IDE housing the concierge agent codebase, verifying first that the MCP configuration has correctly mounted the Application Design Center and Gemini Cloud Assist MCP servers to manage the deployment lifecycle.

[23:15](https://www.youtube.com/watch?v=fkCTifAqVGg&t=1395s) From Codebase to Cloud: Automated Architecture and Deployment with ADC

Manasa demonstrates how an agent can take a raw local application and deploy it onto Google Cloud using automated infrastructure tooling. Starting from a single natural-language instruction—“design a Cloud architecture from our code base”—the agent scans the entire local repository, analyzes its internal components, and deduces how services map and connect to one another. Once it builds that baseline mental model, it invokes the Gemini Cloud Assist MCP tool to translate those application requirements into a complete architecture. It generates Terraform templates, writes them directly back into the local codebase, and registers the deployment specification into the Application Design Center (ADC).

Within ADC, developers are presented with a visual design canvas where they can inspect the architecture, review configurations, and trace component dependencies before anything goes live. Once approved, the agent initiates the deployment pipeline via Cloud Build. In the demonstration, a deployment process that typically takes 10 to 15 minutes is fast-forwarded to reveal automated status checks, deployment logs, and the provisioning of a public Load Balancer IP address. Manasa pulls this IP address directly from the output and loads the live travel concierge application in the browser, noting that bridging the gap between a local prototype and an enterprise-grade cloud deployment usually burns days of engineering time spent matching cloud primitives and applying security baselines. As Manasa explains: “And we managed to do this in a matter of about half an hour using these two tools.”

```mermaid flowchart TD accTitle: Codebase to Cloud Deployment Flow accDescr: Automated transition from local code to cloud architecture using MCP and ADC subgraph Design ["1. Architecture & Design"] A["Local Codebase"] --> B["Agent Code Analysis"] B --> C["Gemini Cloud Assist MCP"] C --> D["Terraform Templates"] D --> E["ADC Visual Canvas"] end

subgraph Deploy ["2. Approval & Deployment"] E --> F["Developer Approval"] F --> G["Cloud Build Pipeline"] G --> H["Google Cloud Infrastructure"] H --> I["Load Balancer IP"] end ```

- The agent required only a single natural-language instruction to parse the entire codebase without manual configuration mapping. - Infrastructure-as-code artifacts were produced as native Terraform templates written directly to the project directory. - ADC provided a visual representation of all cloud resources, relations, and configurations prior to execution. - Automated Cloud Build routines verified operational health and deployment logs, concluding with a functional external Load Balancer IP.

---

[25:52](https://www.youtube.com/watch?v=fkCTifAqVGg&t=1552s) Scaling the Ecosystem: Agent Registry and Agent Gateway

As organizations build multiple autonomous agents and Model Context Protocol (MCP) servers, they immediately hit scaling bottlenecks around visibility, duplicated engineering effort, and perimeter control. Manasa introduces two core architectural answers: the Agent Registry and the Agent Gateway. The Agent Registry serves as a centralized internal catalog and unified source of truth for all deployed agents and MCP tools across an organization. Instead of separate development groups reinventing duplicate capabilities—such as multiple variations of ticket-booking integrations—teams can discover existing services, view their specifications, and consume them directly.

However, discovery without governance leads to architectural chaos and severe security risks. With dozens of agents operating across an enterprise, organizations cannot allow an unconstrained web of agent-to-agent communication. For instance, while a customer-facing travel concierge must communicate with an external ticketing agent to purchase flights, it must be explicitly barred from ever reaching an internal human resources agent. Rather than burdening individual application developers with hand-coding custom authorization and routing rules between models, the Agent Gateway shifts policy enforcement to the infrastructure level. Manasa emphasizes the platform-first role of this mechanism: “You can think of it as the governance layer for your agents and MCP tools.”

- **Agent Registry:** Functions as an enterprise catalog and single source of truth to promote asset discovery and prevent redundant engineering. - **Agent Gateway:** Acts as the centralized governance proxy, enforcing boundary policies regarding which agents and MCP tools are permitted to call one another. - **Platform-Level Policy:** Eliminates developer burden by applying communication guardrails at the infrastructure layer rather than relying on custom application code.

---

[27:58](https://www.youtube.com/watch?v=fkCTifAqVGg&t=1678s) Runtime Security: Ephemeral Agent Identity and Audit Trails

Naz Bayrak steps up to break down runtime production safety into three critical pillars: identity, threat defense, and observability/auditability. The first line of defense is Agent Identity, a native capability designed to give autonomous models their own enterprise credentials. Traditionally, developers assign agents broad, long-lived service account roles whose entire lifecycle, rotation, and privilege scope must be manually administered. If an agent executes an action under a generic service account, incident responders cannot distinguish which specific agent or model version initiated the downstream API call.

Under the new Agent Identity architecture, an agent is dynamically assigned a unique, least-privilege IAM identity at deployment and runtime that disappears when the workload terminates. When the concierge agent accesses a ticketing backend, it signs and authenticates using its own distinct, scoped identity. If anomalous behavior occurs, security operations teams are not left staring at ambiguous service-level calls; they receive a granular log specifying the exact agent instance, version number, and execution context. As Naz states: “And you'll be able to have an irrefutable audit trail to trace back exactly what happened.”

- Traditional architectures require teams to maintain permanent service roles and manual credential lifecycles for agents. - Agent Identity binds ephemeral, runtime IAM credentials to specific agent instances without developer management overhead. - Identifiers and permissions persist only while deployment workloads are actively running and vanish upon shutdown. - Operational logs provide immutable attribution containing the exact agent name, build version, and identity context for every action.

---

[30:06](https://www.youtube.com/watch?v=fkCTifAqVGg&t=1806s) Defending the Frontier: Live Threat Mitigation with Model Armor

Securing live generative AI applications involves protecting sensitive data and company reputation against bad actors without relying on fragile, endless system prompt tweaks. Naz outlines the primary threats confronting customer-facing agents: leaks of personally identifiable information (PII), unauthorized commentary on business competitors, toxic or biased outputs, and deviations from brand voice. Rather than forcing engineers to spend weeks crafting and evaluating defensive prompts that can still be bypassed, Google Cloud introduces Model Armor—an inline security proxy stationed directly between incoming user prompts and the underlying foundation model. Naz summarizes its design objective: “Model Armor resides between the user and your LLM to prevent all of these from happening.”

```mermaid flowchart LR accTitle: Model Armor Security Proxy accDescr: Incoming user requests are filtered by Model Armor before reaching the LLM A["End User / Attacker"] --> B["Model Armor Inspection Layer"] subgraph Inspection ["Threat Filtering Engine"] B -.-> C["PII / Exfiltration Detection"] B -.-> D["Prompt Injection & Jailbreak"] B -.-> E["System Prompt Leakage"] B -.-> F["Malicious URL Interception"] end B -->|"Sanitized Traffic"| G["Foundation LLM / Agent"] B -->|"Blocked Attack"| H["Safe Fallback Response"] ```

To demonstrate its capabilities, Naz runs a live attack suite against the travel application UI. An attacker first attempts data exfiltration by coaxing the model into dumping customer credit card records; Model Armor detects the intent and suppresses the response. Successive attacks escalate through direct prompt injections, creative jailbreaks, and system instruction extraction (“prompt leak”) attacks where the user tries to compel the agent to reveal its internal guidelines. The interface shows the attacker becoming increasingly frustrated as the filter neutralizes safety violations and flags malicious URLs. To validate that the system does not simply drop all user inputs, Naz executes a standard travel query, which passes through cleanly, followed by a final test against sensitive data detection that is immediately blocked.

- Intercepts threats natively in the network path, replacing fragile, long-winded system prompt instructions. - Shields systems against automated data exfiltration attempts targeting PII and financial records. - Mitigates direct and indirect prompt injections, jailbreaks, and attempts to extract system instructions. - Inspects and filters malicious URLs and policy-violating language while allowing legitimate business interactions to flow freely.

---

[33:38](https://www.youtube.com/watch?v=fkCTifAqVGg&t=2018s) Observability: Moving Production Telemetry from Day Two to Day Zero

Manasa returns to address the reality of operating generative AI workloads in production environments, opening with a direct question to the audience: “How many of you have had production applications that have had no issues.” When not a single person raises their hand, she highlights the universal reality of production failures and the critical need for deep observability. The fundamental challenge of modern software operations is diagnosing unexpected behaviors quickly, yet observability is routinely deferred. As Manasa observes: “And one of the things that frustrates me is that observability is often referred to as a day two operation.”

Observability is traditionally treated as an afterthought because implementing it is tedious and expensive. Application developers must manually instrument source code with tracing spans, while platform engineers must extract signals from cloud infrastructure and stitch together complex custom dashboards that correlate application context with system telemetry. Because of this high initial effort, teams rarely implement proper observability at launch ("day zero"), leaving production agents flying blind. Manasa explains that Google Cloud resolved this dilemma by integrating deep telemetry natively into the platform SDKs, capturing essential execution signals automatically from the moment of deployment.

- The audience poll demonstrated that virtually every production system experiences operational faults. - Traditional observability efforts require heavy manual code instrumentation and complex custom cloud dashboard engineering. - High upfront implementation friction causes development teams to defer monitoring strategies until post-launch ("day two"). - Google Cloud embeds baseline telemetry into the agent platform SDK to provide native visibility on "day zero" without manual instrumentation overhead.

[34:54](https://www.youtube.com/watch?v=fkCTifAqVGg&t=2094s) Unified Observability with Application Design Center (ADC)

Application observability is enabled out of the box with the click of a single button. By utilizing Application Design Center (ADC) to deploy the entire application, developers instantly gain platform-level dashboards. What sets ADC apart from standard cloud monitoring is that instead of treating infrastructure as disconnected silos—leaving teams to piece together separate signals from a Cloud SQL Postgres database, individual Cloud Run instances, and other isolated services—it binds every resource together under a single, unified application context. This allows engineering teams to monitor and operate the application as an integrated whole rather than juggling fragmented metrics across a Google Cloud project.

Alongside the core infrastructure metrics, the deployment provides a dedicated Agent App dashboard. This dashboard surfaces specialized GenAI operational telemetry critical for running production AI workloads, including token count consumption and latency metrics segmented by specific models.

- **Unified Context:** ADC eliminates operational silos by correlating signals from Cloud Run, Cloud SQL Postgres, and other resources into a single application-level topology. - **Out-of-the-Box Metrics:** Automated dashboards track real-time GenAI performance indicators, such as request latency per model and cumulative token usage.

---

[35:57](https://www.youtube.com/watch?v=fkCTifAqVGg&t=2157s) Diagnosing GenAI Agents with OpenTelemetry Standards

Once monitoring dashboards detect an anomaly or performance regression, engineers require deep diagnostic instrumentation to isolate the root cause. Google Cloud actively champions OpenTelemetry (OTel), the open, vendor-neutral standard used across the industry to construct resilient telemetry pipelines without vendor lock-in. To address modern AI application stacks, Google Cloud collaborated with the open-source software (OSS) community to extend the core OpenTelemetry protocol, adding native telemetry signals specifically tailored for GenAI architectures.

Before these extensions, telemetry conventions only accommodated traditional application interactions. Because GenAI systems introduce non-deterministic model behaviors, execution trajectories, and external tool calls, extended OpenTelemetry semantics now capture granular model metadata alongside conventional execution traces.

As the speaker attempts to switch over to a live demonstration to display one of these GenAI execution traces, the screen switch encounters a brief technical hitch. The speaker reassures the audience with a smile: “It's worth it, I promise. It just takes a couple of seconds to come up.” After a brief pause waiting for the AV feed, the speaker decides to defer the live trace view until the end of the presentation so the AV team can assist, ensuring only a single screen switch is needed.

---

[38:00](https://www.youtube.com/watch?v=fkCTifAqVGg&t=2280s) End-to-End Traces and Business Analytics with BigQuery

While technical metrics confirm that an application is running cleanly without throwing errors, teams still face an existential operational question: “The thing is, the question here is, how do you know if your agents are actually delivering value to your end customer?” To evaluate real-world impact, the agent uses the BigQuery analytics plugin provided by the Agent Development Kit (ADK). This plugin captures every conversation turn, extracts all conversation metadata and message payloads, and streams them into a structured BigQuery table for high-powered SQL analytics.

```mermaid flowchart LR accTitle: GenAI Trace and Analytics Pipeline accDescr: End-to-end trace execution and conversational data export to BigQuery.

subgraph AgentExecution ["Agent Execution Turn"] U["End User"] --> A["Gemini 2.5 Flash Agent"] A -->|"tool call"| T["London Activities Search Tool"] T -->|"tool result"| A A -->|"final response"| U end

subgraph ObservabilityLayer ["OTel & ADK Analytics"] A -.->|"OTel GenAI spans"| TR["Trace Explorer"] A -.->|"ADK BigQuery Plugin"| BQ[("BigQuery Analytics")] BQ --> S["SQL Sentiment & Token Analysis"] end ```

Switching successfully to the live console, the speaker inspects an end-to-end trace of a single conversational turn. The waterfall trace details every internal hop and latency breakdown: the initial user request, the call to the large language model, the agent invoking an external tool to search for London activities data, the tool returning results to the model, and the model rendering the final output. The trace spans are explicitly tagged with GenAI metadata, highlighting that the agent used Gemini 2.5 Flash with a temperature setting of 0.01 and consumed 3,658 tokens for that single turn—prompting the speaker to joke, “We need to optimize our agent. But anyway, so you can see all of this in one space.”

Turning to the BigQuery console, the speaker reviews interaction records for a single day, capturing timestamps, conversation events, attributes, and user prompts. Running a SQL sentiment analysis query sorts the conversations by customer satisfaction score on a 1-to-5 scale: - **Five-star interaction:** A satisfied user reviewing an itinerary states, *“Yes, day 3 is absolutely perfect. Those are exactly the kind of things I want to do,”* receiving a top rating of 5/5. - **One-star interaction:** Exporting lower-scoring conversations into a Google Sheet reveals a frustrated user complaining, *“This is really not working out, so you can't find anything,”* resulting in a bottom rating of 1/5. - **Token consumption analytics:** A custom dashboard graphs average daily token consumption by interaction type, enabling engineers to identify heavy prompt chains and optimize cost profiles.

---

[42:02](https://www.youtube.com/watch?v=fkCTifAqVGg&t=2522s) Scaling Enterprise Governance with ADC Golden Templates

Deploying, hardening, and instrumenting a single agent requires thoughtful configuration, but scaling that manual process across an enterprise with 10 or 100 agents quickly becomes impossible. As the speaker emphasizes, “Are you going to be able to scale the steps that we talked about one by one for all your agency and organization, scaling manual oversight is really difficult.”

Application Design Center solves this scaling challenge through reusable architectural governance. While ADC can scan an individual developer's repository to generate a bespoke deployment template, an enterprise platform engineering team can instead establish an organization-wide “golden template” for all agentic applications. This golden template bakes in mandatory enterprise policies, MCP tool connection standards, IAM permissions, security guardrails, evaluation gates, and telemetry pipelines by default. Individual developers simply take the pre-approved platform template, configure their specific agent logic and tools, and deploy into production with guaranteed compliance.

---

[43:26](https://www.youtube.com/watch?v=fkCTifAqVGg&t=2606s) Session Recap: The Four Pillars of Production Agents

With two minutes remaining, the presentation transitions back to Naz Bayrak to summarize the core pillars required to move GenAI agents from prototypes to hardened production systems:

1. **Evaluation:** Assessing agents through both response evaluation and trajectory evaluation, using automated metrics to validate multi-step tool execution paths. 2. **Efficient Cloud Deployment & Registries:** Packaging agents efficiently for Google Cloud and establishing centralized registries for discovering and sharing agent tools and capabilities. 3. **Identity, Access, and Threat Modeling:** Enforcing least-privilege IAM specifically mapped to agents at deployment time, and actively mitigating common agent security vulnerabilities. 4. **Observability, Auditability, and Platform Governance:** Implementing comprehensive OTel tracing and BigQuery conversational analytics, while using standardized ADC golden templates to scale compliance and governance across teams without manual overhead.

Naz wraps up by inviting attendees to continue the discussion outside: “Given that I see that we are almost out of time, we will be in the corridor in case any of you would like to ask any questions so that we get to understand your use cases and see how we can help you better.”

## Notable quotes > [08:45](https://www.youtube.com/watch?v=fkCTifAqVGg&t=525s) The real takeaway has been that we are trying to match the serving strategy with the production need, rather than just assume that a default is going to do the trick. — Maria-Irène Călinoiu

> [33:38](https://www.youtube.com/watch?v=fkCTifAqVGg&t=2018s) And one of the things that frustrates me is that observability is often referred to as a day two operation. — Manasa Kandula

> [13:35](https://www.youtube.com/watch?v=fkCTifAqVGg&t=815s) Agents are non-deterministic and this brings some challenges to our evaluation systems. — Naz Bayrak

## Mentioned Alibek Datbayev, Application Design Center, BigBot, BigQuery, Booking.com, Cloud Build, Cloud Run, Cloud SQL, Gemini, Gemini 2.5 Flash, Gemini Cloud Assist, Google Cloud, Google Maps, Google Search, Know Before You Go, Manasa Kandula, Maria-Irène Călinoiu, Model Armor, Model Context Protocol, Naz Bayrak, OpenTelemetry, PostgreSQL, Reels to Reality, Terraform, Vertex AI, Vertex AI Agent Engine

## Video description Simple agents are easy. Reliable agents are hard. Join us for a high-velocity master class on the production-first approach to Gemini Enterprise Agent Platform. We’re breaking down the complex transition from a basic prompt to a hardened enterprise tool. First, connect your agent by using Model Context Protocol (MCP) servers for secure data context. Next, protect it by implementing enterprise-grade security guardrails. Then, inspect it by setting up observability to monitor agent behavior in real time. And finally, perfect it by running automated evaluations to ensure accuracy at scale. In this session, expect no fluff – just the architecture you need to ship with confidence.

Watch more Google Cloud Next 2026 Sessions → https://goo.gle/gcloud-next-2026 Subscribe to Google Cloud Tech → https://goo.gle/GoogleCloudTech

Speakers: Alibek Datbayev, Manasa Kandula, Naz Bayrak, Maria-Irène Călinoiu

BRK2-014 #GoogleCloudNext

[Source](https://www.youtube.com/watch?v=fkCTifAqVGg)

Reading summary

Transitioning generative AI agents from prototypes to production requires moving beyond basic prompt wrappers toward governed cloud architectures, rigorous trajectory evaluations, runtime security, and unified observability, according to engineering leaders from Booking.com and Google Cloud. From Experimental Wrappers to Governed Agent Ecosystems (2:30) Booking.com processes roughly 500,000 requests per second across hundreds of production machine learning models with a target latency around 50 milliseconds for the 99.9th percentile. Following the emergence of generative AI in 2023, the organization evolved from basic API wrappers to in-house models, eventually consolidating access under an enterprise gateway equipped with cost attribution, governance, and monitoring. Practical implementations powered by Gemini include the customer-facing "Know Before You Go" planner grounded in Google Maps and Search, developer data discovery bots, review signal extraction for property partners, and multimodal itinerary generation from social video links. Serving Tiers and Dual-Track Agent Evaluations (8:45) Operationalizing conversational agents revealed that standard on-demand model endpoints lacked the latency predictability required for production. Booking.com adopted provisioned throughput for dedicated tokens and prioritized processing for multimodal workloads. Furthermore, evaluating non-deterministic multi-agent systems requires separating traditional response evaluation from trajectory evaluation, which assesses internal reasoning sequences and tool invocations against baseline datasets. Automated Cloud Deployment and Inter-Agent Governance (20:05) Bridging local prototypes into cloud architectures often stalls over infrastructure complexity. Using Gemini Cloud Assist and Application Design Center (ADC), teams can translate codebases into Terraform templates, preview dependencies on a visual canvas, and deploy to Google Cloud services like Cloud SQL with pgvector via automated Cloud Build pipelines. To prevent redundant development and unconstrained agent communication at scale, organizations employ an Agent Registry for tool discovery and an Agent Gateway to enforce perimeter policies determining which models and tools can interact. Runtime Security via Ephemeral Identity and Inline Guardrails (27:58) Securing operational agents requires moving away from static, broad service accounts. The platform assigns unique, ephemeral IAM identities during runtime that vanish upon workload termination, providing granular attribution in audit logs for every API call. To prevent data exfiltration, prompt injections, system prompt leaks, and harmful URLs, the architecture interposes an inline security proxy, Model Armor, directly between the end user and foundation models rather than relying solely on defensive prompting. Turn-Zero Observability and Standardized Golden Templates (33:38) Rather than treating monitoring as a post-launch task, telemetry is built natively into platform SDKs using extended OpenTelemetry conventions for generative AI. Unified dashboards trace latencies and token counts across cloud resources, while an Agent Development Kit plugin streams conversation turns into BigQuery for sentiment and cost analysis. To scale these governance standards across dozens of workloads without manual oversight, platform teams publish ADC "golden templates" embedding mandatory IAM policies, Model Context Protocol configurations, guardrails, and telemetry by default. Scaling autonomous enterprise agents shifts the primary development bottleneck from initial prompt engineering to robust cloud infrastructure management. While standardized golden templates, inline threat filters, and trajectory metrics provide structural controls, ongoing operational challenges persist around balancing strict execution paths against agent reasoning flexibility.

Source quotes

[8:45] Maria-Irène Călinoiu: The real takeaway has been that we are trying to match the serving strategy with the production need, rather than just assume that a default is going to do the trick. [33:38] Manasa Kandula: And one of the things that frustrates me is that observability is often referred to as a day two operation. [13:35] Naz Bayrak: Agents are non-deterministic and this brings some challenges to our evaluation systems.

Original transcript

[0:13] Hi, everyone. [0:16] Welcome I'm very excited to kick off the session today, [0:21] where we'll talk about how to build a reliable agent [0:25] from prototype to production. [0:28] My name is alibek. [0:30] I'm an engineering manager at booking.com [0:33] leading the AI platform team. [0:35] I'll be sharing the stage with amazing Googlers [0:37] and my colleague Maria. [0:39] They'll introduce themselves later. [0:43] So we've been partnering with Google for many years. [0:48] And this isn't just a one off project. [0:52] It's a battle tested partnership that [0:55] is built on deep trust and technology [0:58] that actually delivers. [1:00] And beyond our partner performance marketing [1:03] partnership with Google. [1:05] We're also harnessing the power of Gemini and Vertex AI [1:09] to build a generic use cases that unlock smarter and more [1:13] automated customer experiences. [1:17] But let me first give you a little bit more introduction [1:20] about our company first. [1:22] So this is our mission. [1:24] We want to make it easier for everyone [1:27] to experience the world. [1:29] And this is something that we ourselves truly believe in. [1:33] It is vital because connecting people with destinations [1:38] is something that fosters this cultural understanding [1:42] and personal growth. [1:43] Because we rediscover ourselves what we like, what we love, [1:46] what we don't know. [1:48] And in fact, many of you, if not all, [1:51] traveled across countries and maybe even continents [1:55] to experience this very conference in person, [1:59] embodying that core mission. [2:03] Can I have a quick show of hands. [2:04] How many of you today traveled internationally. [2:08] Wow that's almost the whole room. [2:12] And how many of you have used booking.com app [2:15] to plan your trip. [2:18] That's exciting. [2:19] I love to see so many hands. [2:21] For those of you who did not raise hands, but maybe [2:25] for the rest as well, I wanted to give you [2:27] a sense of how massive our engine is. [2:30] So let me dive into some of the stats about our platform. [2:36] So we are not just a booking site. [2:40] It's a massive online travel platform. [2:43] Since 2010, we've facilitated over 6.8 billion guest arrivals [2:52] in different accommodations. [2:55] We are present globally, so it's a truly international company. [2:59] We cover 175,000 destinations worldwide, [3:04] and we have offices across 140. [3:08] We have 140 offices across 70 countries with $7,000 employees [3:14] in our headquarters in Amsterdam, Netherlands. [3:17] But the real story here is that we are multi vertical company. [3:24] We've evolved beyond hotels, accommodation, transport, [3:28] attractions, and we are fully integrated [3:32] multi vertical ecosystem. [3:37] But now let's talk about AI. [3:40] So booking.com is not new to AI. [3:42] We've been running predictive machine learning models [3:45] for more than 10 years already. [3:48] Today we run hundreds of models in production [3:51] that generate massive traffic of around half a million requests [3:56] per second, with roughly 50 millisecond at 999. [4:02] But today, we want to focus specifically on the AI. [4:05] So let me give you a glimpse of our AI, AI [4:09] ecosystem and to understand where we're going. [4:14] Let's have a look how our ecosystem matured. [4:18] So first, we started experimenting in 2023. [4:23] Since the ChatGPT moment, you can see the phase 0. [4:27] We had a very simple wrapper kind of application. [4:31] This was our foot in the door in the industry. [4:35] And by early 2024, we've revamped our core stack [4:40] and begun integrating our in-house models by and evolving [4:45] into multi-provider environment. [4:49] But the real shift happened when we moved to a single gateway [4:54] where our internal users can access any model. [4:58] And not only that, but also with a critical addition [5:01] of built in governance and usage monitoring. [5:08] And today we have arrived at comprehensive AI ecosystem. [5:11] You can see we have a lot of capabilities listed here. [5:14] I'm not going to dive into each all of them. [5:17] But we're not just accessing models anywhere. [5:20] We're managing the entire lifecycle [5:23] from advanced evaluations, safety guardrails, [5:25] and production ready agentic capabilities. [5:29] We now have integrated solution that [5:31] allows us scale multiple use cases very quickly. [5:37] And with that, I wanted to hand over to my colleague [5:40] Maria to dive deeper into particular use cases. [5:47] Thank you. [5:49] Hi, everyone. [5:50] I'm Maria, I'm a software engineer also [5:52] in the AI platform at booking.com, [5:54] and I'll build on what alibek just shared [5:56] by showing you a few use cases with Gemini model. [6:00] So the first, I want to focus on is called Know before you go. [6:04] It's actually our first customer facing agentic solution, [6:07] built as a proof of concept together [6:09] with the folks at Google. [6:11] As you may waiting for your trip can be exciting, [6:14] but the planning phase. [6:15] Sometimes it's a bit frustrating. [6:17] It's time consuming. [6:18] So with this project, what we really [6:20] want to do is to bridge the gap between the time of booking [6:23] and when you actually go on the trip. [6:25] So, for example, you can plan how to get from the airport [6:29] to your accommodation, or you can create itineraries [6:31] based on your group, your interest, your constraints. [6:35] This was achieved by leveraging Google Maps and search grounding [6:39] under the hood, and it really shows [6:41] how reasoning and external knowledge [6:43] come together to basically form a cohesive experience [6:47] for the traveler. [6:49] Moving on now to an internal use case. [6:53] It's called big bot. [6:54] It's our internal data registry chatbot. [6:57] Users can ask internal users, of course, [7:00] can ask general questions about the platform. [7:03] They can search for data assets, which [7:04] we know is important in such a big company [7:06] where you have a lot of information. [7:08] And also they can create workflows from natural language. [7:12] The point here is that we're not just [7:14] leveraging Gemini for our end users, [7:16] but also using it internally to enhance developer productivity. [7:22] Now moving on to partner analytics. [7:25] This is a host facing use case. [7:28] So on the partner platform the users [7:30] can go into extract traveler pain points. [7:34] So it's not just we don't just leverage [7:37] LLMs for text generation. [7:40] We use it for signal extraction. [7:42] So a partner can go in and double check. [7:45] Basically extract from comments and reviews. [7:49] The common themes. [7:50] So, for example, they can see that a lot of users, let's say, [7:53] complain about the pool or the breakfast variety. [7:56] And that way they know what to really focus on [7:59] and what matters most for their customers. [8:02] The final use case I want to present [8:04] is focused on multimodality. [8:06] I really love this one. [8:07] It's called rails to reality. [8:09] It takes short form video inspiration [8:11] to real itineraries on the booking.com app. [8:15] And we know that, travel ideas often [8:19] stem from visual inspiration. [8:21] It's not just a proper search query. [8:23] Exactly what you're going for. [8:24] So building an itinerary in booking really [8:27] helps you plan your trip better. [8:29] Essentially, Gemini is a perfect fit for this [8:32] because you can pass YouTube URLs directly to the model [8:36] and it enhances the context of the query. [8:38] And I think, a great example of how [8:41] multimodal I can be super relevant for travel. [8:45] So I've shared a few use cases, but of course, these [8:47] didn't come without their challenges and their learnings. [8:51] All of these examples show us one important thing. [8:54] It's not just about choosing the right model, [8:56] it's about how we serve the LLMs and make [8:59] them easy to use across our organization. [9:03] Most of you may already be familiar, [9:04] may already be familiar with on demand LLMs serving, [9:07] but we found that there are multiple options [9:09] to serve LLMs on Vertex AI. [9:11] We found out about this because we've [9:14] gone through a lot of service tiers [9:16] and a lot of experimentation. [9:18] Essentially, one size does not fit all [9:20] when it comes to serving tiers. [9:22] Initially we were using just on demand, [9:24] so the default Vertex AI serving option. [9:29] But for before you go. [9:30] Our first project, we quickly found [9:32] that it wasn't really fast or scalable enough. [9:35] So we switched to provisioned throughput, [9:37] which is a reserved capacity of tokens per minute. [9:41] But that also isn't a bulletproof solution. [9:43] So then we started looking into priority processing [9:47] for multimodal use cases, for example, real estate reality. [9:52] The real takeaway has been that we [9:54] are trying to match the serving strategy with the production [9:57] need, rather than just assume that a default is [9:59] going to do the trick. [10:01] And the second learning I want to focus on [10:03] is about knowledge sharing and visibility. [10:05] So we are a platform team. [10:07] That means we don't develop these use cases ourselves. [10:10] It's usually self-service or at most we support. [10:13] So it means that building the platform [10:14] is only half the job for us. [10:16] The other half is helping users understand what's available, [10:21] when to use which tool, and how to get [10:22] started in a way that's really accessible and relevant to them. [10:26] So we tailor a lot of content, always try [10:28] to keep it up to date, keep coming up [10:30] with hackathons, provider specific information, [10:33] with the goal to really help people move from experimentation [10:36] to production. [10:38] So if I had to summarize our journey in one sentence, [10:40] it would be this booking.com has been moving [10:44] from isolated experiments to a governed multi-provider [10:48] production ready. [10:48] AI ecosystem. [10:49] And Google is a very important part of that journey, [10:52] especially as we venture further into agentic solutions. [10:56] Thank you. [10:56] Now I'll hand over to manasa and naaz, [10:58] who will share four principles for deploying agents [11:02] into production on Google Cloud. [11:13] Most of us are familiar with building agents. [11:16] However, as we're building them, some natural questions [11:20] arise, how do we ensure the quality of our agents [11:24] are up to our quality standards. [11:26] How do I properly govern my tools and agents [11:30] as I'm building more and more of them. [11:33] And as I am following this journey with my agents, [11:36] how do I make sure I keep them secure [11:38] while they're running in production. [11:40] Today, we are going to cover four main topics in the next 30 [11:44] minutes to help you understand how you [11:46] can go about these challenges. [11:48] First, we'll dive into how do you [11:50] evaluate your agents before you put them into production [11:54] utilizing various techniques. [11:56] Then we're going to give some tips on how you can move [11:59] from code to Cloud quickly. [12:02] We'll then continue with how do you [12:04] have agent identity at deployment time, [12:08] and how do you keep your agents secure in production. [12:12] And then we'll finalize with observability and auditability [12:16] of your agents as you're running them in production. [12:19] You might run into issues that you would like to debug, [12:21] and we'll look into how you can do that effectively. [12:25] Before I begin, I would like to introduce you [12:27] to my colleague manasa kandula, who [12:29] is an application modernization application modernization [12:32] specialist. [12:33] And my name is Barak and I'm an AI specialist. [12:36] And we are both based in Google Amsterdam. [12:40] So we are going to be utilizing a sample demo. [12:43] Coincidentally, this is going to be in travel domain inspired [12:48] from my colleagues from booking, and we'll [12:51] have three main agents in our sample demo. [12:54] The main one will be the London travel concierge. [12:57] This is the main agent that will interact [13:00] with our users, who will give travel itinerary [13:03] recommendations to them. [13:04] It will speak via to with the ticket agent and activities [13:08] agent. [13:09] It will have access to whether MCP server as long as weather [13:13] is very unpredictable, and it will also have its own London [13:16] data MCP server, where it's going to have access [13:20] to proprietary data that resides in Cloud SQL that belongs [13:24] to our company, so that we can give proper recommendations [13:28] to our end users. [13:30] We'll also utilize Agent Engine and Model Armor and more on that [13:33] later. [13:35] So let's go ahead and get started with how [13:37] we can evaluate agents. [13:40] Agents are non-deterministic and this brings some challenges [13:44] to our evaluation systems. [13:46] We can mainly split our methods of evaluating them [13:50] into two components. [13:52] First is response evaluation, where [13:55] we disregard everything that happens within an agent [13:58] and only take a look at the response. [14:00] This is very similar to how you do any AI model evaluation. [14:04] And the second is trajectory evaluation. [14:07] This is unique to agents specifically [14:10] where we look into inner workings of an agent [14:13] and look into which tools did it call with which parameters [14:18] and which agents it collaborated. [14:20] And then we evaluate our golden goal, which [14:24] is our evaluation set against how the agent performs [14:27] and come up to certain decisions. [14:30] Now, I imagine all of you are familiar with human evolution. [14:33] This is what we did four years ago. [14:35] And any person who used an LLM probably has done this. [14:40] You take a look at your response and you [14:41] decide whether if it was good enough or not. [14:44] Ideally, if you're doing it professionally, [14:46] you have certain criteria that you need [14:48] to evaluate the agent against. [14:49] Well, this has some obvious drawbacks. [14:52] It's very subjective. [14:53] It's time consuming and expensive. [14:55] So this brought us to using LLM as Judge. [15:00] This is a commonly used technique [15:02] that was adopted by the industry, where [15:04] you have a secondary LLM that evaluates [15:07] the outputs from the first LLM. [15:09] And this is something that we mainly [15:11] use for response evaluation. [15:13] And it's still pretty valid for evaluating agents. [15:16] Now the advantage of using LLM as Judge is very scalable. [15:21] It's efficient and consistent. [15:24] While it's not perfect, it's not very [15:26] suitable for complex scenarios. [15:28] For example code modernization. [15:30] It does not work very well with LLM as Judge, [15:33] and it's limited by the LLMs capabilities. [15:35] Of course, we can utilize LLMs judge integrate it [15:40] in automated metrics. [15:41] So automated metrics for those of you [15:44] who are AI practitioners probably heard [15:46] about the metrics like IMC, mean absolute error and so on. [15:50] These are the automated metrics designed for agents [15:53] specifically, some of which we inspire from AI evaluation [15:58] like hallucination, groundedness, coherence scores [16:03] that also utilizes LLMs judge. [16:05] But you can also have custom automated metrics [16:08] integrated into your systems. [16:11] Now what I'm mainly going to focus on here [16:14] is the trajectory automated metrics, [16:16] because that is only viable for agents themselves. [16:21] Our agents are built on SDK in our demo. [16:24] So I'm going to walk you through how that works on SDK. [16:28] So when you want to do a trajectory evaluation, [16:31] you decide on which metrics that you would like to look into. [16:35] Let's walk over the first street that is the most commonly used. [16:39] So you can look into exact match. [16:41] So you come up with a golden test [16:44] sets on deciding what is the most ideal path. [16:48] My agent should follow that includes the tools that [16:52] includes the tool, input parameters, and the order [16:55] of those tools that is supposed to be called for the agent [16:58] to come up with a certain output. [17:00] You don't care about the output, but the inner workings [17:03] need to be perfect. [17:05] That is the exact match evaluation. [17:07] So this is specifically relevant. [17:09] For example, our ticket agent, where it's important [17:12] that we ticket the customer correctly [17:14] with the right discount, with the right price and so on. [17:18] Then we have in order match. [17:21] When we do in order match given our agents [17:23] are not deterministic. [17:25] It's OK if it calls some other tools. [17:28] Let's say it decided to call Google Search tool [17:31] when I didn't have that in my golden test set. [17:34] It increased my latency. [17:36] However, it did not change the response [17:38] because it decided it accidentally called it. [17:40] It was not necessary and it just moved on. [17:43] I will still get a correct response [17:45] in order match so you can detour, [17:48] but as long as you get to the right destination, [17:50] meaning you call the right tools, that's OK. [17:53] Then we have any order match. [17:55] This is when you don't really mind. [17:57] As long as the right tools are being called, [18:00] the order is not really important for you. [18:03] Think of us giving recommendations about London. [18:06] You don't really care with which neighborhood [18:08] you start as long as you give the correct recommendations [18:11] to your user. [18:13] So the advantage of automated metrics [18:16] is that they are very scalable and their objective, [18:20] but no methodology is perfect. [18:23] It's difficult to measure when it comes to creativity [18:27] and a subject of cheating, especially [18:29] if you pick the wrong metric for your use case. [18:32] Now I'll show you a demo on how it looks in our sample. [18:40] So here. [18:44] We have the implementation for loading the configuration [18:48] of our agent on SDK. [18:49] So the configuration are the automated metrics [18:51] that we would like to evaluate. [18:53] Here I have the tool trajectory the hallucination the safety [18:56] and so on. [18:57] So that includes the trajectory and the response evaluation. [19:00] Then I am loading an evaluation data set. [19:03] I created this using the SDK user interface [19:06] by having conversations with the agent, [19:08] and I save the conversations that I'm [19:10] happy with as my golden test set. [19:13] You see, it's an adjacent format. [19:15] Later we will run a simulation. [19:18] So we have the input prompts that we prompted our agents to. [19:21] And the agent has access to Golden results. [19:25] We are going to rerun this agent and see which [19:27] results it will come up with. [19:29] Outfit and based on those results, [19:31] we are going to calculate each metric and see how it performs [19:34] and save the results on BigQuery. [19:36] So here, you see in one conversation at every turn, what [19:41] was the input prompt. [19:43] What was the expected response. [19:45] What was the actual response. [19:47] And now you see all the metric results [19:49] saved in a granular level. [19:51] And on the right hand side, we have the aggregated metrics [19:54] for the whole conversation. [19:56] So by looking at that can decide whether if your agents [20:00] are ready to go to production. [20:02] And once you're there it is time to get to Cloud. [20:05] And for that, I will hand over to Vanessa. [20:10] We're going to be switching back and forth often, [20:12] so don't need to clap every time, but we appreciate it. [20:16] So the next step that we cared about [20:18] is making sure that we were able to increase our developer [20:21] velocity. [20:22] By that I mean we want to specifically make sure [20:24] our developers are not burdened by all [20:27] of the bottlenecks that come with taking an agent [20:29] from prototype to production. [20:31] I'll explain what that means in a second. [20:33] So what we've noticed over the last year [20:36] is with the help of code assistance tools, [20:39] people are able to build agents relatively quickly and easily [20:42] in their own local environment. [20:43] But what we see is that developers generally [20:46] get slowed down when they have to take this agent that they've [20:49] been developing locally, running it as a prototype, [20:51] and then take it into a secure Cloud architecture. [20:54] And that obviously makes sense because in the advent of the AI [20:58] era, the rules have changed, the runtimes have changed, [21:01] architectural principles have changed. [21:03] So this adds a lot of cognitive complexity. [21:07] So in order to solve that, we use something [21:09] called Gemini app Cloud Assist. [21:11] And Gemini Cloud Assist is something [21:12] you can think of as your AI architect for Google Cloud. [21:17] So what it does is that it has a really good understanding [21:19] of the resources and infrastructure that Google [21:22] Cloud has available, and it can analyze your code base [21:25] and your intent to be able to translate that into architecture [21:29] that lands well on Google Cloud. [21:31] And if you paid attention to the architecture of our London [21:33] concierge app, there was a PostgreSQL database [21:36] with a vector extension over there. [21:38] So in order to bring that into Cloud SQL onto Google Cloud, [21:42] the agent would suggest using Cloud SQL with Postgres. [21:46] And also in order to make sure it's secure, [21:49] it will also likely suggest that you [21:50] need to make it a private instance along with Gemini Cloud [21:55] Assist. [21:55] We also have the application Design Center, [21:57] and you can think of the application [21:59] Design Center as a place where you [22:01] can store your architectural templates [22:03] and deploy your architectural templates from there. [22:06] So you can work with Gemini Cloud Assist and application [22:09] Design Center together. [22:10] And one more advantage of using ADC. [22:12] I'm going to call it ADC from now on, [22:14] is that it also gives you out of the box metrics from day one. [22:18] And we'll take a look at that a little bit down the line. [22:21] But now we were talking about developers [22:23] and bringing their code to Cloud. [22:26] What would such a user journey look like. [22:28] What would a developer's journey look like. [22:30] Let's say, for instance, you're using Gemini CLI [22:32] as your code assistance tool. [22:34] You would use that to be able to help you generate code. [22:37] And along with the Gemini Cloud Assist and ADC MCP servers, [22:42] you can have them analyze your code base [22:44] figure out the right architecture for your code base [22:47] and also land that architecture on Google Cloud. [22:50] And in that process, if you have any issues, [22:52] you can use these two MCP tools to be [22:54] able to help you troubleshoot. [22:56] So we'll take a look at a recorded demo [22:58] to see what that looks like. [23:01] So the first thing you see is an Ide with our agents code base. [23:06] And first we check to see if our MCP servers are installed, [23:10] MCP tools are installed. [23:12] We have the ADC and the Gemini Cloud Assist MCP [23:14] servers installed. [23:15] And then we make sure that we have a skill installed [23:18] that is able to coordinate with these two [23:20] and make them work to do what you need. [23:23] Now that we have the prerequisites, [23:25] we make sure we have. [23:27] We ask to design a Cloud architecture from our code base. [23:33] So the agent looks through your entire code base, [23:36] searches through it, tries to understand the components of it, [23:39] and tries to figure out how they're connected and related [23:42] to one another. [23:42] I didn't give it any other instructions [23:44] except that one sentence. [23:46] So then after it has a basic understanding, [23:48] it calls Gemini app Cloud Assist MCP [23:50] tool to take this understanding and translate that [23:53] into an architecture. [23:57] So what you see now is that after it's translated it, [23:59] it takes the Terraform version of those templates and downloads [24:03] that directly onto your code base. [24:05] And now it also deploys that template [24:08] onto ADC in the same step. [24:11] So now we can look at the template in a visual design [24:13] Canvas. [24:14] And we can also explore the configuration [24:16] of the different components inside the template [24:18] and how they're related to one another. [24:20] Now that we approve this template, I took a look at it. [24:23] It looked good, I approved it, and now [24:26] the agents are going to deploy this architecture onto Google [24:31] Cloud. [24:32] This is a time consuming process. [24:34] It takes about 10, 15 minutes. [24:35] Obviously it's fast forwarded. [24:38] And it keeps checking to see if there are any issues [24:41] and checks the deployment status until it's complete. [24:45] Now that it is deployed we can actually [24:48] take a look at that application deployed using the ADC as well. [24:53] So we can take a look at the deployment logs in Cloud Build. [24:58] You can see if there's any issues. [24:59] Everything looks fine. [25:00] I went through it very, very quickly. [25:02] And then we can take a look at the outputs of this deployment. [25:06] And the main thing I'm interested in [25:07] is the load balancer IP address, with which [25:10] we can take a look at which we can interact with our travel [25:13] agent right there. [25:16] It seems to work I think. [25:18] Yeah it works. [25:19] So one step further. [25:22] So what I hope to show you with this demo is [25:24] that if you have a process where you have an agent [25:27] or you have an agent application, [25:28] you built it locally, it runs well locally. [25:31] And if you want to bring it to the Cloud, [25:33] it usually takes a developer days simply [25:35] because you have to figure out what [25:37] the right matching component on Google Cloud is. [25:39] How do you configure it using security best practices. [25:42] How do you connect two things together. [25:44] And this is something that takes many days. [25:46] And we managed to do this in a matter of about half an hour [25:48] using these two tools. [25:52] Now we have a bunch of agents and a bunch of servers [25:56] running in our environment. [25:58] So how do you make sure you can use track [26:01] what agents and MCP servers you have running, [26:03] and how do you also make sure that you are reusing tools [26:06] that you've deployed and not building [26:08] new ones that are very similar to the ones [26:10] that you already have. [26:12] So for this, we have something called the registry. [26:14] You can think of this registry as a source of truth [26:17] for your agentic ecosystem, so you can use this [26:21] as a catalog to store your MCP servers and your agents, [26:25] and use this as a source of truth within your organization. [26:27] So one of the advantages it has, one of the main advantages [26:30] is that it helps discovery across your organization. [26:33] So you can see what agents, your teams, other teams [26:36] and your organizations have built. [26:38] And you can also reuse those same tools and agents [26:40] and reduce double work in engineering [26:44] alongside agent registry. [26:46] We also have something called the agent gateway. [26:48] While discovery is an important problem to solve [26:51] and you can do that with the registry, [26:53] you also need something like governance. [26:56] So with teams building lots of agents and tools, [26:59] you have a proliferation of them and you [27:01] want to make sure not every agent [27:03] that you have is able to talk to every other agent [27:06] in your system. [27:07] So for instance, in our travel app, [27:09] we would like our travel app to be [27:10] able to talk to for example, the ticketing agent, [27:14] because we want it to be able to use it to buy tickets [27:16] for our customers, but we don't want [27:19] to talk to our internal HR agent because that [27:21] would be really bad. [27:22] So this is something that typically, [27:24] if you leave this in the land of the developer of developers, [27:27] it adds a lot of burden and it takes [27:29] a lot of engineering cycles to be able to implement that [27:32] at the developer level. [27:34] So we want to pull this up to the platform level. [27:36] And this is what agent gateway does it. [27:38] You can think of it as the governance layer [27:40] for your agents and MCP tools. [27:44] And I wish I about this for the next hour. [27:47] I can't but there's lots of great demos on the floor [27:50] and lots of talks that we'll talk about gateway and agent [27:54] registry, so please be sure to check them out. [27:57] OK, with that, Nas is going to be talking about safety. [28:02] So how do we ensure our agents stay secure while we run them [28:06] in production. [28:08] For that, we are going to look into three main components. [28:12] We are going to get started with identity. [28:15] Then we'll look into the most common threats [28:18] Gen AI applications face while running live, [28:21] and how we can tackle those. [28:23] And finally, we'll wrap up with talking about observability [28:27] and auditability. [28:29] When you put your agent to production [28:31] and if something goes wrong, how you're [28:32] going to work backwards from it to fix the issue that you [28:35] have in place. [28:38] So let's get started with identity. [28:40] We just released it today. [28:43] Wear this is a solution where your agent will [28:47] have a unique IAM at deployment time, [28:51] and you don't have to manage it. [28:53] It will have these privileges access provided by you. [28:57] The advantage of having such solution, [28:59] in the case of our demo, say that our concierge agents [29:03] wants to purchase a ticket. [29:05] It will do so with its own AI, with its own IAM, so [29:10] its own authentication. [29:12] And when something goes wrong and you look back at your logs, [29:15] you're going to see not just a service, that access [29:19] to that agent, but you'll see exactly [29:21] with which agent, with which version that access was made. [29:25] And you'll be able to have an irrefutable audit trail to trace [29:29] back exactly what happened. [29:33] For example, here on the top, we have our concierge agent [29:38] that has a regular service role there. [29:41] It means you will have to manage the whole lifecycle [29:44] of this role. [29:45] And in this second and third agent, [29:47] we defined our agents with identity. [29:50] So at runtime they were assigned an identity. [29:53] You don't have to worry about it. [29:54] If it's not running in deployment then they disappear. [29:58] And then as you're running them in production, [30:01] you'll have access to all of these logs and behaviors. [30:05] Moving forward, how do we make sure [30:07] we prevent bad actors accessing and compromising our agents. [30:13] There are many reasons why you might want to secure your LLM [30:17] applications. [30:18] The common ones would be not wanting [30:22] to leak PII information of your customers. [30:26] Protecting your brand reputation via not sharing information [30:30] about your competitors. [30:32] You don't want to release biased information. [30:37] And also information that is not relevant to your tone of voice [30:41] as a company. [30:43] So for that, we have Model Armor. [30:46] Model Armor resides between the user and your LLM [30:50] to prevent all of these from happening. [30:52] So these are things that you would prompt yourself [30:56] over a long period of time to prevent from happening. [30:59] And you would need to do rigorous tests. [31:02] But if you were to place Model Armor in between, [31:04] then most of these things are tackled on behalf of you. [31:07] So let's see how that works in our demo. [31:11] So here we have the UI of our application. [31:14] We are greeting our agent and then [31:17] we are going to attempt to do data exfiltration. [31:22] We are trying to retrieve the credit card [31:24] information from a user. [31:26] This is detected by our Model Armor [31:29] and our agent is not responding to that. [31:32] Then we'll try to do prompt injection. [31:37] Again detected by Model Armor. [31:54] We'll then have a jailbreak attempt. [31:56] And you'll see we have many creative attacks [31:58] that we can have towards our application. [32:07] Now we are trying to do prompt promptly catch attack [32:09] by trying to retrieve the system instruction from our agent. [32:18] Well, you see the attacker getting frustrated. [32:29] Violating the safety filter. [32:35] And then finally trying a malicious URL [32:38] filter to see if our agent will actually try [32:43] to go ahead and retrieve it. [32:48] So now, to prove you that we did not block every prompt, [32:53] we are making our agent work as the way it's supposed to. [32:58] But we're not going to give up. [33:05] Well, let's see if it can do sensitive data detection. [33:12] Well, this is something that would have taken us [33:16] a lot of prompt engineering to get to while with Model Armor, [33:20] you can just enable it between your user and your language [33:22] model to have all of this in place. [33:25] Given that we talked about identity and agent security [33:30] at runtime, we'll now move on to talking [33:32] about how do we debug issues as they [33:34] arise while running in production hand [33:37] over back to Manasseh. [33:38] So I'm going to start with a question. [33:40] How many of you have had applications run in production. [33:44] How many of you have production applications. [33:47] Almost all of you. [33:49] How many of you have had production applications [33:51] that have had no issues. [33:54] Not a single hand up. [33:55] That proves my point. [33:57] So the thing is, the crux of the story [33:59] is how do you figure out when something goes wrong if you [34:02] don't have observability. [34:04] And one of the things that frustrates me [34:06] is that observability is often referred to as a day two [34:10] operation. [34:11] And there's reason behind this. [34:13] It's mainly because it's not trivial. [34:15] So as an application developer, in order [34:16] to be able to get observability signals, [34:19] you need to instrument your code base. [34:21] Next, as a platform engineer, in order [34:23] to be able to get observability signals across your platform, [34:26] you have to pull these out of your Cloud vendor. [34:28] You have to build beautiful dashboards that [34:30] integrate with your application signals [34:32] so that they actually make sense to you [34:33] and give you information that is valuable. [34:35] So all of this takes a lot of time and engineering effort. [34:38] And that's why it's never done on day 0. [34:41] It's always done as an afterthought, not as [34:43] an afterthought, but afterwards. [34:45] But we actually managed to solve this problem for ourselves [34:48] by doing two things. [34:50] Firstly, we used SDK with a Google Cloud platform [34:54] with which we were able to get application observability out [34:58] of the box with just a click of a single button button. [35:01] And then if you remember earlier, [35:03] I talked about application Design Center ADC. [35:06] We used ADC to deploy our entire application. [35:09] You saw that. [35:09] And that gave us platform dashboards out of the box [35:12] as well. [35:13] And what ADC did on top of that is instead of just giving us [35:16] dashboards and signals for independent resources [35:19] across the Cloud project, it tied them together [35:22] as something that belonged to a single application. [35:25] So we were able to operate the application [35:27] as a whole instead of signals from the Postgres database, [35:30] instead of signals from just that Cloud Run instance, [35:32] et cetera. [35:33] So with that, we were able to simplify our journey pretty [35:36] quickly. [35:38] And alongside with our agent app, [35:40] we also have the Agent app dashboard, [35:42] which has additional metrics that are useful to anybody [35:45] who's running AI apps. [35:46] So you're able to look at things like the token count [35:49] usage, the latency with respect to certain models, et cetera. [35:53] And all of these come out of the box. [35:56] All right. [35:57] Now that we have our dashboards ready, [35:59] how do we actually diagnose issues. [36:02] Once the signals that there's an issue occurs. [36:06] So we at Google Cloud have been using [36:08] I've been a big supporter of OpenTelemetry [36:10] for a very long time, and for those who are not aware of it, [36:13] OpenTelemetry is a standard that is open and is [36:16] used by multiple vendors across the globe, [36:19] and it is there to help you build telemetry pipelines [36:22] independent of vendors. [36:24] So it adds a lot of interoperability [36:26] between different tools, et cetera, and what [36:28] the OSS community and Google Cloud has done together [36:31] is extended the OpenTelemetry protocol to also include signals [36:37] from JNI sources. [36:39] So before when this came up, obviously there was no JNI [36:42] and all the telemetry signals were [36:44] based on normal applications. [36:46] But of course, if you're building and operating [36:48] an JNI app, you need extra signals [36:50] about the model, et cetera that OpenTelemetry gives for you. [36:54] Now let's take a look. [36:56] Sorry, I'm going to go back. [36:58] I need to go to a live demo, please. [37:11] It's worth it, I promise. [37:12] It just takes a couple of seconds to come up. [37:21] Do I need to do something here. [37:28] OK I might have to skip this one if somebody can come and help me [37:38] for the next demo. [37:39] I'll come back to this in a second. [37:41] I was about to show you what one of these traces for JNI [37:43] look like. [37:44] We'll come back to it at the end. [37:46] So I promise you I'll show you this. [37:47] So we only do one switch at the end to a live demo. [37:50] So right now we have our application instrumented. [37:54] It runs beautifully. [37:55] There are no errors that I seem to be picking up, [37:58] but there's still a very important question to answer. [38:00] The thing is, the question here is, how do you [38:02] know if your agents are actually delivering value [38:04] to your end customer. [38:05] Because that is the reason you're building everything. [38:08] So in order to do that, we use something called the BigQuery [38:11] analytics plugin, which comes with application Development Kit [38:15] or SDK. [38:16] So what it does is it takes all the every single conversation [38:20] that you have with your user or with the end user. [38:23] And it takes all of the metadata and the information from it [38:26] and stores it in a structured database. [38:29] And for those of you who know what BigQuery [38:31] is that you can run very powerful analyses on this data [38:35] to have help you deliver business value. [38:40] Yeah Can we go back. [38:43] Thanks this has a demo too. [38:45] So it has to put up. [38:52] Sorry what do I have to click. [38:55] Yeah Great. [38:57] So first let's take a look at the trace. [38:59] So this is a trace for one single turn [39:01] that the user had with the agent. [39:03] So what you see over here is you can see every single step [39:07] and how much time each of these steps [39:09] took before the end user got the response back. [39:12] And I also talked about the fact that it [39:14] has JNI components in it. [39:18] So we go to this traditional view of a trace. [39:20] You see there are several steps over here [39:22] where you call the LLM. [39:23] The LLM calls a tool to do search for data [39:27] London activities data, and then gives this answer back [39:30] to the AI model. [39:31] And then finally the user gets the answer. [39:33] Everything marked with JNI over here [39:36] tells you that this has JNI information in the trace. [39:39] So if we take a look at this see a whole bunch of metadata. [39:42] You can see that we're using Gemini 2.5 Flash. [39:45] The temperature that we used was 0.01. [39:48] The number of tokens I used for this specific turn, [39:50] which was about 3,658. [39:54] We need to optimize our agent. [39:56] But anyway, so you can see all of this in one space. [39:58] So you can have this for every single conversation. [40:01] And this is very useful information [40:03] to have when you're debugging. [40:05] Now, if we went back to BigQuery, [40:06] which wear which also captures our user's interaction [40:10] with our model, what you see over here [40:14] is data for one specific day. [40:17] And yeah so it gives you the timestamp [40:20] gives you the event for every single conversation. [40:23] It gives you the attributes. [40:27] And if I'm not mistaken, it might have hidden that column. [40:30] But you also see the content of the conversation. [40:33] And now with this, I can run very interesting queries. [40:36] So for instance I want to know for this particular day what [40:39] is the user sentiment are how happy [40:42] is each of my user with every single conversation. [40:45] So having taken this data, I run this analysis [40:47] and you see this is sorted descending. [40:50] So that's why you see very happy users in the very top. [40:53] You see that this person was very happy with what we gave. [40:56] Yes, day 3 is absolutely perfect. [40:58] Those are exactly the kind of things I want to do, [41:00] blah blah blah. [41:01] So that conversation is rated five out of five. [41:03] But if we go way down, let's go here instead. [41:06] And this is easier to go. [41:08] This is exported to a Google Sheet. [41:12] So this is an unhappy user. [41:14] It says this is really not working out [41:15] so you can't find anything blah blah blah. [41:18] So that gives you a 1 out of five. [41:20] So just by using this data and running a SQL query, [41:22] I was able to analyze the sentiment of every single user [41:25] conversation. [41:26] And that is very useful information for me. [41:28] And lastly, I can also have a dashboard [41:31] that I built to see the number of tokens [41:33] that I've used on a specific day on average, [41:35] and you see where these tokens are being used. [41:37] And if I wanted to and if I could write good SQL queries, [41:40] I would have drilled down even further [41:41] and looked at more data about where these tokens are [41:45] being used up and which type of interaction, et cetera. [41:48] We can go back to the slides now. [41:56] Presentation all right. [42:02] So we looked at both of these. [42:03] And just to end this, we talked about a lot of different steps [42:08] that we went through in the last 30 minutes [42:10] about how to get our agents into production. [42:13] And let's say you do this for one agent, [42:15] it takes a little bit of effort. [42:16] Let's say you want to do this for 10 agents, [42:18] and let's say you want to do this for 100 agents. [42:20] Are you going to be able to scale the steps that we talked [42:23] about one by one for all your agency [42:27] and organization, scaling manual oversight is really difficult. [42:31] So in order to solve that problem, [42:32] we go back to ADC again. [42:34] Earlier in the demo, we saw that we [42:36] used ADC to be able to take our code base, [42:40] analyze it, and generate a custom template for our code [42:44] directly. [42:45] But what if we have a platform team [42:46] in between that has a golden template for all agentic apps [42:49] in your organization, so they can create an A genetic template [42:53] for any genetic app that is going to be deployed [42:55] with your organization, which has all the policies, [42:58] all the guardrails, all the security requirements built in. [43:02] And then you as a developer, just [43:04] take that template using these MVP tools [43:06] and customize it to run to your app. [43:08] Then you make sure that your application, [43:12] every single application in your organization is compliant. [43:16] I'm going we have two minutes left, [43:17] so I'm going to skip to the very end. [43:19] Check out ADC. [43:20] It's a really cool tool and it really helps build in governance [43:24] at a platform level. [43:26] So with this, I hand this back to Nas. [43:30] As a recap. [43:31] We covered four main topics. [43:34] First, we talked about evaluation. [43:36] How you can do evaluation on agents [43:38] is through response evaluation or trajectory evaluation. [43:41] And we covered a few metrics that you [43:43] could cover when it comes to trajectory automated metrics. [43:47] Then we talked about how do you get [43:49] your code to Cloud efficiently. [43:51] She also covered how do you have a proper registry for your tools [43:55] and agents. [43:57] Then we talked about having an IAM for your agents specifically [44:01] at deployment time, and looked into common threads [44:04] that your agents might face and how you can tackle those. [44:08] She then concluded with how you can do observability [44:11] and auditability, and also as you're [44:14] scaling the amount of agents that you're [44:16] putting into production within your organization, [44:19] how you can utilize templates to scale more effectively rather [44:23] than doing all of these steps one by one. [44:27] Given that I see that we are almost out of time, [44:30] we will be in the corridor. [44:32] In case any of you would like to ask any questions so that we [44:36] get to understand your use cases and see [44:38] how we can help you better. [44:39] Thanks a lot for listening.