Upwork Finds AI Agents Fail Most Real Tasks Without Human Guidance

Rearchitecting Upwork for the AI Era With Upwork And The BBC — Ai4

Ai4

Autonomous AI agents succeed on only 30% to 40% of paid digital jobs on their first pass, but reach human expert parity at 1,000 times the speed when directed by human verifiers.

AI-assisted summary of Rearchitecting Upwork for the AI Era With Upwork And The BBC — original source. Check the original for full context.

Source published: 2026-08-22

Detailed notes

## Background Published on YouTube by the Ai4 channel on August 22, 2026, this fireside chat features Upwork CTO Andrew Rabinovich in conversation with BBC Senior Technology Journalist Thomas Germain. According to the description, the session explores real-world performance data from Upwork's Human + Agent Productivity Index (HAPI) and details how the platform rebuilt its infrastructure around 'Uma' to coordinate human expertise with autonomous agents.

## TL;DR Upwork CTO Andrew Rabinovich argues that autonomous AI agents remain far from replacing human knowledge workers, succeeding only 30% to 40% of the time on real-world qualitative tasks without human guidance. To bridge this gap, Upwork is transitioning into an AI work operating system powered by 'Uma', a meta-agent that decomposes client intent into atomic basis functions and orchestrates hybrid human-machine execution. The platform is also seeing the rise of 'agentic hiring,' where autonomous AI models hire human freelancers via API to act as real-time verifiers for subjective and non-deterministic decisions. Ultimately, Rabinovich rejects the lump-of-labor fallacy, asserting that embedding human experts into native agent workflows via Model Context Protocols (MCPs) will unlock backlogged demand and vastly expand global productive capacity.

## Atmosphere & style The discussion carries a measured, pragmatic tone that cuts through Silicon Valley hyperbole with hard marketplace data. Germain acts as an incisive journalistic foil, pressing on real-world labor anxieties, flawed benchmark claims, and workforce displacement, while Rabinovich responds with the steady, analytical calm of a 25-year machine learning veteran. The conversation moves from a sober reality check on the limits of next-token prediction to a grounded, reassuring vision of human-in-the-loop collaboration.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=BkAY24qUbn0&t=4s) AI is far from replacing human labor because existing models rely entirely on human data and struggle in unconstrained, non-deterministic domains. - [00:04](https://www.youtube.com/watch?v=BkAY24qUbn0&t=4s) Upwork is transitioning from a traditional talent matchmaking marketplace into an AI-native work operating system that translates client intent into orchestrated human-agent deliverables. - [02:17](https://www.youtube.com/watch?v=BkAY24qUbn0&t=137s) Uma serves as an orchestration meta-agent that breaks complex projects down into 500 to 1,000 atomic units of work, delegating each to the optimal mix of specialized agents and human freelancers. - [04:30](https://www.youtube.com/watch?v=BkAY24qUbn0&t=270s) Evaluating qualitative work such as writing and design cannot rely on traditional benchmarks, requiring real marketplace transaction data and client satisfaction as ground truth. - [07:32](https://www.youtube.com/watch?v=BkAY24qUbn0&t=452s) Upwork's Human + Agent Productivity Index (HAPI) shows that autonomous AI succeeds only 30% to 40% of the time on real jobs, but reaches expert-level parity 1,000x faster when paired with human guidance. - [10:37](https://www.youtube.com/watch?v=BkAY24qUbn0&t=637s) Autonomous agents break down on complex workflows due to multimodal prompt ambiguity, statistical next-token prediction limits, and an inability to manage multi-turn task dependencies. - [10:37](https://www.youtube.com/watch?v=BkAY24qUbn0&t=637s) Determining completion for qualitative digital work is an AI-complete problem that lacks the objective stopping conditions found in deterministic domains like mathematics or chess. - [17:35](https://www.youtube.com/watch?v=BkAY24qUbn0&t=1055s) AI agents are transitioning into sources of labor demand through 'agentic hiring,' programmatically hiring human specialists via API to resolve subjective ambiguity and verify quality in real time. - [20:23](https://www.youtube.com/watch?v=BkAY24qUbn0&t=1223s) Model Context Protocols (MCPs) enable millions of freelance specialists to be surfaced directly inside native productivity environments like ChatGPT, Slack, and Claude as prompt extensions. - [20:23](https://www.youtube.com/watch?v=BkAY24qUbn0&t=1223s) AI agent collaboration overcomes the lump-of-labor fallacy by removing headcount bottlenecks and drastically expanding total global work capacity for unexecuted initiatives.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=BkAY24qUbn0&t=4s) Rethinking Upwork: From Talent Matchmaker to Work Operating System

Thomas Germain opens the conversation by addressing the elephant in the room: widespread conference buzz claiming AI will surpass human intelligence and displace all human labor, which poses an existential threat to freelance marketplaces. Andrew Rabinovich, drawing on 25 years of machine learning experience, firmly pushes back, asserting that we are "nowhere close" to replacing humans. AI has simply learned to perform tasks "faster, cheaper, and better"—specifically for workflows humans have already mastered. Because all AI models rely entirely on human-generated or human-collected training data, they struggle profoundly in unconstrained, non-deterministic domains where established playbooks do not exist.

Rabinovich outlines how Upwork is reinventing its core business model. Historically, the platform operated as a matchmaking marketplace connecting clients to freelance talent for digital work. Today, Upwork is pivoting into a full "work operating system" or work delivery platform. In this paradigm, clients no longer need to figure out how to execute a project or specify job roles; they merely express their high-level intent. Upwork’s overarching AI system—Uma—then analyzes the request and orchestrates a hybrid mix of machines and human experts with human oversight to produce the finished deliverables.

```mermaid flowchart TD accTitle: Evolution of Upwork Architecture accDescription: Comparison between Upwork's legacy talent marketplace model and its modern AI-native work operating system.

subgraph Legacy["Legacy Marketplace Model"] L1["Client specifies role/skills"] --> L2["Upwork matches client to freelancer"] L2 --> L3["Freelancer manually produces deliverable"] end

subgraph Modern["Work Operating System Model"] M1["Client defines high-level goal/intent"] --> M2["Uma Meta-Agent decomposes intent"] M2 --> M3["Match atomic units to hybrid talent (Humans + Agents)"] M3 --> M4["Human-in-the-loop oversight & quality verification"] end ```

---

[02:17](https://www.youtube.com/watch?v=BkAY24qUbn0&t=137s) Inside Uma: Intent Translation and Atomic Units of Work

Responding to Germain's questions regarding the architecture of Uma and the deliberate boundaries set for agentic workflows, Rabinovich explains that Uma stands for **Upwork's Mindful AI**. The emphasis on "mindfulness" reflects a strict design philosophy focused on delivering trustworthy, reliable, and explainable outcomes. Crucially, Uma is not designed as an autonomous worker that executes end tasks, but rather as an orchestration "meta-agent" structured across four foundational stages:

1. **Intent Understanding:** Translating high-level client objectives into actionable execution plans (e.g., when a client states *"I want my business to make more money,"* Uma translates this abstract goal rather than forcing the client to specify a software engineer, designer, or marketer). 2. **Decomposition into Basis Functions:** Drawing on 20 years of proprietary digital work data, Uma breaks down complex, exponential job varieties into 500 to 1,000 atomic units of work ("basis functions") that can be combined to assemble any project. 3. **Talent & Tool Orchestration:** Identifying and assigning the optimal combination of human freelancers and specialized AI agents to execute these individual atomic tasks in concert. 4. **Outcome Verification:** Evaluating and validating the final output against client expectations before delivery.

---

[04:30](https://www.youtube.com/watch?v=BkAY24qUbn0&t=270s) Benchmarking Qualitative Work: Ground Truth via Market Feedback

Germain points out a fundamental gap in traditional AI benchmarks: industry evaluations focus predominantly on quantitative, deterministic tasks like coding and mathematics, whereas over 90% of Upwork's marketplace consists of qualitative tasks like copywriting, design, and marketing strategy. Rabinovich explains that for deterministic tasks (such as coding or machine translation), automated verification is straightforward. However, evaluating qualitative output requires ground-truth validation derived from real-world marketplace transactions.

Borrowing from marketplace feedback loops akin to Uber or Airbnb, Upwork treats satisfied client outcomes from completed projects as ground truth. Uma is continuously trained on freelancer-generated deliverables and corresponding client acceptance data. This category-by-category learning process allows Uma to evaluate new agent-generated outputs against historical, verified work. Furthermore, Rabinovich emphasizes that standard deep learning concepts like transfer learning do not apply cleanly across disparate domains; mastering web design evaluation does not automatically translate into evaluating complex cross-functional strategies.

---

[07:32](https://www.youtube.com/watch?v=BkAY24qUbn0&t=452s) The Human + Agent Productivity Index (HAPI)

Turning to Upwork’s research, Germain highlights the **Human + Agent Productivity Index (HAPI)**. Rabinovich contrasts HAPI against standard industry benchmarks: - **Academic Benchmarks:** Static and prone to "hill climbing" (overfitting to pass LSATs or SATs, yet failing basic common-sense tests like counting how many letters 'p' are in "Google"). - **Data Labeling Vendors:** Incentivized to present models as failing with single-digit precision to encourage continued enterprise data spending. - **HAPI Methodology:** Evaluates autonomous agents against dynamically updated, real paid jobs posted on Upwork that possess genuine economic value.

The findings demonstrate that when state-of-the-art AI agents attempt real Upwork jobs autonomously, they struggle significantly on the first pass (succeeding only 30% to 40% of the time). However, when a human expert is placed in the loop—reviewing initial drafts and guiding the agent through iterative refinement—agent performance reaches parity with standalone human experts while completing tasks roughly **1,000 times faster**.

```mermaid flowchart LR accTitle: Human plus Agent Productivity Loop accDescription: Iterative feedback loop showing how human guidance brings AI agent output to expert quality at high speed.

A["Real Job Task"] --> B["AI Agent First Attempt"] B --> C{"Autonomous Success: 30-40%"} C -->|Fails / Ambiguous| D["Human Expert Review & Guidance"] D --> E["Iterative Prompt Adjustments"] E --> B C -->|Refined| F["Expert-Parity Deliverable (1,000x Faster)"] ```

---

[10:37](https://www.youtube.com/watch?v=BkAY24qUbn0&t=637s) Prompt Ambiguity, Modality Limits, and Deterministic vs. Qualitative AI

Rabinovich breaks down the primary technical bottlenecks preventing autonomous agent success: - **Multimodal Ambiguity vs. Text Bottlenecks:** Real-world work involves complex combinations of text, pixels, and audio. Translating a nuanced UX/UI paradigm or brand voice purely through text prompts creates significant ambiguity, causing autonomous models to drift off-target. - **Limits of Next-Token Prediction:** Large language models are statistical pattern-matching engines fine-tuned via RLHF (Reinforcement Learning with Human Feedback). They lack intrinsic intuition, human experience, and the tacit context needed to infer what a client truly means. - **Task Scope and Multi-Turn Complexity:** Simple, single-turn, self-contained tasks (e.g., spinning up a web server or laying out a basic page) are achievable autonomously. Once a workflow requires coordinating front-end code, back-end architecture, content, and marketing assets, generic agents break down.

When Germain compares this to recent claims by OpenAI and Anthropic regarding models solving novel mathematics theorems, Rabinovich differentiates between deterministic problem-solving and qualitative work:

Rabinovich argues that knowing when a qualitative task is truly complete is an "AI-complete problem"—and until human judgment can be mathematically formalized, human-in-the-loop collaboration remains essential.

---

[17:35](https://www.youtube.com/watch?v=BkAY24qUbn0&t=1055s) Agentic Hiring: Transforming Freelancers into Real-Time Verifiers

Addressing the economic impact on knowledge workers, Rabinovich rejects the notion that automation simply shrinks the labor pool, citing historical precedents: the invention of calculators expanded the mathematics field, and moving from punch cards to high-level code created millions of software engineering roles. As software production costs collapse, demand shifts toward bespoke, highly personalized applications.

While low-level mechanical tasks (basic logo generation, simple text translation, boilerplate summaries) are fully automated, a profound shift is occurring on Upwork: **AI agents are shifting from a source of talent supply to a major source of labor demand**.

Under this emerging paradigm of *agentic hiring*: - Frontier systems (such as ChatGPT, Claude, or Gemini) encounter subjective ambiguity or non-deterministic hurdles (e.g., clinical cross-checking, subjective recommendations, nuanced copy reviews). - The AI agent directly hires human freelancers on Upwork via API in real time. - Freelancers transition from executing monolithic, weeks-long projects to serving as expert guides and verifiers who handle high-frequency micro-decisions—potentially answering complex, context-specific queries dozens of times per minute.

Germain concludes the segment by bringing up broader anxieties in the labor market, citing recent reports in *The Guardian* on the burnout of an "infinite work week" managing autonomous tools, and corporate case studies like Klarna replacing in-house customer service teams with AI, only to later rely on gig workers paired with AI systems.

[20:23](https://www.youtube.com/watch?v=BkAY24qUbn0&t=1223s) Optimism Against a Dystopian Labor Market: MCPs and Expanding Capacity

Confronted with the unsettling specter of a "dystopian labor market," Thomas Germain jokingly invites Andrew Rabinovich to "play therapist" and articulate why he remains optimistic about what lies ahead. Rabinovich acknowledges that seismic technological shifts inevitably produce dramatic "oscillations in both directions" as hype, panic, and experimentation collide, but notes that this turbulence continues only until the field eventually settles down and a stable, long-term trajectory emerges. While familiar work habits and paradigms are being thoroughly dismantled and reassembled, Rabinovich identifies two core structural pillars that justify his positive outlook.

The first reason centers on frictionless workflow integration. Rabinovich emphasizes that future work will not force users to abandon their native work hubs; instead, human experts and intelligent agents will collaborate natively within whatever environment teams already inhabit, whether that is ChatGPT, Slack, or Claude. By leveraging Upwork’s Model Context Protocols (MCPs), the platform’s global footprint of roughly 20 million skilled professionals can be surfaced and embedded directly into automated workflows as an on-demand, real-time extension of the prompt layer.

The second pillar of Rabinovich’s optimism lies in dismantling the lump-of-labor fallacy. He argues that the advent of autonomous and assisted agents will vastly increase the sheer volume of work accomplished globally rather than reducing human opportunity. In Rabinovich's view, contemporary teams do not leave initiatives untouched because they lack ideas or ambitions; "the things that we do now is not because we don't know what else to do," he stresses, but simply "because we don't have any resources to do more." By removing the resource bottleneck, agentic tooling combined with human specialists will dramatically expand what organizations can build, leading Germain to close the fireside chat with mutual thanks as time runs out.

```mermaid flowchart LR accTitle: Upwork MCP Architecture for Embedded Human-Agent Workflows subgraph Native["Native Work Environments"] ChatGPT["ChatGPT"] Slack["Slack"] Claude["Claude & Cloud Workspaces"] end subgraph Orchestration["Interoperability Layer"] MCP["Upwork Model Context Protocol (MCP)"] end subgraph Capacity["Expanded Work Capacity"] Agents["Autonomous Agents"] Talent["20M+ Human Specialists"] end

Native --> MCP MCP --> Agents MCP --> Talent ```

- **Equilibrium following disruption**: Large-scale technological transformations initially trigger wild oscillations across markets before settling into stable, lasting operational norms. - **MCP-driven workforce integration**: Through Model Context Protocol implementations, Upwork aims to bridge its pool of roughly 20 million independent professionals directly into agent prompts and third-party productivity environments. - **Resource unblocking over displacement**: Agentic expansion unlocks latent project demand, enabling organizations to tackle backlogged roadmaps that were previously abandoned solely due to resource and headcount constraints.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=BkAY24qUbn0&t=4s) nowhere close — Andrew Rabinovich

> [00:04](https://www.youtube.com/watch?v=BkAY24qUbn0&t=4s) faster, cheaper, and better — Andrew Rabinovich

> [02:17](https://www.youtube.com/watch?v=BkAY24qUbn0&t=137s) I want my business to make more money — Andrew Rabinovich

> [10:37](https://www.youtube.com/watch?v=BkAY24qUbn0&t=637s) AI-complete problem — Andrew Rabinovich

> [20:23](https://www.youtube.com/watch?v=BkAY24qUbn0&t=1223s) the things that we do now is not because we don't know what else to do — Andrew Rabinovich

> [20:23](https://www.youtube.com/watch?v=BkAY24qUbn0&t=1223s) because we don't have any resources to do more — Andrew Rabinovich

> [20:23](https://www.youtube.com/watch?v=BkAY24qUbn0&t=1223s) play therapist — Thomas Germain

> [20:23](https://www.youtube.com/watch?v=BkAY24qUbn0&t=1223s) oscillations in both directions — Andrew Rabinovich

## Mentioned Airbnb, AlphaGo, Andrew Rabinovich, Anthropic, ChatGPT, Claude, Gemini, Google, HAPI (Human + Agent Productivity Index), Klarna, Model Context Protocol (MCP), OpenAI, Slack, The BBC, The Guardian, Thomas Germain, Uber, Uma (Upwork's Mindful AI), Upwork

## Video description Andrew Rabinovich — Chief Technology Officer, Head of AI & ML, Upwork Thomas Germain — Senior Technology Journalist, The BBC

Information Companies are investing billions of dollars in autonomous AI agents with the promise that software will soon complete professional work independently. But what actually happens when you test these systems on real jobs with real stakes?

Andrew Rabinovich, Chief Technology Officer and Head of AI/ML at Upwork, brings a perspective shaped by 25 years in machine learning and one of the world’s largest real-world datasets on digital work. On the Upwork marketplace, more than 800,000 clients post over 3 million jobs each year across hundreds of professional categories, creating a live laboratory for testing the limits of AI agents in production environments.

In this fireside chat, we’ll share insights from Upwork’s Human + Agent Productivity Index (HAPI), which evaluates how advanced AI agents perform on real paid tasks. The results reveal a consistent pattern: agents struggle to reliably complete even well-defined work on their own, but performance improves dramatically when paired with expert human guidance.

Drawing on more than 10 petabytes of proprietary marketplace interaction data, this discussion will also focus on how Upwork rebuilt its platform for the AI era with Uma, an AI-native orchestration layer designed to coordinate human expertise and intelligent agents to deliver work outcomes faster and more reliably.

Join this session to understand what real-world data reveals about the limits of autonomous AI and why the future of work will be built around human-AI collaboration rather than replacement.

[Source](https://www.youtube.com/watch?v=BkAY24qUbn0)

中文简报

Upwork发现：若无人类指导，AI智能体无法完成大多数真实任务自主AI智能体在付费数字任务中首次尝试的成功率仅为30%至40%，但在人类校验者的指导下，能以1000倍的速度达到人类专家水准。 Upwork首席技术官Andrew Rabinovich指出，自主AI智能体远未达到取代人类知识工作者的程度，在缺乏人类指导的情况下，其在现实世界定性任务中的成功率仅为30%至40%。为弥补这一差距，Upwork正在向由元智能体“Uma”驱动的AI工作操作系统转型，该元智能体能够将客户意图分解为原子基函数，并协调人机协同执行。该平台还见证了“智能体雇佣”（agentic hiring）的兴起，即自主AI模型通过API雇佣人类自由职业者，为具有主观性和非确定性的决策提供实时校验。最终，Rabinovich驳斥了工作总量固定谬论，强调通过模型上下文协议（MCP）将人类专家融入原生智能体工作流，将释放积压的需求并大幅拓展全球生产力能力。 AI远未达到取代人类劳动力的程度，因为现有模型完全依赖人类数据，且在无约束的非确定性领域中表现吃力。 (0:04) Upwork正从传统的人才匹配市场转型为AI原生工作操作系统，致力于将客户意图转化为由人类与智能体协同编排的交付成果。 (0:04) Uma作为编排元智能体，可将复杂项目拆解为500至1000个原子工作单元，并将每个单元委派给专用智能体与人类自由职业者的最优组合。 (2:17) 评估写作和设计等定性工作无法依赖传统基准，需要以真实的平台交易数据和客户满意度作为事实依据。 (4:30) Upwork的“人类+智能体生产力指数”（HAPI）显示，自主AI在真实工作中的初次成功率仅为30%至40%，但若辅以人类指导，能以1000倍的速度达到专家水准。 (7:32)

Reading summary

Andrew Rabinovich, drawing on 25 years in machine learning, contends that artificial intelligence is nowhere near replacing human labor because models cannot autonomously resolve qualitative, non-deterministic tasks. Instead, Upwork is restructuring from a talent marketplace into an AI-orchestrated work operating system where agents hire humans for micro-judgment and oversight. From Matchmaker to Work Operating System (0:04) Upwork is pivoting away from its legacy model of matching clients with individual freelancers toward delivering finished work directly. According to Rabinovich, clients increasingly prefer expressing high-level intent—such as wanting a business to increase revenue—rather than specifying job descriptions, software tooling, or role requirements. Upwork manages this shift through Uma ("Upwork’s Mindful AI"), an orchestration meta-agent designed around explainability and reliability rather than end-task autonomy. Drawing on 20 years of digital work data, Uma translates high-level objectives into 500 to 1,000 atomic units of work, which Rabinovich terms "basis functions." The system then assigns these discrete tasks to an optimized combination of specialized AI tools and human freelancers, overseeing progress through to final verification before delivery to the client. Ground Truth in Non-Deterministic Tasks (4:30) Rabinovich highlights a fundamental limitation in standard AI evaluations: while academic benchmarks focus heavily on deterministic disciplines like coding and mathematics with clear stopping conditions, over 90% of Upwork's marketplace involves qualitative work such as copywriting, marketing strategy, and UX/UI design. Because large language models operate on statistical pattern matching and next-token prediction, they lack tacit human context and drift off-target when resolving multimodal ambiguity across text, visuals, and brand voice. To establish reliable ground truth where deep learning transfer mechanisms break down, Upwork trains Uma on verified platform transaction data. Deliverables accepted by satisfied clients serve as ground-truth baselines, mirroring feedback loops found in consumer marketplaces like Uber or Airbnb. Rabinovich asserts that knowing when a qualitative task is truly finished remains an "AI-complete problem" that necessitates human taste and judgment. The Human Plus Agent Productivity Index (7:32) To measure real-world performance beyond static academic tests that encourage model overfitting, Upwork introduced the Human + Agent Productivity Index (HAPI), which tests autonomous systems on real, economically valuable marketplace jobs. The research shows that state-of-the-art autonomous agents succeed on their initial attempt only 30% to 40% of the time. However, when an experienced human professional reviews intermediate outputs and guides the model through iterative prompting, agent performance reaches parity with human experts while completing deliverables roughly 1,000 times faster. Rabinovich argues this massive speedup demonstrates that combining human domain expertise with agentic execution is far more viable than pursuing fully autonomous end-to-end workers. Agents as Buyers of Human Expertise (17:35) Rejecting the idea that automation will shrink the overall labor pool, Rabinovich argues that lower software production costs will unlock latent demand for bespoke applications. Instead of agents serving solely as talent supply, frontier systems like ChatGPT, Claude, and Gemini are becoming purchasers of labor. Under this "agentic hiring" paradigm, autonomous agents reach non-deterministic hurdles or subjective ambiguities and programmatically hire human specialists via API to perform micro-decisions and real-time verification. To integrate this capability natively into existing enterprise software rather than forcing users onto standalone platforms, Upwork is utilizing Model Context Protocols (MCP). These protocols connect its pool of roughly 20 million freelancers directly into environments like Slack or Claude, effectively treating human expertise as an on-demand extension of the prompt layer to unblock backlogged business roadmaps. While Rabinovich expects labor markets to reach a productive equilibrium that unlocks latent project capacity, the model fundamentally alters knowledge work by replacing weeks-long projects with high-frequency micro-evaluations. This leaves open questions about how workers will navigate market volatility and avoid burnout while supervising continuous agent workflows.

精炼阅读摘要

Upwork机器学习专家Andrew Rabinovich指出，人工智能并未走向完全替代人类劳动，而是推动平台从传统人才撮合中介转型为由意图驱动的“工作操作系统”。他认为，在占据主导地位的定性任务中，AI模型受限于概率机制与终止条件判定难题，必须依托人机协同与模型上下文协议（MCP）将人类专业经验转化为实时校验闭环。从撮合交易转向意图驱动的工作操作系统 (0:04) Rabinovich结合其25年机器学习研发经验指出，当前AI仅仅是在人类已熟练掌握的工作流中以“更快、更便宜、更好”的方式执行任务，由于模型完全依赖人类产生的数据，在缺乏现成范式的不受限非确定性领域中表现挣扎。针对行业替代焦虑，Upwork正在将其商业模式从传统的数字人才撮合市场，重构成端到端交付成果的“工作操作系统”。在该架构中，客户无需自行拆解职位需求，只需表达诸如“让企业赚更多钱”等高层意图。核心编排系统Uma（意为“审慎AI”）依托平台20年积累的工作数据，将繁杂多变的数字项目拆解为500至1000个原子工作单元（basis functions），进而协同调度专用AI智能体与人类专业人员共同执行，并由人类全程参与监督与最终验收。定性任务的市场检验与生产力跃升指数 (4:30) 针对传统基准测试局限于代码和数学等确定性任务的缺陷，访谈指出Upwork平台上超过90%的需求属于文案、设计及营销策略等定性工作。Rabinovich强调，定性任务无法依赖自动化规则校验，其底层事实（ground truth）源于类似Uber或Airbnb的市场交易反馈，即真实客户对交付成果的验收与满意度评价，且各垂直门类的评估能力无法通过简单的迁移学习进行跨领域套用。根据Upwork基于真实商业需求发布的“人类+智能体生产力指数”（HAPI），前沿自主智能体在首次独立尝试真实平台任务时的成功率仅为30%至40%。然而，当引入人类专家进行审查并多轮调整提示词进行迭代引导时，产出成果不仅能达到独立人类专家的基准水平，整体完成速度还可提升约1000倍。概率模型技术局限与定性任务的终止难题 (10:37) Rabinovich剖析了阻碍自主智能体落地的深层技术限制。他指出，主流大语言模型本质是经过人类反馈强化学习（RLHF）微调的统计模式匹配工具，依赖下一个标记（next-token）预测，缺乏人类经验与隐性上下文理解。当项目涉及跨前端开发、后端架构、内容和营销等多资产协作时，纯文本提示词在传递品牌风格和界面体验等细微语境时存在严重歧义，导致单次提示极易偏离目标。与数学定理证明或棋类博弈相比，定性创意工作的分水岭在于“终止条件”（stopping condition）。在确定性领域中，目标明确且验证规则固定，机器可借助无限算力进行暴力搜索与并行探索；而定性数字工作涉及人类审美、语境判断与多重妥协，判断任务何时真正完成属于“AI完全问题”（AI-complete problem）。在人类主观判断被形式化数学表达之前，闭环中的人类协作不可或缺。智能体雇佣机制与工作总量的结构性扩张 (17:35) 在评估知识工作者的未来时，Rabinovich驳斥了工作总量固定论（lump-of-labor fallacy），并以计算器推广和高级编程语言普及带来更多就业机会为例，预测软件与数字资产边际成本的下降将激发对定制化应用的超额需求。他强调，AI智能体正从平台的人才供给方转变为劳动力的主要需求方：当前沿大模型遭遇主观歧义或非确定性障碍时，将通过API接口实时在平台上发起“智能体雇佣”（agentic hiring）。通过部署模型上下文协议（MCP），Upwork计划将其覆盖的约2000万专业人才网络直接内嵌于ChatGPT、Claude及Slack等原生协作环境中，使人类专家成为提示词层的实时延伸。自由职业者的工作形态也将从历时数周的单体项目，演化为以极高频率为智能体提供上下文微观决策与主观校验的引导者。Rabinovich认为，企业积压大量待办事项主要受制于执行资源短缺而非缺乏构想，人机整合将从根本上拓宽全球商业项目的产能边界。对话表明，Upwork试图通过架构重组将AI颠覆风险转化为“智能体雇佣人类”的全新需求市场，但将自由职业者转变为高频微观决策校验者的过程，是否会引发劳工过度疲劳或平台监管摩擦，仍是转型中尚未完全解决的现实问题。

Detailed notes — zh

## Background 本视频由行业峰会频道 Ai4 于 2026 年 8 月 22 日发布在 YouTube 平台。内容记录了 Upwork 首席技术官兼 AI/ML 负责人 Andrew Rabinovich 与 BBC 资深科技记者 Thomas Germain 展开的一场炉边对谈，重点基于 Upwork 拥有的海量真实工作交互数据与 HAPI 指数，探讨自主 AI 智能体在真实经济场景下的能力极限，以及 Upwork 如何依托 Uma 元智能体将平台重塑为人机协同的工作操作系统。 ## TL;DR 对谈直面“AI 全面替代人类”的行业炒作，指出完全自主的 AI 智能体在面对真实非确定性任务时独立成功率仅为 30% 到 40%，但在人类专家的闭环引导下交付速度可提升千倍。Upwork 据此转型为“工作交付平台”，借助核心元智能体 Uma 完成模糊意图理解、原子任务拆解与人机协同编排。针对定性产出难以被算法直接验证的本质瓶颈，人类审美与行业经验成为不可替代的验收基准。未来 AI 智能体更将演变为需求方，通过 Agentic Hiring 与 MCP 协议在交互循环中实时“雇佣”人类专家进行高频校准，推动全社会生产力总量的扩容扩张。 ## Atmosphere & style 整场对话呈现出一种剥离概念炒作后极具穿透力的务实与清醒，洋溢着基于 25 年机器学习一线研发沉淀的理性工业气质。BBC 记者 Thomas Germain 的提问步步紧逼，直切“数学竞赛突破是否会攻陷定性领域”、“看门人式无限工作周焦虑”等敏锐痛点；而 Andrew Rabinovich 则沉着自若，以技术史规律和底层统计机制为坐标，逐一化解焦虑，展现出在看清 AI 算法边界后依然坚定押注“人机共生”的建设性乐观。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=BkAY24qUbn0&t=4s) AI 远未达到完全替代人类的程度，其本质仍依赖人类数据且无法独立处理非受限和非确定性的复杂业务场景。 - [00:04](https://www.youtube.com/watch?v=BkAY24qUbn0&t=4s) Upwork 正在从传统的自由职业者撮合市场转型为“工作操作系统”，由 AI 元智能体将客户目标拆解并交由人机协同交付。 - [02:17](https://www.youtube.com/watch?v=BkAY24qUbn0&t=137s) 核心元智能体 Uma 通过意图理解、任务原子化、人机编排及质量验证四步工作流，全局协调人类与垂直智能体。 - [02:17](https://www.youtube.com/watch?v=BkAY24qUbn0&t=137s) 全平台的海量工作可解构为 500 至 1000 个基础“原子工作单元”，通过组合即可构建任何复杂业务流。 - [04:30](https://www.youtube.com/watch?v=BkAY24qUbn0&t=270s) 对于定性任务的质量评估，Upwork 引入交易闭环机制，以客户满意确认的交付物沉淀为基准真值来进行模型校准。 - [07:32](https://www.youtube.com/watch?v=BkAY24qUbn0&t=452s) HAPI 指数显示，AI 智能体独立接单成功率仅 30%~40%，但有人类专家在回路引导时，交付速度可实现千倍跃升并达到专家级质量。 - [10:37](https://www.youtube.com/watch?v=BkAY24qUbn0&t=637s) 多模态需求的 Prompt 文本表达存在天然歧义，加上 Next Token 预测的统计本质，决定了 AI 必须依赖人类提供隐性经验与纠偏。 - [12:29](https://www.youtube.com/watch?v=BkAY24qUbn0&t=749s) AI Agent 擅长执行自包含、低歧义的单轮确定性任务，而在跨领域、多系统联调的多轮复杂协作中极易崩溃。 - [13:46](https://www.youtube.com/watch?v=BkAY24qUbn0&t=826s) 效率工具的提升会因降低边际成本而激发出庞大的定制化需求，推动总需求呈指数级增长而非消灭岗位。 - [13:46](https://www.youtube.com/watch?v=BkAY24qUbn0&t=826s) 在缺乏闭环校验器的定性任务中界定成果优劣属于“AI-complete”的终极问题，人类主观价值判断是不可替代的验收标准。 - [17:42](https://www.youtube.com/watch?v=BkAY24qUbn0&t=1062s) “Agentic Hiring”代表了双边市场的新模式：AI Agent 正变为需求方，在遇到非确定性决策时实时调用人类专家进行微任务校验。 - [19:38](https://www.youtube.com/watch?v=BkAY24qUbn0&t=1178s) 人类工作形态正从长周期的独立执行者转变为高频校准的领航员，全球自由职业者将通过 MCP 等协议无缝内嵌至各类 AI 工作流中。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=BkAY24qUbn0&t=4s) AI 时代的定位转变：从人才撮合市场到工作交付平台针对大会上普遍存在的“AI 将超越人类并全面替代人类劳动力”这一观点，Andrew Rabinovich 明确给出了不同的判断。拥有 25 年机器学习研发经验的他指出，AI 远未达到完全替代人类的程度。目前的 AI 核心能力在于将人类已经探索掌握的技能以“更快、更便宜、更好”的方式执行，其本质完全依赖于人类生成或收集的数据；面对非受限（unconstrained）且非确定性（nondeterministic）的复杂业务场景，AI 依然寸步难行。 Upwork 的平台定位也因此经历了根本性的演进。过去，Upwork 的核心业务是作为自由职业者与客户之间的连接中介，通过双边撮合来完成数字化工作；如今，平台正转型为一套“工作交付平台”或“工作操作系统”（Work Operating System）。在这一架构下，工作内容由机器在人类的监督与引导下共同定义并部分生成。客户来到平台时，仅需提出最终想要实现的目标，而无需自行规划如何落地，平台的 AI 系统则负责将客户的意图转化为由人机协同交付的具体成果。 ```mermaid flowchart LR accTitle: Upwork 业务定位演进图 subgraph Legacy["传统模式：人才撮合市场"] ClientOld["客户提出具体招募需求"] --> MatchOld["平台中介撮合"] --> FreelancerOld["自由职业者执行交付"] end subgraph AI_Era["AI 时代：工作操作系统"] ClientNew["客户定义最终目标（Intent）"] --> UmaOrch["Uma 元智能体拆解与编排"] UmaOrch --> Collab["人类专家 + 垂直 AI 智能体协同"] Collab --> Result["高质量最终交付成果"] end ``` --- ### [02:17](https://www.youtube.com/watch?v=BkAY24qUbn0&t=137s) 核心架构：解构 Uma 元智能体的四步工作流为实现这一转型，Upwork 推出了名为 Uma 的核心智能体工具。Uma 代表“Upwork’s Mindful AI”（Upwork 审慎智能体）。命名中特别强调“Mindful”，旨在突出系统必须具备可信度、可靠性与可解释性。Uma 的定位并非亲自去执行具体的端到端工作，而是作为“元智能体”（Meta Agent）进行全局工作编排与调度。在设计逻辑上，Uma 依托 Upwork 过去 20 年积累的工作数据资产，将复杂的业务交付拆解为四个明确阶段： ```mermaid flowchart TD accTitle: Uma 元智能体四阶段编排流程 A["1. 意图理解 (Intent Translation) 将商业诉求转为明确目标"] --> B["2. 任务原子化 (Atomic Decomposition) 映射至 500~1000 个基函数任务"] B --> C["3. 人机资源编排 (Talent & Agent Orchestration) 调度人类专家与垂直 Agent 协同"] C --> D["4. 成果质量验证 (Quality Verification) 根据客户核心诉求校准产出质量"] ``` - **意图理解（Intent Translation）**：客户往往不会直接提出“需要一名软件工程师或设计专家”，而是表达高层商业诉求（如“我想让公司赚更多钱”）。Uma 的第一步是将模糊的商业目标转化为清晰的技术与业务意图。 - **任务原子化分解（Atomic Decomposition）**：Upwork 发现，海量繁杂的工作本质上都可以解构为一系列基础的“原子工作单元”（Atomic Units of Work）。这些单元如同数学中的“基函数”（Basis Functions），全平台大约仅由 500 到 1000 个代表性任务组成，通过不同组合即可构建出任何复杂工作。 - **人才与智能体协同编排（Talent & Agent Orchestration）**：识别并调度最适合执行各原子任务的实体——无论该实体是人类、垂直 AI 智能体，还是人机组合，让它们协同完成整套工作流，而非仅做零散任务。 - **质量验证（Quality Verification）**：在最终交付前，由 Uma 检验产出结果的质量标准是否与客户的真实需求严格契合。 --- ### [04:30](https://www.youtube.com/watch?v=BkAY24qUbn0&t=270s) 定性交付物的评估挑战与平台准入机制针对定量任务（如写代码、机器翻译）的评测通常可以通过确定性规则或校验 Agent 轻松完成，但 Upwork 平台上超过 90% 的业务属于文案、设计、营销策略等高度主观、定性的领域。为了让 Uma 具备评估定性内容的能力，Upwork 引入了类似 Uber 和 Airbnb 的交易闭环评价体系。当自由职业者交付成果并获得客户满意认可后，该样本便沉淀为特定品类的“标准真值”（Ground Truth）。Uma 通过持续学习这些成功案例，建立起对主观定性成果的比较与评估基准。由于平台的业务广度极大，这种评估能力的构建必须分门别类逐一打磨，通用领域的迁移学习（Transfer Learning）在此类复杂专业场景中几乎无法直接生效。此外，为了维护平台的专业交付声誉，任何外部 AI 智能体在接入平台接单前，都必须通过严格的真实业务基准测试。测试表明，若让完全独立的 AI 智能体直接挑战具有真实经济价值的实际订单，其初次通过率往往极低。 --- ### [07:32](https://www.youtube.com/watch?v=BkAY24qUbn0&t=452s) HAPI 指数：揭示 AI 独立交付与人机协同的性能鸿沟在谈及 AI 能力评估时，Andrew 批评了行业内两类常见的评测偏差： 1. **静态学术基准（Academic Benchmarks）**：存在严重的“爬山算法式过拟合”（Hill Climbing）。开发者针对 SAT、LSAT 等固定题库反复刷分，模型在考试中表现惊人，但在面对“Google 一词中有几个字母 P”这类常识问题时却给出错误回答。 2. **数据标注公司的宣传偏差**：为了持续推销标注数据，刻意强调模型在复杂任务中仅有个位数精度的极差表现。为此，Upwork 推出了基于真实经济订单的“人类 + 智能体生产力指数”（HAPI, Human + Agent Productivity Index）。该指数利用平台上由人类专家成功完成的动态真实任务，对 AI 智能体进行系统性对照评测。 | 评估维度 | 智能体独立执行 (Autonomous Agent) | 人机协同执行 (Human-in-the-Loop) | |:--- |:--- |:--- | | **任务成功率** | 仅为 **30% ~ 40%**（首轮几乎无法直接交付） | **达到人类专家独立执行的同等水平** | | **交付效率** | 频繁陷入歧义，产出偏离预期 | 相比纯人类作业速度提升可达 **1000 倍** | | **交互模式** | 单次 Prompt 依赖，极易产生理解偏差 | 专家多轮引导校准，快速逼近目标 | 实验数据表明，就像用户日常使用 ChatGPT 极少能单靠一条 Prompt 完美解决问题一样，AI 智能体在缺少引导时极易因理解偏差而失败；但若引入领域专家对首轮产出进行审查与多轮修正，人机组合不仅能完全达到专家的交付标准，交付速度更实现了千倍级的跃升。 --- ### [10:37](https://www.youtube.com/watch?v=BkAY24qUbn0&t=637s) 为什么自主智能体需要“人类在回路” 针对为何必须引入“人类在回路”（Human-in-the-Loop）这一问题，核心原因主要包含以下几点： - **Prompt 的多模态表达歧义（Prompt Ambiguity）**：Upwork 上的绝大多数任务具有多模态特征，交付物涵盖文本、像素、音频等多种形式。然而，当前人与智能体的交互主要依赖纯文本。当试图用语言文字去抽象描述某种设计规范、排版风格或交互体验（UX Context）时，文本表达不可避免地存在巨大歧义空间，导致 AI 探索出大量偏离预期的输出路径。 - **人类偏好对齐的本质局限**：大语言模型普遍依赖基于人类反馈的强化学习（RLHF）进行微调。本质上是模型生成选项 A、B、C，再由人类专家进行偏好裁决，以此校准模型的输出风格与倾向。 - **底层算法机制的先天缺陷**：现有的智能体和语言模型本质上依然建立在“下一个 Token 预测”（Next Token Prediction）的统计模式匹配之上。这一底层原理决定了 AI 缺乏真正的直觉、行业沉淀经验，以及在客户语言表达与真实商业意图之间进行深层意图洞察的能力。人类专家的核心价值正是提供这种隐性经验与关键纠偏。 ### [12:29](https://www.youtube.com/watch?v=BkAY24qUbn0&t=749s) AI Agent 的能力边界：确定性任务 vs 跨领域多轮协作 Andrew Rabinovich 指出，处理模糊性（Ambiguity）需要贯穿在工作完成的多个迭代步骤中，这与当前推理模型（Reasoning Models）和思维链（Chain of Thought）的逻辑高度一致——行业之所以需要观察模型的思考链路，正是为了精准定位错误并予以纠正。当 Thomas Germain 问及 Agent 能够独立完成的任务是否存在某种特定模式时，Andrew 解释称，**任务的歧义越少、交互轮次越少，Agent 越容易胜任**。例如，配置 Web 服务器或独立设计单个网页属于相对自包含（Self-contained）的任务；但如果一项工作涉及前端、后端、内容生成以及营销资产之间复杂的多向连接，Agent 就会崩溃，因为它们要么被过度调优至极狭窄的垂直场景，要么过于泛化而平庸无奇。 ```mermaid flowchart TD accTitle: AI Agent Task Viability Spectrum subgraph Easy["高可行性任务（自包含 / 低歧义）"] A["配置 Web 服务器"] B["单页 UI/网页设计"] end subgraph Failure["易崩溃场景（跨领域多轮协同）"] C["前端与后端系统联调"] D["营销物料与核心内容协同"] E["跨职能项目综合交付"] end Easy -->|低复杂度 & 少交互| Success["Agent 可独立完成"] Failure -->|垂直过拟合 / 泛化能力不足| Bottleneck["Agent 逻辑崩溃，需人类介入"] ``` - 独立任务特征：自包含、单轮执行、上下文确定、输入输出边界清晰。 - 复杂任务痛点：涉及多系统协作与多素材对齐时，AI 缺乏全局协同能力。 --- ### [13:46](https://www.youtube.com/watch?v=BkAY24qUbn0&t=826s) 确定性数学 vs 定性评估：为什么人类专业判断是 AI 终极瓶颈针对近期 OpenAI 与 Anthropic 在国际数学竞赛及难题证明上的“军备竞赛”，Thomas 质疑这是否意味着非量化领域的失守只是时间问题。Andrew 明确表达了乐观但理性的态度。他回顾历史指出：**计算器的发明增加了数学家的数量，打孔卡过渡到编程语言增加了软件工程师的数量**。过去软件之所以千篇一律，是因为构建定制化、个性化应用的成本极高；如今借助 AI，每个人都能拥有专属定制的软件（例如高度个性化的邮件客户端），世界对软件的总需求呈指数级爆炸。针对数学领域的突破，Andrew 将智力活动拆解为离散的光谱： 1. **问题定义（What to do）**：由人类提出猜想与定理（无论是 300 年前的费马还是现代菲尔兹奖得主）。 2. **算力穷举（Crunching）**：机器凭借无限记忆、不眠不休与海量并行能力，在此阶段已全面碾压人类。 3. **终止与评估（When to stop）**：判断任务是否完成。在数学等确定性场景中，机器证明完成后仅需执行形式化验证即可；但在定性（Qualitative）领域中，界定交付物是否合格极度困难。 ```mermaid flowchart LR accTitle: Spectrum of Intelligence in Qualitative vs Quantitative Tasks subgraph Phase1["1. 提出问题"] P1["人类提出定理 / 业务诉求"] end subgraph Phase2["2. 运算与探索"] P2["机器暴力搜索 / 深度推理"] end subgraph Phase3["3. 终止与评估"] P3A["确定性任务：自动化形式验证"] P3B["定性任务：依赖人类主观价值判断"] end Phase1 --> Phase2 --> Phase3 ``` Andrew 提到了 AlphaGo 在对战李世石时下出的“第 37 手”——在围棋这种闭环博弈环境中，系统能够一路推演到终局并直接根据胜负判定胜率；但现实中的定性工作没有固定的胜负判定器，唯一可行的评估基准就是人类的判断力（Human Judgment）。Andrew 强调，解决定性评估是“AI-complete”的终极问题，等同于通用人工智能（AGI）的实现；以他从事机器学习 25 年的经验来看，行业离这一步依然相去甚远。 - 技术变革的历史规律：工具效率提升不但不消灭岗位，反而因降低边际成本激发出更庞大的定制化需求。 - 定性与定性的根本区别：定性任务缺乏明确的损失函数与校验器，人类审美与专业经验是唯一验收标准。 --- ### [17:42](https://www.youtube.com/watch?v=BkAY24qUbn0&t=1062s) Agentic Hiring：AI Agent 成为需求方，在循环中实时雇佣人类在人机协同（Human + Agent）范式下，自由职业生态正迎来剧烈重塑。基础的低门槛任务（如简单 Logo 制作、文本翻译、摘要提取）已被全面自动化，人类不再需要从事这类重复劳动。然而，Upwork 双边市场正在发生颠覆性的结构转变：**AI Agent 不仅是供给侧（Supply）的劳动力，正在变成全新的需求侧（Demand）**。 Andrew 提出了**“Agentic Hiring”（智能体雇佣）**概念：当用户在 Claude、Gemini 或 ChatGPT 等模型中进行对话交互时，模型在遇到非确定性决策（如医疗结果诊断、个性化餐厅推荐、关键主观意见）时无法自主验证，此时 Agent 会通过 Upwork 等平台直接实时“雇佣”人类专家。在此模式下： - 人类的角色从“耗费数周独立完成庞大项目的执行者”，转变为“实时校准、引导与验证 Agent 行为的领航员”。 - 专家工作形态被碎片化与高频化解构：一位资深专家可能无需接大单，而是在一分钟内响应 50 次即时验证提问。 ```mermaid flowchart TD accTitle: Agentic Hiring Architecture User["终端用户"] -->|"复杂任务交互"| LLM["前台 Agent（Claude / ChatGPT / Gemini）"] LLM -->|"遇到非确定性 / 定性决策"| UpworkAPI["Upwork MCP / 调度接口"] UpworkAPI -->|"实时路由微任务"| Expert["人类专家（领航员 & 验证者）"] Expert -->|"即时判断反馈（高频校验）"| LLM LLM -->|"综合交付高可靠成果"| User ``` --- ### [19:38](https://www.youtube.com/watch?v=BkAY24qUbn0&t=1178s) 破解“无限工时”与“零工降级”迷思：MCP 驱动的劳动力内嵌未来针对《卫报》等媒体报道的“无限工作周”（用户沦为 24 小时紧盯 Agent 运行的看门人）以及类似 Klarna 裁减客服后又因 AI 故障不得不重新引入零工人员等职场焦虑，Thomas 询问 Andrew 缘何对未来依然保持乐观。 Andrew 坦言，任何重大技术范式转移都会经历双向的震荡与摇摆，直到最终找到长期稳定的平衡点。他的核心信心源于两大支柱： 1. **工作流的原生嵌入**：依托 Upwork 的 Model Context Protocol（MCP）等连接机制，全球超过 2000 万的专业人才储备能够被无缝内嵌到 Slack、ChatGPT、Claude Co-work 等各类工作流中，无需用户脱离现有环境即可调用顶尖人类智慧。 2. **总生产力上限的扩容**：人类当前所做的工作总量受限于资源与时间上限，而非想象力的枯竭；Agent 技术的成熟将释放巨大的潜在需求，使人类能够涉足过去因成本过高而无法启动的创新事业。 - 宏观波动规律：新技术落地初期伴随试错反弹属于正常周期震荡，最终会收敛于高效的人机分工。 - 人才网络新基建：2000 万全球自由职业者的专业知识将以 API / MCP 的形式成为大模型的外部认知扩展模块。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=BkAY24qUbn0&t=4s) AI 远未达到完全替代人类的程度。目前的 AI 核心能力在于将人类已经探索掌握的技能以“更快、更便宜、更好”的方式执行，其本质完全依赖于人类生成或收集的数据；面对非受限且非确定性的复杂业务场景，AI 依然寸步难行。 — Andrew Rabinovich > [00:04](https://www.youtube.com/watch?v=BkAY24qUbn0&t=4s) 客户来到平台时，仅需提出最终想要实现的目标，而无需自行规划如何落地，平台的 AI 系统则负责将客户的意图转化为由人机协同交付的具体成果。 — Andrew Rabinovich > [02:17](https://www.youtube.com/watch?v=BkAY24qUbn0&t=137s) 这些单元如同数学中的“基函数”（Basis Functions），全平台大约仅由 500 到 1000 个代表性任务组成，通过不同组合即可构建出任何复杂工作。 — Andrew Rabinovich > [07:32](https://www.youtube.com/watch?v=BkAY24qUbn0&t=452s) 就像用户日常使用 ChatGPT 极少能单靠一条 Prompt 完美解决问题一样，AI 智能体在缺少引导时极易因理解偏差而失败；但若引入领域专家对首轮产出进行审查与多轮修正，人机组合不仅能完全达到专家的交付标准，交付速度更实现了千倍级的跃升。 — Andrew Rabinovich > [10:37](https://www.youtube.com/watch?v=BkAY24qUbn0&t=637s) 现有的智能体和语言模型本质上依然建立在“下一个 Token 预测”（Next Token Prediction）的统计模式匹配之上。这一底层原理决定了 AI 缺乏真正的直觉、行业沉淀经验，以及在客户语言表达与真实商业意图之间进行深层意图洞察的能力。 — Andrew Rabinovich > [12:29](https://www.youtube.com/watch?v=BkAY24qUbn0&t=749s) 任务的歧义越少、交互轮次越少，Agent 越容易胜任。 — Andrew Rabinovich > [13:46](https://www.youtube.com/watch?v=BkAY24qUbn0&t=826s) 计算器的发明增加了数学家的数量，打孔卡过渡到编程语言增加了软件工程师的数量。 — Andrew Rabinovich > [13:46](https://www.youtube.com/watch?v=BkAY24qUbn0&t=826s) 过去软件之所以千篇一律，是因为构建定制化、个性化应用的成本极高；如今借助 AI，每个人都能拥有专属定制的软件，世界对软件的总需求呈指数级爆炸。 — Andrew Rabinovich > [13:46](https://www.youtube.com/watch?v=BkAY24qUbn0&t=826s) 现实中的定性工作没有固定的胜负判定器，唯一可行的评估基准就是人类的判断力（Human Judgment）。解决定性评估是“AI-complete”的终极问题，等同于通用人工智能（AGI）的实现。 — Andrew Rabinovich > [17:42](https://www.youtube.com/watch?v=BkAY24qUbn0&t=1062s) AI Agent 不仅是供给侧（Supply）的劳动力，正在变成全新的需求侧（Demand）。 — Andrew Rabinovich > [17:42](https://www.youtube.com/watch?v=BkAY24qUbn0&t=1062s) 人类的角色从“耗费数周独立完成庞大项目的执行者”，转变为“实时校准、引导与验证 Agent 行为的领航员”。 — Andrew Rabinovich > [19:38](https://www.youtube.com/watch?v=BkAY24qUbn0&t=1178s) 人类当前所做的工作总量受限于资源与时间上限，而非想象力的枯竭；Agent 技术的成熟将释放巨大的潜在需求，使人类能够涉足过去因成本过高而无法启动的创新事业。 — Andrew Rabinovich ## Mentioned Andrew Rabinovich, Thomas Germain, Upwork, The BBC, Uma (Upwork's Mindful AI), HAPI (Human + Agent Productivity Index), Uber, Airbnb, OpenAI, Anthropic, AlphaGo, 李世石, Claude, Gemini, ChatGPT, Slack, Model Context Protocol (MCP), Klarna, The Guardian (卫报) ## Video description Andrew Rabinovich — Chief Technology Officer, Head of AI & ML, Upwork Thomas Germain — Senior Technology Journalist, The BBC Information Companies are investing billions of dollars in autonomous AI agents with the promise that software will soon complete professional work independently. But what actually happens when you test these systems on real jobs with real stakes? Andrew Rabinovich, Chief Technology Officer and Head of AI/ML at Upwork, brings a perspective shaped by 25 years in machine learning and one of the world’s largest real-world datasets on digital work. On the Upwork marketplace, more than 800,000 clients post over 3 million jobs each year across hundreds of professional categories, creating a live laboratory for testing the limits of AI agents in production environments. In this fireside chat, we’ll share insights from Upwork’s Human + Agent Productivity Index (HAPI), which evaluates how advanced AI agents perform on real paid tasks. The results reveal a consistent pattern: agents struggle to reliably complete even well-defined work on their own, but performance improves dramatically when paired with expert human guidance. Drawing on more than 10 petabytes of proprietary marketplace interaction data, this discussion will also focus on how Upwork rebuilt its platform for the AI era with Uma, an AI-native orchestration layer designed to coordinate human expertise and intelligent agents to deliver work outcomes faster and more reliably. Join this session to understand what real-world data reveals about the limits of autonomous AI and why the future of work will be built around human-AI collaboration rather than replacement. [Source](https://www.youtube.com/watch?v=BkAY24qUbn0)

Source quotes

[0:04] Andrew Rabinovich: AI 远未达到完全替代人类的程度。目前的 AI 核心能力在于将人类已经探索掌握的技能以“更快、更便宜、更好”的方式执行，其本质完全依赖于人类生成或收集的数据；面对非受限且非确定性的复杂业务场景，AI 依然寸步难行。 [0:04] Andrew Rabinovich: 客户来到平台时，仅需提出最终想要实现的目标，而无需自行规划如何落地，平台的 AI 系统则负责将客户的意图转化为由人机协同交付的具体成果。 [2:17] Andrew Rabinovich: 这些单元如同数学中的“基函数”（Basis Functions），全平台大约仅由 500 到 1000 个代表性任务组成，通过不同组合即可构建出任何复杂工作。 [7:32] Andrew Rabinovich: 就像用户日常使用 ChatGPT 极少能单靠一条 Prompt 完美解决问题一样，AI 智能体在缺少引导时极易因理解偏差而失败；但若引入领域专家对首轮产出进行审查与多轮修正，人机组合不仅能完全达到专家的交付标准，交付速度更实现了千倍级的跃升。 [10:37] Andrew Rabinovich: 现有的智能体和语言模型本质上依然建立在“下一个 Token 预测”（Next Token Prediction）的统计模式匹配之上。这一底层原理决定了 AI 缺乏真正的直觉、行业沉淀经验，以及在客户语言表达与真实商业意图之间进行深层意图洞察的能力。 [12:29] Andrew Rabinovich: 任务的歧义越少、交互轮次越少，Agent 越容易胜任。 [13:46] Andrew Rabinovich: 计算器的发明增加了数学家的数量，打孔卡过渡到编程语言增加了软件工程师的数量。 [13:46] Andrew Rabinovich: 过去软件之所以千篇一律，是因为构建定制化、个性化应用的成本极高；如今借助 AI，每个人都能拥有专属定制的软件，世界对软件的总需求呈指数级爆炸。 [13:46] Andrew Rabinovich: 现实中的定性工作没有固定的胜负判定器，唯一可行的评估基准就是人类的判断力（Human Judgment）。解决定性评估是“AI-complete”的终极问题，等同于通用人工智能（AGI）的实现。 [17:42] Andrew Rabinovich: AI Agent 不仅是供给侧（Supply）的劳动力，正在变成全新的需求侧（Demand）。 [17:42] Andrew Rabinovich: 人类的角色从“耗费数周独立完成庞大项目的执行者”，转变为“实时校准、引导与验证 Agent 行为的领航员”。 [19:38] Andrew Rabinovich: 人类当前所做的工作总量受限于资源与时间上限，而非想象力的枯竭；Agent 技术的成熟将释放巨大的潜在需求，使人类能够涉足过去因成本过高而无法启动的创新事业。

Source quotes

[0:04] Andrew Rabinovich: nowhere close [0:04] Andrew Rabinovich: faster, cheaper, and better [2:17] Andrew Rabinovich: I want my business to make more money [10:37] Andrew Rabinovich: AI-complete problem [20:23] Andrew Rabinovich: the things that we do now is not because we don't know what else to do [20:23] Andrew Rabinovich: because we don't have any resources to do more [20:23] Thomas Germain: play therapist [20:23] Andrew Rabinovich: oscillations in both directions

Original transcript

[0:04] Thanks everyone for joining us. Uh to kick us off, let's start sort of big picture. So AI [0:12] rolls around. There's a lot of people at this conference who are talking about it's going to [0:15] be smarter than a human being. It's going to replace all human labor. For a platform that [0:19] connects businesses with freelancers, I could see that being kind of a threat. Could you [0:24] walk me through how you rethought what Upwork is for and how it's going to manage the new [0:30] era of technology? Sure. Um, so first of all, while I've been working on AI for 25 years now, I [0:40] don't think we're anywhere close with AI replacing humans. Um, so far AI has been able to learn h how [0:48] to do things faster, cheaper, and better. things that humans have already learned how to do. Um, [0:56] all of AI is trained on human generated data or in human collected data. Things that humans don't [1:03] know how to do, we have a very hard time doing in unconstrained and nondeterministic domains. [1:07] That's number one. Number two is um Upwork used to be and it's been historically um a very successful [1:18] uh business essentially a talent marketplace or matchmaking for connecting clients to freelancers [1:26] humans on both sides to get digital work done. Um today Upwork is evolving into a work delivery [1:34] platform or a work operating system if you will where the work is defined and in partly generated [1:44] by machines but not without human oversight. Um in particular when clients come to the platform they [1:52] merely need to define what the goal is without to trying to specify how to achieve that goal. And [1:59] the the goal of the AI systems of Upwork uh mainly this UMA meta agent is to really understand how [2:09] work is done and how to convert an intent of the client into deliverables outcome that are produced [2:17] by both humans and machines. You mentioned UMA. So every company needs an AI strategy. You guys [2:23] rolled out this agentic tool. It's one of the first things I think customers see when they [2:27] come to the platform. Can you walk me through what Uma does, but I think just as importantly [2:32] as you were thinking about rolling it out, the tasks you decided you didn't want it to work on, [2:35] like what were the things Uma could have done that you were like this isn't good for an agent [2:39] to tackle, right? Um, so first of all, UMA stands for Upwork's mindful AI and it's the mindful data [2:46] is very important because the systems need to be trustworthy, reliable and explainable. We spend [2:52] a lot of time focusing on explanability um of outcomes. Um as UMA is built as a meta agent, [3:00] its goal is not to actually do work but to facilitate work. In particular, it's it's broken [3:07] up into four components. First is understanding the intent. When a client comes in and says, [3:14] "I want my business to make more money." They don't ask for a software engineer or a designer [3:19] or a marketing campaigner. they just say I want to make more money here's my business. So the first [3:24] goal of UMA is really to translate that into the intent. Then what we've learned uh over the last [3:31] 20 years uh collecting work data on the platform is that jobs while exponentially many in isolation [3:41] they're all comprised of atomic units of work. In these atomic units of work, we think of them as as [3:47] basis function which are fairly small maybe 500 to a thousand tasks that are representative that [3:55] in combination with each other can generate any kind of a job. Third is to identify the talent [4:03] whether it's humans or machines or combinations of humans and the machines that can do these atomic [4:09] units of work in orchestration uh together such that the complete job gets created as opposed to [4:17] individual tasks being done. And then lastly, Uma is able to verify that the quality of the outcome [4:24] is consistent with what the client actually needs. So, Uma is an orchestration agent rather than a [4:30] worker that completes any particular task. So, I think that's interesting in the sense that when [4:37] we're talking about uh agent benchmarks, a lot of the time we're looking at quantitative tasks, [4:43] right? Coding and math and things like that. A lot of things that people come to Upwork for are [4:47] more qualitative, right? Copywriting, design, marketing strategies, and things like that. How [4:52] did you figure out a way for UMA to evaluate the success of these tasks that are more, you know, [4:58] touchyfey and human for lack of a better word? Right. So, as Upwork represents all digital work, [5:05] there's a small portion of this work that's deterministic things like coding or machine [5:12] translation where you can verify the output by using uh another agent and say if the translation [5:17] is correct or not. But to your point, over 90% of all work on the platform is qualitative and is up [5:25] to the receiver or the generator to determine if it's good or not. Um just like in any marketplace, [5:33] um once a transaction is completed, there is a rating system with just like in Uber [5:39] or Airbnb to determine if the outcome is successful or not. So once the work is done, [5:46] we know if this outcome that's been generated by a freelancer is satisfactory to a client that we [5:53] hold as the ground truth and then we teach Uma based on examples that are generated by [6:00] uh freelancers to learn that new outcomes can be compared to the ground truth and identify [6:06] if this is uh correct or not. That's interesting. Was that a rocky start? Like did it take a while [6:11] for UMA to get to a point where it had enough feedback that it was working effectively? Did [6:16] the users like it at first? Do people jump on it or do you have to kind of coax them into using it? [6:21] It is a continual work in progress and it has to be done on a category per category level because [6:29] the the breadth of the platform is so vast that even if we get good at understanding design or [6:36] website development understanding some other more complex things is independent of the things that [6:42] we learn. So concepts like transfer learning uh don't reapply here. Um, one of the other major [6:49] reasons for doing this is we want to introduce AI agents to complete work on the platform together [6:56] with humans or in isolation and to uphold the brand quality where we spend a tremendous [7:04] amount amount of time making sure that the human freelancers deliver high quality work. We also [7:09] want to make sure that highquality agents are present on the platform. So before these agents [7:14] get onto the platform, they have to complete and pass the benchmark to make sure that they're good [7:19] enough. And it turns out that when agents try to complete this benchmark consisting of real jobs [7:26] with right economic value on the platform, these agents cannot perform quite well. So speaking of [7:32] benchmarks, you guys put out a report earlier this year that I think is really interesting, [7:36] not just for Upwork, but kind of looking across the entire economy and the future of work. [7:40] you came up with a metric. You called it the human agent productivity index. Happy sounds kind of [7:45] nice, right? Uh can you tell me what you looked at and uh what the findings were? Correct. Um so [7:52] if we look at evaluation benchmarks for AI systems in general, they tend to focus on static academic [8:03] benchmarks that are very easy to hill climb. Hill climb is a concept where you continue to optimize [8:08] your systems to perform well on a particular data set and then you exceed it and you say my system [8:14] solves SATs or LSAT so it's an incredible system and then you ask it how many letters P are in [8:20] Google and it says two so it clearly overfits on some things and doesn't work on other things. If [8:26] you go to um data labeling companies they want to show you completely the opposite. They want to say [8:32] that language models and AI agents are so terrible performing in singledigit precision quality that [8:40] you need to continue buying data from them in order to improve the performance of these models. [8:45] Our goal on on on Upwork is slightly different. Our goal is to deliver high qu high quality [8:51] outcomes for real tasks to real clients who are willing to pay real dollars for this. So we take [8:57] real jobs that emerge on the platform that are dynamically updated. we know that these jobs are [9:02] successfully completed by human experts and then we try to compare the performance of these tasks [9:10] from agents to humans. So and what we see is that on the first attempt these agents cannot complete [9:18] these tasks at all and that would kind of be consistent with the out with the results provided [9:24] by the labeling companies. But we took this to the next step and we said if you get the human to work [9:30] with the agent basically guided to improve and just like all of us when we use chat GPT it's very [9:36] rare that we give a single prompt and get a single answer right away we have to continue prompting it [9:41] to get to where we want to be. Same thing here. We let the agent perform the task on the first step. [9:46] Then we look a the human expert looks at an output and then guides the agent to adjust to perform [9:52] additional steps. And through this back and forth interactions, agents do end up completing the [9:57] tasks as well as humans. But it turns out that the the combination of human and machine end up [10:03] doing it thousand times faster. Yeah, this is really interesting. uh especially because you [10:08] had a unique data set to work with which is actual jobs that people have gone to Upwork uh for help [10:14] with. That's right. And as I understand it, the report found that agents working on their own [10:19] completed the task successfully. I think it was 30 to 40% of the time, but with a human in the loop, [10:26] it reaches about par with a human expert working on the task alone. I feel like maybe that's [10:32] counterintuitive, not what people would expect to hear. Can you tell me why it is that human [10:37] in the loop helps with these sorts of problems? So the main reason why this happens is uh is due [10:44] to the prompt ambiguity if you will when most of well not most majority of the jobs on Upwork are [10:56] of multimodal nature meaning that the output that's been generated from from a particular [11:02] task is a combination of text pixels audio um and some other modalities. Um the interaction [11:13] with agents today happens purely through text. So when you try to explain a particular design [11:19] paradigm or some user experience context that you want the agent to generate and you put it in text, [11:27] it's basically ambiguous and the agent can go many different ways and they produce outcomes. [11:33] that this is exactly why um all language models are fine-tuned to human preference through this [11:40] reinforcement learning with human feedback. The agent produces multiple outputs and then [11:45] human experts are asked to pick a answer A, B or C in order to tune these models to provide [11:51] the particular preference of the personality that they're looking for. Um at the same time there are [11:58] certain things that human expertise are not yet translated into the agentic uh capabilities. Um, [12:08] all agents and all language models are trained on essentially next token prediction, [12:15] which is basically a statistical pattern matching algorithm that does not have things like intuition [12:22] and experience and understanding of what someone means when they say something else. [12:29] And this sort of um flavor or context of ambiguity needs to be presented at multiple iterations of [12:38] um job completions which is very consistent with how we think about reasoning models and [12:44] this concepts of uh chain of thought right we want to see how the model thinks because we want [12:50] to understand where it makes mistakes so we can correct it so it's very consistent. Was there any [12:55] pattern that you identified in terms of the kinds of jobs that the agents were able to complete on [13:01] their own without human help? Was it random? Was it more quantitative kinds of tasks? Yes. So, [13:07] the less ambiguous and less multi-turn jobs there are, the easier it is for an agent. If the [13:14] task is set up a web server or if the task is to design a web page, those things are more or less [13:21] u self [snorts] contained. But if the process requires multiple connections between front-end, [13:28] backend, content, marketing assets, um, and other components of work delivery, then the agents tend [13:34] to break because they're either tuned to one very specific vertical or they're so generic [13:40] they're not good at anything in particular. So, zooming out a little bit here, uh, Anthropic and [13:46] Open AI over the past couple weeks are kind of competing each other through press announcements [13:51] about how their AIs are beating or solving novel math problems that human beings have never been [13:58] able to complete. Uh, there's a lot of people here at the conference who will tell you that it's only [14:02] a matter of time before AI moves on and is able to perform just as well in other domains that [14:08] aren't again as quantitative. I would imagine you probably disagree with that. You you have a vested [14:14] interest in it in a sense or the same way the data labeling companies want to continue for there to [14:20] be a need for data labeling companies, you need there to continue to be a need for freelancers. [14:25] But it sounds like in general you're optimistic about the future of labor. Um I am. So f first of [14:32] all when um we invented calculators the number of mathematicians have only increased. When we [14:39] went from punch cards to writing code the number of software engineers has increased. The amount of [14:46] software that needs to be generated and the world needs is exponentially large. And the reason that [14:53] we've had we've generated the software that we have is because created bespoke personalized [14:59] applications and systems is just too expensive and everybody just built very very generic things. Now [15:04] we can build highly personalized email clients for you and for me and for everybody else because [15:10] everybody wants their slightly different things. That's number one. Number two, when it comes to [15:15] math, um the reason why machines are able to perform so well is because these models, [15:23] these tasks are deterministic. So if you think about intelligence as a sort of discrete spectrum, [15:30] the first step is to figure out what to do. So the problems that the theorems that the machines [15:37] solved they were formulated by humans whether it's 300 years ago by FMA or 10 years ago by a [15:44] field medalist. The second part of the spectrum is to do the actual crunching where machines are [15:50] actually better than humans already because they have infinite memory. They can multitask. They [15:56] don't have to sleep and they can just try every possible combination just like when we play chess [16:02] or go with machines. And the last part which is also very difficult is to know when to stop to to [16:08] evaluate the outcome and say this is complete or you got to keep going. In the case of mathematics [16:14] it's actually very easy because when the machine says I've completed a proof all you got to do is [16:19] go and check if the proof is correct and if it is you succeed if it's not you say keep going. [16:23] But in in qualitative things it's actually be it's actually very very difficult. That's number one. [16:30] Um the next step is that all of this, if you remember in um Alph Go, there was this magical [16:40] moment of move 37 where the machine invented a move that no human has ever seen. Everybody [16:47] thought it was a disaster, but it turned out to be like this great surprise that won uh the game. [16:52] The reason it works is because in a game-like environment, you can follow all the steps until [16:58] the end and see if you have won the game or you have lost. In qualitative work, you you it's very [17:07] difficult to say if you've completed the task correctly or incorrectly and the only way to [17:12] do this is to have human judgment. And until we get past that moment and this is largely unsolved [17:18] problem and I would argue that this is sort of the pinnacle of AI. If we solve that that's an [17:24] AI complete problem. If we solve that then AGI is solved. Then we get there but I've been working on [17:29] this for 25 years together with the greatest minds in the sort of industry and I don't think we're [17:35] there yet. So if we need a human in the loop that means uh hopeful note there's an increase on the [17:42] premium for human expertise but as we're getting more efficient right I assume the human plus agent [17:50] uh paradigm on upwork means that tasks are getting completed faster. Is this empowering [17:56] for freelancers or is this ultimately something that's going to make the market a lot more [18:00] difficult to navigate for someone that's working on their own? It's quite transformative. Um, [18:06] to your point, low-level jobs, building logos, translating text, uh, writing summaries of things, [18:13] that's completely automated away. Humans don't need to be doing that, just like humans won't [18:18] have to drive cars anymore. Um, what is becoming extremely interesting, and we just spoke about how [18:25] on this two-sided marketplace at Upwork, agents or machines are becoming an additional source of [18:32] supply. What we're starting to see is that agents are actually becoming a completely new source of [18:38] demand. So this concept of agentic hiring where while you're inside Claude or Gemini or Chad GPT, [18:47] you converse with the agent, but at some point the agent can't verify its own result because it's [18:52] not a deterministic task. Whether it's a medical result or recommendation for a restaurant or some [18:58] some personal opinion, it needs human expertise in the loop in real time. So essentially, [19:04] Chad GPT or Claude starts hiring a humans on Upwork in in real time and rather than [19:14] humans having to complete these giant projects on their own, they become guides or verifiers [19:21] of the behavior of the platform. So the types of jobs that humans will be doing will be completely [19:27] different. So a single human expert wouldn't be working on a single project, but rather be [19:32] available to answer immediate questions 50 times a minute. So we're running out of time here, [19:38] but I want to ask you one more question. Uh last time we spoke, we talked about a couple of, [19:43] you know, really discouraging articles that have come out recently about the labor market. [19:47] Uh there was a story in the Guardian about the infinite work week that now because we're all [19:52] constantly managing AI agents, we're all walking around with our thumb and our laptops to keep them [19:56] open, you never really get to relax. There was a similarly discouraging article that came out about [20:04] uh the transfer to more and more gig work. Uh looking at CLA as an example, where they fired [20:10] all of their customer service agents, replaced them with AI, it didn't work. Then they brought [20:15] back gig working customer service agents working together with AI. It feels like we could be moving [20:23] towards a kind of dystopian labor market here. I'm sure you're optimistic. Maybe you can play [20:28] therapist for me. Why are you feeling good about the future in this scenario? Um so whenever such [20:36] big changes occur you see oscillations in both directions until the the field settles down [20:43] and the the trajectory emerges that's stable and sort of uh long-term. Um we are looking at a time [20:55] where certain habits of work are being completely transformed and the main optimism that I carry is [21:06] a that the work that people do wherever they do it will now be able to be done by agents and humans [21:16] without them leaving their environment where whether it's chatgpt or slack or or cloud co-work [21:24] because using upwork's uh MCPs the human talent of the world which is like 20 million people can [21:31] be directly embedded um into your workflows that's number one um and secondly the type the amount of [21:38] work in general that will be done in the future because of these agentic advances is only going [21:44] to increase because the things that we do now is not because we don't know what else to do. [21:49] It's just because we don't have any resources to do more. Well, I think we're out of time, [21:52] but Andrew, thanks so much for the conversation. Thanks everyone for joining us. Thank you.