ALTK-Evolve Matches or Exceeds ACE Agent Performance at a Fraction of the Token Cost

ALTK-Evolve Matches or Exceeds ACE Agent Performance at a Fraction of the Token Cost A comparative evaluation between agentic memory systems ACE and ALTK-Evolve shows that calibrating memory delivery rather than injecting an entire playbook at every step yields equal or superior task accuracy while significantly lowering token usage. Both ACE and ALTK-Evolve refuse to compress learned agent trajectories into generic summaries, instead tracking support counts across individual lessons. While ACE injects its entire evolving playbook on every step, ALTK-Evolve clusters lessons into typed guidelines and selectively retrieves what a model can handle. Evaluated on AppWorld, ALTK-Evolve outperformed ACE on DeepSeek-V3.2 using ~40% of the tokens and tied accuracy on gpt-oss-120b at roughly one-seventh of the inference cost. Shared Principles on Agentic Memory Both ACE and ALTK-Evolve enable LLM agents to learn from execution histories without weight updates or human labeling. The systems share a core design choice: refusing to compress hard-won lessons into brief summaries. Both frameworks intentionally avoid brevity bias and context collapse by preserving granular lessons backed by counters—ACE using per-bullet helpful/harmful counters and ALTK-Evolve using support counts that track how many independent episodes produced each guideline. Consolidation and Delivery Architecture The systems diverge in how they build memory and deliver it during inference. ACE expands a single comprehensive playbook via a Generator-Reflector-Curator loop, deduplicating through embeddings and feeding the entire playbook to the model on every step. ALTK-Evolve consolidates near-duplicate lessons within clusters, preserving combined support counts, and extracts typed guidelines (strategy, recovery, optimization) with causal attribution and subtask granularity. Rather than sending the full memory store continuously, ALTK-Evolve treats delivery as a dial, selecting core and task-specific guidelines based on model capacity. AppWorld Benchmark Findings In-house AppWorld tests using a ReAct code agent demonstrated substantial token savings for ALTK-Evolve. On DeepSeek-V3.2, ALTK-Evolve reached 89.3 Task Goal Completion (TGC) and 80.4 Scenario Goal Completion (SGC) with 263K tokens per task, beating ACE's 80.4 TGC and 73.2 SGC at 634K tokens. On the weaker gpt-oss-120b model, ALTK-Evolve scored 56.0 TGC using 116K tokens compared to ACE's 54.8 TGC using 777K tokens. On the weaker model, selective retrieval outperformed the full playbook on hard tasks by preventing excessive context from overwhelming model capacity.

ALTK-Evolve与ACE对比：以更低Token成本实现高效智能体记忆 IBM Research对比了ALTK-Evolve与ACE两套智能体记忆系统。两者均主张让智能体从过往轨迹中吸取经验且拒绝过度压缩，但ALTK-Evolve通过动态校准的交付机制，在AppWorld基准测试中大幅降低了推理阶段的Token消耗，同时保持或超越了基准准确率。 核心共识：两系统均拒绝将经验压缩为简略概括，分别通过命中计数与“支持度计数”（support count）保留每条经验的重复验证次数以防上下文崩溃。 架构差异：ACE每步向模型注入完整的操作手册（playbook）；ALTK-Evolve则聚类合并同类经验并继承支持度，推理时按模型能力动态检索特定规则或全量注入。 测试表现：在AppWorld基准上，强模型DeepSeek-V3.2使用ALTK-Evolve后达到89.3% TGC，Token消耗仅为ACE的约40%；弱模型gpt-oss-120b则以约七分之一的成本取得了与ACE相当的准确率。 经验的构建与交付差异 ACE通过生成器、反思器与管护器循环扩充统一剧本，并在推理的每一步无差别注入完整内容。ALTK-Evolve则在构建时聚类近似经验并合并支持计数，提取具备因果归因与来源追溯的分类准则；在推理交付时，ALTK-Evolve将其视作可调节的旋钮，针对强模型提供全量集合，针对弱模型则仅提供核心规则及任务检索子集，避免过长上下文干扰模型发挥。 基准测试效率与分级表现 在AppWorld的168个测试任务中，ALTK-Evolve配合DeepSeek-V3.2取得了89.3%任务目标完成率（TGC）和80.4%场景目标完成率（SGC），单任务平均消耗263K Tokens，显著优于ACE的80.4% TGC与634K Tokens。在gpt-oss-120b上，ALTK-Evolve在简单和中等难度任务略逊于ACE，但凭借困难任务中的针对性检索拉高了整体均值，以116K Tokens打平ACE全量注入的777K Tokens表现。

ALTK-Evolve and ACE both let an agent learn from its own trajectories. The difference is what they do with what they learn — and that decides the token bill. Give an LLM agent a realistic multi-step task — split a bill, find a song, reconcile an order across nine simulated apps — and when it fails, it usually isn't for lack of knowledge. It mis-paginates an API, resolves the wrong person, or returns a value when none was asked for. The model knows the APIs; what it hasn't internalized is how to use them reliably . That's learnable from the agent's own history. Two recent systems do exactly this, on the same kind of agent: ACE (Agentic Context Engineering) and our ALTK-Evolve ( introduced here ). Both are a form of agentic memory — turning an agent's past trajectories into reusable lessons and feeding them back at inference time, no weight updates, no human labels. They even agree on the hard part. Where they part ways is delivery . A note on words, because the two systems name things differently: we'll call the raw thing an agent learns a lesson . ACE organizes its lessons into one comprehensive, evolving playbook ; we consolidate ours into individually retrievable guidelines . Same lessons, two containers. What we agree on Both systems refuse to compress. ACE names the failure modes precisely: brevity bias — optimization collapsing toward short, generic instructions — and context collapse — a model asked to rewrite its whole context each step summarizing the detail away. Its answer is to keep a rich, itemized playbook, with a helpful/harmful counter on every bullet, and let the model distill relevance at read time. We reach the same conclusion from the other direction. Every distinct guideline keeps a support count — how many independent episodes produced it — and we never summarize the store down to a handful of rules. A lesson five different tasks discovered is a different object from one that appeared once, and both are worth keeping. So on the core question — should you compress an agent's hard-won lessons into a tidy summary? — ACE and ALTK-Evolve give the same answer: no. Count them, don't collapse them. ACE's per-bullet counters and our support counts are two spellings of the same idea. Where we differ Two places: how the memory is built , and how it's delivered — and it's the delivery difference that shows up in the token bill. Consolidation (how the store is built). ACE grows one playbook through a Generator → Reflector → Curator loop, applying incremental delta updates and de-duplicating by embedding. We cluster near-duplicate lessons and merge within a cluster, support-conserving — when several lessons merge, the survivor inherits their combined count, so the store shrinks without losing the record of how much experience backs each guideline. We also extract typed guidelines — strategy, recovery, and optimization — with causal attribution and provenance back to the source trajectory, and at subtask granularity, so a lesson learned on one app can transfer to another. Delivery (what reaches the model at inference). This is the one that drives the numbers. ACE injects the comprehensive playbook on every step, the same way regardless of model or task. We treat delivery as a dial, not a constant: a small fixed core of high-support guidelines, extended per task with a handful selected for the task at hand (cosine or LLM-guided, priority-weighted) — or, when a model has the headroom to use it, the full consolidated set. The same lessons are available to both agents; the difference is that ACE always sends all of them, and we send however many a given model can actually use. Why it matters On AppWorld, with the same base ReAct agent, running both systems in-house: Model TGC / SGC Tokens/task DeepSeek-V3.2 ACE 80.4 / 73.2 634K ALTK-Evolve 89.3 / 80.4 263K gpt-oss-120b ACE 54.8 / 35.7 777K ALTK-Evolve 56.0 / 37.5 116K On the strong model we're better on both metrics at ~40% of ACE's inference cost . On the weak model we edge ACE 56.0 to 54.8 — close enough that we call it a tie on accuracy (a repeat run of ours landed at 54.8, matching ACE almost exactly, which is within this benchmark's run-to-run noise) — at about one-seventh the cost. A fair word on cost: ACE's own efficiency story is about building its context cheaply. Ours is on a different axis — serving it. Retrieving a few guidelines per task instead of injecting the whole playbook on every step is where the tokens go, and it's the direct consequence of the delivery difference above. Where does the accuracy come from? The by-difficulty breakdown tells two different stories: Figure 1. Post-memory Task Goal Completion by difficulty, ours vs. ACE. On DeepSeek-V3.2 (right) we win Easy, Hard, and Overall; ACE only edges Medium. On gpt-oss-120b (left) ACE leads easy and medium, but per-task selection wins the hard tasks — and the aggregate. Each system improves from its own no-memory baseline (see the by-difficulty reference tables under Method notes below). The two models tell different stories. On gpt-oss-120b, ACE's full playbook has the edge on Easy and Medium — there's enough of the task solved by generic instruction-following that a comprehensive prompt helps more than it distracts. But on Hard tasks, where the model has to pick the right lesson rather than wade through all of them, curated retrieval pulls ahead — and that's the tier that decides the aggregate. On DeepSeek-V3.2 the story flips: the stronger model absorbs ACE's full playbook well enough to edge us on Medium, but we lead Easy, Hard, and Overall — with more capacity to spare, more lessons (delivered our way) keep helping instead of crowding each other out. We give each model its best configuration — the full consolidated set for the strong model, selective retrieval for the weaker one, because a large context overwhelms a weaker model rather than helping it. (Exactly how much to inject, and how it scales across the capability spectrum, is the focus of our follow-up post .) Same lessons, different delivery Both systems refuse to compress an agent's hard-won experience into a tidy summary — that part, we agree on. The difference is whether delivery is fixed or calibrated: ACE sends the whole playbook every step no matter what; we send however much of the guideline set a given model can actually use. That calibration is what bought the numbers above — same-or-better accuracy at a fraction of ACE's inference cost — and on the weaker model, it was the difference between guidance that helped and guidance that got in the way. Try the ALTK-Evolve library — which includes the extraction, consolidation, and retrieval pipeline used here — or read the full technical report for the complete method and ablations. Linked artifacts / references Earlier post: ALTK-Evolve introduction — link ACE (Agentic Context Engineering) — link AppWorld benchmark — link ALTK-Evolve — link Full technical report — link Method notes AppWorld test_normal , 168 tasks. A ReAct code agent (each step writes Python; the environment returns the output). TGC = Task Goal Completion; SGC = Scenario Goal Completion, which requires every variant of a scenario to pass. Memory is mined from train/dev only ; results are single runs (pass@1), as is standard on this benchmark. The ACE numbers are our own runs of the ACE agent, evaluated in-house on the same AppWorld splits and the same base models as ALTK-Evolve (DeepSeek-V3.2 and gpt-oss-120b). The ACE paper reports on a different base model (DeepSeek-V3.1), so running it ourselves keeps the comparison controlled for model and harness. Both systems are the same ReAct agent and differ only in the prompt template — which is why the two no-memory baselines differ (72.0 vs 79.8 TGC); we don't rest the comparison on that baseline gap, only on the claims a prompt tweak can't touch: same-or-better accuracy at a fraction of the tokens. Reference tables DeepSeek-V3.2 — test_normal (168 tasks): System Guidelines TGC SGC Tokens/task ReAct, no memory 0 79.8 64.3 148K ReAct + ACE 106 80.4 73.2 634K ReAct + ALTK-Evolve 191 89.3 80.4 263K gpt-oss-120b — test_normal : System Guidelines TGC SGC Tokens/task ReAct, no memory 0 39.9 21.4 110K ReAct + ACE full 54.8 35.7 777K ReAct + ALTK-Evolve (selected) ~29 56.0 37.5 116K gpt-oss-120b — by difficulty (TGC): Difficulty Baseline ACE ALTK-Evolve Easy 66.7 84.2 82.5 Medium 35.4 60.4 56.2 Hard 19.1 23.8 31.8 Aggregate 39.9 54.8 56.0 DeepSeek-V3.2 — by difficulty (baseline → +memory): the two systems start from different no-memory baselines (79.8 vs 72.0 TGC overall) because of the prompt-template difference above. Tier ALTK TGC ALTK SGC ACE TGC ACE SGC Overall 79.8 → 89.3 64.3 → 80.4 72.0 → 80.4 57.1 → 73.2 Easy 93.0 → 94.7 84.2 → 84.2 78.9 → 84.2 63.2 → 78.9 Medium 81.2 → 97.9 62.5 → 93.8 85.4 → 100.0 75.0 → 100.0 Hard 66.7 → 77.8 47.6 → 66.7 55.6 → 61.9 38.1 → 47.6