ServiceNow CoreAI Introduces AutoSynthData to Train Enterprise Agents Using Automated Curriculum Generation

ServiceNow CoreAI Introduces AutoSynthData to Train Enterprise Agents Using Automated Curriculum Generation ServiceNow CoreAI has developed AutoSynthData, a pipeline designed to convert model failure modes into validated, executable synthetic training datasets for enterprise agents. Tested on EnterpriseOps Gym environments, the framework identifies capability gaps against stronger teacher models and iteratively shifts the training frontier. Useful agentic training tasks require a system specification, a feasible, realistic, and difficult user prompt, and a verifier that is consistent, sound, and complete. AutoSynthData builds curriculum data without exposing original evaluation prompts, relying on capability specification cards derived from evaluation runs. Generation uses a two-phase architecture—Target generation and Multiply expansion—supported by a shared generation controller and environment execution adapter. Quality control involves sample-level positive/negative verification, solver difficulty evaluation, and bounded critique-and-repair loops alongside batch-level meta-reviews. On EnterpriseOps Gym Hybrid, fine-tuning Gemma-4-26B-A4B-it with Qwen3.8-27B demonstrations improved mean Pass@1 by 7.2 percentage points (a 35% relative gain), closing 59% of the gap to the reference model. Task Formulation and Capability Profiling AutoSynthData frames an agentic task through three components: a system specification outlining environment constraints and initialization, a user prompt defining the request, and a verifier. Prompts must be feasible, realistic, and difficult enough to fall within the target agent's unsolved frontier, while verifiers must demonstrate consistency, soundness, and completeness. The pipeline identifies capability gaps by running both the target model and a stronger teacher across diagnostic tasks, converting failure patterns into sanitized capability specification cards without leaking original prompts, entities, or trajectories. Dataset Generation and Scaling Phases Using capability specification cards, AutoSynthData synthesizes training tasks in two core phases. The Target phase produces core training candidates via parallel worker generation, followed by execution, evaluation, and repair. The subsequent Multiply phase expands the dataset by synthesizing novel variations with distinct requests, initial states, and verifiers derived exclusively from validated target samples. Multiplied samples cannot seed further generations, anchoring the process to prevent drift. Architecture execution is split between a centralized controller managing dataset composition and an environment-specific adapter running verification and profiling. Multi-Level Quality Control and Repair To prevent reward hacking and unexecutable trajectories, AutoSynthData applies candidate- and batch-level checks. Candidates are filtered for solver difficulty (typically passing at most one of three target attempts while passing at least two of three teacher trials). Candidates undergo positive verification to test the reference trajectory and negative verification by mutating states to check verifier soundness. Failing candidates are evaluated by a critic that diagnoses errors to attempt targeted repairs within a bounded retry limit. Across generations, batch-level meta-reviews analyze coverage, redundancy, and systematic failure patterns to steer subsequent generator output. Experimental Evaluation on EnterpriseOps Gym The framework was evaluated using supervised fine-tuning across two domains in EnterpriseOps Gym. In the Hybrid domain, using Gemma-4-26B-A4B-it as the target and Qwen3.8-27B as the teacher, 2,000 synthetic tasks generated in roughly 18 hours lifted target model mean Pass@1 by 7.2 percentage points (a 35% relative increase) and raised verifier success from 63.01% to 68.55%, closing 59% of the gap to the reference model. In the ITSM domain, using DeepSeek-V4.1-Flash as the teacher, 1,994 generated tasks increased Pass@1 from 18.77% to 27.18%.

ServiceNow CoreAI推出AutoSynthData：针对企业级智能体的合成训练数据生成框架 ServiceNow CoreAI开发了AutoSynthData框架，通过分析目标模型在特定企业环境中的失败表现和更强教师模型的成功轨迹，自动化生成具备可行性、真实性与校验能力的训练任务，以此弥补模型能力短板并推进持续训练。 任务抽象由系统规范、面向智能体的用户提示词以及验证器三要素构成，需兼顾可行性、真实性及难度。 采用脱敏能力规格卡指导生成，数据构建分为目标（Target）与倍增（Multiply）两个阶段以扩增规模同时避免漂移。 引入样本级正向执行、变异负向校验及批评修复循环，并结合批次级元评审平衡覆盖度与多样性。 在EnterpriseOps Gym实验中，微调后的目标模型在Hybrid域和ITSM域分别取得了7.2和8.41个百分点的Pass@1提升。 企业智能体任务的核心要素 企业模型往往受限于特定系统规则与数据状态。AutoSynthData将任务抽象为系统规范、用户提示和验证器。系统规范界定环境策略与初始状态；用户提示需确保在环境中存在可行轨迹、贴近实际业务需求，且能暴露目标模型的弱点；验证器则需满足一致性、有效性与完备性，既不能过度宽松奖励错误行为，也不能过度严格排斥正确变体。 从模型失败到任务生成与扩增 该框架通过诊断评估分析目标模型失败与教师模型成功的轨迹，提取出能力规格卡。生成器在不接触原始评估提示与轨迹的前提下构建新任务。数据生成采用两阶段推进：首先在Target阶段并行生成核心样本，经执行验证和修复后入库；随后在Multiply阶段对已通过样本进行衍生变异。倍增样本不可作为二次倍增的种子，以防止生成漂移。 样本级质检与批次级动态评审 为保障合成数据质量，系统执行多层质量控制。样本级检测优先筛选目标模型难解而教师模型可解的任务，并实施正向执行检查与负向变异测试，未通过样本交由批评模块定位问题并针对性修复。批次级元评审则持续监测不同任务类别的覆盖率与重复度，动态降低过载类型的生成权重，将预算引导至未覆盖的能力盲区。 动态能力边界与EnterpriseOps Gym实验结果 模型微调后会改变其能力边界，AutoSynthData可随之迁移生成重点以驱动多轮训练或强化学习。在EnterpriseOps Gym的Hybrid领域实验中，利用该流程生成2000个样本对Gemma-4-26B-A4B-it进行微调（以Qwen3.8-27B为教师），使其Pass@1均值提升7.2个百分点，弥补了与基准模型59%的差距；在ITSM领域（以DeepSeek-V4.1-Flash为教师）生成1994个样本后，Pass@1均值也从18.77%提高至27.18%。

Enterprises need agents that work well in their own environments. The work they ask these agents to do is shaped by the systems they use, the rules they follow, and the state of their data. A model may be broadly capable and still struggle with a particular environment: a workflow it handles poorly, a combination of tools it misuses, or a constraint it fails to respect. Those are the weaknesses an enterprise needs to improve. The difficulty is turning those weaknesses into training data. An individual failure tells us something, but training a model requires many new tasks that exercise the same capability in different situations. Those tasks must also be possible to complete in the environment, resemble work someone would actually request, and have a reliable way to check whether the agent succeeded. At ServiceNow CoreAI, we built AutoSynthData to turn those capability gaps into training data. It uses a target model’s failures and a stronger teacher’s successes to decide what the model should learn next, then generates and validates new tasks that exercise those capabilities. As the model improves, the curriculum shifts toward what it still finds difficult. We illustrate the pipeline with EnterpriseOps Gym ( Malay et al., 2026 ), using the released dataset . We begin by describing the environment an agent operates in and what makes a task useful for training. What makes a useful agentic task? An agentic environment defines the world in which an agent operates: the state it can observe and modify, the tools and APIs it can invoke, and the state transitions produced by its actions. A task is instantiated within this environment. We use the following abstraction: task = (system specification, user prompt, verifier) System specification The system specification defines the constraints under which the agent operates, including system instructions, environment policies, and, when applicable, task-specific initialization such as a seeded database state or a set of knowledge articles. The specification must be compatible with the environment’s tools, state, and supported actions. Its instructions should be clear and avoid arbitrary constraints introduced solely to manufacture difficulty. Agent-facing task The user prompt specifies what the user wants the agent to accomplish, together with any user-level constraints. A generated task should satisfy three properties. Feasibility. There should exist at least one trajectory in the current environment that satisfies the user prompt while respecting the system specification. This rules out tasks that depend on unavailable tools, inaccessible knowledge, impossible state transitions, or actions prohibited by policy. Realism. The user prompt should resemble something a user would plausibly ask in the target environment. The space of executable behaviors is usually much larger than the space of realistic workflows. Difficulty. For training, the task should expose a weakness of the current agent. Tasks that are already solved reliably provide little new training signal. The useful region is therefore tasks that are feasible and realistic, but not yet consistently solved. Verifier The verifier determines whether the resulting trajectory successfully completes the task. It should satisfy three properties. Consistency. It should agree with the user prompt, the system specification, and the task-specific environment state. Soundness. It should reject trajectories that fail to satisfy the task or violate relevant constraints. Completeness. It should accept valid solutions rather than encode one particular reference trajectory. These properties matter directly during training. A lax verifier can reward incorrect behavior, while an overly restrictive verifier can penalize valid solutions. Overview Given an environment and a target model, AutoSynthData generates training tasks consisting of a system specification, user prompt, and verifier. The generated tasks are grounded in the environment and selected to provide useful training signal for the current model. AutoSynthData first evaluates the target model in the environment using diagnostic tasks and identifies patterns in the tasks it struggles to complete. A stronger teacher helps characterize which of those tasks are solvable and what successful behavior looks like. AutoSynthData turns the resulting capability gaps into new executable tasks, checks each task in the environment, and uses accepted samples for post-training. Evaluating the updated model reveals which gaps remain and can guide the next round of generation. From model failures to a curriculum AutoSynthData uses evaluation runs in the target environment to identify what the model needs to learn next. In our EnterpriseOps Gym experiment, we run both the target model and a stronger teacher on the evaluation tasks. We examine those runs to identify: the capability being tested; the tools and workflow structure involved; where the target model fails and how the teacher succeeds; the properties that a correct final state must satisfy; the dimensions that can vary while preserving the capability being tested. We distill these findings into sanitized capability specification cards. The evaluation tasks guide what the model should learn, but the generator does not receive their original prompts, entities, trajectories, or verifier details. It receives the cards and uses them to create new tasks with different prompts, states, and solution paths. Generating and scaling tasks Identifying a capability gap tells us what to teach, but training requires many varied tasks that exercise it. AutoSynthData uses the specification card to generate those tasks. Suppose the target model struggles with tasks that require the following workflow: The generator creates new tasks that exercise this workflow, varying the entities, initial environment state, workflow composition, tool combinations, wording, and difficulty. The stronger teacher then demonstrates a successful trajectory for each task. For supervised fine-tuning (SFT), these demonstrations teach the target model how to apply the capability in new situations. AutoSynthData builds the dataset in two phases: first generating and validating core samples, then expanding them into novel variants. Target The target phase creates the core set of training samples from the capability specifications. Workers generate independent tasks in parallel, picking up a new target when they finish. Each candidate goes through validation, execution, solver evaluation, and repair before acceptance. The result is a batch of vetted examples built around what the target model needs to learn. Multiply The multiply phase expands the dataset by creating novel variants of accepted target samples. Each variant has its own user request, environment state, entity configuration, reference trajectory, and verifier, and must pass the same validation and execution checks. A multiplied sample cannot seed another multiplied sample. This anchors expansion to the vetted target set and limits drift across generations. Implementation details To support both phases, AutoSynthData separates generation control from environment-specific execution. A shared controller coordinates generation, quality control, coverage, and dataset construction, while an adapter handles environment execution, task and state management, reference replay, deterministic verification, solver execution, and task profiling. Together, parallel target generation and multiplication provide a path to training-scale datasets. Their usefulness depends on the checks applied to every candidate: the task must be executable, the solution must work, and the verifier must distinguish success from failure. High-quality synthetic data needs more than generation Generating a plausible request is not enough to produce useful training data. A task may be impossible in the target environment, its reference solution may fail when executed, or its verifier may reward the wrong final state. AutoSynthData checks these properties before accepting a task for training. AutoSynthData reviews quality at two levels: individual candidates must pass verification, and batches must provide useful coverage and diversity. Sample-level verification and repair Each candidate must clear a quality-control loop before entering the training dataset. We begin with solver evaluation to measure difficulty. In the configuration used here, we favor tasks the target model solves on no more than one of three trials and the stronger solver solves on at least two of three trials. Candidates also undergo positive and negative verification and a bounded repair process. Positive verification The positive gate asks: Does the intended solution solve the generated task? The pipeline executes the reference trajectory in the target environment and checks the resulting state against the candidate’s verifier. This reveals mismatches among the prompt, initial state, solution, and success criteria. Negative verification The negative gate asks: Do relevant incorrect outcomes fail? For example, it can mutate parts of the expected outcome and confirm that those states no longer pass verification. This catches weak verifiers that award success without requiring the intended behavior. Critique and repair Failed candidates go to a critic before being discarded. The critic examines the sample and its failure, looking for inconsistent state, impossible workflows, incorrect task construction, bad reference trajectories, weak verifier logic, or a mismatch with the intended capability. The critic’s findings guide repairs, with a fixed limit on retries: candidate ↓ failure ↓ critique / diagnosis ↓ targeted repair ↓ run the gates again ↓ accept or retry A repaired task must pass the relevant checks again. The diagnosis guides repairs to the existing candidate rather than requiring generation to start over. Passing these checks makes a sample eligible for training, but individually valid samples can still form a repetitive or unbalanced dataset. AutoSynthData therefore also reviews generation at the batch level. Batch-level review A batch may overrepresent a few easy task families, miss a capability, or reflect too much generation effort spent on a low-yield pattern. A meta-review examines accepted samples, rejected samples, and generation behavior across each batch. It asks: Which task families are overrepresented, and which capability dimensions are missing? Are the same kinds of examples appearing repeatedly? Do particular targets keep failing generation? Are systematic problems appearing in critiques? What guidance should change for the next batch? The controller tracks coverage in the accepted dataset, reduces generation in overrepresented regions, and directs more work toward gaps. When a region repeatedly produces poor candidates, critiques and meta-review guide changes to the generation strategy. These adjustments balance useful learning signal, task quality, coverage, diversity, and low redundancy within the available generation budget and dataset size requirements. Together, these feedback loops improve both individual tasks and the dataset they form: sample-level checks guide candidate repair, while batch-level review guides future generation. Moving the training frontier The useful training distribution changes as the model improves. AutoSynthData treats synthetic data generation as a search for tasks near the target model's capability boundary: difficult enough to expose weaknesses, but solvable enough for the teacher to provide reliable demonstrations. After post-training, we evaluate the updated model in the same environment. Tasks it now solves reliably are less useful for the next training round; persistent failures point to capabilities that still need attention. Those results can guide the next generation round. Our experiments focus on SFT, but the same mechanism could support reinforcement learning (RL): generate tasks that challenge the current policy and provide reliable learning signal, train, then move the generation target with the updated policy. We plan to test this moving, difficulty-calibrated frontier beyond SFT. EnterpriseOps Gym experiments We use EnterpriseOps Gym to test whether this approach improves a model on tasks in a stateful enterprise environment. We generate training tasks in the Gym’s Hybrid and ITSM environments, fine-tune the target model on accepted samples, and evaluate the resulting checkpoints. Hybrid We tested the pipeline on the Hybrid domain of EnterpriseOps Gym using Gemma-4-26B-A4B-it as the target model and Qwen3.8-27B as the teacher. AutoSynthData generated 2,000 synthetic training samples in about 18 hours. We fine-tuned Gemma on this dataset and evaluated the resulting checkpoints on the benchmark. The best checkpoint was epoch 5. Hybrid results The synthetic SFT checkpoint improves mean Pass@1 by 7.2 percentage points, a 35% relative improvement, and raises verifier success from 63.01% to 68.55%. It closes 59% of the original Pass@1 gap between Gemma and the reference model. The training tasks were newly generated from capability specifications; the generator did not receive the original evaluation tasks. The result demonstrates improvement in EnterpriseOps Gym Hybrid, the environment used for this experiment. ITSM We also applied AutoSynthData to the ITSM domain of EnterpriseOps Gym, using Gemma-4-26B-A4B-it as the target model and DeepSeek-V4.1-Flash as the teacher. AutoSynthData generated 1,994 synthetic training samples in 66 hours. Generation took longer than in the subsequent Hybrid run described above, primarily because the ITSM run used a larger teacher model and preceded pipeline optimizations that improved throughput. On ITSM, synthetic SFT raises mean Pass@1 from 18.77% to 27.18%, showing that the approach also improves performance in a second domain. Closing the loop The tasks most useful for training depend on both the environment and the model working in it. AutoSynthData uses the model’s failures to choose what to generate, validates new tasks against the environment, and makes those tasks available for post-training. Our EnterpriseOps Gym results show the value of that approach in a controlled setting. As the model changes, the same process can focus on the gaps that remain.