Fireworks AI Unifies Train-Deploy Stack to Replace Closed Model APIs

Stop Renting Intelligence: The Train-to-Deploy Loop for Specialized AI — Fireworks AI — AI Engineer

AI Engineer

Splitting model training and inference across separate cloud platforms degrades output quality through numerical mismatches, making a co-located train-to-deploy loop essential for open models to beat proprietary alternatives.

AI-assisted summary of Stop Renting Intelligence: The Train-to-Deploy Loop for Specialized AI — Fireworks AI — original source. Check the original for full context.

Source published:

Key ideas

  • Fireworks AI processes over 30 trillion tokens across 10,000+ customer organizations with a concurrent scale of approximately 200,000 requests per second. (3:12)
  • Custom inference optimization using proprietary CUDA kernels and speculative draft models yielded a 13x speedup for Cursor's Composer 2 and a 40x latency reduction for Vercel. (3:47)
  • To prevent quality loss and precision degradation, Fireworks unifies training and inference under matching numerics alongside zero-downtime weight hot-loading. (12:42)
  • Fireworks supports three distinct training paths: an autonomous Training Agent for automated model and hyperparameter selection, Managed Training with UI observability, and a code-level Training API for custom loops and loss functions. (7:22)

Detailed notes

## Background Published on YouTube by AI Engineer on October 11, 2026, this talk features Jetashree Ravi, Tech Lead Manager of Applied ML at Fireworks AI, speaking at the AI Engineer World's Fair in San Francisco. The page description outlines how teams can migrate away from expensive closed models toward specialized open-source alternatives through continuous training, low-level inference optimizations with FireAttention, and zero-downtime deployment pipelines.

## TL;DR Jetashree Ravi argues that engineering organizations can overcome the escalating costs of proprietary APIs by customizing production-ready open models within a continuous train-and-deploy loop. She illustrates how Fireworks AI supports massive scale for clients like Cursor, Vercel, and Genspark using custom CUDA kernels, speculative decoding, and reinforcement fine-tuning (RFT). The presentation breaks down three training approaches—an autonomous Training Agent, Managed Training, and a granular Training API—while emphasizing that co-locating training and inference prevents numerical drift and enables zero-downtime weight hot-loading. Ravi concludes with live platform walkthroughs demonstrating automated fine-tuning planning and one-click production serving profile configurations.

## Atmosphere & style The talk has a crisp, pragmatic, and engineering-focused tone, beginning with an empathetic audience poll acknowledging the sharp cost spikes of proprietary AI before moving swiftly into technical solutions. Ravi presents with calm, measured authority, balancing massive platform metrics with granular technical details on CUDA kernels, precision degradation, and speculative decoding. Watching the session feels like attending a hands-on architectural briefing, reinforced by straightforward UI demonstrations that demystify fine-tuning workflows for practicing machine learning engineers.

## Bulletin - [03:12](https://www.youtube.com/watch?v=13YZ1EB9hjU&t=192s) Fireworks AI processes over 30 trillion tokens across 10,000+ customer organizations with a concurrent scale of approximately 200,000 requests per second. - [03:47](https://www.youtube.com/watch?v=13YZ1EB9hjU&t=227s) Custom inference optimization using proprietary CUDA kernels and speculative draft models yielded a 13x speedup for Cursor's Composer 2 and a 40x latency reduction for Vercel. - [12:42](https://www.youtube.com/watch?v=13YZ1EB9hjU&t=762s) To prevent quality loss and precision degradation, Fireworks unifies training and inference under matching numerics alongside zero-downtime weight hot-loading. - [07:22](https://www.youtube.com/watch?v=13YZ1EB9hjU&t=442s) Fireworks supports three distinct training paths: an autonomous Training Agent for automated model and hyperparameter selection, Managed Training with UI observability, and a code-level Training API for custom loops and loss functions.

## Key takeaways - [00:12](https://www.youtube.com/watch?v=13YZ1EB9hjU&t=12s) Rising costs from closed-source model providers are pushing teams to customize open-source foundation models to match or exceed proprietary capabilities. - [01:27](https://www.youtube.com/watch?v=13YZ1EB9hjU&t=87s) Modern open models such as GLM-4.7 and GLM-5.2 have reached baseline production readiness, especially on complex software engineering benchmarks. - [02:22](https://www.youtube.com/watch?v=13YZ1EB9hjU&t=142s) The operational paradigm has shifted from deploying static model checkpoints to building continuous, iterative train-and-infer feedback loops on live production data. - [03:47](https://www.youtube.com/watch?v=13YZ1EB9hjU&t=227s) Genspark overcame quality plateaus when migrating off closed models by applying reinforcement fine-tuning (RFT) with custom evaluators, achieving a 12% quality improvement. - [05:27](https://www.youtube.com/watch?v=13YZ1EB9hjU&t=327s) The FireAttention engine writes custom low-level CUDA kernels co-designed around specific customer sequence lengths, token ratios, and prompt cache hit rates. - [06:27](https://www.youtube.com/watch?v=13YZ1EB9hjU&t=387s) Speculative decoding on Fireworks allows teams to train their own custom draft models, accelerating token generation speed without sacrificing end-model quality. - [07:22](https://www.youtube.com/watch?v=13YZ1EB9hjU&t=442s) Fireworks structures its training platform into three personas: a guided Training Agent, a point-and-click Managed Training service, and an open Training API for custom code. - [10:07](https://www.youtube.com/watch?v=13YZ1EB9hjU&t=607s) While Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) are widely supported, Reinforcement Fine-Tuning (RFT) requires specialized dataset curation and custom reward evaluation setups. - [12:42](https://www.youtube.com/watch?v=13YZ1EB9hjU&t=762s) Zero-downtime hot-loading, developed originally for Cursor, lets teams hot-swap active model weights on live endpoints without interrupting user requests or draining clusters. - [13:22](https://www.youtube.com/watch?v=13YZ1EB9hjU&t=802s) Training on one platform and serving on another frequently creates numerical mismatches that cause model regressions; co-locating both stages guarantees mathematical fidelity. - [14:22](https://www.youtube.com/watch?v=13YZ1EB9hjU&t=862s) The autonomous Training Agent inspects user training and validation data, formulates an execution plan, selects appropriate architectures (e.g., Qwen 38B), and runs tuning runs upon approval. - [16:17](https://www.youtube.com/watch?v=13YZ1EB9hjU&t=977s) Deploying fine-tuned models from Managed Training is handled in one click via preconfigured operational profiles tuned for latency, high throughput, or time to first token.

## Detailed walkthrough ### [00:12](https://www.youtube.com/watch?v=13YZ1EB9hjU&t=12s) The Shift from Closed to Open Models

Jetashree "Jet" Ravi opens her talk by surveying the audience: she asks how many in the room currently rely on closed-source models such as ChatGPT or Anthropic, and how many are finding that those models are becoming extremely expensive very quickly. Seeing a large show of hands, she points out that companies and developers across the industry are facing the exact same pressure. Teams are actively looking to transition away from proprietary systems toward open-source models for both cost efficiency and raw performance. The central question is how to make the quality and operational performance of open-source models match or beat their closed-source counterparts.

```mermaid flowchart TD accTitle: The Continuous Train-to-Deploy Cycle accDescr: Diagram illustrating the shift from static single-model serving to an iterative train-and-inference feedback loop. subgraph SFT_RFT["Training Phase"] A["Custom Domain Data"] --> B["Training Job (SFT / RFT / DPO)"] end subgraph Deploy["Serving Phase"] B --> C["FireAttention Inference Engine"] C --> D["Production Workload at Scale"] end D -->|"Continuous Telemetry & Data Collection"| A ```

Ravi frames the discussion around three fundamental truths shaping today's AI ecosystem: - First, modern open-source models are genuinely production-ready right out of the box, as evidenced by models like GLM-4.7 and GLM-5.2, particularly when applied to challenging coding tasks. - Second, whenever an open-source model falls short of a closed model, customization is the equalizer: “if open source models don't like beat or match the quality of close source models, the differentiating factor comes from how you customize, train and improve these open source models to match the performance and quality of closed source models.” - Third, the overarching AI engineering stack has undergone a structural shift; instead of deploying a model once as a static artifact, organizations are moving to a continuous train-and-infer loop that continuously refines weights on fresh application data. Fireworks AI positions itself as the infrastructure layer designed to handle both ends of this continuous loop cheaply and at massive scale.

---

[03:12](https://www.youtube.com/watch?v=13YZ1EB9hjU&t=192s) Fireworks at Scale: Cursor, Vercel, and Genspark

To demonstrate what this looks like in practice, Ravi shares Fireworks' operational metrics and customer case studies. Fireworks currently handles over 30 trillion tokens in production, a figure recorded prior to the release of GLM-5.2 that has since surged by at least 5 trillion tokens per day. The platform supports more than 10,000 companies globally, sustaining concurrent traffic of roughly 200,000 requests per second. Backing this footprint, Fireworks has raised around $327 million across funding rounds, including a recent Series C, and operates with a team of approximately 200 people.

Ravi highlights three concrete customer deployments demonstrating the platform's production impact: - **Cursor:** Cursor's Composer 2 model is both trained and hosted on Fireworks. Leveraging the custom FireAttention runtime, Composer 2 delivers 13x faster inference speeds. - **Vercel:** Fireworks helped Vercel fine-tune and host their specialized models on the platform, unlocking a 40x improvement in latency. - **Genspark:** Operating in the prompt-driven system-generation space alongside platforms like Lovable, Genspark hit a strict quality bottleneck when attempting to migrate off closed-source models. Fireworks partnered with them to run reinforcement fine-tuning (RFT), clean their training datasets, and craft dedicated evaluator functions, yielding a 12% end-to-end quality improvement.

---

[05:27](https://www.youtube.com/watch?v=13YZ1EB9hjU&t=327s) The FireAttention Engine and Inference Optimization

Achieving extreme speedups like Vercel's 40x latency reduction relies on Fireworks' proprietary inference engine, known as FireAttention. Ravi explains that between a high-level model architecture and the underlying physical GPUs sits a hardware translation layer of CUDA kernels. Instead of using off-the-shelf primitives, Fireworks writes and maintains its own custom CUDA kernels. When customers arrive with defined Service Level Objectives (SLOs), Fireworks tunes the entire inference stack—from the kernel up—to fit the specific workload profile.

| Optimization Layer | Implementation Details | Customer / Operational Benefit | |:--- |:--- |:--- | | **Custom CUDA Kernels** | Built specifically for the FireAttention runtime layer on GPUs | Low-level execution tuned for exact sequence shapes | | **Workload Co-design** | Tailored to fixed ratios of input tokens, output tokens, and prompt cache hit rates | Meets strict target latency budgets at reduced compute cost | | **Serving Techniques** | Implements disaggregation, quantization, and speculative decoding | Higher per-GPU throughput and sustained concurrency | | **Custom Draft Models** | Users train specialized draft models using their own dataset uploads | Accelerates primary model token generation without quality regression |

As Ravi summarizes the engineering relationship with clients: “most of the time customer like versel in the previous slide come to us and tell us they have x input tokens y output tokens they have zprompt to prompt cache hit rate and they want to meet so and so latency targets and we help them optimize this whole inference stack including that cuda kernel so that we can serve this model at a cheaper price but also at a that can meet their performance target and for their specific workload itself.”

Beyond kernel optimizations, FireAttention incorporates system-level techniques including disaggregation, quantization, and speculative decoding. Fireworks uniquely allows customers to train custom draft models directly on the platform by uploading targeted datasets. A well-trained, specialized draft model dramatically accelerates the main model's decoding velocity, driving up overall throughput while honoring latency caps.

---

[07:22](https://www.youtube.com/watch?v=13YZ1EB9hjU&t=442s) Three Flavors of Training: Agent, Managed, and Training API

While latency optimization becomes critical once a model is deployed, Ravi reiterates that the initial roadblock when moving away from closed APIs is almost always model quality. Fireworks tackles this by providing three distinct training modalities aimed at different engineering personas:

```mermaid flowchart LR accTitle: Three Training Offerings at Fireworks AI accDescr: Chart contrasting the Training Agent, Managed Training, and Training API paths. subgraph UI_Driven["Automated / UI"] A["Training Agent"] -->|"Upload dataset & goal"| B["Auto data cleaning, model selection & HP tuning"] end subgraph Managed["Managed Service"] C["Managed Training"] -->|"API or Dashboard UI"| D["Manual HP, loss curves & W&B sync"] end subgraph Advanced["Code-First"] E["Training API"] -->|"Custom script"| F["Custom training loops, losses, SFT / RFT / DPO"] end ```

1. **The Training Agent:** Geared toward product managers and newer AI builders, the agent is accessed via the web UI. Users provide a natural language description of their objective and link their uploaded dataset. The agent automatically selects an appropriate base model, sanitizes and prepares the dataset, and performs hyperparameter exploration to remove the guesswork from fine-tuning. 2. **Managed Training:** The most popular tier across Fireworks' existing customer base. Accessible via both a REST API and a dashboard UI, users select their model, set hyperparameters, supply their dataset, and initiate the run with a single button click. The platform includes full observability dashboards tracking epoch-by-epoch loss progression, and features native integration with Weights & Biases (W&B) by accepting an API key to stream metrics directly to external dashboards. 3. **The Training API:** Designed for ML engineers who require granular control. “this particular offering of the product lets ML engineers define their own training loop and loss functions,” giving teams the freedom to construct non-standard training routines beyond the constraints of standard managed workflows.

These training tiers support Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Reinforcement Fine-Tuning (RFT). Ravi observes that RFT has surged in demand, though she notes that the automated Training Agent does not yet support RFT datasets or reward tuning; those complex workflows—such as dataset curation and evaluator reward function construction—are currently supported via managed services and bespoke team engagements, directly setting up the live walkthroughs of each tool.

[11:08](https://www.youtube.com/watch?v=13YZ1EB9hjU&t=668s) The Continuous Train-to-Deploy Flywheel

Jetashree Ravi frames the continuous train-to-deploy loop as the defining operating model for modern AI systems. Rather than treating training as a static, one-time exercise, teams build a dynamic pipeline where live production feedback directly informs iterative retraining. As she observes: “I know in the first slide I was talking to you about the realities of AI today where most people prefer doing a continuous loop where they train host train host as they get new and new data and feedback from their users and that is what we are trying to make more seamless on our platform and that's also why we have different training products different offerings as well as making it easier uh for you to host and make it also cheaper and faster for you to host those models at scale itself.”

```mermaid flowchart TD accTitle: The Continuous Train-and-Deploy Flywheel accDescr: A cyclical flow diagram illustrating the loop from training to one-click deployment, monitoring, and retraining from checkpoints. subgraph Cycle["The Train-to-Deploy Loop"] Train["Model Training<br>(SFT, RFT, DPO)"] Deploy["1-Click Production Deployment<br>(Hot-loading weights)"] Monitor["Live Monitoring<br>(Cache hit rate, speculator metrics)"] Collect["Feedback & Metric Collection"] Train -->|"One click"| Deploy Deploy -->|"Telemetry"| Monitor Monitor -->|"Gathers data"| Collect Collect -->|"Retrain from checkpoint"| Train end ```

The execution flow begins the moment training wraps up: the user is a single click away from rolling the trained model into a production environment. Once live, the platform exposes detailed operational telemetry, tracking key serving metrics such as aggregate token volume, KV cache hit rate, and the performance and acceptance efficiency of speculative decoding draft models. These production metrics and user interactions feed back directly into retraining cycles, allowing engineers to pick up from an existing checkpoint and produce updated model weights without friction.

- **GPU Partnerships:** Fireworks AI secures access to large GPU fleets by partnering across multiple major public clouds and specialized neocloud providers. - **Continuous Feedback:** The loop closes by taking real-world inputs, model evaluations, and draft-model speculator statistics to trigger incremental checkpoint retraining.

---

[12:42](https://www.youtube.com/watch?v=13YZ1EB9hjU&t=762s) Hot-Loading and Numerical Fidelity

Two foundational infrastructure capabilities make this tight loop practical without degrading developer velocity or model behavior: zero-downtime hot-loading and strict numerical fidelity across the boundary between training and serving. Ravi reveals that hot-loading originally came out of engineering work with Cursor before becoming a core public capability: “Another thing is we offer is hotloading and this has become really popular especially when we were training models for cursor and this was actually a feature we implemented specifically for cursor and then we thought would be useful for general public while they were training models as well.” When iterating rapidly, waiting for new infrastructure or draining existing serving clusters disrupts user traffic; hot-loading swaps out active weights under an existing endpoint instantly so traffic cutovers happen with zero downtime.

Equally critical is numerical fidelity. A common antipattern across the industry is training a model on Platform A and then converting or migrating the weights to serve on Platform B, which often introduces subtle quality regressions and degraded performance. Explaining why this unified setup matters, Ravi explains: “A lot of the time customers use platform A for training and host on platform B. But what happens when you do that is you'll have issues in quality of the model. And the reason you have those issues is humix is not always compatible when you train on a different platform and host on a different platform. So that's why we offer training and inferencing on the same platform and ensure that uh you won't find any numerical differences after you finish training and once you decide to host this particular model on the platform itself.” Unifying both stages on a single engine ensures arithmetic and kernel consistency throughout.

- **Zero-Downtime Hot-Swapping:** Engineers can swap model weights on an active endpoint with a single click, completely eliminating redeployment delays and service interruptions. - **Kernel & Numerics Alignment:** Co-locating training and FireAttention inference guarantees identical mathematical behavior, preventing precision bugs caused by cross-platform weight translation.

---

[14:22](https://www.youtube.com/watch?v=13YZ1EB9hjU&t=862s) Demo: The Training Agent in Action

Ravi shifts into the first live walkthrough, showcasing the autonomous Training Agent designed to abstract away manual configuration for supervised fine-tuning. The user initiates the process by pointing the agent to two datasets stored in the platform: the training dataset and the evaluation/test dataset. Rather than forcing the practitioner to guess optimal hyperparameters, the user delegates model selection and parameter tuning entirely to the agent.

Once the prompt is submitted, the agent generates an explicit execution plan. It details its supervised fine-tuning (SFT) strategy, explains why it selected a specific base model—in this demonstration, Qwen 38B—and outlines its selected hyperparameter configuration alongside a high-level timeline. The human practitioner reviews the reasoning and hits approve. The agent then executes the run autonomously through discrete phases, printing status milestones as step two and step three complete successfully. Upon conclusion, it renders an end-to-end retrospective covering the final training configuration, performance metrics including accuracy and F1 score, and actionable algorithmic next steps for subsequent iterations.

- **User Inputs:** Pointers to the primary training dataset and the evaluation/test split within Fireworks storage. - **Model Choice:** In the demo run, the agent automatically selects Qwen 38B based on dataset characteristics. - **Structured Plan:** Outlines the SFT plan, justifies the architecture, specifies learning rates and training settings, and requests user approval before provisioning compute. - **Post-Run Telemetry:** Outputs validation loss curves, final accuracy, and F1 scores alongside automated suggestions for follow-up runs.

---

[16:17](https://www.youtube.com/watch?v=13YZ1EB9hjU&t=977s) Demo: Managed Training UI and Production Deployment

For teams that want direct control over their training setup without having to write boilerplate pipeline code, Ravi walks through the Managed Training UI. The user uploads their custom dataset, chooses a base model architecture, points to an evaluation set, and opens optional fine-grained settings to manually override hyperparameter controls. Clicking "Create" kicks off the job immediately, populating the job dashboard where developers can track live loss convergence curves and execution metrics. If a team wants to run quick ablation studies, the UI allows them to clone previous runs directly and tweak individual settings without starting from scratch.

```mermaid flowchart LR accTitle: Managed Training and Serving Deployment Flow accDescr: Step-by-step UI pipeline showing dataset selection, fine-tuning run creation, performance inspection, and one-click deployment using latency/throughput profiles. A["Select Base Model<br>& Upload Data"] --> B["Configure Evaluator<br>& Hyperparameters"] B --> C["Click 'Create'<br>(Kick off training job)"] C --> D["Inspect Loss Curves<br>& Metrics / Clone Run"] D --> E["Select Custom Model<br>in Deployment Section"] E --> F["Apply Serving Profile<br>(Throughput / Fast / Minimal)"] ```

Once the model satisfies performance expectations, turning it into a production endpoint requires no specialized infrastructure work. In the deployment panel, the engineer picks their newly trained custom weights and selects an operational serving profile tailored to specific SLAs:

- **Deployment Presets:** Pre-optimized serving profiles—including *Throughput*, *Fast*, and *Minimal*—allow users to instantly tune deployment clusters based on latency requirements, raw throughput, or time to first token (TTFT). - **Run Reproducibility:** Every managed job logs its exact data manifest, hyperparameter dictionary, and loss progression, letting teams clone and re-run experiments with modified variables. - **Developer Cookbooks for Training APIs:** While Ravi notes that an end-to-end demo for the raw Training API is omitted because enterprise pipelines vary widely, she highlights that Fireworks provides extensive cookbook examples for users writing custom loss functions, training loops, and reinforcement learning scripts.

Ravi concludes the session by reiterating that keeping training and inference inside one tightly coupled ecosystem lowers production costs, eliminates cross-platform quality loss, and lets specialized models match or beat closed models in real-world software.

## Notable quotes > [00:12](https://www.youtube.com/watch?v=13YZ1EB9hjU&t=12s) if open source models don't like beat or match the quality of close source models, the differentiating factor comes from how you customize, train and improve these open source models to match the performance and quality of closed source models. — Jetashree Ravi

## Mentioned Jetashree Ravi, Fireworks AI, Cursor, Composer 2, Vercel, Genspark, Lovable, ChatGPT, Anthropic, FireAttention, GLM-4.7, GLM-5.2, Qwen 38B, Weights & Biases

## Video description Closed models are getting expensive fast. Open models can match them, if you keep training them on your own data.

Jetashree Ravi, who leads part of the applied machine learning team at Fireworks AI, explains how teams move from closed models to open ones without losing quality. Her three truths: open models like GLM 5.2 are production-ready, customization closes the remaining quality gap, and the AI stack is shifting to a continuous train-and-serve loop. She shares Fireworks' scale (about 30 trillion tokens, 10,000+ companies) and customer results: Cursor's Composer 2 trained and hosted on Fireworks with 13x faster inference, a 40x latency improvement for Vercel, and 12% better quality for Genspark with reinforcement fine-tuning. She covers the FireAttention engine and its custom CUDA kernels, speculative decoding with custom draft models, and three ways to train (a training agent, managed training, and a training API for SFT, RFT and DPO), then the flywheel: one-click deploys, monitoring, hot-loading new weights with no downtime, and matching numerics between training and serving. She ends with demos of the training agent and managed training.

In this talk: • Why open models plus customization can match closed ones • FireAttention, custom kernels and speculative decoding • Training agent, managed training and the training API • Hot loading and numerical fidelity for a continuous train-and-deploy loop

SPEAKER Jetashree Ravi, Tech Lead Manager, Applied ML, Fireworks AI LinkedIn: https://www.linkedin.com/in/jetashree-ravi/ GitHub: https://github.com/jraviS2023

LINKS Fireworks AI: https://fireworks.ai Managed fine-tuning docs: https://docs.fireworks.ai/fine-tuning/managed-finetuning-intro Supervised fine-tuning docs: https://docs.fireworks.ai/fine-tuning/supervised-fine-tuning

CHAPTERS 0:00 Intro 1:27 Three truths about open models 2:22 Where Fireworks fits 3:12 Fireworks at scale 3:47 Cursor, Vercel and Genspark 5:27 FireAttention and custom kernels 6:27 Speculative decoding and draft models 7:22 Three ways to train 8:22 Managed training 9:27 Training API 10:07 SFT, RFT and DPO 11:07 The data flywheel 12:42 Hot loading new weights 13:22 Numerical fidelity 14:22 Demo: training agent 16:17 Demo: managed training and deploy 18:42 Wrap-up

Recorded at the AI Engineer World's Fair 2026 in San Francisco. Subscribe for more talks from the engineers building with AI.

AI Engineer: https://ai.engineer YouTube: https://www.youtube.com/@aiDotEngineer X: https://x.com/aiDotEngineer LinkedIn: https://www.linkedin.com/company/aidotengineer/

#FineTuning #OpenSourceAI #AIEngineer

[Source](https://www.youtube.com/watch?v=13YZ1EB9hjU)

Reading summary

Jetashree Ravi of Fireworks AI argues that rising proprietary API costs are driving teams toward open-source models, where sustaining competitive quality and cost efficiency requires unifying training and inference into a continuous, co-located development loop. Economic Pressure and Customizing Open Models (0:12) Escalating expenses from closed-source providers such as OpenAI and Anthropic are prompting organizations to evaluate open-source alternatives. Ravi contends that modern open weights like GLM-4.7 and GLM-5.2 are production-ready out of the box for coding and general tasks, but customization remains the critical differentiator when open models fail to match closed benchmarks. To demonstrate enterprise feasibility, Ravi cites customer deployments: Cursor trained and served its Composer 2 model on Fireworks for a 13x inference speedup; Vercel reduced latency by 40x; and Genspark achieved a 12% end-to-end quality gain by pairing reinforcement fine-tuning (RFT) with custom evaluator functions. Fireworks reports hosting over 30 trillion tokens in production—growing by roughly 5 trillion tokens daily—while serving 10,000 companies and handling 200,000 requests per second. Hardware Co-design with Custom Inference Kernels (5:27) Achieving substantial latency reductions requires moving beyond generic GPU runtimes. Fireworks' FireAttention engine utilizes custom-written CUDA kernels optimized specifically for client workload profiles, aligning the software stack with explicit Service Level Objectives (SLOs), fixed token ratios, and prompt cache hit rates. At the system layer, FireAttention integrates disaggregation, quantization, and speculative decoding. To maximize decoding velocity without causing quality regressions, the platform allows developers to upload domain datasets and train custom draft models directly on the serving architecture, driving higher concurrency while respecting tight latency budgets. Three Tiers of Model Training Modalities (7:22) Because model quality is the primary barrier when migrating off closed APIs, Fireworks offers three training paths tailored to different engineering roles. The no-code Training Agent allows users to provide an objective and a dataset via the UI; the agent cleans the data, selects an architecture (such as Qwen 38B in demonstration runs), and tunes hyperparameters after user plan approval. For standard engineering teams, Managed Training provides an API and dashboard interface with explicit hyperparameter controls, epoch loss curves, and direct Weights & Biases logging. Advanced ML practitioners requiring specialized loss functions or non-standard training loops can use the raw Training API. These systems support Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and RFT, though Ravi notes automated RFT dataset and reward tuning currently require managed engineering engagements. Continuous Deployment, Hot-Loading, and Numerical Fidelity (11:08) Ravi frames modern AI engineering as an ongoing flywheel rather than a static, one-time deployment. Once trained, models deploy in one click to production, where monitoring systems track operational telemetry such as KV cache hit rates and draft-model speculation efficiency. These production interactions provide direct feedback for subsequent checkpoint retraining. Two infrastructure capabilities support this loop: zero-downtime hot-loading and cross-stack numerical fidelity. Hot-loading, originally developed for Cursor, allows teams to swap active model weights on live endpoints without draining clusters or incurring downtime. Additionally, co-locating training and FireAttention inference on the same platform prevents the numerical discrepancies and subtle precision degradation that frequently occur when exporting weights across different vendor runtimes. Fireworks positions tightly coupled training and serving infrastructure as the key to surpassing closed proprietary APIs on cost and performance, though advanced reinforcement fine-tuning workflows still depend on bespoke reward modeling and data engineering rather than fully autonomous automation.

Source quotes

[0:12] Jetashree Ravi: if open source models don't like beat or match the quality of close source models, the differentiating factor comes from how you customize, train and improve these open source models to match the performance and quality of closed source models.

Original transcript

[0:12] Hi everyone. Uh before we get started, I [0:15] actually have a quick question for all [0:17] of you. How many of you actually today [0:19] use a closed source model like chat GPD [0:22] or Anthropic? [0:24] Okay. How many of you feel like it's [0:27] getting really expensive really quickly? [0:30] Okay, that's that's a good percentage of [0:33] you all. And uh just like all of you [0:36] might be thinking of switching from [0:38] closed source model to open source [0:40] models for cost reasons, performance [0:43] reasons, there uh other customers and [0:46] users that are going through the same [0:48] journey. And today I'm going to talk [0:50] about where these open source models [0:54] shine and how to make the performance or [0:57] quality of these open source models [0:59] match the closed source models itself. [1:02] And before I get into that quick [1:04] introduction of myself uh I work at [1:06] fireworks. I lead part of the applied [1:09] machine learning team. My name is [1:10] Jatashri but feel free to call me Jet if [1:13] that's easier. Um let's talk about the [1:16] whole train to deploy loop that that the [1:18] open source models are taking today when [1:20] users are switching from closed source [1:22] to open source models. [1:25] But even before we get into that let's [1:27] talk about some fundamental truths that [1:30] are true today in the AI ecosystem. So [1:33] the three fundamental truths that are [1:35] true uh are one open source models for [1:38] the most part are production ready. If [1:40] any of you have used GLM 5.2 two, you [1:44] know that that is true especially for [1:46] the coding use cases. [1:48] Two, if open source models don't like [1:51] beat or match the quality of close [1:53] source models, the differentiating [1:55] factor uh comes from how you customize, [1:58] train and improve these open source [2:00] models to match the performance and [2:02] quality of closed source models. And [2:05] three, the AI stack itself for most of [2:08] our customers and users is shifting from [2:11] let's just serve a model once once and [2:14] that's done to a more continuous loop [2:17] that is improving from training to [2:19] inferencing uh the same open source [2:22] model itself. [2:24] So that's where fireworks kind of comes [2:27] in. It makes the whole process of you [2:29] having to go through the continuous [2:30] journey of training these open source [2:32] models as well as hosting these open [2:34] source models at scale much more easier [2:36] and also much more cost effective as [2:38] well. And I'll get into how we actually [2:41] do the training itself and different [2:42] flavors of training that we offer [2:44] towards the end of the presentation. But [2:46] I did also want to quickly talk about [2:48] inference before we go to the training [2:50] portion of it which has become really [2:51] popular among our customers and user [2:54] base. [2:55] And before we also talk about inference [2:57] and training, I just wanted to give you [2:59] a quick overview of how uh inference [3:02] looks like especially at scales and what [3:05] are the kind of production impact that [3:08] we have seen among some of our customers [3:10] and user base itself. So today at in [3:13] production at scale uh we serve about 30 [3:17] trillion tokens and it's increasing. Um [3:19] this number was actually preGLM 5.2 [3:22] release. And I'm sure this has pretty [3:24] much kind of increased by about at least [3:26] 5 trillion tokens a day. Now with the [3:28] GLM 5.2 release, we support more than [3:32] 10,000 companies globally and process [3:34] about 200k requests a particular second [3:38] itself. And the kind of production [3:40] impact that we are seeing within our [3:42] customer and user base is very [3:45] significant. So cursor uh if for those [3:48] of you who already use cursor and cursor [3:50] models u the composer 2 model for cursor [3:54] was trained and also hosted on our [3:55] platform today and um it is able to [4:00] achieve 13x faster inference because of [4:03] the fire attention engine that we have [4:05] and I'll speak a little bit about that [4:06] in the next slide as well. Uh Versell is [4:09] also another big customers uh of ours. [4:12] We help them actually fine-tune and also [4:14] host the model on our platform and they [4:17] see a 40x latency improvement. And [4:20] another big use case study is actually [4:22] genpark. Um genpark lovables are in very [4:26] similar field where they actually let [4:28] you design systems with just prompts [4:30] itself. And uh quality is a big [4:34] bottleneck or was a big bottleneck for [4:36] them when they were trying to switch [4:38] from closed source to open source [4:39] models. and we helped them fine-tune. We [4:42] did uh RFD for them. We helped them [4:44] clean the data set, define evaluator [4:46] functions and at the end of the RFT [4:49] training job itself, we saw that there [4:51] was 12% quality improvement for the Gen [4:54] Spark use case. Um also we are a trusted [4:58] platform among all our other customers [5:00] and user base as well and we have raised [5:03] around 327 million CDC uh a while ago [5:07] and we are about 200 people working at [5:09] fireworks. That brings me to the next [5:12] portion of it. I know in the previous [5:14] slide I did mention that we made latency [5:16] faster for a couple of our customers [5:18] such as versell and how we do it is [5:22] quite simple at a at a higher level. Um [5:26] at fireworks we have our own inference [5:28] engine called fire attention. So you [5:30] have a model you have GPUs and the way [5:32] the model sits on top of the GPUs is [5:34] through a translation layer called CUDA [5:36] kernels and as part of fire attention we [5:39] build our own CUDA kernel. So when a [5:41] customer comes to us uh with a workload [5:44] and a performance target in mind then we [5:47] can optimize this particular inference [5:49] engine and the whole AI stack including [5:51] that CUDA kernel for that specific use [5:53] case. So most of the time customer like [5:56] versel in the previous slide come to us [5:58] and tell us uh they have x input tokens [6:00] y output tokens uh they have zprompt to [6:04] uh prompt cache hit rate and they want [6:06] to meet so and so latency targets and we [6:09] help them optimize this whole inference [6:11] stack including that cuda kernel so that [6:13] we can serve this model uh at a cheaper [6:16] price but also at a that can meet their [6:18] performance target and for their [6:20] specific workload itself. uh as part of [6:23] fire attention and inference [6:25] optimization, we do also implement the [6:28] very well-known techniques in the AI [6:31] field which is things like speculative [6:33] decoding, disagregation, quantization. [6:36] In terms of speculative decoding itself, [6:39] we also allow customers to train their [6:41] draft models on our platform. Uh so they [6:44] can even upload the data set very [6:46] similar to how they do for fine-tuning [6:48] for uh training a draft model. uh which [6:51] helps them uh make the model the main [6:54] model uh perform much faster uh meet [6:57] latency requirements also gives you [6:59] better throughput if the draft model is [7:01] trained uh very well and um so once [7:05] people actually people actually get to [7:07] this stage mainly once they're happy [7:09] with model quality but when customers [7:13] are switching from closed source to open [7:15] source models the biggest bottleneck or [7:18] the gap is quality and the The way we we [7:21] try to solve for it is through our [7:23] training product and our training [7:25] product itself comes in three different [7:28] flavors for different segments uh within [7:30] our customer and user base. One being [7:33] the training agent. This is very popular [7:36] among our product managers and builders [7:39] that get that got into this AI [7:42] inferencing and training field very [7:44] recently. So the agent can be accessed [7:47] through our UI and uh you can you you [7:51] just need to link it the data set that [7:52] you have uploaded uh on the fireworks [7:55] platform as well as description on what [7:57] you're trying to do as part of this [7:59] training run itself. Uh it will even let [8:01] you choose the model. It will clean the [8:03] data set for you. It will also help you [8:05] do the hyperparameter uh fine-tuning [8:08] itself because it's not always easy to [8:10] get the right uh hyperparameters itself. [8:14] um when you are doing training for your [8:16] specific workload. The next flavor also [8:20] one of our most popular flavors among [8:22] our customer and user base is our [8:24] managed training and this is what most [8:26] of our customers use today. They can use [8:30] our managed training platform one [8:33] through API and second through the UI [8:36] itself. It is as simple as you inputting [8:39] the hyperparameters that you think will [8:41] be good for your model. choosing a [8:43] model, uploading the data set itself and [8:46] then clicking a button. That's how [8:48] simple we make it on uh fireworks for [8:50] the manage training itself. And uh I [8:53] know one challenge a lot of people have [8:55] is observability once the training run [8:57] itself kicks in. So we also have a good [9:00] UI dashboard that shows you how the [9:02] training uh is happening, how well your [9:04] model is uh learning at each epoch, what [9:07] does the loss function look like. [9:09] Additionally, you can also plug in u [9:12] your weights and biases API key into the [9:15] manage training service itself uh and [9:17] export some of the observability metrics [9:20] to platforms like weights and biases if [9:22] you're already using that. And the third [9:24] and the kind of the most advanced flavor [9:27] of training that we offer is the [9:29] training API itself. [9:32] And the training API is very popular [9:34] among some uh of the ML engineers within [9:37] uh different organizations or within our [9:40] customer base. And in this in this [9:43] particular this particular offering of [9:45] the product lets uh ML engineers define [9:49] their own training loop and loss [9:51] functions. And this becomes important [9:53] when you want to customize your training [9:55] job according to your needs. that is [9:58] that goes well above beyond the managed [10:01] training service that we offer itself [10:03] and these these different uh flavors or [10:07] the products within the training [10:09] ecosystem that we offered are offered [10:12] for supervised finetuning reinforcement [10:15] fine-tuning and even DPO as well. Um [10:18] nowadays I think RFT has become [10:21] extremely extremely popular and if you [10:24] ever need help in helping define a good [10:26] evaluator function then we have worked [10:28] with customers like Genpark that I was [10:30] talking to you about in the previous [10:32] slide to help them define an evaluator [10:35] function clean the data set um and also [10:38] choose the right hyperparameters [10:39] especially for RFD. Unfortunately the [10:42] training agent itself does not support [10:44] RF. So it does not uh help you clean the [10:47] data set or even do the hyperparameter [10:49] tweaking but we are more than happy to [10:52] help you do it if you have a big use [10:53] case in mind. So now that we know all [10:57] about the different um training [10:59] offerings that we have I also wanted to [11:01] show you a quick demo of each of these [11:03] the training agent manage training and [11:05] the training API itself. Uh but before [11:08] we get into that I just wanted to close [11:10] off by saying how the data flywheel [11:13] looks like today on our platform. I know [11:15] in the first slide I was talking to you [11:17] about the realities of AI today where [11:20] most people prefer doing a continuous [11:22] loop where they train host train host as [11:25] they get new and new data and feedback [11:27] from their users and that is what we are [11:29] trying to make more seamless on our [11:31] platform and that's also why we have [11:33] different training products different [11:35] offerings as well as making it easier uh [11:38] for you to host and make it also cheaper [11:41] and faster for you to host those models [11:43] at scale itself. And the way this [11:45] flywheel works is kind of very simple. [11:48] Once these uh models are trained, I will [11:50] show you uh a demo in the next slide [11:53] where the models get trained. It's it's [11:55] you just are one click away from [11:57] deploying these models into production. [12:00] And once you've deployed, you can uh [12:02] monitor. We have u monitoring metrics [12:05] for the number of tokens, cash it rate, [12:08] how your speculator is working, if [12:10] you've trained a speculator. uh we help [12:12] you collect gather uh feedback from all [12:15] the all the metrics that you have [12:17] collected and if you choose to also [12:19] retrain on the same platform from the [12:21] end from the checkpoint that you were [12:24] using to a new checkpoint itself. So [12:27] some of some capabilities on the [12:28] platform that makes these data flywheel [12:30] possible is one definitely having a host [12:33] of uh GPUs. We do have we do partner [12:36] with different cloud and neocloud [12:38] providers which makes it easier for us [12:40] to provide compute and also host these [12:42] models at scale. Another thing is we [12:45] offer is hotloading and this has become [12:47] really popular especially when we were [12:49] training models for cursor and this was [12:53] actually a feature we implemented [12:54] specifically for cursor and then we [12:56] thought would be useful for general [12:57] public while they were training models [12:59] as well. uh what hot loading essentially [13:01] means is when you have a new checkpoint [13:04] uh you you can directly go to live [13:06] deployment without any downtime itself. [13:09] So if you have a deployment and you've [13:11] gone through this continuous loop and [13:13] trained a new checkpoint then you can [13:15] just click a button and swap out the [13:17] weights itself that you were using uh uh [13:20] previously with the new endpoints. [13:22] Another thing that makes it uh easier to [13:25] train on our platform is the numerical [13:27] fidelity. So a lot of the time customers [13:30] use platform A for training and host on [13:32] platform B. But what happens when you do [13:35] that is you'll have issues in quality of [13:38] the model. And the reason you have those [13:40] issues is humix is not always compatible [13:42] when you train on a different platform [13:44] and host on a different platform. So [13:46] that's why we offer training and [13:48] inferencing on the same platform and [13:50] ensure that uh you won't find any [13:53] numerical differences after you finish [13:55] training and once you decide to host [13:57] this particular model on the platform [13:58] itself. And um one other thing that that [14:02] makes it more seamless is because [14:04] training and inference is on the same [14:05] platform. You're just one click away [14:08] from uh deploying this model at [14:10] production scale. [14:12] Now that we have learned all about uh [14:15] the inference and training, I just [14:16] wanted to show you quick demo of [14:18] different training products and how they [14:21] work on our platform. The first portion [14:23] of the demo is showing you how you can [14:25] use the training agent itself. Over here [14:28] you can uh input where your training [14:31] data set lives within the fireworks what [14:33] it's called. Uh you can also uh give it [14:38] the eval data set itself. uh I know [14:41] people have the train and eval or a test [14:44] data set separately. So you give the mo [14:46] you give the agent both the data sets [14:48] and then you ask it uh to fine-tune and [14:51] even select the hyperparameters and the [14:53] model for you. The the agent then goes [14:56] on and tells you the step that it will [14:59] it will take to accomplish the task that [15:01] you have given it. So it it details the [15:03] SFT plan. It tells you why it selected [15:06] the base model. In this case, it's [15:08] selected quen 38B. [15:11] And it'll also give you some information [15:13] on the hyperparameters and what the [15:15] whole training process can look like at [15:17] a high level. And once you're done uh [15:20] once you are uh you feel okay about the [15:22] output that the agent has given, you can [15:24] approve whatever the agent has uh the a [15:27] the agent has outlined. And once you [15:29] approve it, it will actually go through [15:31] different steps and keep you posted on [15:33] if it finishes step two, what does uh it [15:36] look like? If it successfully finishes [15:38] step three, what does that look like [15:40] now? And at the end of the whole [15:42] training run, it will also give you a [15:44] summary of uh the config, the base model [15:49] um and uh different hyperparameters and [15:52] how that impacts it as well. [15:59] And once it's once the whole model [16:01] training is done, it actually gives you [16:03] metrics such as accuracy and F1 score. [16:06] Uh and it also summarizes kind of [16:09] resummarizes and gives you an overview [16:11] of everything that uh that it did. And [16:13] one cool feature is it also gives you [16:15] recommended next step as part of u the [16:18] training run itself. Now the next [16:21] portion of uh this demo actually talks [16:24] about how we can use our uh manage [16:26] training platform. I did mention that [16:29] you just have to select the model data [16:31] set and the hyperparameters and then [16:32] you're just a button click away from [16:34] actually training a model. Say you've [16:37] uploaded uh the data set, chosen a [16:40] model. Uh all you'll have to do is then [16:43] uh go to the go to the u evaluator [16:47] settings, choose the evaluator data set [16:50] and also you get to choose the [16:51] hyperparameter as part of optional [16:53] settings if you decide to have a more [16:55] fine grained hyperparameters for your uh [16:58] training run itself. [17:00] And once you're happy with all the [17:02] configurations including the model and [17:05] the data set itself, you just need to [17:07] create uh you just need to click on the [17:09] button that says create. Um and as soon [17:13] as you click on the button that says [17:14] create, a training job then kick kicks [17:17] out. And you can see that uh the UI [17:20] looks similar to this when a training [17:22] job actually gets completed. [17:26] And if you click on that particular [17:28] training job itself, it'll give you [17:30] information on the data set it used, the [17:32] hyperparameters it used, what the model [17:34] loss itself looks like when the model [17:36] was training. Um, and then you can even [17:39] clone these training runs and reproduce [17:41] it with different settings and [17:43] configurations itself. [17:47] And once you're happy with the [17:48] fine-tuned model, you can go to the [17:50] deployment section, choose the custom [17:53] model that just got uh trained on our [17:55] platform, and then click on deploy once [17:58] you're happy with the deployment [18:00] configurations. Or you can choose a set [18:03] of deployment configurations that we [18:05] have such as throughput, fast, minimal. [18:09] uh and this is uh kind of a template [18:11] that that works well for generic [18:14] workloads that we have optimized based [18:15] on throughput latency uh time to first [18:19] token and other such metrics [18:23] and uh I unfortunately don't have an [18:26] entire demo for our training API because [18:29] the setup itself differs from customer [18:32] to customer but we do have a lot of [18:34] cookbooks on our websites that you can [18:36] see and follow especially if you're [18:38] interested in customizing your own [18:41] training loops and loss functions [18:43] itself. Uh and that's all I have for you [18:46] for today. I just wanted to give you a [18:48] brief overview of how the whole training [18:50] to inferencing loop looks like. Thank [18:52] you very much. And if you have any [18:54] questions, I'm happy to take them in the [18:55] next minute or so that we have.