Scaling 1,600 production agent workflows with Qualcomm

Qualcomm’s customer engineering teams support hundreds of global customers daily across dozens of SDKs and product lines. The work is difficult for agents for the same reason it is difficult for new engineers: partial success does not count. An answer can be fluent and mostly correct and still fail if it cites the previous silicon revision, misses an errata note, or skips one step of a board bring-up sequence.

The baseline system, a leading foundation model with tool access but no access to institutional knowledge, passed fewer than 23% of tasks. The model had never seen the proprietary specifications, internal documentation, help-center content, or decades of support history that determine whether an answer is correct at Qualcomm.

Every number in this study is measured against a held-out suite of production workflow tasks, each with a task-specific rubric. Grading is LLM-as-judge with three independent grading passes per task; we report the task-averaged mean to control for grader variance. A task passes only if all of its rubric criteria pass.

Before any agent touched production, we built a golden set of real support tasks with the pilot team, each with a task-specific rubric. That held-out suite gated every change shipped over the following four months.

Scoring every change against the same suite is what let the deployment move quickly: regressions surface immediately, so improvements compose instead of interfering.

To locate where the 23→98 jump comes from, we ran an ablation: hold the model weights and harness fixed, and add one context system at a time. Retrieval over indexed documentation recovers the largest single share of failures: the model stops inventing register maps and starts citing them. The virtual directory and ContextFS mounts close most of the remaining gap on code-adjacent tasks. Learned memories and sleep-time consolidation convert the long tail: tasks whose answers exist nowhere in any document, only in accumulated support history.

No stage in the ladder involves fine-tuning; the weights are identical in every configuration. This is also why the gap closed in weeks: each stage is a configuration change that ships like software, without a training cycle.

We logged every eval-gated change shipped during the climb. Plotted as residual failure rate on a log axis, the trajectory is close to a single exponential: each change removed a roughly constant fraction of the remaining failures, with a fitted half-life of about five iterations. The failure rate fell from 77% to 2% over 28 shipped changes.

No single change produces a visible knee in the curve; the gains are spread across many small fixes. This matches how the loop operates: graded traces identify the largest remaining failure class, a fix ships behind the eval, and the process repeats. The half-life stayed roughly constant across the four months, including after the major context systems from the ablation had already landed.

The variance collapses alongside the mean: the weekly spread of daily scores narrows from roughly nineteen points in week one to under two points by week sixteen. As memories consolidate, repeated task shapes retrieve the same context and produce the same outcome.

The original deployment plan targeted a 20% productivity improvement over four years. The measured result at month four was 40%. The over-shoot comes from shared knowledge infrastructure: each new team starts from everything previous teams’ agents already learned. Onboarding a team takes one to two days: connect data sources, define agents and workflows, and run.

By month three, the fleet’s 1,600 rubric-graded workflows were producing a continuous stream of verified traces, each a (task, context, trajectory, grade) tuple. This is the corpus the distillation pipeline trains on. The pipeline runs automatically in four stages:

The distilled 32B sits within two points of the teacher. Removing learned memories costs the distilled 8B eight points, which indicates the students inherit the teacher’s retrieval behavior rather than memorizing the graph’s contents; they stay current as the graph continues to learn.

Without institutional context, additional model capability buys little: the frontier is nearly flat, because no model at any size has access to Qualcomm’s internals. With the context graph and distillation, the fleet operates on a strictly dominant frontier. The 8B student resolves tasks at roughly 1/45th the teacher’s cost while giving up six points of pass rate, and the router reserves teacher-tier inference for tasks outside the students’ verified envelope.

To understand what is driving the score improvements, we compared early-deployment rollouts against month-four rollouts on the same held-out tasks. Three behavioral shifts account for most of the gain:

Procurement evaluations of this deployment came down to four questions: time to value, cost at scale, whether the system keeps improving, and whether security can sign off. The table separates figures measured in this deployment from figures modeled on its unit economics.

On governance: the entire system runs inside Qualcomm’s VPC with zero data egress. Agents authenticate through SSO and inherit the permissions of the engineer they act for; there is no separate agent permission model for security to reason about. Proprietary source is exposed to retrieval as metadata only via the virtual directory, and every agent action lands in an auditable trace, the same traces the eval and distillation pipelines consume.

Across four months at Qualcomm, the binding constraint was context, not model capability: the same weights went from 23% to 98% through changes in what the model could see at runtime. Shared knowledge infrastructure kept scaling cheap, with 85 teams and 1,600 workflows onboarded and per-team cost falling throughout. The production fleet then became its own training corpus: automatic distillation converted verified workflow traces into small open-weight models that hold most of the accuracy at a fraction of the cost.

The deployment continues to improve past the measurement window: sleep-time compute writes new memories, and students are periodically re-distilled as the graph changes. We expect further gains from extending distillation to more workflow tiers and from routing on predicted task difficulty rather than task type.

For the workflows behind these numbers and how the platform maps to chip design, fab operations, and field diagnostics, see Context for semiconductors.