Measure and improve a factory

Warp Factories tracks what your factory produces and how well it performs, so you can spot a problem, test a fix, and decide whether to keep it.

Feature	What it tells you
Dashboard metrics	How much work the factory produced, and what it cost.
Scorers	Whether completed conversations meet criteria you define.
Benchmarks	How different configurations perform on the same tasks.
Self-improvement	Which repeated failures get investigated and turned into follow-up work.

Read metrics on the Dashboard page

The Dashboard page shows activity, cost, autonomy, and evaluation results:

Metric	What it shows
Total runs	All agent runs, with breakdowns by agent type, status, source, model, and more.
PRs opened	Pull requests created from factory work.
PRs merged	Pull requests that merged.
Autonomy	The share of the factory’s merged PRs that needed no human input beyond an approving review and the merge itself.
PR cycle time	The median time the factory’s merged PRs took from run kickoff through PR, first review, and merge, with a median for each stage.
Cost per PR	An estimate of credits spent per pull request.
Most expensive PRs	The highest-cost pull requests.
Scorer cards	Results from your Scorers.
Self-improvement PRs	The three newest Self-improvement pull requests, regardless of the selected date range.

Cost per PR is an estimate, not a billing figure: it counts recorded credits and can undercount actual usage.

Caution

PRs merged, Autonomy, PR cycle time, and the detail in Most expensive PRs require the GitHub App and only cover activity from after you install it.

Use the dashboard to pick which runs to investigate, not to conclude what caused a change. Total runs includes evaluation, benchmark, and Self-improvement runs, so a higher run count with a flat PR count could mean harder tasks, retries, or measurement activity.

Configure Scorers

A Scorer uses an LLM judge to classify completed conversations against criteria you write, such as “did the agent run the tests before opening a PR?” Scorers classify conversations rather than grading them on a numeric scale. Keep each Scorer focused on one question so its failures point to a specific fix.

Configure these fields:

• Agent(s) to evaluate - The agents this Scorer applies to. Select at least one.
• Judge instructions - The criteria the judge checks for.
• Judge model - The model that acts as the judge.
• Classifications - The labels the judge can assign, each with a score.
• Pass threshold - The score a conversation needs to pass.
• Sample rate - The share of the selected agents’ completed conversations to evaluate.

While the sample rate is above 0, scoring runs automatically: shortly after a sampled conversation completes, the judge evaluates it and records a classification, a score, and its reasoning. To stop automatic scoring, set the sample rate to 0.

You can also score any single conversation on demand, which is useful for testing new judge instructions before raising the sample rate. Scoring a conversation again replaces its previous result from that Scorer.

Changing Pass threshold updates how past scores display as pass or fail; the recorded results don’t change.

Compare configurations with benchmarks

A benchmark compares configurations of a single agent on the same fixed tasks, so you can test a model, harness, or runner change before adopting it. A benchmark suite includes:

• Agent - The agent whose configurations you compare.
• Tasks - Fixed prompts with success criteria.
• Configurations - The harness, model, and runner combinations to test.
• Scorers - Your classification Scorers, applied to every trial.
• Repetitions - The number of trials per task and configuration.

You can create a benchmark task from a completed run’s detail pane, and Warp copies the run’s input into the task. Add success criteria before you launch.

Every benchmark also runs Correctness, a built-in Scorer that marks each trial as pass or fail against the task’s success criteria. Results show pass rates, cost, and quality for each configuration, with per-task detail. Warp doesn’t combine these signals into one score or pick a winner; you weigh the results and decide. Benchmark credit totals don’t include model usage, so the true cost is higher.

Configure and review Self-improvement

Turn on Self-improvement for each Scorer whose failures you want investigated automatically. Self-improvement groups related failures and files follow-up tasks as ordinary agent runs. A follow-up run can propose changes to application code. It can also improve the factory itself: when you manage your factory as definitions as code, its prompts, skills, and configuration are version-controlled files, so a follow-up run can open a pull request against the factory definition the same way it would against application code. Nothing is adopted without your review.

Each Self-improvement pull request includes a Regressions addressed section that links the failing runs and Scorer results behind it, so you can trace the change back to its evidence.

Run a practical improvement loop

Change one measurable thing at a time:

flowchart LR
  Define[Define a Scorer] --> Baseline[Collect a baseline]
  Baseline --> Inspect[Inspect failures]
  Inspect --> Benchmark[Benchmark a candidate]
  Benchmark --> Adopt[Review and adopt]
  Adopt --> Monitor[Keep monitoring]
  Monitor --> Inspect
  Inspect -.->|Repeated failures| Improve[Self-improvement]
  Improve -.-> Adopt

1. Define a Scorer. Pick one agent and one failure mode you can observe. Write the judge instructions and classifications, then score a few conversations manually and compare the judge’s results against your own review.
2. Collect a baseline. Let automatic scoring run until results reflect normal work. Record the Scorer settings, date range, and relevant costs.
3. Inspect failures. Read the judge’s reasoning and the underlying conversations. Look for causes like missing context, unclear instructions, or missing tools. Turn on Self-improvement when the same failure keeps repeating.
4. Benchmark a candidate. Compare configurations of that agent on the same tasks, with enough repetitions to trust the difference.
5. Review and adopt. If the evidence supports the change, make it. Review Self-improvement pull requests with the same standards as human-authored ones.
6. Keep monitoring. Leave the Scorer active and compare new results against your baseline. Revise the Scorer, or set its sample rate to 0, when its criteria no longer match what your team needs.

Next step

Record an adopted change in factory definitions as code so your team can review the factory configuration.