Replay / shadow evaluation

VectorStep already records everything needed to answer “is this prompt/model/agent change safe to promote?” — per-step rendered prompts and outputs, prompt_hash + agent_version bucketing (see Confidence), human/ deterministic outcome labels with a defined precedence, and promotion-readiness criteria. But editing a prompt resets its calibration bucket by design, and the only way to earn evidence for the new configuration was previously to run it forward in real traffic and wait for marked outcomes to accumulate.

Replay closes that loop: take the most recent labelled step executions from an existing bucket, execute a candidate configuration (a different model, agent, and/or prompt) against the same recorded inputs, auto-grade what can be auto-graded, queue the rest for human marking, and show a side-by-side report — recorded config vs. candidate — an owner can use to decide on promotion.

The safety gate: replay.safe_agents

Replay executes real agent calls against live tools. That’s the point — a mocked tool layer would grade the candidate on stale fiction — and the danger, since a side-effecting agent (one that files tickets, pages people, mutates state) could re-fire actions it already fired once when it was recorded.

replay:
  safe_agents:
    - "gateway:sre-investigation"
    - "openclaw:incident-triage"

safe_agents is an explicit allowlist of "executor:agent" identities the operator asserts are read-only. A replay request is rejected with 403 unless both the recorded step’s agent and the candidate’s agent appear here — even if only the model is changing, the recorded agent still has to be on the list. No replay: block (or an empty safe_agents) means replay is off entirely; every request 403s pointing back at this config key.

Two modes

rendered — resend the recorded step’s exact rendered prompt to the candidate model/agent, verbatim. Valid when the candidate changes model and/or agent only, not the prompt template. Requires the executor that produced the recorded sample to have persisted the actual rendered prompt text (the gateway and openclaw executors both do, via raw_response["prompt"]); a sample recorded before that persistence existed, or by an executor that doesn’t stash it, is reported as unreplayable rather than guessed at.

rerender — render the candidate’s prompt template against the recorded step’s reconstructed Jinja context (the same {{steps.*}}, {{labels.*}}, {{vars}} a production run would have seen). Required whenever the prompt template itself is changing. If context reconstruction fails for a sample — the owning pipeline is no longer loaded, a prior step’s persisted output doesn’t parse, an artifact has aged out of retention — that sample is marked unreplayable, never silently dropped from the report.

Sample selection

A batch samples the most recent K labelled executions of the source bucket, stage: production only (default K=20, hard cap 100). “Labelled” means the same label-precedence chain Confidence uses everywhere else: a human step mark, else a deterministic-check failure, else a run-level fallback — never an unlabelled row. The bucket itself is either an explicit selector (agent, model, provider, prompt_hash, agent_version) or "current", which resolves to whatever bucket the step’s single most recent production execution actually landed in.

The report’s recorded-accuracy figure is exactly this sample set’s existing numbers — the same bucket Calibration would report for it, not a separate recomputation.

Grading the candidate

• Deterministic checks declared on the step run automatically against the candidate’s output. Same asymmetry as production: a failure auto-labels the candidate 0.0 (a strong, computer-verified negative signal); a pass proves nothing on its own and leaves the sample unmarked.
• Everything else goes to the marking queue, where replay-produced steps are flagged REPLAY, and a human marks them exactly like a production step.
• The report recomputes live on every request — there’s no persisted report artifact, so a mark submitted a moment ago is already reflected.
• An optional LLM auto-judge can grade what deterministic checks and human marks leave ungraded — see below. Off by default; a deployment that never sets replay.judge behaves exactly as described here.
• Verifier and grounding never fire during replay. The candidate runs bare — primary call only — so the comparison isolates the one variable under test, and it’s cheaper.

LLM auto-judge (optional)

The judge never touches production calibration or promotion readiness. This is the first thing to trust about it: a judge verdict lives only inside the replay batch’s own report. It is never written into the label precedence chain that Calibration and Promotion readiness read from, so a model’s opinion about a candidate can never contribute to the empirical trust that gates real production runs — by construction, not by convention.

Enable it with a replay.judge block, alongside safe_agents:

replay:
  safe_agents:
    - "gateway:sre-investigation"
    - "gateway:replay-judge"   # the judge is a live agent call too — see below
  judge:
    agent: replay-judge          # default shown; a Gateway agent you configure
    executor: gateway             # default
    timeout_seconds: 120           # default
    max_output_chars: 4000         # default; per side (reference/candidate), before truncation

Absent entirely (no judge: key) means no judging at all — replay grades on human marks and deterministic failures only, exactly as described above.

The judge’s agent must be in safe_agents too. It fires another live agent call in the same batch as the candidate, so an operator who configures a judge and forgets to allowlist it gets a clear 403 at launch — before any candidate executes — rather than a batch that silently grades nothing.

Precedence

A sample’s candidate is graded by the first of these that applies:

1. Human mark — ground truth, nothing overrides it.
2. Deterministic check failure — an objective, operator-written assertion that already failed is dispositive. A judge saying “looks right to me” over a failed check would be the system talking itself out of hard evidence using a soft signal, on exactly the sample where the hard evidence matters most. (A passing check produces no label of its own — same asymmetry as everywhere else — so it never blocks the judge.)
3. LLM judge verdict — only reached when the first two have nothing to say.
4. Nothing — the sample stays ungraded, exactly as without a judge.

Only recorded-correct samples are judged

The judge compares the candidate’s output to the sample’s recorded output — so that comparison is only meaningful when the recording is itself known to be correct. A sample recorded incorrect has no reference worth matching (there are many ways to be wrong); a sample recorded partial has no specific enough reference to compare against. Both are skipped, visibly: the report’s judge_skipped_n and its breakdown say why each sample wasn’t judged, so “the judge graded 12 of 20” reads as design, not as 8 failures.

One consequence worth expecting: a bucket with poor recorded accuracy gives the judge very little to grade against. That’s correct behaviour, not a limitation — a mostly-wrong step has few known-good references to check a candidate against, and no amount of judging manufactures one.

Reading the report

Judge-labelled and human/deterministic-labelled accuracy are reported separately, never blended:

• candidate_accuracy_so_far — unchanged meaning: human marks and deterministic failures only. A report from before this feature existed and one from after use this field identically.
• judge_accuracy — judge-labelled samples only. Read this as “how often a model thought the candidate matched,” not as evidence of correctness on its own.
• combined_accuracy — all sources blended, clearly labelled as mixed, for a quick overall read.

A judge-labelled row in the UI is visually distinct from a human-marked one, shows the judge’s one-sentence reason, and never removes the “mark this” affordance — a judge verdict is a head start on review, not a replacement for it, and a human mark always overrides it (including one that arrives after the judge already ran).

Cost

A batch with judging makes up to K extra LLM calls (bounded below K by the skips above) — at k: 100, up to 100 extra calls. There’s no separate cap; k already bounds it. The cost preview below includes the judge as a second component once it has at least one past batch’s usage to estimate from; until then it reports the judge’s cost as unknown rather than guessing. The judge’s actual token usage is always logged at batch completion, and visible per-verdict in the report, regardless of the preview.

Writing the judge agent

The judge is a Gateway agent like any other — see Writing your grounding judge for the shared failure modes of any constrained comparison judge (model choice, keeping tools: [], sizing max_tokens): the replay judge is answering a different question (does this match a known-correct reference, not is this claim backed by evidence), but the same discipline applies. Its built-in prompt explicitly tells the judge to grade substance over wording, to treat the recorded reference as the standard rather than a matter of taste, and to prefer partial over incorrect when a truncated comparison lands right at the cut point.

Cost preview

A replay batch makes K live LLM calls with no confirmation step by default — k accepts up to 100, and the whole point of replay is testing a candidate, often a more expensive model than production. Before launching, the “Replay against candidate…” form shows a live, advisory cost estimate that updates as you change the mode, model, agent, or K — it never blocks submit, and a failed or pending estimate just leaves the form usable with nothing shown.

The estimate multiplies a mean of the source bucket’s own recorded token counts by the candidate model’s rate, so it’s always presented with its basis, never as a single confident number:

≈ $0.42 for 20 samples · based on 18 recorded calls averaging 3,100 in / 850 out tokens · live catalog price, cross-provider

• A more verbose candidate will exceed it. The estimate assumes the candidate’s token usage resembles the recorded bucket’s — a reasoning model in particular can produce far more output tokens than the model it replaces, and reasoning tokens bill as output. rerender mode can also change input length, since it rebuilds the prompt from current config.
• Unpriced reports unknown, never $0.00. If the candidate model isn’t in your pricing table or the live catalog, or the bucket has no recorded token usage at all, the estimate says so explicitly rather than rendering a misleadingly cheap number.
• The tier label means what it means everywhere else. manual (your own pricing.models table), live_exact (a live OpenRouter price for the exact model), or live_cross_provider (a live OpenRouter price borrowed from a different provider — the weakest, most approximate tier) — same styling as a run’s persisted cost badge, so a weak estimate looks weak the same way everywhere.
• If replay.judge is configured, the preview adds the judge as a second line — but only once at least one past batch has recorded real judge token usage for this step to estimate from; a step that has never been judge-graded before shows the judge’s cost as unknown, not a guess. The judge multiplier is capped at how many of the K samples are actually judge-eligible (recorded label correct — see above), not the full K, since the judge never runs on the rest.
• Never blocks, never gates. The preview doesn’t add a confirmation round trip to the launch API — POST /steps/{step_name}/replay keeps its exact existing contract. The estimate has its own endpoint the UI (or any script) can call separately.

Where the results live

A replay batch is stored as an ordinary pipeline run: one synthetic run per batch, stage: testing, with a replay_of descriptor recording the source bucket, the candidate, the mode, and which recorded samples map to which candidate step executions. Its steps carry the candidate’s prompt_hash/agent_version — so if the candidate is later promoted, these replay-produced marks stay stage: testing and are excluded from the new production bucket. That’s intentional: replay evidence informs the promotion decision, it doesn’t pre-seed the production track record. Production trust is still only earned in production — see Testing vs production stages for why that boundary exists and what else it protects.

Because it’s an ordinary stage: testing run, every metric/aggregate surface that already excludes testing traffic excludes a replay batch automatically — no new exclusion logic anywhere. It shows up in browse surfaces (the runs list, the marking queue) with the usual TESTING badge, and Prometheus exposes it specifically as vectorstep_replay_batches_total and vectorstep_replay_steps_total{grade} (grade: completed, deterministic_failed, execution_failed, unreplayable) — see Observability.

API

# Launch a batch — blocks until every sample has been attempted
POST /steps/{step_name}/replay
# body: {
#   "bucket": "current",                          # or {agent, model, provider, prompt_hash, agent_version}
#   "candidate": {"model": "...", "agent": "...", "prompt_template": "..."},  # any subset
#   "mode": "rendered" | "rerender",
#   "k": 20                                        # optional, default 20, max 100
# }
# → {"status": "completed", "run_id": "<synthetic-run-id>"}

# Recomputed live on every call — reflects marks as they arrive
GET /replays/{run_id}/report
# → {
#     step_name, mode, source_bucket, candidate, k,
#     recorded_distribution: {correct, partial, incorrect},
#     recorded_accuracy, candidate_accuracy_so_far,
#     candidate_graded_n, candidate_total_n, unreplayable_count,
#     judge_accuracy, judge_graded_n,             # judge-labelled samples only, or null/0
#     judge_skipped_n, judge_skipped_breakdown,   # {} / 0 when replay.judge is unset
#     combined_accuracy,                          # human + deterministic + judge, mixed
#     confidence_distribution: [...],
#     rows: [{sample_step_id, recorded_label, recorded_label_source,
#             status, candidate_step_id, candidate_confidence,
#             candidate_summary, deterministic_passed, mark_outcome,
#             candidate_label?, candidate_label_source?,
#             judge_verdict?, judge_reason?, judge_skipped_reason?}, ...]
#   }

judge_accuracy/judge_graded_n/judge_skipped_n/judge_skipped_breakdown/ combined_accuracy are present on every report, including one from a batch run before replay.judge existed or with it left unset — they just carry harmless empty/null values (judge_accuracy: null, the two counts 0, judge_skipped_breakdown: {}, combined_accuracy equal to candidate_accuracy_so_far). candidate_accuracy_so_far itself keeps exactly its original meaning either way.

# Advisory cost preview — same body as launch, but makes no LLM calls and
# creates no run row. Never gates the real launch above.
POST /steps/{step_name}/replay/estimate
# body: identical to POST /steps/{step_name}/replay
# → {
#     total, per_sample, sample_count,               # candidate calls only; total is null, never 0, when unpriced
#     mean_input_tokens, mean_output_tokens,          # the recorded bucket's own averages this is based on
#     tier, basis_n, reason,                          # tier/reason null when total is null; reason may still
#                                                      # appear on a priced result as a caveat (e.g. a guessed provider)
#     judge: {                                        # present only when replay.judge is configured
#       total, per_sample, eligible_count,            # eligible_count = at most K, capped to judge-eligible samples
#       mean_input_tokens, mean_output_tokens, tier, basis_n, reason,
#     } | null,
#     combined_total,                                 # total + judge.total when both known, else null
#   }

Concurrency

Fixed at 3 concurrent candidate executions per batch — sequential is too slow at K=20, unbounded risks hammering a provider. Not configurable.

UI

Minimal by design: a “Replay against candidate…” link on the step insights page opens a plain form (/ui/replays/new?step=...) for the candidate model/agent/prompt/mode/K, with the live cost preview beside the K input; on submit it blocks until the batch finishes and lands on the report page (/ui/replays/{run_id}). Marking a replay-produced step happens on the ordinary run detail page, same widget as any other step.

Scheduled regression replay (golden sets)

Everything above is an operator-triggered batch, answering “would this candidate be safe to promote?” Scheduled regression replay answers a different question — “has the config that’s already live silently gotten worse?” — by re-running a fixed, curated set of past executions against the pipeline’s current production config on a cron schedule, so a provider changing a pinned model’s behaviour or a Gateway agent config drifting is caught by the system rather than by a user.

Golden sets: pinned, not “most recent K”

A golden set is a small, curated collection of specific past step executions an operator hand-picks — not a bucket selector. This is deliberate: ordinary replay samples “the most recent K labelled executions,” which silently changes as new production runs age old ones out of the window. Two nightly runs a week apart would then be comparing against different samples, so a difference between them would mean nothing. A regression suite needs a fixed baseline, so a golden set pins specific pipeline_steps rows by id and never substitutes new ones in on its own.

Curate one from the run detail page — a “+ Add to golden set” link appears next to every completed step, since curation happens naturally while you’re already reviewing that step’s output, not in a separate admin screen you have to remember exists. Manage golden sets (list, view members, re-pin, remove, delete) from Golden sets…, linked off the step insights page next to “Replay against candidate…”.

Unlike ordinary replay’s bucket-scoped sampling, a golden set’s members are not required to share a bucket — spanning models and prompt versions across members is a feature here, not an oversight, since the question is “does the current config still handle these specific cases,” not “how does this bucket compare.”

What this can and cannot detect automatically — read this first

Without deterministic checks or an LLM judge on the step, a scheduled regression run grades almost nothing. At 3am, nobody has marked anything by hand, so the only automatic label available is a deterministic-check failure — and the same asymmetry as everywhere else in VectorStep applies: a passing check produces no label at all. So on a step with no deterministic_checks configured and no LLM auto-judge enabled, an unattended run grades zero samples and detects nothing — the report says so explicitly (“0 of 20 members could be graded automatically”), and VectorStep logs a warning naming the golden set at startup/reload if this is the state you’ve configured. Configuring a judge or adding deterministic checks to the step is what actually closes this gap; without either, this feature only ever alerts on its own “every member came back unreplayable” failure case, not on genuine accuracy drift.

Pinned labels don’t follow later re-marks

A member’s label is copied at the moment it’s pinned, not looked up fresh on every run. If someone later re-marks the original step, the pinned baseline does not move — a golden set is a fixed standard, and a regression suite whose own baseline drifts underneath it would reintroduce exactly the problem pinning exists to prevent. A divergence between a pinned label and the source’s current label is detected and reported on every run (label_divergences in the report), never silently applied — re-pinning a specific member to its current label is an explicit action from the golden set’s page.

A pinned member’s source row can also simply vanish (retention, cleanup). There’s no foreign key holding it in place, on purpose: a database-level cascade would silently shrink the golden set the same way an unpinned “most recent K” would. Instead a missing source makes that one member unreplayable, visibly, with its own reason string, every run — a golden set that’s lost half its members looks obviously broken, not quietly smaller.

Always rerender, never rendered

A scheduled regression replay always executes in rerender mode — there is no way to configure it otherwise, and a config entry with mode: rendered is rejected at load time. rendered mode resends the originally recorded prompt text verbatim, which would make a prompt-caused regression — one of the most common kinds — invisible by construction. The comparison here is specifically “does today’s config, run today, still get this right,” so it has to actually render today’s prompt template.

The allowlist still applies

Nothing about running unattended relaxes replay.safe_agents — if anything, an unattended job firing live tool calls is more reason for the gate, not less. The golden set’s step’s current agent (and the judge’s agent, if configured) must be allowlisted exactly as for an operator-triggered batch; an unlisted agent is refused before the first schedule ever registers.

Configuring a schedule

replay:
  safe_agents:
    - "gateway:sre-investigation"
  regression:
    - golden_set: nightly-triage      # must already exist — created from the UI
      cron: "0 3 * * *"
      alert_on_regressed: 1            # default shown — alert if at least this many members regressed
      alert_on_accuracy_drop: 0.10     # default shown — alert if fresh accuracy is this far below pinned baseline
      notify:
        channel: slack
        template: "Regression in {{golden_set}}: {{n_regressed}}/{{n_members}} regressed, fresh accuracy {{fresh_accuracy}}. {{report_url}}"
        config: {}

This is service config, not pipeline config — a golden set spans runs and its schedule is an operational concern, the same way live-pricing catalog refresh is. Either threshold firing sends the alert; both are configurable, neither is hard-coded. notify is the exact same channel/template/ config shape pipeline notifications already use, so it dispatches through whichever notifier (slack, telegram, webhook) is already configured — including when the underlying batch run is stage: testing, which every replay batch always is. That’s a deliberate exception: the stage-based routing that quietly sends a test pipeline’s notifications to the log instead of the real channel does not apply to a regression alert, since the alert itself is a genuine operational signal, not pipeline test noise. An alert also always fires when the job crashes, or when every pinned member came back unreplayable — silence must never be ambiguous between “all clear” and “the check didn’t run.”

/reload (and a restart) re-registers regression schedules idempotently, same as pipeline schedules — editing this block takes effect without restarting the whole process.

Cost

Every scheduled run makes one live LLM call per golden-set member, every time it fires — there’s no sampling or backoff. A 50-member golden set on a nightly cron is 1,500 calls a month, before counting an optional judge (which adds up to one more call per judge-eligible member). Size golden sets and schedules with that multiplication in mind, especially against an expensive candidate model.

Where the results live

A scheduled run is stored exactly like an operator-triggered replay batch — same synthetic stage: testing run, same report page, same marking-queue integration — with source: regression instead of source: replay so it’s filterable apart from hand-launched batches, and a regression block on the report (/ui/replays/{run_id}) showing baseline vs. fresh accuracy, which members regressed or improved, any label divergences, and the unreplayable count.

There’s currently no separate JSON API for golden set CRUD — golden sets are created and curated from the UI only (/ui/golden-sets).

Multi-step / whole-pipeline replay

Everything above replays one step at a time — deliberately, since isolating the variable under test is exactly what makes a single-step comparison readable. But a config change can have an effect that only shows up downstream: a prompt tweak on step 1 that quietly starves step 3 of a field it used to get, for instance. Single-step replay cannot see that by construction — it only ever re-runs one step against recorded upstream history. Multi-step replay closes that gap: it replays a chain of contiguous steps, each one fed by the previous replayed step’s output, not the stale recording.

rerender only — never rendered

rendered mode resends a sample’s recorded prompt verbatim. For step 1 of a chain that’s coherent, but for step 2 the recorded prompt already has step 1’s old, recorded output baked into it at render time — resending it verbatim would feed step 2 the old output while step 1 was just re-executed with a candidate config. The chain would stop being a chain, and the batch would silently measure nothing. So multi-step replay only ever renders the candidate’s prompt template against a freshly reconstructed context — a request that asks for rendered mode is rejected at validation with that reason, before anything executes.

Sequential steps only

A range may not contain a parallel group or a fan-out. This isn’t an implementation gap — a fan-out’s branch count is resolved at runtime from its over expression, so a replayed fan-out could produce a different number of branches than the run it’s replaying. Per-branch comparison has no defined meaning once the branch counts can disagree. A range that ends before a parallel/fan-out group, or begins after one, is fine — that group’s recorded output is then just upstream history like any other step, exactly as single-step replay already treats it.

Context chaining

For a range [A..Z] replaying against one recorded run:

• Steps before A use that run’s recorded outputs — they were never re-executed.
• Steps A through Z use the replayed outputs produced earlier in this same chain.
• Step A itself has an entirely recorded upstream, so its context is identical to what single-step replay would have built for it.

A step failing mid-chain

If step 2 of a 4-step range fails, steps 3 and 4 have no upstream output to run against. VectorStep abandons the rest of that run’s chain rather than substituting the recorded output and continuing — a chain that’s half replayed and half recorded would produce a result nobody could interpret. That run is reported as partial, distinct from a run that completed every step and one where the very first in-range step never even completed (unreplayable). Other runs in the same batch are unaffected. An abandoned step is reported as ungraded, never as a 0.0.

Sample selection

The sample unit here is a recorded run, not a step’s calibration bucket. A run qualifies when it executed every step in the range (a run that short-circuited partway through can’t be replayed as a chain) and at least one in-range step has a resolvable label — requiring every step to be labelled would select almost nothing, since marking is per-step and sparse. The most recent K qualifying runs are selected, ordered by the run’s most recent in-range execution.

Grading — per step, and per run

There is deliberately no single “did the chain succeed” label. resolve_label has no concept of chain-level success, and inventing a roll-up label here would be a new, unvalidated trust signal layered on top of the per-step ones that already exist. Instead:

• Per step, across runs — each step’s own replayed accuracy against its own recorded accuracy, computed over that step’s own labelled subset (not a batch-wide pool). This is the number that actually localises a regression: if a candidate change to step 3 breaks it, step 3’s accuracy drops while steps 1 and 2 hold steady.
• Per run — whether the chain completed, partially completed, or never got off the ground, plus how its final in-range step graded. Grading itself reuses the exact same precedence chain as everywhere else (human mark > deterministic failure > LLM judge > ungraded) — nothing new to learn here if you already read the sections above.

The report’s grid view puts runs as rows and steps as columns, so “step 3 regressed on 7 of 10 runs” is visible at a glance rather than buried in a flat list.

Cost — multiplies by steps in range

A multi-step batch makes K × steps-in-range live calls, not K. A 5-step range at k: 10 is 50 calls; the same range at k: 50 is 250. Because the multiplier compounds, k’s own cap is lower here (max 50, vs 100 for single-step replay) and the product is capped outright — a request where k × steps-in-range exceeds 500 is rejected before anything executes, rather than relying on k’s cap alone to keep a long range affordable.

Concurrency is the same 3 chains in flight at once as single-step replay’s 3 samples — but each chain runs its steps strictly in order (step N depends on step N−1’s replayed output), so a long range is slower wall-clock than the same K on a single step. A batch is synchronous either way: the launch request blocks until every selected run has been attempted.

Candidate overrides — keyed by step name

candidates is a map from step name to the same {model, agent, prompt_template} shape single-step replay uses. Steps absent from it run with their current configuration — the point of multi-step replay is usually changing one step and observing the downstream effect, not changing every step at once. An empty candidates map is meaningful: it replays the chain completely unchanged, which is the same shape a regression check would take.

API

# Launch a chain replay — blocks until every selected run has been attempted
POST /pipelines/{pipeline_name}/replay
# body: {
#   "pipeline": "incident-response",              # must match the URL
#   "range": {"from": "triage", "to": "remediate"}, # "to" omitted = to the end of the pipeline
#   "candidates": {"remediate": {"model": "..."}}, # keyed by step name; steps absent run unchanged
#   "k": 10                                        # optional, default 10, max 50
# }
# → {"status": "completed", "run_id": "<synthetic-run-id>"}

# Same report endpoint as single-step replay — dispatches on the batch's kind
GET /replays/{run_id}/report
# → {
#     kind: "multi_step", pipeline, range, step_names, candidates, k,
#     recorded_distribution: {step_name: {correct, partial, incorrect}},
#     per_step: {
#       step_name: {recorded_accuracy, recorded_n, candidate_accuracy_so_far,
#                    candidate_graded_n, judge_accuracy, judge_graded_n,
#                    combined_accuracy, abandoned_n},
#       ...
#     },
#     runs: [{source_run_id, sample_index, status,        # completed | partial | unreplayable
#             final_step_label,                           # the LAST in-range step's own resolved label, never a synthesised one
#             steps: {step_name: {status, candidate_step_id?, recorded_label?,
#                                  recorded_label_source?, candidate_label?,
#                                  candidate_label_source?, judge_verdict?,
#                                  judge_reason?, reason?}, ...}}, ...],
#     n_runs, n_completed, n_partial, n_unreplayable,
#   }

A Phase 1 (single-step) report has no kind key at all — GET /replays/{run_id}/report serves both shapes from the same endpoint, and an existing integration reading single-step reports sees no change.

UI

A “Replay a range…” link on the pipeline detail page opens a plain form (/ui/replays/multi-step/new?pipeline=...) for the pipeline, the from/to step names, per-step candidate overrides (as JSON), and K. Submitting blocks until the batch finishes and lands on the same report route as single-step replay (/ui/replays/{run_id}), which renders the runs-as-rows/steps-as-columns grid described above instead of the flat per-sample table when the report’s kind is multi_step.

What replay is not

• Operator-triggered replay doesn’t run on a schedule — every batch is explicitly launched. Scheduled regression replay is the schedule-driven counterpart, and it answers a different question (has the current config regressed) with a different, fixed sample set (a golden set, not “most recent K”).
• Multi-step replay doesn’t produce a single pass/fail verdict for the whole chain — see Grading — per step, and per run for why that’s deliberate, not a missing feature.

These are deliberate boundaries, not oversights — each would need its own design before it’s worth building.

Related

• Confidence — “Proving a change before promotion” ties replay directly to the S/V/G/D formula and calibration bucketing this feature reuses.
• Promotion readiness — the criteria a replay batch’s evidence ultimately feeds a decision about.
• Marking queue — where replay-produced steps needing a human mark show up, flagged REPLAY.
• Testing vs production stages — why a replay batch stays stage: testing even after the candidate it tested gets promoted.