Tutorial: promote your pipeline to production

Every tutorial in this series has run ~/.vectorstep/service/pipelines/alert-triage.yaml in stage: testing — the default — without saying so explicitly. This capstone tutorial makes that visible, then walks through actually earning and executing the promotion to stage: production, using readiness evidence rather than a gut call.

Models used for testing

This series was validated internally against openrouter/anthropic/claude-sonnet-4.5 for first-responder and openrouter/anthropic/claude-haiku-4.5 for every other agent. Other models and providers should work too, but exact wording and confidence numbers can vary by model.

Prerequisites

The pipeline as it stands after however much of the series you’ve done — this tutorial works against any shape of alert-triage.yaml. The more tutorials behind it (confidence threshold, verifier, fan-out, notifications), the more interesting its accumulated run history is for this one; readiness reads real evidence, and a pipeline with only a couple of runs will mostly show insufficient_data, which is itself a fine thing to see once.

1. See what stage: testing has actually been hiding

Re-trigger the pipeline once, the same way you have all series:

curl -X POST "http://localhost:8000/webhook?source=alertmanager&allow_testing=true" \
  -H "Content-Type: application/json" \
  -d @tests/fixtures/alertmanager_critical.json

Open the run in the UI — the amber TESTING badge next to the run status is the first thing stage: testing changes: it’s shown on every browse surface (runs list, dashboard, this run) precisely so testing activity never looks like real traffic by accident.

If you did Route escalations to a real channel, expand the Run log and look for notification_suppressed_testing instead of notification_sent — “[testing] Notification routed to log: escalate → would have been log” — even though a real channel is configured, testing forces every notification through log instead.

What’s genuinely unaffected: the trust-gating mechanics from every earlier tutorial — confidence thresholds, verifiers, grounding, fan-out joins, budget aborts — behave identically in either stage. Only four specific outbound paths are muted by stage: testing (notifications, executor: notify, on_failure.webhook, and the external side of executor: human) — see what testing mutes for the complete list. Nothing about whether the pipeline reasons well is different; only whether its side effects reach the outside world.

2. Build up real run history

Readiness reads accumulated evidence — it can only ever report what this pipeline has already earned, not what it’s capable of in principle. Trigger it several more times:

for i in 1 2 3 4 5; do
  curl -X POST "http://localhost:8000/webhook?source=alertmanager&allow_testing=true" \
    -H "Content-Type: application/json" \
    -d @tests/fixtures/alertmanager_critical.json
  sleep 1
done

Then mark a few outcomes — this is what feeds the accuracy and calibration tiers below; without it they’ll sit at insufficient_data no matter how many runs pile up.

There are two places to do this, and which one you use actually matters:

• Run-level — the Accuracy feedback widget at the bottom of the run detail page (Correct / Partial / Incorrect). One click, one verdict for the whole run.
• Step-level — a separate Correct / Partial / Incorrect control inside each individual step’s own expanded detail panel, next to its parsed output. One verdict for that one step’s execution only.

How they interact: a run-level mark is a fallback — it’s applied to every step in that run for accuracy/calibration purposes, but only for steps that don’t already have something more specific. A step’s own mark always wins over the run-level fallback for that one step; so does an automated deterministic-check failure on that step, if it has one. Nothing below the winning label is even looked at. This is exactly why run detail describes step-level feedback as not affecting “gating or flow control” — that’s about the live run itself behaving no differently, not about readiness. It’s fully counted, and counted first.

In practice: for this series’ pipeline, marking the run is enough most of the time — one meaningful chain of reasoning, one verdict. Reach for a step-level mark instead when a run’s overall outcome doesn’t reflect one particular step fairly — e.g. the run as a whole was fine but one check-upstreams branch reasoned badly and you don’t want that dragging down that branch’s own calibration history, or the reverse (a run escalated because of one bad step, but a different step in the same run behaved correctly and deserves its own accurate record rather than inheriting the whole run’s “incorrect”).

Open a couple of the resulting runs and mark a few — run-level is fine to start with. For finding every step still missing a mark across a whole pipeline (rather than clicking into runs one at a time), see the Marking queue.

3. Look at readiness before you set a bar

Open the pipeline’s detail page (/ui/pipelines/alert-triage) and find the Promotion readiness card — it only appears for stage: testing pipelines. This pipeline has no readiness: block yet, so the card reads “No criteria configured — showing observed evidence at service defaults”, and each step shows an observed: ... chip instead of tier chips — real evidence, just not measured against a bar you chose. Hover one: “Observed at service defaults (bin_width 0.1, n_min 20) — not a bar you chose.”

This is what evidence looks like with nothing configured: not nothing, but not a verdict either.

4. Use the criteria builder

Click the Build criteria button in the Promotion readiness card’s own header — not a per-step tier chip; clicking one of those opens the builder scoped to just that one step instead, which isn’t what we want here. This opens a preview-only builder — turning a knob sends the candidate config to POST .../preview and updates the result against this pipeline’s real accumulated evidence within about 300ms. Nothing is written until you paste something yourself; the banner says so directly: “Preview only — nothing is saved. Copy the YAML and ship it through git.”

Check the Scope dropdown at the top of the builder reads “Whole pipeline (default for every step)” — that’s what it opens to when you use the header button, but confirm it before continuing, since the resulting YAML lands in a completely different place (pipeline-level vs. one step’s own readiness: block) depending on what’s selected here.

Start with two knobs:

• Operational → min_runs: 20, acceptable_statuses: [completed]
• Confidence → min_confidence: 0.80

Watch the live result update per step as you type — a verdict badge (ready / not_ready / insufficient_data) and a narrative explaining why. Two knobs on this same screen are worth reading carefully before you trust them elsewhere, because both are counter-intuitive in the same direction — laxer than they look:

The two classic misreadings

• calibration.n_min is per confidence band, not a total. A step with 100 marked results spread evenly across 10 bands has only 10 in each, and won’t validate at n_min: 20 — look at the fullest band’s own count, never a total.
• Adding a status to acceptable_statuses makes the bar laxer, not stricter. [completed, escalated] accepts runs where a human had to step in — a weaker claim than [completed] alone, even though the longer list reads like a higher bar.

Once the preview looks right, click Copy and paste the generated snippet into ~/.vectorstep/service/pipelines/alert-triage.yaml, at the pipeline level (a sibling of budget:/notifications:):

readiness:
  operational:
    min_runs: 20
    acceptable_statuses: [completed]
  confidence:
    min_confidence: 0.80

5. Reload and confirm the bar applies for real

curl -X POST http://localhost:8000/reload

Refresh the pipeline detail page. The per-step chips should now read operational / confidence verdicts instead of a plain observed: ... badge — pass (green), not_ready (red), or insufficient_data (amber) depending on what the last few triggers actually produced. Click How is this judged? on a step for the same plain-language narrative the builder showed, now reading the real configured bar instead of a preview.

If a tier reads not_ready and you don’t understand why, this is the moment to actually trust the number rather than raise the bar until it turns green — that’s what stage: testing and this whole card exist to surface before real traffic depends on it.

6. Promote

Change one line in the same file:

stage: production

Reload, then trigger it without allow_testing=true this time — this is the concrete difference promotion just made:

curl -X POST http://localhost:8000/reload
curl -X POST "http://localhost:8000/webhook?source=alertmanager" \
  -H "Content-Type: application/json" \
  -d @tests/fixtures/alertmanager_critical.json
# → {"status": "accepted", "run_id": "..."}   — no allow_testing needed any more

What you should see

The new run has no TESTING badge, and — assuming check-upstreams’s confidence_threshold is still the real 0.70 from Route escalations to a real channel rather than the 0.99 used there to force a demo — it should just complete normally, same as every healthy run earlier in this series. That’s not an anticlimax: a promoted pipeline behaving exactly like it did in testing, on the same real evidence, is the entire point.

One consequence worth being explicit about: no notification fires on this run either, even if Telegram is configured — notifications: only reacts to an escalate/abort/notify transition, and a healthy completed run never triggers one. If you were expecting to see something in your phone right now, that’s why nothing arrived — not a bug, and not a testing-stage artifact this time; a production run with nothing to escalate genuinely sends nothing.

The Promotion readiness card is gone entirely from the pipeline detail page — it’s a stage: testing-only tool, not a permanent fixture, since a production pipeline is judged by its live metrics and dashboards from here on, not a pre-promotion checklist.

Seeing a real notification actually fire

To confirm end-to-end that a promoted pipeline’s notifications genuinely reach a real channel — not suppressed, not simulated — force the same escalation used earlier in the series, now for real. In ~/.vectorstep/service/pipelines/alert-triage.yaml:

  - fan_out:
      name: check-upstreams
      ...
      confidence_threshold: 0.99   # deliberately unreachable — see below

curl -X POST http://localhost:8000/reload
curl -X POST "http://localhost:8000/webhook?source=alertmanager" \
  -H "Content-Type: application/json" \
  -d @tests/fixtures/alertmanager_critical.json

This time the run log should show notification_sent, not notification_suppressed_testing — and if Telegram is configured, check your phone; this is the first message in the whole series that actually reaches it for real, rather than being routed to log because the pipeline was still stage: testing.

Put confidence_threshold back to 0.70 once you’ve confirmed it — same as every other time this series has used an artificially unreachable bar to force a demo, it only exists to manufacture the escalation, not to describe how this pipeline should actually run.

7. Close the loop

Nothing about this promotion was automated. Readiness is strictly advisory — the tooling never intercepted the stage: edit, never added a UI toggle, never blocked /reload. It informed a decision you made in a one-line YAML diff, reviewed in git exactly like any other config change. A team that wants an automated gate on top of this can script one against GET /pipelines/{name}/promotion-readiness — the same JSON the builder itself calls — but building that automation is a decision for that team to make deliberately, not something this feature does for you by default.

That’s the end of the tutorial series. Every mechanism from the quick start through this page is now sitting in one real, promoted pipeline — trust knobs, grounding, fan-out, artifacts, notifications, budget, and now a readiness bar it actually earned.

Where next

• Promotion readiness — the full tier reference (operational/confidence/accuracy/calibration), every readiness knob, and the merge rules for a step that adds to or overrides the pipeline’s house standard.
• Testing vs production stages — exactly what stage gates and what it doesn’t, and the full promotion workflow.
• Marking queue — finding every step still missing a human accuracy mark before its tiers can resolve.
• Replay / shadow evaluation — once a step has real calibration history, this re-runs a bucket’s own recorded, labelled inputs against a candidate prompt/model change, so you can compare the two on identical evidence without waiting for fresh production traffic.