Tutorial: turn on the trust knobs

Build your first agent got alert-triage running your own first-responder agent with no gating at all — rung 0 on the trust ladder. This tutorial climbs two rungs on that exact pipeline: a confidence floor, then a second opinion.

Models used for testing

This series was validated internally against openrouter/anthropic/claude-sonnet-4.5 for first-responder and openrouter/anthropic/claude-haiku-4.5 for every other agent introduced later in the series. Other models and providers should work too, but exact wording and confidence numbers can vary by model.

1. Add a confidence floor

Only the triage step’s own two new lines (confidence_threshold, on_low_confidence) are new here — leave name:, trigger: (including dedup: { enabled: false }), context_template:, and everything else in ~/.vectorstep/service/pipelines/alert-triage.yaml exactly as the previous tutorial left it. The full file should now read:

name: alert-triage
description: First-responder agent gathers evidence before anyone escalates
trigger:
  match: { source: alertmanager, severity: critical }
  dedup:
    enabled: false   # still needed — see the quick start's note on this

context_template:
  include:
    - severity
    - summary

steps:
  - name: triage
    executor: gateway
    executor_config:
      agent: first-responder
      session_key: "agent:first-responder:{{pipeline_run_id}}:triage"
    confidence_threshold: 0.70
    on_low_confidence: escalate
    prompt_template: |
      ...                          # unchanged from the previous tutorial

If a re-trigger later in this tutorial comes back deduplicated instead of accepted, that dedup: block got dropped somewhere along the way — check it’s still there before anything else.

Reload and re-trigger it exactly as before:

curl -X POST http://localhost:8000/reload
curl -X POST "http://localhost:8000/webhook?source=alertmanager&allow_testing=true" \
  -H "Content-Type: application/json" \
  -d @tests/fixtures/alertmanager_critical.json

Both tools should still succeed, so the agent reports high confidence and the run completes exactly as before — confidence_threshold only changes behaviour when the bar isn’t cleared. See how confidence and calibration work for what the number means before you rely on it.

See it actually gate

Break one of the tools on purpose to see the other side of the threshold. Rename the known-issues file so the read fails:

mv ~/vectorstep-tutorial/known-issues.md ~/vectorstep-tutorial/known-issues.md.bak

Re-trigger the same webhook. The agent should now report low confidence honestly (per its own soul.md: “a tool failed … say so honestly and score low”), the step should come back below the 0.70 threshold, and the run’s status should show escalated instead of completed — a human reviews it instead of the pipeline proceeding on a guess. Check the Trust panel on the run to see the exact number and why it didn’t clear the bar.

Restore the file when you’re done:

mv ~/vectorstep-tutorial/known-issues.md.bak ~/vectorstep-tutorial/known-issues.md

2. Add a second opinion

A confidence floor only checks the agent’s own number. A verifier adds a second agent call that sanity-checks the primary response before that number is trusted — and it can only lower confidence, never raise it. See Verifiers for the full mode/combination reference; this tutorial uses the defaults.

Add a verifier: block to the same step — again, only additive: name:, trigger: (dedup included), and context_template: at the top of the file are unchanged:

name: alert-triage
description: First-responder agent gathers evidence before anyone escalates
trigger:
  match: { source: alertmanager, severity: critical }
  dedup:
    enabled: false

context_template:
  include:
    - severity
    - summary

steps:
  - name: triage
    executor: gateway
    executor_config:
      agent: first-responder
      session_key: "agent:first-responder:{{pipeline_run_id}}:triage"
    confidence_threshold: 0.70
    on_low_confidence: escalate
    prompt_template: |
      ...                          # unchanged from the previous tutorial
    verifier:
      executor: gateway
      executor_config:
        agent: first-responder
        session_key: "agent:first-responder:{{pipeline_run_id}}:triage-verify"
      combination_strategy: minimum
      trigger:
        always: true

This reuses the same first-responder agent for the verifier, just with a different session_key — enough to see the mechanism work with nothing new to write. In critic mode (the default), the verifier gets the primary’s full response plus a transcript of its tool calls, and critiques the reasoning rather than re-running the task blind.

In production, verify with a different model

A verifier reusing the exact same agent config will tend to agree with itself for the same reasons it made the original call — its disagreement is the informative signal, and that’s more likely from a differently-tuned reviewer. The one-line change is executor_config.model on the verifier block: model: anthropic/claude-opus-4-6, for example, overrides just that call without touching the agent’s own default. For a step that authorises a side effect rather than just informing, prefer mode: independent instead of the critic default — see Verifiers for why.

Reload and re-trigger again. The run’s Trust panel now shows both numbers — primary and verifier — and which one the minimum strategy picked. Audit columns (verifier_agent, verifier_model) record which agent actually ran the verification on that specific run, independent of whatever the config says today.

What you should see

The step now shows two agent calls, not one: the primary result you already know, plus a separate CRITIC panel underneath — the verifier’s own independent write-up, with its own summary, next-step-context, and a different-shaped reasoning breakdown (ASSESSMENT / GAPS / CONFIDENCE_RATIONALE, not the primary’s supports/contradicts/ assumptions), since its job is to review the primary’s work, not re-triage from scratch. In this setup it usually agrees closely with the primary — expected, since it’s the same agent config; that’s exactly the “a verifier reusing itself tends to agree with itself” point above.

Below both, expand TRUST (SHADOW) — click How was this calculated? for a plain-English, numbered walkthrough of exactly how the final confidence was derived: the step’s own self-reported number, what the verifier contributed under the minimum combination strategy, and the resulting number against the confidence_threshold. The row underneath breaks out S (self-report), V (verifier, with its mode and which agent/model actually ran it), and G (grounding) — showing n/a for G, because it isn’t configured on this step yet.

“(SHADOW)” is not a hint that nothing here is real — the confidence threshold and verifier you just added are both genuinely gating this run; that’s why it says completed rather than escalated. The label reflects a separate, coarser distinction: whether the full trust-vector policy (self-report + verifier + grounding + deterministic checks, combined via min()) is what’s deciding the outcome, versus today’s simpler threshold-plus-verifier path. It flips to (enforced) the moment any of grounding, calibration, or a deterministic check gets turned on for this step — which is exactly what the next tutorial does. Run log at the bottom is the literal timestamped event sequence — pipeline started, step started, verifier ran, step completed — useful for seeing where the ~20s actually went (two sequential LLM-plus-tool-call rounds: primary, then critic).

Where next

Go to Turn on grounding next — rung 3, and the tutorial that flips this same step’s Trust panel from (shadow) to (enforced). After that: fan-out, artifacts, notifications, budget, metrics, and finally promoting it to production. Deterministic checks and calibration follow the same additive pattern but aren’t yet their own click-by-click tutorials — see the docs below instead.

Once you’re comfortable with the mechanics:

• Adding trust, one signal at a time — the full ladder, and when climbing further is (and isn’t) worth it.
• Calibration — once this step has real run history, check whether its “90% confident” has actually meant 90%.