Tutorial: turn on grounding

Build your first agent had you watch first-responder’s trace show two real tool calls — fetch against GitHub’s status API, filesystem reading known-issues.md — and told you that’s exactly what grounding checks automatically. This tutorial turns that on: a judge agent cross-references upstream_incident and known_issue against that same trace instead of you eyeballing it, then goes one rung further and lets that judgement actually gate the step.

Models used for testing

This series was validated internally against openrouter/anthropic/claude-sonnet-4.5 for first-responder and openrouter/anthropic/claude-haiku-4.5 for every other agent, including grounding-judge below. Other models and providers should work too, but grounding-judge behaviour in particular can vary noticeably by model — see the note on this in step 3.

Prerequisites

Turn on the trust knobs completed — ~/.vectorstep/service/pipelines/alert-triage.yaml’s triage step already has a confidence_threshold and a verifier. Grounding is executor: gateway-only, which triage already is.

1. Install the grounding-judge agent

cp -r ~/.vectorstep-gateway/samples/agents/grounding-judge ~/.vectorstep-gateway/agents/
curl -X POST http://localhost:18780/reload

POST /reload on the Gateway picks up new agent directories from disk without a restart — unlike adding a new mcp_servers: entry, this doesn’t need the PATH-sensitive full restart from the first tutorial.

Worth reading ~/.vectorstep-gateway/agents/grounding-judge/soul.md before moving on: it has tools: [] and is explicitly told not to use outside knowledge — its only job is to check whether the trace it’s handed backs up a claim, not whether the claim is actually true. A claim that just repeats something the primary agent was already told (the alert’s severity, service, environment) doesn’t need evidence; only claims that go beyond that — a root cause, a lookup result, a ticket ID — do. It’s also explicit that seeing a tool call is not the same as seeing its result: a truncated or missing TOOL RESULT means the claim is unsupported, not “probably fine because the right tool ran” — worth remembering for step 3.

2. Turn on grounding (shadow mode)

In ~/.vectorstep/service/pipelines/alert-triage.yaml, add a grounding: block to the triage step — everything else in the file is unchanged from the previous tutorial:

name: alert-triage
description: First-responder agent gathers evidence before anyone escalates
trigger:
  match: { source: alertmanager, severity: critical }
  dedup:
    enabled: false

context_template:
  include:
    - severity
    - summary

steps:
  - name: triage
    executor: gateway
    executor_config:
      agent: first-responder
      session_key: "agent:first-responder:{{pipeline_run_id}}:triage"
    confidence_threshold: 0.70
    on_low_confidence: escalate
    prompt_template: |
      ...                          # unchanged from the previous tutorial
    verifier:
      executor: gateway
      executor_config:
        agent: first-responder
        session_key: "agent:first-responder:{{pipeline_run_id}}:triage-verify"
      combination_strategy: minimum
      trigger:
        always: true
    grounding:
      agent: grounding-judge

agent: grounding-judge is actually the default — it’s shown explicitly here for clarity. Nothing else is required to turn grounding on: no threshold, no cap, just a score that gets recorded.

Reload and re-trigger:

curl -X POST http://localhost:8000/reload
curl -X POST "http://localhost:8000/webhook?source=alertmanager&allow_testing=true" \
  -H "Content-Type: application/json" \
  -d @tests/fixtures/alertmanager_critical.json

What you should see

The step should still show completed, at the same confidence as before — shadow-mode grounding never changes the outcome. What’s new is a G figure alongside S and V in the Trust panel, plus a per-claim breakdown under it: one row for upstream_incident, one for known_issue, each marked supported (✓) with a short excerpt of the evidence the judge found in the trace, or unsupported (✗) if it couldn’t find any. With both tools working, expect G at or near 1.0 — both claims should be well-supported. “How was this calculated?” now includes a grounding line in its narrative alongside self-report and verifier.

3. Make it actually catch something

Shadow mode is only interesting once you’ve seen it flag a real gap, and the reliable way to force one doesn’t touch first-responder at all — no prompt edits, no soul.md edits, none of the original two-task pipeline changes. In ~/.vectorstep/service/pipelines/alert-triage.yaml, just cap grounding.max_trace_chars down to something absurdly small:

    grounding:
      agent: grounding-judge
      max_trace_chars: 10   # deliberately tiny — see below

Reload and re-trigger — no Gateway reload needed, nothing about the agent or its tools changed:

curl -X POST http://localhost:8000/reload
curl -X POST "http://localhost:8000/webhook?source=alertmanager&allow_testing=true" \
  -H "Content-Type: application/json" \
  -d @tests/fixtures/alertmanager_critical.json

What you should see

G should drop sharply — likely to 0 — with every claim in the per-claim breakdown now marked unsupported, each citing the truncated TOOL_RESULT as the reason. Nothing about what actually happened changed: the same two tool calls ran and returned the same real data, and both of first-responder’s original claims (upstream_incident, known_issue) are exactly as true as they were in step 2. What changed is that max_trace_chars: 10 truncates every tool-result event in the transcript handed to the judge down to about ten characters plus an ellipsis — and the bundled grounding-judge is explicit that a TOOL CALL line alone (which survives untouched — only result content gets truncated) is not evidence, and that a truncated result should be marked unsupported rather than charitably assumed fine. If your evidence text calls out the truncation explicitly rather than just saying “unsupported,” that’s the judge doing exactly what it’s told.

This is the truncation gotcha the reference doc warns about, deliberately provoked rather than stumbled into: a claim that looks exactly like a hallucination from the grounding score alone can actually be a real, true claim whose supporting evidence just didn’t make it into what the judge was shown. In production this usually shows up by accident on a step with unusually long tool output, not because someone set the cutoff to 10 — see Grounding keeps flagging real evidence as unsupported for the full troubleshooting path, including the second, independent truncation point on the Gateway side that raising this setting alone won’t fix.

Judges vary a lot — test yours, don’t assume

Getting a truncated result reliably marked unsupported took real iteration to land on the bundled judge’s exact wording — earlier attempts that didn’t explicitly rule out “a tool was called, so it’s probably fine” let plenty of genuinely-unsupported claims through, and results varied noticeably between models used as the judge. Don’t assume a grounding judge behaves a given way just because the sample does, especially if you swap in a different model or write your own from scratch — verify it against a case where you already know the right answer, the same way this tutorial just did. See Writing your grounding judge, Writing good agents, and Writing good prompts for what actually shapes this behaviour.

4. Turn on enforced grounding

So far G is purely informational. Add enforce: true to make it participate in the gate:

    grounding:
      agent: grounding-judge
      max_trace_chars: 10
      enforce: true

The gate formula becomes combined_trust = min(effective_confidence, G) — see the full formula rather than re-deriving it here. There’s no separate grounding threshold; it reuses the step’s existing confidence_threshold (0.70).

Reload and re-trigger again, with max_trace_chars: 10 from step 3 still in place.

What you should see

With G at or near 0, combined_trust = min(effective_confidence, G) should come out close to 0 regardless of how high the primary’s own confidence is — comfortably below confidence_threshold: 0.70 — so expect the run to escalate. The Trust panel header should now read “Trust (enforced)” instead of “(shadow)”, with a Combined trust figure shown alongside S/V/G. This is the actual payoff: a confident, correct, well-evidenced response still gets treated as untrusted the moment its evidence trail is cut off — grounding enforcement doesn’t know why G is low, only that it is.

5. Clean up

Remove max_trace_chars: 10 from the grounding: block (or set it back to the default, 1500) — it only existed to manufacture this demo, and leaving evidence permanently invisible to the judge defeats the point of having grounding at all. Reload VectorStep again after reverting. Leave grounding: and enforce: true in place; that’s the actual end state this tutorial was building toward. Nothing about first-responder itself was ever touched, so there’s nothing to revert on that side.

A note on how narrow this agent still is

first-responder’s soul.md hardcodes an exact numbered task list — two tools, then summarise. That’s a reasonable shape for a single-purpose tutorial agent, but it doesn’t scale to reusing the same agent across pipelines with different needs. A more general design keeps soul.md to identity and standing behaviour (how carefully to reason, when to say “I don’t know,” what “supported by evidence” means to this agent) and lets each pipeline’s own prompt_template supply what this run specifically needs. See Writing good agents and Writing good prompts for where that line usually belongs, including why a prompt-level ask that conflicts with soul.md’s stated scope tends to lose.

The grounding-judge sample, in contrast, is already written the general way: nothing in its own soul.md is task-specific, which is exactly why the same one agent could judge a completely different pipeline’s claims tomorrow without changes. Most real deployments reuse one judge across many steps rather than writing a bespoke one per pipeline; you’d reach for a specialised judge only when a domain has its own notion of what counts as evidence (legal, financial, medical claims, for instance) that a generic cross-referencer wouldn’t reliably apply. See Writing your grounding judge for when that’s actually worth doing, and what to change.

Where next

Go to Fan out over multiple services next — the next tutorial in the series.

Once you’re comfortable with the mechanics:

• Grounding — the full reference this tutorial walks through hands-on, including deterministic checks (D), the third trust-vector signal this tutorial doesn’t cover.
• Writing your grounding judge — going beyond the bundled sample: model choice, and what to change for a step with many load-bearing claims.
• Grounding keeps flagging real evidence as unsupported — if a claim you’re confident is real gets flagged unsupported, this is almost always a truncation cutoff, not a judge mistake — start here.