Skip to content
VectorStep
Blog
Beta. VectorStep is in beta.Let us know if something breaks.

Authoring synthetic golden-set members

Golden sets pin existing step executions as a fixed regression suite. That’s a chicken-and-egg problem for a pipeline that hasn’t shipped yet: you can’t pin a case that has never run. Synthetic members close that gap — a hand-authored input/expected-output pair, no execution required. This guide walks through the motivating case directly: giving a brand-new pipeline a quality bar before it ever sees production traffic.

The pipeline

Section titled “The pipeline”

A minimal single-step pipeline, on-call-triage.yaml, not yet wired to any real alert source:

name: on-call-triage
trigger:
  match: {}
context_template:
  include:
    - severity
    - summary
steps:
  - name: triage
    executor: gateway
    executor_config:
      agent: on-call-triage-agent
    prompt_template: |
      Alert: {{ summary }}
      Severity: {{ severity }}


      Decide whether this needs a page right now, and say why in one sentence.

Two fields make it into every triage prompt — severity and summary — because context_template.include names them. These same two names are the only fields a synthetic member for this step can supply; anything else 422s naming the unrecognized field (see Bypasses the normaliser for why the field set is fixed to exactly this list).

1. Create the golden set

Section titled “1. Create the golden set”

Same as for a pipeline with real traffic — golden sets aren’t synthetic-only or recorded-only, they hold either kind of member (see Golden sets: pinned, not “most recent K”):

Create it from Golden sets…, linked off the step insights page — name it, scope it to triage. Note the golden_set_id from the URL; you’ll need it for the next step, since synthetic members are authored via API/MCP only, not a form.

2. Author a synthetic member for the obvious case

Section titled “2. Author a synthetic member for the obvious case”

Before any real alert has ever reached triage, pin down what “handled correctly” looks like for a plainly critical one:

Terminal window
curl -X POST https://vectorstep.example.com/golden-sets/$GOLDEN_SET_ID/synthetic-members \
  -H "Authorization: Bearer $OPERATOR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "input": {
      "severity": "critical",
      "summary": "payments-api returning 5xx for all requests, error rate 100% for 4 minutes"
    },
    "expected_output": "Page now — total outage of a payment-critical service for over 4 minutes is an immediate page regardless of time of day."
  }'
# → 201 {"member_id": "...", "golden_set_id": "...", "member_type": "synthetic"}

Or the same call from an MCP client, via VectorStep Service MCP’s add_synthetic_golden_set_member:

add_synthetic_golden_set_member(
  golden_set_id="<golden_set_id>",
  input={
    "severity": "critical",
    "summary": "payments-api returning 5xx for all requests, error rate 100% for 4 minutes",
  },
  expected_output="Page now — total outage of a payment-critical service for "
                   "over 4 minutes is an immediate page regardless of time of day.",
)

expected_output is the reference an LLM judge — if you configure one for triage — compares every future candidate against. Without a judge, this member still replays and reports pass/fail via whatever deterministic checks triage declares; it just can’t be graded purely on prose similarity.

3. Add the edge case you don’t want to wait for

Section titled “3. Add the edge case you don’t want to wait for”

The second reason synthetic members exist: a case you already know is tricky, that you’d rather guard against now than wait for it to embarrass you in production:

Terminal window
curl -X POST https://vectorstep.example.com/golden-sets/$GOLDEN_SET_ID/synthetic-members \
  -H "Authorization: Bearer $OPERATOR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "input": {
      "severity": "info",
      "summary": "payments-api p99 latency 800ms, up from a 200ms baseline, single region"
    },
    "expected_output": "Do not page — elevated but not critical, single region, no error-rate signal. Note it for the morning on-call handoff instead."
  }'

This is the adversarial/edge-case use from the motivating cases: a plausible-looking “should this page?” case where the right answer is restraint, pinned as a permanent guard against a future prompt or model change that starts over-paging.

4. Check what’s pinned

Section titled “4. Check what’s pinned”
list_golden_set_members(golden_set_id="<golden_set_id>")

or GET /golden-sets/$GOLDEN_SET_ID/members — both synthetic members show up with member_type: "synthetic", step_id: null, and the synthetic_input/synthetic_expected_output you authored. If triage later gains real production traffic, pinning a few of those runs as ordinary (member_type: "recorded") members alongside these two is normal — see Mixed membership, not a separate “synthetic golden set” concept.

5. Wire up the schedule

Section titled “5. Wire up the schedule”

Nothing synthetic-specific here — the same replay.regression[] config block any golden set uses (see Configuring a schedule):

replay:
  safe_agents:
    - "gateway:on-call-triage-agent"
  regression:
    - golden_set: on-call-triage-quality
      cron: "0 3 * * *"
      notify:
        channel: slack
        config:
          webhook_url: "${SLACK_WEBHOOK_URL}"

From here on, both members — one confirming the obvious page fires, one confirming the near-miss doesn’t — replay every night against triage’s current config, indistinguishable in the report from a member pinned off real traffic, catching a prompt or model regression before triage has ever handled a single real alert.

Where next

Section titled “Where next”