Tutorial: run deterministic logic as a pipeline step

Every other step type in this series calls an LLM. Sometimes that’s the wrong tool — a threshold check, a unit conversion, a lookup against a local dataset is something you’d rather trust to code that gives the same answer every time than to a model reasoning about it fresh on every run. That’s what executor: shell is for: a pipeline step that runs a pre-approved, checksum-pinned local script, whose output feeds forward to later steps exactly like any other step’s does.

This tutorial is standalone — no agent, no Gateway, no MCP tools. Just the quick start and one script.

What you’ll build

A severity-classifier pipeline with one step: a script that takes an error-rate percentage and classifies it into SEV1/SEV2/SEV3 by threshold — deterministically, the same result every time for the same input, which is exactly the property you want from a severity classification and exactly the property an LLM can’t strictly guarantee. A SEV1 result exits non-zero, which drives the step’s confidence to 0.0 and triggers escalation — no new gating mechanism, the step reuses on_low_confidence like every other step type already has.

Prerequisites

• The quick start completed — VectorStep running, .env with your admin/webhook tokens.

1. Write the script

Scripts live under scripts_dir — ./scripts by default, /etc/vectorstep/scripts in the container (read-only, same reasoning as config.yaml itself: this is host/operator-managed content, never editable through the pipeline-write API). Create it on the host:

mkdir -p scripts
cat > scripts/classify_severity.py <<'EOF'
#!/usr/bin/env python3
import sys

rate = float(sys.argv[1])
if rate >= 10:
    print(f"SEV1: error rate {rate}% is critical (>=10%) — escalating")
    sys.exit(1)   # nonzero exit -> confidence 0.0 -> on_low_confidence fires
elif rate >= 5:
    print(f"SEV2: error rate {rate}% is high (>=5%)")
else:
    print(f"SEV3: error rate {rate}% is within tolerance")
EOF
chmod +x scripts/classify_severity.py

The exit code carrying meaning is deliberate, not incidental — confidence is 1.0 on exit 0, 0.0 on anything else, so a script’s own pass/fail logic drives the step’s existing confidence machinery instead of needing a second expression language bolted onto just this executor.

2. Compute its checksum and allowlist it

This is the one step that can’t be done through the API — security.shell_steps.allowed is config.yaml-only, by design. A pipeline can only ever reference a script by name; only someone with filesystem access to this file can decide what that name actually runs:

sha256sum scripts/classify_severity.py   # or: shasum -a 256 on macOS

Add to config.yaml:

security:
  shell_steps:
    enabled: true
    scripts_dir: ./scripts
    allowed:
      - name: classify-severity
        path: classify_severity.py
        sha256: <the hash you just computed>

enabled is off by default — every existing pipeline that doesn’t use executor: shell is completely unaffected either way. Restart VectorStep to pick this up (config.yaml is loaded once at startup, same as every other security-sensitive setting):

docker compose restart vectorstep
docker compose logs --tail=20 vectorstep

You should see this near the top of the log:

Shell steps are ENABLED (security.shell_steps.enabled: true) — 1 script(s) allowlisted from ./scripts

If it says 0 script(s) instead, the allowlist entry was rejected — check for an ERROR ... security.shell_steps.allowed: line just above naming why (a hash mismatch, a missing execute bit, a path that doesn’t resolve inside scripts_dir).

3. Write the pipeline

This pipeline is triggered by the generic webhook source, which reads a pipeline: field straight out of the payload — bypassing trigger.match entirely — rather than the alertmanager source the rest of this series uses. Cleaner for a standalone tutorial with no risk of colliding with any other pipeline’s trigger.

pipelines/severity-classifier.yaml:

name: severity-classifier
description: Classify alert severity deterministically from an error-rate figure
trigger:
  match: { event: error-rate-spike }

context_template:
  include:
    - metadata.error_rate_pct

steps:
  - name: classify
    executor: shell
    executor_config:
      script: classify-severity
      args: ["{{error_rate_pct}}"]
    confidence_threshold: 0.5
    on_low_confidence: escalate

context_template.include matters here in a way it didn’t in the first tutorial — top-level fields like labels.* are available in every template without being listed, but a generic-source payload’s data block lands in metadata, and that needs to be listed explicitly to flatten metadata.error_rate_pct into the bare {{error_rate_pct}} the step’s args reference. Leave it out and the step fails immediately with UndefinedError: 'metadata' is undefined — before the script ever runs, since Jinja rendering happens first.

Reload:

curl -X POST http://localhost:8000/reload \
  -H "Authorization: Bearer $(grep VECTORSTEP_ADMIN_TOKEN ~/.vectorstep/.env | cut -d= -f2)"

4. Trigger it — critical path

curl -X POST "http://localhost:8000/webhook?source=generic&allow_testing=true" \
  -H "Authorization: Bearer $(grep VECTORSTEP_WEBHOOK_TOKEN ~/.vectorstep/.env | cut -d= -f2)" \
  -H "Content-Type: application/json" \
  -d '{
    "pipeline": "severity-classifier",
    "event": "error-rate-spike",
    "summary": "Error rate spike detected for payments-api",
    "data": {"service": "payments-api", "error_rate_pct": "12.5"}
  }'

What you should see

This was run for real against a live instance — not simulated. 12.5% is above the SEV1 threshold, so the script exits 1:

{
  "confidence": 0.0,
  "summary": "script 'classify-severity' exited 1",
  "next_step_context": "SEV1: error rate 12.5% is critical (>=10%) — escalating\n"
}

Run status: escalated. No LLM was called — next_step_context is the script’s own stdout, byte for byte, truncated to 4000 characters if it ran long. This is the whole point: the same input produces the same classification every time, and a human sees it immediately because the step’s own exit code drove confidence below on_low_confidence’s threshold, not because anything downstream had to re-interpret the output.

5. Trigger it — a lower reading

Same webhook, a smaller error_rate_pct:

curl -X POST "http://localhost:8000/webhook?source=generic&allow_testing=true" \
  -H "Authorization: Bearer $(grep VECTORSTEP_WEBHOOK_TOKEN ~/.vectorstep/.env | cut -d= -f2)" \
  -H "Content-Type: application/json" \
  -d '{
    "pipeline": "severity-classifier",
    "event": "error-rate-spike",
    "summary": "Minor error rate uptick for payments-api",
    "data": {"service": "payments-api", "error_rate_pct": "6.2"}
  }'

6.2% clears the SEV1 bar but not SEV2 — the script exits 0, confidence is 1.0, run status completed:

{
  "confidence": 1.0,
  "summary": "script 'classify-severity' exited 0",
  "next_step_context": "SEV2: error rate 6.2% is high (>=5%)\n"
}

Same script, same deterministic logic, two different outcomes purely from the input — nothing about the pipeline’s gating changed between the two runs.

What’s actually enforced here

Worth being explicit about, since it’s easy to skim past as “just another executor”:

• No free-text command field exists on this executor at all — not “disabled by default like security.allow_shell_checks,” genuinely absent from the schema. executor_config.script must match an allowed[].name entry; there is nothing a pipeline author can put in executor_config that runs arbitrary code.
• The checksum is re-verified before every single execution, not just once at startup — edit scripts/classify_severity.py after enabling this and the very next run fails with a checksum-mismatch error instead of silently running the changed script.
• POST /pipelines/POST /steps reject a step referencing an unlisted script before it’s ever written to disk — try changing script: above to a name you haven’t allowlisted and reload; the error names the step and says plainly that this can’t be fixed by retrying with different pipeline YAML, only by an operator editing config.yaml.
• The subprocess gets no environment by default — unlike the older type: shell deterministic check, which inherits this process’s full environment including provider API keys. executor_config.env: {KEY: "${ENV_VAR}"} opts keys in individually if a script genuinely needs one.

On Kubernetes

Everything above works the same on a cluster, with two differences: the scripts come from a read-only ConfigMap (mounted with defaultMode: 0555, otherwise the service rejects them as not executable), and a script change needs a pod restart rather than a reload. See Changing config on Kubernetes → Shell-step scripts.

Where next

• Executors — the full executor: shell reference: cpu_seconds/mem_bytes/ file_bytes resource limits, timeout_seconds, and the process containment behind them.
• Configuration reference — security.shell_steps in full, alongside allow_shell_checks and template_sandbox.
• Skills — Running a skill’s bundled scripts — the same content-vs-execution-approval model, ported to the Gateway for a skill’s own bundled scripts, invoked by an agent rather than a pipeline step directly.
• Adding trust, one signal at a time — how a deterministic step like this one fits alongside confidence thresholds and verifiers in the broader trust model.