Skip to content
VectorStep
Blog

Webhook says accepted but the run 404s

Symptom: POST /webhook returns {"status": "accepted", "run_id": "abc123"} — a clean, successful-looking response — but GET /runs/abc123, or opening that run in the UI, comes back 404. The alert wasn’t dropped; a different run handled it. This page explains why, and how to confirm it.

Why this happens

Section titled “Why this happens”

VectorStep guarantees that two overlapping triggers for the same alert never run as two separate pipelines at once — that’s a deliberate, database-enforced safety property, not an edge case. If two webhook calls for the same alert arrive close together, only one of them actually creates a run.

The catch is in the timing: the “accepted” response is sent before that check finishes, so both callers can be told “accepted,” each with their own run_id — and then, a moment later, one of those two run_ids turns out to belong to a run that was never actually created. That run_id then 404s, permanently, on every future request. See Webhook sources — Race safety for the exact mechanics.

This is not data loss. The alert itself was still handled — by the run that won — you just weren’t handed that run’s real run_id.

How to confirm this is what happened

Section titled “How to confirm this is what happened”
  1. Note the alert’s identity (for Alertmanager, the fingerprint label; for a generic source, the idempotency_key you sent).
  2. Check /ui/runs (or GET /runs?pipeline_name=...) around the same timestamp for a different run against the same pipeline. If you find one, that’s almost certainly the run that actually processed your alert.
  3. If the pipeline has trigger.dedup.enabled: false (common for pipelines you’re actively re-triggering by hand, since it’s a “quick-start convenience” for exactly that), don’t assume dedup is fully off — the database-level protection above still applies regardless of this setting. dedup.enabled: false only turns off the window-based re-fire suppression (a completed run within N seconds), not the in-flight race guard.

What actually causes the race in practice

Section titled “What actually causes the race in practice”

The most common trigger is firing the same alert multiple times in quick succession — deliberately, while testing, or because a monitoring system (Alertmanager included) genuinely does resend a still-firing alert on its own evaluation interval. If you’re scripting repeated triggers for testing (building up run history, load-testing a pipeline), wait for each run to actually finish before firing the next one, rather than firing on a fixed short interval — these runs can take anywhere from 30 seconds to a couple of minutes depending on the pipeline, and firing faster than that reliably reproduces this exact symptom:

Terminal window
run_id=$(curl -s -X POST "http://localhost:8000/webhook?source=alertmanager&allow_testing=true" \
  -H "Authorization: Bearer $WEBHOOK_TOKEN" \
  -H "Content-Type: application/json" \
  -d @webhooks/alertmanager_critical.json | python3 -c "import json,sys; print(json.load(sys.stdin).get('run_id',''))")


while :; do
  status=$(curl -s "http://localhost:8000/runs/$run_id" -H "Authorization: Bearer $ADMIN_TOKEN" \
    | python3 -c "import json,sys; print(json.load(sys.stdin).get('status','?'))" 2>/dev/null)
  [ "$status" != "running" ] && [ -n "$status" ] && break
  sleep 5
done

Where next

Section titled “Where next”