Observability

The service exposes three complementary ways to see what it’s doing: cumulative Prometheus metrics for trends, per-run OpenTelemetry traces for drill-down, and rotating log files for the raw event stream. This page covers all three.

Prometheus metrics

GET /metrics exposes Prometheus text-format metrics, computed from pipeline_runs / pipeline_steps at scrape time. All counters are cumulative all-time totals — use rate()/ratios in PromQL for escalation rate, per-agent success rate, and verifier veto frequency rather than relying on pre-baked percentages.

Every metric below is scoped to stage=production — a stage=testing pipeline (the default, see Pipeline stages) contributes to none of them, including the metrics that query pipeline_steps without otherwise touching pipeline_runs.

Metric	Type	Labels	Description
vectorstep_pipeline_runs_total	counter	pipeline, status	Total runs by pipeline and terminal status
vectorstep_pipeline_runs_in_progress	gauge	—	Runs currently in status=running
vectorstep_pipeline_steps_total	counter	pipeline, step_name, executor, agent, model, provider, status	Total steps by pipeline, step, executor, agent, model, provider, and status. pipeline/step_name/model/provider are what let a Grafana dashboard reconstruct the per-step and per-model success-rate breakdowns the Pipelines/Steps/Agents Insights UI pages compute directly from the DB — the UI is for a quick look, this metric is for a real dashboard or alert. NULL agent/model/provider (non-gateway executors) are bucketed as "".
vectorstep_pipeline_step_duration_seconds	histogram	pipeline, step_name, executor, agent, model, provider	Step execution duration (buckets: 1, 2, 5, 10, 30, 60, 120, 300, 600, 1200, +Inf seconds)
vectorstep_verifier_runs_total	counter	agent	Steps where a verifier ran, by primary agent
vectorstep_verifier_overrides_total	counter	agent	Verifier runs where the verifier lowered the primary’s effective confidence
vectorstep_pipeline_tokens_total	counter	team, pipeline, step_name, executor, agent, model, provider, direction	Cumulative input/output tokens consumed, broken down by owning team (Team attribution) for cost attribution. direction is input/output. NULL team/model/provider are bucketed as "" rather than dropped, so unattributed spend stays visible. Steps from executors that don’t report tokens (openclaw, human, webhook) are excluded rather than padded as zero.
vectorstep_pipeline_cache_tokens_total	counter	team, pipeline, step_name, executor, agent, model, provider, direction	Cumulative Anthropic prompt-cache tokens. direction is read/write. Billed at their own rates and not included in vectorstep_pipeline_tokens_total, which keeps counting only input and output. Only steps that reported cache tokens contribute. See Prompt-cache tokens.
vectorstep_human_approval_decisions_total	counter	team, pipeline, decision	Cumulative human step approve/reject decisions. decision is approved/rejected, derived from primary_confidence (1.0/0.0 — see the executor’s contract). Timeouts leave primary_confidence NULL and are excluded rather than miscounted as either outcome. NULL team is bucketed as "".
vectorstep_pipeline_feedback_total	counter	pipeline, outcome	Cumulative human accuracy feedback. outcome is correct/partial/incorrect.
vectorstep_step_feedback_total	counter	pipeline, step_name, agent, model, provider, outcome	Cumulative per-step human accuracy feedback, production-scoped. outcome is correct/partial/incorrect.
vectorstep_step_grounding_score	histogram	pipeline, step_name, agent, model, provider	Shadow-mode grounding score (G) distribution per step, production-scoped (buckets: 0.1, 0.2, …, 1.0, +Inf). Only steps with a grounding: block that produced a score contribute — NULL/not-computed steps are excluded, not padded as zero.
vectorstep_step_deterministic_check_total	counter	pipeline, step_name, outcome	Cumulative whole-step deterministic-check outcomes, production-scoped. outcome is passed/failed — passed only when every declared check for that step run passed. Steps with no deterministic_checks: declared are excluded.
vectorstep_human_approvals_pending	gauge	team	Currently pending human step approvals, awaiting a response on whichever channel (Telegram/Slack/Teams) that team is routed to. Unlike every other metric here this isn’t derived from the database — pending approvals are in-memory only — so it reflects only this process’s current state, not a historical/cumulative total. NULL team is bucketed as "". Always emits at least a zero-valued series so the metric doesn’t disappear from dashboards when nothing’s pending. Excludes stage=testing pending approvals, same as every other series on this page.
vectorstep_pipeline_cost_total	counter	pipeline, team, model, provider, source	Cumulative step cost in pricing.currency’s units, split by pricing tier (source is manual|live_exact|live_cross_provider). Unpriced steps (no tier match, or no token data) are excluded rather than padded as zero, same treatment as the tokens metric. One honest series rather than a separate approximate metric — use sum without(source) for the total, or filter on source for just the tiers you trust. See Cost accounting.
vectorstep_team_budget_ratio	gauge	team	Month-to-date spend / pricing.team_budgets[team], UTC calendar month. Only teams with a configured budget get a series. Advisory only — never gates a run.
vectorstep_runs_resumed_total	counter	pipeline	Total runs resumed after a restart. See Durability & resume.
vectorstep_replay_batches_total	counter	step_name	Total replay batches launched via POST /steps/{name}/replay. See Replay / shadow evaluation.
vectorstep_replay_steps_total	counter	step_name, grade	Total replay sample outcomes. grade is completed|deterministic_failed|execution_failed|unreplayable. See Replay / shadow evaluation.

Standard python_* / process_* / python_gc_* process-health metrics are included automatically via prometheus_client’s default collectors.

OpenTelemetry tracing

While /metrics gives aggregate, all-time counters, distributed tracing gives a per-run drill-down — one trace per pipeline run, with a span for every step, branch, verifier, and underlying LLM call. Use /metrics to spot trends (“escalation rate is up this week”); use traces to answer “why was this run slow / why did this step escalate”.

Tracing is disabled by default and adds zero overhead until enabled — the OTel SDK’s default TracerProvider is a no-op.

Enabling tracing — add to config.yaml:

observability:
  otel:
    enabled: true
    exporter: otlp          # otlp | console
    endpoint: http://localhost:4318/v1/traces   # OTLP/HTTP collector endpoint
    service_name: vectorstep-service

exporter: console prints spans to stdout — useful for local verification without a collector. exporter: otlp (default) sends spans via OTLP/HTTP to a collector (e.g. Grafana Alloy/Tempo).

Span hierarchy — each pipeline run is its own trace root:

pipeline.run: <team>/<pipeline>       (vectorstep.pipeline.name, vectorstep.run.id, vectorstep.source, vectorstep.team, vectorstep.run.status)
├── <step name>                       (vectorstep.span.kind=step, vectorstep.executor, vectorstep.agent, confidences, vectorstep.model, vectorstep.provider, vectorstep.prompt_hash, vectorstep.agent_version)
│   ├── gen_ai.<executor>             (vectorstep.span.kind=gen_ai — the LLM call itself)
│   └── <step>:verifier|:independent   (vectorstep.span.kind=verifier, vectorstep.verifier.mode, vectorstep.confidence)
│       └── gen_ai.<executor>
├── <group name>                      (vectorstep.span.kind=parallel_group, vectorstep.join_strategy, vectorstep.branch_count, vectorstep.agent_version)
│   ├── <branch name>                 (vectorstep.span.kind=branch, vectorstep.executor, vectorstep.agent, vectorstep.confidence, vectorstep.model, vectorstep.provider)
│   │   └── gen_ai.<executor>
│   └── ... (concurrent siblings)
├── <fan_out name>                    (vectorstep.span.kind=fan_out, vectorstep.join_strategy, vectorstep.prompt_hash, vectorstep.agent_version)
│   ├── <fan_out name>/0              (vectorstep.span.kind=branch, vectorstep.executor, vectorstep.agent, vectorstep.confidence)
│   │   └── gen_ai.<executor>
│   └── ... (one branch per runtime list item)
└── ...

vectorstep.prompt_hash/vectorstep.agent_version are set on the <step name> span (from that step’s own prompt_template) and the <fan_out name> group span (all branches of a fan-out share one template, so a single hash is meaningful there) — but deliberately not on individual branch spans or the <group name> parallel_group span’s prompt_hash, since each parallel branch has its own distinct template and no single hash would be meaningful at the group level. vectorstep.agent_version is set wherever a single agent’s output is available, including the parallel group span. Neither is added as a Prometheus metric label — unbounded cardinality, since every prompt/agent edit mints a new, permanently-retained label value.

The root span name is pipeline.run: <team>/<pipeline-name> when a team is attributed (e.g. pipeline.run: payments/alert-triage-critical), or pipeline.run: <pipeline-name> for unattributed runs. This makes the Name column in Grafana Tempo immediately useful without needing to expand attributes. vectorstep.team is also set as a span attribute when present, so traces can be filtered/grouped by team in PromQL or Tempo query expressions.

Each run’s structured log event (step_started, verifier_ran, branch_completed, etc. — see Run storage) is also recorded as a span event on the current span, so the structured run log and the trace timeline tell the same story from two angles.

gen_ai.* span attributes:

Attribute	Description
gen_ai.system	Executor name (openclaw, openclaw_ws, gateway)
gen_ai.request.model	Model override requested, if any
gen_ai.response.model	Actual model used, from executor metadata
vectorstep.gateway.duration_ms	Backend-reported call duration
vectorstep.agent	Agent name

Forward compatibility with the VectorStep Gateway: every gen_ai.* span injects the current traceparent/tracestate (W3C Trace Context) into the outbound agent request sent to the OpenClaw/VectorStep Gateway WebSocket APIs. Today’s gateways ignore these extra params. Once the VectorStep Gateway adds its own OTel instrumentation, its LLM/tool-call spans will automatically nest under the corresponding gen_ai.* span — giving one connected trace from webhook → pipeline → step → individual LLM/tool calls, with no further changes needed on the VectorStep side.

Sending this to a backend

Both signals are standard, vendor-neutral output — /metrics is plain Prometheus text format, and traces are OTLP/HTTP. VectorStep has no Grafana-specific (or any other backend-specific) integration beyond the config above: point any Prometheus-compatible scraper at /metrics, and point observability.otel.endpoint at any OTLP-compatible collector — Grafana Alloy, the OpenTelemetry Collector, or a vendor’s own ingest endpoint all work the same way from VectorStep’s side. See OpenTelemetry’s docs for the collector and backend side of this setup — this page only covers what VectorStep itself emits.

Logging

The service writes rotating log files under the directory set by logging.dir (/data/logs by default in the container, auto-created) — omit logging.dir entirely to log to stdout only, which is the usual choice inside a container or Kubernetes pod. Two files are produced:

• service.log — application logs (rotating, 10 MB × 5 files).
• access.log — HTTP access logs, kept separate from service.log so a busy webhook endpoint’s request/response lines don’t drown out actual application events when you’re reading through them.

Both files rotate at 10 MB, keeping 5 generations. logging.level controls verbosity (INFO by default). logging.format chooses between the default text line and json (one object per line); see Correlating logs with traces below. See Deployment for the full logging: block in context, and Configuration reference for every field.

Correlating logs with traces

Every log line from the service and the Gateway carries four fields, so you can go from a line to its trace and from a run to its lines:

2026-11-06T10:15:30 INFO src.pipeline.runner [trace_id=0af7651916cd43dd8448eb211c80319c span_id=b7ad6b7169203331 run=3f2a… step=triage] Step 'triage' verifier: …

• trace_id / span_id: the active OpenTelemetry span. - when tracing is off.
• run / step: the pipeline run and step. - outside a run. They are set per run and per branch, so lines from concurrent runs and parallel branches stay apart. The service sends the run and step to the Gateway with each agent call, so the Gateway’s lines carry them even when tracing is off. Older Gateways ignore them and show -.
• Pipeline name: also set for the whole run and sent to the Gateway, but only in the JSON format (pipeline) and on exported records (vectorstep.pipeline.name), so you can filter a log backend by pipeline. The text line leaves it out to stay short.

A sub-pipeline is its own trace and its own run, with its own IDs.

Set logging.format: json (in both the service and the Gateway) for one JSON object per line. Fields that have no value are left out:

{"ts":"2026-11-06T10:15:30.123Z","level":"INFO","logger":"src.pipeline.runner","message":"…","trace_id":"0af7…","span_id":"b7ad…","run_id":"3f2a…","step":"triage","pipeline":"alert-triage"}

An exception field holds the traceback when there is one. The JSON format is scrubbed for credentials exactly like the text format. The text format changed in this release: it now has the bracketed field block before the message, so anything that parses the old line needs updating, or switch to json. Access-log lines are unchanged.

The run’s trace ID is stored with it and shown on the run page. Set observability.otel.trace_url_template to turn it into a link to your tracing backend:

observability:
  otel:
    enabled: true
    trace_url_template: "https://tracing.example.com/trace/{trace_id}"   # whatever URL opens a trace by ID in your backend

The template must be an http(s) URL containing {trace_id}; anything else is ignored with a warning. Runs made before this release, and runs made with tracing off, have no trace ID. The ID is also in the run API as trace_id, so it reaches the MCP get_run tool.

To find a run’s log lines, filter your log backend on the trace_id (or run_id) field. How you write that filter depends on the backend and on how you ship the logs; with logging.format: json the fields are already structured.

Exporting logs over OTLP

Logs can also be sent to your collector alongside stdout, the same way traces are:

observability:
  otel:
    enabled: true
    endpoint: http://collector:4318/v1/traces
    logs:
      enabled: false              # default
      endpoint: null              # default: the traces endpoint with /v1/traces -> /v1/logs
      level: INFO                 # may be stricter than logging.level
      capture_content: none       # none | output | output_and_prompt  (service only)
      capture_content_max_chars: 4000

• It is additional to stdout, not a replacement. When the collector is down the exporter drops records, so stdout (and the optional log files) remain the record during exactly that kind of incident. Keep them on.
• Records carry the trace and span IDs, vectorstep.run.id, vectorstep.step and vectorstep.pipeline.name, and use the same service.name as traces, so your backend can link a trace to its logs and back. Loki shows these attributes with underscores (vectorstep_step, vectorstep_pipeline_name).
• Credentials are scrubbed from the message, its arguments, attributes and the exception text before export (same best-effort backstop as above).
• A dead collector does not slow or fail a run: export happens on a background thread with a bounded queue.
• If logs.endpoint is unset and observability.otel.endpoint does not end in /v1/traces, the service refuses to start rather than send logs to a guessed URL.
• Auth uses the standard OTEL_EXPORTER_OTLP_HEADERS (or OTEL_EXPORTER_OTLP_LOGS_HEADERS) environment variable, as for traces. Access logs are not exported.

Step output capture is off by default. With capture_content: output, each finished step (and each parallel or fan-out branch) sends its parsed output to the backend as a log record named vectorstep.step.output; output_and_prompt also sends the rendered prompt as vectorstep.step.prompt. Each is cut at capture_content_max_chars, with vectorstep.content.truncated set when it was.

Caution

Step output and prompts can contain customer data, and this copies them to a third-party store with its own retention and access rules. Leave it off unless you have decided that is acceptable. The database stays the system of record. Content goes only to the OTLP backend, never to stdout or the log files. Scrubbing recognises a short list of credential shapes, so it is no substitute for deciding what you are willing to send. Tool calls and tool results are not captured. The Gateway has no content capture.

Secrets in logs

Every log handler scrubs the final line before it is written, so a credential that appears in a message, a %s argument or an exception traceback is replaced with <redacted>. It recognises Telegram bot tokens (which sit in the Bot API URL path), Slack webhook URLs and tokens, bearer tokens and sk- provider keys. This is a best-effort backstop for those known shapes, not a guarantee. A credential in some other format can still reach a log, so keep secrets out of anything you ask the service to log. The Gateway applies the same scrubbing. Earlier versions could write a Telegram bot token to the log; if you ran one of those with Telegram enabled, rotate the token and purge old log files.