Changing config on Kubernetes

VectorStep is built to sit in an engineering team’s delivery flow: a pipeline or agent changes in git, and the running system picks it up without a restart. Both services hot-reload. On Kubernetes the only real question is how the changed files get into the pod, and there are three supported answers. This page explains each, tells you how to choose, and points at copy-and-tweak templates for all three.

Short version

Not sure which to pick? Use Route A — API push. It’s the only route that checks a change before it touches the running system, it goes live in seconds, and it needs no changes to the shipped manifests. Pick Route C if your platform team requires every object in the cluster to be declared in git — either applied by a pipeline (C1) or reconciled by Argo CD / Flux (C2).

Git is the source of truth. VectorStep never writes to git — the pod’s volume is a deployed copy of what’s in your repo. (The UI’s pipeline pages are read-only previews for exactly this reason: “copy the YAML and ship it through git”.) Whichever route you choose, the workflow is the same as on a Linux VM: change a file in git, CI delivers it, the service reloads.

What reloads, and what doesn’t

There are two processes and they reload independently.

You changed	It lives in	Reload
Pipelines, library steps	the service — /data/pipelines, /data/steps	POST /reload on the service
pricing:, replay:, security policies	the service — config.yaml, /data/security_policies	POST /reload on the service (once the ConfigMap has synced)
Agents (agent.yaml, soul.md), skills	the Gateway — /data/agents, /data/skills	POST /reload on the Gateway (one call reloads both)
auth, server, database, TLS, mcp_servers, providers, the Gateway’s writes_enabled	config.yaml (ConfigMap)	restart the pod (kubectl rollout restart) — for tool servers see Adding MCP tool servers
Shell-step scripts and their sha256 pins	a read-only ConfigMap at /etc/vectorstep/scripts, and security.shell_steps in config.yaml	restart the pod — the script list is read once at startup

A reload swaps the in-memory registry from disk; in-flight runs are not interrupted, and scheduled pipelines are re-registered. Every reload and every write endpoint needs the admin token — the service’s is VECTORSTEP_ADMIN_TOKEN, the Gateway’s is VECTORSTEP_GATEWAY_ADMIN_TOKEN (both come from the Secrets in the shipped manifests).

Caution

Changing an agent and reloading the service does nothing — agents belong to the Gateway. If an agent change “didn’t take”, check which one you reloaded.

Choosing a route

	A — API push	B — copy files	C — GitOps ConfigMap
How it works	HTTPS calls to the write API	kubectl exec + tar into the pod, then reload	ConfigMaps mounted read-only + a reload sidecar
Checked before going live	Yes — a bad change is rejected	No — but rolled back automatically if the reload is rejected	No — reload is rejected; the bad file stays (fix forward)
What delivers it	CI, over HTTPS	CI, with kubectl access (RBAC provided)	C1: CI running kubectl apply (RBAC provided) · C2: Argo CD / Flux — no CI credentials at all
Exposes anything outside the cluster	an Ingress, or an in-cluster runner	no	no
Time until live	seconds	seconds	about 1–2 minutes (Kubernetes ConfigMap sync)
Agents	✅	✅	✅
Skills	✅	✅	✅ — packed as one tarball each
Works with Argo CD / Flux	no	no	yes — see C2
UI/API edits	allowed	allowed	blocked — writes_enabled: false

Templates for all three live in the k8s/delivery/ folder of the VectorStep-Dist repo, next to the base manifests: scripts you copy into your config repo, GitLab CI examples, a least-privilege RBAC identity, and the Deployment patches for route C. The scripts are plain bash and work from any CI — GitLab, GitHub Actions, Jenkins — or from a laptop. The gitlab-ci.example.yml files are thin wrappers and are templates to adapt.

Route A — API push

CI calls the write API to create or update each file. Each call is validated first, written atomically, and followed by a reload in the same request — so one call both ships the change and makes it live, and a bad change is rejected and leaves the running config untouched.

push.sh wraps this for a whole config directory:

export VECTORSTEP_URL=https://vectorstep.example.com   VECTORSTEP_ADMIN_TOKEN=...
export GATEWAY_URL=https://gateway.example.com         GATEWAY_ADMIN_TOKEN=...

./push.sh validate config    # dry run — changes nothing; run on every merge request
./push.sh apply    config    # create-or-update everything; live immediately

config/
  steps/*.yaml
  pipelines/*.yaml
  agents/<name>/agent.yaml + soul.md

It applies in dependency order (steps, agents, then pipelines), is idempotent, and exits non-zero if any file is rejected, printing the reason:

ok    step      service-health-check.yaml
ok    agent     cfg-agent
FAIL  pipeline  broken.yaml
        trigger: Field required
        steps: Input should be a valid list

Things to know:

• validate checks against what is already live. A pipeline that uses a step added in the same change fails validation until that step has been applied. That’s expected, not a bad file; push.sh prints a hint.
• Deleting a file in git doesn’t delete it from the cluster. Remove it with DELETE /pipelines/{name} (or /steps, /agents).
• Reaching the Gateway. The shipped manifests expose the Gateway as a ClusterIP Service only. Run the CI job on a runner inside the cluster and use the in-cluster URL (http://vectorstep-gateway.<ns>.svc:18780), or — if you must — expose it with gateway-ingress.example.yaml. That Ingress exposes an admin-token-protected write API; restrict it to your CI runners’ addresses and keep the token in a masked, protected CI variable. If an in-cluster runner is blocked by the example NetworkPolicy, uncomment its ci-runners rule.
• Prefer a runner-less route? Use B or C — they expose nothing.

Route B — copy files

This is the closest equivalent to “copy the file onto the VM and reload”. CI needs only kubectl access; no VectorStep token ever leaves the cluster, because the reload is made from inside the pod using the admin token already in its environment.

./sync.sh service config     # config/pipelines, config/steps  -> service pod
./sync.sh gateway config     # config/agents,    config/skills -> gateway pod

sync.sh is built to be safe on a live system:

1. snapshots the target directories in the pod,
2. copies your files in,
3. reloads — and if the reload is rejected, it restores the snapshot, reloads again, and exits non-zero so the CI job fails.

So a bad file never stays on disk. Without REMOVE_MISSING=1 it only adds and overwrites files; with it, the target directories are made to match your repo exactly (anything not in git is deleted — use this only if all config is git-managed).

Give CI a least-privilege identity with ci-rbac.example.yaml: it can look at the two Deployments and exec into their pods, and nothing else — it can’t read Secrets or change workloads. pods/exec is a powerful verb (those pods hold the admin tokens), so store the credential as a protected CI variable and prefer a short-lived token (kubectl create token vectorstep-ci --duration=15m).

Doing it by hand is the same thing:

POD=$(kubectl -n vectorstep get pod -l app=vectorstep -o name | head -1)
tar -C config/pipelines -cf - . | kubectl -n vectorstep exec -i "$POD" -c vectorstep -- tar -C /data/pipelines -xf -
kubectl -n vectorstep exec "$POD" -c vectorstep -- python -c "
import os, urllib.request as u
print(u.urlopen(u.Request('http://localhost:8000/reload', method='POST',
      headers={'Authorization': 'Bearer ' + os.environ['VECTORSTEP_ADMIN_TOKEN']})).read().decode())"

(The container images don’t include kill or curl, so kubectl exec ... kill -HUP won’t work — use the Python one-liner above. SIGHUP itself is honoured if you can deliver it.) Files copied in only take effect when the reload runs.

Route C — GitOps ConfigMaps

For teams where everything in the cluster is declared in git. Pipelines, steps, agents and skills live in ConfigMaps mounted read-only into the pods, and a small reloader sidecar turns “the ConfigMap changed” into a hot reload.

How it fits together:

• Pipelines and steps are ConfigMaps (vectorstep-pipelines, vectorstep-steps) mounted over /data/pipelines and /data/steps.
• Agents are one flat ConfigMap (vectorstep-agents) with keys <agent>.agent.yaml and <agent>.soul.md. ConfigMap keys can’t contain /, so an init container and the sidecar assemble the agents/<name>/ directories from it. Adding an agent is therefore just a ConfigMap change — no Deployment edit, no restart.
• Skills are directories with nested files, so they travel as one tarball each (incident-triage.tar.gz) in a single ConfigMap (vectorstep-skills). The same init container and sidecar unpack them into /data/skills, keeping the nesting and the execute bit on scripts. apply.sh packs them for you from config/skills/<name>/, deterministically, so an unchanged skill produces an unchanged ConfigMap and nothing reloads. A tarball that is unsafe (a path that escapes the skill, a symlink) or has no SKILL.md is rejected, that skill keeps its previous version, and the reload is reported as rejected so the CI job fails.
• The config-reloader sidecar (it runs inside the VectorStep image, so there’s nothing extra to pull or vet) checks the mounted files every 10 seconds and calls POST /reload when they change.
• writes_enabled: false in each service’s config.yaml makes the write API refuse with a clear 403 writes_disabled — the mounts are read-only, so letting someone try would only produce a confusing error. validate and /reload keep working. See writes_enabled.

The mounts, the sidecar and the reload behaviour are identical whichever way the ConfigMaps reach the cluster. There are two ways to do that:

	C1 — a pipeline applies them	C2 — Argo CD / Flux reconcile them
Who creates the ConfigMaps	apply.sh, run by your CI	a GitOps controller, from manifests in git
Anyone runs a script	yes, on every change	no
CI needs cluster access	yes (kubectl)	no
Job fails if the reload is rejected	yes	no — watch the sidecar log instead

One-time setup (both)

1. Set writes_enabled: false in both services’ config.yaml (ConfigMap), apply, and restart.
2. Mount the ConfigMaps and add the sidecar to each Deployment. The two patch files are ordinary Kubernetes strategic-merge patches:

kubectl -n vectorstep patch deploy/vectorstep         --patch-file service-patch.yaml
kubectl -n vectorstep patch deploy/vectorstep-gateway --patch-file gateway-patch.yaml

(Under C2 you make the same change in your GitOps repo instead — see below.)

C1 — a pipeline applies them

Create the ConfigMaps from your config directory, then do it again whenever something changes:

./apply.sh config vectorstep

config is laid out the same as for route A:

config/
  pipelines/*.yaml
  steps/*.yaml
  agents/<name>/agent.yaml + soul.md
  skills/<name>/SKILL.md (+ reference/, scripts/, …)

Packing skills needs python3 on the machine running apply.sh.

apply.sh rebuilds the ConfigMaps from the directory (so route C mirrors git exactly — deleting a file in git removes it), does nothing if nothing changed, and then waits for the sidecar to report the reload and fails the job if it was rejected. Only the service(s) whose ConfigMaps actually changed are waited on. Run it from any CI job that has kubectl access — ci-rbac.example.yaml gives it a least-privilege identity.

C2 — Argo CD or Flux reconcile them

Here nobody runs a script. Your GitOps controller watches a repo, applies the manifests in it, and the sidecar does the rest: the controller updates the ConfigMap, Kubernetes syncs it into the pod, the sidecar notices and reloads.

What goes in the GitOps repo:

• The Deployment changes from service-patch.yaml and gateway-patch.yaml, applied as kustomize patches: (Argo CD renders kustomize natively; Flux uses a Kustomization).
• The ConfigMaps, generated from your config files. The file names in the agents ConfigMap are the flat keys the sidecar expects, so list them with key=path:

kustomization.yaml

configMapGenerator:
  - name: vectorstep-pipelines
    options: { disableNameSuffixHash: true }   # keep the names stable — see the note below
    files:
      - pipelines/service-health.yaml
      - pipelines/disk-pressure.yaml
  - name: vectorstep-steps
    options: { disableNameSuffixHash: true }
    files:
      - steps/check-pod.yaml
  - name: vectorstep-agents
    options: { disableNameSuffixHash: true }
    files:
      - first-responder.agent.yaml=agents/first-responder/agent.yaml
      - first-responder.soul.md=agents/first-responder/soul.md
  - name: vectorstep-reloader
    options: { disableNameSuffixHash: true }
    files:
      - reloader.py              # copy from k8s/delivery/c-gitops-configmap/
patches:
  - path: service-patch.yaml
    target: { kind: Deployment, name: vectorstep }
  - path: gateway-patch.yaml
    target: { kind: Deployment, name: vectorstep-gateway }

• The skills ConfigMap, which kustomize can’t build because skills are packed into tarballs. Run pack-skills.py in CI and render the ConfigMap into the repo:

python3 pack-skills.py config/skills build/skills
kubectl create configmap vectorstep-skills --from-file=build/skills \
  --dry-run=client -o yaml > gitops/vectorstep-skills.yaml

Commit vectorstep-skills.yaml next to the kustomization (list it under resources:). The packing is deterministic, so the file only changes when a skill does.

Add a file to the list when you add a pipeline, step or agent. (A generator can’t use wildcards; if you’d rather not maintain the list, have a CI job render the ConfigMaps with kubectl create configmap … --from-file=… --dry-run=client -o yaml and commit the result.)

Two things differ from C1, both because the controller — not a job — is applying the change:

• Nothing fails a job when a reload is rejected. A change that doesn’t pass validation is rejected at reload and the running config stays as it was, but the controller still reports the sync as healthy. Catch bad files before merge by running ./push.sh validate config on the merge request (the validate endpoints still work with writes_enabled: false), and alert on the sidecar’s REJECTED log line or on config_errors in GET /health.
• The names must stay stable. Kustomize’s default behaviour of adding a hash to a generated ConfigMap’s name would roll the pod on every change instead of hot-reloading it, which is why disableNameSuffixHash is set on each generator above. (Set it per generator rather than globally — the one exception is shell-step scripts, which want the hash.)

Skill scripts

A skill can bundle scripts, but an agent can only run one that an operator has approved in the Gateway’s config.yaml, pinned by sha256 (see Running a skill’s bundled scripts). That list is read once at startup, so on route C a script change is a restart-class change, like shell-step scripts:

1. Change the script in git and compute its new sha256 (sha256sum).
2. Put the new sha256 under security.skills.allowed in the Gateway’s config.yaml ConfigMap.
3. Apply both ConfigMaps (apply.sh for the skill, kubectl apply for the config) and restart the Gateway: kubectl -n vectorstep rollout restart deploy/vectorstep-gateway.

If you change a pinned script and skip the pin, the reload still reports success, but the Gateway drops that script from its allowlist and logs security.skills.allowed: … sha256 mismatch — dropping entry. The agent can no longer run it until the pin and a restart catch up. Scripts you haven’t approved are unaffected, and skill content (SKILL.md and reference files) still hot-reloads normally.

Upgrading an existing route C install

If you set up route C before skills were supported, bring it up to date in this order — the order matters:

1. Copy the current apply.sh, reloader.py and the new pack-skills.py from the c-gitops-configmap/ folder into your config repo.

2. Run ./apply.sh config vectorstep. This creates the new vectorstep-skills ConfigMap (empty if you have no skills/ yet) and refreshes vectorstep-reloader with the new sidecar code.

3. Re-apply the current gateway-patch.yaml:

   kubectl -n vectorstep patch deploy/vectorstep-gateway --patch-file gateway-patch.yaml

   The Gateway restarts once to pick up the skills mount. Re-applying over the old patch is safe — it only adds the new volumes, mounts and variables.

Do step 2 before step 3: the patched Deployment mounts vectorstep-skills, and a pod can’t start while a ConfigMap it mounts doesn’t exist. The service needs nothing — service-patch.yaml is unchanged. Under route C2, add the skills ConfigMap and the updated Gateway patch to your GitOps repo instead.

Timing

About 70 seconds from the ConfigMap changing to both services reloaded — Kubernetes syncs a changed ConfigMap into a pod asynchronously (typically a minute or so) and the sidecar notices within its 10-second interval. That’s a Kubernetes property, not something VectorStep adds.

Limits to know about

• No pre-flight check. A pipeline that fails validation is rejected at reload (the running config is untouched), but the bad file is now in the ConfigMap, so a pod restart would skip it. Catch it earlier by running push.sh validate on the merge request and fix forward promptly (see Recovering from a bad file).
• ConfigMaps are capped at 1 MiB each. All skills share one ConfigMap, and binary data is base64-encoded, so budget about 750 KB of packed skills in total (text compresses well, so that is a lot of Markdown). apply.sh checks the size first and, if you’re over, lists the largest skills. If you outgrow it, shrink the big reference files or deliver skills with route A or B, which have no such limit.
• Never use a subPath mount for any of this: a subPath ConfigMap mount never receives updates.

Shell-step scripts

executor: shell steps run pre-approved scripts from security.shell_steps.scripts_dir (see Run a shell step). Scripts are deliberately not part of routes A, B or C: they must never be reachable through the write API, so they don’t live on the /data volume. On Kubernetes they are a read-only ConfigMap mounted at /etc/vectorstep/scripts, and that is true whichever route you use for everything else.

shell-scripts.sh and the manifests are in the k8s/delivery/ folder of the VectorStep-Dist repo, with the route templates.

1. Mount the scripts. Uncomment the scripts volume and volumeMount in service/deployment.yaml. The volume must set defaultMode: 0555:

volumes:
  - name: scripts
    configMap:
      name: vectorstep-shell-scripts
      defaultMode: 0555      # required — see below

A ConfigMap volume mounts files as 0644 by default, and the service refuses a script without the execute bit. Leave defaultMode out and the log says security.shell_steps.allowed: rejected — … script file is not executable (missing +x) and 0 scripts are allowlisted.

2. Enable shell steps and pin each script. In the service’s config.yaml, set security.shell_steps.enabled: true and list every script with its sha256. shell-scripts.sh prints the entries for you:

classify-severity.sh

./shell-scripts.sh sha256 scripts
# allowed:
#   - name: classify-severity
#     sha256: 5bb15e0e…

3. Create the ConfigMap and restart:

./shell-scripts.sh apply scripts vectorstep

apply creates or updates the vectorstep-shell-scripts ConfigMap from the directory and, if anything changed, does a rolling restart and waits for it. It does nothing when the scripts are unchanged.

Changing a script

Scripts take effect on a restart, not a reload. The allowlist and its sha256 pins are read once at startup, on purpose, so what is allowed to run can’t change underneath a running service. Updating the ConfigMap without restarting makes the next shell-step run fail its checksum check (the running service still holds the old sha256) until the pod restarts. On v0.1.17 and earlier the failure is a less obvious FileNotFoundError for the script’s path — same fix, restart the pod.

The order for a change:

1. Edit the script.
2. Run ./shell-scripts.sh sha256 scripts and put the new sha256 in config.yaml.
3. Apply the config ConfigMap, then run ./shell-scripts.sh apply scripts — it restarts the pod, which picks up the new script and the new pin together.

Forget step 2 and the service rejects the script on the sha256 mismatch rather than running it. Catch that on the merge request instead:

./shell-scripts.sh check scripts config.yaml    # fails if any script's sha256 isn't pinned

apply needs permission to create and patch ConfigMaps and to patch the vectorstep Deployment; ci-rbac.example.yaml has an optional commented rule for this. If your CI can’t restart the Deployment, run it with RESTART=0 and restart separately.

With Argo CD or Flux (route C2)

Put the scripts ConfigMap in your kustomization without disableNameSuffixHash. Kustomize then adds a content hash to its name and rewrites the reference in the Deployment, so every script change rolls the pod — exactly the restart a script change needs. Keep the config.yaml ConfigMap that holds the pins hashed too, so a pin and its script always arrive in the same rollout.

configMapGenerator:
  - name: vectorstep-shell-scripts       # hashed on purpose: a change rolls the pod
    files:
      - scripts/classify-severity.sh

and set defaultMode: 0555 on the volume in your Deployment patch.

writes_enabled

A new setting, true by default (nothing changes unless you set it). When false, every write endpoint — POST/PUT/DELETE on /pipelines, /steps (service) and /agents, /skills (Gateway) — returns

{"detail": {"type": "writes_disabled", "message": "Pipeline and step writes are disabled on this deployment (writes_enabled: false) — its config is managed declaratively. Change the files in git and apply them (e.g. update the ConfigMap, then POST /reload)."}}

with HTTP 403, instead of the bare 500 Internal Server Error you’d otherwise get from a read-only filesystem. Reads, the /validate endpoints and /reload are unaffected. The check runs after the admin-role check, so an unauthenticated caller gets a 401 and can’t use it to probe the setting. The service re-reads it on /reload; the Gateway reads it at startup.

Recovering from a bad file

A pipeline or step file that fails validation can end up on the volume with routes B and C (route A rejects it up front, and sync.sh rolls it back). What happens next depends on whether the service is running or starting:

• Running, then a reload: the reload is rejected with the file and the reason, and the running config is untouched. Nothing is lost.
• Starting (a pod restart, a node drain): the service skips the bad file, logs an ERROR, and starts with everything else — releases after v0.1.10. One bad file no longer takes every other pipeline down with it. The cost is that the skipped pipeline handles no alerts until it is fixed, so it is deliberately hard to miss: an ERROR line per file and a summary (STARTED WITH 1 CONFIG FILE(S) SKIPPED (…)) in the log, "config_errors": N on GET /health (which still returns 200, so probes keep passing), and a red banner on every UI page naming the file and the first line of the error.

To recover, fix or remove the file and reload — the banner and the config_errors count clear once every file loads. Note that while a skipped file is still on disk, a reload and writing any other pipeline through the write API are rejected with an error naming that file (a change is only accepted if the whole directory is valid); pushing a corrected version of the bad pipeline itself, or removing it, unblocks everything.

On route C the fix is to correct the file in git and re-apply (re-run apply.sh, or let your GitOps controller sync). On v0.1.10 and earlier, a bad file on disk makes the service refuse to start — the pod goes into CrashLoopBackOff with a ValidationError for that file in its log. If that happens on a PVC-backed install, remove the file with a one-off pod that mounts the same volume, then scale back up:

kubectl -n vectorstep scale deploy/vectorstep --replicas=0
kubectl -n vectorstep apply -f - <<'YAML'
apiVersion: v1
kind: Pod
metadata: { name: vs-recover }
spec:
  restartPolicy: Never
  securityContext: { runAsUser: 1000, runAsGroup: 1000, fsGroup: 1000 }
  containers:
    - name: r
      image: ghcr.io/bantex01/vectorstep:latest
      command: ["sh", "-c", "ls /data/pipelines; rm -v /data/pipelines/THE-BAD-FILE.yaml"]
      volumeMounts: [{ name: data, mountPath: /data }]
  volumes: [{ name: data, persistentVolumeClaim: { claimName: vectorstep-data } }]
YAML
kubectl -n vectorstep logs vs-recover && kubectl -n vectorstep delete pod vs-recover
kubectl -n vectorstep scale deploy/vectorstep --replicas=1

Confirming a reload worked

• The service’s POST /reload returns {"status": "reloaded", "pipelines_loaded": N} — check N.
• The Gateway’s returns the loaded agent and skill lists.
• The logs record it: kubectl logs deploy/vectorstep | grep Reloaded.
• On route C (C1 and C2) the sidecar logs change detected … reload ok (or REJECTED with the reason): kubectl logs deploy/vectorstep -c config-reloader.

Related

• Kubernetes install
• Write API — the endpoints route A calls
• REST API → Reload and Creating agents → Hot reload
• Securing a deployment — the admin tokens used here
• Crash with exit code 132 on an Apple M4 VM — if you test on a local multi-node VM cluster
• Scaling & availability — why this is single-replica today