How We Built and Validated sandboxctl with a Governed Agentic SDLC

A security-sensitive Kubernetes tool became a test of something larger: whether human-approved obligations, specialist AI agents, blocking review gates, and layered evidence could operate as one software-delivery system.

The interesting question in agentic software engineering is no longer whether a model can produce code. It is whether an organization can turn model-driven work into a delivery process that remains inspectable when requirements change, reviewers find risk, tools fail, context is compacted, and the target system can affect real infrastructure.

Product-status disclosure: OPI Forge is in private beta. This article documents an internal, instrumented delivery case study. It does not announce general OPI Forge availability or a packaged sandboxctl release.

The engineering problem was not just writing a CLI

sandboxctl is a Rust command-line tool for creating and managing Kubernetes namespaces intended to run hostile workloads inside Kata virtual machines. Its design combines runtime isolation, strict admission controls, custom role-based access, short-lived user credentials, resource quotas, storage constraints, and default-deny Cilium network policy.

That breadth made it a demanding delivery test. A defect could live in ordinary code, but it could also hide in Kubernetes API semantics, authorization and impersonation, credential publication, controller ownership, rollback behavior, network-policy attribution, evidence persistence, or cleanup. A superficial “the CLI ran” result would not be enough.

The approved specification and threat model therefore behaved as a binding contract. They defined product behavior, protected targets, accepted limitations, review triggers, and named acceptance scenarios. The implementation could adapt as it encountered the repository and the cluster, but it could not silently weaken those obligations.

Why the scope matters: This is a point-in-time case study of a security-sensitive infrastructure implementation. It is stronger evidence than a small coding demonstration, but it is not a universal benchmark for every codebase, team, or delivery task.

Two worktrees created two autonomy profiles

The project did not begin by asking one agent to “build the system.” It began with a separate specification and threat-model worktree. That phase was intentionally human-intensive because its decisions established the obligations that later automation would have to respect.

During the specification phase, the human operator reviewed material decisions one by one. The record includes approval or correction of quota values, revocation behavior, identity rules, and security language. It also includes stopping the process when it drifted into unsupported reasoning. That concentration of human work was a governance feature: the phase was deciding what the system was allowed to become.

After the baseline was approved, the operating mode changed. The implementation worktree recorded 2,094.3 minutes of summed subagent compute over a 2,441.6-minute wall-clock span. The human retained authority over scope, privileged access, live mutation, acceptance, and handoff requirements, but most bounded delivery work moved to specialist agents.

The pattern is more useful than a binary “autonomous or not” label:

Use human judgment heavily where the work defines obligations and accepted risk, then increase automation inside the approved boundary.

The work-share figures are method-specific estimates, not wall-clock autonomy percentages. Agent work is the sum of timed launch durations; eligible human work is derived from qualifying message intervals. Because concurrent agent durations are summed independently, the percentages must not be read as the share of elapsed time without human involvement.work-share

We used specialists, not one general coding agent

The implementation worktree recorded 14 role classes. The six most active were coders, QA engineers, code reviewers, security reviewers, Kubernetes technical reviewers, and architects.

These are launch counts, not unique people, persistent personas, or separate foundation models.launches

The point of role separation was not to create theatrical job titles. It was to produce different failure lenses. A slice could be correct in one dimension and still be blocked in another. That happened repeatedly:

• Code review found command and state-machine defects.
• QA found evidence that could report PASS without proving the claimed outcome.
• Security review found reachable privilege, secret-handling, race, and trust-boundary problems.
• Kubernetes review found assumptions that did not match real API and controller behavior.
• Architecture and guidelines review protected the relationship between binding specifications and subordinate implementation artifacts.

Measured compute reinforced that operating model:

Verification was the largest category in this implementation worktree. That is an observed property of this case, not a prescription that every project should reproduce the same ratio.

Review operated as control flow

In many AI-assisted workflows, review happens after a large patch already exists. Here, review verdicts controlled whether the work could advance. A must-fix result routed the finding back into implementation, followed by focused re-review from the relevant specialist.

The observed pattern was:

implementation → must-fix → repair → focused re-review → pass

The process does not prove that every possible defect was found. It shows that recorded blockers were treated as execution state rather than optional commentary.

Six findings that materially changed the result

1. Authorization could fail open. A security review found a path in which a Kubernetes authorization evaluationError could be treated too permissively. The repair made uncertainty fail closed and passed focused re-review.

2. Traceability could report a false PASS. QA found that failed evidence for a requirement could still allow the traceability layer to pass. The gate was repaired so failed evidence remained blocking.

3. Duplicate evidence could mint mutation authority. A QA review found that duplicate prerequisite reports could satisfy the write-permit gate. The permit was tightened to exact prerequisite validation and exact mutation-plan binding.

4. Credential publication could become partial. Reviewers found crash states in which a credential artifact set could be only partly visible, along with an encoding mismatch. Publication was redesigned as a crash-atomic directory transaction with strict parsing and deterministic recovery tests.

5. Network deletion could race. Review found that deletion preconditions were not enforced strongly enough. The flow was changed to bind exact resource identity and state to the permitted mutation.

6. Cleanup could falsely report success. Late in the live-harness review, collision refusal and cleanup accounting were found to be incomplete. The harness was made fail closed, cleanup became exact, and the full local and live acceptance sequence was rerun after the repair.

One of the clearest examples concerned network-policy attribution. A request returning HTTP 503 was not accepted as proof that removing a Cilium exception caused the denial. The final test also verified policy absence, an empty managed-policy inventory, healthy backend and route state, a ready EndpointSlice and Pod, and stable Service identity.

That distinction matters beyond Kubernetes:

An observed failure is not automatically proof of the cause a test hoped to exercise. The evidence must rule out realistic alternatives.

Continuity came from durable state, not one perfect context

The implementation worktree lasted more than forty hours of wall-clock span and recorded 12 context compactions. It also recorded 932 transcript files, 12,927 turns, and 17,211 tool calls. This was too much work to treat one model context as the project’s memory.

Continuity instead came from artifacts outside the active context: accepted specifications, source files, tests, recorded review findings, subagent outputs, repository state, and explicit checkpoints. Those durable surfaces made it possible to resume after compaction or provider failure without treating the project as a lost conversation.

The cache telemetry does not establish an exact monetary cost or savings figure. The supplied evidence does not include pricing, cache-write tokens, or a separately reported reasoning-token category. We therefore do not convert the token record into a dollar claim.

Interpretation: Resumability is supported by the recorded compactions and continued delivery. The telemetry does not prove lossless semantic equivalence across every compaction; it shows that the work continued through durable project state.

What validation established

No single layer was allowed to stand in for the entire system. The project separated accepted source authority, static traceability, local verification, live acceptance, external platform evidence, and human authorization.

A source hash does not prove implementation parity. A passing unit-test suite does not prove every cluster behavior. A selected live phase does not prove every accepted requirement. Each layer answers a different question.

The committed validation record reports:

• all 7 of 7 local-gate commands passed;
• 416 Rust tests passed;
• 0 Rust tests failed;
• 1 Rust test was ignored as an explicit subprocess entry point;
• cargo audit passed;
• the default all-phase reference-cluster run exited with code 0;
• its machine-readable summary reported PASS;
• global preflight reported 7 of 7 checks passed;
• cleanup passed; and
• final exact absence checks passed.

Those live results are repository-reported for this publication because the private generated run directory—commands, raw evidence rows, logs, audit output, and manifests—is not being published here.repository-reported The OPI telemetry separately records the final QA launches that performed the post-mutation local gate and fresh all-phase live rerun.

What this evidence supports

• The implementation worktree was instrumented and review-heavy.
• The committed local-gate result passed in the documented source state.
• The repository reports a passing point-in-time reference-cluster run with exact cleanup.
• Final validation was rerun after the last material harness repairs.

What it does not establish

• That every tracked obligation was live-tested.
• That every deployment environment will behave identically.
• That the system is independently audited or production ready.
• That OPI has a universal speed or cost advantage.

The canonical local gate

Every command had to run and exit successfully. A command that did not run was not counted as a pass.

The live harness supported narrower modes, but only the default all-phase mode could be described as the documented full acceptance run. Preflight-only, selected-phase, core-only, or cleanup-only results were not allowed to inherit broader scope.

What the process delivered

The final merged change contained 10 commits and changed 56 files. Relative to the original base, it recorded 76,649 additions and three deletions. Those figures describe repository scale, not model authorship, semantic complexity, or productivity.

Security-sensitive product behavior

The result included a Rust CLI for sandbox creation, verification, user credentials, access management, and exact network exceptions.

Isolation and authorization controls

The implementation combined Kata runtime selection, strict admission, custom RBAC, short-lived credentials, quotas, storage restrictions, and default-deny Cilium policy.

Evidence and traceability

The repository included requirement mappings, secret-safe evidence, lifecycle provenance, finalization, and a detached seal designed to avoid circular self-attestation.

Validation and handoff

The delivery surface included a local gate, live acceptance harness, deterministic classifier, operator and user guidance, parity policy, and a reusable agent-operable skill.

A deterministic line-based reconstruction model mapped the delivered artifact surface to a large nominal conventional engineering estimate. We do not convert that estimate into “person-days saved” or a speedup. There was no matched human team building the same system under controlled conditions. The measured OPI worktree and the reconstruction model are different measurement systems.

What the evidence does not prove

The strongest case studies disclose their boundaries before the call to action. Four matter most here.

Two accepted artifacts remain unresolved

The repository does not yet contain an exact compliant ClusterRole/sandboxctl-operator manifest or its complete manual installation procedure. The executable high-trust path used a cluster-administrator kubeconfig. Repository-wide specification parity is therefore not claimed.

No independent third-party audit

The project received extensive internal security, QA, code, and platform review. No independent red team, penetration test, certification, or third-party security audit was supplied. Internal adversarial review is valuable; it is not independent assurance.

No universal performance or cost claim

The case establishes measured behavior for these two worktrees. It does not establish a universal speedup, dollar cost, cost saving, or best-in-class result. The cache record is not a billing statement, and the deterministic development-cost estimate is not an observed baseline.

No claim that every obligation passed live testing

The project tracked 343 product-traceability obligations, while a broader accepted-document inventory can be derived as 370 normative obligations. Those are document and traceability inventories—not live-pass counts. The live harness exercised a defined set of cluster-observable scenarios.

Residual platform and product risks remain

The accepted threat model retains limitations involving image supply-chain controls, FQDN policy on shared public IPs, individual certificate revocation, already-open streams, asynchronous Pod deletion, worker image pulls, and Kata, Cilium, admission, CSI, or platform defects. “Secure” and “isolated” are scoped engineering claims, not guarantees.

Six lessons for governed agentic software delivery

1. Govern obligations before scaling execution. Use human authority where the work determines product behavior, accepted security risk, and release boundaries.
2. Give reviewers power over workflow state. A finding that cannot block progression is advice, not a delivery gate.
3. Separate review disciplines. Correctness, QA, security, and platform semantics fail differently. Differential expertise finds non-overlapping problems.
4. Keep proof scoped to the layer that produced it. Compilation, unit tests, live acceptance, traceability, and audit evidence answer different questions.
5. Externalize project memory. Specifications, tests, findings, repository state, and checkpoints must outlive any one model context or provider call.
6. Rerun acceptance after material change. An earlier PASS becomes stale when the harness, product, configuration, or evidence rules change.

The result is not a story about removing people from software delivery. It is a story about moving human judgment to the decisions where it carries the most leverage, then surrounding delegated execution with enough structure that work can be directed, challenged, repaired, and closed with visible evidence.

Methodology and evidence definitions

This article is based on two machine-generated OPI telemetry digests, Git and pull-request history, accepted specifications and threat model, the implementation, a deterministic development-cost report, and the committed validation record. High-stakes claims were constrained to the scope those materials support.

Continue reading

• Beyond PRDs: Building an Adaptive AI Software Factory explains why delivery systems need adaptation, explicit review gates, and durable state—not a rigid requirements conveyor belt.
• AI Agent Harnesses: Why the Same Model Produces Different Results examines how context, tools, recovery, and governance change the economics and quality of model-driven work.
• Explore OPI Forge for Chrysent’s private-beta product for governed agentic software delivery.

work-share: Automated work share sums concurrent agent launch durations independently and combines them with an interval-based estimate of eligible human work. It is not a wall-clock percentage of time without human involvement. launches: One launch is one bounded subagent execution. The 466 figure must not be described as 466 unique agents, people, or models. repository-reported: The committed validation record and OPI QA launch summaries report the live result. The private live-run directory was not attached for independent inspection in the publication evidence package.