Engineering case study · July–November 2026

Engineering a Governed AI Development System

A case study in AI-assisted software engineering, system reliability, and technical governance.

An applied systems-engineering case study in AI-assisted software development, automation, verification, and controlled autonomy.

Human-directed, AI-assisted engineering. This is not evidence of software-security credentials, industrial-controls safety competence, or unaided coding.

Executive summary

The problem. I was building software with Claude Code agents across several personal repositories: this portfolio site, application prototypes, and a local-model research lab. As the agents gained tools and permissions, the records show capabilities accumulating unchecked, agents reporting success they had not earned, and permissions widening one approval at a time.

The approach. Over four months I designed and directed a governed development environment around Claude Code, built on four rules:

01A capability is trialled before it is exposed, and its exposure follows measured use.
02The agent that builds a change never certifies it.
03Rules are enforced by the runtime wherever it supports them, and enforced rules are canary-tested where possible.
04Ambitious ideas are tested as hypotheses, with gates that can fail.

The outcome. By November 2026, five external-communication and delegation tools were denied and other connected-service writes required approval, both observed in fresh sessions. Skill exposure was cut by usage. Every remediation gate separated builder, verifier and authorizer. A repaired test pipeline passed 2,574 of 2,574 tests. A month of experiments rejected moving routine work to local models, apart from one narrow lane. Controls that remain untested, or exist only as written rules, are named on this page.

My part. Claude Code agents wrote nearly all the code and configuration. I set the direction, made the architectural and evaluation decisions, approved changes (each commit, push and merge individually in the recorded October–November gates), challenged conclusions and operated the system.

Problem and requirements

How the requirements changed

The records do not say why the project began. They do show what was being controlled at each stage.

Original requirements (July): written standing rules for agent actions, staged approvals, and a lab for vetting third-party skills before use.

August–September: A goal to “reduce future dependence on Claude Code” led to a local-model lab, where a read-only claims auditor checked reported results. Agents’ commits there later recorded correcting “three false claims” and admitted “I published a number I could not produce”. In late August, a written rule established that readiness to commit is never authorization to commit.
October: An audit found more than 150 skills competing for a listing budget of about 1% of the context window; only 3 of about 60 skills in one suite were in use (an agent’s measurement). Exposure had to follow usage. Continuous integration began running the repository’s four checks.
November: Audit findings were ranked “by actual security exposure and operational impact, rather than number of findings closed”, then addressed through three gated changes.

Architecture

What the runtime enforces, and what it only tells the model

Solid heavy borders are runtime-enforced; dashed borders are instructions or advice; dotted borders are separate boundaries outside the verified scope. On narrow screens the diagram scrolls sideways.

A change moves through the system like this: my brief; a builder session working on a branch or worktree; a fresh-context, read-only verifier; my separate authorization of the commit, the push and the merge; then CI and deployment.

Claude Code evaluates supported tool invocations against its permission rules. Hooks and downstream execution by external tools have separate execution boundaries and must be evaluated independently.

Runtime permissions do not provide complete isolation of delegated processes or downstream integrations. Those boundaries remain outside the verified scope of this project.

Enforced or instructed

Permission deny rule: Yes; A denied tool is removed from the agent’s tool list.
Permission ask rule: Yes; The call needs approval; unattended runs refuse it.
Disabled permission-bypass mode: Yes; A setting. That it caused the observed fallback to normal mode is inferred.
CLAUDE.md rules, including per-instance git authorization: No; Instructions the model reads. Followed in the recorded gates, not technically locked.
Read-only verifier agents: Partly; The tool allowlist is enforced. Their independence is of context, not of model.
Hooks: Separate; The runtime runs them on its own events; they are not evaluated as tool calls.
Downstream execution by external tools: Separate; Permission rules gate the call into a tool, not what that tool or its server does next. Not verified here.
Git worktrees: No; They separate working copies, not processes or permissions. A worktree is not a security sandbox.
CI on merges to main: Not verified; CI runs on merges. Whether it is a required check, or a prerequisite for deployment, was not verified.

Four decisions

The architectural decisions that shaped it

Decision 01

Select capabilities by evidence, and size exposure to use

Problem: Plugins and skills can arrive with broad defaults, including hooks that run without the model choosing to. Installing broadly had already exhausted the skill-listing budget.
Alternatives: Install broadly and let the model choose.
Remove large suites entirely.
Approach: Trial a capability, then adopt it, scope it down, freeze it or retire it, and set its exposure from observed use: listed for the model, user-invocable only, or off. This practice emerged as the environment matured and was formalised by the October audit; it does not describe every earlier adoption.
Evidence: July: a design tool was audited and pinned with hash manifests, and a security scanner was tested against unsafe fixtures.
October: an audit removed 6 plugins and 2 marketplaces, applied 53 skill overrides, and scoped one suite to its 3 used skills.
Of 23 external repositories reviewed, two “Adopt” verdicts became none once agents read the code. One candidate was trialled under my typed scope (“project scope, per-edit hook neutralized, one audit”).
Tradeoff: Trials take time.
Limitations: Superpowers (July) and Impeccable (August) were adopted with no recorded security review.
The grades and usage counts were produced by agents.

Decision 02

Separate builder, verifier and human authorizer

Problem: When the same agent built and checked its work, it reported results it could not reproduce.
Alternatives: Builder self-review.
A proposed bounded-autonomy exception for agents (drafted in October, never merged).
Approach: The builder implements. A fresh-context, read-only verifier checks. I authorize.
Evidence: July: a written rule, “Builder self-review is not independent review”.
August: a read-only claims auditor was validated against a fixture, then used to recompute lab results.
October: in a QA pilot, a report-only session found defects, a separate builder fixed three, and a separate reviewer found no regression across 21 routes at 11 widths.
November: every remediation gate was independently reviewed.
Tradeoff: More sessions per change.
Limitations: Every reviewer was a Claude session; no human engineer other than me reviewed the work.
The November closeout pass was the builder’s.
The pilot’s production check did not record which build was deployed.

Decision 03

Put authority in runtime controls, not only in model instructions

Problem: Written rules can be misread, and permission rules can be inert without anyone noticing.
Alternatives: Rely on instructions alone.
Approach: Enforce rules through permission settings where Claude Code supports them, and canary-test them where possible. Where the runtime cannot enforce a rule, keep the rule narrow. From late August, a written rule required a one-time authorization for each commit, push and merge: “READY_TO_COMMIT is readiness, never authorization.”
Evidence: August: on that Claude Code version, a canary showed that a deny rule for one file-writing tool did not fire while the rule for the editing tool did. The rules were switched to the form that fires.
November: user-level rules denied five external-communication and delegation tools, required approval for 28 other write tools, and disabled bypass mode. Fresh sessions showed the five tools gone (94 → 89 visible), reads working, and gated writes refused when unattended.
Tradeoff: More prompts; unattended runs cannot make gated writes.
Limitations: The approval result was not isolated against a control, because both control runs hung.
One group of approval rules is untested at runtime.
Coverage was checked only against the tools connected at the time; plugin-provided servers are not covered by these rules.
The git rule was followed in every recorded October–November gate; earlier work ran partly under standing approvals.
Runtime permissions do not isolate delegated processes or downstream integrations, and agent processes are not OS-sandboxed.

Decision 04

Test local-model delegation as a gated hypothesis

Problem: A local-model lab set out to move routine engineering onto local models, keeping Claude as teacher, reviewer and escalation path.
Alternatives: Adopt on promising headline numbers.
Approach: A 42-task corpus, preregistered A/B tests with baselines, qualification gates, kernel-level checks for zero network egress, and read-only auditors recomputing reported facts.
Evidence: One model was rejected, scoring 90.0% against 96.7%.
A private, read-only lane qualified, with zero remote egress.
The second milestone first failed (8/26). It later scored 14/26 against a 9/26 keyword baseline, but all 64 responses chose the same candidate, so the model’s contribution was not measurable. The baseline and the response pattern are agent-reported in lab documents.
Auditors rejected the third milestone’s architecture, and a bounded follow-on failed 5 of 5 validations.
Tradeoff: A month of work produced mostly negative results. The lab’s 697 commits measure activity, not productivity.
Limitations: Claude remained the builder; local models are qualified only for the narrow read-only lane.
Prompts are missing for most lab sessions, and the result narratives were written by agents.

Failure investigations

Three failures that exposed architectural weaknesses

A passing security grade from zero files

Observed: In July a security scanner graded deliberately unsafe synthetic fixtures C/62, C/62 and D/50, but gave a skill directory A/100 after scanning 0 files.
Verified: The scanner’s own report records that grade alongside zero files scanned.
Unresolved: Why nothing was scanned. The scanner was never adopted, and no decision record says why.
Corrective action: None recorded for the scanner itself. Separately, later practice confirms that a control actually ran, with canaries and fresh-session tool listings, before its result is trusted.
Verification: The raw report was re-read during the evidence reconstruction.
Lesson: A check that inspects nothing is a failure, not a pass.

Local-model lab: concurrency and incomplete containment

Observed: In late August a second writer was acting on the lab during measured experiments. A quarantine marked the spending path neutralized, yet the quarantined line still made three warm-up inference calls and spent one experiment run, which aborted at 35 of 64.
Verified: The records show an unidentified second writer, and that the quarantine did not stop all execution (medium confidence: seen in the records, not re-checked against every primary source).
Probable contributing factors (inferred): The authority to spend runs rested on written instructions and session discipline; an exclusive lock was introduced only as part of the response.
Unresolved: Who the second writer was, how the run was spent, and why many lab transcripts are missing.
Corrective action: I chose read-only analysis, stood the competing session down, halted work before re-authorizing it by role, and shut down the local runtime by hand. A fresh clone and an exclusive lock were decided on, though the recovery itself ran in a worktree.
Verification: Partial. The quarantine was never formally reversed, and the lab closed in mid-September with the incident unexplained.
Lesson: Single-writer discipline needs a mechanism, not a sentence.

Tests discovered inside nested Git worktrees

Observed: The root test run collected 516 test files, 435 of them from Git worktrees stored inside the repository.
Verified: Vitest’s default discovery included the worktrees placed under .claude/worktrees/, and setting its exclude option replaces the defaults rather than extending them.
Unresolved: None material to the fix.
Corrective action: exclude: [...configDefaults.exclude, ".claude/worktrees/**"], built by a builder session, reviewed by a fresh-context verifier, and merged after separate authorizations.
Verification: Discovery fell to 81 files, 2,574 of 2,574 tests passed, and CI and deployment succeeded. No change in production behaviour is claimed.
Lesson: Worktrees isolate working copies, not tools: anything that walks the repository sees them.

A smaller example: a third-party skill in the personal scope silently shadowed this repository’s own /retro skill, because personal skills take precedence. The audit proposed renaming the project skill; I overruled it and disabled the third-party copy, and both repositories were then confirmed to load the project skill.

Outcomes

What changed, with its qualifications

External repositories evaluated
23 reviewed; none adopted outright; 1 trialled and kept at local scope
Medium confidence: agent verdicts, reconstructed from a session record.
Skill exposure
53 overrides; 6 plugins and 2 marketplaces removed
High for the configuration; the usage counts behind it were measured by agents.
Test discovery
516 → 81 files
High: observed.
Test suite after the fix
2,574 of 2,574 passing
High. A test count is not a measure of coverage.
Remediation
3 gates, each separately authorized, independently reviewed, backed up, with a rollback path
The closeout pass was the builder’s; three audit findings were dropped without a decision.
Local-model delegation
1 narrow read-only lane qualified; broader replacement rejected
High for the results recorded in commits. The migration did not succeed.

Commit messages show the test suite growing from 144 tests in July to 2,574 in November. That is size, not coverage, and the lab’s commit count is activity, not productivity.

Lessons

What changed in how I work

01Test the control, not the configuration: an inert deny rule and a zero-file “A” would both pass a desk review.
02Separate implementation from verification.
03Treat negative results as valid outcomes.
04Make single-writer discipline mechanical, with a lock checked before any spend or commit.
05Preserve decision provenance: many decisions survive only in agent-written commits or session records that age out.
06Record the deployed commit with every production check.
07Give every audit finding an explicit disposition; three were dropped silently.

Contribution and attribution

Who did what

Me: Direction, requirements and scope; architectural and evaluation decisions; approval of changes, individually for each commit, push and merge in the recorded October–November gates; challenging conclusions, such as the /retro overrule; and operating the system, including a manual runtime shutdown and running an auditor’s tests myself.
Claude Code: Nearly all implementation and configuration, automated test execution, commit messages and audit reports, and every AI-only review.
ChatGPT: Planning, selected briefs, technical critique and editorial support. One locally recorded brief says it was drafted in ChatGPT; the rest of this role is as I report it.

“Independent review” on this page means a separate Claude session with a fresh context and read-only tools. No human engineer other than me reviewed the work.

Not established by this work

That local models can replace Claude.
Any security certification, or the absence of vulnerabilities.
Isolation of delegated processes or downstream integrations.
CI as a required check or deployment prerequisite.
Independent human review.

Evidence

What you can inspect, and what is held privately

What you can inspect directly is this page: the sanitized summaries below, and the architecture diagram above. The repositories behind it are private, and the lab repository has no public remote, so the identifiers below are listed for traceability only and cannot be opened from here.

Test-discovery fix
Before: 516 test files collected, 435 from nested worktrees. After: 81 files; 2,574 of 2,574 tests passing; CI and deployment succeeded on the merged commit.
Identifier:
Merge 8cef863 (PR #39)
Held in:
Private repository
Limit:
No production-behaviour change is claimed.
Continuous integration
Test, lint, typecheck and build run on merges to main since October 2026.
Identifier:
Commits eb17ba7 → dcee6a9
Held in:
Private repository
Limit:
Whether CI is a required check or a deployment prerequisite was not verified.
Per-instance git authorization
“READY_TO_COMMIT is readiness, never authorization.”
Identifier:
Commit 48293ee
Held in:
Private repository
Limit:
A written rule, not a technical lock.
Local-model qualification
One model rejected (90.0% vs 96.7%); one read-only lane qualified; second milestone negative at 8/26, later 14/26 against a 9/26 baseline with the model’s contribution not measurable; third milestone’s architecture rejected; follow-on 5 of 5 validations failed.
Identifier:
Commits e63c3f7c, 42a6b0a0, 81037865, 24dda17d, af88abb3
Held in:
Local lab repository, no remote
Limit:
Result narratives were written by agents.
Deny-rule canary
A deny rule for one file-writing tool did not fire; the rule for the editing tool did; rules were switched.
Identifier:
Commit 5e37be32
Held in:
Local lab repository, no remote
Limit:
Specific to that Claude Code version; not re-tested.
Permission controls
Five tools denied, 28 approval-required, bypass mode disabled; 94 → 89 visible tools in fresh sessions.
Identifier:
Settings backup and gate session record
Held in:
Private records, not published
Limit:
Approval result not isolated against a control; one group of approval rules untested.

Session transcripts, configuration files and tool reports are retained privately for traceability and are not published.

← The engineering lab See it at the Garage software desk →