The other half of Greentic

▸ Start here

Skip the parts you already know. The sections worth your time in order are: Where our lanes collide (new information, and the thing most likely to cost you a week if you hit it unprepared), Operating rhythms (you have never run the develop/main release lanes), then the subsystem sections as reference when you touch them.

Two things that are true of this lane and not of yours

⚡ Where our lanes collide

All figures re-derived 2026-07-31 · re-run before trusting them

This is the section that does not exist anywhere else, and the one worth reading closely. The two lanes have never been integrated, and several places where they touch are load-bearing but unguarded.

The divergence, measured

Two cherry-equivalent commits across seven repos. Nothing has ever been ported between the lanes. Where you find byte-identical files — the deployer's bundle-upload module, for instance — that is pre-fork ancestry, not evidence that porting works. Do not cite it as one.

Three seams that are load-bearing and unguarded

What this means practically

• Do not assume a fix ports. With ~0 cherry-equivalence, every cross-lane change is a fresh implementation, not a cherry-pick. Budget accordingly.
• Deploy-spec first. Any convergence plan that does not start by resolving the three-way fork will stall on it later, at higher cost.
• The seams fail quietly. A floating branch pin and a presence-only binary check both produce confusing failures far from their cause. Both are cheap to harden and worth doing before any integration work.

§ Running this lane

Verified on 2026-07-31 · toolchain 1.95.0 · targets host + wasm32-wasip2

Your workspace already works, so this covers only what is different about operating the develop/main lane. Every command below was executed before being written down.

1 · Switch the fleet onto the lane

./update-repos.sh --clone        # ensure every repo is present and fetched
./switch-to-develop.sh --pull    # move all repos to develop and fast-forward
./switch-to-default.sh           # …and back again

# Dirty trees are skipped by default; --stash if you want them moved anyway.
./switch-to-develop.sh --stash --dry-run

2 · Toolchain — one rule that will surprise you

./sync-toolchain.sh --check      # fails if any repo has drifted

3 · The product toolchain — and why it may downgrade you

cargo binstall gtc
gtc install       # resolves the pinned toolchain manifest and installs exact versions
gtc doctor        # verifies the install

4 · Verify you match the reference machine

These drift. Re-derive rather than trusting the table — <binary> --version on your own box, and compare against crates.io.

5 · Build and test a repo

# The full local gate — fmt, clippy, tests, and repo-specific checks.
bash ci/local_check.sh

# Pack building, from a pack workspace:
greentic-pack build

Useful workspace scripts

§ What you're absorbing

Built 2026-03-23 → 2026-07-31 · Lane develop / main · Scale 109 repos, 1,688 merged PRs of context

Nine subsystems, listed below in the order they are documented. You will not need most of this on day one — it is reference material for when you touch a given area.

How the work arrived — and why that shape matters to you

The monthly profile below tells you where the codebase is mature and where it is young. April was the CI and toolchain unification, so that substrate is settled and you should not need to touch it. July was the air-gap update train and webchat routing — the newest code, and the least soaked. If something breaks in your first month, that is where to look first.

The repos you will actually live in

Of 109 repos touched, ten carry most of the weight. Expect to spend the majority of your time in the first two.

Set your expectations about the work mix

The commit mix is 1,319 chore · 699 ci · 468 feat · 465 fix · 95 docs · 39 test — roughly two-thirds infrastructure and maintenance rather than features. That is the honest shape of running a 141-repo fleet, and it will be the shape of your weeks too. Feature work here is only affordable because the substrate underneath it is mechanised; budget accordingly, and be suspicious of any plan that assumes otherwise.

§ The stack, from this lane's side

You know this pipeline — it is here for one reason: the layer boundaries are also the ownership boundaries. Your lane lives mostly at the Component and Flow end, authoring what runs. This lane owns the right-hand side: how packs become bundles, how bundles reach an environment, and how the operator enforces what a component declared. The seams in Where our lanes collide all sit at the Pack/Bundle join.

§ How the fleet is organised

Four artifacts govern the whole fleet. Learn these and most of the machinery becomes legible.

REPO_MANIFEST

.github/toolchain/REPO_MANIFEST.toml — the registry. 85 entries carrying toolchain variant, ordered publish lists, version ranges, pipeline flags, binary classification and cached tiers. Every pipeline script reads it. Note it is not the full inventory — ~59 repos on disk are not onboarded.

dep-graph/

The derived dependency graph. build_graph.py scans every Cargo.toml on disk (705 crates), derives 295 Rust edges plus 52 hand-declared architectural edges, and computes tiers by topological longest path. Tiers are derived, never hand-edited — the tier = N in the manifest is a cached copy. To change a tier, change the dependency and regenerate.

The three lanes

main = stable (1.1.x) · develop = dev (1.2.0-dev.N) · research = experimental (1.3.0-research.N, not yours). Each has its own publish cadence and registry behaviour.

CLAUDE.md / AGENTS.md

The workspace guide at the root, kept current and consulted constantly. AGENTS.md is a symlink to it, so editing one updates both.

Tier structure

Tier 0 has no internal dependencies and publishes first; each tier above waits for the one below. Current range is 0–11 with zero cycles. The shape is worth internalising because it predicts your blast radius: a change to greentic-types (tier 1, 60 consumers) cascades through most of the fleet, while a change to greentic-designer (tier 10) affects nothing downstream.

§ Operating rhythms

This is the part that is hardest to reconstruct from the code, so it is written out explicitly.

The three release lanes

How a release actually happens

The signal is the version change in Cargo.toml — there are no release labels or out-of-band metadata. Merging a version bump to main triggers tag creation, which fans out to binary builds and the registry publish. Keep this property; it is what makes Cargo.toml version, git tag and artifact version permanently agree.

Binary bifurcation — read this before publishing anything

On the develop lane, binary-shipping crates publish under a sibling <crate>-dev name so that cargo binstall resolves dev builds without polluting the stable namespace. Libraries keep their canonical name. Crates that are both a library and a binary publish the library under the canonical name as a pre-release and the binary as the -dev sibling. Three manifest fields drive this; get them wrong and the publish either fails or silently lands in the wrong namespace.

Forward-porting main → develop

# Clean merges push straight to develop; conflicts open a PR per repo.
INPUT_DRY_RUN=true bash .github/scripts/forward-port.sh   # preview first
bash .github/scripts/forward-port.sh

What to check, and when

§ Conventions in this lane

You have your own working practices and this is not an attempt to replace them. But these six rules are load-bearing here, and each was written after a specific incident in this lane — dropping one tends to reproduce the failure that created it. They also explain several architectural choices that otherwise look over-built.

Evidence first

No status claim — "done", "fixed", "passing" — without a fresh verification command, its full output read, and the exit code checked. Green CI is not evidence that a gate ran; the run log has to contain the gate's own output.

Design gate

For non-trivial work, a written plan reviewed before code. The plans/ tree holds ~45 completed design documents; the gate is applied over the spec, not after implementation.

Mutation-prove

Tests written by an agent are not trusted until deliberately mutated code makes them fail — re-run personally, not on the agent's report. A uniform mutation that breaks everything proves nothing; a name-filtered harness that matches no tests exits 0 and looks green.

Fail closed

An absent "no" is never a "yes". Missing credential, unreadable secret, unknown pack kind, declined webchat path — each is an error, never a silent fallback. This rule was written after finding the same class of silent-failure bug ten times over.

Surgical scope

Every changed line traces to the request. Adjacent code is not "improved"; orphaned imports from one's own change are removed, pre-existing dead code is reported but left alone.

Re-derive

Documented counts and versions carry the command that regenerates them. The workspace guide instructs readers to re-derive rather than trust the prose — because the prose goes stale daily and a stale number is worse than no number.

01 Fleet CI/CD & GitHub infrastructure

Peak April 2026 · ~714 CI PRs · 85+ repos, 2 orgs · 39 reusable workflows

The starting condition was 85 repositories with hand-copied CI, six different Rust toolchain versions, no rustfmt.toml anywhere, 52 repos still on master, and 85 of 224 workflow files with no permissions: block. The fix was to stop maintaining CI per repo and start maintaining it once.

What was delivered

• A reusable-workflow library — 39 workflow_call workflows in the central greenticai/.github repo. Per-repo CI became a 5–15 line caller. pr-ci.yml is the unified PR baseline (fmt + clippy + test + build), variant-dispatched for host vs wasm.
• Fleet rollout waves, each a single-day mechanical PR blast: dependency review (57 repos), concurrency groups (36), toolchain pin to 1.94.0 (50), Codex fork-PR guard (38), scheduled-failure Slack notifier (76), branch invariants (36), nightly cron consolidation into a single 03:XX UTC window (42), version-tag/publish sync (32).
• Toolchain standardisation — two canonical rust-toolchain.toml variants in .github/toolchain/, distributed by sync-toolchain.sh with --dry-run / --direct / --check modes. Six fragmented versions consolidated to 1.94.0, then 1.95.0.
• Default-branch standardisation — master → main across 52 repos, preceded by an audit of 143 open PRs (130 auto-retargeted, 2 closed, 3 with hardcoded refs fixed).
• Security hardening — explicit permissions: blocks in three batches; CodeQL from zero repos to fleet-wide; fork-PR guards on the Codex remediation workflows that hold secrets; branch-protection rulesets on every repo eligible for them.
• Onboarding automation — onboard-repo.sh applies the entire convention set to a new repo in one command (toolchain, rustfmt, CI callers, dependency review, branch invariants, Slack notifier, MSRV, manifest registration), with a remote mode that needs no local checkout.
• GitHub App migration — PATs replaced by the greentic-ci App across both orgs, with a per-org token-minting helper.
• Coverage policy gate — nightly cargo-llvm-cov + nextest driven by a per-repo coverage-policy.json, with distinct exit codes for build failure (4), policy violation (5), bad configuration (2) and cache poisoning (127).

Decisions

Code

main-pre-release-guard:
  if: github.event_name == 'pull_request' && github.base_ref == 'main'
  runs-on: ${{ inputs.runner }}
  steps:
    - uses: actions/checkout@v6
      with:
        submodules: ${{ inputs.checkout-submodules }}
    - name: Reject pre-release versions on main
      shell: bash
      run: |
        set -euo pipefail
        offenders=$(
          find . -name Cargo.toml -not -path './target/*' -print0 \
            | xargs -0 grep -HnE '^version = "[0-9]+\.[0-9]+\.[0-9]+-(dev|alpha|beta|rc)\.' \
            || true
        )
        if [ -n "$offenders" ]; then
          echo "::error title=Pre-release on main::..."
          exit 1
        fi

#!/usr/bin/env bash
set -euo pipefail
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$ROOT"

echo "==> cargo fmt --check"
cargo fmt --all -- --check
echo "==> cargo clippy"
cargo clippy --all-targets --all-features -- -D warnings
echo "==> fixture tool prerequisites"
ci/ensure_fixture_tools.sh
echo "==> replay deployer scaffolds"
cargo run --features internal-tools --bin replay_deployer_scaffolds
echo "==> cargo test"
cargo test --all
echo "==> cargo build --no-default-features (baseline)"
cargo build --no-default-features

name: Assert Branch Invariants

# Why this exists (Phase D of plans/binary-bifurcation.md):
#   pr-ci.yml's main-pre-release-guard catches `0.x.y-dev.NN` versions at PR
#   time, but only via the PR-level event. Direct pushes to `main` (possible
#   on private repos under the GitHub free plan, where ruleset enforcement is
#   not available) bypass it.
#   This workflow runs on `push: [main]` AND `pull_request: [main]` so the
#   guard fires on both paths.
#
# Lane behavior:
#   main:    FAIL on `*-dev.*` version OR `<base>-dev` package name
#   develop: WARN if a manifest-declared binary basename is missing

Gotchas found the hard way

• Inside a reusable workflow, github.workflow resolves to the caller's name — concurrency groups built from it collide whenever a release workflow calls ci.yml. Fix: hardcode a literal prefix.
• App installation tokens are per-org, not per-App. A greenticai token hitting a private greentic-biz repo gets a privacy-preserving 404, not a 403 — so it reads as "repo doesn't exist".
• A PR's status-check rollup cannot contain a push-only job. Nightly lock PRs auto-merged "green" on a gate that had never run; the real gate fired for the first time after merge.
• A reusable workflow never appears in gh run list under its own name — its jobs nest inside the calling run, so it can produce artifacts that look impossible if you only check run lists.
• Branch protection without required status checks lets red PRs merge. greentic-types had enforce_admins: true and no status checks; a PR merged with clippy failing and halted the nightly tier train.
• Widening a CI trigger can expose a structurally-broken job. Adding develop to a pull_request filter made semver-checks run for the first time on a pre-release lane where it can never pass — there is no stable baseline for v1.2.*.
• Runner disk exhaustion kills large builds silently — it surfaces as a null conclusion with an unretrievable log. Fixed centrally with a best-effort free-disk step reclaiming 12–15 GB.
• Private repos on the Free plan receive no org secrets, even at visibility=all. Tag-on-version-bump then "succeeds" while publishing nothing, because the App-token mint fails and the fallback GITHUB_TOKEN cannot trigger workflows.

Status

done 39 reusable workflows · App migration · branch rename (52 repos) · CodeQL fleet-wide · auto-merge and branch cleanup on all 67 dev-publish repos · Renovate retired.

open greentic-runner CI is fmt-only — a cross-org private dependency blocks the shared reusable. greentic-types has no post-merge build at all (pull_request + workflow_call only), which let a corrupted lockfile survive three days undetected. The local_check.sh-vs-pr-ci.yml blind spot has not been surveyed fleet-wide. macOS runner spend is ~$645/month against a ~$50–110 target.

02 Release pipelines, versioning & the dependency graph

3 publish lanes · 85 manifest repos · tiers 0–11 · 347 graph edges · 705 crates

With no root workspace, publishing order is not something Cargo can work out. It has to be derived. This category is the machinery that turns 141 loose repositories into something that can be released as a coherent product.

What was delivered

• REPO_MANIFEST.toml — the fleet registry: 85 entries carrying toolchain variant, ordered publish lists, version-track ranges, pipeline flags (dev-publish-enabled on 67), binary classification, and cached tiers. Every pipeline script reads it.
• The dependency graph — dep-graph/build_graph.py scans every Cargo.toml on disk (705 crates, 141 repos), derives 295 Rust edges plus 52 hand-declared architectural edges, and computes tiers by topological longest path. Tarjan SCC detects cycles. sync_manifest_tiers.py --check is the CI gate against drift. Deliberately does not read the manifest — on-disk source is the only authority.
• Nightly develop lane — cron at 02:00 UTC, dispatching per-repo publishes in tier order 0→10 through a three-stage chain (prepare → release-binaries across a six-target matrix → publish). A lower_tier_published flag forces downstream rebuilds even without source changes. Post-tier, a lock-sync job runs scoped cargo update across every develop repo and opens bot PRs.
• Binary bifurcation — 20 binary-shipping crates across 17 repos publish under a sibling <crate>-dev name on the develop lane, so cargo binstall resolves with no flags while the stable namespace stays clean. Dual-role crates (library + binary under one package name) publish the library as a pre-release under the canonical name and the binary as the -dev sibling.
• Fleet version trains — a force-jump that unified a fragmented dev lane (repos at 0.2.x, 0.5.x, 0.6.x) onto a single base across 39 PRs; the 1.1 GA content promotion across 66 repos; the 1.2 pre-release lane at 59/59 repos.
• Forward-porting — five main→develop waves between July 6th and 30th, each opening PRs across 30–50 repos, with the failure modes documented as they were discovered.
• Workspace scripts — eight at the root plus ten in .github/scripts/, several of which ship their own unit tests (test_bump_cargo_versions.py, test_rewrite_binary_name.py, test_assert_branch_invariants.py).

Decisions

Code

Tier formula:
    tier(repo) = 0                                if repo has no internal deps
    tier(repo) = 1 + max(tier(producer))          otherwise

Strongly-connected components (cycles) are condensed into a single super-node
before tier computation; all SCC members share the same tier. Non-trivial
SCCs are reported separately.

#   binary-crates            — Subset of `publishes` that ship a binary. On develop, each
#                              publishes as `<name>-dev` (Phase C of binary-bifurcation).
#   dual-role-binary-crates  — Subset of `binary-crates` that ALSO publish a library under
#                              the same `[package].name`. Library keeps publishing as `<name>`;
#                              binary gets a sibling `<name>-dev` copy-publish.
#   binary-bins              — Map of `package -> bin name` overrides for binary-crates whose
#                              `[[bin]].name` differs from the `[package].name`.

[[edges]]
from = "greentic"
to   = "greentic-operator"
kind = "runtime-delegation"
note = "gtc CLI dispatches to greentic-operator subprocess"

[[edges]]
from = "greentic"
to   = "greentic-bundle"
kind = "runtime-delegation"
note = "gtc CLI dispatches to greentic-bundle subprocess"

Gotchas found the hard way

• A path dependency hides the version floor from every CI gate. { path = "../crate", version = ">=0.2" } builds against the in-tree copy; the floor is only consulted when something resolves from the registry. Three releases shipped uncompilable this way.
• The -dev namespace is shared between develop and research. Research 1.3 outranks develop 1.2 by semver, so <crate>-dev can serve a research binary to a develop consumer.
• Squash-merging a forward-port destroys ancestry. The comparison keeps showing main ahead and the next wave re-opens the same conflicts. Forward-ports must merge, never squash or rebase.
• The stable-release script bumps Cargo.toml but not Cargo.lock — so every binary cut needs an explicit workspace update before merge, or the --locked binary build fails.
• An exact pin in a workspace member breaks dev-publish. The publish job stamps every member to a run-id version, which the pin cannot match — and PR CI cannot see it, because PR CI resolves through the path dependency.
• Yanked crates stay usable under --offline but fail a fresh resolve. A test that scaffolds a lockless project fails while the main build stays green.
• dev-publish reports SKIPPED, not failed, when the binary stage fails. A green nightly summary can hide the fact that nothing was built — assert the crate actually reached the registry.
• A forward-port wave is not just the conflicting PRs. One wave had 17 conflict PRs resolved while 15 clean PRs sat open, including tier-0 foundations.
• Cron schedules run on the default branch only. A nightly fix on develop must be back-ported to main or it never takes effect.
• $(cargo …) inside command substitution hangs — sccache holds the pipe open, and the process sits at zero CPU in anon_pipe_read. Redirect to a file instead.

Status

healthy Nightly develop lane, mostly green since 2026-07-10, 67 repos enrolled. Graph clean: 0 cycles, max tier 11. Develop lane at 59/59 repos on the 1.2 pre-release base. Bifurcation complete across 20 crates.

dead The weekly stable lane has not run since 2026-05-04, and that run failed — last success was 2026-04-16. Stable releases since then have been cut by hand. A revive-or-retire decision is owed. Consequently main only advances by manual forward-port, and roughly a dozen main-only commits across five repos remain unported.

03 Core runtime & foundation crates

~1,718 commits across 30 repos · stable ABI greentic:component@0.6.0 · wasmtime 43→45

These are the crates everything else depends on — greentic-types alone has 60 consumers. A change here cascades through the entire fleet, which is why most of the decisions below are about making change cheap rather than about features.

What was delivered

• The stable component ABI — greentic:component@0.6.0, with two mandatory exports: describe() returning a self-describing manifest, and invoke() taking CBOR payloads. A lighter @1.0.0 world coexists for runners that need schema introspection without the full contract. CI guards in three repos hard-fail if any downstream repo redefines the canonical WIT package.
• CBOR canonical serialization — a serialize-then-canonicalize pipeline that sorts map keys, rejects floats, rejects tags, and rejects non-text keys, so that identical data always produces identical bytes. That property is what makes Blake3 content-addressing of pack manifests and idempotency keys possible at all.
• Capability sandboxing — a five-crate capability system with validated cap:// URIs, plus network allow-lists and a .gmap policy file format the operator evaluates deny-by-default.
• Multi-tenancy plumbing — TenantCtx threaded through every invocation, telemetry span and session key, and a deterministic session-key format so a conversation resumes identically regardless of which runtime instance handles the next turn.
• The flow model — YAML authored, parsed to a document, lowered to an intermediate representation of nodes and routes, serialized to CBOR for the runtime.
• The pack format — a ZIP carrying a CBOR manifest, an SBOM, flows, WASM binaries and assets, Blake3-hashed and Ed25519-signed with an embedded certificate chain.
• Runner work — revision-keyed runtime so multiple revisions coexist, runtime config injection, a resolver for runtime:// URIs, and a full telemetry attribution chain from deployment rollout ids through to OpenTelemetry spans.
• Fleet migrations — the wasmtime 43→45 migration across every consumer, and four full version trains (0.4→0.5, the 1.1 unification, 1.2-dev, 1.3-research).

Decisions

Code

world component {
  import control;
  export node;
}

/// Context that accompanies every invocation across Greentic runtimes.
#[derive(Clone, Debug, PartialEq, Eq, Hash)]
pub struct TenantCtx {
    pub env: EnvId,
    pub tenant: TenantId,
    pub team: Option<TeamId>,
    pub user: Option<UserId>,
    pub session_id: Option<String>,
    pub flow_id: Option<String>,
    pub node_id: Option<String>,
    pub provider_id: Option<String>,
    pub trace_id: Option<String>,
    pub attributes: BTreeMap<String, String>,
    pub deadline: Option<InvocationDeadline>,
    pub attempt: u32,
    pub idempotency_key: Option<String>,
    pub impersonation: Option<Impersonation>,
    // … further identity and tracing fields
}

/// Encode a value as canonical CBOR bytes.
pub fn to_canonical_cbor<T: Serialize>(value: &T) -> Result<Vec<u8>> {
    let mut interim = Vec::new();
    into_writer(value, &mut interim)
        .map_err(|err| CanonicalError::Encode(err.to_string()))?;
    canonicalize(&interim)
}

pub fn canonical_session_key(
    tenant: impl AsRef<str>,
    provider: impl AsRef<str>,
    anchor: Option<&str>,
    user: Option<&str>,
) -> SessionKey {
    SessionKey::new(format!(
        "{}:{}:{}:{}",
        tenant.as_ref(), provider.as_ref(),
        anchor.unwrap_or(DEFAULT_CANONICAL_ANCHOR),
        user.unwrap_or(DEFAULT_CANONICAL_USER)
    ))
}

PATTERN='^package\s+greentic:component@'
MATCHES="$(rg -n --hidden --glob '!.git/*' --glob '*.wit' \
  --glob '!**/target/**' --glob '!**/out/**' \
  --glob '!crates/vendor/**' --glob '!**/wit-staging/**' \
  "$PATTERN" . || true)"
if [[ -n "$MATCHES" ]]; then
  echo "ERROR: greentic-pack must not define canonical greentic:component WIT."
  exit 1
fi

Gotchas found the hard way

• A dependency update can silently break canonical hashing. A transitive bump enables serde_json's preserve-order feature, switching maps from sorted to insertion order — anything hashing a stringified value quietly changes its answer.
• Extracting an AsyncFnMut helper breaks Send inference downstream. The error surfaces at the tokio::spawn site in another crate, not at the helper. A refactor that looked like a clean simplification broke three host APIs; the fix was to inline the loops back and add a compile test asserting the futures are Send.
• Two wasmtime versions in one lockfile compile on Linux and macOS and hard-fail on Windows — the MSVC linker rejects two copies of the same rlib where ELF and Mach-O tolerate them.
• Hotfixes merged straight to main never trigger the forward-port workflow, which only runs after a stable release or on manual dispatch. Develop diverges silently.
• The YAML crate is aliased under two different names across the fleet via package renames — the import name proves nothing, only the package = key does.
• #[serde(default)] cannot see sibling fields — the default runs with no access to the rest of the struct, so "absent" and "null" cannot be distinguished that way.
• Hot-reload without a carried store registry silently wipes live conversations — every in-flight turn gets no reply and no error.

Status

stable types 1.1.1 · flow 1.1.1 · pack 1.1.5 · component 1.1.1 · runner 1.1.5 · operator 1.1.5 · i18n 1.1.1. Develop at 1.2.0-dev, research at 1.3.0-research.

open Canonical hashers still rely on stringified-value ordering rather than sorting keys explicitly — the durable fix is owed. Three repos remain on old wasmtime versions. greentic-runner CI runs no clippy and no host-target tests.

04 Deployment, environments & bundles

~1,250 commits across 7 repos · deployer PRs #155 → #540 · 4 cloud targets

This is the largest category by volume, and the one with the clearest single architectural idea: the environment, not the bundle, is the thing you deploy to. Everything else follows from making Environment a first-class, persisted entity rather than a string label.

What was delivered

• The next-gen foundation (May) — an 1,822-line design document, then a ten-step implementation: the greentic-deploy-spec contract crate, an EnvironmentStore with a filesystem backend, the op <noun> <verb> CLI shape, environment bootstrap, a revision-transition state machine, an audit log with an ownership gate, and a registry binding capability slots to native handlers.
• Four deployment targets, each an env-pack rather than a hardcoded branch: local process, Kubernetes (~2,500 lines of manifest rendering, bound ServiceAccount identity, Vault bootstrap), AWS ECS (eight PRs in a single day — task sets, Fargate launch config, target-group pooling, per-deployment ALB listener rules), and Google Cloud Run (typed target seam with an in-memory fake, ETag-guarded traffic updates, verified live on europe-west1 and torn down the same day).
• The env-manifest apply engine — a declarative document describing an entire environment, applied by one command. Fifteen step kinds, two TOCTOU re-hash gates, a --check mode that validates and diffs without mutating, and idempotency keys derived from a hash covering every mutated field.
• An operator store server — SQLite-backed HTTP service exposing the full environment API for remote management, with bearer-token RBAC, ETag optimistic concurrency, durable audit with a retention watermark, and backup/restore for disaster recovery.
• One bundle, one command — gtc start <x>.gtbundle auto-creates an environment, deploys, warms a revision and serves it. The E2E suite migrated to this path and the readiness gate moved from parsing stdout to probing /readyz.
• op env destroy — atomic rename-to-tombstone under a file lock as the commit point, then purge; a failed purge leaves a reapable tombstone rather than a half-destroyed environment.

Decisions

Code

#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum CapabilitySlot {
    Deployer, Secrets, Telemetry, Sessions, State, Revocation,
    Messaging, Extension,
}

#[async_trait]
pub trait Deployer: std::fmt::Debug + Send + Sync {
    async fn stage_revision(&self, env: &Environment, revision_id: RevisionId)
        -> Result<StageOutcome, DeployerError>;
    async fn warm_revision(&self, env: &Environment, revision_id: RevisionId,
        answers: Option<&Value>) -> Result<WarmOutcome, DeployerError>;
    async fn drain_revision(&self, env: &Environment, revision_id: RevisionId)
        -> Result<DrainOutcome, DeployerError>;
    async fn archive_revision(&self, env: &Environment, revision_id: RevisionId,
        answers: Option<&Value>) -> Result<ArchiveOutcome, DeployerError>;
    async fn apply_traffic_split(&self, env: &Environment,
        deployment_id: DeploymentId, answers: Option<&Value>)
        -> Result<TrafficSplitOutcome, DeployerError>;
}

pub fn with_builtins() -> Self {
    let mut registry = Self::new();
    for handler in BUILTIN_HANDLERS {
        registry.register(Box::new(*handler))
            .expect("built-in handler paths are unique");
    }
    registry.register(Box::new(
        super::local_process::LocalProcessDeployerHandler::default(),
    )).expect("local-process deployer handler path is unique");
    #[cfg(feature = "creds-aws")]
    registry.register(Box::new(super::aws::AwsEcsDeployerHandler::default()))
        .expect("aws-ecs deployer handler path is unique");
    #[cfg(feature = "creds-gcp")]
    registry.register(Box::new(
        super::gcp_cloudrun::GcpCloudRunDeployerHandler::default(),
    )).expect("gcp-cloudrun deployer handler path is unique");
    registry.register(Box::new(super::k8s::K8sDeployerHandler::default()))
        .expect("k8s deployer handler path is unique");
    registry
}

{
  "schema": "greentic.env-manifest.v1",
  "environment": { "id": "@@ENV_ID@@", "name": "cloudrun-demo" },
  "trust_root": "bootstrap",
  "packs": [
    { "slot": "deployer", "kind": "greentic.deployer.gcp-cloudrun@1.0.0",
      "pack_ref": "builtin",
      "answers": { "project": "@@PROJECT@@", "region": "@@REGION@@",
        "access_mode": "@@ACCESS_MODE@@", "runtime_image_digest": "@@IMAGE_DIGEST@@" }
    },
    { "slot": "secrets", "kind": "greentic.secrets.dev-store@1.0.0", "pack_ref": "builtin" }
  ],
  "bundles": [
    { "bundle_id": "cloudrun-demo", "bundle_source_uri": "@@BUNDLE_URI@@",
      "bundle_digest": "@@BUNDLE_DIGEST@@",
      "route_binding": { "path_prefixes": ["/"] } }
  ]
}

Gotchas found the hard way

• Cloud Run never delivers GET /healthz to your container — Google's frontend intercepts it and returns a branded 404 page. Probe /readyz. Kubernetes uses /healthz legitimately, in-cluster. Same path, opposite correct answer per platform. Found live.
• A bundle has two runtime pack roots and the deployer only scanned one — provider packs were invisible to the revision runtime. The false green: the generic revision ingress returns 200 for any POST, so the endpoint looked healthy.
• A library fix does not ship until every embedding binary is rebuilt. The deployer is both a binary and a library, pinned via lockfile by three other binaries; a floor bump is a behaviour floor, not an API one, so nothing fails to compile against the old copy.
• Two green PRs can merge cleanly and leave main not compiling. One added a field, the other added a struct literal, CI last ran 41 hours earlier — different file regions, no textual conflict, red main.
• A stale pack-list reference from a pre-1.1 environment bricks the whole environment, because the loader bails on the first bad reference and healthy deployments become collateral. Missing and invalid must be handled differently, and the lexical .. check has to run before touching the filesystem.
• gtc start opens a public tunnel when public_base_url is unset and a messaging pack is served — and the disable flag does not win, because the override happens first.
• env apply writes host config at step 2 but deploys bundles at step 4, so a manifest that declares a bundle and names it as default fails on first apply if the default is validated eagerly.
• bundle build can emit an empty 4 KB bundle and exit 0, surfacing one command later at deploy. Size is the cheap probe — a real bundle is megabytes.
• Bundle id must come from the manifest, never the filename stem. Two differently-named files declaring the same id would otherwise create rival deployments instead of blue-greening one.
• An exhaustive destructure is a deliberate tripwire. A new field on the environment manifest forced an explicit decision about whether the update-broadcast surface should accept it; binding to _ would have compiled and silently opened a fleet-rebinding vector.

Status

shipped One-bundle-one-command · Kubernetes with Vault · Cloud Run · session-sticky routing · env-manifest apply phases 1–2 · operator store server · deploy-spec 0.2.6.

develop only op env destroy, named local environments, the AWS ECS deployer and the full remote-control surface exist on develop but not on main — the stable binary still returns a stub.

open Vault cold start (~15 s after idle) is worked around with a keep-warm loop, not fixed. Env-manifest phases 3–4 not started. Azure and single-VM deployers not started. The --target flag still routes to the legacy Terraform path rather than env-packs; unification was decided on 2026-07-30 but sequenced behind other work.

05 Updater, trust & air-gapped distribution

2026-06-30 → 2026-07-31 · 31 days · from zero to air-gapped fleet distribution

What was delivered

• The update plan — a signed statement carrying an environment manifest as opaque JSON, with a monotonic sequence number for anti-rollback, a nonce, artifact digests, and an optional binaries list. Signed with Ed25519 over a pre-authentication encoding.
• The staging state machine — atomic plan admission under a single lock hold, then download → staged → applying → applied, with single-flight apply admission and channel-scoped state so a channel migration cannot wedge a host.
• Binary self-update — verified atomic binary swap with a rollback copy, an exec(2) auto-restart, a boot-attempts counter, and an anti-rollback tombstone. The rollback path was demonstrated end to end: a deliberately boot-broken version detected, restored and re-executed in 58 seconds.
• Push and poll — a server-sent-events transport alongside the polling loop, including the fix for a shutdown hang where a blocking SSE read wedged runtime teardown.
• The trust model — a did:web document on its own domain, separate from the plan server, so one compromise does not defeat both gates. Per-environment trust roots, a resolver that derives key ids from raw public keys, and a CLI verb that anchors an environment on the fleet root.
• Air-gap Tier 1 — a signed .gtupdate envelope: export on the connected side, carry it across on physical media, import into a per-environment content-addressed store on the other side, with digest preflight, receipts and garbage collection.
• Air-gap Tier 2 — one import point serving the entire in-gap fleet over HTTP. An import can write a static serving directory; runtimes poll it and fall back to a blob mirror for binaries. The serving guide ships with nginx and caddy configurations that were verified by running real containers.
• Toolchain distribution — gtc install resolving an OCI manifest of exact version pins, a one-command gtc up bootstrap, and a dedicated :airgapped channel carrying the dev-lane binaries the air-gap verbs need.
• Production infrastructure — a Cloudflare Worker plan store with Durable Objects live at updates.greentic.cloud, and a coordinated publisher that emitted 84 signed binary artifacts (14 packages × 6 targets) in its first run.

Decisions

Code

pub struct UpdatePlan {
    pub schema: String,
    pub plan_id: String,
    pub env_id: String,
    pub sequence: u64,
    pub created_at: DateTime<Utc>,
    pub nonce: String,
    pub target: serde_json::Value,
    pub artifacts: Vec<PlanArtifact>,
    #[serde(default, skip_serializing_if = "Vec::is_empty")]
    pub binaries: Vec<BinaryArtifact>,
    pub compat: CompatRequirements,
    pub rollback: RollbackPolicy,
}

pub fn pae(payload_type: &str, payload: &[u8]) -> Vec<u8> {
    let mut out = Vec::with_capacity(payload_type.len() + payload.len() + 32);
    out.extend_from_slice(b"DSSEv1 ");
    out.extend_from_slice(payload_type.len().to_string().as_bytes());
    out.push(b' ');
    out.extend_from_slice(payload_type.as_bytes());
    out.push(b' ');
    out.extend_from_slice(payload.len().to_string().as_bytes());
    out.push(b' ');
    out.extend_from_slice(payload);
    out
}
// Test asserts: pae("t", b"hi") == b"DSSEv1 1 t 2 hi"

pub fn swap_binary(
    new_binary: &Path, target: &Path, opts: &SwapOptions,
) -> Result<SwapOutcome, BinSwapError> {
    let new_bytes = fs::read(new_binary)?;           // read once — no TOCTOU
    if let Some(expected) = &opts.expected_digest {
        verify_digest_bytes(&new_bytes, expected)?;  // fail closed before mutation
    }
    let prev_path = prev_path_for(target);
    fs::copy(target, &prev_path)?;                    // rollback copy
    fs::File::open(&prev_path)?.sync_all()?;          // fsync for crash safety
    atomic_install(&new_bytes, target, parent)?;      // temp → fsync → chmod → rename
    Ok(SwapOutcome { prev_path })
}

<push_to>/
  plan/
    plan.json      — raw plan bytes (the signature covers these exact bytes)
    plan.json.sig  — raw signature envelope bytes
    meta           — {"sequence": <u64>, "plan_sha256": "<hex>"}
  blobs/
    sha256-<hex>   — one file per plan-referenced digest

Write order: blobs first → plan.json → plan.json.sig → meta LAST.
Each write is atomic; a per-destination lock serialises concurrent pushes;
an incoming sequence lower than the existing meta is refused.

Gotchas found the hard way

• Use notify_one, never notify_waiters. The former stores a permit, so an event fired before the waiter is listening is still delivered. The latter drops it, and the update is lost forever.
• A blocking SSE read wedges runtime shutdown for up to 15 minutes. The read can never return inside any grace window, so the fix is to detach the runtime at shutdown rather than to add a timeout.
• Broadcast bootstrap can deadlock. Older runtimes reject every broadcast-addressed plan — but the binary swap that would fix them runs after plan admission. Affected hosts need an out-of-band install.
• gtc install actively downgrades newer binaries, because it pins exact versions from the toolchain manifest, not whatever is newest on the registry. Publishing a crate does not reach users until the manifest is refreshed.
• The manifest version must equal a published release tag. A label with no matching release 404s self-update for every host whose version differs.
• Digest format asymmetry is load-bearing: plans carry sha256:<hex> with a colon; served blobs use sha256-<hex> with a hyphen. The URL must use the hyphen form.
• Plan bytes are pretty-printed JSON; the statement inside the signature envelope is compact. Confusing the two changes the digest and breaks verification silently.
• wrangler secret put with piped stdin stores a value that does not match what you sent — and reports success. The production upload token diverged on first deploy this way.
• "Environments: N" in the publish notification is an upload count, not fleet reach. It says how many plans were uploaded, not how many hosts received or applied them.
• Dev-lane binaries never report their publish version — --version shows the manifest version, not the run-id in the archive filename, so dev self-update is a guaranteed 404.
• promote is a point-in-time digest copy, not an alias — every forced republish silently strands the stable tag until it is re-promoted.

Status

shipped Updater foundation (1.1.2 stable) · air-gap Tiers 1 and 2 · push and poll · auto-restart with boot-failure rollback · coordinated publisher (live) · did:web trust anchor (live) · gtc up · the :airgapped toolchain channel · production plan store (live).

descoped The turnkey in-gap plan server — deliberately not built.

open Phase D (trust-root rotation: key roles, threshold verification, two-phase commit, recovery journal) is blocked on a dependency publish. Garbage collection does not cover push-to directories, so a serving directory grows unbounded. A re-import does not repair a corrupt mirror blob. The notify endpoint is reachable off-host and should be gated.

06 Messaging, webchat & hot-reload routing

May–July 2026 · greentic-start + greentic-messaging-providers · shipped in start 1.1.37

The serving runtime has to take an inbound webhook from Slack, Teams, Telegram or a browser and route it to the right tenant, the right bundle, the right flow, and the right revision — while the environment is being reloaded underneath it. This category is that routing problem and the concurrency problems it exposed.

What was delivered

• Multi-bundle webchat URL routing — a URL grammar of /{tenant}[/{bundle}[/{flow}]] that multiplexes many bundles on one tenant, backed by a 1,700-line routing module with pre-built bundle and flow indices. Per-bundle and per-flow URLs are printed at boot.
• "Standard Option A" — a pack-provided single-page webchat UI supersedes the built-in console, which is retained as a fallback for bundles that ship no UI. A separate UI-only pack exists for bundles that need a browser tier without a messaging provider.
• Hot-reload granularity — a three-way reload outcome (Unchanged / RoutingOnly / Full) so that flipping a default bundle rebuilds four routing tables instead of re-instantiating WASM. Conversation loss during reload went from 8-in-300 to 0-in-300.
• Session-sticky revision routing — provider-specific conversation hints (Telegram chat.id, Slack channel, webchat conversation id) pin a conversation to a revision, with generation-monotonic eviction so a stale drain-phase request cannot steal a pin from the active revision. In-memory or Redis-backed, with TLS.
• Tenant/team binding correctness — the default tenant changed from demo to default; admin requests naming no tenant are now rejected rather than guessed; unknown webchat tenants return 404 instead of failing open; mixed-tenant environments refuse to activate under a scoped secrets backend.
• Scoped messaging endpoints — a full endpoint CLI plus an auto-resolution ladder (HeaderWins > Hit > Miss > Ambiguous > NoImpl > Skipped) across six shipping providers.

Decisions

Code

pub(crate) fn forwarded_ws_scheme(headers: &[(String, String)]) -> &'static str {
    let is_https = headers
        .iter()
        .find(|(k, _)| k.eq_ignore_ascii_case("x-forwarded-proto"))
        .and_then(|(_, v)| v.split(',').next())
        .is_some_and(|proto| proto.trim().eq_ignore_ascii_case("https"));
    if is_https { "wss" } else { "ws" }
}

#[derive(Clone, Debug, PartialEq, Eq, Hash)]
pub(crate) struct RevisionStoreKey {
    pub deployment_id: String,
    pub revision_id: String,
    pub tenant: String,
    pub team: String,
    pub customer_id: String,
    pub bundle_id: String,
}

fn torn_write_newly_unbacked(
    current_unbacked: &std::collections::BTreeSet<String>,
    previously_unbacked: &std::collections::BTreeSet<String>,
) -> Vec<String> {
    current_unbacked
        .iter()
        .filter(|id| !previously_unbacked.contains(id.as_str()))
        .cloned()
        .collect()
}

#[derive(Clone, Eq, Hash, PartialEq, Debug)]
pub struct DedupKey {
    pub deployment_id: DeploymentId,
    pub tenant: String,
    pub team: String,
    pub user_id: String,
    /// Flow discriminator for webchat multi-flow bundles.
    pub flow_hint: Option<String>,
}

Gotchas found the hard way

• Making a slow path fast exposes latent torn-read bugs. Cutting reload from seconds to milliseconds made a pre-existing non-atomic two-file write observable for the first time.
• :latest never reliably rotates on the provider registry — pin :stable or an explicit version. Observed drift: :latest at 0.5.6 while :stable was 0.5.8.
• A pack-list entry keys on the pack filename, not the manifest id, while the flow index reads the manifest — so renaming a .gtpack re-scopes its secrets and produces 500s.
• Slack interactive payloads arrive form-encoded, not as JSON, so a JSON parse yields null and instance identification cannot find the app id. Multi-app deployments hit an ambiguity error.
• A debouncer only flushes on tick boundaries at a quarter of its timeout. Widening a debounce to "harden" a timing test widens the tick proportionally and breaks it deterministically.
• GET /chat returning 200 proves nothing about the message path. A static asset served fine while the actual POST 404'd on a tenant-name mismatch.
• Mixed-tenant environments boot locally but fail under Vault. The dev store is unscoped, Vault is scoped — the mismatch is invisible until production.
• Building a pack and publishing a pack are gated by different mechanisms — a filesystem glob decides what builds, a matrix file decides what publishes. A pack can be built, tested, and permanently unpublishable.

Status

shipped All of the above is on the stable lane in greentic-start 1.1.37, pinned in the :stable toolchain manifest, and validated by a demo asserting 58 and 43 checks across declarative and imperative deployment paths on a pure stable toolchain.

deferred Slack form-encoded interactive routing (needs the server to forward the raw body); Teams JWT signature verification — phase 1 decodes and validates claims but does not cryptographically verify, so a forged token with a correct audience is admitted; revision-aware event routing; the unauthenticated webchat token endpoint.

07 Instance identity & slot prefill — a nine-day cross-repo train

2026-05-29 → 2026-06-06 · ~50 PRs · 14 repositories · 5 architectural tiers

This one is worth calling out separately because of its shape: a single coherent feature delivered as a wave through the entire dependency stack — WIT contract, then types, then runtime, then serving host, then providers, then components, then a worked example — in nine calendar days.

The problem

One messaging provider deployment could only serve one bot identity. Running two Teams bots on the same runtime was impossible: the provider type ("teams") was the only routing discriminator available, so every inbound webhook for that provider class went to the same flow regardless of which bot actually received it. Operators needing multiple bots had to deploy a separate runtime per bot — which defeats the multi-tenant architecture entirely.

A second gap sat alongside it. When a user typed "Start an NDA between Acme and us, due 2026-07-15", the router could find the right flow, but the structured data inside that sentence was thrown away. The user then re-typed the counterparty and the date into a form. There was no slot-extraction component, no way to carry the originating utterance through the dispatch directive, no prefill field on the card, and no flow-level schema declaring what a flow expects.

The train, in dependency order

Decisions

Code

interface instance-identity-api {
  identify-instance: func(input-json: list<u8>) -> option<string>;
}

interface instance-identity-describe-api {
  describe-identify-instance: func() -> option<list<u8>>;
}

world instance-identity {
  export instance-identity-api;
}

world instance-identity-describe {
  export instance-identity-api;
  export instance-identity-describe-api;
}

pub const SLOT_SCHEMA_METADATA_KEY: &str = "greentic.slot_schema";

// in FlowDoc / FlowIr:
#[serde(default, skip_serializing_if = "Option::is_none")]
pub slot_schema: Option<Value>,

pub(crate) fn extract(input: &SlotExtractionInput) -> SlotExtractionOutput {
    let utterance = input.utterance.as_str();
    let mut slots: Vec<ExtractedSlot> = Vec::new();
    let mut filled: Vec<String> = Vec::new();
    let mut missing: Vec<String> = Vec::new();
    for def in &input.slot_definitions {
        let matched = match def.slot_type {
            SlotType::String  => extract_string(utterance, def),
            SlotType::Enum    => extract_enum(utterance, def),
            SlotType::Number  => extract_number(utterance, def),
            SlotType::Date    => extract_date(utterance, def),
            SlotType::Boolean => extract_boolean(utterance, def),
        };
        // …

async fn handle_worker_invoke(
    req: Request<Incoming>,
    state: Arc<ServeState>,
    peer_is_loopback: bool,
) -> Result<Response<Full<Bytes>>, Response<Full<Bytes>>> {
    if !peer_is_loopback {
        return Err(error_response(
            StatusCode::FORBIDDEN,
            "worker invoke is restricted to loopback callers",
        ));
    }
    let activation = state.current();
    let body_bytes = read_body_limited(req).await.map_err(|_| { /* … */ })?;
    let worker_req: WorkerInvokeRequest =
        serde_json::from_slice(&body_bytes).map_err(|err| { /* … */ })?;
    // resolve deployment by asserted tenant, dispatch to revision, return replies

Gotchas found the hard way

• Telegram's webhook body carries no bot identity at all — the only discriminator is an HTTP header. That single fact forced the wrapper payload design: the host must pass headers, not just the body.
• Every implementation must accept both the wrapper shape and a bare body. Shipping wrapper-only would break older hosts and every existing test harness.
• An AsyncFnMut helper extracted from the probe loop broke Send inference in the serving host, three repos downstream. The refactor was reverted, the loops re-inlined, and a compile test added to prevent the same tidy-up from being reattempted.
• Provider invocation fails closed on cross-pack ambiguity — if two packs in a revision bind the same provider type, the runtime refuses to guess rather than picking one.
• Empty and whitespace secret tokens are rejected explicitly, otherwise they match falsely against the admit table.

Status

shipped The WIT contract, flow model, slot extractor, card prefill and worker gateway are on main. The runtime identity APIs and the six provider implementations are on develop. open The multi-endpoint conformance test and the concepts documentation page were planned and never landed. Multi-turn slot filling was explicitly out of scope.

08 Secrets, capability security & trust

Mar–Jul 2026 · ~430 commits across 7 repos · 14 findings fixed

Two threads run through this category: consolidating a scattered secret lifecycle into one library, and closing a series of concrete security holes — several of which were found by adversarial review or by running the code rather than reading it.

What was delivered

• Secrets library consolidation — canonical identity functions, a data-driven generation model, capped generators, a sink abstraction, and collision-free naming for cloud backends. The trigger was concrete: the deployer's cloud path never saw generated secrets at all, so a JWT signing key and a webhook secret simply never reached the cloud.
• Vault workload identity — replaced secret values in Kubernetes with workload identity. Pods get a service account; the deployer renders only references; the runtime exchanges the pod's projected token for a short-lived Vault token and resolves references at run time. Proven end to end on a live cluster.
• Security tooling at fleet scale — CodeQL across 63 repos, dependency review across 59, automated remediation on a shared workflow with a fork-PR guard, and a long-red fuzz and audit pipeline repaired after three latent breakages were unstacked.
• Six concrete vulnerability fixes, detailed below.

Security findings fixed

Decisions

Code

// Same invariant for the trust root: a plan that seeds or rotates the
// env's signing-key trust anchor would gain permanent signing authority.
// Stripped at sign time by `strip_non_applyable_blocks`; refused here as
// the fail-closed backstop.
if manifest.trust_root.is_some() {
    return Err(OpError::InvalidArgument(
        "update plan target declares a `trust_root` block: a plan may not \
         re-point the trust root it is verified against. Configure the \
         trust root with `op env apply`."
            .to_string(),
    ));
}

/// 26 bytes of SHA-256 (208 bits, 52 hex chars — the most that fits beside a
/// 10-char prefix inside the 63-char limit). Collision-resistant, not injective:
/// two ids CAN in principle share a stamp, and one that did would be classified
/// `SecretOwnership::Ours`. At 208 bits that is infeasible to hit by accident
/// or to construct. Do not shorten it — an earlier 32-bit digest here had
/// findable collisions between two valid env ids, which is a full bypass of the
/// ownership boundary.
pub(crate) fn env_owner_stamp(env_id: &str) -> String {
    use sha2::{Digest, Sha256};
    let digest = hex::encode(&Sha256::digest(env_id.as_bytes())[..26]);
    let readable: String = env_id.to_ascii_lowercase().chars()
        .map(|c| if c.is_ascii_alphanumeric() || c == '-' || c == '_' { c } else { '-' })
        .take(10).collect();
    format!("{readable}-{digest}")
}

const RAW_TEXT_ALPHABET: &[u8] =
    b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_-";

/// Upper bound on a generated secret's declared `length`. The value comes from a
/// pack manifest, so a malformed or hostile pack could otherwise turn a single
/// integer into an unbounded allocation / CPU sink (`base64url`/`hex` further
/// scale it). No real secret approaches this; a pack asking for more is rejected
/// loudly rather than silently clamped.
const MAX_GENERATED_LENGTH: usize = 4096;

pub fn render_vault_network_policy(p: &VaultInfraParams) -> Value {
    json!({
        "apiVersion": "networking.k8s.io/v1",
        "kind": "NetworkPolicy",
        "metadata": { "name": "allow-vault-ingress-from-workers", "namespace": p.namespace },
        "spec": {
            "podSelector": { "matchLabels": { "app.kubernetes.io/component": VAULT_NAME } },
            "policyTypes": ["Ingress"],
            "ingress": [{ "from": [{
                "namespaceSelector": {
                    "matchLabels": { "kubernetes.io/metadata.name": p.worker_namespace }
                },
                "podSelector": {
                    "matchLabels": { "app.kubernetes.io/component": WORKER_COMPONENT }
                },
            }], "ports": [{ "protocol": "TCP", "port": VAULT_PORT }] }],
        },
    })
}

Gotchas found the hard way

• kubectl port-forward bypasses network policies — it reaches the pod over loopback. A port-forward that "works" can completely mask a pod-to-pod block that produces 401s in production.
• Two normalisers in the same path are deliberately asymmetric — one segment preserves hyphens, the next folds them to underscores. Making them consistent writes to a path nothing will ever read.
• The operator has no Vault backend at all — Vault lives only in the serving host, and the runner has a third, separate backend system. Three same-shaped abstractions with similar names coexist.
• env destroy leaves cloud and Vault secrets behind — only provider infrastructure is torn down.
• The spec crate contains dead source files that are never declared as modules, so any grep or code-index reports type definitions from code that does not exist at runtime.
• Private repos on the Free plan receive no org secrets — which silently breaks both token minting and publishing. The fix for one repo was to make it public.

Status

complete Secrets consolidation (dev and stable) · Vault workload identity, proven end to end · trust-root takeover closed with dual defence · cloud secret ownership · CodeQL and dependency review at fleet scale · fuzz and audit pipeline repaired.

open The provision/promote engine is scaffolded but has no production callers yet. Secret rotation returns not-implemented. Cloud and Vault secrets are orphaned on environment destroy. The bundled-secrets backend — which would collapse N cloud secret objects into one per environment — is designed but unbuilt.

09 Demos, E2E, QA & verification discipline

18 demos · 10 E2E workflows · 10 conformance suites · a measured, not guessed, performance answer

Demos here are not marketing artifacts — they are the integration tests. Several of the most consequential bugs in this whole body of work were found because a demo asserted something a unit test could not.

What was delivered

• A runnable demo catalog — 18 demos, each producing a signed pack, bundled with providers and answer documents, published to a container registry and releases.
• Five new CI gates, each written in response to a real shipped defect: bundle shape verification, build-mode assertion, provider minimalism, anonymous-pull verification, and an app-pack completeness check.
• Provider minimalism — the rule that a bundle vendors the browser tier plus whatever it actually configures, and nothing else. Measured effect across the fleet: 222 MB → 154 MB, −31%, with the worst offenders down 76–79%.
• E2E across six platform targets plus provider, cloud, browser, telemetry and store suites; migrated from parsing stdout for readiness to probing a readiness endpoint.
• Conformance suites asserting signed manifests, 14 mandatory telemetry span keys, deployer idempotency, and message deduplication.
• Self-checking demos — demo scripts that double as assertion suites, so a silently-green run is structurally impossible.

Three bugs that only demos could find

The performance question, settled by measurement

Should the runner cache pre-instantiated WASM components across invocations? Two independent code-reading analyses estimated the cost at 4–8 seconds. The measurement:

Throughput stayed flat at ~12k ops/s from one thread to sixteen — no contention cliff. The estimate was wrong by five orders of magnitude. The dead cache field was deleted rather than wired up, and the lesson was written down: measurement beats code-path reasoning.

Verification discipline

This is the part that generalises beyond Greentic. The rules below were each written after a specific failure.

Mutation-prove

Agent-written tests are validated by deliberately breaking the production code and confirming the test fails — re-run personally, not taken on report. Five distinct fake-test patterns were catalogued: tests that re-derive the logic instead of calling it; fixtures that put the target first so a first-entry shortcut is indistinguishable from a correct match; construction by struct literal so nothing between test and assertion can break; hand-built fixtures blind to a changed discriminator; and a name-filtered harness that matches nothing and exits 0.

No-fail-fast

The default test runner stops at the first failing binary, so a repository can hold several independent failures while showing one. Verification runs must not stop early.

All targets, all features

Linting only the library skips test-module code, so lints pass locally and fail in CI.

Check the real target

A host build skips architecture-gated blocks entirely — a WASM component can pass every local gate and still fail to compile for the target it actually ships to.

One adversarial round

Before merge, one round of adversarial review, then fix what is valid and in scope, then simplify once. Never loop until clean: findings drift off the PR's subject, and an out-of-scope finding becomes a follow-up rather than scope creep.

Design gate first

For non-trivial work, the spec is reviewed by independent readers — one per lens, each asked to find where the spec is wrong and to anchor every finding to a real line of code. On one deployment feature this falsified four load-bearing claims that had survived careful solo reading.

Run every command you document

A fabricated command is prose: it compiles, passes every gate, and often reads better than the real one. One documentation review caught three commands describing verbs and flags that did not exist.

Code

const REQUIRED_OTEL_KEYS: &[&str] = &[
    "service.name",
    "greentic.pack.id",
    "greentic.pack.version",
    "greentic.flow.id",
    "greentic.node.id",
    "greentic.component.name",
    "greentic.component.version",
    "greentic.tenant.id",
    "greentic.team.id",
    "greentic.user.id",
    "greentic.session.id",
    "greentic.run.status",
    "greentic.capability",
    "greentic.artifacts.dir",
];

pub fn assert_deployer_idempotent(
    deployer: &Path, config: &Path, extra_args: &[&str],
) -> Result<()> {
    let first = run_deployer(deployer, config, extra_args)
        .context("first deployer apply failed")?;
    let second = run_deployer(deployer, config, extra_args)
        .context("second deployer apply failed")?;
    if first != second {
        bail!("deployer is not idempotent: outputs differ between runs");
    }
    Ok(())
}

# lib.sh — shared by demo.sh and demo-start.sh.
# Sourced, never executed. The caller must set `HERE` and `PORT` before
# sourcing; everything else is passed as an argument so the two demos can
# differ freely in how they bring an environment up.
#
# The one piece of shared STATE is FAILURES: `fail`/`expect` bump it, and each
# script exits on it. Both demos double as checks, so a silent green run must
# be impossible.

FAILURES=0
fail() { bad "$*"; FAILURES=$((FAILURES + 1)); }
expect() {
  if [[ "$3" == *"$2"* ]]; then ok "$1"
  else fail "$1 -- expected '$2', got '$(printf '%s' "$3" | head -c 70)'"; fi
}

Gotchas found the hard way

• grep -x 'bundle.lock.json' matches bundle-lock.json — the dot is a regex wildcard. Use a literal file test when the distinction is the whole point of the check.
• A toolchain install prefetched 92 registry blobs all-or-nothing with no resume, so three retries were three fresh chances to lose the job rather than three chances to finish. Dropping the unused prefetch cut install from 256 s to 77 s.
• pgrep -x silently matches nothing for names over 15 characters — it compares the kernel's truncated command field.
• strings only reports printable ASCII runs — an em dash splits a marker phrase into two runs and the match silently fails.
• Docker's 64 MB default shared memory caused 525 spurious test failures that presented as unrelated database migration errors. Same commit at 1 GB: 2 failures.
• Never pkill -f in a demo script — the pattern matches the calling shell's own command line and kills the session.
• A demo showing identical output on three different URLs was a mislabelled pack, not broken routing. Check the asset before debugging the router.

Status

verified Demo catalog at 1.1.7 with all five gates · 17 bundles republished and verified byte-reproducible · cloud update demo live · designer database CI gap closed on both lanes · E2E migrated to the environment path.

open The stable E2E promotion gate has its build foundation done but the orchestrator, full-demo suite and reviewer gating remain. Integration strategy phase 2 has been stalled since mid-May. The durable fix for the registry client's retry behaviour is owed but blocked on the dead stable release lane. Two demo bundle packages serve stale content indefinitely.

⚠ Landmines

The cross-cutting traps most likely to catch you, collected from all nine subsystems. Each cost real debugging time at least once. The per-subsystem sections above carry the rest.

▸ Pick up here

Ordered by what unblocks the most, not by size. Everything below is known and tracked; nothing here is a surprise discovered while writing this document.

P0 Do these first

P1 Structural — do before building new features

P2 Feature continuation

P3 Deferred by decision — revisit consciously

These were deferred deliberately, not forgotten. Two are worth re-examining early:

• Teams JWT signature verification (phase 2). Phase 1 decodes and validates claims but does not cryptographically verify, so a forged token with the correct audience is admitted. security-relevant — this is the one deferred item with a real edge; decide whether the deployment model makes it acceptable.
• The unauthenticated webchat token endpoint, which mints valid signed tokens from pack presence alone.
• Slack form-encoded interactive routing — needs the server to forward the raw body.
• Revision-aware event routing; multi-turn slot filling; Azure and single-VM deployers.
• The bundled-secrets backend — designed but unbuilt; collapses N cloud secret objects into one per environment, which is a real cost saving at scale.
• macOS runner spend, roughly $645/month against a $50–110 target. Mitigation path identified, not executed.

Two structural problems with no known fix

Flagged so you do not spend time rediscovering that they are hard rather than merely undone.

• The -dev binary namespace is shared between our two lanes — and yours wins. Both lanes publish binaries under the same <crate>-dev name on crates.io, and 1.3.0-research.* numerically outranks 1.2.*. So a cargo binstall <crate>-dev on the develop lane can resolve to your research build. One package was observed serving a twelve-day-stale research binary to develop consumers. There is no structural fix today — only checking which lane a -dev version actually came from before trusting it. Worth folding into the P0 lane decision.
• Environment destroy orphans cloud secrets. Only provider infrastructure is torn down; entries in cloud secret managers and Vault survive. Manual cleanup is currently owed after every destroy.

§ Access you'll need

You already have most of these. Listed so the gaps are obvious — the ones you are least likely to hold today are Cloudflare, the cloud deployment accounts, and crates.io publish rights across the whole namespace. Ask Vlad for the credential inventory and handover procedure; that is deliberately not recorded here.

§ Where the knowledge lives

This document is a map, not the territory. When you need depth, these are the primary sources — in the order you will want them.

The one idea to carry forward

If you take nothing else from this document: the through-line in every subsystem is make the invariant mechanical. Tiers are derived, not declared. Canonical bytes are enforced by a canonicaliser, not by convention. Ownership of a cloud secret is a 208-bit stamp, not a naming agreement. A trust root cannot be re-pointed by the thing it authorises — at sign time and again at apply time. A demo that could pass silently is not considered a demo.

That principle is also the honest answer to the lane problem. The two lanes did not drift because anyone was careless — they drifted because nothing mechanical was stopping them. A floating branch pin, a presence-only binary check, and a shared -dev namespace with no lane discriminator are all places where the invariant lived in someone's head. Whichever way you decide the P0 lane question, the durable part of the answer is a gate rather than an intention.