hatchery

Declare, deploy and monitor the house estate: dokku apps, bare containers, databases, jobs, secrets and DNS.

Hatchery keeps one declaration for each stack: a manifest, the kind files beside it, and the tofu files it writes. The estate runs on three boxes: the laptop, the mini and the opi. A stack holds dokku apps, docker containers that dokku does not own, postgres clusters and their databases, and jobs that launchd or systemd runs. Secret keys move out of the config sidecars into secrets files and vault documents, and the forge's CI secrets come from vault too. MWServer was the first target, but nothing in the model is MWServer-specific. Service kinds and their environment contracts are data, so a new service needs no change to the core.

A job beats

hatchery install --jobs writes each declared job so its program runs inside one small wrapper, /bin/sh -c <script> hatchery-beat <stack> <job> <pulse>, in front of the program. When a run ends the wrapper posts a beat to <pulse>/api/beats, where pulse is the manifest's publish target, with the key in ~/.roost_node_key. pulse sets the beats against the declaration to say which job is late or failed. The wrapper ends with the program's own exit code whether pulse answered or not, and curl gives up after two seconds without a connection. Only a job that ends its run is wrapped: a job with a schedule, and on Linux a run-at-load oneshot such as box-boot. A job that is kept alive, a service started at login on a Mac and a hook another unit starts all run as declared, and so does every job of a manifest that publishes nowhere. No job's own script changes.

On a host stack that declares the resident with fetched: roost, the wrapper is roost beat in its place (house#149): <home>/.local/bin/roost beat <stack> <job> --pulse <pulse> -- <program...>, where <home> is the home the stack's job programs are written under, since launchd reads no $HOME. It posts the same beat with the same limits and exits with the program's code. A stack with no fetched: roost, or whose programs name no home, keeps the shell line.

The estate file

estate.json in the state folder names every manifest of the estate once:

{ "manifests": ["~/.config/hatchery/hatchery.json", "estate/hatchery.json", "air/hatchery.json"] }

An entry is a path relative to the file's own folder, an absolute path, or one that starts with ~. hatchery finds the file at HATCHERY_ESTATE, or at ~/.config/hatchery/estate.json, which is a link to the state folder's file. A verb that takes several manifests, such as secrets ledger, secrets rotate, install and declared, reads them all when no --manifest is given. A verb that takes one, such as config set, deploy and status --stack, reads the manifest that holds the stack it was given. A stack name that two manifests hold is refused with both paths. --manifest overrides the file for one run, and a machine with no estate file reads the one default manifest as before.

Commands

Every command that reads a manifest takes -m, --manifest <path>. See Finding the manifest for the search order when you do not give one. A command that takes an SSH target also accepts a saved host name with a leading @, for example @opi. A command that writes usually needs --yes, and shows its plan without it.

Stacks and services

Command	What it does
stack new <name> --tofu-dir <dir> [--backend <b>] [--host <target>] [--environment prod|staging|dev] [--set key=value]	Create a stack in an empty directory: write the tofu files, run tofu init, and record the stack in a manifest.
stack list	List the stacks a manifest declares.
stack clone <source> <target> [--environment <env>] [--create --tofu-dir <dir> --yes] [--apply] [--db full|schema|none]	Plan what a clone of a stack carries, and create the clone with --create. Also takes --backend, --cluster, --host, --port, --network, --gated and --show-secrets.
stack rm <stack> [--yes --confirm <stack>] [--purge]	Show what tofu destroy removes, and destroy the stack with --yes and the name repeated. --purge also deletes the tofu directory.
service new <stack> <name> --kind <kind> --image <ref> [--domain <d>] [--port <n>] [--network <net>]	Write a new service into a stack: its tofu declaration, image variable, config and manifest entry. Also takes --gated, --mint-keypair, --no-vault, --dry-run and --yes.
new <stack>/<name> --kind static|node|swift|board [--domain <d>]... [--storage <path>]... [--full] [--dir <path>] [--yes]	Make a new app on the estate. The short form, a spike, makes the kind, the declaration, the scaffold, the app on the box with its ports pinned and the tunnel route. The kind declares delivery: git, so the declaration has no deploy block and the first push builds the app (house#147). Each --domain is a name the declaration carries, and each one in the zone gets a tunnel route; with none, the app answers at <name>.jimmyhoughjr.net. Each --storage is a mount the new kind file declares, and the declaration names it as dokku storage: the first after the app, as fixit is, and each other one with the last part of its path added, as rookery-work is. A kind file that is there already keeps its own storage; --full adds the forge repo with the remote forge, the workflow, the vault registration and a database when the kind asks. Each step re-runs to the same place, and a second run with --full promotes a spike. The apply is aimed at the new app's own resource, so the changes a stack's plan holds for its other apps are named and left alone. Without --yes every step prints and nothing changes (house#82).
route <stack>/<service> [--yes] [--dry-run] [--zone <zone>]	Publish the tunnel route of each name the service declares in the zone: a proxied CNAME to the account's one healthy remote-managed tunnel and an ingress rule to dokku's nginx at http://localhost:80. Each part is read first and written only when it is missing, so a second run says each name is routed already, and a name outside the zone is skipped with the reason. The Cloudflare token is CF_API_TOKEN or ~/.cf_api_token, read in process and never printed. new runs the same code for its route step. Without --yes, or with --dry-run, each name prints what would be done (house#155).
deploy <stack>/<service> [--image <ref>] [--yes] [--dry-run]	Deploy a service by the delivery its kind file declares, and by no other road. image writes the image into the stack's tofu variables and manifest and applies the plan, or with --image on a dokku app runs the image deploy with its port map pinned; without --image it brings tofu back to the image the manifest declares. git pushes the declared checkout's main to the box's dokku remote for the app, on the host that declares the checkout. board runs roost's status on the host whose job runs it, over ssh from another machine. A git or board deploy reads the port map before and after, and sets it back if the build moved it. A git push and an image deploy first read dokku's deploy lock with apps:report <app> --app-locked, read it again every five seconds while it is held, and after sixty seconds refuse with "a deploy of is in progress; try again in a minute" (house#161). A refusal prints its sentence alone and exits 1, with no usage text. --image on a git or board service is refused. Without --yes the verb prints the delivery and the line it would run, and runs nothing. serve's /api/deploy takes the same road.
up <stack>[/<service>] [--yes]	Start the services a stack declares. Dokku stacks only.
down <stack>[/<service>] [--yes]	Stop the services a stack declares. Dokku stacks only.
restart <stack>[/<service>] [--yes]	Restart the services a stack declares. Dokku stacks only.

Every verb that names a service takes it as one token, <stack>/<service>, and a bare <stack> means every service the stack declares. The old shapes, two tokens and the --stack and --service flags, are still read and each says the new form on standard error. A write prints its plan and changes nothing without --yes; --apply in deploy is read as --yes for a time and says so. Ruled on 2026-10-03 on house#66 (house#67).

Needs

A service's after list names the apps that must be serving before it starts, and hatchery box order reads that order and checks that each app in it serves now. The boot job box-boot reads the same needs at every boot from the declaration hatchery install places on the box (house#140). after says only the order, never what breaks when the thing it names is not actually answering, and a boot order job can read clean while a service that started too early runs degraded for hours (house#137).

needs says more. Each entry names a service, as <stack>/<service> or a bare name where exactly one stack declares it, and carries:

• kind: start when the service does not recover on its own if the need was not up when it started, or run when it asks again on its own.
• for: a few words for a report, saying why the need exists.
• road: how the two services reach each other. Left out, it derives: box when the two stacks declare the same host, public otherwise. Write it only to override a derivation that cannot be made, such as two containers sharing one docker network — network and lan are override-only words, never derived.

after keeps working with no edit: it reads as a need of kind start, so hatchery box order and every existing manifest are unaffected. A service may also declare ready, a command that runs inside its container and exits 0 when it answers, for a need with no healthPath to ask instead.

hatchery needs reads every manifest the estate file names and prints the layers those needs put the estate in — layer 0 needs nothing, and a service sits one layer past the latest of what it needs, worked out fresh each run rather than read from a file. --answers asks each service its own question, over HTTP where it declares healthPath, or ready over ssh and docker exec otherwise; it reaches every box named and changes none of them. hatchery restart, down and deploy print an "also touched" section before their --yes gate, naming every service whose needs point at the one in hand.

Watching

Command	What it does
status [--stack <s>] [--timeout <seconds>]	Report the live health of each declared service. The exit code fails on unreachable or degraded.
events [--stack <s>] [--limit <n>] [--history <file>]	Show the health changes that serve recorded in the history file.
serve [--bind <addr>] [--port <n>] [--token <t>] [--watch-interval <seconds>]	Serve the API on this machine and poll health on its own clock. GET / answers the API's index as JSON, and the coop's Stacks page is the face. Also takes --history, --alert-webhook, --pulse and --pulse-key-file.
serve --install / serve --uninstall	Install serve with the given options as a launchd agent, or remove that agent.
declared [--manifest <m>]... [--answers] [--publish] [--publish-to <url>]	Print what the manifests declare as JSON, with no config values. Each service with a kind file carries configKeys, the keys its kind's environment block declares, as {key, why, secret} and never a default, an example or a value, and [] for a kind that declares none. --publish sends it to pulse with the roost node key. --answers prints one name kind backend line per service.
needs [--manifest <m>]... [--answers]	Print the layers a service's needs put the estate in, with each need's kind and road. --answers asks each service its own question and changes nothing.

Configuration

Command	What it does
config validate <config.json> --service <kind> --backend <b>	Check a config map against the environment contract of a service kind.
config audit [--stack <s>]	Read the live config of each declared service and check it against its contract.
config sync <stack>[/<service>] [--dry-run]	Copy the live config into the declared sidecar, so the two agree.
config set <stack>/<service> KEY=VALUE... [--yes]	Merge values into a service's config. KEY= with no value removes the key. Without --yes the keys print and nothing is written.
config split <stack>[/<service>] [--dry-run]	Move a service's secret keys out of its sidecar into a secrets file of their own.

Boxes

Command	What it does
box init [<host>] [--backend dokku|appPlatform|cloudRun] [--yes]	Check the prerequisites of a backend, and with --yes fix what is missing on a dokku box. Also takes --cluster, --project, --key and --network.
box order <host> [--manifest <m>]...	Read the boot order that the after lists and the needs declare, and check that each app in it serves now. It writes nothing; the order is in ~/.local/share/hatchery/declared.json on the box, read by the boot job box-boot.
box scan [<target>]	Read the apps, containers and jobs that run on a box or a platform, and sort each one as declared, hatchery-shaped or foreign.
box adopt <target> <app> --stack <s> [--kind <k>] [--kind-file <path>] [--dry-run]	Declare an app or a container that already runs on a box into a stack. Also takes --replace, --refresh-config, --name and --no-vault. It reads the app's delivery off the box and writes it into the kind file.
box adopt <target> --stack <s> --deliveries [--yes]	Read how each app of the stack reached the box and fill delivery in its kind file. Without --yes it prints each app's delivery and writes nothing.
box space <box> [--free]	Show the free disk on a box, and with --free remove the build cache and the images that no container uses.
install [-m <manifest>]... [--host <target>] [--yes] [--jobs] [--json] [--publish]	Read what each host holds against its declaration and the forge's main, and with --yes install the difference.
host add <name> <target>	Save an SSH target under a name.
host list	Show the saved targets, the targets stacks use, and the targets ~/.roostrc names.
host rm <name>	Remove a saved target.
doctor [--backend <b>] [--host <target>] [--stack <s>]	Check the prerequisites to manage a stack on a backend.
setup [--backend <b>]	Print the steps that get a box or a platform ready for hatchery.

Kinds and DNS

Command	What it does
kind add <path>	Add a hatchery-kind.json file to the registry beside the manifest.
kind list	List the kinds the registry declares.
dns render <kind>	Print the dnsmasq configuration that a resolver's kind declares.

A kind file's delivery says how its service reaches the box (house#125):

• image: dokku pulls a built image with git:from-image. Only this delivery puts a deploy block in the dokku declaration.
• git: a push of the app's main to the box's dokku remote, which builds it there. Every scaffold of new writes this.
• board: roost's status run writes a status board to the box.

A git checkout is declared on a host stack, in its settings as "checkouts": "coop=~/repos/coop, ~/repos/fixit", where a bare path names the app by its last folder. A job's program under ~/repos/<app> declares that checkout too. The checkout's dokku remote must name the app, so a clone of another repo with the same folder name is refused.

box adopt reads the delivery from one dokku fact: git:from-image records the image as the app's git-source-image, and a git push leaves it empty. The status app is a board when a host stack declares a job that runs roost status.

A kind file without the key reads as git when new wrote it from a scaffold, and as image otherwise, which is what each app on the estate was rendered with before the key existed. The aliksend/dokku provider runs git:from-image on the create and on every update of an app that declares a deploy. Before the first push no image exists, so the create failed and the provider destroyed the app in the same apply, and on every later update the image replaced the repository a push had built (house#147).

Databases

Command	What it does
db provision --server <container> --database <name> --admin <user@host|local> [--owner <role>] [--app-user <role>]	Create a database, its roles and its grants on a postgres server. With --network, it also creates an absent server. Also takes --host, --port and --publish.
db provision --declared <stack>/<service>	Make sure that every database the manifest declares on that cluster exists, with its roles and grants.
db adopt <target> <container> --stack <s> [--dry-run]	Declare the databases inside a postgres cluster that already runs on a box.

No password a provision mints or holds goes into an argument: the server's superuser password reaches docker run through an env file made from standard input, mode 600 and removed after, and every statement that carries a password reaches psql on standard input, so ps on either machine shows the host, the user and the database and nothing else. The one exception is gcloud sql users set-password --password, which has no standard input form.

Secrets and vault

Command	What it does
secrets holders <stack>/<service>	Print who holds each of a service's rotatable secrets.
secrets rotate <stack>/<service> [<key>...] [--dry-run] [--yes]	Replace declared secrets in order: the issuer, then each holder, then each restart.
secrets sync <stack>/<service> [--dry-run]	Set a service's secret keys into its vault document.
secrets ledger [-m <manifest>]... [--due <days>] [--json] [--publish]	List every declared secret with its issuer, liveness, dates and what is owed next. --json prints one document, and --publish sends it to pulse's /api/ledger with the node key, where the coop's Tokens page reads it.
secrets issued <stack>/<service> <key>... [--on <day>]	Stamp the day a person issued a new value for these keys, so the ledger stops putting them up for rotation.
secrets place <stack>/<service> | --host <host> [--yes]	Write the secret files a declaration holds on a host, a file, a key in a file or a key in ~/.roostrc, from the record by name. Without --yes the plan prints.
secrets adopt <stack>/<service> | --host <host> [--yes]	Read the secret files a host holds into the record, for each declared holder the record has no value for, and never print one.
secrets record <stack>/<service> <key>	Take one issued value from standard input into the stack's file of rotated values, for a key a person issues.
vault login [--name <label>]	Sign this machine in to vault through the browser, and store the operator token.
vault status	Show who this machine is signed in to vault as.
vault logout	Remove this machine's vault token file.
vault requires <stack>/<service> [--dry-run]	Set on vault what a declared app requires of a person, from the service's vaultRequires.
forge seed <name>	Store a CI secret in vault's forge app. The value comes on standard input.
forge secrets <owner/repo>... [--name <secret>]... [--replace]	Set the forge's CI secrets from vault on repositories that do not have them.
forge prune <package>... [--keep <n>] [--protect <prefix>]... [--protect-deployed <dokku@host:app>]... [--box <target>] [--yes]	Delete old versions of forge container packages, and keep the newest and the protected ones.
state init --directory <dir> [--remote <owner/name>] [--push] [--dry-run]	Set up a state directory that encrypts its secrets and backs them up.
state status	Report the secrets that are not in the encrypted backup.
state seal	Encrypt the backup again, so it agrees with the disk.
state verify	Decrypt the backup and check it against the manifest.
state fill <stack> [--directory <dir>] [--yes]	Fill ports, deploy, config and checks into an imported tofu state from its own plan, with no apply.

The secrets and vault commands take --manifest more than one time, to read several manifests. The vault commands take --vault <url> to act on a vault other than the default.

Design notes

• declaration-source-design.md: the manifest and kind files become the one source, and hatchery writes every tofu file. Proposed on 2026-09-16, not started.
• exposure-design.md: exposure providers that make a cloned stack reachable. The none, platform and cloudflare-local providers are in the code.
• app-platform-pass.md: the full provider pass for DigitalOcean App Platform, and after it Cloud Run. The listed slices are shipped, and cloudflare-api custom domains wait.

Details

Cloning a stack

stack clone reads a stack's config — live from the box when the backend can answer, the declared sidecar otherwise, and it says which — and decides, key by key, what a copy into another environment should do: carry it, rewrite the names in it, mint a fresh one, or refuse and say why. Nothing that points at the source's database or grants the source's authority is ever copied. Optional keys the source sets ride along; unset ones aren't mentioned. Without --create it only reports; with --create --tofu-dir <dir> --yes it bootstraps the stack, scaffolds the services, layers the carried values on, and prints the config set commands for what still needs a person.

serve answers the same two steps at /api/stack/clone/plan and /api/jobs/clone. The plan carries every key's fate before anything is written, and the create runs the same bootstrap, scaffold and layering the CLI runs, then names the keys only a person can supply, as service new does. No secret value leaves serve. The plan carries the names and what happens to each one.

History and alerts

hatchery serve polls on its own clock (--watch-interval, default 30s), so watching does not depend on a browser tab being open. Every observed change — a service worsening, recovering, or degraded for a new reason — is appended to <manifest>.history.jsonl beside the manifest, printed on the serve console, and answered at /api/history. hatchery events reads the same file from the CLI. --alert-webhook <url> POSTs worsening transitions (only worsening; recovery is in the record but never wakes anyone) as JSON with a ready-made text line, so anything chat-shaped can consume it directly.

Finding the manifest

A manifest lives next to the config files it names, because configFile resolves relative to the manifest's own directory — which is rarely the directory you are standing in. So a command with no --manifest looks in three places, in order:

1. ./hatchery.json
2. $HATCHERY_MANIFEST
3. ~/.config/hatchery/hatchery.json

An explicitly passed --manifest is used exactly as given and never falls back, because second-guessing it would hide a typo. Finding nothing lists everywhere it looked.

From nothing to something

There was no path from an empty directory to a running service without writing HCL yourself — service new could only add to a configuration that already existed, and the lab's own was written by hand. stack new closes that:

hatchery stack new newlab --host dokku@192.168.0.103 --tofu-dir ~/infra-state/newlab
hatchery service new newlab paylab2 --kind payment-gateway \
  --domain paylab2.opi --image payment-gateway:arm64-6be9967
hatchery config set newlab/paylab2 DATABASE_HOST=... DATABASE_USER=... DATABASE_PASSWORD=... --yes

It writes versions.tf, providers.tf and variables.tf, runs tofu init, and records the stack in a manifest beside them. An existing configuration is never written over.

serve answers the same steps at /api/stacks/new and /api/services/new, and the coop's Stacks page starts a new app through them. hatchery serve does not need a manifest to start, because a manifest must not be needed to reach the tool that writes the first manifest.

A key hatchery cannot supply is left out of the config entirely rather than written as "". The dokku provider rejects a zero-length config value outright (string length must be at least 1), so an empty placeholder makes a brand-new stack fail to plan at the worst possible moment.

Serve is the engine, and the coop is the face

hatchery serve holds the control routes and draws no page. Jimmy ruled on 2026-10-22 that the coop is the face and serve is the engine (house#40, house#151). serve drew a dashboard of its own on the air's loopback, and a phone could not reach it. The coop's Stacks page now draws the whole declaration per service and forwards each act to serve with the serve token, as the rotate button has done since house#48.

GET / answers the API's index as JSON: every route, with one sentence on what it does. No route answers HTML.

A verb runs the same from a shell, from serve and from a job, and the frame test holds it: Tests/FrameTests runs every verb's read-only form as serve runs a child, under env -i with launchd's PATH, from /, with no --manifest (house#159).

The coop's acts each run as a serve job, and the page follows the job through GET /api/jobs?id=&from=:

Act	Route	What the job runs
deploy	POST /api/jobs/deploy {target, image?, confirm}	hatchery deploy <stack>/<service> --yes as a child of serve, by the road the kind file's delivery names. An image app needs an image ref, because a deploy with none is a tofu reconcile, and that is the apply.
config set	POST /api/jobs/config-set {target, key, value, confirm}	The write /api/config/set makes, into the declared config file, sealed again after. The lines name the key and never the value.
apply, the plan	POST /api/jobs/apply/plan {stack, confirm}	tofu plan -out into a private file, read back with tofu show -json. Each change is a line, and a create, a delete or a replace refuses the apply with the resource named.
apply, the run	POST /api/jobs/apply {stack, confirm, plan}	tofu apply of the plan file that plan job kept, once. tofu refuses the file if the state moved after the plan.
new app	POST /api/jobs/new {target, kind, confirm}	hatchery new <stack>/<name> --kind <scaffold> --yes, the short form, so the app is a spike.
rotate	POST /api/jobs/rotate {target, key, confirm}	hatchery secrets rotate --yes, then the ledger to pulse.

GET /api/jobs with no id answers every job since serve started, newest first, each as {id, act, target, state, startedAt}. The act is one of deploy, config-set, plan, apply, new, rotate, clone and destroy, and startedAt is milliseconds since the epoch. The list carries no line of any transcript, and serve keeps it in memory, so a restart of serve empties it (house#167).

A restart is not a serve job. The coop forwards it to pulse's /api/restart/<app>, which checks the declaration first (house#70). Every job route refuses a production stack, and the estate is dev.

The mark is a cracked egg on the same perch roost draws its rooster on — same 64×64 field, same flat geometry, same #C4602A. roost owns machines; hatchery owns the stacks that hatch on them, and the two marks are meant to read as siblings.

$ hatchery serve
hatchery serving http://127.0.0.1:7878
  bound to loopback — not reachable from the LAN
  from elsewhere: ssh -L 7878:localhost:7878 <this-host>

It does not run inside what it manages

hatchery manages deployments, so it deliberately is not one. Running it as an app on the box it administers means a restart of that stack kills the tool mid-action, and the moment you most need it — box wedged, apps down — is exactly when it would not be there. It is a local process. There is no Dockerfile here on purpose.

Getting a box ready in the first place

hatchery doctor tells you dokku is not answering; it cannot tell you how to put dokku there. hatchery setup is that half — installing dokku, authorizing your key for the dokku user, creating the shared docker network, provisioning the database role hatchery refuses to invent. serve answers the same list at /api/setup.

setup only prints the steps. hatchery box init runs them as assertions: each step is checked first, fixed only when it is missing, and checked again after. Without --yes it runs only the checks, so nothing changes on the box until you ask for it. serve does not run these steps, because they touch a machine's package manager, its firewall and its SSH configuration.

Prerequisites, checked before anything is written

hatchery doctor checks the things that otherwise surface halfway through tofu init — as a provider error that never mentions the missing SSH key that actually caused it:

  ok   tofu installed: OpenTofu v1.12.5
  ok   ssh client: OpenSSH_10.2p1, LibreSSL 3.3.6
  FAIL box reachable: ssh: connect to host 10.99.99.99 port 22: Operation timed out
       -> the box is not reachable from here; check you are on the same network
  --   dokku responds: the box could not be reached

One cause produces one failure: an unreachable box skips the dokku check rather than reporting a second thing to fix. serve's /api/preflight runs the same checks.

The SSH target is how hatchery reaches the box to create and manage apps, the user must be dokku — that account is what turns an SSH command into a dokku command — and your key must already be authorized for it.

Incomplete configuration

/api/stacks carries a config field for a service whose declared config is missing required keys, with the keys named.

It is separate from health on purpose. That service answers HTTP and reads responding, while its declaration could not boot if applied — a service can run happily on values its config file no longer contains. Health is about what is running; the triangle is about what is declared.

Computed from the declared file rather than the box, so it costs a local read on a ten-second poll instead of an SSH round trip per service. config audit is the one that asks the box.

Saved SSH targets

An address is the same thing in six places, and retyping it is how 192.168.0.103 becomes 192.168.0.130 in exactly one of them.

hatchery host add opi dokku@192.168.0.103
hatchery stack new lab --backend dokku --host @opi --tofu-dir ~/infra-state/lab
hatchery host list

The list is saved names, plus targets already in use by a stack, plus anything roost advertises. roost configures boxes and hatchery configures the stacks on them, so roost is the one that knows a machine exists — ~/.roostrc already declares ROOST_DOKKU_HOST, and hatchery reads it rather than making you type an address written down two directories away. Nothing had to change in roost for that to work. The file is read, never sourced: sourcing an rc file to learn one variable would run whatever else is in it.

Host telemetry belongs on the same seam and is deliberately not here. roost already ships a per-node reporter with launchd and systemd installers; a second set on the same box would be two answers to one question. The dokku guide points at roost's installer instead.

Tearing a stack down

hatchery stack rm <name>                        # shows the destroy plan, changes nothing
hatchery stack rm <name> --yes --confirm <name>  # actually destroys it

The name has to be repeated back, because this is the one action with no undo. The tofu directory and the config files are left on disk: they hold the state file and real secrets, and forgetting a declaration should not also delete the only record of what was there.

Looking at one service

/api/logs reads dokku logs, with the colour codes stripped and each line classified by severity. /api/config answers what the service runs with, each secret replaced by a fingerprint before the value leaves the process, beside the contract issues config audit reports.

Confirmation is enforced on the server

A dialog on a page that asks for a service's name is a convenience. It is not the control, because anything can post to the endpoint. So every mutating request carries the name of what it is about to change, and the server refuses a mismatch:

$ curl -XPOST localhost:7878/api/lifecycle -d '{"stack":"mwlab","service":"mwlab","action":"restart","confirm":"wrong"}'
{"error":"confirmation did not match; expected 'mwlab' to change it"}

Applying a deploy to a production stack is refused outright and stays on the CLI. The page is the wrong place for the one action with no undo.

Binding and tokens

The default bind is 127.0.0.1, which nothing off the machine can reach, so no token is needed to get started. Binding anything else requires --token and is refused without one — this process holds SSH access to every stack it manages, so an open port is an open shell by proxy.

hatchery serve                                   # loopback, no token
hatchery serve --bind 0.0.0.0 --token "$(openssl rand -base64 24)"

The token is compared without an early exit, so how long the check takes says nothing about how much of it matched.

service new authors a service that does not exist yet — the declaration, its image variable, its config, and its manifest entry:

  write paylab2.tf
  append to variables.tf
  write paylab2.config.json
    APP_URL                  composed
    DATABASE_DB              composed
    DATABASE_PASSWORD        NEEDS VALUE
    GATEWAY_ADMIN_TOKEN      minted (32 bytes)
    KEYPAIR_JWKS             minted (RSA-2048, RS512)

  3 key(s) need values before this will boot:
    DATABASE_PASSWORD — the database role already exists; a new password would not connect

Where secrets come from

Until now they came from nowhere: the values were set by hand on the box once, and everything since — pull-mwlab-config.sh, then config sync — copied them back down. That works for a service that already runs and has nothing to say about one that does not.

So each declared key gets an origin, and the distinction is the whole point:

Origin	Meaning
minted	hatchery generated it; nothing else has a claim on the value
shared	copied from a sibling, because the value has to match across the stack
composed	derived from what the manifest already declares
NEEDS VALUE	only a person or a third party has it

Three rules, each from the lab rather than from a preference. A signing key is shared when the stack already has one — the services verify each other's tokens, so a fresh key is a key nothing accepts — and minted only when there is no sibling to share with. Database credentials are never invented, because the role exists before the service does; a minted password produces a service that cannot connect. Third-party credentials are never invented, and saying so is more useful than a placeholder that looks filled in.

KEYPAIR_JWKS is minted as RSA-2048/RS512 rather than an elliptic curve, because that is what the live keys are. CryptoKit has no RSA, so hatchery shells out to openssl and reads the PKCS#1 structure directly — nine DER integers that are exactly a private RSA JWK's fields.

Not everything is an environment variable. Stripe credentials for the lab live in the database, so the origins above cover the config half of the problem and not the whole of it.

Rotating secrets, and the ledger

A secret is declared in its kind file with a rotation: who issues the value, and who holds it. hatchery turns it over in one order, the issuer, then every holder, then every restart, and the same declaration is what the ledger lists.

"NODE_KEY": {
  "secret": true,
  "class": "sharedKey",
  "rotation": {
    "issuer": { "type": "vaultSecret", "app": "roost", "name": "NODE_KEY" },
    "holders": [
      { "type": "dokkuConfig", "app": "pulse", "key": "NODE_KEY" },
      { "type": "file", "host": "jimmy@opi.jimmyhoughjr.net", "path": "~/.roost_node_key" }
    ]
  }
}

Issuer	Who mints the value
vaultAppKey	vault, a new app key for the service
vaultS3Key app	vault, a new S3 key for an app
vaultSecret app name	vault, a named secret in an app's document
postgresRole server role	a new password on a database role
vaultReseal	hatchery, and vault re-seals every app secrets document and S3 key under the value before any holder takes it
random bytes	hatchery, random bytes
manual recipe	a person, by the recipe; the rotation refuses and prints it

Holder	Where the value lands
dokkuConfig app key [restart]	an app's config on the box, set with --no-restart, then one ps:restart (rolling) or stopStart
roostrc host key	a key in a host's ~/.roostrc
file host path	one file on one host, over ssh or local
launchdEnvironment host label key, systemdEnvironment host unit key	a job's environment on a host, then the job restarts
vaultSecret app name	a named secret in an app's document, written by the run unless it is the issuer's own app and name, then the app restarts
gitCredential host forge user	the credential git's helper on the host holds for that user at the forge, approved from standard input; read at each push, nothing restarts
dockerLogin host registry user	docker's login on the host for that user at the registry, docker login --password-stdin; read at each pull, nothing restarts
dokkuRegistryLogin host registry user	dokku's own global registry login on its box, registry:login --global --password-stdin; read at each image deploy, nothing restarts

A value never goes into a command's arguments, because ps on the Mac and on the box shows every argument to every account, house#46. A dokkuConfig holder sends the key and value as JSON on the ssh connection's standard input to dokku --quiet config:import --format=json --no-restart <app> -, which merges it into the app's config. The import never restarts the app. The restart step restarts each app once, after every holder has the value, because on 2026-10-01 an import without --no-restart redeployed the coop for 85 seconds inside the hold step. A vaultSecret holder on another app or name takes the value through vault's admin route, in the request body, because on 2026-10-01 rookery restarted on its own unchanged NODE_KEY and its posts to pulse got 401. Every other holder runs a small script over ssh that reads the value from standard input into a mode 600 temp file, stops when the input is empty, sets the holder from the file, and removes the file when the shell exits. The postgresRole issuer sends its ALTER ROLE to psql over docker exec -i the same way. An error that echoes the value reaches the report with the value withheld.

A key another service turns over is declared as { "rotation": { "owner": "<stack>/<service>" } }, and its row points at the owner. A kind shared by several jobs declares a key once, and only the jobs whose config or secrets file carries it are planned, so a shared kind rotates once per issuer.

A key that nothing rotates is declared as { "rotation": { "none": "<reason>" } }, house#50. The reason is required, and a none beside an issuer, holders or an owner fails when the kind file loads. The ledger shows the reason as the row's next step, never puts the key up for rotation, and gives it no reminder. rotate and rotate --all skip the key and print the reason. The lab database passwords of mwstack-pg-dev and mwstack-pg-staging on the box declare it, because a person resets them only when the lab is rebuilt.

secrets rotate --all runs every declared rotation on every manifest. Before its first issuer it probes every host it will touch, and a silent host refuses the whole run, because a value minted in vault with no holder able to take it is the failure of 2026-09-27. Lines print as each step finishes. A person-issued key is refused with its recipe.

Every secret declares a class, ruled on 2026-09-30 in house#56. The class decides the check before anything is minted and the shape of the run, and the issuer still says who mints. A class and an issuer that do not fit fail when the kind file loads.

Class	Issuers that fit	Before the run	The run
token	vaultAppKey, vaultS3Key, manual	the issuer answers	mint, place, check the new value works, revoke the old
sharedKey	vaultSecret, random, manual	every holder's host answers	one value onto every holder, restart each
sealingKey	vaultReseal, manual	the issuer is vaultReseal, the operator token works, and every holder is running; a manual issuer is refused	mint, re-seal, every holder, every restart, then vault opens every document
password	postgresRole, manual	the database answers	the role change first, then every holder
address	manual	none	refused with the recipe; a person renames it on the device

A URL that always carries a password is a password, because the URL belongs to the password. A shared key whose receiver takes a list declares "list": true, and its value overlaps: the new value joins the list, every holder moves, then the old value leaves. Today only vault's kiosk tokens, upload tokens and S3 keys do. A secret with no class cannot run: the rotation refuses that row, nothing guesses a class from its name, the ledger shows the class as owed, and the rest of the run goes on.

vault's SESSION_SECRET is a sealingKey. vault seals every app's secrets document and every S3 key under a key derived from it, so a new value locks every document with no symptom until an app restarts, which is what happened on 2026-09-28. Its issuer is vaultReseal, house#45, and its holders are vault and pulse, both stopStart. The run mints a value and calls POST /api/admin/reseal, which opens every document under the old value, seals it under the new one and refuses while any one does not open. Then vault's and pulse's secrets files take the value, both configs take it with --no-restart, vault restarts, pulse restarts, and the run reads GET /api/admin/reseal until the restarted vault opens every document. A dokku holder's own secrets file takes a held value too, where that file already declares the key, so the next apply of that service does not put the old value back.

If the check fails, vault kept the old set in DATA_DIR/reseal-backups/<stamp>/, named in the report. Stop vault, copy that directory's files over DATA_DIR/authz/, set the old value back on vault and pulse and in both secrets files from the state backup sealed before the run, and start both. A check that fails on every file with vault still on the old value needs no rollback: set the new value from vault's secrets file on vault and restart it. A manual sealingKey issuer is still refused, and the ledger shows it as a re-seal with its recipe.

A host's own secret files are holders too: a file holder is the value whole, an envFile holder is a KEY=value line in a file a unit reads, and roostrc is that holder with the path fixed (house#121). secrets place --host <host> --yes writes every such holder the declaration names on the host from the record, which is the stack's rotated.secrets.json and each service's own secrets file, the same values secrets sync puts into vault's document; a blank box takes every file it holds in that one run, and nobody types a value. A holder the record has no value for is named and skipped: secrets adopt --host <host> --yes reads it from a box that still holds it, and pbpaste | secrets record <stack>/<service> KEY takes one a person issued. One kind file serves every job of a host stack, so a value recorded under any job of the stack is found. install reads each declared secret file as a row, present or missing, and names the place line for a host that lacks one.

secrets ledger lists every declared secret: its issuer, whether it is known to work, the day it was issued, its expiry, and what is owed next. The dates live in ~/.config/hatchery/ledger.json on the Mac that runs the rotations, names and dates and never a value. A key the ledger has never seen with a date shows the day it first saw it and is put up for rotation; a finished rotation stamps the day, and secrets issued <stack>/<service> KEY stamps a key a person turned over by hand. A held key shows its owner's day. A key no API can check, an Apple private key or a Google client secret, never reads live, and a typed expires on such a key is a review date, when a person looks at the issuer's console. --due <days> prints the reminders owed inside the window and exits 1 when any is, which is how the daily publish carries a reminder to the board.

secrets ledger --publish sends the same rows as one document to pulse's /api/ledger with the roost node key, beside the declaration, and the coop's Tokens page draws them with the owed tokens first. A row with a rotation of none has next set to none and carries the kind file's reason in noneReason. house-rotate publishes after every run, and the air runs it daily.

Installing what a host declares

A merge to hatchery, roost or the house skill changed nothing on a box until a hand ran an installer there, and five traps in the house skill traced to that gap. hatchery install closes it from the declaration: each declared job names the program it runs, and the program lives in a checkout, is a built binary, or is a copy of a file from a checkout. Each of those and the job's own plist or unit is one row per host.

$ hatchery install -m ~/infra-state/air/hatchery.json -m ~/infra-state/box/hatchery.json
  jimmy@opi.jimmyhoughjr.net
    THING               KIND      INSTALLED     WANTED        STATE
    roost               checkout  876998f       ab5fed7       behind
    dokku-reconcile.sh  copy      676807740da1  5e127985e63a  differs
    opi-backup          job       3f36742f535e  3f36742f535e  level

One script per host reads every fact in one round trip. A checkout is measured against the forge's main, read from this machine, so a clone whose origin is GitHub still answers for the forge. A checkout that holds the forge's main and more reads ahead, and the forge is what wants a push. A copy is measured against the file in its checkout on that host. A job file is measured against the rendering in its canonical form, plutil on a Mac and the unit stripped of comments on Linux, so the executor's tabs and a comment say nothing. A binary is measured by the sha bin/install writes beside it.

Without --yes the table prints and nothing changes. With it every host is probed first and a silent one refuses the run before anything is written; then a checkout fetches the forge's main and fast-forwards, a binary is rebuilt by its checkout's bin/install, a copy is placed by its tool's installer, and a missing job file is written and the supervisor reads it again. A copy's installer runs with HATCHERY_UNITS_DECLARED=1, so it places the file and writes no unit: the declaration owns the unit, and on 2026-10-13 the reconcile's installer wrote its three units in its own form, which read differs until --jobs. After a checkout moves, the rows after it are read again from the host, so a copy that follows the checkout installs in the same run. A dirty checkout is never touched. A job file that differs is rewritten only with --jobs, because a hand-written unit may hold what the declaration cannot yet say, such as an OnFailure line. --publish sends the report to pulse beside the declaration, where the coop's Estate page reads it.

Each host stack's first row is its declaration file, ~/.local/share/hatchery/declared.json (house#140). It is the document hatchery declared --placed prints: every stack of every manifest read, with no findings, and with at set to the newest manifest's write so the same manifests give the same bytes. The row compares the file's sha256 on the host with the document's, reads level, missing or differs, and with --yes writes the document from standard input and moves it into place. The boot job box-boot reads it at every boot, so the box needs neither pulse nor the forge to know its needs, and a change to a need in the manifests reads differs on the next install with nothing published.

The read also lists the job files on the host that carry hatchery's mark: a unit under ~/.config/systemd/user whose first line is # Written by hatchery., a plist under ~/Library/LaunchAgents that holds <!-- Written by hatchery. -->, and the boot order unit hatchery box order wrote on 2026-09-15, which is known by its Description=hatchery line. A marked file that no declaration names reads not declared, and with --yes it is disabled and stopped, or booted out on a Mac, and removed with any script under ~/.local/share/hatchery that its ExecStart names. A file with no mark is a person's or another tool's, and the verb never reads it this way.

The root door is the one thing the verb reads and never installs. A host stack that declares "door": "/usr/local/sbin/house" in its settings gets one door row per house-* script in the house repo's box directory on the forge's main, hashed on the host against the file root installed, and a script that differs or is missing is drift that stays: installing a root step widens what a session may do as root, so the report names the one command a person runs instead, house-door push and then ssh -t <host> "sudo bash ~/house-box/install-door". doorSource and doorInstaller in the same settings name another tool's directory and another installer (house#80).

A Python venv is a row too (house#122). A host stack that declares "venvs": "meshtastic router tapo" in its settings gets one venv row per name, after the roost checkout and before the jobs. Each name is a pinned list in roost, venvs/<name>.txt, and roost's bin/install-venv.py in the host's checkout does the reading: it answers what the venv holds and what the list wants as two short hashes of the pins, by name and version. A venv that holds its pins reads level, one that is not there reads missing, one that holds other than its list reads differs, and a checkout that has no recipe yet reads unknown. With --yes the recipe makes a missing venv and makes again one that differs, as a copy is placed again. A job that runs from a venv that was made again holds the old files until it restarts, and the run says so.

A built binary can come from the forge's package registry instead of a checkout on the host (house#132). A host stack declares it in its settings, as "fetched": "roost, statusgen=status@roost": each entry is <tool>, <tool>=<package> or <tool>=<package>@<repo>, and the package and the repo default to the tool. Each entry is one fetched row, placed at $HOME/.local/bin/<tool>, whether a job runs it or not. A job whose program is .local/bin/<tool> for a listed tool reads as that row and never as a binary built from a checkout, because a fetched tool is declared and never guessed from its path. The row reads the sha in <path>.sha against the forge's main of the repo, as a built binary does: level, behind, missing, or unknown when the forge has no such repo or the host is not of the platform the build is for.

A build is published to the forge's generic registry as generic/<package>/<sha>/<file>, where <sha> is the commit of the repo's main it was built from and <file> is <tool>-linux-arm64 for Linux and <tool>-macos-arm64 for macOS, with a <file>.sha256 beside it that holds the file's sha256 as its first word. With --yes the host fetches both from https://forgejo.jimmyhoughjr.net/api/packages/jimmy/generic/<package>/<sha>/<file> with curl and no token, checks the sha256, writes the file to <path>.fetching, makes it executable, signs it again with codesign --force --sign - on a Mac, moves it into place, writes <path>.sha, and runs <path> --version, whose first line the run prints. A 404 or a bad hash stops the row with the reason, and the old binary and its .sha stay as they were. macOS kills a copied binary whose signature the copy voided, so the signing is not optional there. ForgePackages.genericURL and ForgePackages.genericFile hold the URL and the file name for both sides.

Signing in to vault

Every value hatchery mints in vault goes through vault's admin routes, and the credential for those routes used to be the signed-in admin's browser cookie, copied out of devtools by hand and passed in VAULT_SESSION. hatchery vault login replaces that. It binds a free loopback port, opens vault's own browser sign-in, and vault redirects the operator token back to that port. The value never passes through an argument or the shell history, and no person reads it.

$ hatchery vault login
  sign in at:
    https://vault.jimmyhoughjr.net/auth/cli?port=51234&name=mini.local
  signed in to https://vault.jimmyhoughjr.net as jimmy@example.com
  the token is at /Users/you/.config/hatchery/vault/vault.jimmyhoughjr.net.token

The token lands in ~/.config/hatchery/vault/<vault host>.token, mode 600 in a directory of mode 700, written through the file system and never through a shell. One file per host, so a lab vault signs in beside the estate's and neither takes the other's place. The token is checked against vault before it is written, so a file on the machine is always a credential vault answered for.

Every admin call looks for a credential in one order:

Where	What the call carries
VAULT_OPERATOR_TOKEN in the environment	Authorization: Bearer <token>
the token file for the vault's host	Authorization: Bearer <token>
VAULT_SESSION in the environment	Cookie: vault_session=<session>

The variable comes first, so a one-off run against another identity needs no sign-out. The session stays last, so a person who already has one keeps working. A machine holding none of the three is told run: hatchery vault login, in place of the devtools recipe it used to print.

hatchery vault status asks vault who the credential belongs to rather than trusting the file. hatchery vault logout removes the file and calls no route. A token is revoked in vault's console, so a token that got loose stays loose until it is revoked there.

Keeping the backup current

The files holding those secrets cannot be committed, and cannot be gitignored and forgotten either. A state directory marked with .age-recipient keeps an encrypted copy — secrets.tar.age — produced by its own seal.sh, and that archive is what version control actually stores.

The mechanism was fine and the workflow was not. Sealing was a separate command a person had to remember after hatchery had already written the file, so a scaffolded stack left a freshly minted signing key on exactly one disk, and nothing said so for a day.

So hatchery seals for you. Any command that writes a secret — stack new, service new, config set, config sync, a deploy --apply that rewrites terraform.tfstate — re-seals the directory afterwards and says that it did. serve's config routes do the same.

$ hatchery state status
/Users/you/infra-state
  last sealed  2026-08-06T12:20:45Z
  ok           every secret on disk is in the backup

state status compares SHA-256 hashes against secrets.manifest, so it never decrypts and needs no age identity — cheap enough to run on every check. It reports three things: a secret that was never sealed, one that changed since it was, and one still in the archive after being removed from disk. hatchery state seal re-seals by hand when you want it.

Hatchery runs the directory's own seal.sh rather than reimplementing it. That script decides what counts as a secret and which key it encrypts to; a second implementation drifting from it is precisely how a file ends up believed-sealed and absent.

Sealing never fails the command that triggered it. The config write already succeeded, and reporting it as a failure would invite someone to write the value again — so a seal problem is reported as a warning beside the result.

Committing and pushing the sealed archive is still yours to run. The archive lands in the working tree; getting it off the machine is a git operation on your repository, and hatchery does not make those on your behalf.

Filling an imported state

An import never reads deploy, and reads ports and config only for the keys the state already knows. So every app in estate and sites planned four changes, and hatchery new refused both stacks (house#146). An apply is not the cure, because the dokku provider runs config:set, ports:set, git:from-image and ps:restart on each update.

hatchery state fill <stack> plans, and writes the plan's own after values for those four attributes into terraform.tfstate. It refuses the whole run when a resource plans a create, a delete, a replace, or a change to another attribute. It prints attribute names only, because config holds secrets. Without --yes it writes nothing. With --yes it keeps the old state as terraform.tfstate.backup-<time> and adds one to the serial. --directory plans and fills a private copy of the stack instead of the stack itself, for a rehearsal. Reseal the state directory after a fill.

A dokku declaration leaves GIT_REV out of its config, because dokku rewrites it on each deploy, and declares no config when the map is then empty. Change a stack's declarations to this form before the fill, or the state takes GIT_REV in. After the fill, tofu plan shows only the differences that are real on the box.

config audit checks what each service is running with rather than what a file claims:

mwlab  [dokku]
  mwlab: 21 keys, 0 error(s), 0 warning(s)
  paylab: 10 keys, 0 error(s), 0 warning(s)
  comlab: 9 keys, 0 error(s), 0 warning(s)

The distinction matters. dokku config:set merges rather than replaces, so a key set by hand never reaches the declared file and nothing reports the difference. The lab had exactly that: three keys were live on the app and absent from the file, and only reading the app found them.

status asks each service what it says about itself and composes the stack:

mwlab  [dokku]  responding
  mwlab                   responding   88ms    no health endpoint
  paylab                  responding   85ms    HTTP 200, no readiness report
  comlab                  responding   85ms    HTTP 200, no readiness report

The states run worst to best: unreachable, degraded, responding, ready. A stack reports the worst state among its services. responding means the service answered without a readiness report, which is what an older image looks like — better than silence, worse than an answer we can read. The exit code fails on unreachable or degraded only, so an image without a health route does not fail a script.

Two details that came from the lab rather than from a document. A dokku service is probed at the box with a Host header, because a lab vhost usually has no public DNS record. And the readiness path follows the estate convention rather than a path hatchery invented: /health for everything the shared microservice template generates, /healthz for gsx-gateway.

deploy moves a service to a new image through the declaration that owns it:

mwlab: mwlab_image
  mhehmsoth/mwserver2:arm64-0630f31-health -> mwserver2:arm64-abc1234
  manifest updated to mwserver2:arm64-abc1234
  tofu plan: changes pending
  ...
  nothing applied; re-run with --apply --yes

This is the part that most wants to be a shortcut and most must not be. The image is a tofu-declared attribute, so setting it at the box would leave the declaration stale and tofu plan permanently dirty — the same two-owners mistake config sync exists to undo. So deploy writes the tofu variable, runs tofu plan -detailed-exitcode, and shows what that would do. Nothing is applied unless --apply --yes says so.

Three things it will not do quietly:

• If the plan stops evaluating after the write, the variables file is put back and the failure is reported. A configuration that no longer parses blocks every other change to the stack, which is worse than an undeployed image.
• The manifest moves only after tofu agrees the write evaluates. Writing it first would leave the declaration claiming an image that never planned.
• If the manifest and the tofu variable already disagreed, it says so before overwriting — that means someone moved the image outside hatchery.

The edit to variables.tf is textual rather than a parse-and-reprint, because the comments in that file explain how each image was built and where it came from. Exactly one line changes.

Why this exists

A stack's configuration currently lives in at least five places that disagree with each other: a DO app-spec template, gitignored dokku config dumps, .auto.tfvars.json files, the DO console, and a password manager. Nothing reconciles them, and nothing tells you whether a deployed app is actually serving — only whether its last deploy succeeded.

hatchery config validate is the first answer to that. It encodes the environment contract per service and backend, and it already catches a real latent failure: the 2026-07-28 cutover retired the discrete DATABASE_HOST / DATABASE_PORT / … keys on App Platform, so a lab config promoted to production as-is would not boot.

Design

Layering

All logic lives in HatcheryKit. Frontends are thin, so everything of consequence is reachable by tests. A future macOS GUI is a view layer over the same types — it does not get its own logic.

State ownership

hatchery owns the declaration — a committable manifest recording which stacks should exist, which services they run, and where each service's config sidecar lives.

It deliberately does not own live state. What is actually running is queried from the backend on demand. The two are kept apart because the alternative has already bitten us: the mwlab OpenTofu state is local-only and unbacked-up, was once recovered from an ephemeral scratch directory, and losing it would orphan three running apps.

Secrets live in sidecar files that are never committed. The manifest records where config lives, never what it contains.

Backends

Backend	Target	Create	Read config back
dokku	Self-hosted — the opi box and the Pi fleet, over SSH	yes	yes
appPlatform	DigitalOcean App Platform — the production tenant plane	yes	no
aws	AWS App Runner — a container, a URL, a health check	yes	no
cloudRun	Google Cloud Run — App Runner's closest sibling	yes	no
host	The docker daemon on a box over SSH, for containers and jobs that dokku does not own	yes	yes

hatchery once reported that App Platform could not be created, on the grounds that a spec is YAML applied through doctl. That was wrong: digitalocean_app is a first-class resource whose spec.service block takes an image, an environment, a port and a health check. What is genuinely blocked is reading a spec back — every SECRET value returns as EV[...] ciphertext — so config audit and config sync refuse for the three cloud backends rather than reporting drift they cannot see. Creating and reading are separate questions and only the second has a no.

Fly.io was considered and deferred. Its tofu provider is fly-apps/fly v0.0.23 and offers fly_app, fly_machine and fly_ip with no service abstraction and no health-check block, so a service would have to be assembled from machines by hand — a materially worse fit than the three above. flyctl would be the better route if it is wanted.

Each backend answers for itself. A provider supplies its own settings, its own readiness check and its own setup guide, so hatchery doctor --backend aws asks about credentials and a region while --backend dokku asks about a reachable box and an authorized key — and neither has to know the other exists:

$ hatchery doctor --backend aws
  AWS App Runner
  ok   tofu installed: OpenTofu v1.12.5
  FAIL aws cli: not found on PATH
       -> brew install awscli
  --   aws credentials: the aws cli is not installed
  --   aws region: the aws cli is not installed

hatchery setup --backend <name> prints what that backend needs from nothing, including the settings it takes. serve answers the same guide at /api/setup?backend=<name>.

Backend settings

What a backend needs is declared by the backend, not asked for separately by the CLI and the API:

Backend	Settings
dokku	host, ssh_key, db_admin, exposure, exposure_admin
appPlatform	region, token*
aws	region, access_role_arn
cloudRun	project, region
host	host, ssh_key, platform

* secret: read from the environment at apply time, never written to the manifest.

hatchery stack new crlab --backend cloudRun --tofu-dir ~/infra-state/crlab \
  --set project=my-proj --set region=europe-west1

Before this, --host was a dokku concept every other backend ignored, a region was a bare parameter threaded through three call sites, and the page carried a needsHost flag so it could special-case one backend. Adding a provider meant editing all of them; now it means declaring an array. A missing required setting is refused by name, and so is a key the backend does not have:

$ hatchery stack new crlab --backend cloudRun --tofu-dir …
Error: cloudRun needs project; pass --set <key>=<value>, or run `hatchery setup --backend cloudRun`

$ hatchery stack new crlab --backend cloudRun --tofu-dir … --set projekt=typo
Error: cloudRun has no setting 'projekt'; it declares: project, region

Nothing here assumes a particular box. Which settings exist, whether one of them is an SSH target at all, and whether a value is stored or read from the environment are all the backend's answer to give.

App Runner rather than ECS or Fargate for the first AWS target, because hatchery's model of a service is an image, some environment, a domain and a health path — which is what App Runner takes. Fargate would need a VPC, subnets, a load balancer and target groups generated alongside every service, none of which hatchery has an opinion about. ECS can arrive as its own backend rather than as a flag on this one.

The AWS backend has not been applied against a live account. The generated configuration passes tofu validate against the real hashicorp/aws provider schema and tofu fmt -check, which means the resource shape and attribute names are right — not that a deploy succeeds. Treat the first real apply as the test.

The two are not interchangeable. App Platform dropped discrete DATABASE_* keys in the connection-string cutover; the lab still runs pre-cutover images, so the same keys remain legal there. EnvContract encodes that difference rather than pretending one contract fits both.

deploy writes tofu and does not talk to the box, so it works on every backend. Live config reads work on dokku and host only. up, down and restart work on dokku only, and they refuse the other backends by name.

They arrive as implementations behind a protocol rather than as branches inside each verb, so a new provider is a new file rather than an edit to seven of them.

Relationship to roost

roost owns machines. hatchery owns stacks.

roost handles the host plane: the box itself, dokku and node telemetry. It never learns what MWServer is.

hatchery handles the application plane: the stack definition, the environment contract, images, database wiring, lifecycle, application-level health.

hatchery publishes a URL's tunnel route itself, with hatchery route, because placing a name is hatching (house#128, house#155). roost route once did it through publish-route.sh, and on 2026-10-23 serve's PATH found the Swift resident as roost, which has no route verb, so the console's first new app stopped at its route. roost runs any roost-<cmd> on PATH as a plugin, so a roost-hatch link to hatchery makes roost hatch work. bin/install does not make that link.

Where they actually overlap today

That division reads cleanly and is not yet true in the code. Both tools drive the same box over the same SSH connection, and three commands do the same work:

	roost	hatchery
logs	ssh dokku@h logs <app> -n 200	ssh dokku@h logs <app> --num 200
restart	ssh dokku@h ps:restart <app>	the same command
read config	config:show	config:export --format json

Two front doors to one command is harmless. Config writes are not.

roost config <app> KEY=VALUE runs dokku config:set on the box. hatchery deliberately refuses to do that: it writes the declaration and lets tofu apply, so that tofu stays the single owner of what the app runs with. A value set through roost is therefore invisible to the declaration, and tofu plan starts describing something that is no longer true.

That is exactly the drift hatchery config audit and config sync exist to find — the lab had three such keys live on mwlab and absent from its file, which is what prompted those commands. So the two tools do not merely duplicate here; they disagree about who owns config, and roost wins silently on the box.

Until that is settled in code rather than in this file: prefer hatchery config set over roost config <app> K=V, and run hatchery config audit after any use of the latter.

What hatchery does not rebuild

MacWorkStack-infra already solves local dev (compose.yaml + db/init/) and tenant creation (new-tenant.sh — key generation, database and pool creation, spec render, validate, create, poll). hatchery shells out to those rather than redesigning them.

Two deploy behaviours it must respect:

• An env-only spec update reuses the already-resolved image digest and does not re-pull the tag. Only doctl apps create-deployment re-resolves.
• Verify a deployment by digest, never by tag.

Development

swift build
swift test