v2 · the machine that learned to test itself
Version 1 drew this same machine on 2026-07-21 and ended on three things it did not do: it never ran the app's own tests, one of its checks tested production and so ran the whole pipeline backwards, and its most sensitive alarm was wired to a cut cable. Twenty-four hours later all three are closed — and the work that closed them was set in motion by the very outage the machine had failed to catch. This is the same journey, re-measured on 2026-07-22: nine boxes became ten, the red dashed link went green, and the honest gaps got smaller. Click any box to open it.
Three acts, now ten stations. A change enters at box 1 and is in front of users after box 9; box 10 never stops running. The one link that pointed the wrong way in v1 now points forward.
Everything here happens before the change is allowed into the main line of the codebase, and every check here is now hermetic: it builds whatever it needs from scratch and touches nothing real. This is the cheap place to be wrong, and in v2 it is finally the place the app is tested too.
The only step on the whole page that no machine is watching. Everything downstream exists because this step is done by a human at 1am.
Either Swift code in the iOS app, or — for anything touching the database — a new migration file dropped into supabase/migrations/ with a timestamp in its name.
as long as it takes
Not measured, and not measurable. Nothing is automated here.
Nothing. This is the step where mistakes are made, not caught.
Nowhere yet. It shows up two or three boxes later, or — if it slips through — in front of a user.
migration — a numbered text file of instructions for the database, applied once, in order. NoonFox never edits a database by hand: it writes a file, and every copy of the database (the throwaway test one, staging, production) is brought to the same shape by applying the same files in the same order. The files are the source of truth; the databases are just the result.
Why the database is still the scary half
App code can be rolled back by shipping a new build. A database migration that drops a column has already destroyed the data by the time anyone notices. That asymmetry is unchanged in v2 — it is why the last two boxes still require a human to type a confirmation. What changed is that the app half is no longer un-tested: it just isn't the dangerous half.
A pull request is a proposal: "here is a bundle of changes, please let them into the main line." Nothing is merged yet. Opening it is what fires the starting gun for every automatic check.
The branch is pushed to GitHub and a pull request is opened against main. GitHub then looks at every workflow file in the repository and starts the ones that listen for pull_request.
seconds
The click itself. The waiting starts at box 3.
Nothing on its own — but it is the trigger without which none of the checks would ever run.
N/A.
Path filters — now they route to the app too
Each workflow declares which files it cares about. The database workflows only wake if something under supabase/** changed; the new app-test workflow (box 4) only wakes if FindIT2/**, the tests, or the project file changed. A pure database change no longer drags the app tests along, and a pure app change no longer drags the database battery along. Each change pays for exactly the checks it earns.
presubmit — a check that runs before a change is allowed in, not after. The opposite is finding out in production. In v1 one presubmit check quietly broke this promise by testing production; in v2 every check in Act 1 is genuinely presubmit again.
Several workflows fire at the same moment and run concurrently, so the wait is the length of the longest one, not the sum. The big change from v1: the check that talked to production is gone, and two new hermetic checks took its place — one of which reproduces the exact outage that started all of this. Today the longest is about 137 seconds. Any single red one blocks the merge button.
≈ 137 s
The hermetic PostgREST canary, now the slowest gate. The replay job is ≈ 116 s alongside it.
Broken schemas, un-replayable migrations, security rules that leak, permission errors on the app's real write path, drifting config, unregistered source files, unpinned dependencies.
A live database. Every check here builds what it needs from scratch. Nothing here can be made flaky by someone else's half-finished experiment.
A red ✗ next to the pull request, plus an email from GitHub. Never an email when it goes green again — see where errors show up.
The checks — open each one
.github/workflows/edge-schema-refs.yml · job "check" · median 116 s
This was the conceptual centrepiece of v1, and it still is — but it grew a limb. In v1 the hermetic replay proved the schema and ran 33 in-database invariant tests. In v2 the SQL contract suite runs here too, against the same throwaway database. That suite used to live in the check that talked to production; moving it onto the replay is half of how the backwards link got fixed.
hermetic — sealed off from the outside world. The job spins up a brand-new, empty Postgres inside the CI machine, replays every migration file in order from 000_baseline.sql, runs its tests against that, then destroys the whole thing. No staging, no production, no credentials, no network. Run it a thousand times, get the same answer a thousand times.
What runs inside the 116 seconds
pgTAP — a testing framework that runs inside the database itself. Tests are written in SQL and report pass/fail like any suite. RLS — row-level security: the database's own rules about which rows each signed-in user may see or change. Many of these tests exist purely to prove one user cannot read another user's photos.
A real catch, from v1's own day
A powerful new database function, reachable by the app, was caught from the files alone in under two minutes, before it reached staging, production, or a user. That argument for the hermetic approach is exactly why the SQL contracts were moved to live beside it.
.github/workflows/edge-schema-refs.yml · job "api-contracts-hermetic" · median 137 s · added #781
This check did not exist in v1, and it is the direct answer to the outage that opens this whole story. In July a database permission error broke photo capture for four days without a single automated check going red, because no test spoke to the database the way the real app does — through the full web layer, with a real signed-in user, doing a real write.
Why the replay check could not catch it
The schema replay (check 1) talks to Postgres directly. The bug lived one layer up: in how the app's data-sync connector phrases its write, the database quietly needed a permission it had been denied, and only that exact phrasing, through the web interface triggered the error. A direct database test can be perfectly green while the real path is broken. So this check boots the entire local stack — the web interface, the sign-in service, the database — creates a genuine test user, and sends the connector's byte-for-byte write request.
Proven by making it fail on purpose
Before this check was trusted, its guard was deliberately sabotaged on a scratch branch — the exact permission was revoked — to confirm the check turns red with the real outage's fingerprint. It did, naming the precise error code:
A healthy run instead reports a harmless 23502 (a not-null complaint about the deliberately empty test row) — proof the write reached the database and was permitted, just empty. The difference between those two codes is the difference between v1's four-day outage and a two-minute red check.
42501 vs 23502 — two Postgres error codes that tell opposite stories. 42501 is "you are not allowed to do that" — the outage. 23502 is "you're allowed, but you left a required field blank" — exactly what a deliberately-empty canary write should get. The canary passes on the second and screams on the first.
.github/workflows/supabase-tests.yml · one job now, not two
In v1 this workflow had two jobs and one of them — the SQL contract suite that phoned production — was the broken one. That job has been removed entirely. What remains is the good half: a dependency-pinning guard, storage-path guards, a check that four separate copies of the same text-normalising function still agree byte for byte, an AI-spend cap, and the health-check rules' own unit tests. All hermetic.
.github/workflows/xcode-project-sync.yml · ≈ 22 s · 7 jobs in parallel
Unchanged from v1, and still quietly excellent. One job answers a boringly specific question that has cost real days: is every new Swift file actually listed in the Xcode project file? A file on disk but not registered simply never compiles — the build is green and the code does nothing. The other six are shrink-only ratchets: each counts a known bad pattern and refuses to let the count go up. Debt can be repaid, never taken on.
Sign-in settings — password rules, token lifetimes, which providers are on — live in a dashboard where they can be changed with a mouse and no record. "Drift" is the gap between what the repository claims and what is actually set. This check measures that gap. It runs both on pull requests and on a schedule, because drift isn't caused by pull requests — it is caused by somebody clicking something on a Tuesday. In v2 it has company on the schedule: a second, broader drift sweep now watches the database schema the same way (box 10).
This is the box that closes v1's first honest gap. Yesterday the sentence was: "the app's 2,396 tests never run in CI, not once." Today they run on every pull request that touches the app, and a green merge is no longer possible while one of them is red.
A dedicated Mac — NoonFox's own, not a rented one — checks out the branch, regenerates the Xcode project, and runs the full 2,396-test unit suite in the iOS Simulator.
.github/workflows/unit-tests-macos.yml
≈ 81 s
A warm run. Cold, with nothing cached, ≈ 189 s. It runs alongside box 3, so it adds nothing to the wait.
Every regression in the app's own logic that a unit test covers — the entire body of code people actually hold in their hands, which in v1 no automated check touched.
A red ✗ on the pull request, exactly like the database checks.
self-hosted runner — a build machine NoonFox owns, rather than one rented by the minute from GitHub. Apple builds can only run on Macs, and GitHub's rented Macs cost roughly ten times a Linux minute. At the rate this project merges, a dedicated Mac mini pays for itself many times over — so the runner is a real machine on a real desk, labelled findit2-mini.
The three things that make a self-hosted runner trustworthy
v1 drew a red dashed line between box 3 and box 4 and explained, at length, why the arrow sometimes pointed backwards. Here is how it was straightened — and the small, elegant idea underneath it.
The flaw, restated
One presubmit check asked its questions of the live production database. A pull request adding a new column asked production about a column production did not have, so the check stayed red until production already had the change. Production therefore got the change first, and the merge followed — backwards from the order every other box assumes.
The fix is a shape, not a patch — and the shape is: ask the same question in three correct places
The contract suite wasn't wrong to exist. It was in the wrong place. So the one prod-touching check became three:
Nothing in this act is automatic. There is still no auto-deploy to production anywhere in this system, by design — a bad migration applied automatically to production is an outage, and that has happened once. What is new: the last box now runs the full live contract suite against whatever was just deployed.
The pull request is merged. From this moment the change is part of the official history, and main is what every environment is supposed to be brought into line with.
GitHub merges the branch. Note what does not happen: nothing is deployed. No database is touched. Merging changes the files, not the world.
a click
Only possible once every check from boxes 3 and 4 is green.
It is the enforcement point, not a check: the green checks are what make the button pressable.
GitHub refuses the merge and says which check is red.
Staging is a second, complete copy of the backend that no user ever touches. It exists so the first time a migration meets a real, running system, the stakes are zero.
A person opens the deploy: migration workflow and dispatches it by hand, filling in three inputs: the environment (staging), the path to the migration file, and — only for production — a typed confirmation.
.github/workflows/deploy-migration.yml
≈ 21 s
The machine part. The waiting-for-a-human part is unbounded.
A migration that applies cleanly to an empty test database but not to a real one that already holds data.
A red job on the workflow run.
workflow_dispatch — "only starts when a human presses the button." There is still no code path anywhere in the repository that deploys a migration on its own.
The steps it runs, identically for both environments
Staging and production get byte-identical treatment, which is the point: whatever staging survived, production will experience the same way.
The hermetic checks in Act 1 already proved the schema is valid. This step exists for the things a throwaway database physically cannot prove, because it has no world around it.
1. That the sync engine replicates the new shape correctly to phones. 2. That the server-side functions deploy. 3. A real client sync round-trip — a device sending and receiving actual data.
by hand
A person looking at a running system. Not a workflow, not measured.
Integration failures: the schema is fine, and yet the pieces don't fit together once they're live.
In front of the person doing it — which is exactly why it must not be skipped.
Correctness before, integration after — the division that shapes everything
Correctness — does the schema apply, do the security rules hold, does the app's write get permitted — is now proven entirely before merge, on hermetic throwaways, where the answer is deterministic. Integration — do the live parts still talk to each other — is proven after merge, on staging. v2 sharpened this line: the SQL contracts that used to blur it (correctness questions asked of a live system) are back on the hermetic side of it.
The same workflow as box 6, aimed at the real database that real people's photos live in. The only differences are one dropdown value and one word typed by hand.
The same dispatch form, with environment = prod and a third input where the operator must type PROD in capitals. Then the identical apply → ledger → verify → contracts sequence.
≈ 21 s
Same workflow, same measured cost as staging.
The typed confirmation catches the only failure mode left at this point: a human who is moving too fast.
A red job on the run — and, if the migration half-succeeded, the new contract step catches it before anyone leaves the page.
Still no automatic promotion, still on purpose
A database migration cannot be rolled back the way a build can — once a column is dropped, the data is gone. So the human dispatch is the promotion gate. There is no "auto-promote to production" switch to find, because none was ever built.
Three things now confirm the change landed: the deploy's own read-back, the full live SQL contract suite run against the environment just deployed to, and a canary that behaves like a real user. The middle one is where v1's prod-touching check went to live correctly.
The read-back confirms the expected shape. The contract suite then asks the just-deployed environment every contract question — the very questions that used to (wrongly) gate the merge. The canary fires after every successful deploy.
automatic
Part of the same run, plus the canary immediately after.
A migration that reported success but left the environment violating a contract — caught here, against that environment, blocking the deploy rather than a pull request.
A red job on the deploy run; a red canary run.
Why "after deploy" is the right home for a live check
A live contract test asks: "is this running environment in the shape the contracts require?" That is a perfectly good question — it is just not a question about a pull request, which has no running environment of its own. It is a question about a deploy. So it now runs where a deploy can answer it, blocking the deploy if the answer is no. Same test, same assertions; the only thing that changed is that it can no longer hold a merge hostage to production's current state.
canary — named after the bird taken down coal mines. This one signs in as a genuine test user and sends the app's exact write request to every synced table. Non-destructive: every field is sent as NULL, so the database can check whether the write would be permitted without any row ever being stored. It is the live twin of the hermetic canary in box 3 — same request, real environment.
Everything above is triggered by a change. These run on a clock whether anything shipped or not. v2 adds a fifth GitHub watcher — a nightly schema drift sweep — and, more importantly, finally proved that the most sensitive alarm here can wake somebody up.
The only jobs in the system with no human waiting on the result, which is what lets them be slow. There are still two entirely separate clocks — that distinction is unchanged from v1, and still matters more than any single job. What changed: GitHub's clock gained a schema drift sweep, and the database clock's alarm was finally proven to reach a person.
Clock 1 · GitHub Actions — now 5 jobs, all times UTC
backup-nightly · ≈ 294 s · 02:30
Copies the database out to Cloudflare R2, on completely different infrastructure. Still the longest-running job in the system, and the one whose failure would matter most in the way that is hardest to notice.
powersync-health · ≈ 54 s · 05:20
Checks the sync service is genuinely alive — because a dead sync instance can keep serving stale cached data and look healthy from the outside for days.
drift-watch · ≈ 180 s · 05:40 · NEW
The nightly schema sweep — the third and final home of v1's misplaced contract check. It does two things: runs the full live SQL contract suite against production, and dumps the schema of production, staging, and a fresh migration replay and compares all three. If the deployed schema has drifted from the files — a column added by hand, a migration merged but never deployed — it files a GitHub issue with the exact diff.
That is a watcher earning its keep on its first real week: it noticed that a data-integrity rule lived in the files and on staging but had never been promoted to production, and it said so without anyone asking. Boring for months, loud once — this was the "once".
auth-config-drift · ≈ 23 s · 06:00
The sign-in-settings drift check from box 3, on a timer as well — because drift is caused by someone clicking a dashboard, which produces no pull request to check.
canary: authenticated write · 06:15 + after every deploy + manual
Signs in as a real test user, sends the app's exact write to every synced table, confirms none is rejected. The live twin of the hermetic canary in box 3; non-destructive, every field NULL.
Clock 2 · pg_cron — 12 jobs inside the database
Scheduled by the database, run by the database, invisible to anyone watching the repository. Most are housekeeping. One is the alarm — and in v1 that alarm was the third honest gap.
v1's third gap was that this alarm had never once been proven to reach a person. In v2 it was proven the only way that counts: by firing it. A synthetic alert was raised on staging, and following it end to end surfaced three separate silent breaks — a report that was captured but never flushed to the alerting service before the machine that captured it shut down; a monitor whose check-in address had been wrong for fifteen days; and a staging alert channel that had never delivered a single event. All three were fixed, and the re-fired alarm arrived — in a real inbox.
The rule that governs this now: no detector counts as coverage until a test-fire proves it reaches a human.
Re-measured in seconds on 2026-07-22. Bars are to the same scale, so shapes are comparable at a glance. The pull-request checks run at the same time, so what you wait for is the longest bar among them — not their sum.
The pull-request wait grew from about 114 seconds to about 137 — the price of the new hermetic canary that reproduces the outage. For that 23 extra seconds, the app is now tested (in parallel, adding nothing to the wait) and the exact four-day outage class is caught before merge. And the single most expensive line in the whole pipeline is still not a test: it is the 88 seconds spent booting an empty Postgres from nothing.
Still four places, but the map has shifted in NoonFox's favour: the outage class moved from "the app is broken in someone's hand" up to "a red check on a pull request", and the database alarm now escapes the database.
The normal channel for every pull-request check — and the channel now caught two things it never used to: the app's own regressions (box 4) and the database-permission outage class (the hermetic canary, box 3).
The gotcha still worth knowing: GitHub emails when a check fails, not when it goes green again — so the last email always points at the old failed run. The reliable answer is gh pr checks <PR>, never the newest email.
If deploy: migration fails — on staging or production — it shows as a red job on that run. Since a human dispatched it and is watching, this failure always has an audience. In v2 it fails on one more thing: a violated live contract, caught by the new verify step.
The app reports its own distress to Sentry: dead-letter events (a change that could not be sent and was set aside) and queue-wedge events. In v1 the most sensitive of these was, in effect, shouting into a disconnected line.
Fixed in v2: the report is now flushed to Sentry before the reporting machine shuts down, and the delivery was proven end to end into a real inbox.
pipeline_health_check runs inside the database four times an hour, writes findings to a pipeline_alerts table, and raises alerts in Sentry. Its most sensitive rule trips on a single client-side dead letter.
What changed: in v1 that alarm had never once been proven to reach a person, and a root-row data-loss discard was filed at the same low severity as routine noise. Both are fixed — the severity is raised for unrecoverable losses, and the alarm was test-fired all the way to a human. See the closed gaps →
drift-watch (box 10) adds a fifth place a problem can surface: a GitHub issue, filed automatically overnight, when the deployed schema drifts from the files. It caught a real one on its first week — a data-integrity rule that had reached staging and the codebase but never production.
This is the section v1 existed to earn. It named three things the machine did not do; here is each one, shut, with the honest note that the work was set off by the outage the machine had failed to catch. A shorter list of smaller, still-true gaps follows — because a diagram that only shows the fixed parts manufactures confidence just as surely as one that hides the broken parts.
v1 said: "There is no xcodebuild workflow at all in the repository. The 2,396 Swift unit tests run only on a developer's own Mac, when that developer chooses to run them. Nothing runs them automatically on a pull request."
how it shut
A dedicated Mac — findit2-mini, a machine NoonFox owns rather than rents — now runs the full 2,396-test suite on every pull request that touches the app, in about 81 seconds, in parallel with the database checks so it adds nothing to the wait. A floor of 2,000 makes a zero-test vacuous pass impossible. Box 4 on the diagram is this gap, filled.
The self-hosted Mac was a deliberate cost choice: Apple builds only run on Macs, GitHub's rented Macs cost roughly ten times a Linux minute, and at this project's merge rate a Mac mini on a desk pays for itself many times over.
v1 said: "The SQL contract suite sends its questions to the live production database. A new migration's test cannot pass until production already has that migration — so production gets the change before the merge that was supposed to authorise it."
how it shut
The one prod-touching check was split into three correctly-placed ones: the contracts run hermetically before merge (box 3), against the just-deployed environment after deploy (box 9), and on a nightly drift sweep against production (box 10). The merge gate no longer depends on production's current state, so it is a real gate again — the mended link in Act 1.
The underlying principle, now enforced: a merge gate must be a pure function of the change in front of it. A live-environment question is a fine question — it just belongs to a deploy or a schedule, never a pull request.
v1 said: "The mirror that copies client-side dead letters to the server was broken… during a four-day total outage it never fired once… and even with a signal, the rule fires at WARN, not CRITICAL. No detector counts as coverage until a test-fire proves it reaches a human."
how it shut
The broken mirror was repaired, and the severity was raised so that an unrecoverable root-row data loss pages loudly instead of whispering at the same level as routine noise. Then the alarm was test-fired end to end — which is the only proof that counts, and which promptly exposed three more silent breaks in the delivery path: a captured report never flushed before its machine shut down, a monitor check-in address wrong for fifteen days, and a staging channel that had never delivered a single event. All three fixed; the re-fired alarm landed in a real inbox.
The honest coda: v1 predicted this alarm was broken. It was more broken than v1 knew — the diagram found one cut cable; test-firing found four. That is the argument for test-firing every detector, made in one incident.
Two clocks, no shared screen. GitHub Actions and the database's own scheduler still share no dashboard, no alerting path, no failure history. Seventeen scheduled jobs, two places to look, and nothing that shows them together. This is now the largest gap on the board.
Nine functions still capture without flushing. The flush fix that saved the alarm's own reports has not yet been applied to the other nine server-side functions; their error reports are best-effort in the same way the alarm's used to be. Tracked, to be fixed as each is next deployed.
One migration guard is queued, not built. A ratchet that would refuse a destructive migration lacking an explicit acknowledgement is designed but not yet shipped. Until it is, "don't drop a column by accident" rests on review, not on a machine.
Every term used above, defined once, in the order you would meet them. The five new to v2 are marked.
Re-measured 2026-07-22. Durations are medians of recent runs unless marked otherwise; where a number was not captured, this page says so rather than estimating. v1 was measured 2026-07-21 — one day, three gaps, closed.