The goal-state architecture for the Shmili audio pipeline: audit first, then an explicit planner that decides which findings are fixed together and which need their own iteration, then loops that converge and stop. Written after the 78-episode sweep, which exposed what the old shape could not answer.
Every element below is tagged built (runs today), partial (exists but incomplete), or proposed (designed, not yet built).
The design is mostly about which actor owns which decision. Everything else follows from that.
Denoise, repair, levels, encode, detection, scoring. Reproducible and free. A script cannot have an opinion that changes on Tuesday.
Judgment that needs reading: routing to advisors, spotting sound-design cues, adjudicating flags, building review pages.
Taste, and every irreversible act. Outranks all meters by rule, not by courtesy. Publishing is his alone.
The old shape ran production first and audited the finished master. That made one question unanswerable: was this defect in the recording, or did we cause it? Auditing the raw capture at ingest turns every later score into a delta from a known baseline.
┌────────────────────────────────────────────────────────────────────┐
│ 0 INGEST new recording lands (Voice Memos → mini) │
└──────────────────────────────┬─────────────────────────────────────┘
▼
TRANSCRIBE (S) whisper -mc 0 → timestamped transcript
│
┌─────────────────┴──────────────────┐
▼ ▼
╔═════════════════════════╗ ╔══════════════════════════════╗
║ AUDIT-0 · BASELINE ║ ║ RUBRIC B · STORYTELLING ║
║ Rubric A on the RAW ║ ║ runs ONCE, here, forever ║
║ capture ║ ║ ║
║ "what does this ║ ║ describes a performance ║
║ recording ALREADY ║ ║ that cannot be re-performed ║
║ have, before we ║ ║ → feeds SELECTION and the ║
║ touch anything?" ║ ║ trim conversation ║
║ ║ ║ → NEVER a to-do list ║
╚════════════╤════════════╝ ╚══════════════════════════════╝
│
│ ◀── THE BASELINE. every later score is a DELTA from here.
▼
FIX PLANNER what needs doing, and what may be done together [Fig 3]
▼
╔══════════════ REPAIR + VERIFY LOOP ══════════════╗ [Fig 4]
╚═════════════════════════╤════════════════════════╝
▼
PRODUCTION STAGES — every mutation gated by a re-audit:
trim ──▶[AUDIT]──▶ sound design ──▶[AUDIT]──▶ opening+master ──▶[AUDIT]
(A) (L+S) (S)
│ │ │
└── the delta is ATTRIBUTED to the stage that caused it ──┘
▼
REVIEW PAGE (L) ──▶ AMIT LISTENS (A) ──▶ notes ──▶ back into the loop
▼
PUBLISH (A) ◀── the only irreversible act, and it stays his
This part runs today and is the most battle-tested piece of the system. The order is not negotiable: each stage assumes the previous one ran.
-original.wav ──── the untouched iPhone capture ─────────────────┐
(never pruned: it is BOTH the donor and the reference) │
│
denoise() restore_floor.py identity_patch.py│
┌──────────────┐ ┌───────────────────┐ ┌──────────────────┐│
│ SUBTRACTIVE │───▶│ holes ← room tone │───▶│ corr < 0.55 → │◀┘
│ ONLY │ │ from the SAME │ │ splice the │
│ DFN3 @ 12 dB │ │ recording │ │ ORIGINAL back in │
└──────────────┘ └───────────────────┘ └────────┬─────────┘
generative = banned a pause may never sit │
24 dB deletes words >6 dB below room floor ▼
repair_dropouts.py
(deleted-speech guard)
│
▼
-clean.wav
An audit produces many findings at once. Some belong in one pass; some must be their own iteration. Deciding that explicitly, before touching audio, is the step the old pipeline lacked — and its absence is why episode 1 took five repair rounds, with rounds three onward trading one artifact for another.
Timestamp overlap is only one of four axes:
| axis | what it decides |
|---|---|
| SCOPE | global (touches every sample) vs local (a window). A global op can never share a pass with local patches — it changes the ground the local detectors measure against. |
| OVERLAP | do the sample ranges collide, including crossfade tails? Two patches 40 ms apart with 30 ms ramps are not independent — they are one edit. |
| TOOL | same tool = one pass by design. debump.py batches its windows internally and its 2% selection guard only means anything per-pass. |
| GATE | auto vs human. A cough that might be an in-breath is not a scheduling problem, it is a decision. It never enters an automatic batch. |
findings[] (t_start, t_end, class, tool, scope, gate, severity)
│
▼
┌─────────────────────────────────────────────────────────────┐
│ 1 SPLIT BY GATE │
│ auto ──▶ planner human ──▶ approval queue (Amit) │
└────────────────────────┬────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────────┐
│ 2 SPLIT BY SCOPE │
│ GLOBAL ──▶ ordered first, one per pass, re-DETECT after │
│ LOCAL ──▶ continue to clustering │
└────────────────────────┬────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────────┐
│ 3 CLUSTER LOCAL FINDINGS ON THE TIMELINE │
│ guard = 2 × crossfade (60 ms room-tone, 40 ms band-cut) │
│ ├── disjoint by > guard ──▶ INDEPENDENT │
│ └── overlap or < guard ──▶ MERGE into ONE edit │
│ spanning both, with ONE crossfade pair │
└────────────────────────┬────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────────┐
│ 4 PACK INTO BATCHES │
│ all INDEPENDENT clusters → one batch → ONE rebuild │
│ (the expensive unit is verification, not the edit) │
└────────────────────────┬────────────────────────────────────┘
▼
plan.json
batch 0: [global: denoise] → re-detect after
batch 1: [global: EQ + loudnorm] → re-detect after
batch 2: [17 knocks, 3 holes, 1 dup] ← all disjoint
queue : [1 cough @5:31 — may be a breath → Amit]
A rebuild costs ~19 seconds; its verification costs minutes (the Whisper word-diff runs at ~5.6× realtime). So batch everything provably independent — then recover attribution by bisection when a batch fails.
apply batch (21 fixes) ──▶ re-audit
│
┌───────┴────────┐
▼ ▼
PASS FAIL
│ │
keep all split in half, re-run
1 verify │
┌────┴────┐
▼ ▼
PASS FAIL ──▶ split again …
culprit isolated in log₂(n): ~5 verifications for 21 fixes, not 21
Every repair is a splice. These five rules make “did the stitching go wrong?” a checkable assertion instead of a listening question.
| 1 LENGTH | Every edit is sample-count neutral. No cue, VTT offset or timestamp ever moves. |
| 2 ONE RAMP | 20 ms raised-cosine for band-cut windows, 30 ms for room-tone patches. A merged edit gets one ramp pair, never one per finding. |
| 3 DONOR | Fill comes from the SAME recording; identity-patch fill passes mild_denoise first (raw fill was heard as “רעש סטטי”). |
| 4 SEAM | Sample step at every seam < 0.05, and no seam lands inside a word. |
| 5 UNTOUCHED | Every sample outside the planned ranges is bit-identical, or explained by exactly one global gain. This is the strongest rule — it is an assertion that either holds or does not. |
A full audit after every batch is waste. Four tiers, each answering a different question:
| tier | answers | cost | when |
|---|---|---|---|
| T0 FULL | what IS this recording? the baseline | heavy — both rubrics, whisper, NISQA, LTASS | once, at ingest |
| T1 SCOPED | did this batch do what it claimed and break nothing? | cheap — touched classes + the 5 stitch invariants | after every batch |
| T2 REDETECT | where are the defects NOW? | medium — full detection, no scoring | after any global op |
| T3 DELIVERED | is the shipped file conformant? | small — TP, loudness, encode, the join | after the final encode |
T2 exists because of a lesson we paid for: after a global change the detectors' thresholds refer to a file that no longer exists. That is how repair_dropouts.py flagged 72 phantom regions on an Auphonic master and dragged its floor from 26.8 to 19.8 dB.
┌──────────────┐
│ T1 SCOPED │
│ after batch │
└──────┬───────┘
▼
┌──────────────────────────────────────────────┐
│ five questions, in this order │
├──────────────────────────────────────────────┤
│ 1 batch FAILED? ──▶ BISECT │ diagnostic
│ 2 a GLOBAL op ran? ──▶ T2 redetect│ mandatory
│ 3 the fix CREATED findings? ──▶ new batch │ mandatory
│ 4 human queue returned? ──▶ new batch │ unlocked
│ 5 Amit heard something? ──▶ new batch │ outranks all
└───────────────┬──────────────────────────────┘
│ all five = no
▼
══ CONVERGED ══ ──▶ T3 on the delivered mp3 ──▶ ship
round 1 ──▶ round 2 ──▶ round 3 ──▶ ✋ STOP. Offer the choice.
DO NOT attempt round 4.
the signature to recognise: verification says CLEAN
and Amit's ear says NOT FIXED
↓
the LOCALISATION is wrong, not his ear.
the honest move is the residuals register,
not another round. (t01, rounds 3–5)
Question 3 is the one people forget: repairs create findings. Floor restoration leaves micro-gaps behind (t01 went 229 holes → 46 → 4 across rounds) and every splice creates a seam that did not exist. A clean batch means “re-look”, not “done”.
Typical episode: one T0, one or two T2, two or three T1, one T3 — five to seven audit runs, of which exactly one is expensive.
Persist the facts; recompute the opinions. This is what makes rubric revision cheap and what keeps old numbers comparable.
audit run ──▶ measurements.json (durable, rubric-agnostic:
LUFS, TP, hole census, floor percentiles,
NCC pairs, F0 bands, NISQA windows …)
│
▼
score(measurements, rubric@v1.3) ──▶ 89.2
│
rubric → v1.4 │ (archive percentiles replace house numbers)
▼
score(measurements, rubric@v1.4) ──▶ 91.5
◀── NO AUDIO RE-AUDITED. Pure recomputation.
L1 PER FINDING audit → fix → re-audit minutes
└─ stop rule at 3 rounds, then it becomes Amit's call
L2 PER STAGE every mutation gated, delta attributed per episode
└─ turns "which stage injured this file?" from
archaeology into a lookup
L3 PER ARCHIVE the ledger rewrites the rubric per sweep
├─ Tier B house numbers ──▶ archive percentiles
│ micro-gap 4.43/min ──▶ 1.75/min (the old bar was
│ the DAMAGED archive's median)
│ pause floor [-75,-55] ──▶ [-62,-52], plus a second
│ band for the noisy-room family
└─ new metrics must EARN gate status:
report-only → monotonicity → retrospective →
anti-sterility guard → Amit's A/B → GATE
Named honestly, because a diagram that only shows what exists is a sales brochure.
A sidecar per delivered file: source, stages, parameters, script versions, score, rubric version. The entire t74 investigation was three hours answering a question a manifest makes a lookup.
Nothing protects against a pipeline change breaking what worked. In two days we swapped the denoiser, rewrote thresholds and changed the opener with no such check. t01, t41, t74 are the natural goldens.
78 masters are about to be swapped on a live third-party feed with no defined undo beyond per-file backup convention.
Transcript yield is a scored sub-check and it is what search engines index, yet transcripts regenerate ad hoc. They deserve their own track and gate.
The architecture ends at PUBLISH, but the archive exists for the kids. There is no route for “Hili said the helicopter part sounds wrong” to become a timestamped finding.
The last gate before publishing says “a minimum total score, set by Amit”. It has never been set.
Two smaller rules worth adopting: content-addressed caching everywhere (the pattern already exists — OPENING_TRIM_DB is baked into the cached opener's filename so a changed value regenerates instead of silently reusing a stale asset), and a Rubric B variant for scripted episodes, since the live Amit-and-Hili recordings break its assumption that the show is improvised.
| today | goal state | |
|---|---|---|
| audit runs | once, on the finished master | at ingest, then at every mutation |
| a score means | absolute; mixes our damage with the room | a delta from the recording's own baseline |
| storytelling rubric | never run, on any episode | once at ingest; feeds selection, never a fix list |
| fix granularity | whole-episode rebuild | per finding, planned into batches |
| “which stage broke it” | archaeology | attributed automatically |
| rubric numbers | engineering judgement | archive percentiles, self-updating |
| coverage | ~52% of the technical rubric | ~90% + the listening set |