4 · Who said what¶
Turning audio into text is basically solved. Turning it into who said what is less obvious, and it is where the queryable data actually lives. Some tools get attribution for free: Teams and Zoom can lean on the structure they already have, per-speaker channels and per-account logins, the same move as using the structure the feed hands you one chapter back. It gets harder the moment it is a single mixed stream: a podcast, a panel, a deposition, which is most audio worth querying. And attribution sits upstream of almost every step that produces value, so an error here propagates. Every derived fact, count and timeline inherits it.
The obvious way to solve a hard AI problem is to point the biggest model at it: a frontier LLM over every few seconds of speech. That works, but it is wasteful when a simpler model does the same job, and here one largely can. The archive is big. This one show is 352 episodes, ~475 hours, roughly 20 days of continuous audio, and you re-run all of it every time you change a prompt. Cost scaling with the hours you ingest is true of any approach, so the lever is not raw power but matching each sub-problem to the cheapest model that solves it. The extra engineering that takes amortises over every re-run, precisely because the cost is proportional to the hours processed.
Breaking the problem into manageable parts¶
Recovering speaker structure from a single stream is really three jobs, and matching each to the right tool is the whole economic argument in miniature:
- Diarization decides when each distinct voice is active, and how many
distinct voices there are, without knowing any identities. This is the
deep-learning part, and it is one I lean on as a black box rather than open
up:
pyannote1 runs neural models over the audio and clusters the result into speaker turns. Overall it does an excellent job, but it can miscount, merging two similar voices into one or splitting one voice into several, and that miscount becomes a failure mode further down. - Identification decides whose voice each turn is. Because we own the roster, a closed set of recurring hosts rather than open-world speaker search, a classical per-speaker Gaussian Mixture Model over MFCC features is enough: cheap, CPU-only, no training infrastructure. It is also simple enough to take a closer look at, which I do a bit further down.
- LLM refinement cleans up the residue with a small local model2, the handful of turns the first two stages leave uncertain, using the surrounding dialogue. It is reserved as a scalpel, not pointed at the whole transcript.
Deep learning where it pays for itself, decades-old classical ML where it does not, and an LLM only on the leftovers. Identification could in principle do the diarization too, scoring every segment against every host and cutting where the winner changes, but running the two independently and asking them to agree buys a confidence a single model would not. Whether the GMM alone matches the combination is something I never measured, and it is an optimisation for later.
These stages stack on each other's output, so most of the engineering is defending against the previous stage's failures, not its happy path, and none of those failures show up in a demo. Four stories from this stage, working downstream from the raw audio toward the graph: a transcription loop, the identifier and its two bugs, the scalpel for phantom speakers, and the check that proves the result.
The loop: a symptom in the graph, a cause in the transcript¶
The clue turned up downstream, in the graph: one statement repeated over and over, thirteen identical copies carrying thirteen separate timestamps. Tracing it back upstream, the transcript had the same sentence thirteen times in a row:
[00:14:02 → 00:14:03] Und das ist genau der Punkt.
[00:14:03 → 00:14:04] Und das ist genau der Punkt.
[00:14:04 → 00:14:05] Und das ist genau der Punkt.
… (10 more)
Whisper had not misheard anything. It had fallen into a decoder loop: it predicts text token by token against the audio, and on a thin or ambiguous stretch it can lock onto a phrase and re-emit it once per timestamp tick. The sentence was spoken once, so that first copy is real. The twelve that follow are hallucinated. Left alone they become twelve fake facts downstream, each with its own timestamp.
The fix lives at the source, in transcript assembly, and it is unglamorous: collapse runs of consecutive, byte-identical segments, keep the first, and stretch its end over the absorbed copies so the span stays continuous for diarization overlap.
def dedupe_repetitions(text_data):
out = []
for entry in text_data:
text = entry["text"].strip()
if out and text and out[-1]["text"].strip() == text:
out[-1]["timestamp"][1] = entry["timestamp"][1] # absorb the duplicate
else:
out.append(entry)
return out
Exact-match only, no fuzzy collapse, so genuine repetition ("nein, nein, nein") survives. Across the corpus this removed 5,853 duplicate segments, 449 distinct loops across ~180 episodes. The lesson is one that recurs through this stage: a symptom you notice downstream often originates several steps upstream, and the cheapest fix lives at the source, not where you first saw it.
The GMM: classical ML, and two bugs left over from 2024¶
Identification does not need deep learning here. A Gaussian Mixture Model is a compact statistical portrait of a voice. Take a few minutes of someone speaking and slide a 25-millisecond window along it in 10-millisecond steps. Because the windows overlap, that comes to about a hundred frames a second:
Reduce each frame to MFCC features (a short summary of its spectral shape) and fit a handful of Gaussians to that cloud of features. Each recurring host gets one such model. To label a turn, score its frames against every host's model and take the best fit.
Does it actually tell two people apart? Score a single 25-millisecond frame against both hosts' models and it is right only about three times in four, unreliable on its own. But each model's score is a per-frame average, so a longer window tightens both voices' score distributions around their true centres without moving them, until the two no longer overlap. By a full turn of a few seconds they are cleanly separated and identification is essentially perfect. Same score, more frames:
That is the whole reason a decades-old technique is enough here: no single frame is decisive, but averaging over a turn is. Making it correct and fast cost two ordinary bugs, both leftovers from the 2024 prototype. One of the quieter risks of shelving a project for two years is that its mistakes sit and wait for you. Fortunately both were easy fixes once found.
Correctness, a silent library API. GaussianMixture.score returns the mean
per-frame log-likelihood, not the total. Code that assumed a sum was silently
length-biased, so longer clips scored "better." The fix is to use the mean
deliberately: for one clip scored across every model, mean log-likelihood is
exactly the comparable quantity.
lls = [m.score(fv) for m in models] # mean per-frame LL: length-invariant,
i = int(np.argmax(lls)) # comparable across models for a fixed clip
Performance, seeking in the wrong place. The original step spawned one
ffmpeg per diarization turn with output seeking, which decodes the whole
episode from the top and throws it away, once per clip, hundreds of times per
episode. Decode each episode once and slice it in memory instead. That alone
took identification from minutes per episode to ~3.1 s.
The scalpel: an LLM on the turns nothing cheaper can resolve¶
First, how a turn even reaches the LLM. A diarization label is treated as
unreliable when it is thin or undecided: too few turns to trust, or an acoustic
vote with no clear winner, where the frame-by-frame scores split roughly evenly
instead of favouring one host. pyannote also sometimes carves a short
backchannel, a laugh or an "mhm," into its own phantom cluster, and the GMM
cannot identify a sub-second clip: the likelihoods are a near-tie and can even
land on a host who is not in the episode at all. In one Hannah-and-Jens episode
a one-second cluster was confidently labelled "Frederik Banning," who was never
there. Those are exactly the turns handed to the LLM.
What matters is what it is shown. Take a real case. Diarization split one of
Marco's own sentences clean in half, and the acoustic model labelled the
orphaned second half "Ulrich," the co-host, on a near-tie guess. This is the
window the LLM sees, with every uncertain label hidden behind ?, including the
target itself (>>>):
Marco: … Es war ja eine, ja, so eine Whistleblower-
>>> ? : Whistleblower-Geschichte, die aber irgendwie von der …
wie heißt denn diese Frau? Ich kann es mir nie merken.
Marco: Jedenfalls hat die sich sehr auf den Whistleblower
konzentriert und daraus dann ihre Story hergeleitet …
Read the dialogue and the answer is obvious: the ? line finishes Marco's own
sentence. He trails off on "so eine Whistleblower-" and picks straight back up
with "Whistleblower-Geschichte." The LLM reassigns it to Marco.
The masking is what makes that work. An earlier version left the acoustic guess
visible, >>> Ulrich: Whistleblower-Geschichte …, and the model simply agreed
with it and rubber-stamped the wrong label. Hiding it forces the model to reason
from the conversation, who is mid-sentence and who answers whom, instead of
parroting a guess. The answer is snapped back to the known roster, with a
fallback to the nearest reliable neighbour. The LLM touches a handful of turns
per episode, not thousands.
The verifier: proving the fix, from the feed¶
The podcast's RSS description names the cast in prose, "In dieser Folge diskutieren Hannah und Ulrich …", a human-authored signal the pipeline already has. A guard extracts those first names and cross-checks them against the speakers actually attributed, running both before and after the scalpel stage.
not_named = sorted(detected - named) # a voice attributed to someone the
if not_named: # feed description never mentions
problems.append(("detected but not named", not_named))
On one episode the raw acoustic map carried a phantom Ulrich Voss, a single
near-tie turn the description never mentions, so the guard flagged it. After
refinement reassigned that turn, the same check came back clean and reported
resolved by refinement. Only a discrepancy that survives refinement exits
non-zero and blocks the build.
It is the same move as the metadata consistency check: a second, independent source, here the human-written description, played off against the pipeline's own output, so their agreement is what lets you trust an acoustic guess rather than take it on faith. Next: turning those attributed turns into a graph of facts, not chatter.
-
pyannote.audio, an open-source neural speaker-diarization toolkit. Further reading, pretrained models and the papers behind them: https://github.com/pyannote/pyannote-audio. ↩ -
The local model here is gemma-3-12b, the 4-bit build
mlx-community/gemma-3-12b-it-4bit, run on-device via Apple's MLX. ↩