7 · Ideas are cheap¶
By now the mechanisms are in place: entities and relations extracted from the dialogue, then resolved down to one node per real-world thing. This chapter collects the failure modes, the places where building those mechanisms went wrong in instructive ways, and where instinct sent me somewhere the evidence didn't.
Two patterns recur. The first: the fix you're certain about barely moves the number, while a setting you configured once and forgot decides the whole result. The second: the symptom shows up a layer away from the bug, so the obvious reading is the wrong one. Both are cheap mistakes to make, and the best insurance against a confident wrong intuition is to measure.
A note on scale first. If ideas are cheap, the testing has to be cheap too, or you never see the end of it. Re-running extraction over ~475 hours of audio for every hunch would be far too slow to iterate on, so the numbers here come from the 12-episode proof-of-concept subset: 20 hand-picked factual chunks, small enough to turn a change around in minutes.
What follows is a collection of quite typical debugging stories, the intent is to highlight that in practice, building up such a pipeline is rarely ever a plug and play type of deal for input data that isn't trivial or comes already nicely structured.
Two ideas that sounded decisive¶
The relation-extraction step hands a small local model a chunk of diarized dialogue and asks it to extract the factual relations in it, who did what to whom. Two things looked like they would obviously decide how well it worked: how I formatted the transcript for the model, and how I worded the extraction prompt. Both are the kind of idea that's cheap to have and sounds plainly correct.
The thorough option to evaluate these ideas is expensive: read all twenty chunks and check every extracted relation against the transcript by hand. Raw relation count is the cheap substitute, and a trap - a careless configuration emits more relations, most of them spurious, so counting everything rewards the wrong behaviour. So a larger model does the reading instead: each relation is checked against the transcript by the 30-billion-parameter Qwen3-Coder, judging what the 12-billion Gemma extracted, and two numbers come out of it - correct facts, the count it confirms, and precision, the share of everything extracted that survives. Automating the judgement does not shut off inspection: every relation, its verdict, and the judge's own reasoning are written to disk, so any figure below can still be opened and checked by hand.
Reformat the transcript. Diarization produces an indented, script-like layout,
one Speaker: block per turn, long turns wrapped across lines. Tidying that into
clean single-line text, I assumed, would help the model concentrate on the content.
The idea wasn't idle: un-wrapping had already fixed a real bug one stage earlier,
where the entity extractor grabbed newlines mid-name and produced mangled
duplicates (later in this chapter).
If dropping the wrapping helped there, surely it helped here.
Measured across the 20 chunks (the wrapped production layout, one line per turn, and a fully flattened run-on line):
| transcript format | correct facts | precision |
|---|---|---|
| wrapped (production) | 131 | 71% |
| one line per turn | 130 | 79% |
| flattened run-on | 120 | 75% |
The layout barely moves it. Wrapped and one-line-per-turn are a dead heat; fully flattening costs ~8% of correct facts, so the turn boundaries carry a little signal and the indentation carries none. A real effect, but a small one, and not the make-or-break I'd expected.
Ask for more. The second idea: make the prompt exhaustive, instructing the model to find a relation for every entity it can, rather than the terser production wording. More instruction, more coverage.
This one genuinely helps. At a fixed reply budget, the exhaustive prompt recovers more correct facts at the same precision:
| relation prompt | correct facts / chunk |
|---|---|
| current (production) | 6.2 |
| exhaustive | 7.7 |
So both cheap ideas were real, one worth about a fact per chunk, the other about one-and-a-half, but modest, and neither is the thing that decides whether extraction works at all.
The factor that decided it¶
That thing was the reply's token budget. Extraction sets max_tokens to 2048,
one line that reads like defensive boilerplate, a guard against a runaway model, the
kind you write once and stop seeing. It turned out to be load-bearing. When an
earlier experiment dropped it, bundled in with the two changes I was actually
testing, the server fell back to its own much smaller default. Long replies were
then cut off mid-object, and because a JSON reply is valid right up to the cut,
nothing raised an error: the parser simply failed and the relations silently
vanished. I read the collapse as one of the changes I meant to be testing doing the
damage. It was the missing cap.
That single setting is the difference between "extraction is broken" and "extraction is fine." With no cap, the highest-volume configuration fails to parse 90% of the time and recovers almost nothing; with a working cap it recovers a full set:
| reply budget | correct facts / chunk | parse-fail |
|---|---|---|
| capped (2048) | 8.1 | 0% |
| capped (4096) | 8.1 | 0% |
| no cap (server default) | 0.5 | 90% |
Best configuration (wrapped + exhaustive), 20 chunks.
Two things are worth drawing out. First, the cap dwarfs the two ideas I'd been fussing over: it swings the result several times more than the transcript format or the prompt wording do:
Second, having found the lever, I checked the other direction too, doubling the cap from 2048 to 4096 changes the correct-fact count by nothing. The setting sits on a plateau, well past the point where it constrains the output, so the only failure mode is having no cap at all. That's not leaving a potential open loop: once a knob looks decisive, nudge it further and confirm the result stops moving, so you know you measured a plateau and not the near side of a cliff.
And the swallowing was a decision, but one I had to remember I'd made before the bug made any sense. On the 2024 cloud version the extractor was GPT-4o, which in those days would occasionally return not-quite-valid JSON. So I'd told the pipeline to catch an unparseable reply, skip that chunk, and move on. Nothing was cached yet, so a skipped chunk was just that run's small loss. For this kind of experiment it was a solid trade-off, a rare dropped chunk across 475 hours being noise rather than something to fail a whole run over. That held while the failures stayed rare and random. But a budget set too low does not fail randomly. It truncates the same long chunks every time, so the tolerance I'd written for the occasional blip was now quietly absorbing a systematic hole. A broken extraction came back looking like a number I could tune. The repair was to teach the code to tell a parse failure apart from a genuinely empty result, and to record the failure rather than store it as nothing.
The token cap fooled me by hiding. The next three bugs fooled me the other way, by pointing at the wrong layer. Same discipline settles both: distrust the obvious reading, and find the cheapest test that shows where the fault actually is.
Orphans were a wrapping bug, not a relation gap¶
228 entities landed in the graph with no edges at all. The obvious reading, that the NER model simply finds entities the LLM never relates, was only half right. 134 of the 228 were mangled duplicates: GLiNER2 had run on the line-wrapped chunk rendering, so a span grabbed the newline and indentation in the middle of a name and produced a broken twin of an entity that was already connected under its clean spelling. Cleaning names at the Neo4j boundary, collapsing whitespace and rejoining words split across a wrap, merged each twin back into its real node. That left a genuine remainder of roughly 94 truly unconnected entities, which get a mention with provenance rather than an invented relation, so no entity is left stranded by a formatting artefact.
This is the same wrapping quirk that made "just flatten the transcript" look like an obvious win one stage over, at relation extraction. There, unlike here, it turned out not to matter.
Fuzzy-snapping the endpoints¶
This is the seam the two-model split leaves behind,
and it is the inverse of the pattern that runs through the rest of this project.
Usually it is two independent methods agreeing that lets you trust a fact. Here the
two methods disagree about the very same thing. The NER extractor has already
settled an entity's canonical name, and the relation model, reading freely, writes
that same entity's name slightly differently as a relation endpoint: a declension
(heimische versus heimischen), an extra qualifier, a difference in casing. An
exact-match join treats the two spellings as two different nodes and silently drops
the fact.
The fix reconciles the two from the most confident match to the least: an exact
match, then a case-insensitive one, then whole-word containment, and only as a last
resort a high-cutoff fuzzy match, with word-boundary guards so that a short name like
Land is never absorbed into a longer one like Deutschland that merely contains
it. Held to that order, the snapping recovered ~145 relations (+6%) across 148
snaps without introducing false joins.
Don't assume the model, verify it¶
Nothing here actually lied. The relation cache keys each result on the model name I configured, on the reasonable-sounding assumption that the server at that address was serving that model. It wasn't. A rebuild had defaulted its base URL to the wrong port, so cache-miss chunks were quietly extracted by Qwen and written under the Gemma key. The symptom was a quality cliff: twice as many relations per chunk, 11% vague predicates against Gemma's 1%, and the statement count ballooning from 2,014 to 2,536, all of it noise.
The server could have told me which model it was serving the whole time. I just
never asked. The fix is a one-time GET /v1/models check at startup that fails
loudly when the served model isn't the one configured. The lesson generalises past
this one bug: don't assume a remote service is what you set it to, confirm it. (On
the honest path the same cache is a real win: a warm rebuild of the entity pass
drops from 3 minutes 11 seconds to 4.8 seconds, about 40×, and deleting before
writing keeps rebuilds idempotent, after Neo4j had drifted to 2,879 statements
against the 1,954 in the clean cache.)
Where the effort belongs¶
Strip these episodes down and one rule is left standing: spend effort where it moves the number, and distrust the reading that comes for free. With the reply budget set right, a small local model is robust to how you format its input and responds only mildly to prompt wording, so the token ceiling, the entity set, and the write-time filters are worth tuning; the input layout I reached for first is not. And when a metric moves the wrong way, the cause is rarely where the symptom sits: missing relations were a text-wrapping bug, a quality cliff was a wrong port, dropped facts were an exact-match join. Defaults are how you get a system working quickly, and most are reasonable, but working and performing are different bars, and the default that's load-bearing for your case never announces itself: the system runs, the output looks fine, and it's quietly wrong.
The same instinct-versus-evidence lesson runs through the resolution work one chapter back, with the sign flipped: there, reaching for a bigger model to fix over-merging made it worse. The full extraction setup (every rendering, every judge verdict, and the prompt and cap sweeps) is kept as a standalone, re-runnable experiment.