2 · Going local¶
Capable local models were one of the big reasons to pick this project back up again - but the change that really carries the weight is where the computation happens. Running it without the cloud wasn't realistic for me in 2024. In 2026 the whole thing runs on my regular laptop, and no data leaves the building.
If you're in the EU, this may matter to you. With the regulatory landscape shifting, keeping audio and transcripts on hardware you control means privacy, sovereignty and GDPR stop being blockers, because nothing is sent to a third party in the first place.
Moving the pieces¶
Going local was largely a stage-for-stage substitution: a cloud model swapped for a small specialised local one at each step. Only the final Neo4j load was already local in the 2024 version.
| stage | 2024 (cloud) | 2026 (local) |
|---|---|---|
| transcription | whisper-large-v3 on dedicated AWS instances leaves |
mlx-whisper (large-v3) stays |
| entity extraction | GPT-4o via LLMGraphTransformer leaves |
GLiNER2-large stays |
| relation extraction | GPT-4o leaves | gemma-3-12b-it-4bit (MLX) stays |
| storage & cache | S3, feeding a local Neo4j leaves stays | local disk + Neo4j stays |
leaves = data leaves the building · stays = stays on the workstation.
Back in 2024 the work was spread across whatever machine was convenient - Whisper ran on dedicated AWS instances, for example - and S3 was simply the shared store that let me run a job anywhere and still have the data to hand without extra plumbing. Neo4j itself already ran locally, fed from S3. So going local mostly meant pulling the compute-heavy stages back onto one workstation.
Why now, and not in 2024¶
Nothing about the approach changed. The tools underneath it did. The 2024 pipeline leaned on GPT-4o for extraction - a frontier-grade API by the standards of that year - and there was no local model in the same class available with the hardware I had at the time, nor a runtime to serve one well. Two things closed that gap: small instruct models that are genuinely capable at extraction, good enough to stand in for that API on this job, and Apple's MLX runtime, which makes running them on a workstation fast rather than a science project. By 2026 the local models are more powerful and the hardware to run them is more available, which is what makes this worth a serious try now.
Some things don't change¶
Whichever model you reach for, there is serious engineering between raw output and a pipeline you would trust - and that is most of the rest of this case study. A better model moves the starting line, but it does not remove the work. The output still has to be steered with prompts and filtered, and mistranscribed names still have to be resolved before the graph holds together - the work of chapters 5 through 7. Nothing about the model you pick spares you that. "Local" was an architecture decision that still had to be earned on quality.
What local is actually for¶
Not speed, and not a smaller bill: caching works against an API just as well, and the
~25× jump from running GLiNER2 on the Apple MPS backend (~0.05 s per chunk) instead
of the CPU (~1.4 s) is a GPU story, not a local one. What you actually get is ownership of
the whole stack, and it shows up in two ways.
Reproducibility. Pinned to a specific model sitting on your own disk, you can re-run the exact same extraction months later and get the exact same result, where a hosted API can quietly change or retire the model under you.
A cost you can see. When you feel the compute first-hand, in fan noise and wall-clock time, you build sensible caching and leaner pipelines in from the start, instead of learning it from a surprise bill. That cost is easy to forget exists while someone else carries it - today's AI subscriptions are heavily subsidised. In 2025 OpenAI ran an operating loss of around $21 billion on $13 billion of revenue, spending far more to serve users than it charges them. Local strips that cushion away and puts the real number in front of you, which turns out to help more than it hurts.
None of this makes the loop slow: a cold 12-episode re-extraction is ~80 minutes and a warm rebuild is under a minute, comfortably fast on one workstation - and that cheap loop is what funds the measure-change-remeasure work the later chapters lean on.