Murmel v5: small, fast Dutch speech recognition that stays quiet when nobody speaks

Play Whisper large-v3 turbo a recording of rain, traffic or a humming fridge, and two out of three times it writes something anyway. "TV Gelderland 2021". "Bedankt voor het kijken!". "ZANG EN MUZIEK", repeated dozens of times. On the same speech-free clips, our new 128M-parameter model writes something on 4% of them, and nothing that looks like a subtitle credit.

That is one of the things we worked on for version 5 of murmel-flash, the family of speech recognition models behind murmel.nl. This piece covers what changed, how the models do on public Dutch benchmarks next to OpenAI, NVIDIA and Mistral, and what comes next: open weights for the smallest model are coming soon.

Built from scratch, for Dutch

murmel-flash is not a fine-tuned Whisper. It is a compact architecture of our own, built in plain PyTorch and trained from scratch on Dutch speech, with English as a second language. That means we can shape it around what Dutch users actually say: names, compounds, numbers, meetings that drift from one topic to the next, and people who talk over each other.

Version 5 comes in two sizes. murmel-flash-m-v5 has 128 million parameters, small enough for a laptop CPU. murmel-flash-l-v5 has 397 million and is the more accurate one. For comparison: Whisper large-v3 has 1.55 billion.

How it does on open Dutch benchmarks

We compared seven models on four commonly used open Dutch test sets: FLEURS, Common Voice 17, Multilingual LibriSpeech and VoxPopuli, about 15,800 clips in total. Every model gets exactly the same audio, one clip at a time, on the same GPU, scored with the same text normalisation, each with its own recommended decoding and no external language model. The benchmark code will be published together with the open model, so anyone can rerun it.

Word error rate in %, lower is better. Normalised: lowercase, no punctuation, numbers written out, fillers removed.

In short:

• murmel-flash-m-v5 makes 18% fewer errors than Whisper large-v3, while being 12 times smaller and 5 times faster.
• Against Whisper large-v3 turbo it makes 30% fewer errors, at a sixth of the size.
• It matches Mistral's Voxtral Mini, a model 37 times its size.
• murmel-flash-l-v5 makes the fewest errors of all seven: 33% fewer than Whisper large-v3 at a quarter of its size, and 10% fewer than NVIDIA's Canary at less than half its size.
• On debate speech (VoxPopuli), murmel-flash makes less than half the errors of Whisper large-v3.

Every model that makes fewer errors than murmel-flash-m-v5 is at least three times larger.

To be fair to the competition: on clean, read-aloud sentences (FLEURS and Common Voice), Whisper and Canary are still better than our small model. The difference shows where people speak more freely. On VoxPopuli, recordings of debates in the European Parliament, Whisper large-v3 gets almost one word in four wrong; murmel-flash gets one in ten.

Silence should stay silent

The most annoying failure of speech recognition is not a misheard word. It is text that appears out of nowhere. On music, noise or silence, Whisper often falls back on what subtitles tend to contain: credits, thank-yous for watching, the name of a regional broadcaster.

We tested this directly: speech-free clips of traffic, rain, kitchens, crowds, machines, hum and silence, none of which the models had seen before.

What murmel-flash still writes on those clips is short: a "ja", a "dank u wel". Two things got us here. The decoder is refereed by a second, faster head that has to agree there is speech before any text comes out. And in version 5 the model was trained on hours of noise with empty transcripts, so it learned that nothing said means nothing written.

Numbers the way you would type them

Phone numbers, postcodes, bank account numbers and amounts are where transcripts usually go wrong, and where an error actually costs something. murmel-flash writes spoken numbers in the format you would type yourself:

On our own test set of real recordings with spoken numbers, murmel-flash-m-v5 gets 11.5% of words wrong and the larger model under 10%, against 14.9% for Whisper large-v3.

Thank you for your input

A big thank you to everyone who uses Murmel and tells us where it goes wrong. Your error reports, your questions, and the recordings that users of murmel.nl chose to share with us show us exactly where the model still stumbles. Every version is better because of it.

Version 4 was a big step: it learned to handle all sorts of audio, from meetings and phone calls to interviews, lectures and broadcasts. Version 5 is the next step. After the main training, we went looking for the words the model still got wrong, such as names, loanwords, compounds and rare words, wrote new, varied training sentences around them, and trained the model specifically on those. Keep the feedback coming: it decides what we work on next.

Coming soon: open weights for the small model

Open weights for murmel-flash-m-v5 are coming soon, free for non-commercial use. We will also share the evaluation code later, so anyone can rerun the benchmarks. Follow us to stay up to date.

Want to use murmel-flash commercially, in a product, a service or your organisation's own workflow, or need one of the larger, more accurate models self-hosted? Reach out to us at murmel@the-ai-factory.com and we will find a licence that fits.

The larger models stay behind murmel.nl and the hosted API, running on servers in the Netherlands. And we are already training the next, larger generation.

Murmel v5 in short: available now on murmel.nl

Murmel v5 is live today for everyone who uses murmel.nl. No new account or settings needed: every new transcription runs on the new model.

• What it is: murmel-flash v5, a Dutch speech recognition (speech to text) model by The AI Factory, trained from scratch on Dutch speech, with English as a second language.
• Where to use it: on murmel.nl, to transcribe recordings of meetings, interviews, phone calls, lectures and podcasts, with speaker recognition, and to make subtitles for video. Developers can use the same model through the Murmel API.
• Privacy: audio is processed on servers in the Netherlands.
• Accuracy: murmel-flash-l-v5 makes the fewest errors of the seven models tested on open Dutch benchmarks (7.1% average word error rate on FLEURS, Common Voice 17, MLS and VoxPopuli).
• Open weights: murmel-flash-m-v5 (128M parameters) is coming soon as open weights, free for non-commercial use.

What we learned

Size is not everything. A model a twelfth of Whisper's size, trained carefully for one language, can beat it on that language. But the details decide whether people trust a transcript: no text out of nowhere, numbers in the right format, names spelled right, and the option to keep audio on your own device. Those are not things you get from a bigger model. You get them by looking at the mistakes, one by one, and fixing them.

Want to try it, build something with it, or tell us where it goes wrong? We would love to hear from you: info@the-ai-factory.com.