Multichannel Dutch speaker recognition: telling voices apart, even when they sound alike
A new model for who speaks when, built for Dutch, that uses stereo and multiple microphones instead of throwing them away
The hardest recording for speaker recognition isn't a meeting with ten people. It's a conversation between two colleagues of the same age, both women, both from the same region. A model that tells speakers apart by timbre and pitch hears two almost identical voices. You may know the result: a transcript where half the meeting is attributed to one person, or where someone changes name halfway through a sentence.
You can try to solve that with a bigger model that learns to hear even finer differences in a voice. I went the other way.
Make the problem simpler, not the solution more complex
In many recordings, the information about who is talking is already there. Phone calls are often recorded in stereo: one side left, the other right. In a podcast or interview, every speaker has their own microphone. And in an online meeting, every participant's computer receives a separate audio stream.
Until now, most systems, Murmel included, threw that information away straight away: everything was mixed down to one mono track, and then a model had to pull it apart again. That's a hard problem you created yourself.
The new model keeps the channels apart. The question "which of these two nearly identical voices is this?" then often becomes a much simpler one: "on which microphone is this loudest?" Only when that isn't enough, for example because two people share a microphone, does the voice itself come into play.
Setups the model understands
A channel may carry one speaker or several, and there may be more people in the recording than there are channels. The model uses up to four channels. Below are the most common setups. In each drawing you see on the left who sits where, in the middle what arrives on each channel (coloured by who is speaking, faded when someone is picked up from further away), and at the bottom what Murmel makes of it.
Online meeting with the notetaker
When you send the Murmel notetaker to a Google Meet, Teams or Zoom call, it now records in stereo. Every participant arrives as their own stream, and the notetaker spreads them over left and right. Two people who sound alike usually end up on different channels. You don't need to do anything for it.
Phone call
Many phone systems and call recording tools store a call with each side on its own channel. Then the problem is nearly solved before the model starts. On more than 20 hours of Dutch phone calls, the wrong speaker or no speaker is marked 3.8% of the time.
Interview or podcast
With a lavalier or headset per person and a recorder or audio interface that stores every channel separately, each microphone mostly hears one person. A bit of crosstalk is no problem: the difference in volume between the microphones tells who is talking.
Meeting room with table microphones
In a meeting room not everyone has their own microphone, and they don't need to. Two or three microphones spread over the table already split the problem: the people on the left side of the table are distributed differently than those on the right. Only people sharing a microphone still have to be told apart by voice.
Hybrid meeting
Part of the group in the room and part online? Record the room microphone on one channel and the call audio on a second. That removes the biggest source of confusion: someone remote is never mixed up with someone in the room.
One phone on the table
Most recordings are still plain mono: a phone on the table, a voice memo. There the model has no channels to lean on. It still does better than community-1, because it was trained on exactly the hard case: Dutch voices that sound alike.
Trained on voices that sound alike
The model is trained from scratch, on Dutch and English conversations in mono, stereo and with multiple microphones. Because real recordings of exactly the hard case are scarce, I simulated many of them: conversations built from individual speakers, deliberately picking voices with a similar pitch. Two men between 100 and 135 hertz, two women between 180 and 220. That way the model learns not to lean on the easy difference between a low and a high voice.
It works in two steps. First it looks at all channels at once, a few seconds at a time, and decides which voices are active and where they are loudest. Then those pieces are grouped into speakers across the whole recording, using a voiceprint of each speaker. At every moment it picks one speaker, so every word in the transcript belongs to exactly one person.
How it does
I compared the model with pyannote community-1, a widely used open-source method for speaker recognition that many services run on. Both got the same Dutch recordings from the Spoken Dutch Corpus (CGN), community-1 as mono, as it was designed. The measure is the diarisation error rate: the share of time where the wrong speaker, no speaker or an extra speaker is marked.
| Dutch test set | Recordings | Murmel | community-1 |
|---|---|---|---|
| Spontaneous conversations | 24 | 4.6% | 10.1% |
| Conversations between similar voices | 7 | 6.3% | 14.9% |
| Varied speech | 21 | 6.9% | 10.1% |
| Phone calls (stereo) | 171 | 3.8% | 9.9% |
Diarisation error rate in %, lower is better, with the usual quarter-second margin around each change of speaker.
The gap is largest where it matters most. In conversations between voices of the same sex, mostly between two women, community-1 marks the wrong speaker almost 15% of the time. Murmel 6.3%. That's a small set of seven recordings, but the pattern is the same in every round I measured.
I also tested what users actually see: words in a transcript attributed to the wrong person. On our own recordings of Dutch Google Meet calls, that's 4.1% of words when the notetaker records in stereo, and 5.1% when the same recording is mono. With community-1 it was 7.5%.
To be fair: on English meetings with four or five participants, community-1 is still slightly better, about 12.5% against 9.5% on the well-known AMI meetings. Murmel is built for Dutch, and that's where I focused first. English meetings are next.
Tips for a recording that's easy to pull apart
- Record in stereo if you can. Many recorders and call recording tools have a stereo setting. Turn it on and leave it: don't convert to mono yourself before uploading.
- A microphone per speaker is best, but two microphones on either side of the table already help a lot.
- Hybrid meeting? Put the room and the call on separate channels.
- Separate tracks per person, for example from podcast software? Send them together through the API and they become one recording with multiple channels.
How to use it
In the studio and with the notetaker, speaker recognition is always on. Upload a stereo or multi-channel file as it is, and the model uses the channels by itself. The notetaker now records online meetings in stereo.
Through the Murmel API you send a stereo file as audio, with identify_speakers. Recorded every speaker separately? Send the tracks as repeated channels fields (2 to 8 files of the same recording, starting together). You pay once, for the longest track:
curl -X POST https://api.murmel.eu/v1/transcribe \
-H "X-API-Key: $MURMEL_API_KEY" \
-F channels=@anna.wav \
-F channels=@bram.wav \
-F identify_speakers=trueOn your own infrastructure
Recordings not allowed to leave your network? The speaker model is also available under licence to run on your own servers, together with the Murmel speech models. Audio and transcripts then stay inside your own environment. Reach out at murmel@the-ai-factory.com, or read more in Run Murmel on your own servers.
In short
- What it is: murmel-flash-speakers-multichannel-v1, a model by The AI Factory that decides who speaks when, using up to four channels of a recording.
- What it's good at: Dutch conversations, and especially voices that sound alike.
- Accuracy: 4.6% diarisation error on Dutch conversations, against 10.1% for pyannote community-1; 3.8% against 9.9% on phone calls.
- Where to use it: in the studio on murmel.nl, with the notetaker for Google Meet, Teams and Zoom, and through the API. On servers in the Netherlands, or under licence on your own infrastructure.
What I like most about this model isn't the model itself, but that the problem got smaller as soon as I stopped throwing information away. Got a recording where it still goes wrong? Tell me at info@the-ai-factory.com: exactly those recordings make the next version better.