Urdu speech recognition for voice agents
The base model wrote no Urdu at all. After adding Urdu to its tokenizer and fine-tuning on 121 hours of speech, the latest test run transcribes code-switched Urdu–English at a 22.1% word error rate, fast enough to stream on live calls. This is ongoing work, and the numbers on this page move as the model improves.
- Where
- Miras AI, Lahore
- Status
- In progress, since September 2026
- Built with
- NVIDIA NeMo, Nemotron 3.5, PyTorch
- Code
- Proprietary; happy to walk through it in an interview
The problem
Miras AI builds voice agents. In Pakistan, callers speak Urdu, and most of them switch to English mid-sentence: میرا order ابھی تک deliver نہیں ہوا (my order still hasn't been delivered). Off-the-shelf speech recognition either ignores Urdu or forces everything into one language, so the agent mishears exactly the words that carry the request.
The goal was a model that understands Urdu, keeps the English words, and is fast enough that a caller never waits on transcription.
Data
121 hours of transcribed Urdu speech, including Urdu–English code-switching. Scores are reported as normalised word and character error rates, so formatting differences such as punctuation are not counted as mistakes.
Architecture
- Extend the tokenizerThe base model produced no Urdu output, so before training I extended its tokenizer with a new Urdu token.
- Fine-tuneTrained on the 121 hours with NVIDIA NeMo on a vGPU T40.
- Stream in chunksAudio is decoded in 320 ms chunks with 240 ms of lookahead, giving 0.56 s of algorithmic latency.
- Commit text per chunkEach chunk takes 67 ms of compute at p95, well inside its 320 ms budget.
Results
| Metric | Base model | Latest test |
|---|---|---|
| Word error rate (normalised) | ≈100% | 22.1% |
| Character error rate | ≈100% | 17.0% |
| English words kept in mixed speech | none | 74.6% |
| Algorithmic latency | – | 0.56 s |
| Compute per chunk, p95 | – | 67 ms |
| Real-time factor | – | 0.18 |
A real-time factor of 0.18 means one second of audio is processed in 0.18 seconds, 5.5 times faster than it is spoken.
Where it breaks
- A 22.1% word error rate is roughly one word in five. That works when a language model downstream reads the transcript for intent; it is not good enough for verbatim records.
- About a quarter of English words in mixed sentences are still lost or mis-transcribed. Those are often product names and order terms, the words an agent most needs.
- The 0.56 s is algorithmic latency. Network and telephony delay come on top of it in a real call.
- 121 hours is small for a language as varied as Urdu. Regional accents and noisy narrowband phone audio are the next things to measure separately before trusting the headline figure everywhere.
Next case study: AI phone intake for US law firms