Dev Blog

Localizing a voice agent is a linguistics problem dressed as an engineering one.

Grace Hynes

Grace Hynes

Voice Linguist Specialist

Nov 17, 2026

·

7

min read

Picture an accounts payable team in Riyadh picking up a call about an overdue invoice. The agent greets them in natural Saudi Arabic, gets to the point quickly, and then reads out a balance like a newsreader doing the evening headlines in Modern Standard Arabic.

Recently, we prepared our voice agent to speak Arabic, a language spoken by over 400 million people worldwide, but for which model support varies substantially by variety and task. The scenario above was what we found in early testing while optimizing our voice pipeline for Dialectal Arabic. The conversation sounded local until a large number came up, and then the agent jumped to a mixed dialect/textbook register, sometimes saying the same amount differently from one run to the next. On a collections call, that’s a real problem. The number is the whole point of the call.

Case Study: Arabic (hard mode)

Arabic has unique linguistic features that are historically challenging for language technology, especially when deployed in real-time domains. This is due to regional dialects with variable writing conventions, diglossia, prevalence of homographs, and other quirks of orthography.

Many available language technologies for Arabic target Modern Standard Arabic, a “high” form of Arabic used primarily in academic and professional texts (aka, widely available data). The majority of Arabic speakers speak one of 30+ dialects.

To deploy a voice agent quickly and at scale, without the time-consuming process of re-training a model from scratch, we “localize” our prompts with critical glossaries, rules-based approaches, and cultural context for optimal performance in our domain.

Our voice agent needs to recall large amounts of financial information to relay to busy people at work. While LLMs at high effort levels could perform well over text, we can’t expect a user to sit through those levels of latency on the phone.

Here we marry linguistics, engineering, and user experience to localize a standard agent pipeline for scalable releases.

  1. Consider how we want the voice agent to sound, then determine which pipeline provides the best results for a given domain.
  2. Bootstrap LLMs, STT, and TTS engines with old-fashioned NLP techniques, prompting, and custom tool calls that the agent can draw on for things it may be bad at on its own, while prioritizing latency.

Problem 1

Given our voice agent is conversational, our goal was to deploy an agent that could engage in a commonly-spoken dialect of Arabic, such as Egyptian and Saudi, rather than Modern Standard Arabic (MSA).

Large monetary amounts were a particular challenge in our testing. We found that responses would use the requested dialect overall but then express a number with MSA case endings or MSA words mixed with dialect forms, sometimes switching between runs.

The direction of conversion matters. A recognition pipeline may turn spoken number words into digits through inverse text normalization. Speech synthesis needs to realize those values as speech, either through its own normalization or through text prepared upstream.

We combined a dialect-specific glossary with a deterministic number grammar to normalize financial amounts, dates, and invoice numbers in the call’s contexts. Our rationale was to avoid a tool round trip for every possible amount and keep the prompt focused on the call’s relevant financial data.

Explicit verbalization grammars are an established speech-processing technique. Our work applied that approach to the monetary expressions and dialect requirements of our agent.

While the verbalization grammar helps consistently steer the agent to Saudi number forms, the TTS step adds additional complications.

3 balances, with/without a verbalization glossary, run through eleven_v4_turbo, reviewed by a native speaker evaluator.

Each amount: what the voice said, what the reviewer heard. Three balances (3,869,967.52, 2,011,300.11 and 1,194.06 SAR) read with and without the Saudi glossary, with word-level IPA and native reviewer notes.

While the amounts using the Saudi glossary outputs were overall ranked more natural by our native reviewer, we found a few cases where we risked intelligibility for the sake of adhering to our chosen dialect.

Adding diacritics to certain words can improve pronunciations where TTS fails to output the correct vowel, such as in in the plural form of “halala,” but inconsistently overall, sometimes depending on where a word sits in a sentence.

We discovered we need a pronunciation add-on in our TTS step (eleven_v4_turbo) to avoid garbling some common phrases. The TTS pipeline fails to apply the correct phonological processes to the more dialect-specific orthography.

This will be an ongoing challenge as we continue to improve our agent and there is a clear need for better Arabic TTS models, but the space looks to be evolving at a rapid pace.

Plural form of “halala” (هللات) (Saudi currency unit)

Problem 2

Our voice agent saves AR-teams time by tracking down and directly engaging with AP contacts.

We make outbound calls, navigate through phone trees and the receptionist's desk, and often speak to multiple people on one phone call.

Here is where Arabic is challenging for a text-based voice agent system without additional tooling, as Arabic speakers assign gender inflections based on with whom they are speaking. On top of this, critical information can get lost in the pipeline. Some of these gender inflections are only present in spoken form without diacritics. Any additional information on gender picked up by the STT may not be passed to the LLM , such as in homographs: «معك», «لك», «عندك» (with you, for you, at yours), where the ending, ك, differentiates gender but is not typically marked to denote vowel differences.

Without any prompt rules concerning gender inflection or clear evidence in the conversational context, the LLM (gpt-4.1, gpt-6.1 terra) drifted inflections in our testing, switching between masculine and feminine address. We were also concerned about situations where the LLM would need to change gender address mid-call. What might happen if a user transferred the agent to a different speaker? What about the likely event where gender cues might not appear to the LLM until much later in the conversation?

If we are able to establish a clear rule on the address needed, the agent could usually hold the reference throughout a conversation, either for gendered singular form, or in gender-neutral plural forms.

We saw evidence that some dialect speakers were more tolerant of the agent defaulting to a plural, gender-neutral form of address (Saudi), than others (Egyptian), though more data is needed. Plural address also has the the added benefit of avoiding homographs that will confound the TTS output. This is our established default behavior in our Saudi agent until we gather more data on conversational dynamics in Arabic and the typical calling sequence in our domain.

In human-to-human discourse, we track multiple streams of non-linguistic information that allow us to navigate situations like this. When establishing a voice agent pipeline, we needed to consider all of the avenues information can reach the voice agent in order to establish firm rules. Future directions here could include speaker diarization, tracking conversational context across calls, better rules on conversational engagement, and speech-to-speech models.

Problem 3

Finally, we establish cultural context for the agent.

What are the working days in a given country for scheduling a callback?

Should the agent’s register be more formal or casual?

What other languages and dialects might the agent encounter while calling in this domain?

Do speakers tend to code-switch between English or other languages in technical contexts? Is the agent pipeline able to handle this both on the STT and TTS side?

In American English voice agents, these prompting rules might be presupposed, but we need to be explicit when localizing.

Thanks to our native speaker review partner, Hani Abdulrahman, as well as data provisioners and reviewers from Spirelight. We also discussed extensively with our extremely knowledgeable peers at Rime, Retell, and LiveKit concerning this topic.

References

  1. UNESCO. (n.d.). World Arabic Language Day.
  2. Ryding, Karin C. (2005). A Reference Grammar of Modern Standard Arabic. Cambridge University Press. Introduction, pp. 1–9. Publisher excerpt. Chapter DOI: 10.1017/CBO9780511486975.002.
  3. Habash, Nizar, Fadhl Eryani, Salam Khalifa, Owen Rambow, Dana Abdulrahim, Alexander Erdmann, Reem Faraj, Wajdi Zaghouani, Houda Bouamor, Nasser Zalmout, Sara Hassan, Faisal Al-Shargi, Sakhar Alkhereyf, Basma Abdulkareem, Ramy Eskander, Mohammad Salameh, and Hind Saddiki. (2018). Unified Guidelines and Resources for Arabic Dialect Orthography. Proceedings of LREC 2018, pp. 3628–3637. European Language Resources Association.
  4. Robinson, Nathaniel Romney, Shahd Abdelmoneim, Kelly Marchisio, and Sebastian Ruder. (2025). AL-QASIDA: Analyzing LLM Quality and Accuracy Systematically in Dialectal Arabic. Findings of the Association for Computational Linguistics: ACL 2025, pp. 22048–22065. DOI: 10.18653/v1/2025.findings-acl.1137.
  5. Albasiri, Enas. (2024). Computational Approaches to Linguistic Challenges in Arabic Speech Recognition. Doctoral dissertation, CUNY Graduate Center.
  6. Ritchie, Sandy, Richard Sproat, Kyle Gorman, Daan van Esch, Christian Schallhart, Nikos Bampounis, Benoît Brard, Jonas Fromseier Mortensen, Millie Holt, and Eoin Mahon. (2019). Unified Verbalization for Speech Recognition & Synthesis Across Languages. Interspeech 2019, pp. 3530–3534. DOI: 10.21437/Interspeech.2019-2807.
  7. Albasiri, Enas, Myungjong Kim, Nourchene Ferchichi, and Oluwatobi Olabiyi. (2026). Inverse Text Normalization for Arabic Numbers in Streaming ASR. Proceedings of the Third Workshop on Computation and Written Language (CAWL 2026) @ LREC 2026, pp. 101–106. ELRA Language Resources Association (ELRA). DOI: 10.63317/4wy5qhn7npqa.
  8. Alhafni, Bashar, Nizar Habash, and Houda Bouamor. (2022). User-Centric Gender Rewriting. Proceedings of NAACL-HLT 2022, pp. 618–631. Association for Computational Linguistics. DOI: 10.18653/v1/2022.naacl-main.46.
  9. Hawasly, Majd, Hamdy Mubarak, Ahmed Abdelali, and Ahmed Ali. (2025). DialG2P: Dialectal Grapheme-to-Phoneme. Arabic as a Case Study. Proceedings of The Third Arabic Natural Language Processing Conference, pp. 466–471. Association for Computational Linguistics. DOI: 10.18653/v1/2025.arabicnlp-main.38.
  10. Hamed, Injy, Fadhl Eryani, David Palfreyman, and Nizar Habash. (2024). ZAEBUC-Spoken: A Multilingual Multidialectal Arabic-English Speech Corpus. Proceedings of LREC-COLING 2024, pp. 17770–17782. ELRA and ICCL.
  11. Lodagala, Vasista Sai, Lamya Alkanhal, Daniel Izham, Shivam Mehta, Shammur Absar Chowdhury, Aqeelah Makki, Hamdy S. Hussein, Gustav Eje Henter, and Ahmed Ali. (2025). SawtArabi: A Benchmark Corpus for Arabic TTS. Standard, Dialectal and Code-Switching. Interspeech 2025, pp. 4793–4797. DOI: 10.21437/Interspeech.2025-2573.
  12. CAMeL Lab. (2023). CODA*: Conventional Orthography for Dialectal Arabic. CAMeL Lab Guidelines, version 0.5, April 16, 2023.
  13. McAuliffe, Michael, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. (2017). Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi. Interspeech 2017, pp. 498–502. DOI: 10.21437/Interspeech.2017-1386. Version 3.4.2 used, with the English MFA acoustic model.
  14. Xu, Qiantong, Alexei Baevski, and Michael Auli. (2022). Simple and Effective Zero-shot Cross-lingual Phoneme Recognition. Interspeech 2022, pp. 2113–2117. DOI: 10.21437/Interspeech.2022-60. Phoneme recognition models used: wav2vec2-lv-60-espeak-cv-ft and wav2vec2-xlsr-53-espeak-cv-ft.
  15. Radford, Alec, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. (2023). Robust Speech Recognition via Large-Scale Weak Supervision. Proceedings of the 40th International Conference on Machine Learning, PMLR 202, pp. 28492–28518. Models used: Whisper large-v3 and large-v3-turbo, run locally.
  16. Boersma, Paul, and David Weenink. (2026). Praat: Doing Phonetics by Computer. Computer program, version 7.0.01.

Grace Hynes

Grace Hynes

Voice Linguist Specialist

Linguist working on voice agent evals and analysis.

We’re hiring

If you’re looking for a new exciting challenge, Stuut might be the place for you.

View job openings

Related articles

November 17, 2026

Grace Hynes

Localizing a voice agent is a linguistics problem dressed as an engineering one.

What we learned localizing Stuut's voice agent for Saudi Arabic: keeping large amounts in dialect, handling gendered address on calls, and giving the agent the cultural context it needs.

Read More

October 14, 2026

Emma Teng

Turning Human Review into Improvements: How Stuut Agent Learns from Its Coworkers

How Stuut Agent turns everyday human review decisions into prioritized fixes: one-sentence case descriptions, stable weekly patterns, root-cause investigation, and a few engineer-reviewed changes.

Read More

See all articles

Setup time to learn more

About

Case Studies

Blog

Careers

Partnerships

Arcade

Trust Center

Try AI Calling

Calculate Stuut’s Value

Terms of Services

Privacy Policy

LinkedIn IconTwitter IconInstagram IconYouTube Icon

© 2026 All Rights Reserved

Brand & Website by Day Job     Web dev by Jomor Design

Based in NYC, backed by Insight Partners, Andreessen Horowitz, M12, and others