Choosing an LLM API for Arabic: What Our Benchmarks Show

OpenAI, Anthropic, and Google all publish Arabic as a supported language across their current model families, and in the broad sense that's accurate — you can send a GPT, Claude, or Gemini model a prompt in Modern Standard Arabic and get back a fluent, grammatically sound response. For a translation task, a summarization task, or a well-formed customer inquiry written in formal Arabic, all three are genuinely capable, and the gap between them on that narrow slice of the problem is smaller than vendor marketing implies. If your only requirement is "handle clean, formal Arabic text," you have three good options and the choice mostly comes down to cost, latency, and whatever infrastructure you're already standardized on.

But almost none of our clients' real traffic looks like clean, formal Arabic. A WhatsApp inbox for a Saudi retail brand, a voice line for a Cairo clinic, or a support queue for a Dubai logistics operator sees Gulf and Egyptian dialect, French- and English-loanword code-switching, voice notes with regional accents, and a steady stream of Arabizi — Arabic written in Latin script with numerals standing in for letters that don't exist in English ("3" for ع, "7" for ح, and so on). That is the actual test, and it's where the published benchmarks stop being useful, because none of the major labs publish detailed, dialect-segmented Arabic evaluation results. The MSA fluency you see in a demo is not evidence of dialect competence, and treating it as such is the single most common mistake we see clients make when picking a model.

Start with dialect accuracy, and be specific about which dialect — Gulf, Egyptian, and Levantine Arabic diverge enough in vocabulary and idiom that a model tuned or heavily reinforced on one can visibly stumble on another. Build a small test set from real transcripts or messages your business already has (support logs, call recordings, old WhatsApp threads), not synthetic examples you write yourself, because your own writing will unconsciously drift toward the more formal register a model handles easily. Run the same test set across every candidate model and score it against a task-specific rubric: did it correctly extract the appointment time, the product name, the complaint category — not just "did the response sound fluent."

Code-switching and Arabizi deserve their own test pass entirely. Real customers switch mid-sentence between Arabic and English, drop brand names and technical terms in Latin script inside an Arabic sentence, and a meaningful share of younger Gulf and Egyptian users write entire messages in Arabizi out of habit, not necessity. A model that handles formal Arabic beautifully can still fail to parse "7abibi fein el order bta3y" as a request to track an order — and if it fails silently, producing a confident but wrong answer instead of asking for clarification, that failure mode is worse than an outright error, because nobody catches it until a customer complains. Test this explicitly rather than assuming it falls out of general Arabic competence, because it does not.

Latency and cost are not secondary concerns for Arabic specifically — they're where the tradeoffs actually bite. Arabic text tokenizes less efficiently than English on most current tokenizers, which means the same conversation costs more tokens and can run slower in Arabic than in English on the same model, sometimes by a wide margin. That difference changes the economics of a high-volume WhatsApp or voice deployment enough that it belongs in the same evaluation spreadsheet as accuracy, not as an afterthought once you've already picked a model on quality alone.

Data residency is the fourth axis, and it's often the one that eliminates candidates before accuracy testing even starts. Saudi PDPL and similar Gulf frameworks push regulated data — health records, financial data, government-adjacent workflows — toward regional hosting or, in some sectors, an explicit prohibition on cross-border transfer. If your use case touches that kind of data, the question isn't just which model performs best on your Arabic eval set, it's which of the models that pass your eval set can actually be deployed under your regulatory constraints. That can rule out a straightforward API call to a foreign region entirely and push you toward a regional cloud deployment or an on-premises option — which is where open-weight Arabic-tuned models become relevant, not as a compromise on quality but as the only architecture that satisfies the residency requirement at all.

The open-weight ecosystem for Arabic has matured enough that it belongs in the conversation, not as a fallback for teams that can't afford a frontier API, but as a deliberate choice when data residency or full model control is a hard requirement. Several groups — Gulf-region research labs among them — have released Arabic-focused or Arabic-tuned open-weight models specifically to address the dialect and residency gaps that closed-source frontier models leave. Running one of these on infrastructure you control, whether that's a private cloud region in-country or on-premises hardware, removes the cross-border data transfer question entirely, because the data never leaves your environment in the first place.

The tradeoff is real and worth stating plainly: open-weight models typically require more engineering effort to reach the same reliability bar a well-chosen frontier API clears out of the box, and the smaller ones can lag on general reasoning and long-context tasks even when their Arabic fluency is strong. The right framing is not "open-weight versus frontier API" as a universal ranking, but a decision tree — start from your residency and control requirements, and let those constraints tell you which category of model you're even allowed to shortlist before comparing quality within that shortlist.

We don't have a single house model we default to, and we're wary of any vendor who does, because the right choice genuinely changes based on the client's dialect mix, channel, latency budget, and regulatory posture. Before recommending anything, we run our own Arabic evaluation suites against the specific candidate models for that client's actual use case — built from real transcripts where we can get them, covering the dialect the client's customers actually speak, including a code-switching and Arabizi segment, and scored against the task the model needs to perform rather than generic fluency. Where residency rules the API options out entirely, that eval runs against the open-weight and regional-hosting candidates instead, and we say so plainly rather than quietly defaulting to whichever model is easiest for us to integrate.

This is also why we treat model selection as an ongoing decision rather than a one-time one. Providers update their models on their own schedule, and an update that improves English reasoning can just as easily shift Arabic dialect behavior in either direction without a corresponding announcement. Any deployment we run gets its eval suite re-run whenever the underlying model changes, not just at launch, because the alternative is finding out about a regression from a customer instead of a dashboard.

Before you commit budget to a specific provider for an Arabic-facing product, get concrete answers to five questions: which dialects, specifically, does your customer base actually use, and do you have real samples to test against? Does the model handle code-switching and Arabizi in a test you ran yourself, not a claim you read? What does the same conversation cost and how long does it take in Arabic versus English on your actual traffic pattern? Does your data classification require regional hosting or an on-premises deployment, and does that rule out any candidate before quality even enters the conversation? And who re-tests the model the next time the provider ships an update? Answering these five honestly, with your own test data, will tell you more than any published Arabic-language benchmark currently available — because none of them are built on your customers' actual Arabic.