Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Short answer: DarijaBench reports high scores for four AI models on a small set of written Moroccan Darija tasks, but it does not show that they understand the language broadly or reliably. Its 60 items cover translation, sentiment classification, and cultural questions; the creator says the score difference between the models was not statistically significant.
What DarijaBench tests
Soufian Zaari’s DarijaBench consists of 60 items, divided evenly among three tasks. The overall score averages results across those categories. Answers are graded automatically with keyword checks, rather than assessed by a panel of human judges. Zaari’s benchmark report
- Darija-to-French translation: The model translates Moroccan Darija into French.
- Sentiment classification: The model identifies sentiment in Darija text.
- Moroccan cultural questions: The model answers questions about Moroccan culture written in Darija.
This is a focused test of short written responses. It is not an evaluation of spoken Darija, extended conversation, or every way people use the language.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →What the reported scores show
In Zaari’s report, Gemini 3.7 Flash, Claude Sonnet 5, and Claude Opus 4.7 each scored 0.95; GPT-5.6 Luna scored 0.90. In an October 2, 2026 update, Zaari reported an exact McNemar p-value of 0.25 for the comparison and described the result as a four-way tie at n = 60. The observed five-point gap therefore does not establish a winner. These are the creator’s results, not an independently replicated comparison. Zaari’s benchmark report
The category results also show why a single overall score needs context. Zaari says sentiment was the easiest task: Gemini answered 20 of 20 items correctly. Translation was the hardest; Gemini scored 85%. The report points to idioms and French loanwords whose use varies in Darija as sources of difficulty. Those findings describe this test set, not model capability in general.
Why a high score is not proof of broad understanding
The sample is small
Each category contains just 20 items. Zaari describes 60 items as a small test and estimates that roughly 200–300 items would be needed to detect a five-point gap reliably. That is the author’s estimate, not an independently verified power analysis. A compact set can provide a useful snapshot, but the result may shift with different examples.
Keyword grading can miss valid answers
Automatic keyword checks are simple to apply, but they can mark an acceptable response wrong if it expresses the right idea in different words. Conversely, matching expected terms does not by itself prove that a model understood the question. The scoring method limits how much can be inferred from the total.
The task coverage is narrow
The three categories do not establish how models handle regional variation, code-switching between Darija and other languages, different scripts, idioms beyond those sampled, or everyday conversation. The benchmark creator recommends adding more idioms, code-switching, and regional variants in a harder version. Whether the reported performance carries over to those settings remains unresolved by this test.
Rank #3
How this fits with other Darija evaluations
Other work suggests that adapting models specifically for Darija can improve results on particular evaluation suites, but it is not a replication of Zaari’s benchmark.
A January 2025 ACL Anthology record for Shang and colleagues’ Atlas-Chat paper describes fine-tuned models in 2B, 9B, and 27B sizes. It reports that Atlas-Chat-9B achieved a 13% performance boost over a larger 13B model on DarijaMMLU. This is evidence about a different model and test suite, not confirmation of the four scores above. ACL Anthology record for Atlas-Chat
Rank #4
AtlasIA has also described a community arena with nearly 300 Darija prompts, model outputs compared head to head, and user votes on accuracy, fluency, and cultural alignment. Human preference voting differs from keyword grading, and that project’s post describes older model versions. It should not be treated as a current leaderboard without checking the live project. AtlasIA project
There are two resources called DarijaBench
Zaari’s 60-item benchmark is not the only resource with this name. An MBZUAI-Paris dataset card describes a broader research dataset covering summarization, six translation directions, and sentiment analysis. The two resources have different scopes, so a score or task description from one should not be attributed to the other. MBZUAI-Paris DarijaBench dataset card
Best Value
What to check when comparing Darija model results
A score is meaningful only in relation to the task and method that produced it. Before comparing two evaluations, check:
- Task: Is the evaluation testing translation, sentiment, cultural knowledge, summarization, or another ability?
- Size and coverage: How many examples are included, and do they represent the dialects and situations you care about?
- Scoring: Are answers checked against keywords or references, or judged by people? What kinds of errors can that method miss?
- Language form: Does the test include code-switching, regional forms, idioms, and the scripts people actually use?
- Date and model version: Which model release was tested, and when? Results from older versions may not describe current systems.
For this particular benchmark, the key distinction is between a strong result on a compact set of written tasks and a claim of broad language understanding. The former is what the report supports; the latter remains unproven.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
