Findings
What the runs show beyond the ranking. Every headline on this page is computed from the published results, so it stays true as models are added.
Knowing the script is not knowing the words
Each line is one model, from how well it swaps Tifinagh and Latin letters (left) to how well it knows the words (right). Almost every model handles the alphabet well, but few know the vocabulary.
Every model reads Tifinagh better than it writes it
Tifinagh → English asks what a word means; English → Tifinagh asks the model to spell the word itself. Averaged over the three dialects, the median model scores 3.5 times higher reading than writing.
- Writing (English → Tifinagh)
- Reading (Tifinagh → English)
| Model | Writing (English → Tifinagh) | Reading (Tifinagh → English) |
|---|---|---|
| GPT-6.1-Sol | 15.8 | 43.9 |
| GPT-6-Astra | 15.8 | 43.0 |
| Claude Opus 5.5 | 15.2 | 39.4 |
| GPT-6-Luna | 8.1 | 28.2 |
| DeepSeek v4.1 Flash | 8.8 | 24.8 |
| Claude Sonnet 5.5 | 5.1 | 20.7 |
| GLM 5.3 Flash | 5.1 | 18.8 |
| Claude Haiku 5.5 | 3.4 | 12.6 |
| MiniMax M3.1 Flash | 0.4 | 2.4 |
Kabyle is the easiest dialect for every model
Meaning score in each dialect, both meaning tasks together. Read the gap with care: the Kabyle lexicon is the smallest of the three and its Tifinagh spellings were generated from the Latin script, so it is not quite the same test.
- Standard Moroccan Amazigh
- Tarifit
- Kabyle
| Model | Standard Moroccan Amazigh | Tarifit | Kabyle |
|---|---|---|---|
| GPT-6.1-Sol | 21.1 | 23.3 | 45.2 |
| GPT-6-Astra | 20.4 | 23.4 | 44.4 |
| Claude Opus 5.5 | 15.8 | 22.1 | 44.0 |
| GPT-6-Luna | 9.8 | 10.9 | 33.8 |
| DeepSeek v4.1 Flash | 9.4 | 10.2 | 30.7 |
| Claude Sonnet 5.5 | 5.1 | 8.2 | 25.3 |
| GLM 5.3 Flash | 6.0 | 6.9 | 23.1 |
| Claude Haiku 5.5 | 2.4 | 4.7 | 16.9 |
| MiniMax M3.1 Flash | 0.9 | 1.3 | 2.0 |
Spelling conventions move script scores by up to 19 points, and TifinaghBench by at most 1
Dictionaries write the same sounds differently, and the Tarifit dictionary drops the schwa irregularly. Strict scoring accepts only the dictionary’s own spelling; the official scores also accept the standard conventions. That rescues many transliterations but hardly any meanings: a model that does not know a word gains nothing from spelling rules.
Script score
- Strict spelling
- Conventions accepted
| Model | Strict spelling | Conventions accepted |
|---|---|---|
| GPT-6.1-Sol | 77.5 | 96.5 |
| GPT-6-Astra | 77.7 | 96.4 |
| Claude Opus 5.5 | 77.6 | 96.0 |
| GPT-6-Luna | 70.8 | 88.3 |
| DeepSeek v4.1 Flash | 77.3 | 93.1 |
| Claude Sonnet 5.5 | 73.6 | 89.4 |
| GLM 5.3 Flash | 65.4 | 80.6 |
| Claude Haiku 5.5 | 70.1 | 81.3 |
| MiniMax M3.1 Flash | 48.5 | 56.5 |
TifinaghBench
- Strict spelling
- Conventions accepted
| Model | Strict spelling | Conventions accepted |
|---|---|---|
| GPT-6.1-Sol | 29.0 | 29.9 |
| GPT-6-Astra | 28.5 | 29.4 |
| Claude Opus 5.5 | 26.3 | 27.3 |
| GPT-6-Luna | 17.7 | 18.2 |
| DeepSeek v4.1 Flash | 16.3 | 16.8 |
| Claude Sonnet 5.5 | 12.3 | 12.9 |
| GLM 5.3 Flash | 11.7 | 12.0 |
| Claude Haiku 5.5 | 7.9 | 8.0 |
| MiniMax M3.1 Flash | 1.4 | 1.4 |
Wrong Tifinagh spellings are rarely near misses
For English → Tifinagh, the character error rate counts the edits needed to reach an accepted spelling, divided by its length. Over all answers, the median model would have to change 82% of the characters: most wrong answers are a different word, not a typo.
| Model | Character error rate |
|---|---|
| GPT-6.1-Sol | 66.3 % |
| GPT-6-Astra | 65.5 % |
| Claude Opus 5.5 | 67.8 % |
| GPT-6-Luna | 81.9 % |
| DeepSeek v4.1 Flash | 79.1 % |
| Claude Sonnet 5.5 | 81.7 % |
| GLM 5.3 Flash | 87.5 % |
| Claude Haiku 5.5 | 86.7 % |
| MiniMax M3.1 Flash | 90.9 % |