Claude Sonnet 5.5
AnthropicClaude CodeReasoning: high
Claude Sonnet 5.5 gets about 1 vocabulary question in 10 right. It does best in Kabyle. Its weakest spot is writing Standard Moroccan Amazigh words in Tifinagh. It swaps Tifinagh and Latin letters well (89.4 out of 100).
Rank 6 of 9Tied with GLM 5.3 Flash
TifinaghBench score
12.9out of 100
95% interval 12.1–13.6
12.3 with strict spelling
0–100, with 95% interval
Script score
89.4out of 100
95% interval 88.3–90.5
73.6 with strict spelling
Transliteration score, 0–100
Scores by dialect and task
Share of the score earned in each cell, 0–100. The transliteration column measures the script, not the vocabulary.
Standard Moroccan Amazigh
- Tifinagh → English
- 9.467 correct, 53 partial, 880 wrong of 1000
- English → Tifinagh
- 0.99 correct, 0 partial, 991 wrong of 1000
- Tifinagh ↔ Latin
- 97.4974 correct, 0 partial, 26 wrong of 1000
Tarifit
- Tifinagh → English
- 14.1104 correct, 72 partial, 817 wrong of 993
- English → Tifinagh
- 2.222 correct, 0 partial, 971 wrong of 993
- Tifinagh ↔ Latin
- 78.0780 correct, 0 partial, 220 wrong of 1000
Kabyle
- Tifinagh → English
- 38.6326 correct, 117 partial, 554 wrong of 997
- English → Tifinagh
- 12.0120 correct, 0 partial, 877 wrong of 997
- Tifinagh ↔ Latin
- 92.8928 correct, 0 partial, 72 wrong of 1000
| Tifinagh → English | English → Tifinagh | Tifinagh ↔ Latin | |
|---|---|---|---|
| Standard Moroccan Amazigh | 9.467 correct, 53 partial, 880 wrong of 1000 | 0.99 correct, 0 partial, 991 wrong of 1000 | 97.4974 correct, 0 partial, 26 wrong of 1000 |
| Tarifit | 14.1104 correct, 72 partial, 817 wrong of 993 | 2.222 correct, 0 partial, 971 wrong of 993 | 78.0780 correct, 0 partial, 220 wrong of 1000 |
| Kabyle | 38.6326 correct, 117 partial, 554 wrong of 997 | 12.0120 correct, 0 partial, 877 wrong of 997 | 92.8928 correct, 0 partial, 72 wrong of 1000 |
By dialect
Meaning tasks only
- Standard Moroccan Amazigh5.1
- Tarifit8.2
- Kabyle25.3
By task
- Tifinagh → English20.7
- English → Tifinagh5.1
- Tifinagh ↔ Latin89.4
Tifinagh → English grades
Free-text meanings, graded by the judge or matched exactly.
- Correct
- Partial
- Wrong
- Failed
Standard Moroccan Amazigh67 correct, 53 partial, 880 wrong of 1000
Tarifit104 correct, 72 partial, 817 wrong of 993
Kabyle326 correct, 117 partial, 554 wrong of 997
Character error rate
For exact-match answers: the smallest edit distance to an accepted answer, divided by its length. Lower is better; it shows near misses that a 0 score hides.
| English → Tifinagh | Tifinagh ↔ Latin | |
|---|---|---|
| Standard Moroccan Amazigh | 85.9% | 1.2% |
| Tarifit | 91.4% | 18.1% |
| Kabyle | 67.9% | 2.5% |
The run
- Setup
- Official setup
- Harness
- Claude Code 2.1.302@anthropic-ai/claude-agent-sdk 0.3.303
- Judge
- Claude Haiku 5.5 (Claude Code)
- Sample
- 1,000 words per dialect, seed 1
- Batch size
- 100
- Sessions
- 97 answer, 30 judge
- Tokens
- 736,459 in · 477,488 out · 238,873 reasoning
- Model cost
- $5.98 (API-equivalent; run on a subscription)
- Judge cost
- $0.22
- Dates
- 20 Oct 2026, 15:40 UTC → 20 Oct 2026, 15:56 UTC
- CLI version
- 0.1.0
Dataset checksums
SHA-256 of each dictionary file, so the run can be checked once the datasets are released.
- Standard Moroccan Amazigh
633505d9ccd2244f9a53b64f56bd153c428cc3dbd7229066845404bf3d7c5060 - Tarifit
d144a5619aa31cc06ffcc4eb26c361c65ca2c71451dfdfbea7fa134206481c4f - Kabyle
ed3a9882ac0f4ef10630917a302fd7d9ff6ac706654bb317b94c715f6b4cf9ea