Run it on your model

The benchmark is a command-line tool. It drives Codex, Claude Code or OpenCode through their official SDKs, closed-book, and writes one results folder per run.

Requirements

Node.js 22.13 or newer
At least one harness installed and signed in: Codex, Claude Code or OpenCode
Claude Code for the official judge, Claude Haiku 5.5
The datasets (see below)

The datasets are not published yet: we are waiting for permission from the dictionaries’ rights holders. Until then the CLI needs your own copy in the documented format. Expected dataset format

Quick start

The CLI asks for the harness, model, dialects, tasks, judge and sample size, then shows live progress.

git clone https://github.com/Itri-Cloud-Labs/TifinaghBench.git
cd TifinaghBench/cli
npm install
npm run dev

The official setup

Leaderboard runs use exactly these settings, changing only the harness, model and reasoning level. Runs are batched 100 words per session, which keeps a full run within a $20 subscription.

npm run build
node dist/cli.js \
  --harness=<codex|claude-code|opencode> \
  --model=<model id> \
  --reasoning=<level> \
  --dialects=ircam,tarifit,kabyle \
  --tasks=tif-to-en,en-to-tif,transliterate \
  --judge-harness=claude-code --judge-model=claude-haiku-5-5 \
  --sample=1000 --batch-size=100 --seed=1 -y

If a run stops at a usage limit, run the same command again after the limit resets: it resumes where it stopped.

Export a shareable copy

Full results quote the dictionaries, so only the score-only export may be published.

node dist/cli.js --export-public results/<run folder> \
  --output ../results/<run folder>

Submit your results

Open a pull request that adds the exported folder to results/. The leaderboard and this site rebuild from it.

Open the repository