Mahak (مِحَكّ)
Open community benchmark evaluating LLM Arabic fluency on authentic everyday tasks
Why This Was Built
General LLM benchmarks (MMLU, Chatbot Arena) rely heavily on translated English benchmarks that fail to evaluate authentic Arabic syntactic flow, legal terminology, and cultural nuance.
The Architectural Answer
A community-annotated benchmark scoring models across authentic Arabic prompts with public Elo ranking matrices, blind side-by-side judging, and native agent evaluation interfaces.
Key Features & Architecture
Technical Specifications
- Primary Languages
- TypeScriptGoAstroSQL
- Frameworks & Tooling
- Cloudflare WorkersReactD1Tailwind CSS
- Architecture & Execution Model
- Edge-rendered ranking matrix, double-blind pairwise voter engine, Go CLI with recovery manifests, and MCP server
- Licensing Model
- WaqfDPL-Isnad 1.0 (waqf)
- Release & Lineage
- Active release cycle since 2026 · Maintained by @JadMadi
Frequently Asked Questions
How does Mahak differ from standard benchmarks like ArabicMMLU?
ArabicMMLU tests translated multiple-choice knowledge. Mahak evaluates authentic generation, dialectal nuances, formal legal drafting, and tone adaptation.
Where can I access the live leaderboards?
The public matrix and side-by-side arena are live at https://mahak.waqf.dev.
How do agents or contributors submit model outputs?
Agents can submit outputs via the Go CLI (`mahak start` and `mahak sync`), REST API (`POST /api/outputs/batch`), or the Model Context Protocol (MCP) server.
How are benchmark rankings calculated?
Rankings use the Wilson score lower bound at 95% confidence with sample-quality filtering on the matrix, alongside an Elo rating system (K-factor 32 under 30 matches, then 16) in the Arena.