Claude Opus 4

Crowdsourced pairwise human preference rankings of LLM responses. Higher Elo means more frequently preferred by users.

American Invitational Mathematics Examination 2024 problems. Three-digit integer answers; very hard for non-reasoning models.

164 hand-written Python programming problems scored by passing unit tests. Saturated for frontier models.

Continuously refreshed coding benchmark drawing from LeetCode, AtCoder, and Codeforces; reduces benchmark contamination.

Real GitHub issues solved end-to-end. Verified subset is a 500-task human-validated slice of SWE-bench.

Massive Multi-discipline Multimodal Understanding; college-exam level questions with images across 30+ subjects.

Math reasoning over visual contexts (charts, figures, geometry).

Long-context retrieval and reasoning suite. We report the 128k token effective-context score.

Median sustained output speed in tokens per second on the model's first-party API for medium-length prompts. Higher is faster.

Median time from request to first output chunk in milliseconds on the model's first-party API for medium-length prompts. Lower is snappier; reasoning models are penalised here because they think before talking.

Contamination-free average across LiveBench's seven task categories (reasoning, coding, agentic coding, mathematics, data analysis, language, instruction following). Questions are rotated every six months and ground-truth answers are objective, removing the need for LLM-as-judge scoring.

Data-analysis subcategory of LiveBench. Table comprehension, CSV / spreadsheet reasoning, SQL-style joins, and chart interpretation. Refreshed every six months with new tables and questions to minimise contamination.