Benchmarks
Humanity's Last Exam
v32,500 expert-written closed-ended questions across dozens of subjects (multiple-choice and exact-match, ~14% with images), graded by an LLM judge. Official CAIS harness.
knowledgeqa8 subsets2,500 samplesvisionHumanity's Last Exam: Diamond
v21,000 refined HLE questions, 500 reasoning and 500 knowledge, graded by an LLM judge. Official CAIS harness, no tools.
knowledgeqa2 subsets1,000 samplesvision