Sign in

API docs

Everything in the web app is available over HTTP. The CLI uses the same API.

Base URLhttps://evalhub.newmeasure.ai/api/v1

Quickstart

  1. 1 Create a key on the API keys page. It starts with evalh_ and is shown once.

    export EVALHUB_API_KEY=evalh_...

  2. 2 Price a run, then submit it.

    curl -s https://evalhub.newmeasure.ai/api/v1/estimate \
  -H "Authorization: Bearer $EVALHUB_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"benchmark_id": "hle", "model_id": "anthropic/claude-opus-5-5", "reasoning": "high"}'

curl -s https://evalhub.newmeasure.ai/api/v1/runs \
  -H "Authorization: Bearer $EVALHUB_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"benchmark_id": "hle", "model_id": "anthropic/claude-opus-5-5", "reasoning": "high"}'

  3. 3 Poll the run until status is succeeded or failed, then fetch the download links.

    curl -s https://evalhub.newmeasure.ai/api/v1/runs/$RUN_ID -H "Authorization: Bearer $EVALHUB_API_KEY"
curl -s https://evalhub.newmeasure.ai/api/v1/runs/$RUN_ID/download -H "Authorization: Bearer $EVALHUB_API_KEY"

Benchmark and model ids come from GET /benchmarks and GET /models, or the Benchmarks and Models pages.

Authentication

Send your key as a bearer token on every request except GET /benchmarks and GET /models. A key acts as your account, so keep it secret; revoke it on the API keys page if it leaks.

Authorization: Bearer evalh_...

Conventions

Format
JSON request and response bodies.
Money
Integer micros: 1000000 = $1.00. A run costs 10% of the model API cost.
Ids
Runs are UUIDs. Benchmarks and models use the string ids returned by the catalog endpoints.

Errors

Non-2xx responses have the body {"error": "message"}.

StatusMeaning
400Invalid request: a missing or bad field, an unknown benchmark or model.
401Missing, malformed or revoked API key.
402Not enough credit to start a run or download results.
404Not found, or it belongs to someone else.
409The resource isn't in the right state yet, e.g. downloading a run that hasn't succeeded.
429Rate limited: too many runs or requests today.

Catalog

GET/benchmarks

No auth

Benchmarks you can run. options_schema is the JSON Schema for the benchmark's options; subsets are the subsets the benchmark itself defines.

Example
curl -s https://evalhub.newmeasure.ai/api/v1/benchmarks
Response
{
  "benchmarks": [{
    "id": "hle", "name": "...", "description": "...", "family": "...", "category": "...",
    "version": "...", "options_schema": { ... }, "subsets": [{ "id": "...", ... }],
    "requires_vision": false, "judge_model_id": "..." | null
  }]
}

GET/models

No auth

Models you can run against. Pass one of reasoning_levels as reasoning; it defaults to default_reasoning.

Example
curl -s https://evalhub.newmeasure.ai/api/v1/models
Response
{
  "models": [{
    "id": "anthropic/claude-opus-5-5", "display_name": "...", "provider": "anthropic",
    "reasoning_levels": ["low", "medium", "high"], "default_reasoning": "medium",
    "supports_vision": true, "input_usd_per_mtok": 5, "output_usd_per_mtok": 25
  }]
}

Runs

POST/estimate

API key

Price a run without submitting it. Each unit is one subset, priced from earlier runs (history) or token counts (tokens).

Request body
{
  "benchmark_id": "hle",
  "model_id": "anthropic/claude-opus-5-5",
  "reasoning": "high",          // optional
  "options": { ... },           // optional, validated against options_schema
  "subsets": ["..."]            // optional, default all
}
Example
curl -s -X POST https://evalhub.newmeasure.ai/api/v1/estimate \
  -H "Authorization: Bearer $EVALHUB_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"benchmark_id": "hle", "model_id": "anthropic/claude-opus-5-5", "reasoning": "high"}'
Response
{
  "estimate_micros": 12340000,
  "units": [{ "subset": "...", "price_micros": 12340000, "basis": "history" }]
}

Errors · 400 invalid config, unknown benchmark or model, or a vision benchmark on a model without vision.

POST/runs

API key

Submit a run. Same body as POST /estimate.

Example
curl -s -X POST https://evalhub.newmeasure.ai/api/v1/runs \
  -H "Authorization: Bearer $EVALHUB_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"benchmark_id": "hle", "model_id": "anthropic/claude-opus-5-5", "reasoning": "high"}'
Response
201 { "run": { ...see GET /runs/{id} } }

Errors · 400 invalid config · 402 balance below $50.00, or the estimate exceeds your available credits (balance minus the estimates of your pending runs) · 429 daily run limit reached.

GET/runs

API key

Your runs, newest first. Same fields as a single run, without units and scores.

Example
curl -s https://evalhub.newmeasure.ai/api/v1/runs \
  -H "Authorization: Bearer $EVALHUB_API_KEY"
Response
{ "runs": [{ ... }] }

GET/runs/{id}

API key

One run. status is pending, succeeded or failed. metrics and scores (the benchmark's official score) are filled once the run has succeeded. Poll this until the status is final.

Example
curl -s https://evalhub.newmeasure.ai/api/v1/runs/$RUN_ID \
  -H "Authorization: Bearer $EVALHUB_API_KEY"
Response
{
  "run": {
    "id": "uuid", "status": "succeeded",
    "benchmark_id": "hle", "benchmark_version": "...", "model_id": "...", "reasoning": "high",
    "options": { ... }, "subsets": ["..."],
    "estimate_micros": 12340000, "charge_micros": 11980000,
    "metrics": { ... } | null, "error": null,
    "created_at": "...", "finished_at": "...",
    "units": [{ "subset": "...", "status": "succeeded", "n_samples": 500, "price_micros": 11980000, "metrics": { ... } }],
    "scores": [{ "label": "Accuracy", "subset": null, "value": 41.2 }]
  }
}

Errors · 404 run not found (or not yours).

GET/runs/{id}/download

API key

Signed download links, valid for one hour: results.json (scores and one row per task, subset ""), then the benchmark's own logs as one logs.tar.gz per subset.

Example
curl -s https://evalhub.newmeasure.ai/api/v1/runs/$RUN_ID/download \
  -H "Authorization: Bearer $EVALHUB_API_KEY"
Response
{
  "files": [
    { "subset": "", "name": "results.json", "url": "https://..." },
    { "subset": "...", "name": "logs.tar.gz", "url": "https://..." }
  ]
}

Errors · 402 balance is negative · 404 run not found · 409 run hasn't succeeded.

Billing

GET/balance

API key

Your credit balance, the estimates reserved by your pending runs, what is available for new runs, and the latest ledger entries (purchases, charges and adjustments).

Example
curl -s https://evalhub.newmeasure.ai/api/v1/balance \
  -H "Authorization: Bearer $EVALHUB_API_KEY"
Response
{
  "balance_micros": 87660000,
  "reserved_micros": 12340000,
  "available_micros": 75320000,
  "ledger": [{ "id": "...", "amount_micros": -11980000, "kind": "charge", "ref": "...", "note": "...", "created_at": "..." }]
}

POST/credits/checkout

API key

Start a Stripe Checkout to buy credits, between $10 and $10,000. Open the returned URL in a browser.

Request body
{ "amount_usd": 100 }
Example
curl -s -X POST https://evalhub.newmeasure.ai/api/v1/credits/checkout \
  -H "Authorization: Bearer $EVALHUB_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "amount_usd": 100 }'
Response
{ "url": "https://checkout.stripe.com/..." }

Requests

POST/benchmark-requests

API key

Ask for a benchmark we don't have yet. GET on the same path lists your requests.

Request body
{ "name": "SWE-bench Pro", "url": "https://...", "notes": "..." }   // only name is required
Example
curl -s -X POST https://evalhub.newmeasure.ai/api/v1/benchmark-requests \
  -H "Authorization: Bearer $EVALHUB_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name": "SWE-bench Pro"}'
Response
201 { "request": { ... } }

Errors · 429 more than 20 requests in a day.

CLI

The evalh command wraps the same API. Log in with an API key (or set EVALHUB_API_KEY; set EVALHUB_API_URL to point at another host).

evalh login
evalh benchmarks
evalh models
evalh estimate hle -m anthropic/claude-opus-5-5 -r high
evalh run hle -m anthropic/claude-opus-5-5 -r high -o key=value -s <subset> --wait
evalh runs
evalh status <run-id> --wait
evalh download <run-id> -d ./results
evalh balance --ledger

A run can also come from a YAML file with benchmark, model, reasoning, options and subsets:

# run.yaml
benchmark: hle
model: anthropic/claude-opus-5-5
reasoning: high

# then
evalh run -c run.yaml --wait