SWE-bench Multimodal

History

JavaScript issues with visual context. Docker harness required.

This benchmark is indexed in the Benchlist registry. We're working on a deeper guide, request priority coverage → if you'd like this article expanded.

How SWE-bench Multimodal is graded

The canonical run uses temperature 0.0 and max_tokens 8192. 619 problems graded by accuracy. License: CC0-1.0.

Common pitfalls when reporting SWE-bench Multimodal

The same number can mean very different things depending on how it was produced. The biggest failure modes specific to this benchmark:

• See /methodology for general benchmark-reporting pitfalls.

Live Benchlist leaderboard

Top attested scores from the Benchlist registry, hydrated client-side from /api/runs.json. Self-reported numbers are de-prioritised, attested results from a real signed transcript always rank above vendor-disclosed ones.

How to ship a SWE-bench Multimodal score that nobody can challenge

Hosted runner, POST a job and we email the verify URL when it's done:

curl -X POST https://benchlist.ai/api/v1/run \
  -H "Authorization: Bearer $BENCHLIST_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "service": "anthropic-claude",
    "model": "claude-sonnet-4.5",
    "benchmark": "swe-bench-multimodal",
    "runs": 1,
    "limit": 50,
    "proof_system": "signed",
    "inference_api_key": "managed"
  }'

Self-hosted, install benchlist-runner via pip, point it at your inference key, get a signed run.json:

pip install benchlist-runner
benchlist run swe-bench-multimodal --service anthropic-claude --model claude-sonnet-4.5 --limit 50
benchlist publish run.json

FAQ