2,000 photos · 671 species and subspecies
How well can AI tell spiders apart? This repository contains the frozen tasks, model answers, scores, and Python CLI behind the experiment. Every model sees the same photos and the same 20 possible names. The best run got 49.85% right.
Read the story · Run it yourself · Full leaderboard · Benchmark protocol
Accuracy is correct species / 2,000 photos. Failed and missing answers count as incorrect.
Every score, cost, and run file
| Model | Effort | Accuracy | Correct / 2,000 | Failures | Est. USD | Evidence |
|---|---|---|---|---|---|---|
| Gemini 3.8 Flash | high | 49.85% | 997 | 4 | $27.11 | Run |
| Gemini 3.8 Flash | medium | 48.45% | 969 | 0 | $8.66 | Run |
| GPT-6 Astra | medium | 47.70% | 954 | 3 | $56.63 | Run |
| GPT-6 Astra | high | 47.50% | 950 | 4 | $97.53 | Run |
| Claude Fable 5.1 | high | 43.10% | 862 | 2 | $54.19 | Run |
| Claude Fable 5.1 | medium | 41.75% | 835 | 3 | $33.65 | Run |
| Muse Spark 1.3* | high | 39.10% | 782 | 0 | $0.93 | Run |
| Muse Spark 1.3* | medium | 38.75% | 775 | 0 | $0.79 | Run |
| GLM 5.3 Flash | high | 36.75% | 735 | 3 | $1.46 | Run |
| GPT-5.6 Sol | high | 33.65% | 673 | 1 | $42.89 | Run |
| GPT-5.6 Sol | medium | 32.65% | 653 | 1 | $25.89 | Run |
| DeepSeek V4.1 Flash | high | 26.40% | 528 | 92 | $8.52 | Run |
| GPT-5.6 Terra | medium | 23.30% | 466 | 0 | $14.37 | Run |
| GPT-5.6 Terra | high | 22.80% | 456 | 0 | $20.53 | Run |
| GPT-5.6 Luna | medium | 21.95% | 439 | 48 | $1.44 | Run |
| GPT-5.6 Luna | high | 21.50% | 430 | 45 | $4.99 | Run |
Estimated cost per 2,000-photo run: $0.79–$97.53. The dollar axis is logarithmic.
Muse Spark 1.3 uses discounted Contributor pricing, with provider training and data use enabled.
Each bar covers all 2,000 photos. “Same family” excludes same-genus answers; “No valid answer” includes failed and missing responses.
Download the numeric chart data.
Python 3.12+.
git clone https://github.com/qforge-dev/spider-bench.git
cd spider-bench
python3 -m venv .venv
source .venv/bin/activate
pip install -e .
spider-bench benchmark modelsProvider setup and pilot run · Reproduce the 2,000-photo comparison · CLI guide
A spider escaped. Call security.
Fortunately, we have a specialist for this.
Species identification: inconclusive. Spider removal: enthusiastic.
Code: MIT. Photos: individual licenses and attribution.