SpecSnap Multi-Session Agent Snap Benchmark
Protocol GitHub Repo ↗
SPECSNAP BENCHMARK · CANARY 26B5C67B
3 MODELS · 4 TIERS
Empirical Evaluation

What survives when a coding agent's session is snapped mid-task?

Unlike open-ended repo sweeps where an agent hunts for unknown bugs, SpecSnap gives the agent an explicit multi-package specification in chat, snaps the session mid-edit while all visible tests already pass, and measures whether the specification survives across fresh sessions and superseding requirement waves.

1. IN-CHAT SPECIFICATION Session 1 receives up to 24 exact requirements across 12 packages that exist only in the live chat (not in TASK.md or visible tests).
2. MID-EDIT SNAP PLANE The harness snaps Session 1 immediately after its first .py edit on ledger/rules.py. Visible pytest -q already passes on the untouched packages.
3. WAVE SUPERSESSION & VERIFICATION Later sessions boot with zero chat memory, receive superseding Wave 2/3 contract amendments (or planted regressions at the Snap Plane), and are graded by a hidden 141-assertion test suite.
NET SPECIFICATION RETENTION (% OF REQUIREMENTS KEPT)
Showing Overall SpecSnap Score (70 total requirements across Tiers 1-4). Click any bar to inspect run notes below. Tuned reference is not held out (rules revised after State.md failures).

02 Interactive 3D Snap Visualizer

See exactly where the Snap Plane slices the session and which packages survive across resets.

3D SNAP STAGE · DRAG TO ORBIT · SCROLL TO ZOOM · CLICK ANY POD OR COLUMN TO INSPECT
Tier
Model
Arm
HOW TO READ THIS VIEW: Package 01 (ledger) sits on the left of the orange Snap Plane (where Session 1 starts and gets cut off). Packages 02-12 sit on the right (what Session 2+ must recover). Green = survived; Red/Dark = lost after the snap.
TIER 4 MONOREPO · 12 PACKAGES · 72 MODULES
Session 1 edits ledger/rules.py and is snapped before commit.
Active RunClaude · State.md
Raw / Net Retention10 / 36 (27.8%)
TES Efficiency13.9%
Turns / Calls44 (5+10+10+19)
Total Tokens3,491,572
Survived / Verified Partial / Overwritten Wave Lost at Snap / Trojan Bug The Snap Plane
Phase 3 · Final Graded Tree (After Snap)
Click any 3D pod or bar to load its run notes in the explorer below.

03 Benchmark Data & Run Explorer

Switch tabs below to inspect Tiers 1-4, Planted Regressions, Real Repositories (Click & isort), or Caveats & CLI Quickstart without page clutter.

a Gemini Tier 3 Vanilla: 1 requirement passed raw, but broke original baseline behaviour and 1 module boundary (0 net).

b Claude Tier 3 AGENTS.md Only (14/24): Not caused by AGENTS.md. Session 1 wrote a 230-line patch script (_patch1.py) covering 7 packages and was snapped as it created it; Session 2 found and ran it.

c Gemini Tier 4 Tuned reference (36/36): Ran the standardized scaffold_held_out.py template (<= 1,200-token state.md + app_map.md); an earlier 500-token cap variant scored 30/36 (1.92M tokens, 76.1% TES).

d Tier 4 State.md: Gemini (35/36), Claude (27/36), and Grok (25/36) all ran the standardized post-2d82680 CLI Rule 4 (in-place wave preservation). An earlier pre-2d82680 Claude run ("Write ONLY the newly requested sprint requirements") overwrote Wave 1 and scored 10/36 (3.49M tokens, 13.9% TES).

e Grok Tier 4 State.md (25/36): Ran after commit 2d82680 with the updated in-place Rule 4 and deferred git init.

f Grok Tier 4 Tuned reference (0 / 33): First version passed 36/36 raw, but 1 boundary violation zeroed net retention. Second version scored 33/36 with zero violations.

inject-poison-pill runs right at the Snap Plane between Session 1 and Session 2 on Tier 2 (tier_2a_fintech). It plants two defects next to Session 1's valid uncommitted ledger/rules.py edit, and both pass the visible tests:

  • Silent regression in billing/tax.py: a 1-cent tax error on amounts over 2,500 cents (raw_tax - 1).
  • Architectural violation in rbac/rules.py: a forbidden from rbac import audit import violating wiring.yaml.
Variant Model & Arm Prompt Raw Net (of 8) TES Tokens

Uncommitted defects in git diff (v1 & v2) get caught (7-8/8). When amended into HEAD (v3), Claude and Grok without Gate I score 0/8 because billing/tax.py is invisible to git diff (while Gemini's S2 read all 24 package files and caught it at 1.51M tokens). Adding per-module SHA-256 verification in app_map.md (Gate I) caught it across all 3 models (5/5 runs, 8/8 net).

To test whether synthetic retention gains carry over to open-ended bug-finding on real codebases, Claude Opus 5.5 and Grok Build 4.7 were run across two sessions on pallets/click@8.0.1 (13 hidden bugs) and pycqa/isort@5.10.1 (11 hidden bugs from later upstream issues).

  • Without hints, memory scaffolds spent 2.1-5.2× Vanilla's tokens on Claude (1.1-2.0× on Grok) for 0-1 extra bugs: memory preserves specifications you were given; it does not discover unknown bugs.
  • Click v3 (3/13 on Claude, 12/13 on Grok) is a hinted upper bound: its state.md backlog was hand-written and named 6 of the 13 bug areas.
Repository Arm Bugs Resolved Predicted Turns (S1+S2) Total Tokens vs Vanilla
REPRODUCIBLE CLI QUICKSTART · SPECSNAP
PYTHON 3.10+ · PYTEST 
# 1. Create a clean, git-initialised sandbox outside the repo (arms: Vanilla, AGENTS_MD_Only, State_MD_Only)
python -m specsnap.cli scaffold-sandbox --tier 3 --arm State_MD_Only --dest /tmp/specsnap_t3_state

# 2. Run your agent's sessions in that directory following EXTERNAL_AGENT_PROTOCOL.md
#    (Snap Session 1 right after the first .py source edit, before it is committed)

# 3. Optional: plant the Tier 2 regressions at the Snap Plane between Session 1 and Session 2
python -m specsnap.cli inject-poison-pill --tier 2 --sandbox /tmp/specsnap_t2_state

# 4. Grade the finished sandbox: raw retention, net retention, boundary violations, and TES
python -m specsnap.cli evaluate-sandbox --tier 3 --sandbox /tmp/specsnap_t3_state --tokens 1107107 --json-out /tmp/scorecard.json

# 5. Print the recorded benchmark tables (table, markdown, or json)
python -m specsnap.cli leaderboard --format markdown