🔬 Trainsect — git bisect for training runs

How it works — causal replay-bisection

1. Deterministic replay. A training step is a pure function of (model, optimizer, scheduler, batch, RNG) state. Checkpoints store all of it with pre-update semantics. Before trusting any bisection, Trainsect replays a segment twice and compares SHA-256 parameter hashes — if they disagree, bisection halts.
2. Collapse predicate (eval-based, never train-spike). A checkpoint counts as collapsed only by its eval loss. Transient train-loss spikes on a healthy model never count.
3. Checkpoint binary search. Predicate P(c) = "checkpoint c is healthy AND replaying forward reproduces the collapse" is monotone, so binary search brackets the trigger to one checkpoint interval in O(log n) replays.
4. Counterfactual shard bisection. Replay the trigger interval with each candidate shard's batches replaced by benign substitutes. The survivor must pass necessity (collapse vanishes without it) and sufficiency (collapse reproduces with only it).
5. Evidence report. JSON + human-readable report with minimal-repro commands, so anyone can re-run the proof.

Honest scope + limitations

• CPU, deterministic, single-process stacks only. GPU replay is expected to fail the bit-exact fidelity check (nondeterministic kernels) — the tool aborts honestly rather than guessing.
• Forensic mode never names unproven batches. Without recorder data, exact batch/step identity is unrecoverable; an honest bracket with a confidence interval beats a precise-sounding guess.
• Distributed training (DDP/FSDP) is not supported.

Forensic mode — checkpoints alone

diagnose takes the forensic path automatically when a directory lacks recorder data. It reconstructs batch orders from the known pool under multiple seeds (3 for the bracket search, 8 for confirmation) and names a shard only if necessity + sufficiency + specificity all hold and frozen-weight loss ranking agrees. Otherwise it reports UNPROVEN with what evidence would settle it. Forensic mode never sets a batch fingerprint or trigger step, and its verdict is never CONFIRMED (causal).

Measured on the two demo runs, stripped of recorder data: the P0 validation run (MLP) diagnoses FORENSIC (probable), shard-2, bracket [40, 80) — collapse reproduces under 8/8 reconstructed orders. The transformer demo diagnoses UNPROVEN, no shard named — its collapse is concentration-dependent and reconstructed orders cannot reproduce it, so no claim is made.

Methodology — the novel claim

Every existing training-debugger bisects the wrong axis:

• Code axis — git bisect (and rldk's bisect) finds the bad commit by re-running training per commit.
• Release axis — culprit bisects checkpoint releases across runs (robotics inference stacks).
• Approximate axis — TRAK / influence functions / TracIn assign scores to samples via linearization, projection, or first-order Taylor approximations.

Trainsect bisects the training-time axis itself: it treats the optimizer trajectory as a deterministic, replayable computation, reloads optimizer state at step t, replays forward, and binary-searches the exact step-range where the collapse triggers. Then it bisects the data-shard axis with counterfactual replay (collapse disappears iff shard S is excluded) to name the exact shard, batch, and sample.

The output is not a ranked hypothesis. It is a falsifiable, re-runnable minimal repro. No existing tool combines (a) deterministic optimizer-step replay across checkpoints, (b) binary search over the checkpoint timeline, and (c) causal data-shard bisection.

Deterministic replay

A training step is deterministic given complete state: θ_t+1, opt_t+1, rng_t+1 = STEP(θ_t, opt_t, batch_t, rng_t). Checkpoints capture model + optimizer + scheduler state; the one-line recorder hook captures dataloader cursor, batch sequence, RNG states, worker seeds, and determinism flags. Before trusting any bisection, Trainsect replays t → t+k twice and compares SHA-256 parameter hashes — if they disagree, bisection halts and reports the divergence.

Collapse detection

Two stages. Stage A is cheap and runs on logs: change-point detection on the eval-loss curve (metric predicate + binary segmentation). Stage B is causal and runs on replay: P(c) = checkpoint c is healthy (its own eval loss below threshold) AND replaying forward c → B reproduces the collapse. The predicate is eval-based, not train-loss-spike-based — a healthy model can show transient train-loss spikes that a spike-based predicate mistakes for collapses. Eval loss is the collapse signal; per-step train loss pinpoints the trigger batch inside an already-established collapse bracket.

Checkpoint binary search + shard bisect

Checkpoints at steps {c₀ < c₁ < … < c_n}, known-bad point B. P(c) is monotone in c, so binary search brackets the trigger to one checkpoint interval in O(log n) replays. Then counterfactual shard bisection: replay the trigger interval with each candidate shard's batches replaced by benign substitutes from a known-clean shard. The survivor must pass necessity and sufficiency. Batch → sample: within the trigger batch, the offending sample is the one whose per-sample loss dominates the batch. The evidence report ships minimal-repro commands for all three counterfactuals.

Instrumented mode — the one-line recorder hook

start_recording(...) / record(step, batch, ...) / stop_recording(...) — record() is called after the step's update, preserving pre-update checkpoint semantics. Measured overhead 0.7–1.0% of step time. A hook-recorded run replays bit-exact.

Gap verification (2026-10-14)

Checked against rldk (bisects code commits), culprit (bisects checkpoint releases across runs, robotics), merriv (bisects precomputed labels), NeuralDBG (in-loop hypotheses without replay), and TRAK/influence functions (approximate scores, not causal replay). Manual data bisection exists only as practitioner folklore. Nothing does deterministic optimizer-step replay + checkpoint binary search + data-shard bisect.

Full methodology: docs/P0-methodology.md in the repo.