≈8.3% less first-passage train time than F1_seal (3.866 h) — 3/3 seeds

The metric

Noise floor: measured stack drift ~6% step-time. Single-seed |ΔT%| < ~6%: do not publish as a sealed under-baseline claim. Sealed candidates all ≥6%.

Challenge mapping: CHALLENGE_TERMS.html · clock rules: FAIR_CLOCK.html.

What we tried

Scale: 13 workspaces · each has a nocap volume · 153 runs/ entries (incl. partials). Codes: Clock-W, Claim-W1/W2, Research-W1…W7, Archive-W1…W3.

Workspace roles

Validated into the claim recipe: Keller Jordan Muon · B32×accum16 · timing-cheap split-QKV · frozen unigram prior · parallel + fastpath · WSD wu476/wd1192 · z-loss 0 · NS 5 · raw FineWeb · EMA off.

What the numbers say

Three sealed candidates first-cross the quality bar; each is ≥6% under sealed F1_seal on the matched A100/torch pin. Mean ΔT% ≈ +8.3%. Published % is vs F1_seal — not vs joint stock (matched first-cross ΔT% is N/A).

Matched pairs

Published % compares candidate T to F1_seal, not to joint stock. Seed 2029 vs F1_Claim-W2: +7.5%. Canon: NUMBERS_CANON.html. Short scoreboard: preview-schema.html.

Compute / GPU-hours

Three tiers, three clocks. Do not collapse to one GPU-hour number. Tiers 1–2 measure trainer-internal train_time_ms. Tier 3 measures Modal billed A100 container time (train_time_ms ≠ billed).

Which number to cite? Claim speed uses sealed train_time_ms (F1_seal / candidates) — not these campaign totals. For campaign spend/effort: quote the triple 72.03 / 110.18 / 147.91 and name the clock.

Ratios. Billed all-apps vs volume: 1.34× (147.91 / 110.18). Billed NoCap vs volume: 1.16× (128.28 / 110.18). Billed vs local: 2.05×. Local is a pulled subset of volume — not apples-to-apples with billed.

Why tiers diverge. Local < volume: not all volume runs pulled; volumes hold more theory / archive / claim clocks. Billed > volume: container wall clock (image/setup, CUDA init, failed/killed partials without usable CSV, idle while container held) vs trainer-loop train_time_ms. Archive notebook-kernel A100 (~19.63 h) drives most of all-apps vs NoCap-named.

Billed method. modal billing report --show-resources --json; keep A100-40GB rows; gpu_hours = cost_usd / (0.000583 $/s × 3600) (public list price). Report $ is pre-credit. Assumes base-region preemptible; region/non-preempt multipliers would overstate hours.

Storage (volume bytes)

Across 13 nocap volumes: ~302 GiB under runs/ (almost all .pt checkpoints; text logs ~2 MiB), ~56 GiB sum of data/ (~5.6 GiB unique FineWeb pack replicated on many workspaces), ~420 GiB grand total.

Caveats. Tiers 1–2: train_time_ms is trainer-loop wall-clock — excludes idle GPU outside the timer, container spin-up, failed spawns with no CSV; partials at measured max only; volume may double-count partial+resume dirs. Tier 3: not native GPU-hours; notebooks and non-NoCap apps inflate all-apps 147.91 vs NoCap 128.28; local 72.03 h is not comparable 1:1 to the billed window. Near-duplicate local logs with different bytes count separate.

Sources: GPU_HOURS_LOCAL.md · internal volume inventory · Modal billing export.

Failures

Hard non-claim. No matched candidate-vs-stock first-cross ΔT%. Stock 1337/2029: never crossed. Seed 4242: stock arm not run. Public % vs F1_seal only.

Full ledger: FAILURE_LEDGER.html.

Recipe

Claim-stack v1. Seeds 1337, 2029, 4242. Stock control = same stack with joint QKV (muon_split_qkv=0).

Model / data

GPT-2 d12 · raw FineWeb · seq 1024 · official disjoint val

Batch

B32 × accum16 · 524,288 tok/update · 4768 steps

Muon

LR 0.02 · mom 0.95 · WD 0.1 · NS 5

Adam (embed/lm_head)

LR 0.0018 · WD 0.1

Schedule

warmup 476 · warmdown 1192 · zloss 0 · frozen unigram prior

Candidate delta

muon_split_qkv=1 · match_joint=0

Environment

A100-SXM4-40GB · torch 2.13.0+cu130 · 1 GPU · bf16 · compile on

Attribution

Keller Jordan Muon + campaign timing-cheap split-QKV (B32 systems path)

Pin card: RECIPE_CARD.html. Narrative source: COMPREHENSIVE_REPORT.md.

Verification + links

Recompute claim ΔT% from public data/*.csv: gate → clock → cross. Implementation and run logs: challenge repository · kit/ · claim-stack v1 · reproduce notes · F1_seal logs. (Repo access may be required until the challenge tree is opened.)

Worked example — s2029_cand.csv: step 4736, train_time_ms = 12,840,827.6 → T_h = 3.5669 → ΔT% = +7.7%.

What verify checks vs what literature teaches. Recompute above is CSV arithmetic only. Optimizer and layout primers (Muon, B32, split-QKV) do not set the sealed %. Campaign totals under Compute are inventory clocks, not claim ΔT%.

Future research

The sealed 3/3 result measures the full claim stack—Muon routing, B32×accum16, WSD, and timing-cheap split-QKV—against sealed F1_seal on a fixed A100/torch pin. The reported ΔT% reflects that combined recipe; it does not isolate which component drove the wall-clock gain. Where measured, joint-QKV stock never crossed; seed 4242 has candidate only. Matched candidate-vs-stock first-cross ΔT% remains N/A.

Verified adjacent work on Practical Efficiency of Muon, WSD schedules, small-batch and gradient accumulation, AlgoPerf time-to-result evaluation, and AdamW motivates tighter controls but does not validate this stack. Priorities:

Compute tiers are separate. Campaign totals 72.03 / 110.18 / 147.91 h are inventory clocks (train_time_ms ≠ billed) — see Compute.

Muon hedge. Attribution: Keller Jordan Muon + timing-cheap split-QKV (B32 systems path). Sealed ΔT% is the claim-stack, not Muon alone. Public sealed kit: TRAINER_DELTA.html · kit/ (this site) · kit/ on GitHub ↗.