NoCap campaign — comprehensive report

Evidence state: 2026-07-16 (seals) · volume inventory 2026-07-17
Purpose: submission packaging — methods, timeline, failures, seals, components
Status: multi-seed under-baseline vs sealed F1. Under our sealed F1 baseline. Matched cand-vs-stock ΔT% = N/A. Public % vs F1_seal only.

Sources: NUMBERS_CANON.md, nocap-repo/results/claim_stack_v1/MULTI_SEED_CLAIM.md (+ seed summaries / PREREG.md), FAILURE_LEDGER.md, CAMPAIGN_TIMELINE.md, CAMPAIGN_RUNS_OVERVIEW.md, VOLUME_INVENTORY.local.md, CHALLENGE_TERMS.md, FAIR_CLOCK.md, RECIPE_CARD.md.

Verdict

On three independent seeds (1337, 2029, 4242), the claim-stack candidate (Muon B32 + timing-cheap split-QKV) first-crosses val_disjoint ≤ 3.3821 about 8% faster than the sealed stock AdamW F1 baseline (3.866 h; mean ΔT% ≈ +8.3%), with terminal CE ≈ 3.37. Where measured, the joint-QKV stock control misses the quality bar.

Local seals are campaign evidence. They are not organizer acceptance.

Challenge terms

Official source: BottleCapAI/NoCap-Test (master README).

They want algorithmic ideas, not LR-only tuning or copied Modded-NanoGPT tricks.

Mapping

Fair clock

Published speed metric = first-passage train time:

1. Train on sealed recipe / pin.
2. Log official val_disjoint (claim-stack: every 128 steps).
3. First step with val_disjoint ≤ 3.3821.
4. Read train_time_ms at that row → T = train_time_ms / 3.6×10^6.
5. Compare to public denominator:

ΔT% = 100 × (3.866 h - T_M) / (3.866 h)

Positive ΔT% = fewer hours to first hit the quality bar.

Pin (required for cardinal %)

Not the metric

Noise floor

Measured stack drift ~6% step-time is the empirical prior for single-seed noise. \|ΔT%\| < ~6% (single seed): do not publish as win. Claim-stack sealed candidates all ≥6%.

Recipe (claim-stack v1)

Hardware / software

Shared recipe (both claim-stack arms)

Candidate vs stock control

Stock control = joint-QKV Muon stack, not AdamW F1.
F1_seal / F1_Claim-W2 are separate AdamW denominators for under-baseline ΔT%.

Attribution: Muon → Keller Jordan. Timing-cheap split-QKV → this campaign’s banked mechanism.

Sealed denominators

Exact F1_seal ms @ cross: 13,918,667.5 → CSV hours ≈ 3.866297; publish 3.866.
Public headline ΔT% uses F1_seal. Seed 2029 may also report vs F1_Claim-W2 (+7.5%).

Do not publish % vs historical notebook F1 (~229.1 min).

Claim-stack seals (3/3)

Aggregates: mean cand T ≈ 3.544 h · mean ΔT% ≈ +8.3%. All three candidates first-cross at step 4736.

Explicit matched-pair statement

Published under-baseline percentages compare candidate T to F1_seal, not to joint stock.

Pass checklist (claim-stack era)

1. Candidate first-crosses ≤ 3.3821 — 3/3
2. Candidate T vs F1_seal ≥ 6% — 3/3 (+8.1%, +7.7%, +9.2%)
3. Terminal CE cand ≤ stock — where stock ran
4. Soft: stock may miss — observed on 1337 and 2029
5. Same-workspace F1 on Claim-W2 for seed-2029 ratios

Ops notes (non-metric)

• Fire-and-forget spawn() after nested .remote() cancels killed earlier work.
• Seed 4242: first spawn rejected (A100-SXM4-80GB). Seal is relaunched 40GB.

Timeline (causal order)

1. Early waves → failed finals

Absolute quality bar: disjoint val ≤ 3.3821 on GPT-2 ~124M / FineWeb, one A100.

2. Fair-clock reform → sealed F1 denominator

Pin: A100-SXM4-40GB · torch 2.13.0+cu130. Metric T = train_time at first logged disjoint val ≤ 3.3821.
F1_seal on Clock-W: 3.866 h. Public speed denominator.

3. Failed full claims (pre–component bank)

Board-v3 800-step screens: no promote.

4. Component bank (research workspaces)

• B32×accum16: ~6% faster core step-time vs B16 (tokens/step fixed at 524,288).
• Timing-cheap split-QKV (block NS5, match_joint=0): multi-seed MECHANISM-REPEAT @800 · mean terminal gain ≈ +0.015 CE · time ratios ≤ ~1.01. Persistence sealed @2400 on Research-W4 seed 5151 · +0.0147 CE · no late reverse.
• Through 2026-07-14 evening: under-baseline complete recipes still 0. Bank existed; full claim did not.

5. Tier A hparam screens — all NO-GO

Defaults kept: mom 0.95, Adam WD 0.1, warmdown 200 (proxy; full claim uses wu476/wd1192).

6. Claim-stack seals (3/3 candidates under F1)

Recipe: Muon B32 + timing-cheap split-QKV candidate vs joint-QKV stock control. Full 4768. Exact A100-40 + torch 2.13. Result: 3/3 under F1_seal (candidate vs F1, not matched stock speedup).

Causal spine (one line)

Failed finals → fair clock + F1_seal → Muon compose misses → bank systems + cheap split → Tier A keeps defaults → claim-stack 3/3 under sealed F1.

Negative results (failure ledger)

Hard non-claim (claim-stack era): no matched candidate-vs-stock first-cross ΔT%.

What was tried (family level · all workspaces)

Scale: 13 workspaces · each has a nocap volume · 153 runs/ entries (incl. partials). Codes: Clock-W, Claim-W1/W2, Research-W1…W7, Archive-W1…W3.

Workspace roles (sealed narrative)

Volumes do not share across workspaces.

Components banked into the claim recipe

Explicit non-claims

1. Local seals ≠ organizer acceptance.
2. No matched candidate-vs-stock first-cross speedup. Joint stock missed quality on 1337/2029; 4242 had no stock arm.
3. No % vs historical notebook F1.
4. No board % from 800-step proxies.
5. Candidate recipe ≠ joint stock. Headline compares candidate to AdamW F1_seal.
6. Do not inflate means beyond sealed rounding.

Compute

Three tiers — different clocks. Workspaces: Clock-W / Claim-W* / Research-W* / Archive-W*. Local artifacts only · do not deploy. No GPU launches.

Ratios (same campaign window). Billed all-apps vs volume: 1.34× (147.91 / 110.18). Billed NoCap-apps vs volume: 1.16× (128.28 / 110.18). Volume vs local: local is a pulled subset.

Why the tiers diverge. Local < volume: not all volume runs were pulled. Billed > volume: Modal bills container wall clock (setup, CUDA init, failed/killed partials, notebook-kernel GPU); train_time_ms is trainer-internal only. Archive-W* notebooks (~19.63 h) drive much of the all-apps vs NoCap gap.

Billed method. modal billing report --show-resources --json per workspace code; keep resource == "A100-40GB"; gpu_hours = cost_usd / (0.000583 $/s × 3600). NoCap split = app description contains nocap. Report $ is pre-credit; hours assume base-region preemptible list price.

Storage (volume bytes)

Across 13 nocap volumes: ~302 GiB under runs/ (almost all .pt checkpoints; text logs ~2 MiB), ~56 GiB sum of data/ (≈5.6 GiB unique FineWeb pack, replicated on many workspaces), ~420 GiB grand total. Source: VOLUME_BYTES.local.md (local-only · do not deploy).

Local family rollup (from pulled CSVs)

Caveats. Tiers 1–2: train_time_ms is wall-clock inside the training loop as logged — excludes idle GPU, container spin-up, failed spawns without CSV; partial runs at measured max only. Tier 3: not a native GPU-hour field; derived from resource cost; region/non-preempt multipliers would overstate hours; local 72.03 h is not apples-to-apples with full billed window. No GPU reservation or deploy in the production of this report.

Submission packaging notes

Bundle / email: pending. Next packaging step is writeup + organizer submission — not more science GPUs unless an operator asks.

Related local paths