Challenge terms — official rules vs this campaign
Audience: strangers auditing whether our numbers match the BottleCapAI benchmark.
Official source: BottleCapAI/NoCap-Test (master README).
Our measurement: FAIR_CLOCK.html · GLOSSARY.html
What BottleCapAI asks for
Open benchmark (Modded-NanoGPT fork) on one GPU (reference: RTX 4090).
| Official rule | Summary |
|---|---|
| Optional goal | val loss ≤ 3.3821 on FineWeb faster than stock baseline. |
| Non-4090 | Benchmark baseline on your setup; report speedup vs that baseline. |
| Fair comparison | Same treatment for baseline and candidate (e.g. don’t raise LR only on candidate). |
| Documentation | IDEA.md with motivation, method, results. Negative results welcome. |
| Submission | git bundle create <first>-<last>.bundle --all → [email protected] (speedup % in subject; negative OK). Beat baseline → add RESULTS.md. |
| Status | First competition finished Nov 2025; self-paced submissions still welcome. |
They want algorithmic ideas, not LR-only tuning or copied Modded-NanoGPT tricks. Sub-baseline runs with a clear idea still matter.
Metric: cross-entropy on FineWeb val (we log val_disjoint). README allows other sequence lengths if the full val set remains a valid probability model.
How this campaign maps those rules
| Official | Our mapping |
|---|---|
| Quality ≤ 3.3821 | First logged val_disjoint ≤ 3.3821 (horizon 4768). GLOSSARY. |
| Speed vs baseline | First-passage train time T from train_time_ms at first-cross (compile warmup in). FAIR_CLOCK. |
| Non-4090 rule | Denom = our sealed stock AdamW on pinned stack — not the 4090 leaderboard row. |
| Fair comparison | One sealed denom; no matched speedup vs joint stock where stock missed quality. |
| IDEA.md | Campaign docs + FAILURE_LEDGER.html. |
| Bundle | Campaign packaging for organizers. |
Public denominator
| Label | Workspace | Pin | First-cross T |
|---|---|---|---|
| F1_seal | Clock-W | Stock AdamW · A100-SXM4-40GB · torch 2.13.0+cu130 | 3.866 h @ 4736 |
ΔT% = 100 × (3.866 h - T_candidate) / 3.866 h
Current claim (honest)
- Candidate: claim-stack v1 (Muon B32 + timing-cheap split-QKV), seeds 1337 / 2029 / 4242.
- vs F1_seal: 3/3 under denom — +8.1% / +7.7% / +9.2%; mean ≈ +8.3% (~3.544 h).
- Under our sealed F1 baseline (not Nov 2025 prize framing).
- Not matched candidate-vs-joint-stock speedup (stock missed; 4242 no stock arm).
- Submission pending to [email protected].
Attribution for any submission: Muon (Keller Jordan); timing-cheap split-QKV from this campaign’s bank.
Non-claims
- 4090-table speed without re-benchmarking 4090 baseline.
- Cardinal % from 800-step proxies or historical notebook F1 (~229.1 min).
- “Beat joint stock” — stock missed on matched pairs.
Copy-safe wording: FAIR_CLOCK § non-claims.
Related
| Doc | Use |
|---|---|
| GLOSSARY.html | Terms (val_disjoint, F1_seal, workspaces) |
| disjoint-val-explained.html | Pedagogy: held-out val · bar · first-passage |
| FAIR_CLOCK.html | Speed metric, noise floor, pin |
| CAMPAIGN_OVERVIEW.html | Narrative |
| DOCS_INDEX.html | Full catalog |
Live: https://nocap.think2earn.com