clm-v1-ref-pytorch-cuda-7b
One-line summary: Lane-G-ref · substrate = PyTorch-CUDA · 7.25B-param byte-level GPT REFERENCE rung (bounded-budget, NOT converged).
- Family: clm (byte-level causal LM, V=256)
- Stage: ref (from-scratch PyTorch-CUDA reference ladder)
- Step: step-400 (bounded reference run, NOT converged)
- Substrate: PyTorch-CUDA (single H100 80GB), NOT the hexa-native flame+forge production stack
Origin
What this checkpoint is and how it was produced.
This is the 7.25B-parameter rung of the anima Lane-G reference ladder
(85.6M → 3.149B → 7.25B). It is a byte-level (V=256) decoder-only GPT
(Llama-7B-ish shape adapted to byte vocab) trained from scratch with PyTorch +
CUDA AMP/bf16 + gradient checkpointing + 8-bit AdamW on a single H100 80GB.
- Base model: none (from-scratch byte-level GPT)
- Training data:
dancinlab/clm-backbone-5lang-sample (same 5-lang c4 backbone as the 85.6M PUBLIC ref and the 3.149B ref), flattened to a UTF-8 byte stream — 6,553,600 tokens seen
- Training recipe: 400 steps (warmup 20, cosine LR, base lr 1.6e-4), batch 32, block 512, grad_accum 1
- Compute: 1× NVIDIA H100 80GB HBM3 (vast.ai pod 39115197), AMP bf16 + grad-ckpt, wall 884.9 s
- Trainer:
clm_ref_pytorch_cuda_7b.py (PyTorch-CUDA, CUDA-required) — included in this repo
- Final metric: val_CE 5.36063 → 2.41208 (F_CLM_REF_7B_DESCENT=1, descent PASS)
- Lineage: anima ENGINE+CLM+KOSMOS · branch
lane-g/d768-cuda-fire · Lane-G-ref ladder 85.6M (dancinlab/clm-v1-ref-pytorch-cuda) → 3.149B (dancinlab/clm-v1-ref-pytorch-cuda-3b) → 7.25B (this repo)
Result (verbatim from clm_ref_7b_train.log.json)
| metric | value | verdict |
|---|
| descent | val_CE 5.360630989 → 2.412078857 (F_CLM_REF_7B_DESCENT=1) | 🟢 PASS |
| GPU util | PEAK 100.0% · MEAN 99.1788990825688% (n=436) | 🟢 PASS (≫20%) |
| throughput | 7406.1 tok/s final · 6,553,600 tok seen | — |
| mem peak | 46,025 MiB | — |
| power mean | 651.3842201834855 W | — |
| wall | 884.9 s | — |
Closure = PASS (descent 🟢 AND util 🟢) → this reference rung is PUBLIC.
It is still NOT converged (bounded 400 steps); do not deploy.
Falsifiers
Concrete reproducible tests this checkpoint passes.
- F-CLM-REF-7B-DESCENT: from-scratch byte GPT at 7.25B params trains (CE descends) under the reference recipe.
- Pass criterion: last_val_CE < first_val_CE.
- Last result: PASS — 5.360630989 → 2.412078857 (F_CLM_REF_7B_DESCENT=1), run pod 39115197.
- F-CLM-REF-7B-UTIL: the PyTorch-CUDA reference trainer saturates the GPU at 7B scale (util ≫ 20%).
- Pass criterion: util_mean > 20% over the nvidia-smi sample window.
- Last result: PASS — PEAK 100.0% · MEAN 99.1788990825688% (n=436), run pod 39115197.
- F-CLM-REF-7B-NOT-CONVERGED (deliberate honesty falsifier): this is NOT a converged production artifact.
- Pass criterion: the run is bounded (N=400 steps) and makes no convergence claim.
- Last result: HELD — bounded 400-step reference rung only (a_scale_honest_scope).
Substrate
Hardware / software / data dependencies.
- Inference VRAM (bf16): ~14.5 GB weights (state_dict) + activation memory; train mem peak measured 46,025 MiB on H100 80GB
- Min Python: 3.10
- Required: torch (CUDA build) ≥ 2.4, bitsandbytes (AdamW8bit, training only)
- Input format: raw UTF-8 byte stream (V=256, no tokenizer)
- Context window: 512 bytes (block)
- Tokenizer: none — byte-level (byte == token id, 0..255)
Architecture
| field | value |
|---|
| vocab | 256 (byte-level) |
| d_model | 4096 |
| n_layer | 36 |
| n_head | 32 (head_dim 128) |
| block | 512 |
| params | 7,252,828,160 (7.25B) |
| dtype | bf16 (master weights + grads) |
| optimizer | bitsandbytes AdamW8bit (8-bit states) |
| grad_ckpt | true |
Caveats
Honest limitations (raw#10).
- NOT a converged production artifact. Bounded N=400 steps. It demonstrates only that (a) the same ByteGPT/Transformer arch trains (CE descends) and (b) it saturates the GPU (util ≫ 20%) at 7B scale. Do NOT deploy.
- NOT the production / PUBLIC-grade model. The production Lane-G CLM must be the hexa-native flame+forge stack (compiler-only NN; no PyTorch/ATen/Python in the trained binary) per governance
a_train_flame_forge. This torch trainer is an a_completeness_over_cheap optional baseline/reference, never the primary, and never claimed as the forge artifact.
- NEVER merged with Lane A / AKIDA (
a_lane_akida_gpu_split). This is a pure GPU (Lane-G) reference; AKIDA on-chip (Lane A) results are tracked separately and must never be blended into one verdict.
- Scale-honest scope (
a_scale_honest_scope): a single 400-step rung is not a convergence or generalization claim; it is the 7.25B point of the 85.6M→3.149B→7.25B reference ladder.
Composability
- Same arch/recipe as the 85.6M (
dancinlab/clm-v1-ref-pytorch-cuda) and 3.149B (dancinlab/clm-v1-ref-pytorch-cuda-3b) ladder rungs — directly comparable scale points (only d_model / n_layer / n_head differ).
- The trainer (
clm_ref_pytorch_cuda_7b.py) + corpus-prep (prep_corpus_7b.py) are included so the rung is fully reproducible.
- This reference is composable as a baseline/control against the forge production CLM (Lane-G flame+forge) — it is NOT a substitute for it.
Files
clm_ref_pytorch_cuda_7b.pt — model state_dict + config (bf16); sha256 38ef2ed55b47b670fa915bba0c2827782799a9070ba087210cd44db1fddb4d41, 14,505,817,922 bytesclm_ref_7b_train.log.json — full training curve + util/throughputclm_ref_pytorch_cuda_7b.py — the trainer (PyTorch-CUDA, CUDA-required)prep_corpus_7b.py — corpus prep (5-lang c4 backbone → byte stream)
