You can deploy this model and start earning money today!

sahilj2701/
race-nanochat-d24

Name: race-nanochat-d24
Availability: InStock

Views

No views yet

RACE-nanochat d24 (base)

A full-RACE-attention GPT (depth-24, ~1.384B params) pretrained with nanochat's base pipeline, with every attention layer replaced by RACE attention (a hash-bucket linear-attention mechanism).

Params: 1,384,122,122 | layers: 24 | n_embd: 1536 | heads: 12
Context length: 2048 | vocab: 32768
Training: 5.84B tokens (ClimbMix), bf16, global-causal (window_pattern=L), no fp8
Step: 5568 (final)

Usage

This is a base (not chat-tuned) model — it continues text. Run it with one command from the companion repo branch:

bash

git clone -b race-chat https://github.com/sahiljoshi515/RACE_Attention.git
cd RACE_Attention/chat && bash run.sh

The launcher downloads these weights, JIT-builds the RACE CUDA kernel (needs an NVIDIA GPU + CUDA 12.x), and drops into an interactive prompt.

Files are laid out to match nanochat's NANOCHAT_BASE_DIR: base_checkpoints/d24/model_005568.pt, .../meta_005568.json, tokenizer/tokenizer.pkl, tokenizer/token_bytes.pt.

sahilj2701/race-nanochat-d24

RACE-nanochat d24 (base)

Usage

sahilj2701/
race-nanochat-d24