A 10.1M-parameter decoder-only base language model trained from scratch on a deterministic FineWeb-Edu subset.
This is a small educational and experimental model. It is not a production assistant and should not be relied on for factual, medical, legal, financial, safety-critical, or other consequential advice.
Model details
Property
Value
Parameters
10,096,896
Layers
8
Hidden width
256
Attention heads
4
SwiGLU hidden width
960
Vocabulary
8,192 BPE tokens
Context window
1,024 tokens
Architecture
Decoder-only Transformer
Position encoding
RoPE
Normalization
RMSNorm
Embeddings
Tied input/output embeddings
The architecture was implemented from scratch in PyTorch. This repository includes custom Transformers-compatible configuration and modeling files.
Training
The base model was trained from scratch on 200,015,872 packed tokens selected from the FineWeb-Edu sample-10BT corpus. Documents were selected deterministically by hashing document IDs rather than taking the first source shards. Text was tokenized with a locally trained byte-level BPE tokenizer, an <eos> token was appended after every document, and tokens were packed into fixed 1,024-token blocks.
Base training used AdamW, cosine learning-rate decay, gradient clipping, bfloat16 autocast, and a global batch size of 65,536 tokens on an NVIDIA RTX 3090.
Evaluation
Best validation loss: 3.5592 (next-token validation).
These losses are internal held-out validation measurements. Base and SFT losses use different objectives, and losses from the 8K and 16K tokenizers are not directly comparable. No broad academic benchmark suite or human preference evaluation was run.
Usage
Install the dependencies:
pip install -r requirements.txt
Run the included example after cloning this repository:
python generate.py --model . --prompt "The purpose of education is"
trust_remote_code=True is required because this is a custom architecture. Review configuration_fineweb.py and modeling_fineweb.py before loading code from the Hub.
Limitations
The model is very small by modern language-model standards and often produces incorrect, repetitive, incoherent, or fabricated text.
Its knowledge is limited to patterns learned from the training data; it has no live information retrieval, tools, memory, or reliable sense of current time.
The 1,024-token context window includes both the prompt and generated output.
FineWeb-Edu is English-focused web data and can contain errors, biases, sensitive material, and uneven topic coverage.
The SFT variants received a small amount of instruction tuning and are not robust chat assistants.
The model has not been evaluated for safety, fairness, memorization, or production deployment.
Training data and attribution
FineWeb-Edu, licensed ODC-By 1.0, was used for base pretraining.
The source datasets are not redistributed in this model repository and remain subject to their respective licences and attribution requirements.
Licence
The model weights and original code in this repository are released under the Apache License 2.0. This does not replace or override the licences of the training datasets.
