Every weight in this model was produced by this project. There is no from_pretrained, no base checkpoint, and no adapter: the tokenizer, the architecture code, the data pipeline and the training loop are all bespoke, and training started from a random initialisation.

Trained on a single RTX 5070 Ti (16 GB).

Architecture

Written by hand — no transformers model classes were used to define it.


Type	decoder-only, pre-norm transformer
Parameters	351,650,304
Layers	18
d_model	1280
Attention heads	20 query / 4 KV (GQA)
Head dim	64
FFN	SwiGLU, hidden 3456 (≈ 8/3 × d_model)
Normalisation	RMSNorm (pre-norm) + QK-norm
Positions	RoPE (θ=10000) — no learned position embeddings
Context	1024
Vocab	32,768 (own byte-level BPE, padded to a multiple of 128)
Biases	none, in any linear layer
Embeddings	tied between input and LM head

Parameter split: 41,943,040 embedding (11.9%) / 309,705,984 transformer blocks.

Training


Tokens seen	13,676,103,680
Steps	26,082
Schedule	warmup-stable-decay (WSD)
Optimizer	fused AdamW, betas (0.9, 0.95), wd 0.1 on 2D params only
Precision	bf16 autocast, fp32 master weights
Hardware	1 × RTX 5070 Ti 16 GB

Evaluation

Bits per byte is the headline metric. This model's vocabulary (32,768) differs from GPT-2's (50,257), and cross-entropy per token is not comparable across tokenizers. BPB divides that out and is directly comparable.

Metric	This model	GPT-2 124M
Parameters	351,650,304	124,439,808
Validation loss	1.7331	—
Bits per byte	0.5873	—
HellaSwag (0-shot, len-norm)	pending	29.55%

Validation loss is measured on a held-out slice of the training corpus and is not comparable to GPT-2's OpenWebText number. Bits per byte is, because it is tokenizer-independent.

Data provenance and licensing

Source	HF id	License
fineweb-edu	`HuggingFaceFW/fineweb-edu`	ODC-By 1.0

Attribution: FineWeb-Edu (HuggingFaceFW/fineweb-edu), ODC-By 1.0

ODC-By requires attribution, which is given above. Whether pretraining-data licenses flow through to model weights is legally unsettled; permissively licensed sources were chosen deliberately as the most defensible posture. Model weights are released under cc-by-nc-4.0.

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("Dikshan1234/LLMPretrain")
model = AutoModelForCausalLM.from_pretrained("Dikshan1234/LLMPretrain",
                                             trust_remote_code=True)

inputs = tok("Once upon a time", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=100, do_sample=True,
                     temperature=0.8, top_p=0.95)
print(tok.decode(out[0]))

trust_remote_code=True is required because the architecture (RoPE + SwiGLU + GQA + QK-norm) is not a stock transformers model. The two files it loads, configuration_llmpretrain.py and modeling_llmpretrain.py, are in this repo and are short enough to read.

Sample generations

===== step 26000 =====

--- [greedy] 'Once upon a time'
Once upon a time, the Company’s sole business was to sell and distribute the Company’s products. The Company’s products were sold to a limited number of customers. The Company’s products were sold to a limited number of customers. The Company’s products were sold to a limited number of customers. The Company’s products were sold to a limited number of customers. The Company’s products were sold to a limited number of customers. The Company’s products were sold to a limited number of customers. The Company’s products were sold to a limited number of customers. The Company’s

--- [t=0.8] 'Once upon a time'
Once upon a time, the Company would issue capital stock of the Company upon obtaining the consent from the holder of more than 5% or more than 10% of the Company’s outstanding shares of common stock. The right of the holder of more than 5% of the Company’s outstanding shares of common stock to elect to exercise such rights upon request is not subject to any minimum equity or voting rights.

The Company has a dividend reinvestment plan under which the Company may purchase shares of the Company’s common stock in the open market at the fair market value on the date of purchase.

The Company has

--- [greedy] 'The capital of France is'
The capital of France is a combination of the French national currency, the euro, and the British pound. The euro is the currency of the European Union. The British pound is the currency of the United Kingdom.

The Company’s primary foreign currency exposure is the euro. The Company’s primary foreign currency exposure is the British pound. The Company’s primary foreign currency exposure is the euro.

The Company’s primary foreign currency exposure is the British pound. The Company’s primary foreign currency exposure is the euro.

The Company’s primary foreign currency exposure is the British pound. The Company

--- [t=0.8] 'The capital of France is'
The capital of France is a highly diversified country encompassing a variety of economic zones. France’s total area is approximately 4,500,000 km2. France’s largest geographical area is located in the Western part of the country, while the rest of France is mainly made up of the central regions and the eastern part of the country. The French people are among the most highly educated of all nations in the world. France is also one of the largest producers and exporters of steel, copper, aluminum and iron in the world. France’s economy and culture are characterized by a high level of individual achievement and innovation

--- [greedy] 'In 1969, humans first'
In 1969, humans first discovered the human genome. The genome is the complete set of all human genes. The genome is the complete set of all human genes.
The human genome is the complete set of all human genes. The genome is the complete set of all human genes.
The human genome is the complete set of all human genes. The genome is the complete set of all human genes.
The human genome is the complete set of all human genes. The genome is the complete set of all human genes.
The human genome is the complete set of all human genes. The genome is the complete set of all

--- [t=0.8] 'In 1969, humans first'
In 1969, humans first became aware of a
bacterial contamination in a milk facility in France.

The
FDA has issued a complete response letter, or CRL, for the E. coli contamination at the facility, and the agency has indicated that if it is unable to complete such a CRL, it will issue a
suspended stay until the issue can be resolved. The CRL required that the site be shut down, which was accepted by the agency in January, 2020. The site is currently under a CRL for a different site.

In January 2015, the FDA issued its


--- [greedy] 'The three states of matter are'
The three states of matter are:

•

the solid state, which is the state of matter that is in the solid state;

•

the liquid state, which is the state of matter that is in the liquid state

Limitations

This is a base model trained on a modest token budget. It has had no instruction tuning, no RLHF and no safety alignment. It will produce confidently wrong text, and it reflects whatever biases exist in its training corpus. It is a research and educational artifact, not a product.