You can deploy this model and start earning money today!

zyberg2091/
fineweb_pretrained_model_37M

Name: fineweb_pretrained_model_37M
Availability: InStock

Views

No views yet

FineWeb Pretrained Model - 37.8M Parameters

A 37.8M-parameter decoder-only language model implemented and pretrained from scratch in PyTorch.

Model summary

Parameters: 37.8M
Layers: 6
Hidden dimension: 512
Attention heads: 8
KV heads: 4
Context length: 256
Vocabulary: 20,000
Architecture: grouped-query attention, RoPE, pre-norm transformer
Training data: 250M-token FineWeb slice
Training: approximately 61,000 steps and 500M token passes
Precision: float32

Intended purpose

This model is a research and educational artifact for studying small-language-model pretraining, architecture implementation, and parameter allocation. It is not intended as a production assistant.

Loading

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "zyberg2091/fineweb_pretrained_model_37M"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
)