You can deploy this model and start earning money today!

DanielTanasescu/
raw-250m

Name: raw-250m
Availability: InStock

random-initialization

research

text-generation

apache-2.0

Views

No views yet

raw-250m

UNTRAINED MODEL — RAW INITIALIZED WEIGHTS

This repository contains a raw foundation-model starting point.

The weights are randomly initialized and have not been pretrained. The model will not generate meaningful text until a pretraining stage is completed.

This release is intended for:

architecture experiments
pretraining research
transformer testing
educational purposes

Architecture

Field	Value
Parameters	248.91M
Layers	10
Hidden size	1280
Attention heads	20
KV heads	5
Head dimension	64
FFN	3776 SwiGLU
Context length	4096 tokens
Vocabulary	49157
Tied embeddings	True
RoPE theta	500000
Precision	bfloat16

Initialization

Method: gpt2_scaled
Seed: 42
Deterministic initialization

Memory

Weights: 497.81 MB
KV cache: 12.80 KB/token
AdamW training state estimate: 3.98 GB

Usage

This model is not pretrained.

Example loading code:

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Quantum47Vision/raw-250m"
)

tokenizer = AutoTokenizer.from_pretrained(
    "Quantum47Vision/raw-250m"
)

DanielTanasescu/raw-250m