You can deploy this model and start earning money today!

0ldev/
Lara-350M

Name: Lara-350M
Availability: InStock

text-generation-inference

endpoints_compatible

Views

No views yet

Lara-350M (Pretrained)

A 350M parameter LLaMA-style language model pretrained from scratch on a custom corpus. This is the base pretrained checkpoint — it has not yet been instruction-tuned or fine-tuned with any persona.

Architecture

Type: LLaMA (decoder-only transformer)
Parameters: 439,403,520
Layers: 24
Hidden size: 1024
Attention heads: 16
KV heads: 16
Vocab size: 64,000
Max sequence length: 2048
Activation: SiLU (SwiGLU)
Normalization: RMSNorm

Tokenizer

Important: This model uses a custom tokenizer trained from scratch on the training corpus. It is NOT a standard LLaMA, GPT-2, or any other pre-existing tokenizer.

Type: BPE (Byte-Pair Encoding)
Vocab size: 64,000
Special tokens: BOS (<s>), EOS (</s>), UNK (<unk>)
Training: Trained alongside the model on the same corpus

Do NOT use a standard LLaMA tokenizer with this model — the token IDs will be completely wrong and produce garbled output.

Training

Architecture: Custom LLaMA-350M (trained from scratch)
Training: 100,000 iterations with AdamW optimizer
Batch size: 32 effective (2 micro-batch x 8 grad accum x 2 GPUs)
Learning rate: 3e-4
Hardware: 2x NVIDIA RTX 3060 (12GB VRAM each)
Corpus: Custom pretraining corpus

What This Is Not

This is a raw pretrained language model. It has:

No instruction tuning
No persona or personality
No RLHF or DPO alignment
No guardrails or safety training

It predicts the next token based on patterns learned during pretraining. To use it as an assistant, it would need further fine-tuning (SFT, RLHF, etc.).

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("0ldev/Lara-350M")
tokenizer = AutoTokenizer.from_pretrained("0ldev/Lara-350M")

inputs = tokenizer("The meaning of life is", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

License

Apache 2.0