It is a raw base model, not an instruction-tuned assistant. It is intended for research, benchmarking, continued pretraining, and small-model experimentation. If you are looking for a base model that is better at prompt continuation you should use Veyra2 Mango 30M Base instead.

Model Details

Property	Value
Parameters	49,303,040
Architecture	Llama-like causal LM
Layers	16
Hidden size	512
Attention heads	8
KV heads	2
Head dim	64
Intermediate size	1408
Vocabulary size	8192
Context length used in training	1024
Activation	SwiGLU / SiLU
Normalization	RMSNorm
Attention	GQA with QKV-Norm
Positional encoding	RoPE
Weight tying	Tied input embeddings and LM head
Training tokens	Approximately 19.99B
Training precision	bfloat16
Optimizer	Muon

Tokenizer

Special tokens:

<|bos|>: 0
<|eos|>: 1
<|pad|>: 2
<|unk|>: 3
<|im_start|>: 4
<|im_end|>: 5

Training Data

The model was trained on a 20B-token pretraining mixture.

FineWeb-Edu sample-10BT: 45%
Cosmopedia-v2: 30%
DCLM-Edu: 15%
FineMath finemath-3plus: 10%

Training Summary

Final step: 9532
Tokens seen: 19,990,052,864
Tokens per step: 2,097,152
Global batch size: 2048
Sequence length: 1024
Last train loss: 2.5062
Final eval loss: 2.4978
Final eval perplexity: 12.16

Evaluation

SciCloze Eval

SciCloze-900 accuracy: 47.56%
SciCloze-900 correct: 428 / 900

SciCloze-900 repository: veyra-ai/SciCloze-900

SciCloze-900 subject breakdown:

Biology: 53.67%
Physics: 45.67%
Chemistry: 43.33%

Other Evals

Zero-shot results:

Open SLM Bench verified:

Average: 38.81%
PIQA acc_norm: 62.13%
ARC-Easy acc_norm: 42.47%
ARC-Challenge acc_norm: 23.29%
HellaSwag acc_norm: 31.28%
ArithMark-3.0 acc: 31.60%

Additional local evaluations:

Winogrande acc: 50.12%
OpenBookQA acc_norm: 28.80%
BoolQ acc: 60.67%
SciQ accuracy: 72.70%
SciQ normalized accuracy: 64.20%

Usage

This model uses custom Transformers remote code because it includes QKV-Norm and a custom architecture definition.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "veyra-ai/Veyra2-Apricot-50M-Base"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.float16,
    device_map="auto",
)

prompt = "In the 19th century"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=120,
        do_sample=True,
        temperature=0.6,
        top_p=0.9,
        repetition_penalty=1.1,
        use_cache=True,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Notes on Generation

Veyra2-Apricot-50M-Base is a raw base model. It is not instruction tuned and should not be expected to behave like a chat assistant. Open-ended generations can be unstable, repetitive, or factually unreliable. It's not a polished assistant.

Intended Use

This model is intended for:

small language model research
continued pretraining
benchmarking
science-oriented cloze and multiple-choice evaluation
experimentation with compact causal LMs

Limitations

Not instruction tuned
Not RLHF tuned
Not safe for factual or high-stakes use without additional validation
Can hallucinate names, citations, species, references, and technical claims
Open-ended text may drift off-topic
Context length during training was 1024 tokens

Citation

If you use this model, please cite the model repository:

veyra-ai/Veyra2-Apricot-50M-Base