Property	Value
Parameters	1,147,766,784 (~1.15B)
Layers	24
Attention heads	16 (GQA: 4 KV heads)
Hidden dim	2048
FFN	SwiGLU
Normalization	RMSNorm
Position	RoPE with YaRN context extension
Context	4,096 tokens (extensible via YaRN)
Tokenizer	Custom SentencePiece, 32K vocab
Architecture	Mini-Llama style: RoPE, GQA, SwiGLU, RMSNorm, Flash Attention

Model Family

Model	Params	Status
Orion Atlas 1B	1.15B	🟡 Training
Orion Atlas 3B	~3B	📋 Planned
Orion Atlas 7B	8.77B	🏗️ Architecture released
Orion Atlas 14B	~14B	📋 Planned
Orion Atlas 37B	~37B	📋 Planned

The 7B model uses a Mamba-2 Hybrid + Differential Attention architecture — the first known combination of these techniques. See orion-atlas-7b.

Training

Pre-training data: FineWeb-Edu, SlimPajama, StarCoder, The Stack, OpenWebMath, Cosmopedia, GPT4All, Tulu-SFT (~17B tokens)
Hardware: NVIDIA H100 80GB HBM3 (RunPod)
Framework: Custom PyTorch training loop
Optimizer: AdamW, lr=3e-4, cosine decay
Batch: 40 × 2048 tokens (effective ~82K tokens/step)
Speed: ~50,000 tok/s

Design Goals

Built for agentic tasks — tool calling, structured JSON output, multi-step reasoning. The 1B serves as the seed model for progressive distillation up to the 37B flagship.

Usage

This model uses a custom architecture (not HuggingFace Transformers compatible). See model.py for inference code.

# Coming with weights release

Citation

bibtex

@misc{palermini2026orionatlas,
  title={Orion Atlas: A Mamba-2 Hybrid Architecture with Differential Attention for Agentic Language Models},
  author={Avery Palermini},
  year={2026},
  institution={Cendrix AI},
}

asvuep/orion-atlas-1b

Orion Atlas 1B

Status

Architecture

Model Family

Training

Design Goals

Usage

Citation

asvuep/
orion-atlas-1b