Pink Elephant 48B is a production-ready Mixture-of-Experts (MoE) large language model and the flagship release from Pink Elephant Limited. Upcycled from the proven Pink Elephant 14B dense foundation and instruction-tuned across code and mathematics, it delivers the reasoning depth of a dense 48-billion-parameter network while activating only ~30% of its parameters for every token — 47.7B parameters of total capacity, ~14.7B active at inference.

This repository ships the fully merged, ready-to-run model: the trained LoRA expert adapters and the learned router are fused directly into the base weights. No adapter setup, no PEFT plumbing — just load and generate.

The central promise: every token is routed to a hand-picked council of specialist experts — dense-scale intelligence at a fraction of the compute.

The Architecture: A Council of Experts

Pink Elephant 48B replaces the single monolithic feedforward network of a conventional transformer with a cabinet of eight expert networks per layer, orchestrated by a learned router.

8 experts per layer — each with its own specialized weights and perspective
Top-2 sparse routing — the router selects the two most qualified experts for every token; the other six rest
A trained router — fine-tuned end-to-end alongside the experts, so assignments sharpen continuously with every step

Every token therefore travels through only 2 of 8 experts. This is sparse computing at its most elegant: maximum capability, minimum energy, delivered in real time.

Figure 1 — The Pink Elephant 48B sparse MoE transformer block: a learned router selects 2 of 8 expert networks for every token.

Specification	Value
Total parameters	47.7B (~48B)
Active parameters / token	~14.7B (30.7% of the network)
Architecture	Mixture of Experts — 8 experts, top-2 routing
Transformer layers	40
Hidden size	5,120
Expert hidden size	8,960
Attention heads	40 (10 KV heads, head dim 128)
Router	Learned gating, full-precision training, auxiliary balancing
Context window	16,384 tokens
Vocabulary size	100,352
Precision	bfloat16
License	MIT — free for commercial use

48B of Knowledge, a Quarter of the Compute

Capacity of a dense 48B — a vast reservoir of knowledge and reasoning, spread across specialists that never compromise on depth
~3.2× compute savings — with only ~30% of parameters active per token, inference costs a fraction of a dense model of the same scale
No quality compromise — dense-scale capacity and sparse-scale speed, in a single deployable artifact

Figure 2 — Only ~30% of the network activates per token: dense-scale capacity at 3.2× lower compute.

Trained to Write Code and Reason Mathematically

Pink Elephant 48B was instruction-tuned on ~100,000 curated examples spanning two of the most demanding reasoning domains:

Software engineering — executable code generation and instruction following (self-oss-instruct, 49,661 examples)
Mathematical reasoning — rigorous step-by-step problem solving (OpenR1 Math, 50,000 examples)

Training: 1,000 optimization steps, AdamW with a cosine learning-rate schedule (peak 1.5e-4), LoRA (r=32, α=64) applied to the expert projections, with the router trained in full precision.

Training loss and learning-rate schedule

Figure 3 — Training and validation loss over 1,000 optimization steps, with the cosine learning-rate schedule (left-to-right panels).

An Elegant Scaling Story

Pink Elephant 48B was upcycled, not trained from scratch. The feedforward weights of the established pink-elephant-llm-14b dense foundation were mathematically split and replicated into eight complementary experts — preserving the parent model's knowledge while multiplying capacity roughly threefold. Scaling through intelligence, not brute force.

Figure 4 — Product lineage: from the dense 14B foundation to the upcycled MoE base, refined through SFT, and shipped as the merged 48B.

Getting Started

Load the model directly — no adapters required:

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "pinkelephantlimited/pinkelephant-llm-48b"

tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",            # or "cuda" / "cpu"
    trust_remote_code=True,
)

chat = [{"role": "user", "content": "Write a Python function that checks if a number is prime."}]
prompt = tok.apply_chat_template(chat, tokenize=True, add_generation_prompt=True, return_tensors="pt")
with torch.inference_mode():
    out = model.generate(
        prompt.to(model.device),
        max_new_tokens=256,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
    )
print(tok.decode(out[0][prompt.shape[1]:], skip_special_tokens=True))

Lower-footprint inference

For consumer GPUs, the model can be loaded in 4-bit using standard transformers tooling (requires bitsandbytes):

python

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    load_in_4bit=True,
    device_map="auto",
    trust_remote_code=True,
)

Note: this model uses a custom architecture shipped in the repository (modeling_pink_elephant.py). Loading it requires trust_remote_code=True.

The Refined Successor: 48B-S

The Pink Elephant journey continues. pinkelephant-llm-48b-s takes this flagship through a complete second epoch (steps 1,557 → 3,114), cutting training loss 32% (0.0616 → 0.0419) and merging into a single ready-to-run artifact. If you are starting fresh, use the 48B-S.

Product lineage: from the dense 14B core to the upcycled MoE base, refined through SFT, and shipped as the merged 48B — then refined again into the 48B-S.

Release Family

Repository	Purpose
`pinkelephant-llm-48b-s`	The refined flagship (two epochs) — recommended for new deployments
`pinkelephant-llm-48b` (this repo)	Merged production model — the first-epoch flagship
`pinkelephant-llm-48b-moe-base`	Upcycled MoE base (bf16) — for research and merging
`pinkelephant-llm-48b-adapters`	LoRA adapters + trained router, published every 100 steps
`pink-elephant-llm-14b`	The dense 14B foundation
`pinkelephant-llm-48b-sft-tokenized`	The instruction-tuning dataset (tokenized)

Evaluation

Benchmark results for the family are published on their respective model cards. The 48B-S refined flagship reports 70.12% pass@1 on HumanEval and 38.40% on MATH-500 (both greedy), with per-problem records under evals/ in that repository. The 14B lineage reports 71.34% pass@1 on HumanEval and 63.40% on MATH-500. The 48B-S HumanEval figure demonstrates the same performance band as the parent at roughly a third of the per-token compute.

Verification & Test Report

The full release and verification documentation is published alongside this model:

Document	Contents
Release Report	The complete release narrative — architecture, lineage, training, and quality evidence
Verification Test Report	Instrumented end-to-end test results on real hardware
User Manual	What / which / where / how for every public artifact across the whole family

What was tested

The published artifact was downloaded, loaded, and exercised end-to-end on an NVIDIA RTX PRO 6000 Blackwell Server Edition (102 GB VRAM, 160 GB RAM, PyTorch 2.11.0+cu130, transformers 5.14.1) at native bfloat16 — no quantization, no CPU offload:

Test	Result
Artifact download	95.38 GB across 20 safetensors shards, ~175–190 MB/s
Model load	Clean — 47,691,290,048 parameters; only `original_inv_freq` flagged UNEXPECTED (benign cached RoPE tensor)
VRAM residency	96.0 GB of 102 GB occupied after load; 93.8% occupancy; peak allocation 95.4 GB
Generation throughput	19.1 tokens/sec greedy (128 tokens in 6.7 s)
Functional suite	5 of 6 prompt categories produced correct output

Functional smoke test (greedy, 64 tokens)

Prompt	Result
`def is_prime(n):`	✅ Correct primality function, continues to `is_perfect_square`
The capital of France is	✅ "Paris. So, the answer is Paris..."
What is 17 * 23?	✅ Decomposes 17×20 + 17×3 = 391 correctly
Write a python function that returns the factorial of n.	✅ Correct recursive factorial with edge-case guards
`def fibonacci(n):`	✅ Correct canonical recursion
Explain MoE in one sentence	⚠️ Empty under greedy decoding — the model emitted EOS immediately on this single prompt; sampling or `min_new_tokens` restores output (analyzed in the Verification Test Report, Chapter 10)

Every result above is reproducible via the step-by-step recipe in the Verification Test Report (Chapter 11), which requires only a GPU with ≥ 102 GB VRAM and this repository.

Limitations

Fine-tuned primarily on code and mathematical reasoning; general-knowledge coverage inherits from the 14B parent
Long-horizon, multi-step agentic workflows may benefit from explicit tool-use training
Outputs should be reviewed before use in regulated or safety-critical decisions

License & Commercial Use

Released under the MIT License — free for commercial and research use, modification, and redistribution, with or without attribution.

Pink Elephant Limited — publishing large, sovereign language models.