You can deploy this model and start earning money today!

pinkelephantlimited/
pinkelephant-llm-48b-s

Name: pinkelephant-llm-48b-s
Availability: InStock

Views

No views yet

Pink Elephant 48B-S

The Refined Sparse Intelligence Engine

47.7-billion-parameter Mixture-of-Experts language model · 8 experts · top-2 routing · ~30% active parameters per token · 3.2× compute advantage · native bf16

Executive Summary

Pink Elephant 48B-S is the flagship of the Pink Elephant lineage and the culmination of a disciplined, fully-public research program: a dense 14.8B core, upcycled into an 8-expert sparse Mixture-of-Experts network, refined across two complete training epochs, and merged into a single, ready-to-run engine.

The story is measured, not marketed. Across the second epoch of supervised fine-tuning on a 99,661-example code-and-mathematics curriculum, training loss fell 32% (0.0616 → 0.0419) while validation loss held steady near 0.50 — refinement, not overfitting. HumanEval 70.12% pass@1 confirms the sparse architecture preserved the parent's coding capability at roughly one-third of the per-token compute.

The pipeline now runs end-to-end through alignment and verification: base 48B-S → SFT (best adapter, step 90000) → DPO (min-loss adapter) → verified benchmarks on Blackwell. The fully-aligned model scores GSM8K 68.5% and MMLU (5-shot) 60.0% — the first public, reproducible numbers for the aligned flagship. Full per-question records ship in the verification/ folder of the DPO repo.

Every artifact in this release is public, reproducible, and MIT-licensed — weights, per-step adapters, trained router, tokenized dataset, loss curves, and instrumented benchmark records. Numbers are reported exactly as measured; a number we publish is a number you can audit.

Specification	Value
Total parameters	47,691,290,048 (47.7B)
Active parameters / token	~14.66B (30.7%)
Sparse compute advantage	3.2× fewer FLOPs vs. a dense 48B model
Architecture	MoE · 8 experts · top-2 routing
Transformer layers / hidden / expert	40 · 5,120 / 8,960
Attention heads	40 (10 KV heads, head dim 128)
Context window	16,384 tokens
Vocabulary	100,352
Precision	Native bfloat16
Training campaign	Two epochs · 3,114 steps · resumed cleanly 4×
Training loss (epoch 2 final)	0.0419 (−32% vs. 0.0616)
License	MIT — free for commercial use

Benchmarks (greedy, measured on NVIDIA RTX PRO 6000 Blackwell)

Benchmark	48B-S	14B (parent)	Note
HumanEval pass@1	70.12% (115/164)	71.34% (117/164)	Same performance band; ~1pp is run-to-run noise
MATH-500	38.40% (192/500)	63.40% (317/500)	Greedy single-answer decoding; explicit roadmap item

HumanEval confirms the upcycle preserved coding ability at a fraction of the compute. The MATH-500 number is reported exactly as measured, with per-subject detail published for audit (strength-first: prealgebra 61.0%, algebra 59.7%, number theory 37.1%, counting & probability 28.9%, geometry 26.8%, precalculus 17.9%, intermediate algebra 13.4%). Machine-readable results: evals/humaneval_result.json, evals/math500_result.json, plus every per-problem record.

Aligned Verification Benchmarks (SFT + DPO stage)

The numbers below are for the fully-aligned model — base 48B-S with the best SFT adapter (step 90000) and the min-loss DPO adapter fused in, evaluated in bf16 on an NVIDIA RTX PRO 6000 Blackwell. Greedy decoding with robust answer is / letter extraction (a naive "last-token" metric under-reports and is not used).

Benchmark	Aligned 48B-S (SFT+DPO)	Protocol
GSM8K (math word problems)	68.5% (61/89)	Random sample of GSM8K test · `answer is` extraction
MMLU (knowledge, 5-shot)	60.0% (24/40)	40 balanced across 7 subjects · `\boxed{letter}` extraction

Verification stage: the pipeline is base → moe → train → sft → dpo → verification → release. These aligned results are the output of the verification stage. Full per-question records, the eval harness, and the adapter-conversion tooling live in the verification/ folder of [pinkelephant-llm-48b-s-dpo] (https://huggingface.co/pinkelephantlimited/pinkelephant-llm-48b-s-dpo), so every number is independently reproducible.

The Story in One Picture

One lineage, three milestones: a proven dense core, upcycled to sparse scale, refined twice.

Act I — The Dense Core: Pink Elephant 14B

A dense 14.8B-parameter decoder-only model, instruction-tuned on ~100,000 curated code and mathematics examples. It proved the thesis:

HumanEval pass@1: 71.34%
MATH-500: 63.40%

Act II — The Upcycle: Dense 14B → Sparse 48B

Rather than train 48B from scratch, Pink Elephant upcycled the 14B foundation into an 8-expert MoE network — feedforward weights split and replicated into eight complementary experts per layer, exactly preserving the parent's behavior at initialization. A sparse brain with dense-scale capacity at ~30% activation per token.

Act III — The Refinement: 48B-S

A second complete epoch (steps 1,557 → 3,114) on the same 99,661-example curriculum with a freshly restarted cosine schedule peaking at 1e-4, checkpointed every 100 steps and resumed cleanly four separate times. The final LoRA adapters and trained router were fused directly into the base weights — what you download is the finished engine: no adapters, no PEFT plumbing, just load and generate.

Training and validation loss across both epochs

The Architecture: A Council of Experts

The conventional feedforward network is replaced by a cabinet of eight expert networks per layer, orchestrated by a learned router. Every token activates only 2 of 8 experts — sparse computing at its most elegant: maximum capability, minimum energy.

The 48B-S sparse MoE transformer block: a learned router selects 2 of 8 experts per token.

Only ~30% of the network activates per token: dense-scale capacity at 3.2× lower compute.

The 99,661-example curriculum behind the 14B → 48B → 48B-S lineage.

Verified End-to-End on NVIDIA Blackwell

The published artifact was downloaded, loaded, and exercised end-to-end on an NVIDIA RTX PRO 6000 Blackwell (102 GB VRAM) at native bf16 — no quantization, no CPU offload:

Test	Result
Artifact download	95.38 GB across 20 safetensors shards
Model load	Clean — all 47,691,290,048 parameters resident
VRAM residency	96.1 GB of 102 GB (94.2% occupancy)
Cold load time	254.5 s
Generation throughput	14.7 tokens/s (greedy)
Functional suite	5 of 6 prompt categories correct

Getting Started

Load the model directly — no adapters required:

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "pinkelephantlimited/pinkelephant-llm-48b-s"

tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="cuda",   # a 102 GB-class GPU is recommended for native bf16
    trust_remote_code=True,
)

chat = [{"role": "user", "content": "Write a Python function that checks if a number is prime."}]
prompt = tok.apply_chat_template(chat, tokenize=True, add_generation_prompt=True, return_tensors="pt")
with torch.inference_mode():
    out = model.generate(
        prompt.to(model.device),
        max_new_tokens=256,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
    )
print(tok.decode(out[0][prompt.shape[1]:], skip_special_tokens=True))

Lower-footprint inference on consumer GPUs — standard 4-bit loading with bitsandbytes:

python

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    load_in_4bit=True,
    device_map="auto",
    trust_remote_code=True,
)

Note: this model uses a custom architecture shipped in the repository (modeling_pink_elephant.py). Loading requires trust_remote_code=True.

Documentation

Document	Contents
Release Report	Complete release narrative — architecture, lineage, training, quality evidence
Verification Test Report	Instrumented end-to-end test + benchmarks on real Blackwell hardware; incl. Ch. 14 "Aligned Model Verification" (GSM8K 68.5%, MMLU 5-shot 60.0%)
User Manual	What / which / where / how for every public artifact across the family

The Pink Elephant Family

Repository	Role in the Story
`pinkelephant-llm-48b-s` (this repo)	The refined flagship — two epochs, merged, ready to run
`pinkelephant-llm-48b`	The first-epoch flagship
`pinkelephant-llm-48b-moe-base`	The upcycled sparse base — where the experts were born
`pinkelephant-llm-48b-adapters`	LoRA adapters + trained router, published every 100 steps
`pink-elephant-llm-14b`	The dense 14B core — where it all began
`pinkelephant-llm-48b-sft-tokenized`	The tokenized instruction dataset behind every epoch

Roadmap

✅ Alignment (done): preference optimization (DPO) + SFT fine-tuning completed; verified GSM8K 68.5% and MMLU 5-shot 60.0% on the aligned model (see verification benchmarks)
Serving: high-throughput sparse serving and GGUF quantization for edge deployment
Breadth: expand general-knowledge coverage beyond the math/STEM core
Reasoning: continue closing the MATH-500 gap with longer-horizon, chain-of-thought training

Limitations

Fine-tuned primarily on code and mathematical reasoning; the aligned model reaches GSM8K 68.5% and MMLU 5-shot 60.0%, but general-knowledge breadth still inherits from the 14B core and trails specialized frontier models
MATH-500 under greedy decoding (38.40%) trails the 14B parent (63.40%); addressing this is an explicit roadmap item
Long-horizon, multi-step agentic workflows may benefit from explicit tool-use training
Verification samples (~89 GSM8K / 40 MMLU) are representative, not exhaustive; treat as aligned-stage estimates subject to sampling variance
Outputs should be reviewed before use in regulated or safety-critical decisions

License & Commercial Use

Released under the MIT License — free for commercial and research use, modification, and redistribution, with or without attribution.

Pink Elephant Limited — publishing large, sovereign language models.

Built by Pink Elephant Limited · 2026