You can deploy this model and start earning money today!

pinkelephantlimited/
pinkelephant-llm-48b-s-sft

Name: pinkelephant-llm-48b-s-sft
Availability: InStock

supervised-fine-tuning

Views

No views yet

Pink Elephant 48B-S — The SFT Adapter Treasury

100 full-precision LoRA adapters from a 100,000-step supervised fine-tuning campaign · published checkpoint by checkpoint

Executive Summary

This repository is the training treasury of the Pink Elephant SFT program — the second act in our open alignment pipeline, after the 48B-S release and before Direct Preference Optimization (DPO).

At its heart are 100 full-precision LoRA adapters — the complete learning journey of a 47.7-billion-parameter sparse intelligence engine, captured at 1,000-step resolution from step 1,000 to step 100,000. Every adapter is standalone, loadable, and evaluable.

The campaign was run with an industrial-standard QLoRA recipe — the same cost-efficient methodology Meta and the open-weight ecosystem use to fine-tune flagship-scale models. The frozen base is stored in 4-bit NF4 purely as the memory-efficient training vehicle; all computation ran in native bf16, and the published adapters themselves are full precision. This is the efficient way large models get fine-tuned — and it is exactly why the whole 47.7B model was fine-tuned on a single GPU.

The story is measured, not marketed. Training loss fell from 0.65 → ~0.44 and held-out evaluation identified step 90,000 as the best adapter (CE 0.5392) — the checkpoint we recommend for release. Every number here is real, logged, and reproducible from the public artifacts in this repository.

This is the treasury before the crown. The SFT stage is complete; DPO is next on the road map. Investors, auditors, and engineers can watch — and reproduce — exactly how a flagship model learns to follow instructions, step by step.

Where This Sits in the Lineage

One lineage, three milestones — and this treasury is where the flagship learned to follow instructions.

What Makes This Repository a Treasure

Asset	What it is	Value
100 full-precision adapters	`sft/sft-lora-step1000.pt` … `sft/sft-lora-step100000.pt`	The entire training journey at 1,000-step resolution — a living time-lapse of learning
The training vehicle	The 47.7B MoE network in 4-bit NF4, compute in bf16	Whole-model fine-tuning on a single 96 GB GPU — cost-efficient, industry standard
Full campaign record	`training/sft/` — report, eval results, manifest, notebooks	Auditable from first step to release decision
Reproducibility kit	pinned `requirements.txt`, `SETUP_GUIDE.md`, corrected notebook	Anyone can re-run the campaign and verify the numbers
MIT license	—	Free for commercial use, modification, and redistribution

The Fine-Tuning Campaign at a Glance

Specification	Value
Base model	Pink Elephant 48B-S — MoE, 8 experts, top-2, 47.69B parameters
Method	QLoRA — frozen base in 4-bit NF4 (double-quantized) for storage, all training compute in bf16
Adapters	Full precision — LoRA weights stored natively, mergeable into the bf16 base
LoRA rank / alpha / dropout	32 / 64 / 0.05
LoRA target modules	`qkv_proj`, `o_proj`, `gate_up_proj`, `down_proj`
Optimizer	PagedAdamW8bit, lr 2e-4, cosine, 50-step warm-up
Sequence length / batch	1,024 tokens · micro-batch 4 · grad-accum 8 (effective 32/step)
Dataset	OpenHermes-2.5 — 1,001,551 tokenized instructions (ChatML, 1,024 cap)
Training steps	100,000
Adapters published	100 (every 1,000 steps)
Hardware	Single NVIDIA RTX PRO 6000 Blackwell, 96 GB VRAM (~28–30 GB resident)
Final training loss	~0.44 at step 88,999 (near-plateau)
Held-out best	step 90,000 — 0.5392 CE
Recommended release	`sft/sft-lora-step90000.pt`
License	MIT — free for commercial use

The Learning Curve: Watch a Mind Get Sharp

Training loss fell steadily from 0.65 in the early campaign to a plateau near 0.44 — a healthy, stable SFT convergence over 100,000 steps. The single high point at step 69,019 is a documented resume artifact / bad batch, not a divergence; the loss immediately returned to the falling trend.

The Early-Stop Decision, Made on Evidence

After training, three released checkpoints were measured on a clean held-out set (2,000 sampled sequences the model had never seen):

Held-out evaluation across release checkpoints

Checkpoint	Held-out CE loss
step 80,000	0.5424
step 90,000	0.5392 ← best
step 100,000	0.5578

Held-out loss fell from step 80k → 90k, then rose at step 100k (+0.0185) — the classic signature of a model beginning to overfit its training distribution in the tail. This empirically confirms the early-stop at ~100k decision. There is no justification for another 150,000 steps; the released adapter is step 90,000.

The Journey, Published Step by Step

Every 1,000 steps produced a full-precision adapter, so the model's growth can be watched — and evaluated — at any moment of its life:

sft/
├── sft-lora-step1000.pt      ← the first checkpoint
├── sft-lora-step2000.pt
├── ...
├── sft-lora-step90000.pt     ← best held-out (0.5392) — recommended release
├── ...
└── sft-lora-step100000.pt    ← the end of the campaign

Each adapter is standalone (weights + config) and loads onto the base model, which ships in native bf16 from the flagship repository. 100 adapters. One continuous story. This is what transparent, industrial-standard model development looks like.

Loading the Adapters

1. Load the base + LoRA adapter

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model, set_peft_model_state_dict
from huggingface_hub import hf_hub_download

REPO = "pinkelephantlimited/pinkelephant-llm-48b-s-sft"
BASE = "pinkelephantlimited/pinkelephant-llm-48b-s"

tok = AutoTokenizer.from_pretrained(BASE, trust_remote_code=True)
tok.pad_token = tok.eos_token
model = AutoModelForCausalLM.from_pretrained(
    BASE, torch_dtype=torch.bfloat16,
    trust_remote_code=True, device_map="cuda",   # or load_in_4bit=True for a smaller footprint
)
model = get_peft_model(model, LoraConfig(
    r=32, lora_alpha=64,
    target_modules=["qkv_proj", "o_proj", "gate_up_proj", "down_proj"],
    lora_dropout=0.05, bias="none", task_type="CAUSAL_LM",
))

# load any published adapter — e.g. the recommended release
sd = torch.load(hf_hub_download(repo_id=REPO, filename="sft/sft-lora-step90000.pt",
                                repo_type="model"), map_location="cpu", weights_only=False)
set_peft_model_state_dict(model, sd["adapter"])   # ← the correct PEFT loader

Note for engineers: adapters must be loaded with set_peft_model_state_dict, not model.load_state_dict(..., strict=False). A subtle key mismatch (lora_A.weight vs lora_A.default.weight) can silently load nothing. This was discovered, fixed, and documented during our own audit — see training/sft/SETUP_GUIDE.md.

2. Generate with the chat template

python

chat = [{"role": "user", "content": "Explain what a Mixture of Experts model is in one sentence."}]
prompt = tok.apply_chat_template(chat, tokenize=True, add_generation_prompt=True, return_tensors="pt")
out = model.generate(prompt.to(model.device), max_new_tokens=256, do_sample=True, temperature=0.7,
                     eos_token_id=[tok.eos_token_id, tok.convert_tokens_to_ids("<|im_end|>")])
print(tok.decode(out[0][prompt.shape[1]:], skip_special_tokens=True))

3. Merge an adapter into the bf16 base (optional)

python

from peft import PeftModel
model = PeftModel.from_pretrained(model, hf_hub_download(repo_id=REPO, filename="sft/sft-lora-step90000.pt"))
merged = model.merge_and_unload()
merged.save_pretrained("./pink-elephant-48b-s-sft-merged")   # a single bf16 model, no adapters

Reproducibility — The Campaign Can Be Re-Run

Everything needed to reproduce the training is public in training/sft/:

Artifact	Contents
`requirements.txt`	Pinned, verified environment (transformers 5.14.1, bnb 0.50.1, peft 0.20.0, accelerate 1.14.0)
`SETUP_GUIDE.md`	Full GPU-workstation recovery procedure — validated from scratch
`sft_notebook_cells.json`	The corrected, complete training notebook (incl. `torch.library` patch + loss persistence)
`SFT_REPORT.md`	The training report in markdown
`heldout_eval.json`	Machine-readable held-out evaluation results
`RELEASE.json`	The release manifest (adapter = step 90,000)

One hard-won lesson, documented publicly: transformers must stay on 5.14.1. Version 5.15.x changed gradient-checkpointing defaults and crashes immediately with bitsandbytes 4-bit training. The pinned environment reproduces the campaign exactly.

Why QLoRA? The Efficient Way Flagships Are Fine-Tuned

Fine-tuning a 47.7B MoE flagship is a compute-and-memory problem. QLoRA answers it the industry-standard way:

The frozen base lives in 4-bit NF4 — a memory-efficient training vehicle that keeps the whole model resident on a single 96 GB GPU, not a quality compromise on the model itself.
All training compute runs in bf16 — full-precision gradients, full-precision optimizer states for the adapters.
The adapters are full precision — what you deploy and merge is exact; nothing is lost to quantization.
Result: ~28–30 GB peak VRAM for a 47.7B fine-tune. That is the difference between a laboratory and a production-capable single-GPU pipeline.

For full-precision deployment, the base model ships natively in bf16 in the flagship repository (pinkelephant-llm-48b-s, verified end-to-end on Blackwell at 96 GB residency).

Compute & Efficiency: The Numbers That Matter

The whole 100,000-step campaign ran on a single GPU — an engineering fact that separates a laboratory from a production discipline:

Efficiency metric	Value
Model fine-tuned	47.7B parameters (8-expert MoE)
Peak training VRAM	~28–30 GB on an RTX PRO 6000 Blackwell (96 GB)
Memory efficiency vs. bf16 fine-tune	~3.4× lower peak VRAM
Full campaign	100,000 steps, 1,001,551 instructions, single GPU
Checkpoint cadence	every 1,000 steps, pushed to Hugging Face live

This is fine-tuning at laboratory scale on a single workstation — the same campaign that would otherwise demand a multi-GPU cluster. Every checkpoint, every number, every log is public. That is what disciplined, transparent, investor-auditable engineering looks like.

What This Means for Investors

Capital efficiency. The entire 100,000-step campaign ran on one GPU. Your capital goes into the work, not into GPU fleets — the same result others buy with a cluster.
Unit economics. A 47.7B fine-tune costs almost nothing per run, so we can run many experiments per day — iterate faster, fail faster, ship faster. More shots on goal per dollar than teams with ten times the hardware budget.
Engineering mastery. Single-GPU fine-tuning of a 47.7B MoE flagship means the team has already cleared the hardest infrastructure barriers — quantization, PEFT plumbing, memory budgeting, crash-proof resumption. Those skills don't disappear with more hardware; they compound.
More with less. The open ecosystem rewards the team that does the most with the least. This is the same discipline that made DeepSeek famous — and it is a moat, not a weakness.

Learning rate schedule over the campaign

The cosine schedule with a 50-step warm-up — textbook curriculum discipline across 100,000 steps.

The Road Map: From SFT to the Crown

This repository is stage two of a four-stage industrial alignment pipeline:

Base model — Pink Elephant 48B-S ✅ (see pinkelephant-llm-48b-s)
Supervised Fine-Tuning — this repository ✅ 100,000 steps, 100 full-precision adapters, complete
Direct Preference Optimization (DPO) — next, on top of the released SFT adapter
Further alignment & deployment — RLHF, serving, quantization

The SFT stage is done, documented, and reproducible. DPO reuses the same QLoRA infrastructure. The treasure grows.

The Pink Elephant Family

Repository	Role in the Story
`pinkelephant-llm-48b-s`	The refined flagship — two epochs, merged, native bf16, ready to run
`pinkelephant-llm-48b-s-sft` (this repo)	The SFT adapter treasury — 100 full-precision LoRA adapters + the efficient training vehicle
`pinkelephant-llm-48b`	The first-epoch flagship
`pinkelephant-llm-48b-moe-base`	The upcycled sparse base — where the experts were born
`pinkelephant-llm-48b-adapters`	LoRA adapters + trained router, published every 100 steps
`pink-elephant-llm-14b`	The dense 14B core — where it all began
`pinkelephant-llm-48b-sft-openhermes-1m`	The tokenized instruction dataset behind the SFT campaign

Documentation

Document	Location
SFT Training Report (PDF)	SFT_REPORT.pdf
SFT Training Report (markdown)	`training/sft/SFT_REPORT.md`
Setup & recovery guide	`training/sft/SETUP_GUIDE.md`
Release manifest	`training/sft/RELEASE.json`
Held-out evaluation	`training/sft/heldout_eval.json`

Flagship documentation (in pinkelephantlimited/pinkelephant-llm-48b-s):

Document	Contents
Release Report	The 48B-S release narrative — architecture, lineage, training, quality
Verification Test Report	Instrumented end-to-end test + benchmarks on real Blackwell hardware
User Manual	What / which / where / how for every public artifact in the family

Trust, Verified

100 of 100 checkpoints confirmed live — every step from 1,000 to 100,000 was verified present and downloadable in this repository.
Every number is logged — training loss, held-out loss, and the release manifest are published in machine-readable form (training/sft/).
The campaign is reproducible — the pinned environment and corrected notebook let anyone re-run the exact same training.
One honest correction, fully documented — the resume-path loader bug (SETUP_GUIDE.md) was found during our own audit and published with the fix. Transparency is the brand.

Limitations, Stated Honestly

The 100 adapters reflect per-session training; a resume-path bug was found and fixed post-training (documented in SETUP_GUIDE.md). The held-out numbers are per-adapter quality, which is exactly how the release decision was made.
Generation quality is verified by spot-checks and held-out loss; formal benchmark scores (HumanEval, MATH-500) for the SFT adapter are part of the broader verification program.
DPO — the natural next stage — is on the road map but not yet run.

License & Commercial Use

Released under the MIT License — free for commercial and research use, modification, and redistribution, with or without attribution.

Pink Elephant Limited — publishing large, sovereign language models.

Built by Pink Elephant Limited · 2026