Industrial-standard, end-to-end verification of the FINAL aligned model. This repo is the single home for every tool, notebook, and recorded result used to verify the complete production model — not an intermediate stage.

text

pink-elephant-llm-14b (dense 14B)
    └─add weights / upcycle──▶ 48b-moe-base (8-expert MoE) ──train──▶ 48b
    └─remaining epochs──▶ 48b-s (flagship, s = smarter)
    └─+SFT best (step90000)──▶ +DPO best (dpo-best)──▶ FINAL MODEL  ⇐ verified here
    └─GGUF conversion──▶ run locally via Ollama / LM Studio

Pipeline stage index: base → moe → train → sft → dpo → verification (this) → release

After verification, the model is converted to GGUF for local use in Ollama / LM Studio.

Why a dedicated repo

Verification targets the whole final model (base 48B-S + SFT + DPO merged), which is the artifact users actually download and run. Keeping verification separate from the DPO training repo reflects that: the SFT/DPO repos are training stages; this repo is the acceptance gate for the finished model.

Tooling inventory

Path	Purpose
`tooling/structural/verify_architecture.py`	S1. Verifies the released model's config, parameter scale, and full `base_model:` lineage chain against the architectural contract. CPU-only, CI-runnable.
`tooling/structural/verify_weights.py`	S2. Verifies SFT + DPO LoRA adapters are shape-consistent with the base (hidden/intermediate/expert dims) so they merge cleanly. CPU-only.
`tooling/adapter_to_peft.py`	Converts the pipeline's custom `.pt` LoRA adapters (SFT `sft-lora-step.pt`, DPO `dpo-lora-step.pt`) into standard PEFT dirs that `PeftModel.from_pretrained` consumes.
`tooling/run_benchmarks.py`	S3–S6. Loads base 48B-S + merges SFT best + DPO best, then runs GSM8K + MMLU. Shell-friendly alternative to the notebook.
`tooling/smoke_test.py`	S4. Minimal generation sanity check of the final model.
`tooling/benchmarks/gsm8k.py`	Raw GSM8K evaluation script (batch, checkpointed — used in the live run).
`tooling/benchmarks/mmlu_5shot.py`	Raw 5-shot MMLU script (balanced stratified sampling — used in the live run).
`tooling/benchmarks/mmlu_chunk.py`	Chunked MMLU runner (progress-safe for long runs).
`tooling/run_all.py`	Orchestrates the CPU-side checks (structural + weights) in one command.
`notebooks/verify_full_model.py`	Single notebook running all six stages (S1–S6) on one Blackwell host.
`results/`	Recorded verification outputs (JSON).
`report/ALIGNED_VERIFICATION_REPORT.pdf`	This verification report (print-ready).
`requirements.txt`	GPU-benchmark dependencies.

Quick start

CPU-only structural + weight checks (no GPU)

bash

pip install -r requirements.txt   # torch optional for S1, needed for adapter inspection
python tooling/run_all.py --structural \
    --sft /path/sft-lora-step90000.pt --dpo /path/dpo-best.pt \
    --json results/

Full GPU verification (Blackwell, ~96 GB, bf16)

bash

# On the verification host:
python tooling/adapter_to_peft.py sft-lora-step90000.pt peft/sft
python tooling/adapter_to_peft.py dpo-best.pt           peft/dpo
python tooling/run_benchmarks.py            # GSM8K + MMLU (S3–S6)
# or run notebooks/verify_full_model.py (single notebook, all of S1–S6)

Recorded results (this build)

Populated from live verification runs on our Blackwell host. See results/ JSON files.

Benchmark	Result	Detail
GSM8K (math reasoning)	68.5%	61/89 on a random sample; `answer is` extraction
MMLU (knowledge, 5-shot)	60.0%	24/40 balanced across 7 subjects; letter extraction
Structural contract	PASS	16/16 checks (config, params, lineage)

Results gallery


Headline benchmarks: GSM8K 68.5% · MMLU 60.0%	MMLU (5-shot) accuracy by subject
Structural pre-release gate: 16/16	Verification stages across the whole final model

GPU / environment

Requires a Blackwell GPU host (~96 GB), runs in bf16.
Model load ≈ 3 min, held in kernel memory across notebook cells (kernel-global persistence).
Base: pinkelephantlimited/pinkelephant-llm-48b-s. Adapters: pinkelephantlimited/pinkelephant-llm-48b-s-dpo (sft/sft-lora-step90000.pt, dpo/dpo-best.pt).

License

MIT — Pink Elephant Limited.

pinkelephantlimited/pinkelephant-llm-48b-s-verification

Pink Elephant 48B-S — Verification