You can deploy this model and start earning money today!

cognica/
Cognica-PoE-v1.0-1.3B-stage-summary

Name: Cognica-PoE-v1.0-1.3B-stage-summary
Availability: InStock

Views

No views yet

Cognica-PoE-v1.0-1.3B-stage-summary

Paper: Product of Experts as Scalable Local Learning: Modular Construction at 1.3B Parameters (Jeong, 2026)

Not a production summarization model. This release is empirical validation of PoE's post-hoc specialist construction at 4-layer depth for a sparse-ratio task — paper Section 6.5 extended.

A 164 M-parameter summarization-domain SFT specialist stage trained directly on the frozen Cognica-PoE-v1.0-1.3B-base. The stage is a sibling of the chat/math/code/tool stages — all five branch from the same base, not from each other. Use the cascade loader with this repo (or compose stage heads additively in a custom inference script; see the paper).

Unlike chat/math/code/tool, summarization has a sparse assistant ratio (short summary after long document — assistant tokens are typically 6-10 % of each training example). Early attempts (v1-v5) mode-collapsed into empty-assistant completions. The shipped v9 recipe pairs end-token down-weighting (<|assistant_end|> logits weighted 0.3x) with label smoothing 0.02 to prevent the loss from being dominated by the "stop" decision.

What this artifact does demonstrate:

Sparse-ratio SFT is tractable with loss reweighting. Val bpb exhibits a phase transition at step 2500 -> 3000 (3.785 -> 3.147) as the model transitions from "predict end" to "predict summary". Best saved checkpoint s3600 reaches val bpb 3.1073.
Extractive multi-sentence summaries work. On CNN/DailyMail-style inputs with "Multi-sentence summary:" cue, the model extracts salient sentences from the source, stops cleanly (with repetition_penalty >= 1.1), and preserves most facts.
Length-constraint suffixes are not honored. The "Summary (one sentence):" cue produces multi-paragraph output; the SFT data did not teach explicit length conditioning.
Post-warmup divergence is real. After the best region (s3000-s3900) val bpb drifts upward to 4.27 by step 10981. The shipped checkpoint is not the final step — save_every=100 + eval_every=25 was used specifically to capture the best-val region.

Quick start

python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "cognica/Cognica-PoE-v1.0-1.3B-stage-summary"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model.eval()

# Chat-format prompt (the SFT data was formatted with SmolTalk special tokens):
BOS, USR_S, USR_E, ASS_S = 32759, 32760, 32761, 32762
doc = (
    "The Federal Reserve announced a 0.25 percentage point increase in interest rates on "
    "Wednesday, marking the fifth rate hike this year. Fed Chair Jerome Powell cited "
    "persistent inflation concerns and strong employment data as key factors. Markets "
    "reacted with mixed signals, as the Dow Jones fell 200 points while the tech-heavy "
    "Nasdaq remained steady."
)
prompt = f"{doc}\n\nMulti-sentence summary:"
ids = [BOS, USR_S] + tokenizer.encode(prompt) + [USR_E, ASS_S]
input_ids = torch.tensor([ids], device=model.device)

with torch.no_grad():
    out = model.generate(
        input_ids,
        max_new_tokens=200,
        do_sample=True,
        temperature=0.3,
        top_k=50,
        repetition_penalty=1.15,
        no_repeat_ngram_size=3,
        pad_token_id=BOS,
    )
print(tokenizer.decode(out[0, len(ids):]))

repetition_penalty=1.15 plus no_repeat_ngram_size=3 is strongly recommended — without both, the model loops on trailing phrases. The included generation_config defaults follow this.

Architecture

Component	Detail
Parent	`cognica/Cognica-PoE-v1.0-1.3B-base` (PoE alpha=0.0, d24, step 26430, val bpb 0.7209)
New transformer layers	4 appended at positions 24-27 (d24 -> d28)
Frozen layers	24 (all base layers)
Dual-head	Yes — additive specialist `lm_head_stage` (shape 32768 x 1536, zero-init at training start)
Final projection	`logits = lm_head_base(x) + lm_head_stage(x)`
Total params	1,547,699,986 (~1.55 B)
Trainable params at training	163,577,912 (~164 M, 10.6 %)
Shipped delta	28 tensors, 213,909,560 params, 408 MB (bf16 safetensors)
VE pattern	Preserved from base — 12 value-embeds at layers [1, 3, ..., 23]; new layers carry no VE

Training


Objective	Cross-entropy over assistant turns only; end-token weighted 0.3x; label smoothing 0.02
Data	`EdinburghNLP/xsum` x 2 epochs + `abisee/cnn_dailymail` 3.0.0 x 2 epochs -> 981,804 train convs (+ 512 val)
Case augmentation	First-user greetings duplicated with case variants -> 981,804 -> 983,715 conversations
Sequence length	2,048
Per-GPU batch	8 x 2,048
World size	4 (1 node x 4 x A100 80 GB)
Total batch size	65,536 tokens/step
Steps	10,981 (~2 xsum + 2 cnndm full passes)
Optimizer	MuonAdamW with per-group LR scaling
Matrix LR	3.0 x 10^-4
`lm_head_stage` LR	1.0 x 10^-4, weight decay 0.15
Init LR fraction	0.15
Warmup / warmdown	5 % / 60 %
Eval / save cadence	every 25 / 100 steps
Best checkpoint shipped	step 3,600, val bpb 3.1073

Validation bpb trajectory (saved checkpoints)

Step	val bpb	Phase
100	4.2339	early, predicting-end
500	4.1235	"
1000	4.0131	"
1500	3.9574	"
2000	3.9062	"
2500	3.7850	pre-transition
3000	3.1467	phase transition
3100	3.1252	best region
3200	3.1543	"
3300	3.1333	"
3400	3.1667	"
3500	3.1251	"
3600	3.1073	best saved (shipped)
3700	3.1490	"
3800	3.1320	"
3900	3.1342	"
4000	3.1709	drift begins
5000	3.1954	"
6000	3.3791	"
7000	4.3915	warmdown degradation
10000	4.2688	"
10981	final	DO NOT SHIP

The phase transition at step 2500 -> 3000 is the learnable moment — before it, the model minimizes loss by always predicting <|assistant_end|> (exploiting the sparse-assistant reward); after it, extractive summary tokens become favored. The end-token down-weighting is the mechanism that lets the model escape the early trap.

Stacking further stages

base_model_name_or_path supports chaining. Point a new stage repo's config at this repo and the cascade loader will resolve base -> stage-summary -> new stage transparently. The loader folds each ancestor's lm_head_stage into the effective lm_head_base at load time, so all specialist heads compose additively into the final projection (logits = lm_head_base + Sigma_k lm_head_stage_k). See the paper for the formal account of this construction.

Files

File	Purpose
`config.json`	Model + stage config (`base_model_name_or_path`, `new_layers=4`, `frozen_layers=24`, full `stage_training` block)
`delta.safetensors`	28-tensor stage delta (bf16, 408 MB)
`modeling_cognica_poe.py`	Cascade loader + `_GPT` with dual-head forward (same code as the base repo)
`configuration_cognica_poe.py`	`CognicaPoEConfig` with stage fields
`tokenization_cognica_poe.py`	Byte-level tokenizer (unchanged from base; includes the numeric-token decode fix)
`tokenizer.pkl`, `tokenizer_config.json`, `special_tokens_map.json`, `token_bytes.pt`	Tokenizer assets (unchanged from base)
`convert_stage_delta.py`	Converts a nanochat `save_stage_delta` `.pt` file into `delta.safetensors`

Limitations — explicit list

Research preview at 1.3 B. Capacity-constrained. Produces mostly-extractive multi-sentence summaries; paraphrasing prompts often hallucinate (spelling of names, dates, expanded fact lists).
Length constraint not learned. "Summary (one sentence):" suffix produces multi-paragraph output; the SFT data did not teach explicit length conditioning. Treat as a multi-sentence extractive summarizer only.
Sparse-ratio instability. Without end-token down-weighting the training collapses; with it, the best-val region is narrow (s3000-s3900). save_every=100 + eval_every=25 was needed specifically to capture this window.
Greedy decoding loops. Use repetition_penalty >= 1.1 and no_repeat_ngram_size=3 or sampling.
No RLHF / preference tuning / safety tuning.
Assistant-only loss — validation bpb is measured on assistant tokens only; do not compare to the base's full-text train_val_bpb = 0.7209.

Citation

bibtex

@article{jeong2026poe,
  title  = {Product of Experts as Scalable Local Learning: Modular Construction at 1.3B Parameters},
  author = {Jeong, Jaepil},
  year   = {2026},
  institution = {Cognica, Inc.},
  doi    = {10.5281/zenodo.19547653},
  url    = {https://doi.org/10.5281/zenodo.19547653}
}

@misc{cognica-poe-stage-summary-2026,
  title  = {Cognica-PoE-v1.0-1.3B-stage-summary: Summarization-domain dual-head specialist (4-layer) over a PoE base (research preview)},
  author = {{Cognica, Inc.}},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/cognica/Cognica-PoE-v1.0-1.3B-stage-summary}}
}

License

Apache 2.0 — see LICENSE and NOTICE. Same terms as the base model. Training datasets (XSum, CNN/DailyMail) each carry their own licenses and are acknowledged in NOTICE.