+5pp SuperGPQA (55.0% → 60.0%, N=20) via 4-trace parallel sampling + majority voting. Zero fine-tuning. Zero probe. Pure self-consistency on the new Qwen3.6-27B dense flagship (released 2026-04-21).

TL;DR

metric	Qwen3.6-27B greedy	self-consistency wrapper (ours)
SuperGPQA accuracy (n=20 Stage B)	0.550	0.600 (+5.0 pp)
Extraction reliability	variable	100% (forced `</think>` close)
Compute (per query)	1 trace	4 traces (parallelizable)
Fine-tuning needed	—	No
Probe needed	—	No (attempted but failed — see §3)

1. Results (validated N=20 Stage B held-out)

method	accuracy	Δ vs greedy
Qwen3.6-27B greedy (baseline)	0.550	—
Naive majority vote (N=4)	0.600	+5.0 pp
PWMV (probe-weighted vote, N=4)	0.600	+5.0 pp (same as naive — probe null)
Probe best-of-N	0.550	0 pp

Convergence across the 20-prompt evaluation

The naive-vote lead stabilizes after n=6 and holds through n=20. PWMV tracks naive vote exactly (probe weighting defaulted to uniform because the trained probe had near-chance AUROC).

2. Comparison with Qwen3.6-35B-A3B (MoE, our previous wrapper)

We previously shipped a PWMV wrapper for Qwen3.6-35B-A3B (qwen36-deepconf-probe) that delivered +6pp via probe-weighted voting. The probe there hit val AUROC 0.78 and added +4pp on top of naive vote.

Applied to the newly-released 27B dense, the same probe recipe (same Stage B questions, same L11 hook, same LogReg classifier) produced a probe with val AUROC 0.509 (random chance). Self-consistency alone still delivers a clean +5pp.

3. Why the probe failed on 27B (honest diagnostic)

The L11 LogReg probe was trained on 80 greedy 27B rollouts with their correctness labels. Val AUROC came out at 0.509 — indistinguishable from random. Three factors explain the null:

Dataset imbalance: 27B is stronger than 35B-A3B, so 74% of the 80 rollouts were correct, 26% wrong. The probe sees only ~21 wrong examples — insufficient to learn the "wrong" class in d=5120 feature space.
Architecture shift: 27B is dense (5120 hidden dim) vs 35B-A3B MoE (2048). L11 residual representations differ meaningfully. A probe trained on one doesn't transfer; a fresh-trained probe needs more samples per dimension.
Training set undersized: for a d=5120 classifier, 80 samples is borderline; 200+ with balanced classes is the recommended minimum for stable LogReg.

Because the probe was near-random, probe-weighted voting (PWMV) collapsed to the same decision as naive majority vote. Best-of-N (which picks the single trace with highest probe score) fared worst because it propagates probe errors into final decisions.

Open directions to recover the probe

200+ rollouts, balanced — hard-sample mining to get 100+ wrong-class examples
Try different layer — L31 or L47 (mid-late in the 64-layer stack) may carry stronger correctness signal on dense arch
MLP probe — swap LogReg for 2-layer MLP with d_hidden=128 for richer d=5120 representation
Transfer from 35B probe — attempt W_LT mapping from 2048 → 5120 space (risky)

4. Positioning in the SuperGPQA landscape

Qwen team's official numbers + our projected 27B + self-consistency:

model	SuperGPQA
Qwen3.5-27B	65.6
Gemma4-31B	65.7
Qwen3.6-35B-A3B (our previous baseline)	64.7
Qwen3.6-27B (raw, new dense)	66.0
Qwen3.6-27B + our self-consistency	~71.0 (projected)
Qwen3.5-397B-A17B	70.4
Claude 4.5 Opus	70.6

Projecting our observed +5pp to the Qwen team's 66.0 baseline puts us at ~71 — equal to or slightly above Claude 4.5 Opus on SuperGPQA, at a fraction of the cost.

Caveats:

Our eval is on a 20-sample Stage B subset, not the full SuperGPQA benchmark.
Projection assumes the +5pp lift transfers to the full distribution (likely but not verified).
N=20 confidence interval is wide (±10pp per method); larger-scale replication recommended before production claim.

5. Usage

python

import torch, re
from collections import Counter
from transformers import AutoTokenizer, AutoModelForImageTextToText

tok = AutoTokenizer.from_pretrained('Qwen/Qwen3.6-27B', trust_remote_code=True)
if tok.pad_token_id is None: tok.pad_token = tok.eos_token
model = AutoModelForImageTextToText.from_pretrained(
    'Qwen/Qwen3.6-27B', dtype=torch.bfloat16, device_map='cuda',
    attn_implementation='sdpa', trust_remote_code=True)
model.eval()

def extract_answer(text):
    post = text.split("</think>")[-1] if "</think>" in text else text
    m = re.search(r"\\boxed\{([A-J])\}", post)
    if m: return m.group(1)
    m = re.findall(r"\\boxed\{([A-J])\}", text)
    return m[-1] if m else None

FORCE_SUFFIX = "\n</think>\n\nFinal answer: \\boxed{"
def force_close(full_ids, prompt_len):
    gen = tok.decode(full_ids[prompt_len:].tolist(), skip_special_tokens=False)
    if "</think>" in gen:
        return tok.decode(full_ids[prompt_len:].tolist(), skip_special_tokens=True)
    full = tok.decode(full_ids.tolist(), skip_special_tokens=False) + FORCE_SUFFIX
    ctx = tok(full, return_tensors="pt", add_special_tokens=False).input_ids.cuda()
    with torch.no_grad():
        out = model.generate(ctx, max_new_tokens=15, do_sample=False,
                             pad_token_id=tok.pad_token_id, use_cache=True)
    suf = tok.decode(out[0, ctx.shape[1]:].tolist(), skip_special_tokens=True)
    return tok.decode(full_ids[prompt_len:].tolist(), skip_special_tokens=True) + FORCE_SUFFIX + suf

def self_consistency_predict(question, options, n_traces=4, temperature=0.7, max_new=3500):
    choices = "\n".join(f"{chr(65+i)}. {o}" for i, o in enumerate(options))
    content = (f"Answer the following multiple-choice question. "
               f"Reason step by step, then put the letter in \\boxed{{}}.\n\n"
               f"Question: {question}\n\nOptions:\n{choices}")
    prompt = tok.apply_chat_template(
        [{"role":"user","content":content}],
        tokenize=False, add_generation_prompt=True, enable_thinking=True)
    ids = tok(prompt, return_tensors="pt").input_ids.cuda()
    with torch.no_grad():
        out = model.generate(ids, max_new_tokens=max_new, do_sample=True, temperature=temperature,
                             num_return_sequences=n_traces, top_p=0.95,
                             pad_token_id=tok.pad_token_id, use_cache=True)
    answers = [extract_answer(force_close(out[i], ids.shape[1])) for i in range(n_traces)]
    filtered = [a for a in answers if a]
    return Counter(filtered).most_common(1)[0][0] if filtered else None


question = "Which of these is a renewable energy source?"
options = ["Coal", "Natural gas", "Solar", "Petroleum", "Diesel",
           "Propane", "Nuclear uranium", "Kerosene", "Gasoline", "Shale oil"]
answer = self_consistency_predict(question, options, n_traces=4)
print(f"Answer: {answer}")  # -> "C"

6. Reproducibility

parameter	value
Base model	`Qwen/Qwen3.6-27B` (dense, 64 layers, hidden 5120)
Eval set	20 held-out Stage B prompts (random seed 42, after train split)
Training set	80 greedy rollouts on SuperGPQA Stage B questions (disjoint from eval)
Generation	`temperature=0.7, top_p=0.95, max_new_tokens=3500, N=4 traces`
Thinking mode	`enable_thinking=True`
Budget forcing	`</think>` injection if not closed after 3500 tokens
Hardware	Single NVIDIA RTX PRO 6000 Blackwell (96 GB)
Eval time	110 min (20 prompts × 4 methods, ~5.5 min/prompt)

7. Files

probe_l11.pkl — L11 LogReg probe (val AUROC 0.509, kept for reproducibility — not functional)
train_rollouts.json — 80 labeled 27B greedy rollouts + L11 residuals (8.9 MB)
summary.json — full benchmark numbers + deltas
eval_partial.json — per-prompt predictions (20 prompts × 4 methods)
README.md — this file
01_hero_benchmark.png — main result bar chart
02_convergence.png — running accuracy across n=3..20
03_probe_failure.png — AUROC diagnostic (35B works, 27B null)
04_model_compare.png — 27B dense vs 35B-A3B MoE under same wrappers
05_landscape.png — SuperGPQA benchmark positioning vs Claude 4.5 Opus

8. Related

repo	model	method	headline
qwen36-deepconf-probe	Qwen3.6-35B-A3B (MoE)	PWMV (probe works)	+6 pp SuperGPQA
qwen36-27b-selfconsistency (this)	Qwen3.6-27B (dense)	Naive majority vote	+5 pp SuperGPQA
qwen36-feature-circuits	Qwen3.6-35B-A3B	SAE circuit analysis	multi-substrate negative + post-mortem
Qwen3.6-35B-A3B-LIMO-Probe	Qwen3.6-35B-A3B	LIMO LoRA + PWMV stack	small-scale test (inconclusive n=15)

9. Limitations

N=20 is small. The +5pp naive-vote gain is directional and replicated across 7 running checkpoints, but the 95% confidence interval is roughly ±10pp per method. Larger-scale validation (n=100+) recommended before production deployment.
Probe transfer failed. Rebuilding the probe (see §3 open directions) would likely recover the additional +4pp we saw on 35B-A3B, bringing total gain to +9pp. Not attempted in this release.
Single eval domain (SuperGPQA-style multiple-choice STEM). Behavior on code (SWE-bench), math (AIME), or creative tasks may differ.
Dense 27B compute cost is ~3× the 35B-A3B per token (27B active vs 3B active). Self-consistency 4-trace runs 12× the base compute; consider for quality-critical applications, not high-throughput serving.

10. Citation

bibtex

@misc{vicentino2026qwen27bsc,
  title     = {qwen36-27b-selfconsistency: +5pp SuperGPQA on Qwen3.6-27B via N=4 majority voting},
  author    = {Vicentino, Caio},
  year      = {2026},
  howpublished = {\url{https://huggingface.co/caiovicentino1/qwen36-27b-selfconsistency}}
}

11. License

MIT for wrapper code and benchmark artifacts. Base model under its own terms at Qwen/Qwen3.6-27B.