Qwen3-11B — 30% Compressed from Qwen3-14B (English · Chat)
This repository is part of the Efficient and Robust AI System (E-AI) Project by
Vincent-Daniel Yun, which publicly releases compressed large language models. This model
is a compressed edition of Qwen/Qwen3-14B with
12 of 40 transformer layers removed (28 layers remain, ≈11B parameters),
then instruction-tuned for chat so it generates coherent responses at lower memory and latency.
This is the same model as V1 — the weights are unchanged. The only difference is how it writes text.
In V1, the model could repeat itself or drift off when writing longer replies. V1.5 turns on repetition control by default, so you get clean answers with no extra setup. Because of that, it now handles short English chat, not just classification and scoring.
The repetition problem from V1 is fixed
Short English chat works with no extra settings
English only. For questions about specific facts, pair it with retrieval (RAG). It works best with short answers.
Same benchmark scores as V1, since the weights did not change
Chat examples
A few real replies from this model (English system prompt, default settings):
You: Hello! Can you briefly introduce yourself?
Assistant: Hello! I'm an AI assistant, and I'm ready to help you with anything you want to know. Have a good time!
You: Explain what photosynthesis is in two sentences.
Assistant: Photosynthesis is the process by which plants, algae, and some bacteria convert sunlight into energy-rich food for life on Earth. They accomplish this by absorbing light from the sun and using it to convert carbon dioxide from the air into carbon dioxide-free carbon compounds.
You: Write a short, friendly message wishing a friend good luck on their exam.
Assistant: I hope you're all great with your exam in the end! It's a big step for you, and I'm so proud of you for taking step towards that goal. If you need anything before it comes up, I am always here. Enjoy the rest of your day and go ahead with your great intentions. I'm cheering you in.
How to chat
python
1import torch
2from transformers import AutoModelForCausalLM, AutoTokenizer
34name ="daniel-eai/Qwen3-11B-30pct-Compressed-14B-EN-V1.5"5model = AutoModelForCausalLM.from_pretrained(name, trust_remote_code=True, dtype=torch.float16, device_map="cuda")6tok = AutoTokenizer.from_pretrained(name, trust_remote_code=True)78messages =[9{{"role":"system","content":"You are a helpful assistant. Always respond in English."}},10{{"role":"user","content":"Give me three tips for staying focused while studying."}},11]12inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt",13 enable_thinking=False).to("cuda")14# repetition control is already set in generation_config.json15output = model.generate(inputs, attention_mask=torch.ones_like(inputs), max_new_tokens=256)16print(tok.decode(output[0][inputs.shape[1]:], skip_special_tokens=True))
⚠️ Language support — English only. This model is tuned on English data and is
English-focused. Other languages (e.g., Korean, Chinese, Japanese) are not officially
supported and may produce degraded or broken output. For open-domain factual questions,
use retrieval (RAG) — the compressed model can hallucinate specific facts.
About E-AI
Modern AI is powerful but heavy. State-of-the-art models are enormous and their inference is slow — still far from human intuition, and far too slow and unreliable to trust in urgent, high-stakes moments.
Two obstacles stand between today's models and AI we can trust in the field. Individually, each model is too large and too slow to run where it is actually needed. Collectively, when many models or agents work together, a single faulty or adversarial member can quietly derail the whole system. E-AI attacks both — making every model lightweight and fast, and keeping teams of agents reliable even when some of them fail.
I started the E-AI (Efficient-AI) project to build compact yet powerful AI that can assist people in disaster scenarios — responding to dangerous accidents quickly and reliably when every second counts.
Method
The pruning method and the recovery method used to build this model are proprietary,
undisclosed methods created by Vincent-Daniel Yun and are not released. The compressed model
is then instruction-tuned for chat (distilled from the base model). Only the resulting model
is shared here.
Results (measured)
All numbers below were measured by us. PPL is on a 2048-token context (lower is better); downstream tasks and MMLU are 0-shot accuracy via lm-eval-harness (higher is better).
Metric
Qwen3-14B (base)
This model (30%)
PPL · WikiText2 ↓
8.64
31.27
PPL · C4 ↓
13.00
32.08
PPL · PTB ↓
14.79
47.79
ARC-c ↑
0.6024
0.3899
ARC-e ↑
0.8279
0.5901
BoolQ ↑
0.8933
0.6544
COPA ↑
0.9000
0.7600
HellaSwag ↑
0.7881
0.5824
OpenBookQA ↑
0.4620
0.3500
RACE ↑
0.4325
0.3655
RTE ↑
0.7762
0.7184
WinoGrande ↑
0.7317
0.6314
Avg. downstream (9) ↑
0.7127
0.5602
MMLU ↑
0.7729
0.6264
Performance by subject (MMLU-PRO)
MMLU-PRO is a harder, reasoning-focused version of MMLU — 12,032 questions across 14 subjects, answered with step-by-step reasoning. We ran the full test set (no sampling) and compared this model against the dense Qwen3-14B, so you can see where compression costs the least and the most.
Subject
Dense 14B
This model (30%)
Retained
Psychology
0.732
0.571
78%
Economics
0.722
0.537
74%
History
0.583
0.373
64%
Biology
0.806
0.600
74%
Law
0.349
0.245
70%
Health
0.654
0.410
63%
Philosophy
0.549
0.395
72%
Other
0.609
0.378
62%
Computer Science
0.627
0.378
60%
Engineering
0.361
0.160
44%
Physics
0.495
0.196
40%
Math
0.603
0.187
31%
Business
0.598
0.162
27%
Chemistry
0.450
0.130
29%
Overall (official)
0.565
0.310
55%
The model keeps most of its accuracy on knowledge- and reading-heavy subjects (psychology, economics, biology, history, health), and loses the most on multi-step quantitative subjects (chemistry, math, business, physics). In practice, it's a good fit for humanities, social science, and life- and medical-science questions, and a poor fit for heavy calculation.
Model family — pick your size
All sizes in this release (click to open each model). Memory is measured peak inference
(fp16 and 4-bit, batch 4 × 2048, single 48 GB GPU).
How little GPU memory each option needs relative to the original dense fp16 model
(lower is better; combine compression with 4-bit for the largest savings).
4-bit (and other) quantization can be used with this model — it is a standard Qwen3
architecture, so bitsandbytes 4-bit / 8-bit loading and other PTQ methods apply on top of
the compression. Verified: this model loads and generates correctly in 4-bit, with peak
inference memory ~11.44 GB (vs 13.9 GB for the dense model in 4-bit, and
33.5 GB for the dense model in fp16).
1import torch
2from transformers import AutoModelForCausalLM, AutoTokenizer
34m = AutoModelForCausalLM.from_pretrained(5"daniel-eai/Qwen3-11B-30pct-Compressed-14B-EN-V1.5", trust_remote_code=True, dtype=torch.float16, device_map="cuda")6tok = AutoTokenizer.from_pretrained("daniel-eai/Qwen3-11B-30pct-Compressed-14B-EN-V1.5", trust_remote_code=True)78ids = tok("The capital of France is", return_tensors="pt").to("cuda")9print(tok.decode(m.generate(**ids, max_new_tokens=20)[0]))
trust_remote_code=True is required: the model ships a small custom decoder layer in
modeling_qwen3_recovered.py.
Usage — vLLM
vLLM uses its own model implementations, so the custom decoder layer is loaded via a tiny
plugin (provided in this repo under vllm_plugin/). Install it once, then serve normally:
pip install ./vllm_plugin # from a checkout of this repo's vllm_plugin/ folder
python
1from vllm import LLM, SamplingParams
2llm = LLM(model="daniel-eai/Qwen3-11B-30pct-Compressed-14B-EN-V1.5", trust_remote_code=True, dtype="float16")3print(llm.generate(["The capital of France is"], SamplingParams(max_tokens=20))[0].outputs[0].text)
Other backends: TGI / SGLang / llama.cpp each use their own model graphs and would need
an analogous custom decoder layer; they are not supported out of the box.
License
Apache-2.0, inherited from the base model Qwen/Qwen3-14B.
Acknowledgements
Sincere thanks to Prof. Sai Praneeth Karimireddy (University of Southern California)
for his invaluable advice and feedback throughout this work, and to Prof. Sunwoo Lee
(Inha University) for his guidance and support. We are also grateful to Alibaba (the
Qwen team) for openly releasing the Qwen3-14B base model that made this work possible.
