See the FP16 reference card for the full training recipe, eval numbers, and known limitations (particularly the GSM8K regression vs base — use base Qwen2.5 or Qwen2.5-Math-7B for math-heavy workflows).

Quantization details

Method: GPTQ via llm-compressor
Scheme: W4A16 (4-bit weights, fp16 activations)
Group size: 128
Desc-act: true
Symmetric: false
Calibration: 512 samples from the training set, 2048 tokens each
Ignored layers: lm_head (kept in full precision)
Size on disk: ~4.5 GB

Inference (vLLM)

python

from vllm import LLM, SamplingParams
llm = LLM(model="ramankrishna10/npc-agentic-7b-v3-gptq-4bit", dtype="float16")
out = llm.generate(
    ["Design an event-sourced microservice with exactly-once command handling."],
    SamplingParams(max_tokens=1024, temperature=0.7, top_p=0.9),
)
print(out[0].outputs[0].text)

Citation

If you use NPC Agentic 7B in your work, please cite:

bibtex

@misc{bachu2026npcagentic7b,
  title        = {NPC Agentic 7B: A Single-GPU QLoRA Recipe for a Laptop-Scale Conversational Model},
  author       = {Bachu, Rama Krishna},
  year         = {2026},
  month        = may,
  publisher    = {Zenodo},
  version      = {v1},
  doi          = {10.5281/zenodo.19954103},
  url          = {https://doi.org/10.5281/zenodo.19954103},
  note         = {Preprint}
}

Paper: https://doi.org/10.5281/zenodo.19954103

ramankrishna10/npc-agentic-7b-v3-gptq-4bit

NPC Agentic 7B — GPTQ 4-bit

Quantization details

Inference (vLLM)

See also

Citation

ramankrishna10/
npc-agentic-7b-v3-gptq-4bit