You can deploy this model and start earning money today!

RthItalia/
nano_compact_3b_qkvfp16

Name: nano_compact_3b_qkvfp16
Rating: 2.5 (12 reviews)

Views

No views yet

Nano Compact 3B QKV-FP16

RthItalia/nano_compact_3b_qkvfp16 is the validated compact self-contained variant derived from Qwen/Qwen2.5-3B-Instruct.

This release is not the old overlay artifact zip. It is the final exported Hugging Face folder that loads directly with transformers plus trust_remote_code=True.

What This Variant Is

This runtime uses a mixed policy:

q_proj, k_proj, v_proj: fp16
o_proj and most of the remaining body: Nano compact format
model.embed_tokens: quantized single copy
lm_head: tied custom head over the quantized embeddings

The goal of this policy is practical balance between:

disk size
VRAM footprint
quality relative to the true 8bit baseline

Validated Runtime Envelope

model size: 2.3432 GB
allocated after load: 2.3432 GB
peak generation VRAM: about 2.44 GB

True 8bit baseline used for comparison:

allocated after load: 3.1703 GB
peak generation VRAM: about 3.21 GB

Quality Claim

The quality claim for this release is intentionally narrow:

it was compared against the true 8bit baseline on a small internal prompt suite
it is not claimed to match the full original model in every task
it is not claimed to outperform the base model

More aggressive variants reached better size or VRAM numbers, but failed the quality gate against the true 8bit reference.
qkvfp16 was the first variant that restored acceptable behavior on the validation smoke suite while preserving a substantial memory advantage.

How To Load

python

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

repo_id = "RthItalia/nano_compact_3b_qkvfp16"

tok = AutoTokenizer.from_pretrained(
    repo_id,
    use_fast=True,
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
    device_map="cuda",
    dtype=torch.float16,
).eval()

Example

python

messages = [
    {"role": "user", "content": "Explain what a neural network is in exactly 3 simple sentences."}
]

text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inp = tok(text, return_tensors="pt").to(next(model.parameters()).device)

with torch.no_grad():
    out = model.generate(
        **inp,
        max_new_tokens=120,
        do_sample=False,
        repetition_penalty=1.08,
        eos_token_id=tok.eos_token_id,
        pad_token_id=tok.eos_token_id,
    )

print(tok.decode(out[0][inp["input_ids"].shape[-1]:], skip_special_tokens=True))

Requirements

pip install torch transformers accelerate safetensors

bitsandbytes is not required for this final exported winner variant at runtime.

Important Notes

trust_remote_code=True is required.
The custom runtime uses a NanoTiedHead implementation that ties output logits to the quantized embedding table without registering the embedding module twice.
Custom linear layers use chunked forward paths to keep peak VRAM under control.

Limitations

Validation was narrow and engineering-driven, not a full benchmark suite.
This release is specifically tuned around Qwen/Qwen2.5-3B-Instruct.
It should be treated as a compact experimental runtime artifact, not as a general scientific proof of broader architectural claims.

License Note

This release should be described as a dual-license or dual-layer research distribution.

Built with Qwen.

The intended reading is:

Qwen-derived materials retain the relevant Qwen Research License obligations
Nano-specific runtime, packaging, and documentation changes are additional repository-authored research components

For redistributed copies, keep:

LICENSE
NOTICE
clear indication of modified files where applicable

The Hugging Face metadata still stays at license: other because this is not accurately described by a single simple SPDX identifier.