You can deploy this model and start earning money today!

bschooled/
caeleste-speech-qwen2.5-0.5b-instruct-nf4

Name: caeleste-speech-qwen2.5-0.5b-instruct-nf4
Availability: InStock

Views

No views yet

qwen2.5-0.5b-instruct-nf4

4-bit NF4 (bitsandbytes) quantization of Qwen/Qwen2.5-0.5B-Instruct.


Upstream repository	`Qwen/Qwen2.5-0.5B-Instruct`
Upstream revision	`7ae557604adf67be50417f59c2c2f167def9a775`
Upstream license	`apache-2.0`
Quantization	bitsandbytes NF4, double quantization, bf16 compute
Original size	0.93 GiB
Quantized size	0.44 GiB

Text LLM used for the chat/completions path.

Changes from upstream

The weight tensors have been replaced with their NF4-quantized equivalents and a quantization_config block was added to config.json. Tokenizer, processor, and any remote-code files are copied unmodified from the upstream revision above. No architecture, vocabulary, or generation defaults were altered.

Requirements

Loading requires bitsandbytes and a supported accelerator. NF4 kernels are available for NVIDIA sm_75+ (Turing and newer) and for AMD ROCm on RDNA3 / RDNA3.5 / CDNA.

python

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "bschooled/caeleste-speech",
    subfolder="qwen2.5-0.5b-instruct-nf4",
    device_map={"": "cuda:0"},
)

License

This derivative is distributed under the upstream license, apache-2.0. All upstream copyright, patent, trademark and attribution notices are retained. Refer to the upstream repository for the authoritative license text.

bschooled/caeleste-speech-qwen2.5-0.5b-instruct-nf4

qwen2.5-0.5b-instruct-nf4

Changes from upstream

Requirements

License

bschooled/
caeleste-speech-qwen2.5-0.5b-instruct-nf4