You can deploy this model and start earning money today!

Views

No views yet

Qwen3.8 4B Distilled — GGUF

Name: Qwen3.8_4B_Distilled_GGUF
Availability: InStock

GGUF quantizations of Ma7ee7/Qwen3.8_4B_Distilled for use with llama.cpp, LM Studio, Ollama, Jan, and other GGUF-compatible applications.

Qwen3.8 4B Distilled is a 4-billion-parameter reasoning model created by distilling outputs from Qwen3.8-Max into the smaller Qwen3-4B-Thinking-2507 student model.

The full model was fine-tuned on r0b0tlab/qwen3.8-max-distillation-50k, a dataset of responses and reasoning traces generated by qwen3.8-max-preview.

Model Lineage and Naming

The name Qwen3.8 4B Distilled describes the model's distillation lineage:

Teacher model: qwen3.8-max-preview
Student/base model: Qwen/Qwen3-4B-Thinking-2507
Full-precision fine-tune: Ma7ee7/Qwen3.8_4B_Distilled
Resulting model size: Approximately 4 billion parameters
Distillation dataset: r0b0tlab/qwen3.8-max-distillation-50k
Distribution format: GGUF

This is a Qwen3-architecture student model distilled from Qwen3.8-Max-generated outputs.

The repository does not claim that the underlying architecture or original weights are from Qwen3.8-Max. Qwen3.8-Max is the teacher whose generated responses and reasoning traces were used as training targets for the 4B student.

This repository contains quantized conversions of that independently fine-tuned student model. It is not an official Qwen or Alibaba release.

Model Details

Property	Value
Model type	Decoder-only causal language model
Architecture	Qwen3
Parameters	Approximately 4B
Student/base model	`Qwen/Qwen3-4B-Thinking-2507`
Teacher model	`qwen3.8-max-preview`
Full model	`Ma7ee7/Qwen3.8_4B_Distilled`
Training method	Sequence-level supervised distillation
File format	GGUF
Primary task	Reasoning and conversational text generation
Primary language	English
Thinking mode	Enabled

Available Quantizations

This repository includes the following quantization levels:

Quantization	Description
`Q4_K_M`	Recommended starting point with a strong balance of size, speed, and quality
`Q5_K_M`	Higher fidelity with moderately greater RAM and storage usage
`Q8_0`	Highest-fidelity option in this repository, with the largest memory footprint

For most local users, Q4_K_M is the recommended starting point.

Use Q5_K_M when you have additional memory and want to retain more quality. Use Q8_0 when fidelity matters more than storage or RAM usage.

What “Distilled” Means Here

This model uses sequence-level knowledge distillation.

The smaller student was trained on complete responses produced by the larger teacher. This transfers parts of the teacher's behavior, reasoning patterns, solution structure, and response style without copying the teacher's architecture or weights.

Therefore:

The architecture and original student weights come from Qwen3-4B-Thinking-2507.
The distillation targets come from Qwen3.8-Max-generated outputs.
The resulting checkpoint remains a 4B Qwen3 model.
These GGUF files are quantized versions of that 4B distilled model.
The model is not expected to reproduce the full capabilities of Qwen3.8-Max.

Quick Start with llama.cpp

A recent llama.cpp build can download and run a quantization directly from Hugging Face.

Q4_K_M

bash

llama-cli \
  -hf Ma7ee7/Qwen3.8_4B_Distilled_GGUF:Q4_K_M \
  --jinja \
  --conversation \
  --temp 0.6 \
  --top-p 0.95 \
  --top-k 20 \
  --repeat-penalty 1.05 \
  -c 32768 \
  -n 4096

Q5_K_M

bash

llama-cli \
  -hf Ma7ee7/Qwen3.8_4B_Distilled_GGUF:Q5_K_M \
  --jinja \
  --conversation \
  --temp 0.6 \
  --top-p 0.95 \
  --top-k 20 \
  --repeat-penalty 1.05 \
  -c 32768 \
  -n 4096

Q8_0

bash

llama-cli \
  -hf Ma7ee7/Qwen3.8_4B_Distilled_GGUF:Q8_0 \
  --jinja \
  --conversation \
  --temp 0.6 \
  --top-p 0.95 \
  --top-k 20 \
  --repeat-penalty 1.05 \
  -c 32768 \
  -n 4096

llama.cpp Server

Start a local OpenAI-compatible server:

bash

llama-server \
  -hf Ma7ee7/Qwen3.8_4B_Distilled_GGUF:Q4_K_M \
  --jinja \
  --host 127.0.0.1 \
  --port 8080 \
  -c 32768 \
  -ngl 99

The llama.cpp web interface will normally be available at:

http://127.0.0.1:8080

Example API request:

bash

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen3.8_4B_Distilled",
    "messages": [
      {
        "role": "user",
        "content": "Solve x^2 - 5x + 6 = 0 and explain your reasoning."
      }
    ],
    "temperature": 0.6,
    "top_p": 0.95,
    "max_tokens": 4096
  }'

Ollama

Run the Q4_K_M quantization directly from Hugging Face:

ollama run hf.co/Ma7ee7/Qwen3.8_4B_Distilled_GGUF:Q4_K_M

Other available options:

bash

ollama run hf.co/Ma7ee7/Qwen3.8_4B_Distilled_GGUF:Q5_K_M
ollama run hf.co/Ma7ee7/Qwen3.8_4B_Distilled_GGUF:Q8_0

Local Ollama Modelfile

After downloading a GGUF file, create a file named Modelfile:

text

FROM ./qwen3-4b-thinking-2507.Q4_K_M.gguf

PARAMETER temperature 0.6
PARAMETER top_p 0.95
PARAMETER top_k 20
PARAMETER repeat_penalty 1.05
PARAMETER num_ctx 32768

SYSTEM You are a helpful reasoning assistant. Think carefully and provide a clear final answer.

Create and run the model:

bash

ollama create qwen3.8-4b-distilled -f Modelfile
ollama run qwen3.8-4b-distilled

If your downloaded GGUF has a different filename, replace the path after FROM with the actual local filename.

LM Studio

Open LM Studio.
Search for Ma7ee7/Qwen3.8_4B_Distilled_GGUF.
Select and download a quantization.
Load the model.
Confirm that the Qwen chat template is active.
Start with a context length of 16,384 or 32,768 tokens.
Increase the context length only when sufficient memory is available.

Suggested sampling settings:

Setting	Value
Temperature	`0.6`
Top-p	`0.95`
Top-k	`20`
Repetition penalty	`1.05`
Maximum output tokens	`4096` or higher

Context Length

The model inherits its context configuration from Qwen3-4B-Thinking-2507. Actual usable context in a GGUF application depends on:

Available system RAM
Available VRAM
Quantization level
KV-cache precision
GPU offloading configuration
The llama.cpp or application build

A context length of 16K or 32K is a practical starting point for local use. Longer contexts can require substantially more memory because of KV-cache growth.

GPU Offloading

To offload as many layers as possible to the GPU:

bash

llama-cli \
  -hf Ma7ee7/Qwen3.8_4B_Distilled_GGUF:Q4_K_M \
  --jinja \
  --conversation \
  -ngl 99 \
  -c 32768 \
  -n 4096

For CPU-only inference, omit -ngl 99 or set a smaller number of GPU-offloaded layers.

Recommended Generation Settings

Setting	Recommended value
Temperature	`0.6`
Top-p	`0.95`
Top-k	`20`
Repetition penalty	`1.0`–`1.1`
Maximum output tokens	`4096` or higher

For difficult mathematics, programming, or long-form reasoning, allow enough output tokens for the model to complete both its reasoning and final answer.

Thinking Output

The model inherits a thinking-oriented chat format from Qwen3-4B-Thinking-2507. Depending on the inference application and reasoning parser, visible reasoning may be displayed in a form similar to:

text

<think>
Reasoning process
</think>

Final answer

Some applications may hide the thinking section, place it in a collapsible panel, or display it separately from the final answer.

Distillation Dataset

The full model was trained on:

r0b0tlab/qwen3.8-max-distillation-50k

The dataset contains teacher-generated examples across areas including:

Mathematics
Programming
General reasoning
Scientific reasoning
Instruction following
Limited tool use

Teacher responses were generated by qwen3.8-max-preview. Visible <think>...</think> reasoning traces were retained when present in the dataset.

Intended Uses

This model is intended for local experimentation with:

Mathematical reasoning
Programming and code generation
Logical reasoning
Scientific question answering
General instruction following
Long-form problem solving
Local conversational assistants
Research into teacher-to-student distillation
CPU and consumer-GPU inference

Limitations

This is a quantized 4B student model and does not contain the complete capabilities of Qwen3.8-Max.
Quantization may reduce quality relative to the full-precision model.
Lower-bit quantizations may degrade more noticeably on difficult reasoning or programming tasks.
Distillation transfers patterns from teacher-generated outputs; it does not copy the teacher's architecture or weights.
Teacher-generated answers may contain factual, mathematical, or programming errors.
Visible reasoning traces should not automatically be assumed to be correct.
The model may hallucinate information or produce confidently incorrect answers.
The training mixture is predominantly English.
The training dataset includes examples originating from common evaluation benchmarks.
Results on overlapping benchmarks may be contaminated and should not be treated as clean evaluations without additional controls.
Outputs should be reviewed before use in high-stakes, medical, financial, legal, or security-sensitive settings.

License and Training-Data Notice

This model repository is published under the Apache License 2.0.

This license does not override the licenses, attribution requirements, or usage terms associated with:

The full-precision fine-tuned model
The Qwen3 base model
The Qwen3.8-Max teacher provider
The distillation dataset
The original datasets from which prompts were sourced

Users are responsible for reviewing the relevant upstream model, dataset, and provider terms before redistribution or commercial use.

Acknowledgements

This model builds upon work from:

The Qwen team for Qwen3-4B-Thinking-2507
r0b0tlab for the Qwen3.8-Max Distillation 50K dataset
llama.cpp and the GGUF ecosystem
Hugging Face

Citation

Distillation Dataset

bibtex

@misc{r0b0tlab2026qwen38distillation50k,
  title        = {Qwen3.8-Max Distillation 50K},
  author       = {r0b0tlab},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/r0b0tlab/qwen3.8-max-distillation-50k}}
}

Disclaimer

Qwen3.8 4B Distilled GGUF is an independent community release by Ma7ee7.

It is not produced, endorsed, or officially released by the Qwen team, Alibaba, or Alibaba Cloud.

Ma7ee7/
Qwen3.8_4B_Distilled_GGUF

Qwen3.8 4B Distilled — GGUF

Model Lineage and Naming

Links

Model Details

Available Quantizations

What “Distilled” Means Here

Quick Start with llama.cpp

Q4_K_M

Q5_K_M

Q8_0

llama.cpp Server

Ollama

Local Ollama Modelfile

LM Studio

Context Length

GPU Offloading

Recommended Generation Settings

Thinking Output

Distillation Dataset

Intended Uses

Limitations

License and Training-Data Notice

Acknowledgements

Citation

Distillation Dataset

Disclaimer