Views
No views yet
llama-cpp-python and llama.cpp.| File | Quantization | Approx. size | RAM/VRAM (n_ctx=8k) | Quality |
|---|---|---|---|---|
gpt-oss-20b-Q4_K_M.gguf | Q4_K_M | ~8 GB | ~9 GB | Good — recommended for 12 GB GPUs |
gpt-oss-20b-Q5_K_M.gguf | Q5_K_M | ~10 GB | ~11 GB | Better — recommended for 16 GB GPUs |
gpt-oss-20b-Q8_0.gguf | Q8_0 | ~15 GB | ~16 GB | Best — recommended for 24 GB GPUs |
llama-cpp-python (Python)1from llama_cpp import Llama
2
3llm = Llama.from_pretrained(
4 repo_id="ggfox00000/gpt-oss-20b-GGUF",
5 filename="gpt-oss-20b-Q4_K_M.gguf", # or Q5_K_M / Q8_0
6 n_ctx=8192,
7 n_gpu_layers=-1, # offload everything to GPU (CUDA build required)
8)
9
10out = llm.create_chat_completion(
11 messages=[
12 {"role": "system", "content": "Tu es un assistant utile."},
13 {"role": "user", "content": "Résume cette réunion en français."},
14 ],
15 temperature=0.2,
16 max_tokens=1024,
17)
18print(out["choices"][0]["message"]["content"])1pip install --no-cache-dir --index-url https://abetlen.github.io/llama-cpp-python/whl/cu121 \
2 llama-cpp-pythonllama.cpp (CLI)1# Pull a specific file
2huggingface-cli download ggfox00000/gpt-oss-20b-GGUF gpt-oss-20b-Q4_K_M.gguf \
3 --local-dir ./gguf
4
5# Run with llama-cli
6./llama-cli -m ./gguf/gpt-oss-20b-Q4_K_M.gguf \
7 -n 512 --temp 0.2 -p "Quel est le meilleur réglage pour la cuisson d'un steak ?"huggingface_hub.snapshot_download("openai/gpt-oss-20b") — sharded safetensorsllama.cpp/convert_hf_to_gguf.py … --outtype f16 → F16 GGUFllama.cpp/build/bin/llama-quantize <fp16.gguf> <out.gguf> <quant> for each of Q4_K_M, Q5_K_M, Q8_0scripts/quantize_to_gguf/quantize.sh (in the original Gilbert × May-IA project) for the full reproducible workflow.