Views
No views yet
Revisions & Branches
- main — placeholder landing branch. The canonical README lives here; model files may be minimal.
- NVFP4 - 4-bit weights / 4-bit activations (but acts like 16-bit activations)
- W4A16 — Symmetrical AWQ 4‑bit weights / 16‑bit activations builds and related assets are published under this revision.
- W8A16 — Symmetrical AWQ 8‑bit weights / 16‑bit activations builds and related assets are published under this revision.
- W8A8-FP8_BLOCK — 8‑bit weights / 8‑bit activations, FP8 quality but BLOCK style, to use Cutlas on Blackwell SM12.0 (Needs latest VLLM)
This repository hosts multiple quantizations of the finetuned parent model for vLLM using the compressed-tensors runtime format.
TL;DR
- This repo is quantized (e.g., AWQ W4A16, AWQ W4A16_ASYM, and INT8 W8A16) for vLLM.
- Load with vLLM using
--quantization compressed-tensors(select the branch with your desired quant).- Typical AWQ recipe: group_size=128, keep
lm_headin higher precision; uses the upstream Mistral‑Instruct chat template.
.safetensors (model-00001-of-XXXXX.safetensors + model.safetensors.index.json)config.json with compressed-tensors metadatatokenizer.json, tokenizer.model)chat_template.jinjaREADME.mdExact file list may vary by release; see Files and versions.
pip install vllm1# Example: tensor parallel across 8 GPUs
2CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 vllm serve TheHouseOfTheDude/Behemoth-R1-123B-v2_Compressed-Tensors --quantization compressed-tensors --tensor-parallel-size 8 --max-model-len 32768 --gpu-memory-utilization 0.70 --dtype bfloat16 # or float16 on GPUs without strong BF161curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
2 "model": "TheHouseOfTheDude/Behemoth-R1-123B-v2_Compressed-Tensors",
3 "messages": [
4 {"role":"system","content":"You are Behemoth, helpful, precise, and safe."},
5 {"role":"user","content":"Outline a retrieval pipeline for legal documents."}
6 ],
7 "max_tokens": 512,
8 "temperature": 0.7,
9 "top_p": 0.95
10 }'Note:compressed-tensorsis a vLLM runtime format. Loading this artifact directly in vanilla 🤗 Transformers is not supported; use vLLM for inference. If you need Transformers inference, use a different export (e.g., GPTQ/AWQ.safetensorscompatible with Transformers) or full‑precision weights.
chat_template.jinja is present, it is applied automatically by apply_chat_template within serving stacks that support it.temperature: 0.4–0.9 (0.6–0.8 common)top_p: 0.9–0.95max_new_tokens: 256–2048+repetition_penalty: 1.05–1.15--max-model-len and batch size accordingly.