Behemoth-X-123B-v2.1 — Quantized (compressed-tensors for vLLM)

TL;DR

• Quantized with branch W4A16 (INT4 weights / A16 activations).
• Serve with vLLM using --quantization compressed-tensors.
• Calibration uses a chat-formatted dataset; 512 samples at 2048 tokens max seq length. :contentReference[oaicite:0]{index=0}

Revisions & Branches

The main branch is a landing page (model card + links). All runnable artifacts live under per-revision branches.

• main — placeholder / landing page
• W4A16 — 4-bit weights / 16-bit activations builds and runtime assets

• main: https://huggingface.co/TheHouseOfTheDude/Behemoth-X-123B-v2.1_Compressed-Tensors/tree/main
• W4A16: https://huggingface.co/TheHouseOfTheDude/Behemoth-X-123B-v2.1_Compressed-Tensors/tree/W4A16

What’s inside (per revision)

• Sharded quantized weights (*.safetensors) + index (model.safetensors.index.json)
• config.json with compressed-tensors metadata (weight_format, quantization, quantization_config, etc.)
• Tokenizer artifacts (tokenizer.json, tokenizer.model, merges/vocab if applicable)
• Optional: chat_template.jinja (inherits the parent finetune’s chat style)

Exact files can vary by branch; see Files and versions for each revision.

Quantization recipe (based on the provided script; parameterized for W4A16)

• Method: AWQ via llm-compressor (one-shot), targeting Linear layers, with lm_head ignored during quantization. :contentReference[oaicite:1]{index=1}
• Calibration dataset: neuralmagic/LLM_compression_calibration (split: train), which contains message-structured conversations rendered with tokenizer.apply_chat_template. :contentReference[oaicite:2]{index=2}
• Calibration samples: 512 (num_calibration_samples=512). :contentReference[oaicite:3]{index=3}
• Max calibration sequence length: 2048 (max_seq_length=2048). :contentReference[oaicite:4]{index=4}
• Group size: 128 groups for weight quantization. :contentReference[oaicite:5]{index=5}
• Export: saved with save_compressed=True so vLLM reads compressed-tensors metadata. :contentReference[oaicite:6]{index=6}

The attached script shows an INT8 configuration; this release applies the same pipeline with num_bits=4 for the W4A16 branch (INT4 weights, A16 activations). All other knobs remain as above (dataset, samples, sequence length, targets, ignore list, group size). :contentReference[oaicite:7]{index=7}

Quickstart — vLLM (compressed-tensors)

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
vllm serve TheHouseOfTheDude/Behemoth-X-123B-v2.1_Compressed-Tensors \
  --quantization compressed-tensors \
  --tensor-parallel-size 8 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.70 \
  --dtype bfloat16

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "TheHouseOfTheDude/Behemoth-X-123B-v2.1_Compressed-Tensors",
    "messages": [
      {"role":"system","content":"You are Behemoth-X — helpful, precise, and safe."},
      {"role":"user","content":"Give three robust strategies for long-context retrieval."}
    ],
    "max_tokens": 512,
    "temperature": 0.7,
    "top_p": 0.95
  }'

Note: compressed-tensors is a vLLM runtime format. Loading directly with vanilla 🤗 Transformers is not supported.
For Transformers, use a compatible quant (e.g., GPTQ/AWQ export) or full-precision weights.

Prompting / Chat Template

• Keep a concise system message to set behavior/tone.
• Structure user prompts clearly; enumerate steps for multi-part tasks.

Intended use

• General instruction-following assistants
• Long-form drafting & summarization
• RAG/agent pipelines (pair with a retriever/tool layer)

Always review the parent/base model’s license and evaluate on your domain before production use.

Lineage

• Finetuned parent: https://huggingface.co/TheDrummer/Behemoth-X-123B-v2.1
• This repo: Quantized child of the finetune (compressed-tensors for vLLM)

Hardware tips (rule-of-thumb)

• 123B-class models benefit from multi-GPU (tensor parallel) for throughput.
• Long contexts are KV-cache heavy — tune --max-model-len and batch size.
• Use BF16 on GPUs with native support; otherwise FP16.
• Enable P2P/NVLink where possible; consider CUDA Graphs if stable.

Changelog

• v2.1 (current) — Initial compressed-tensors release; branch W4A16 published; model card marked Quantized.