Views
No views yet
TL;DR
- Load with
--quantization compressed-tensorsin vLLM.- Sharded
.safetensors, tokenizer files, and (if present) achat_template.jinjaare included.- Quant recipe: W4A16_ASYM, typical group_size=128, keeping
lm_headin higher precision.
.safetensors (model-00001-of-XXXXX.safetensors + model.safetensors.index.json)config.json with compressed‑tensors metadatatokenizer.json, tokenizer.model, etc.)chat_template.jinja (inherits upstream formatting if applicable)See the upstream card for architecture details, training info, context length, and the canonical chat template.
--dtype selectable)["lm_head"] (final projection kept in higher precision)Why compressed‑tensors?
The model is exported/packed for vLLM. Generic AWQ loaders expecting raw AWQ tensors are not supported. Use vLLM with--quantization compressed-tensors.
pip install vllm1CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
2vllm serve TheHouseOfTheDude/Agatha-111B-v1-AWQ-W4A16_ASYM \
3 --quantization compressed-tensors \
4 --tensor-parallel-size 8 \
5 --max-model-len 131072 \
6 --gpu-memory-utilization 0.70 \
7 --dtype float16--dtype: Use float16 if BF16 is slower/unsupported on your GPUs; otherwise bfloat16 can be used.--max-model-len: tune to your memory + throughput goals. Long contexts primarily cost KV‑cache memory.--tensor-parallel-size to match your GPU count.1curl http://localhost:8000/v1/chat/completions \
2 -H "Content-Type: application/json" \
3 -d '{
4 "model": "TheHouseOfTheDude/Agatha-111B-v1-AWQ-W4A16_ASYM",
5 "messages": [
6 {"role":"system","content":"You are Agatha, helpful and precise."},
7 {"role":"user","content":"Give me 3 bullet points on resilient networks."}
8 ],
9 "max_tokens": 256,
10 "temperature": 0.7,
11 "top_p": 0.95
12 }'1curl http://localhost:8000/v1/completions \
2 -H "Content-Type: application/json" \
3 -d '{
4 "model": "TheHouseOfTheDude/Agatha-111B-v1-AWQ-W4A16_ASYM",
5 "prompt": "Write a short haiku about sliding-window attention:\\n",
6 "max_tokens": 120,
7 "temperature": 0.7
8 }'apply_chat_template:1from transformers import AutoTokenizer
2
3model_id = "TheHouseOfTheDude/Agatha-111B-v1-AWQ-W4A16_ASYM"
4tok = AutoTokenizer.from_pretrained(model_id, use_fast=True, trust_remote_code=True)
5
6messages = [
7 {"role": "system", "content": "You are Agatha, helpful and precise."},
8 {"role": "user", "content": "Summarize sliding-window attention in 3 lines."}
9]
10
11input_ids = tok.apply_chat_template(
12 messages, tokenize=True, add_generation_prompt=True, return_tensors="pt"
13)If this repo ships achat_template.jinja, it will be used automatically; otherwise it falls back to the upstream model’s template.
temperature: 0.6–0.9top_p: 0.9–0.95max_tokens: 256–2048 (or higher if your context allows)repetition_penalty: 1.05–1.15 (optional)min_p / top_k: optional per preference--max-model-len or batch size to fit memory.--quantization compressed-tensors.safetensors; upstream tokenizer/template preserved.