You can deploy this model and start earning money today!

oxfrug/
Llama-3-8B-instruct-awq

Name: Llama-3-8B-instruct-awq
Availability: InStock

Views

No views yet

AI-Sweden Llama-3-8B-instruct — AWQ W4A16

Official Hub files are BF16 and GGUF. EXL2 lives at oxfrug/Llama-3-8B-instruct-exl2. This repo is the vLLM / TGI pack.

python

# vLLM
from vllm import LLM
llm = LLM("oxfrug/Llama-3-8B-instruct-awq")

hf download oxfrug/Llama-3-8B-instruct-awq --local-dir ./Llama-3-8B-instruct-awq

Format is compressed-tensors from llmcompressor, not classic AutoAWQ GEMM files.

Tool: llmcompressor AWQModifier + QuantizationModifier(scheme="W4A16_ASYM")
lm_head left in higher precision
Calibration: 256 rows × 512 tokens from a Nordic/English wiki mix already used for EXL2 Swedish-heavy cal (sv_heavy.rows.jsonl). Not the frozen eval suite.
Hardware: RTX 3090

Meta Llama 3 Community License. Keep NOTICE. Derivative of Meta Llama 3 via AI Sweden’s instruct tune.

Prompt	Output
`Vad heter Sveriges huvudstad? Ett ord.`	`Huvudstaden i Sverige är Stockholm.`
`Name the capital of Sweden in one word.`	`Huvudstaden i Sverige är Stockholm.`

Same Swedish-first habit as the BF16 parent. This is not a suite score.

Pack	Repo
EXL2 4.0–6.0 bpw	oxfrug/Llama-3-8B-instruct-exl2
Eval suite (do not train)	oxfrug/nordic-instruct-eval-v2