You can deploy this model and start earning money today!

TevunahAi/
Llama-3.1-8B-Instruct-TevunahAi-GPTQ

Name: Llama-3.1-8B-Instruct-TevunahAi-GPTQ
Availability: InStock

safetensors

llama

meta-llama/Llama-3.1-8B-Instruct

Views

No views yet

Meta Llama-3.1-8B-Instruct - TevunahAi Ultra-Hybrid GPTQ with EoRA

Model Details

Property	Value
Base Model	meta-llama/Llama-3.1-8B-Instruct
Architecture	Llama 3.1 Dense Decoder-only Transformer
Parameters	8B
Context Length	128,000 tokens
Languages	EN, DE, FR, IT, PT, HI, ES, TH (8 languages)
Quantization	TevunahAi Ultra-Hybrid GPTQ + EoRA
Original Size	~16 GB (BF16)
Quantized Size	~6-7 GB
Compression	~60% reduction
Active VRAM	~7.08 GB (with inference overhead)
License	Llama 3.1 Community License

Architecture Breakdown

Meta Llama-3.1-8B-Instruct is a multilingual instruction-tuned model trained on 15T+ tokens:

Layer Composition (32 total layers)

32 Transformer Decoder Layers: Standard Llama architecture
32 Attention Heads: GQA with 8 KV heads (4:1 ratio)
Hidden Size: 4,096
Intermediate Size: 14,336
Vocab Size: 128,256
SiLU (Swish) Activation: Smooth gating function
RMSNorm + RoPE: theta=500000 for extended context
SwiGLU MLP: gate_proj + up_proj + down_proj

Why This Matters

128K context window: Process very long documents
8 languages: EN, DE, FR, IT, PT, HI, ES, TH
SFT + RLHF: Instruction-tuned with preference optimization
Tool use support: Function calling capabilities
15T+ training tokens: Extensive pretraining

Quantization Strategy

TevunahAi Ultra-Hybrid Mixed-Precision with EoRA-2048 Boundary Layers

This quantization uses EoRA (Error-optimized Low-Rank Adaptation) - NVIDIA's technique for recovering quantization error through learned low-rank adapters applied during the quantization process.

Component	Precision	EoRA Rank	Rationale
Layer 0 (all projections)	INT8	2048	Maximum error correction at input - errors propagate through entire model
Layer 31 (all projections)	INT8	2048	Maximum error correction at output - directly affects token prediction
Attention Q/K/V/O (layers 1-30)	INT8	128	Quality preservation for attention
MLP gate/up/down (layers 1-24)	INT4	128	Maximum compression in middle layers
MLP gate/up/down (layers 25-30)	INT8	128	Higher precision near output
Embeddings	BF16	-	Preserved for 128K vocab accuracy
LM Head	BF16	-	Preserved for output quality

Why EoRA-2048 Boundaries?

Layer 0: First layer errors compound through all 31 subsequent layers
Layer 31: Final layer directly determines next token prediction
Rank 2048: Maximum error correction capacity (~8-10% recovery vs ~5% for rank 128)
224 layer-specific rules: Not a blanket quantization - each projection optimized individually

Calibration

2,048 samples (8x industry standard of 256)
4,096 sequence length
Diverse datasets: UltraChat, SlimOrca, Code-Feedback, Orca-Math
Premium calibration for multilingual instruction following

Performance Benchmarks

Qualitative Tests (8/8 passed)

Test	Result	Speed	Details
Basic Instruction	✅ PASS	37.4 tok/s	Clear, helpful response
Reasoning	✅ PASS	37.1 tok/s	Sheep puzzle - correct answer (9)
Code Generation	✅ PASS	42.1 tok/s	Palindrome - 4/4 elements
German	✅ PASS	38.7 tok/s	7/7 German indicators
French	✅ PASS	48.6 tok/s	7/7 French indicators
Spanish	✅ PASS	42.6 tok/s	7/8 Spanish indicators
Summarization	✅ PASS	46.9 tok/s	4/4 key terms
Creative Writing	✅ PASS	47.2 tok/s	Valid haiku structure

Quantized Model Benchmarks (lm-eval-harness, 0-shot)

Task	Score	Metric	Stderr
Winogrande	76.00%	acc	±4.29%
HellaSwag	67.00%	acc_norm	±4.73%
TruthfulQA MC2	51.94%	acc	±4.15%
ARC-Challenge	50.00%	acc_norm	±5.03%

Quick test with 100 samples per task.

Inference Performance

Metric	Value
VRAM Usage	7.08 GB
Generation Speed	37-48 tok/s
Load Time	~36 seconds
Tests Passed	8/8

Multilingual Quality Examples

German - Explained AI concepts fluently:

Künstliche Intelligenz (KI) ist eine Disziplin, die sich mit der Entwicklung 
von Computerprogrammen beschäftigt, die Fähigkeiten wie Lernen, Problemlösung, 
Entscheidungsfindung und Kommunikation aufweisen...

French - Described Eiffel Tower accurately:

La Tour Eiffel est un monument emblématique de Paris, construit pour 
l'Exposition Universelle de 1889 et conçu par les ingénieurs Gustave Eiffel 
et Maurice Koechlin...

Spanish - Listed exercise benefits comprehensively:

Hacer ejercicio regularmente ofrece una variedad de beneficios para la salud 
física y mental...

Usage

GPTQModel (Recommended)

python

from gptqmodel import GPTQModel
from transformers import AutoTokenizer

model = GPTQModel.from_quantized(
    "TevunahAi/Llama-3.1-8B-Instruct-TevunahAi-GPTQ",
    device_map="auto",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(
    "TevunahAi/Llama-3.1-8B-Instruct-TevunahAi-GPTQ",
    trust_remote_code=True
)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain quantum computing in simple terms."},
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer([text], return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.7,
    top_p=0.9,
    do_sample=True,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Transformers

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "TevunahAi/Llama-3.1-8B-Instruct-TevunahAi-GPTQ",
    device_map="auto",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
    "TevunahAi/Llama-3.1-8B-Instruct-TevunahAi-GPTQ",
    trust_remote_code=True
)

# Use same generation code as above

vLLM (Production)

bash

pip install -U vllm

vllm serve TevunahAi/Llama-3.1-8B-Instruct-TevunahAi-GPTQ \
    --max-model-len 8192 \
    --trust-remote-code

Installation

pip install gptqmodel transformers>=4.48

Multilingual Capabilities

Llama-3.1-8B supports 8 languages with strong performance:

Language	Code	Quality
English	EN	⭐⭐⭐⭐⭐
German	DE	⭐⭐⭐⭐⭐
French	FR	⭐⭐⭐⭐⭐
Italian	IT	⭐⭐⭐⭐
Portuguese	PT	⭐⭐⭐⭐
Hindi	HI	⭐⭐⭐⭐
Spanish	ES	⭐⭐⭐⭐⭐
Thai	TH	⭐⭐⭐⭐

Multilingual Examples

python

# German
messages = [{"role": "user", "content": "Erkläre mir künstliche Intelligenz."}]

# French
messages = [{"role": "user", "content": "Décrivez la Tour Eiffel."}]

# Spanish
messages = [{"role": "user", "content": "¿Cuáles son los beneficios del ejercicio?"}]

Known Issues

Tokenizer regex warning: Can be safely ignored or fixed with fix_mistral_regex=True when loading tokenizer

Memory Requirements

Inference (quantized model)

Context Length	VRAM Required
Short (4K)	7-8 GB
Medium (16K)	10-12 GB
Long (32K)	14-16 GB
Extended (64K)	20-24 GB
Full (128K)	32+ GB

Tested on: RTX 5000 Ada (32GB) - 7.08 GB active VRAM during inference

Quantization (reproduction)

GPU: RTX 5000 Ada 32GB
RAM: 32GB+ recommended
Method: Full GPU acceleration

Quantization Details

Specification	Value
Method	GPTQ + Ultra-Hybrid + EoRA
Quantizer	GPTQModel
EoRA Boundary Rank	2048 (layers 0 & 31)
EoRA Standard Rank	128 (layers 1-30)
Calibration Samples	2,048 (8x industry standard)
Sequence Length	4,096 tokens
Group Size	128
desc_act	False
sym	True (symmetric quantization)
Bits (default)	4
Layer Rules	224 custom precision rules

Use Cases

Ideal for:

💬 General chat and assistance
🌍 Multilingual applications (8 languages)
💻 Code generation and explanation
📝 Content creation and writing
📄 Long document processing (128K context)
🔧 Tool use and function calling
🏠 Local deployment (7GB VRAM)

Technical Specifications

Specification	Value
Model Family	Meta Llama 3.1
Variant	8B-Instruct
Total Parameters	8B
Total Layers	32
Hidden Size	4,096
Intermediate Size	14,336
Attention Heads	32
KV Heads	8 (GQA)
Activation	SiLU (Swish)
Normalization	RMSNorm
Position Encoding	RoPE (theta=500000)
Context Length	128,000
Vocab Size	128,256
Training Tokens	15T+
Languages	8
Post-training	SFT + RLHF

Acknowledgments

Meta AI for developing the Llama 3.1 model family
NVIDIA for the EoRA (Error-optimized Low-Rank Adaptation) technique used in this quantization
GPTQModel team for the excellent quantization framework

License

Llama 3.1 Community License - See Meta's license terms for usage restrictions and requirements.

Citation

bibtex

@software{llama31_8b_gptq_2025,
  title = {Meta Llama-3.1-8B-Instruct - TevunahAi Ultra-Hybrid GPTQ with EoRA},
  author = {TevunahAi},
  year = {2025},
  note = {Ultra-Hybrid GPTQ with EoRA-2048 boundary layers for maximum quality retention},
  url = {https://huggingface.co/TevunahAi/Llama-3.1-8B-Instruct-TevunahAi-GPTQ}
}

@misc{llama31_2024,
  title = {The Llama 3 Herd of Models},
  author = {Meta AI},
  year = {2024},
  url = {https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct}
}

https://huggingface.co/TevunahAi