f16: Full precision (largest, best quality) - ~3GB
q4_k_m: 4-bit quantization (smallest, recommended for most use cases) - ~1GB
q5_k_m: 5-bit quantization (balanced quality/size) - ~1.2GB
q8_0: 8-bit quantization (high quality, larger size) - ~1.8GB

Usage with Different Backends

1. llama.cpp (Recommended for GGUF)

Installation

bash

# Clone llama.cpp
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
mkdir build && cd build
cmake .. -DGGML_CUDA=ON
cmake --build . --config Release -j

Basic Usage

bash

# Using llama-cli
./llama.cpp/build/bin/llama-cli \
    -m path/to/intent-classifier-q4_k_m.gguf \
    -p "Context:
Previous search: Smartphones
State: SEARCH_RESULTS
Last command: show_list
Products (6): iPhone 15 Pro, Samsung Galaxy S24, OnePlus 12, Google Pixel 8, Xiaomi 14, Nothing Phone 2
Product count: 6

Query: show me that one"

# Using llama-server (for API access)
./llama.cpp/build/bin/llama-server \
    -m path/to/intent-classifier-q4_k_m.gguf \
    --port 8080

Python Example

python

from llama_cpp import Llama

# Load model
llm = Llama(
    model_path="path/to/intent-classifier-q4_k_m.gguf",
    n_ctx=2048,  # Context window
    n_threads=4  # Number of CPU threads
)

# Prepare prompt (ChatML format)
prompt = "<|im_start|>system\nClassify the user's intent based on the query and context. Intent can be: search, show_detail, go_back, or close.<|im_end|>\n<|im_start|>user\nContext:\nPrevious search: Smartphones\nState: SEARCH_RESULTS\nLast command: show_list\nProducts (6): iPhone 15 Pro, Samsung Galaxy S24, OnePlus 12, Google Pixel 8, Xiaomi 14, Nothing Phone 2\nProduct count: 6\n\nQuery: show me that one<|im_end|>\n<|im_start|>assistant\nshow_detail<|im_end|>"

# Generate
response = llm(
    prompt,
    max_tokens=128,
    temperature=0.7,
    stop=["<|im_end|>", "<|im_start|>"]
)

print(response['choices'][0]['text'])

2. Ollama

Import GGUF Model to Ollama

Option 1: Using Modelfile (Recommended)

Create a Modelfile:

dockerfile

FROM ./intent-classifier-q4_k_m.gguf

TEMPLATE "<|im_start|>system\nClassify the user's intent based on the query and context. Intent can be: search, show_detail, go_back, or close.<|im_end|>\n<|im_start|>user\nContext:\nPrevious search: Smartphones\nState: SEARCH_RESULTS\nLast command: show_list\nProducts (6): iPhone 15 Pro, Samsung Galaxy S24, OnePlus 12, Google Pixel 8, Xiaomi 14, Nothing Phone 2\nProduct count: 6\n\nQuery: show me that one<|im_end|>\n<|im_start|>assistant\nshow_detail<|im_end|>"

PARAMETER temperature 0.7
PARAMETER num_predict 128
PARAMETER stop "<|im_end|>"
PARAMETER stop "<|im_start|>"

Import the model:

ollama create intent-classifier -f Modelfile

Option 2: Direct Import

bash

# Import GGUF file directly
ollama import intent-classifier-q4_k_m.gguf

Usage with Ollama

bash

# Command line
ollama run intent-classifier "Context:
Previous search: Smartphones
State: SEARCH_RESULTS
Last command: show_list
Products (6): iPhone 15 Pro, Samsung Galaxy S24, OnePlus 12, Google Pixel 8, Xiaomi 14, Nothing Phone 2
Product count: 6

Query: show me that one"

# With context
ollama run intent-classifier "Context: Previous search: Smartphones\nQuery: show me that one"

Python API

python

import requests

# Generate
response = requests.post(
    "http://localhost:11434/api/generate",
    json={
        "model": "intent-classifier",
        "prompt": "Context:
Previous search: Smartphones
State: SEARCH_RESULTS
Last command: show_list
Products (6): iPhone 15 Pro, Samsung Galaxy S24, OnePlus 12, Google Pixel 8, Xiaomi 14, Nothing Phone 2
Product count: 6

Query: show me that one",
        "stream": False,
        "options": {
            "temperature": 0.7,
            "num_predict": 128,
            "stop": ["<|im_end|>", "<|im_start|>"]
        }
    }
)

print(response.json()["response"])

Chat API

python

import requests

response = requests.post(
    "http://localhost:11434/api/chat",
    json={
        "model": "intent-classifier",
        "messages": [
            {"role": "system", "content": "Classify the user's intent based on the query and context. Intent can be: search, show_detail, go_back, or close."},
            {"role": "user", "content": "Context:
Previous search: Smartphones
State: SEARCH_RESULTS
Last command: show_list
Products (6): iPhone 15 Pro, Samsung Galaxy S24, OnePlus 12, Google Pixel 8, Xiaomi 14, Nothing Phone 2
Product count: 6

Query: show me that one"}
        ],
        "stream": False
    }
)

print(response.json()["message"]["content"])

3. vLLM (For Merged Models)

Note: vLLM works best with HuggingFace format models. Use the merged model instead of GGUF.

Installation

pip install vllm

Usage

python

from vllm import LLM, SamplingParams

# Load merged model (not GGUF)
llm = LLM(
    model="mudasir13cs/E-commerce-intent-classifier",
    trust_remote_code=True,
    max_model_len=2048
)

# Prepare prompt
prompt = "<|im_start|>system\nClassify the user's intent based on the query and context. Intent can be: search, show_detail, go_back, or close.<|im_end|>\n<|im_start|>user\nContext:\nPrevious search: Smartphones\nState: SEARCH_RESULTS\nLast command: show_list\nProducts (6): iPhone 15 Pro, Samsung Galaxy S24, OnePlus 12, Google Pixel 8, Xiaomi 14, Nothing Phone 2\nProduct count: 6\n\nQuery: show me that one<|im_end|>\n<|im_start|>assistant\nshow_detail<|im_end|>"

# Sampling parameters
sampling_params = SamplingParams(
    temperature=0.7,
    max_tokens=128,
    stop=["<|im_end|>", "<|im_start|>"]
)

# Generate
outputs = llm.generate([prompt], sampling_params)
print(outputs[0].outputs[0].text)

vLLM Server

bash

# Start server
python -m vllm.entrypoints.openai.api_server \
    --model mudasir13cs/E-commerce-intent-classifier \
    --trust-remote-code \
    --port 8000

# Use OpenAI-compatible API
curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "intent-classifier",
        "prompt": "Context:
Previous search: Smartphones
State: SEARCH_RESULTS
Last command: show_list
Products (6): iPhone 15 Pro, Samsung Galaxy S24, OnePlus 12, Google Pixel 8, Xiaomi 14, Nothing Phone 2
Product count: 6

Query: show me that one",
        "max_tokens": 128,
        "temperature": 0.7
    }'

4. Text Generation Inference (TGI)

Note: TGI works with HuggingFace format models. Use the merged model.

Installation

bash

# Using Docker (recommended)
docker pull ghcr.io/huggingface/text-generation-inference:latest

Usage

bash

docker run --gpus all \
    -p 8080:80 \
    -v /path/to/model:/data \
    ghcr.io/huggingface/text-generation-inference:latest \
    --model-id mudasir13cs/E-commerce-intent-classifier \
    --trust-remote-code

Python Client

python

from text_generation import Client

client = Client("http://localhost:8080")

response = client.generate(
    prompt="<|im_start|>system\nClassify the user's intent based on the query and context. Intent can be: search, show_detail, go_back, or close.<|im_end|>\n<|im_start|>user\nContext:\nPrevious search: Smartphones\nState: SEARCH_RESULTS\nLast command: show_list\nProducts (6): iPhone 15 Pro, Samsung Galaxy S24, OnePlus 12, Google Pixel 8, Xiaomi 14, Nothing Phone 2\nProduct count: 6\n\nQuery: show me that one<|im_end|>\n<|im_start|>assistant\nshow_detail<|im_end|>",
    max_new_tokens=128,
    temperature=0.7,
    stop_sequences=["<|im_end|>", "<|im_start|>"]
)

print(response.generated_text)

5. Transformers (For Merged Models)

Note: Use the merged HuggingFace model for Transformers.

python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load merged model
model = AutoModelForCausalLM.from_pretrained(
    "mudasir13cs/E-commerce-intent-classifier",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
    "mudasir13cs/E-commerce-intent-classifier",
    trust_remote_code=True
)

# Prepare input
messages = [
    {"role": "system", "content": "Classify the user's intent based on the query and context. Intent can be: search, show_detail, go_back, or close."},
    {"role": "user", "content": "Context:
Previous search: Smartphones
State: SEARCH_RESULTS
Last command: show_list
Products (6): iPhone 15 Pro, Samsung Galaxy S24, OnePlus 12, Google Pixel 8, Xiaomi 14, Nothing Phone 2
Product count: 6

Query: show me that one"}
]

# Apply chat template
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

# Generate
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=128,
    temperature=0.7,
    do_sample=True,
    pad_token_id=tokenizer.eos_token_id
)

# Decode
response = tokenizer.decode(
    outputs[0][inputs.input_ids.shape[1]:],
    skip_special_tokens=True
)
print(response)

Prompt Format

The model uses ChatML format (Qwen2.5's native format):

<|im_start|>system
Classify the user's intent based on the query and context. Intent can be: search, show_detail, go_back, or close.<|im_end|>
<|im_start|>user
Context:
Previous search: Smartphones
State: SEARCH_RESULTS
Last command: show_list
Products (6): iPhone 15 Pro, Samsung Galaxy S24, OnePlus 12, Google Pixel 8, Xiaomi 14, Nothing Phone 2
Product count: 6

Query: show me that one<|im_end|>
<|im_start|>assistant
show_detail<|im_end|>

Input Structure

System Message:

Classify the user's intent based on the query and context. Intent can be: search, show_detail, go_back, or close.

User Message (Context + Query):

Context:
Previous search: [category]
State: [SEARCH_RESULTS|PRODUCT_DETAIL|INITIAL]
Last command: [show_list|show_item|go_back|close]
Products (N): [product1, product2, ...]
Product count: N

Query: [user query]

Expected Output:

show_detail

Example Prompts

Example 1: Pronoun Resolution

Context:
Previous search: Smartphones
State: SEARCH_RESULTS
Last command: show_list
Products (6): iPhone 15 Pro, Samsung Galaxy S24, OnePlus 12, Google Pixel 8, Xiaomi 14, Nothing Phone 2
Product count: 6

Query: show me that one

Example 2: Ellipsis Expansion

Context:
Previous search: Laptops
State: SEARCH_RESULTS
Last command: show_list
Products (5): MacBook Pro, Dell XPS, HP Spectre, Lenovo ThinkPad, ASUS ZenBook
Product count: 5

Query: under 50000

Example 3: Ordinal Reference

Context:
Previous search: Headphones
State: SEARCH_RESULTS
Last command: show_list
Products (4): Sony WH-1000XM5, Bose QuietComfort, AirPods Max, Sennheiser Momentum
Product count: 4

Query: show me the second one

Quantization Comparison

Format	Size	Quality	Use Case
f16	~3GB	Best	Maximum quality, sufficient VRAM
q8_0	~1.8GB	Excellent	High quality, moderate VRAM
q5_k_m	~1.2GB	Very Good	Balanced quality/size
q4_k_m	~1GB	Good	Smallest size, limited VRAM

Performance

Inference Speed: Optimized for CPU and GPU (CUDA)
Memory Usage: Significantly lower than original models
Quality: Minimal quality loss with quantization
Test Accuracy: 100% (394/394) on test set

Download

Download the desired quantization level:

bash

# Using huggingface-cli
huggingface-cli download mudasir13cs/E-commerce-intent-classifier-gguf \
    intent-classifier-q4_k_m.gguf \
    --local-dir ./models

# Or download all formats
huggingface-cli download mudasir13cs/E-commerce-intent-classifier-gguf \
    --local-dir ./models

Related Models

Merged Model (HuggingFace Format): mudasir13cs/E-commerce-intent-classifier
- Full model in HuggingFace format
- Can be used with Transformers, Unsloth, or other HF-compatible libraries
- Suitable for further fine-tuning or inference
Original LoRA Adapter: See merged model repository for LoRA adapter details

Backend Comparison

Backend	Format	Best For	Pros	Cons
llama.cpp	GGUF	CPU/GPU inference, edge devices	Fast, low memory, cross-platform	Limited to GGUF format
Ollama	GGUF	Local development, easy deployment	Simple API, auto-manages models	Requires model import
vLLM	HF	High-throughput serving	Very fast, batching support	Requires HF format, more memory
TGI	HF	Production serving	Optimized serving, Docker support	Requires HF format
Transformers	HF	Research, fine-tuning	Full flexibility, easy integration	Slower inference, more memory

Requirements

For GGUF Models (llama.cpp, Ollama)

llama.cpp (for C/C++ usage)
llama-cpp-python (for Python usage)
Ollama (optional, for Ollama backend)
CUDA (optional, for GPU acceleration)

For Merged Models (vLLM, TGI, Transformers)

vLLM (for high-throughput serving)
Text Generation Inference (for production serving)
Transformers (for research/fine-tuning)
CUDA (recommended for GPU acceleration)

Citation

If you use this model, please cite:

bibtex

@software{ecommerce_agent_models,
  title = {E-commerce Agent Models - Intent Classifier},
  author = {Syed Mudasir},
  year = {2025},
  url = {https://huggingface.co/mudasir13cs/E-commerce-intent-classifier-gguf}
}

License

Apache 2.0

Notes

GGUF models are optimized for inference, not training
Use q4_k_m for most production deployments
f16 format is recommended for maximum quality if VRAM allows
For training or further fine-tuning, use the merged HuggingFace model
The model was trained on English e-commerce data and performs best on similar queries