from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

# Load model and tokenizer
model = AutoModelForSequenceClassification.from_pretrained("surendrapratap/gemma-2b-classifier-gguf")
tokenizer = AutoTokenizer.from_pretrained("surendrapratap/gemma-2b-classifier-gguf")

# Classify text
text = "Your text here"
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)

with torch.no_grad():
    outputs = model(**inputs)
    logits = outputs.logits
    predicted_class_id = logits.argmax().item()
    probabilities = torch.softmax(logits, dim=-1)
    confidence = probabilities[0][predicted_class_id].item()

print(f"Predicted class: {predicted_class_id}")
print(f"Confidence: {confidence:.4f}")

GGUF Conversion for llama.cpp

To convert this model to GGUF format for use with llama.cpp:

bash

# Clone llama.cpp
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp

# Build llama.cpp
make

# Convert to GGUF
python convert_hf_to_gguf.py /path/to/this/model --outdir ./

# Quantize (optional)
./llama-quantize model.gguf model-q4_0.gguf q4_0

MediaPipe LLM Usage

python

import mediapipe as mp
from mediapipe.tasks.python import genai

# Load with MediaPipe LLM (after GGUF conversion)
llm = genai.LlmInference.create_from_options(
    genai.LlmInferenceOptions(model_path="model.gguf")
)

# Generate response
response = llm.generate_response("Your text here")
print(response.generated_text)

Model Performance

Inference Speed: ~50-100ms per sample (CPU)
Memory Usage: ~2GB (full precision), ~500MB (quantized)
Accuracy: Optimized for the specific classification task

Deployment Options

Server Deployment: Use PyTorch directly with FastAPI/Flask
Mobile/Edge: Convert to GGUF and use with llama.cpp
Web: Convert to ONNX.js or TensorFlow.js
MediaPipe: Use GGUF format with MediaPipe LLM

Files Included

model.safetensors: Model weights in SafeTensors format
config.json: Model configuration
tokenizer.json: Tokenizer configuration
tokenizer.model: SentencePiece tokenizer model
label_encoder.pkl: Label encoder for class names
training_metadata.json: Training information

License

This model follows the Gemma license terms. Please ensure compliance when using in production.

Citation

bibtex

@misc{gemma-2b-classifier-gguf,
  title={Gemma 2B Text Classifier (GGUF Compatible)},
  author={Your Name},
  year={2024},
  publisher={Hugging Face},
  url={https://huggingface.co/surendrapratap/gemma-2b-classifier-gguf}
}