from llama_cpp import Llama
from huggingface_hub import hf_hub_download

# Download model
model_path = hf_hub_download(
    repo_id="yamraj047/my_optimal_model-GGUF",
    filename="my-optimal-model-Q4_K_M.gguf"
)

# Load model
llm = Llama(model_path=model_path, n_ctx=2048, n_threads=4)

# Generate text
response = llm("Your prompt here", max_tokens=300)
print(response['choices'][0]['text'])

📊 Specifications

Size: ~4.07 GB (vs 14.5 GB original)
Quantization: Q4_K_M (mixed precision)
Quality: ~98% of original FP16
Speed: 2-4 min on free CPU vs 20+ min on GPU
Context: 32K tokens supported
Hardware: CPU only - no GPU needed!

💻 Use with Gradio

python

from llama_cpp import Llama
from huggingface_hub import hf_hub_download
import gradio as gr

model_path = hf_hub_download(
    repo_id="yamraj047/my_optimal_model-GGUF",
    filename="my-optimal-model-Q4_K_M.gguf"
)

llm = Llama(model_path=model_path, n_ctx=2048, n_threads=4)

def chat(message, history):
    response = llm(message, max_tokens=400, temperature=0.7)
    return response['choices'][0]['text'].strip()

demo = gr.ChatInterface(
    fn=chat,
    title="🤖 My Optimal Model Assistant"
)

demo.launch()

🔄 Model Versions

This is the GGUF quantized version of the merged FP16 model.

Version	Size	Quality	Use Case
Original FP16	14.5 GB	100%	GPU inference
GGUF Q4_K_M	4.07 GB	98%	CPU inference

📜 License

Apache 2.0