from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "yamraj047/my_optimal_model-merged",
    torch_dtype=torch.float16,
    device_map="auto"
)

tokenizer = AutoTokenizer.from_pretrained("yamraj047/my_optimal_model-merged")

inputs = tokenizer("Your prompt here", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_length=200)
print(tokenizer.decode(outputs[0]))

Convert to GGUF

Base: mistralai/Mistral-7B-v0.1
LoRA Adapter: yamraj047/my_optimal_model
Format: Full model (not adapter)
Size: ~14.5 GB
Precision: FP16

This model can be converted to GGUF format for CPU inference:

python

# Use the Nepal Legal GGUF conversion script
ORIGINAL_MODEL = "yamraj047/my_optimal_model-merged"
GGUF_REPO = "yamraj047/my_optimal_model-GGUF"
# ... run conversion script

Files Structure

Same format as standard HuggingFace models:

model-*.safetensors - Model weight shards
config.json - Model configuration
tokenizer.json, tokenizer.model - Tokenizer files
generation_config.json - Generation parameters

License

Apache 2.0