Llama-3.1-8B-Instruct fine-tuned with LoRA on a solar energy FAQ dataset, quantized to Q4_K_M GGUF — runs on any platform, any OS, no CUDA required.


Format	GGUF Q4_K_M (safe — no pickle, no `.bin`)
Size	4.6 GB (original: 16 GB float16)
Platforms	Mac / Windows / Linux / CPU / GPU
Tools	llama-cpp-python · Ollama · LM Studio · Jan · GPT4All

Install & Run

Option 1 — Python API (llama-cpp-python)

Step 1 — Install llama-cpp-python (choose one):

bash

# Mac Apple Silicon — Metal GPU acceleration (FAST):
CMAKE_ARGS="-DGGML_METAL=on" pip install llama-cpp-python

# NVIDIA GPU Linux/Windows — CUDA 12.4 (FAST):
pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu124

# CPU only — any platform, no GPU needed (slower ~5 tok/s):
pip install llama-cpp-python

Step 2 — Run:

python

from llama_cpp import Llama

# Auto-downloads the GGUF from HF on first run (~4.6 GB)
llm = Llama.from_pretrained(
    repo_id="ankur1423/solar-faq-gguf",
    filename="*.gguf",
    n_ctx=2048,          # context window
    n_gpu_layers=-1,     # -1 = all layers on GPU; set 0 for CPU-only
    verbose=False,
)

# Single question
response = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are a knowledgeable assistant for a solar energy company. Answer questions accurately about solar products, manufacturing, and company operations."},
        {"role": "user",   "content": "What is a BOM?"},
    ],
    max_tokens=512,
    temperature=0.1,
    top_p=0.9,
)
print(response["choices"][0]["message"]["content"])

Multi-turn conversation:

python

from llama_cpp import Llama

SYSTEM = "You are a knowledgeable assistant for a solar energy company."

llm = Llama.from_pretrained(
    repo_id="ankur1423/solar-faq-gguf",
    filename="*.gguf",
    n_ctx=4096,
    n_gpu_layers=-1,
    verbose=False,
)

history = [{"role": "system", "content": SYSTEM}]

while True:
    user = input("You: ").strip()
    if not user or user.lower() in {"exit", "quit"}:
        break
    history.append({"role": "user", "content": user})
    resp = llm.create_chat_completion(history, max_tokens=512, temperature=0.1)
    answer = resp["choices"][0]["message"]["content"].strip()
    print(f"Assistant: {answer}\n")
    history.append({"role": "assistant", "content": answer})

Option 2 — Ollama (no Python, no code)

bash

# Install Ollama: https://ollama.com
ollama run hf.co/ankur1423/solar-faq-gguf

One command — downloads and runs interactively.

Option 3 — LM Studio (GUI, Windows/Mac/Linux)

Download LM Studio
Search ankur1423/solar-faq-gguf in the model browser
Download → Chat

Option 4 — Jan App (GUI, offline)

Download Jan
Go to Hub → search ankur1423/solar-faq-gguf
Download → Chat

Option 5 — llama.cpp CLI (raw, fastest)

bash

# macOS (homebrew):
brew install llama.cpp

# Linux:
sudo apt install llama.cpp   # Ubuntu 24.04+

# Then download GGUF and run:
llama-cli \
  -m solar-faq-Q4_K_M.gguf \
  --system-prompt "You are a knowledgeable assistant for a solar energy company." \
  -i --color -c 2048

Platform Support Matrix

Platform	Backend	RAM needed	Speed
Mac M1/M2/M3/M4	Metal GPU	6 GB	Fast
NVIDIA GPU (Linux/Windows)	CUDA	6 GB VRAM	Fast
CPU — Mac / Windows / Linux	llama.cpp CPU	6 GB RAM	~5 tok/s
Google Colab (free tier)	CPU or T4 GPU	6 GB	OK
Ollama (any OS)	auto-detect GPU/CPU	6 GB	Fast / OK
LM Studio / Jan / GPT4All	auto-detect	6 GB	Fast / OK

Minimum: 6 GB RAM/VRAM. Works on most modern laptops with no GPU.

Generation Parameters (recommended)

Parameter	Value	Notes
`temperature`	0.1	Low → factual, consistent answers
`top_p`	0.9	Nucleus sampling
`max_tokens`	256–512	FAQ answers are concise
`n_ctx`	2048	Context window (increase to 4096 for long conversations)

For creative/varied responses, raise temperature to 0.5–0.7.

Prompt Format (Llama-3 chat template)

This model uses the Llama-3 chat template. The prompt format is:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are a knowledgeable assistant for a solar energy company.<|eot_id|>
<|start_header_id|>user<|end_header_id|>

What is a BOM?<|eot_id|>
<|start_header_id|>assistant<|end_header_id|>

llama-cpp-python's create_chat_completion() handles this automatically.

Training Details


Base model	`meta-llama/Meta-Llama-3.1-8B-Instruct`
Fine-tuning method	LoRA (rank 8, 8 layers)
Dataset	~62 solar energy FAQ Q&A pairs
Training iterations	300
Learning rate	1e-4 (cosine decay → 1e-5)
Batch size	2
Max sequence length	1024 tokens
Framework	MLX-LM 0.31+ on Apple Silicon
Quantization	GGUF Q4_K_M via llama.cpp
Size reduction	16 GB float16 → 4.6 GB (−71%)
Training hardware	MacBook M4 16 GB unified memory
Training time	~20 minutes

What is GGUF Q4_K_M?

GGUF (GPT-Generated Unified Format) is a safe, portable model format used by llama.cpp.

Q4_K_M = 4-bit quantization, K-quant method, Medium size/quality tradeoff:

Most weights stored in 4 bits (vs 16 bits in float16)
Quality loss: minimal (~0.1–0.5% perplexity increase vs float16)
Speed: faster than float16 on CPU due to smaller memory bandwidth

No pickle tensors, no arbitrary code — HF security scanner marks this as safe.

Limitations

Domain-specific (solar FAQ) — best for solar energy questions; falls back to base Llama-3 behavior outside training domain
English only
Small dataset (~62 pairs) — may not generalize to all solar topics
Fine-tuned on Q4_K_M base, so further quantization artifacts possible

License

This model is derived from Meta Llama 3.1, which is licensed under the Meta Llama 3 Community License. Use is subject to Meta's acceptable use policy.

ankur1423/fine-tune-test-2

Solar FAQ — GGUF Q4_K_M (4.6 GB)

Install & Run

Option 1 — Python API (llama-cpp-python)

Option 2 — Ollama (no Python, no code)

Option 3 — LM Studio (GUI, Windows/Mac/Linux)

Option 4 — Jan App (GUI, offline)

Option 5 — llama.cpp CLI (raw, fastest)

Platform Support Matrix

Generation Parameters (recommended)

Prompt Format (Llama-3 chat template)

Training Details

What is GGUF Q4_K_M?

Limitations

License

ankur1423/
fine-tune-test-2