SmolLM2 1.7B Fungi RAG Agent Distill LoRA GGUF
This repository contains a fine-tuned LoRA adapter for HuggingFaceTB/SmolLM2-1.7B-Instruct,
exported in GGUF format for llama.cpp and also included as PEFT adapter files.
This is not a standalone full GGUF model. Load it with a compatible SmolLM2 1.7B Instruct
base GGUF, such as the Hugging Face SmolLM2 1.7B Instruct Q4_K_M GGUF.
Files
gguf/smollm2-1.7b-fungi-rag-agent-distill-checkpoint-800-lora-f16.gguf:
llama.cpp LoRA adapter exported from the best measured checkpoint.peft/adapter_model.safetensors and peft/adapter_config.json:
original PEFT LoRA adapter files for Transformers/PEFT workflows.evaluation/:
evaluation summary, report, and plots from the real Fungi RAG multi-agent workflow.
Intended Use
The adapter was tuned for a fungi research-agent workflow that uses:
rag.search for evidence-grounded mycology answers,safety.review for wild-mushroom edibility, field-identification, and safety-sensitive prompts,- numeric source citations from retrieved evidence,
- explicit uncertainty handling when evidence is partial, missing, or conflicting.
It is best treated as a planner/safety/research-agent adapter, not as a general-purpose chat model.
Usage With llama.cpp
1llama-cli \
2 -m /path/to/smollm2-1.7b-instruct-q4_k_m.gguf \
3 --lora gguf/smollm2-1.7b-fungi-rag-agent-distill-checkpoint-800-lora-f16.gguf \
4 -p "<|im_start|>system
5You are a fungi RAG assistant. Use rag.search for evidence-backed answers and safety.review for safety-sensitive claims.<|im_end|>
6<|im_start|>user
7Tell me whether a white-gilled wild mushroom under oak is safe to eat.<|im_end|>
8<|im_start|>assistant
9" \
10 -n 180 -c 2048 -ngl 99 --temp 0
Training Data
Fine-tuning used the linked dataset:
Prepared split counts:
| Split | Examples |
|---|
| Train | 3,387 |
| Validation | 298 |
| Test | 300 |
| Total | 3,985 |
The dataset was generated for the project-owned fungi RAG system and validated for schema,
tool-call contract, citation integrity, and safety boundaries before training.
Source Code
Project codebase:
Evaluation report in the GitHub repo:
Training Summary
Base model: HuggingFaceTB/SmolLM2-1.7B-Instruct
Adapter type: LoRA
LoRA rank: 16
LoRA alpha: 32
LoRA dropout: 0.05
Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Best measured checkpoint: step 800
Final run step: 846
Validation results:
| Metric | Baseline | Checkpoint 800 | Change |
|---|
| Eval loss | 2.8869 | 0.3857 | -86.64% |
| Implied perplexity | 17.94 | 1.47 | -91.80% |
| Mean token accuracy | n/a | 0.8883 | n/a |
Evaluation Summary
The adapter improved sampled agentic behavior accuracy from 33.3% to 47.6%, with the strongest gains in:
- unsafe wild-mushroom field-ID refusal,
- uncertainty handling,
- safety-sensitive answer style.
In the real multi-agent workflow, the adapter produced safer and more qualified prose, but some outputs
failed the current validator because they used parenthetical citations like (5) instead of required
bracket citations like [5]. See evaluation/report.md for details.
Runtime on an NVIDIA GeForce GTX 1660 SUPER through llama.cpp CUDA:
| Evaluation | Generation throughput | Peak VRAM |
|---|
| Behavior prompts | 61.0 tokens/sec | 3,098 MB |
| Real workflow prompts | 65.6 tokens/sec | 3,092 MB |
Safety And Limitations
This model is for academic fungi research-assistant experiments only.
Do not use it for:
- mushroom edibility decisions,
- field identification,
- medical advice,
- dosage or treatment decisions,
- emergency toxicology decisions.
For wild-mushroom exposure or ingestion, contact poison control or emergency medical services.
For identification, use a qualified local expert with full specimen evidence.
