dragon-llama-3.1-gguf is RAG-instruct trained on top of a Llama-3.1 base model.
Benchmark Tests
Evaluated against the benchmark test: RAG-Instruct-Benchmark-Tester
1 Test Run (temperature=0.0, sample=False) with 1 point for correct answer, 0.5 point for partial correct or blank / NF, 0.0 points for incorrect, and -1 points for hallucinations.
--Accuracy Score: 94.0 correct out of 100
--Not Found Classification: 70.0%
--Boolean: 90.0%
--Math/Logic: 72.5%
--Complex Questions (1-5): 4 (Above Average - table-reading, causal)
--Summarization Quality (1-5): 4 (Above Average)
--Hallucinations: No hallucinations but a few instances of drawing on 'background' knowledge.
For test run results (and good indicator of target use cases), please see the files ("core_rag_test" and "answer_sheet" in this repo).
The inference accuracy tests were performed on this model (GGUF 4_K_M) not the original Pytorch, and it is possible that the original Pytorch may score higher, but we have chosen to use the quantized version as it is most representative of the likely use of the model for inference.
Any model can provide inaccurate or incomplete information, and should be used in conjunction with appropriate safeguards and fact-checking mechanisms.
How to Get Started with the Model
To pull the model via API:
from huggingface_hub import snapshot_download
snapshot_download("llmware/dragon-llama-3.1-gguf", local_dir="/path/on/your/machine/", local_dir_use_symlinks=False)
Load in your favorite GGUF inference engine, or try with llmware as follows:
from llmware.models import ModelCatalog
# to load the model and make a basic inference
model = ModelCatalog().load_model("llmware/dragon-llama-3.1-gguf", temperature=0.0, sample=False)
response = model.inference(query, add_context=text_sample)
Details on the prompt wrapper and other configurations are on the config.json file in the files repository.
