RefuelLLM-2-small, aka Llama-3-Refueled, is a Llama3-8B base model instruction tuned on a corpus of 2750+ datasets, spanning tasks such as classification, reading comprehension, structured attribute extraction and entity resolution. We're excited to open-source the model for the community to build on top of.
Architecture - Llama-3-Refueled is built on top of Llama-3-8B-instruct which is an auto-regressive language model that uses an optimized transformer architecture.
This repository contains weights for Llama-3-Refueled that are compatible for use with HuggingFace. See the snippet below for usage with Transformers:
python
1>>>import torch
2>>>from transformers import AutoModelForCausalLM, AutoTokenizer
34>>> model_id ="refuelai/Llama-3-Refueled"5>>> tokenizer = AutoTokenizer.from_pretrained(model_id)6>>> model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16, device_map="auto")78>>> messages =[{"role":"user","content":"Is this comment toxic or non-toxic: RefuelLLM is the new way to label text data!"}]910>>> inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True).to("cuda")1112>>> outputs = model.generate(inputs, max_new_tokens=20)13>>>print(tokenizer.decode(outputs[0]))
Training Data
The model was both trained on over 4 Billion tokens, spanning 2750+ NLP tasks. Our training collection consists majorly of:
Human annotated datasets like Flan, Task Source, and the Aya collection
Synthetic datasets like OpenOrca, OpenHermes and WizardLM
Proprietary datasets developed or licensed by Refuel AI
Benchmarks
In this section, we report the results for Refuel models on our benchmark of labeling tasks. For details on the methodology see here.
Provider
Model
LLM Output Quality (by task type)
Overall
Classification
Reading Comprehension
Structure Extraction
Entity Matching
Refuel
RefuelLLM-2
83.82%
84.94%
76.03%
88.16%
92.00%
OpenAI
GPT-4-Turbo
80.88%
81.77%
72.08%
84.79%
97.20%
Refuel
RefuelLLM-2-small (Llama-3-Refueled)
79.67%
81.72%
70.04%
84.28%
92.00%
Anthropic
Claude-3-Opus
79.19%
82.49%
67.30%
88.25%
94.96%
Meta
Llama3-70B-Instruct
78.20%
79.38%
66.03%
85.96%
94.13%
Google
Gemini-1.5-Pro
74.59%
73.52%
60.67%
84.27%
98.48%
Mistral
Mixtral-8x7B-Instruct
62.87%
79.11%
45.56%
47.08%
86.52%
Anthropic
Claude-3-Sonnet
70.99%
79.91%
45.44%
78.10%
96.34%
Anthropic
Claude-3-Haiku
69.23%
77.27%
50.19%
84.97%
54.08%
OpenAI
GPT-3.5-Turbo
68.13%
74.39%
53.21%
69.40%
80.41%
Meta
Llama3-8B-Instruct
62.30%
68.52%
49.16%
65.09%
63.61%
Limitations
The Llama-3-Refueled does not have any moderation mechanisms. We're looking forward to engaging with the community
on ways to make the model finely respect guardrails, allowing for deployment in environments requiring moderated outputs.
