This version of Smaug is based on the Qwen1.5-72B-Chat model and has undergone further fine-tuning. It is specialised in the areas of reasoning and coding.
It outperforms Qwen1.5-72B-Chat on MT-Bench, as shown below.
MT-Bench
We ran MT-Bench with the Qwen conversation template.
Model
First Turn
Second Turn
Average
Qwen1.5-72B-Chat
8.59
8.08
8.34
Smaug-2-72B
8.86
8.20
8.53
HumanEval
We ran HumanEval with pass@1 with the Qwen conversation template. Smaug-2 outperforms Qwen1.5-72B-Chat by approximately 10%:
Model
pass@1 (%)
Qwen1.5-72B-Chat
56.7
Smaug-2-72B
66.5
This version of Smaug uses new techniques and new data compared to Smaug-72B, and more information will be released later on. For now, see the previous Smaug paper: https://arxiv.org/abs/2402.13228.
Model Details
Model Sources [optional]
Repository: [More Information Needed]
Paper [optional]: [More Information Needed]
Demo [optional]: [More Information Needed]
Uses
Direct Use
[More Information Needed]
Downstream Use [optional]
[More Information Needed]
Out-of-Scope Use
[More Information Needed]
Bias, Risks, and Limitations
[More Information Needed]
Recommendations
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
