This is a 32K version of Smaug-Llama-3-70B-Instruct. It uses PoSE (
https://arxiv.org/abs/2309.10400) and LoRA (
https://arxiv.org/abs/2106.09685) adapter transfer. More details are coming soon.
The prompt format is unchanged from Llama 3 70B Instruct.
1import transformers
2import torch
3
4model_id = "abacusai/Smaug-Llama-3-70B-Instruct"
5
6pipeline = transformers.pipeline(
7 "text-generation",
8 model=model_id,
9 model_kwargs={"torch_dtype": torch.bfloat16},
10 device_map="auto",
11)
12
13messages = [
14 {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
15 {"role": "user", "content": "Who are you?"},
16]
17
18prompt = pipeline.tokenizer.apply_chat_template(
19 messages,
20 tokenize=False,
21 add_generation_prompt=True
22)
23
24terminators = [
25 pipeline.tokenizer.eos_token_id,
26 pipeline.tokenizer.convert_tokens_to_ids("<|eot_id|>")
27]
28
29outputs = pipeline(
30 prompt,
31 max_new_tokens=256,
32 eos_token_id=terminators,
33 do_sample=True,
34 temperature=0.6,
35 top_p=0.9,
36)
37print(outputs[0]["generated_text"][len(prompt):])
Score vs selected others (sourced from: (
https://lmsys.org/blog/2024-04-19-arena-hard/#full-leaderboard-with-gpt-4-turbo-as-judge)). GPT-4o and Gemini-1.5-pro-latest were missing from the original blob post, and we produced those numbers from a local run using the same methodology.
Note that we believe the number of tokens/verbosity of the model strongly influences the GPT-4 judge in this case, and at least partially explains the improvement in Arena-Hard score for the 32K model.
GSM8K The GSM8K numbers quoted here are computed using a recent release
of the
LM Evaluation Harness.
The commit used by the leaderboard has a significant issue that impacts models that
tend to use
: in their responses due to a bug in the stop word configuration for
GSM8K. The issue is covered in more detail in this
GSM8K evaluation discussion.
The score for both Llama-3 and this model are significantly different when evaluated
with the updated harness as the issue with stop words has been addressed.
Detailed results can be found
here
