Stockmark-100b-instruct-v0.1 is an instruction tuned version of
stockmark-100b, a 100 billion parameter LLM developed by
Stockmark Inc.1import torch
2from transformers import AutoTokenizer
3from peft import AutoPeftModelForCausalLM
4
5prompt_template = """### 指示:
6{instruction}
7
8### 応答:
9"""
10
11tokenizer = AutoTokenizer.from_pretrained("stockmark/stockmark-100b-instruct-v0.1")
12model = AutoPeftModelForCausalLM.from_pretrained("stockmark/stockmark-100b-instruct-v0.1", device_map="auto", torch_dtype=torch.bfloat16)
13
14instruction = "生成AIとは？"
15prompt = prompt_template.format(instruction=instruction)
16input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(model.device)
17with torch.inference_mode():
18 tokens = model.generate(
19 input_ids,
20 max_new_tokens = 256,
21 do_sample = True,
22 temperature = 0.7,
23 top_p = 0.95,
24 repetition_penalty = 1.08
25 )
26
27output = tokenizer.decode(tokens[0], skip_special_tokens=True)
28print(output)
We excluded categories that require calculation and coding, and use remaining 60 questions for evaluation.
For local LLMs, we measured the inference time using AWS Inferentia2.
