モデルの概要

Model Details

Model Description

• Developed by: Kota Miyano
• Model type: Causal Language Model
• Language(s) (NLP): 日本語
• License: CC-BY-NC-SA (商用NG)
• Finetuned from model: llm-jp-3-13b

Uses

Direct Use

Out-of-Scope Use

Bias, Risks, and Limitations

Recommendations

How to Get Started with the Model

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

model_id = "llm-jp-3-13b-finetune"
model = AutoModelForCausalLM.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)

prompt = "### 指示\n質問に答えてください。\n### 回答\n"
input_ids = tokenizer(prompt, return_tensors="pt").input_ids
output = model.generate(input_ids, max_new_tokens=100)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Training Details

Training Data

Training Hyperparameters

• Training regime: Batch Size：1
  Gradient Accumulation Steps：2
  Learning Rate：5e-5
  Epochs：1
  Sequence Length：512
  Optimizer：Paged AdamW 32-bit
  

Evaluation