Model Card for Model ID

Model Details

Model Description

• Developed by: [More Information Needed]
• Funded by [optional]: [More Information Needed]
• Shared by [optional]: [More Information Needed]
• Model type: [More Information Needed]
• Language(s) (NLP): [More Information Needed]
• License: [More Information Needed]
• Finetuned from model [optional]: [More Information Needed]

Model Sources [optional]

• Repository: [More Information Needed]
• Paper [optional]: [More Information Needed]
• Demo [optional]: [More Information Needed]

Uses

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch
from peft import PeftModel, PeftConfig

model_name = "mssfj/gemma-2-9b-bnb-4bit-chat-template"
lora_weight = "mssfj/gemma-2-9b-4bit-magpie"

quantization_config = BitsAndBytesConfig(
    load_in_4bit=False,
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=False
)

base_model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=quantization_config,
    device_map="auto"
    )

model = PeftModel.from_pretrained(base_model, lora_weight)

tokenizer = AutoTokenizer.from_pretrained(model_name)

input="""日本で一番高い山は？
"""

messages = [
    {"role": "system", "content": """あなたは誠実で優秀な日本人のアシスタントです。あなたはユーザと日本語で会話しています。アシスタントは以下の原則を忠実に守り丁寧に回答します。
    - 日本語で簡潔に回答する
    - 回答は必ず完結した文で終える
    - 質問の文脈に沿った自然な応答をする
    """},
    {"role": "user", "content": input},
]

input_ids = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

outputs = model.generate(
    input_ids,
    max_new_tokens=512,
    temperature=0.2,
    do_sample=True,
    eos_token_id=tokenizer.eos_token_id,
    pad_token_id=tokenizer.pad_token_id,
    early_stopping=True,
)

response = tokenizer.decode(outputs[0][input_ids.shape[1]:], skip_special_tokens=True)

Direct Use

Downstream Use [optional]

Out-of-Scope Use

Bias, Risks, and Limitations

Recommendations

How to Get Started with the Model

Training Details

Training Data

Training Procedure

Preprocessing [optional]

Training Hyperparameters

• Training regime: [More Information Needed]

Speeds, Sizes, Times [optional]

Evaluation

Testing Data, Factors & Metrics

Testing Data

Factors

Metrics

Results

Summary

Model Examination [optional]

Environmental Impact

• Hardware Type: [More Information Needed]
• Hours used: [More Information Needed]
• Cloud Provider: [More Information Needed]
• Compute Region: [More Information Needed]
• Carbon Emitted: [More Information Needed]

Technical Specifications [optional]

Model Architecture and Objective

Compute Infrastructure

Hardware

Software

Citation [optional]

Glossary [optional]

More Information [optional]

Model Card Authors [optional]

Model Card Contact

Framework versions

• PEFT 0.13.2