You can deploy this model and start earning money today!

serpdotai/
llama-oasst-lora-30B

Views

No views yet

LoRA weights only and trained for research - nothing from the foundation model. Trained using Open-Assistant's dataset. Shout-out to Open-Assistant and LAION for giving us early research access to the dataset.

Sample usage

python

import torch
import os
import transformers
from peft import PeftModel
from transformers import LlamaTokenizer, LlamaForCausalLM

model_path = "decapoda-research/llama-30b-hf"
peft_path = 'serpdotai/llama-oasst-lora-30B'
tokenizer_path = 'decapoda-research/llama-30b-hf'

model = LlamaForCausalLM.from_pretrained(model_path, load_in_8bit=True, device_map="auto") # or something like {"": 0}
model = PeftModel.from_pretrained(model, peft_path, torch_dtype=torch.float16, device_map="auto") # or something like {"": 0}
tokenizer = LlamaTokenizer.from_pretrained(tokenizer_path)

batch = tokenizer("\n\nUser: Are you sentient?\n\nAssistant:", return_tensors="pt")

with torch.no_grad():
    out = model.generate(
        input_ids=batch["input_ids"].cuda(),
        attention_mask=batch["attention_mask"].cuda(),
        max_length=100,
        do_sample=True,
        top_k=50,
        top_p=1.0,
        temperature=1.0
    )
print(tokenizer.decode(out[0]))

The model will continue the conversation between the user and itself. If you want to use as a chatbot you can alter the generate method to include stop sequences for 'User:' and 'Assistant:' or strip off anything past the assistant's original response before returning.

Trained for 4 epochs with a sequence length of 2048 on 8 A6000s with an effective batch size of 120.

Training settings:

json

lr: 2.0e-04
lr_scheduler_type: linear
warmup_ratio: 0.06
weight_decay: 0.1
optimizer: adamw_torch
LoRA config:

target_modules: ['q_proj', 'k_proj', 'v_proj', 'o_proj']
r: 64
lora_alpha: 32
lora_dropout: 0.05
bias: "none"
task_type: "CAUSAL_LM"