You can deploy this model and start earning money today!

jhlim8/
ListenerLM

peft

safetensors

meta-llama/Llama-2-7b-chat-hf

adapter

apache-2.0

Views

No views yet

Model Card for Model ID

LLM Adapters Fine-tuned for an individual's Preferred Listening Style.

Model Details

QLoRA Adapters Fine-Tuned on meta-llama/Llama-2-7b-chat-hf on data made from Conversational Roleplay

Model Description

Model type: LLM
Language(s) (NLP): English
License: Apache 2.0
Finetuned from model [optional]: meta-llama/Llama-2-7b-chat-hf

Model Sources [optional]

Repository: https://github.com/jhlimm8/ListenerLM

Uses

As an example of how a LLM fine-tuned on a specific listening style differs from current LLMs.

Out-of-Scope Use

Anything beyond having the LLM act as a Listener for you.

How to Get Started with the Model

First install some packages

!pip install  --upgrade \
  "transformers==4.36.2" \
  "datasets==2.16.1" \
  "accelerate==0.26.1" \
  "evaluate==0.4.1" \
  "bitsandbytes==0.42.0" \
  # "trl==0.7.10" # \
  # "peft==0.7.1" \

# install peft & trl from github
!pip install git+https://github.com/huggingface/trl@a3c5b7178ac4f65569975efadc97db2f3749c65e --upgrade
!pip install git+https://github.com/huggingface/peft@4a1559582281fc3c9283892caea8ccef1d6f5a4f --upgrade

import torch; assert torch.cuda.get_device_capability()[0] >= 8, 'Hardware not supported for Flash Attention'
# install flash-attn
!pip install ninja packaging
!MAX_JOBS=4 pip install flash-attn --no-build-isolation

then run

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
model_path = "meta-llama/Llama-2-7b-chat-hf"
peft_path = "jhlim8/ListenerLM"

# needed for access to llama 2, can just directly apply here https://huggingface.co/meta-llama/Llama-2-7b
# and then get the token from https://huggingface.co/settings/tokens
huggingface_token = '' 

bnb_config = BitsAndBytesConfig(
    load_in_4bit = True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant = True
)
model = AutoModelForCausalLM.from_pretrained(
    model_path, 
    quantization_config=bnb_config,
    attn_implementation="flash_attention_2",
    device_map={'':'cuda:0'}, # have to specifically set each layer to device 0 when training with single gpu (sus i know)
    torch_dtype=torch.bfloat16, 
    token=huggingface_token
)
model = PeftModel.from_pretrained(model, peft_path, is_trainable=False, token=huggingface_token)
model.to("cuda:0") 
model.eval()
tok = AutoTokenizer.from_pretrained(peft_path, token=huggingface_token)

Training Data

Conversations obtained through Conversational Roleplay

Training Procedure

QLoRA