You can deploy this model and start earning money today!

DarkJian/
PrefixForQuality

Views

No views yet

Model Summary

The finetuned models for CodeLlama-7b-Instruct-hf, Phi-2, StarCoder2-7b after using rank_loss_prefix_tuning. This repo only provide prefix_params for each model, you can load the base model from HuggingFace's offical repo:

json

{
  "StarCoder2-7b": "bigcode/starcoder2-7b",
  "CodeLlama-7b-Instruct-hf": "codellama/CodeLlama-7b-Instruct-hf",
  "Phi-2": "microsoft/phi-2",
}

Load and use the model

python

# load prefix_codellama for example
import os
import torch
from transformers import AutoTokenizer, AutoConfig
from prefix_models import CodeLlamaPrefixCausalLM


lm_path = 'codellama/CodeLlama-7b-Instruct-hf'
prefix_path = './CodeLlama-7b-Instruct-hf'

prefix_config = config_from_pretrained(prefix_path)

lm_config = AutoConfig.from_pretrained(lm_path)
lm_config.n_prefix_token = prefix_config.n_prefix_token
lm_config.prefix_dropout = prefix_config.prefix_dropout
lm_config.n_control = prefix_config.n_control

kwargs = dict()
kwargs['torch_dtype'] = torch.bfloat16
kwargs['device_map'] = 'auto'
kwargs['trust_remote_code'] = True

model = CodeLlamaPrefixCausalLM.from_pretrained(lm_path, **kwargs, config=lm_config)

for param in model.prefix_params:
  torch.nn.init.zeros_(param)

tokenizer = AutoTokenizer.from_pretrained(lm_path)

prefix_file = os.path.join(prefix_path, 'pytorch_model.bin')
model.prefix_params.load_state_dict(torch.load(prefix_file))
model.resize_token_embeddings(len(tokenizer))

# generate
prompt = "def hello_world:\n"
input_ids = tokenizer(prompt, return_tensors='pt').input_ids
model.generate(
  input_ids,
  do_sample=True,
  num_return_sequences=1,
  temperature=0.4,
  top_p=0.95,
  max_new_tokens=300,
  pad_token_id=tokenizer.pad_token_id,
  use_cache=True,
  control_id=0,
)

DarkJian/PrefixForQuality

Model Summary

Load and use the model

DarkJian/
PrefixForQuality