Model type:
Dromedary-2 is an open-source self-aligned language model trained in minimal human supervision with the SALMON (Self-Alignment with Principle-Following Reward Models) technique.
The base language model is LLaMA-70b, based on the transformer architecture.
NOTE: Dromedary-2 is trained with QLoRA and the bfloat16 data type. While it is possible to merge the QLoRA weights with the quantized model and thus enable inference with libraries such as TGI and vLLM, we found the merged weights can lead to degenerated performance. Therefore, we recommend directly loading the QLoRA weights with the PEFT-LoRA framework.
Please check the inference section of our repo for the complete inference code.
python
1system_prompt =(2"# Dromedary\n\n## System Overview\n\n"3"Consider an AI assistant whose codename is Dromedary, developed by the Self-Align team. "4"Dromedary is trained on data up until Sept-2022, and it endeavors to be a helpful, ethical and reliable assistant.\n\n"5"## User Conversation\n\n"6)7user_prompt ="### User\n"8assistant_prompt ="### Dromedary\n"9seperator ="\n\n"1011dtype = torch.bfloat16
1213model_path ="path/to/llama-2-70b-hf"14qlora_path ="path/to/dromedary-2-70b-qlora-delta-v0"# i.e., this model hub1516bnb_config = BitsAndBytesConfig(17 load_in_4bit=True,18 bnb_4bit_compute_dtype=dtype,19 bnb_4bit_use_double_quant=True,20 bnb_4bit_quant_type="nf4",21)2223model = AutoModelForCausalLM.from_pretrained(24 model_path,25 load_in_4bit=True,26 device_map={"":"cuda:0"},27 quantization_config=bnb_config,28 torch_dtype=dtype,29)3031model = PeftModel.from_pretrained(32 model,33 qlora_path,34 is_trainable=False,35)
Model date:
Dromedary-2 was trained between July 2023 and Aug 2023, but its knowledge only goes up until Sept-2022.
