You can deploy this model and start earning money today!

jdpressman/
Mistral-Morpheus-7B

Name: Mistral-Morpheus-7B
Availability: InStock

peft

safetensors

Views

No views yet

First iteration of the default generator LoRa for MiniHF. This model still functions as a base model while writing more coherent text.

Training procedure

This model was trained starting from the MiniHF Mistral SFT evaluator. It was created using the MiniHF Reinforcement Learning From AI Feedback pipeline:

accelerate launch rlaif_generator.py --resume minihf_evaluator_mistral_7b_v0.1 --output-path mistral_h_eval --kl-weight 1.0 --constitution hermes/hermes_constitution.txt --prompts hermes/hermes_prompts.txt --length 256 --batch-size 4 --grad-accum-steps 8

The tuning script was modified to use the AdamW optimizer with weight decay:

opt = optim.AdamW(model.parameters(), lr=1e-5, weight_decay=1e-2, betas=(0.9, 0.98))

This weight decay is based on the observation that RL tuning mode collapse can be undone by interpolating the weights of the base model with that of the RL tuned model. Here the specific recipe was to start from the MiniHF SFT evaluator, then apply weight decay and the KL penalty towards the base model weights to inject entropy back into the policy.

Prompt Bank and Constitution

The prompt bank using during tuning is in the hermes_prompts.txt file found in this repo, the constitution in hermes_constitution.txt

Configuration

The following bitsandbytes quantization config was used during training:

quant_method: bitsandbytes
load_in_8bit: False
load_in_4bit: True
llm_int8_threshold: 6.0
llm_int8_skip_modules: None
llm_int8_enable_fp32_cpu_offload: False
llm_int8_has_fp16_weight: False
bnb_4bit_quant_type: nf4
bnb_4bit_use_double_quant: True
bnb_4bit_compute_dtype: float16

Framework versions

PEFT 0.5.0
PEFT 0.5.0
PEFT 0.5.0

jdpressman/Mistral-Morpheus-7B

Training procedure

Prompt Bank and Constitution

Configuration

Framework versions

jdpressman/
Mistral-Morpheus-7B