About the GPTQ version

• Quantized to 4-bits 128g using GPTQ-for-LLaMA.
• Intended for use with Oobabooga Text Generation WebUI.

Loading model in Oobabooga WebUI

• Use same parameters as the original model, which can be found in the original repo linked below.
• Use AutoGPTQ loader.

Information about original model

Alpaca 7B Native Enhanced

📃 Model Facts

• Trained natively on 8x Nvidia A100 40GB GPUs; no LoRA used
• Trained on the largest & most accurate dataset yet
• Enhanced Programming Capabilities
• First Alpaca model to have conversational awareness

🚀 Quick Start Guide

You are an AI language model designed to assist the User by answering their questions, offering advice, and engaging in casual conversation in a friendly, helpful, and informative manner. You respond clearly, coherently, and you consider the conversation history.

User: Hey, how's it going?

Assistant: Hey there! I'm doing great, thank you. What can I help you with today? Let's have a fun chat!

Step 5. Change the settings to match this screenshot:

📚 Training

We used 8x Nvidia A100 40GB GPUs for training this model. Training time took ~3 hours and resulting loss was 0.4761 over 3 epochs. The command used for training is as follows

torchrun --nproc_per_node=8 --master_port=3045 ./stanford_alpaca/train.py --model_name_or_path ./llama-7b-hf --data_path ./alpaca-7b-nativeEnhanced/training_files/alpaca-megaset-fixed.json --fp16 True --output_dir ./output_7b --num_train_epochs 3 --per_device_train_batch_size 2 --per_device_eval_batch_size 2 --gradient_accumulation_steps 16 --evaluation_strategy "no" --save_strategy "steps" --save_steps 200 --learning_rate 2e-5 --weight_decay 0. --warmup_ratio 0.03 --lr_scheduler_type "cosine" --logging_steps 1 --fsdp "full_shard auto_wrap" --fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer' --tf32 True

Common errors while training

• CUDA Out of Memory error
  • This is because your GPUs do not have a minimum of 40GB of vram. The weakest GPU that we've been able to successfully train on has been Nvidia A100 40GB. Even with 8 of these, the vram usage was almost always right up at the limit. If you have 40GB GPUs and are still running into this error, try halving the per_device_train_batch_size and per_device_eval_batch_size and doubling the gradient_accumulation_steps. If you have more than 40GB of vram per GPU and wish to train faster, the opposite applies.
• LLaMATokenizer error
  • This happens because you forgot to fix tokenizer_config.json in the llama-7b-hf directory. The fix is to rename LLaMATokenizer to LlamaTokenizer in that file.
• RuntimeError: CUDA error: invalid device ordinal
  • This error occurs when your nproc_per_node is set to a number greater than how many GPUs you have installed in your system. You can check how many GPUs you have installed by running nvidia-smi.
• torchrun is not recognized
  • This error occurs when you have a python version older than 3.10. Follow the instructions in the training instructions file to install miniconda and get python 3.10 set up. Circumventing this error by running python -m torch.distributed.run will not work. Many of the dependencies require python 3.10 and will fatally error out at the start of training.
• KeyError
  • This happens when your JSON training data is broken in some way. Try running the dataset_validator.py in the training_files folder to find the broken key.

📝 Notes

• The main version of this model is in the hugging face transformers data type. The other one (.pth) format is provided purely for experimental use with llama.cpp and is not guaranteed to have conversational awareness.
• This model exhibits weird behavior when quantized to 4 bits. This might be due to the complexity of the model. We recommend the smallest quantization to be 8 bits, but this is untested.
• This model is slightly underfitted. We observed that training the model with a smaller gradient accumulation size benefitted the response quality.
• This model appears to have full conversational awareness. This means that provided you're running the model in the same configuration we detailed in the Quick Start Guide, you should be able to hold very detailed conversation with the AI without issues. There is a limit to it's memory, and it's 2048 tokens. Beyond that, it'll forget details and will need to be reminded.

🔧 Dataset

1. Increased accuracy since the original stanford_alpaca dataset had many errors.
2. Better knowledge in programming
3. More training data

👨‍💻 Credits