On GPUs with bfloat16 support, the generate.py script will automatically convert the weights and consume about ~14 GB.
For GPUs with less memory, or ones that don't support bfloat16, enable quantization (--quantize llm.int8):
python generate.py --quantize llm.int8 --prompt "Hello, my name is"
See python generate.py --help for more options.
You can also use GPTQ-style int4 quantization, but this needs conversions of the weights first:
With the generated quantized checkpoint generation works as usual with --quantize gptq.int4, bringing GPU usage to about ~5GB. As only the weights of the Linear layers are quantized, it is useful to use --dtype bfloat16 even with the quantization enabled.
We provide a simple training scripts in finetune_lora.py and finetune_adapter.py that instruction-tunes a pretrained model on the Alpaca dataset using the techniques of LoRA and Adapter.
Download the data and generate a instruction tuning dataset:
python scripts/prepare_alpaca.py
Run the finetuning script
python finetune_lora.py
or
python finetune_adapter.py
It is expected that you have downloaded the pretrained weights as described above.
The finetuning requires at least one GPU with ~24 GB memory (GTX 3090). Follow the instructions in the script to efficiently fit your GPU memory.
Note: For some GPU models you might need to set torch.backends.cuda.enable_flash_sdp(False) (see comments at the top of the script).
More details about each finetuning method and how you can apply it to your own data can be found in our technical how-to guides.
Finetuning How-To Guides
These technical tutorials illustrate how to run the finetuning code.
We provide a simple training script based on Fabric if you want to venture into pre-training on RedPajama, a reproduction of the original LLaMA dataset.
Conversion scripts for our optimized streaming PackedDataset are included.
Follow this guide to start pre-training on the RedPajama dataset:
Look at train.py for a starting point towards pre-training / fine-tuning using Lightning Fabric.
We welcome all individual contributors, regardless of their level of experience or hardware. Your contributions are valuable, and we are excited to see what you can accomplish in this collaborative and supportive environment.
