This repo provides the training checkpoint of LLaMA on the alpaca_data_gpt4 dataset via LoRA [MLP] on 4xA100(80G).
He et al. 2022 gave an insight that FFN can better utilize modification at larger capacities.
We modify the running scripts to
1torchrun --nproc_per_node=4 finetune.py \
2 --base_model '/cache1/chtan/large_models/llama-hf/llama-13b' \
3 --data_path './alpaca_data_gpt4.json' \
4 --output_dir './gpt4-alpaca-lora_mlp-13b' \
5 --batch_size 128 \
6 --micro_batch_size 32 \
7 --num_epochs 10 \
8 --learning_rate 1e-4 \
9 --cutoff_len 512 \
10 --val_set_size 2000 \
11 --lora_r 16 \
12 --lora_alpha 16 \
13 --lora_dropout 0.05 \
14 --lora_target_modules '[gate_proj,down_proj,up_proj]' \
15 --train_on_inputs \
16 --group_by_length
Examples
Instruction: Tell me about alpacas.
gpt4-alpaca-lora_mlp-13b: Alpacas are members of the camelid family, which also includes llamas, camels, and vicunas. They originated in the Andes Mountains of South America and were domesticated by the Incas for their soft, warm fleece. Alpacas are smaller than llamas, standing at an average height of 36 inches and weighing between 100 and 200 pounds. They have a distinctive appearance, with long, shaggy necks and ears, and a curved, black nose. Alpacas are social animals and live in herds, with a dominant male, or a stud, at the head. They are gentle and easy to care for, and are often kept as pets or for their fleece, which is used to make clothing, blankets, and other textiles. Alpacas are valued for their soft, lightweight fleece, which is warmer and more durable than wool. They are also known for their gentle nature and are often used as therapy animals.
Alpaca-LoRA: Alpacas are members of the camelid family and are native to the Andes Mountains of South America. They are known for their soft, luxurious fleece, which is used to make clothing, blankets, and other items. Alpacas are herbivores and graze on grasses and other plants. They are social animals and live in herds of up to 20 individuals.
Stanford Alpaca: Alpacas are small, fluffy animals related to camels and llamas. They are native to Peru and Bolivia, and were first domesticated around 5,000 years ago. They are kept mainly for their fine, soft fleece, which is used to make knitwear and other garments. Alpacas are herd animals and live in small family groups, led by an older male. They are highly social animals and can form strong bonds with their owners.
text-davinci-003: Alpacas are a domesticated species of South American camelid, similar to llamas. They are native to the Andes Mountains in South America and are kept as livestock for their soft, luxurious wool. Alpacas are social animals and live in herds of up to 20 individuals. They are typically kept in small herds of two to five animals, and are relatively easy to care for. Alpacas are herbivores and feed on grass, hay, and other vegetation. They are also known for their gentle and friendly nature, making them popular as pets.
Instruction: Write a Python program that prints the first 10 Fibonacci numbers.
gpt4-alpaca-lora_mlp-13b:
Here is a Python program that prints the first 10 Fibonacci numbers:
1def fibonacci(n):
2 a, b = 0, 1
3 for i in range(n):
4 print(a, end=' ')
5 a, b = b, a + b
6 print(a)
7
8fibonacci(10)
This program defines a function called fibonacci that takes an integer n as an argument. Inside the function, we initialize two variables a and b with the values 0 and 1, respectively. Then, we use a for loop to iterate from 1 to n. For each iteration, we update the values of a and b using the Fibonacci sequence formula: a, b = b, a + b. After the loop is finished, we print the values of a and b using the print function. Finally, we call the fibonacci function with the value 10 as an argument, which will print the first 10 Fibonacci numbers.
Alpaca-LoRA:
1def fibonacci(n):
2 if n == 0:
3 return 0
4 elif n == 1:
5 return 1
6 else:
7 return fibonacci(n-1) + fibonacci(n-2)
8
9print(fibonacci(10))
Stanford Alpaca:
1numbers = [0, 1]
2
3for i in range(2, 11):
4 numbers.append(numbers[i-2] + numbers[i-1])
5
6print(numbers)
7# Output: [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
text-davinci-003:
1# Fibonacci numbers are the numbers in the following integer sequence, called the Fibonacci sequence, and characterized by the fact that every number after the first two is the sum of the two preceding ones.
2
3# Fibonacci sequence: 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, ...
4
5# Program:
6
7# Initialize the first two numbers of the Fibonacci sequence
8a = 0
9b = 1
10
11# Print the first 10 numbers of the Fibonacci sequence
12print(a)
13print(b)
14
15for i in range(2,10):
16 c = a + b
17 a = b
18 b = c
19 print(c)
[1] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, Graham Neubig: Towards a Unified View of Parameter-Efficient Transfer Learning. ICLR 2022
