Views
No views yet
o_proj and down_proj parameters of these added layers initialized to zero, mirroring the approach used in LLaMA Pro.
It's important to note that this configuration has not undergone fine-tuning. Therefore, when fine-tuning, ensure that only every fourth layer is adjusted,
while all other layers remain frozen.1slices:
2 - sources:
3 - model: google/gemma-2b-it
4 layer_range: [0, 3]
5 - sources:
6 - model: google/gemma-2b-it
7 layer_range: [2, 3]
8 parameters:
9 scale:
10 - filter: o_proj
11 value: 0.0
12 - filter: down_proj
13 value: 0.0
14 - value: 1.0
15
16 - sources:
17 - model: google/gemma-2b-it
18 layer_range: [3, 6]
19 - sources:
20 - model: google/gemma-2b-it
21 layer_range: [5, 6]
22 parameters:
23 scale:
24 - filter: o_proj
25 value: 0.0
26 - filter: down_proj
27 value: 0.0
28 - value: 1.0
29
30 - sources:
31 - model: google/gemma-2b-it
32 layer_range: [6, 9]
33 - sources:
34 - model: google/gemma-2b-it
35 layer_range: [8, 9]
36 parameters:
37 scale:
38 - filter: o_proj
39 value: 0.0
40 - filter: down_proj
41 value: 0.0
42 - value: 1.0
43
44 - sources:
45 - model: google/gemma-2b-it
46 layer_range: [9, 12]
47 - sources:
48 - model: google/gemma-2b-it
49 layer_range: [11, 12]
50 parameters:
51 scale:
52 - filter: o_proj
53 value: 0.0
54 - filter: down_proj
55 value: 0.0
56 - value: 1.0
57
58 - sources:
59 - model: google/gemma-2b-it
60 layer_range: [12, 15]
61 - sources:
62 - model: google/gemma-2b-it
63 layer_range: [14, 15]
64 parameters:
65 scale:
66 - filter: o_proj
67 value: 0.0
68 - filter: down_proj
69 value: 0.0
70 - value: 1.0
71
72 - sources:
73 - model: google/gemma-2b-it
74 layer_range: [15, 18]
75 - sources:
76 - model: google/gemma-2b-it
77 layer_range: [17, 18]
78 parameters:
79 scale:
80 - filter: o_proj
81 value: 0.0
82 - filter: down_proj
83 value: 0.0
84 - value: 1.0
85
86merge_method: passthrough
87dtype: bfloat16
88
89
90# Function to freeze layers
91
92from transformers import AutoModelForCausalLM
93
94def update_layer_gradients(model, n):
95 """
96 Enables gradients only for every nth layer within the model's layers, starting from the layer after the 0th.
97
98 :param model: The model instance, assumed to be of type GemmaForCausalLM or similar.
99 :param n: Interval at which layers after the first will have their gradients enabled, indicating they are newly added.
100 """
101 layers = model.model.layers # Access the ModuleList containing the layers
102
103 for i, layer in enumerate(layers):
104 if i % n == (n - 1): # Enables gradients for every nth layer, starting from the layer after the 0th
105 print(i)
106 for param in layer.parameters():
107 param.requires_grad = True
108 else:
109 for param in layer.parameters():
110 param.requires_grad = False
111
112# Load the model
113model = AutoModelForCausalLM.from_pretrained("/Users/gayalshamane/Documents/mergekit/gemma-2b-it-expanded")
114
115
116# Update layer gradients, specify the correct value for n based on your model's architecture
117n = 4 # Example: update every 4rd layer, starting from the first layer after the 0th, adjust this value as needed
118update_layer_gradients(model, n)
119