Views
No views yet
o_proj and down_proj parameters of these added layers initialized to zero, mirroring the approach used in LLaMA Pro.1slices:
2 - sources:
3 - model: mistralai/Mistral-7B-Instruct-v0.2
4 layer_range: [0, 4]
5 - sources:
6 - model: mistralai/Mistral-7B-Instruct-v0.2
7 layer_range: [3, 4]
8 parameters:
9 scale:
10 - filter: o_proj
11 value: 0.0
12 - filter: down_proj
13 value: 0.0
14 - value: 1.0
15
16 - sources:
17 - model: mistralai/Mistral-7B-Instruct-v0.2
18 layer_range: [4, 8]
19 - sources:
20 - model: mistralai/Mistral-7B-Instruct-v0.2
21 layer_range: [7, 8]
22 parameters:
23 scale:
24 - filter: o_proj
25 value: 0.0
26 - filter: down_proj
27 value: 0.0
28 - value: 1.0
29
30 - sources:
31 - model: mistralai/Mistral-7B-Instruct-v0.2
32 layer_range: [8, 12]
33 - sources:
34 - model: mistralai/Mistral-7B-Instruct-v0.2
35 layer_range: [11, 12]
36 parameters:
37 scale:
38 - filter: o_proj
39 value: 0.0
40 - filter: down_proj
41 value: 0.0
42 - value: 1.0
43
44 - sources:
45 - model: mistralai/Mistral-7B-Instruct-v0.2
46 layer_range: [12, 16]
47 - sources:
48 - model: mistralai/Mistral-7B-Instruct-v0.2
49 layer_range: [15, 16]
50 parameters:
51 scale:
52 - filter: o_proj
53 value: 0.0
54 - filter: down_proj
55 value: 0.0
56 - value: 1.0
57
58 - sources:
59 - model: mistralai/Mistral-7B-Instruct-v0.2
60 layer_range: [16, 20]
61 - sources:
62 - model: mistralai/Mistral-7B-Instruct-v0.2
63 layer_range: [19, 20]
64 parameters:
65 scale:
66 - filter: o_proj
67 value: 0.0
68 - filter: down_proj
69 value: 0.0
70 - value: 1.0
71
72 - sources:
73 - model: mistralai/Mistral-7B-Instruct-v0.2
74 layer_range: [20, 24]
75 - sources:
76 - model: mistralai/Mistral-7B-Instruct-v0.2
77 layer_range: [23, 24]
78 parameters:
79 scale:
80 - filter: o_proj
81 value: 0.0
82 - filter: down_proj
83 value: 0.0
84 - value: 1.0
85
86 - sources:
87 - model: mistralai/Mistral-7B-Instruct-v0.2
88 layer_range: [24, 28]
89 - sources:
90 - model: mistralai/Mistral-7B-Instruct-v0.2
91 layer_range: [27, 28]
92 parameters:
93 scale:
94 - filter: o_proj
95 value: 0.0
96 - filter: down_proj
97 value: 0.0
98 - value: 1.0
99
100 - sources:
101 - model: mistralai/Mistral-7B-Instruct-v0.2
102 layer_range: [28, 32]
103 - sources:
104 - model: mistralai/Mistral-7B-Instruct-v0.2
105 layer_range: [31, 32]
106 parameters:
107 scale:
108 - filter: o_proj
109 value: 0.0
110 - filter: down_proj
111 value: 0.0
112 - value: 1.0
113
114merge_method: passthrough
115dtype: bfloat16from transformers import AutoModelForCausalLM
def enable_grad_only_every_nth(model, n):
"""
This function configures the specified model to enable gradient calculations exclusively for every nth layer, starting
from the first layer (0-indexed), to accommodate newly added blocks for training. Concurrently, it freezes the gradients
for all other components of the model, including the embedding layers and the model's head. This setup is particularly
useful for fine-tuning processes where only a subset of layers are targeted for updates, ensuring efficient training and
adaptation of newly integrated layers while maintaining the pre-trained behavior of other model components.
"""
# Freeze embeddings.
for param in model.model.embed_tokens.parameters():
param.requires_grad = False
# Freeze lm_head.
for param in model.lm_head.parameters():
param.requires_grad = False
# Enable gradients for every nth layer
layers = model.model.layers # Access the ModuleList containing the layers
for index, layer in enumerate(layers):
if (index + 1) % n == 0: # Enables gradients for every nth layer, starting from the layer after the 0th
for param in layer.parameters():
param.requires_grad = True
else:
for param in layer.parameters():
param.requires_grad = False
model = transformers.AutoModelForCausalLM.from_pretrained(
"arcee-ai/Mistral-7B-Instruct-v0.2-expanded"
)
# Update layer gradients, specify the correct value for n based on your model's architecture
n =5
enable_grad_only_every_nth(model, n)