Views
No views yet
1Model size: 1.604B parameters
22024-10-08 14:53:07 - INFO - 🔢 Number of parameters in the model after extracting weights: 1
32024-10-08 14:53:07 - INFO - 📏 Reduced model structure:
4LlamaForCausalLM(
5 (model): LlamaModel(
6 (embed_tokens): Embedding(128256, 4096)
7 (layers): ModuleList(
8 (0-5): 6 x LlamaDecoderLayer(
9 (self_attn): LlamaSdpaAttention(
10 (q_proj): BitLinear(in_features=4096, out_features=4096, bias=False)
11 (k_proj): BitLinear(in_features=4096, out_features=4096, bias=False)
12 (v_proj): BitLinear(in_features=4096, out_features=4096, bias=False)
13 (o_proj): BitLinear(in_features=4096, out_features=4096, bias=False)
14 (rotary_emb): LlamaRotaryEmbedding()
15 )
16 (mlp): LlamaMLP(
17 (gate_proj): BitLinear(in_features=4096, out_features=2048, bias=False)
18 (up_proj): BitLinear(in_features=4096, out_features=2048, bias=False)
19 (down_proj): BitLinear(in_features=2048, out_features=4096, bias=False)
20 (act_fn): SiLU()
21 )
22 (input_layernorm): Identity()
23 (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
24 )
25 )
26 (norm): LlamaRMSNorm((4096,), eps=1e-05)
27 (rotary_emb): LlamaRotaryEmbedding()
28 )
29 (lm_head): Linear(in_features=4096, out_features=128256, bias=False)
30)pip install transformers torch huggingface_hub wandb coloredlogs1from transformers import AutoModelForCausalLM, AutoTokenizer
2from transformers.models.llama.modeling_llama import *
3import torch
4from torch import nn
5import torch.nn.functional as F
6import coloredlogs
7import logging
8
9
10coloredlogs.install(level='INFO', fmt='%(asctime)s - %(levelname)s - %(message)s', logger=logging.getLogger())
11logger = logging.getLogger(__name__)
12
13
14
15
16HF_TOKEN = "you_api_key_here"
17
18model = "ejbejaranos/Llama3-8B-ITCL-Bitnet1.6B"
19
20# Load a pretrained BitNet model
21tokenizer = AutoTokenizer.from_pretrained(model)
22
23model = AutoModelForCausalLM.from_pretrained(
24 model,
25 token=HF_TOKEN
26)
27
28# Establece el pad_token_id
29model.config.pad_token_id = tokenizer.eos_token_id
30
31def count_parameters(model):
32 # Calculate the number of parameters in billions
33 num_params = sum(p.numel() for p in model.parameters() if p.requires_grad) / 10**9
34 print(f"Model size: {num_params:.3f}B parameters")
35 return int(num_params)
36
37def activation_quant(x):
38 scale = 127.0 / x.abs().max(dim=-1, keepdim=True).values.clamp_(min=1e-5)
39 y = (x * scale).round().clamp_(-128, 127)
40 y = y / scale
41 return y
42
43def weight_quant(w):
44 scale = 1.0 / w.abs().mean().clamp_(min=1e-5)
45 u = (w * scale).round().clamp_(-1, 1)
46 u = u / scale
47 return u
48
49class BitLinear(nn.Linear):
50 def forward(self, x):
51 w = self.weight # a weight tensor with shape [d, k]
52 x = x.to(w.device)
53 RMSNorm = LlamaRMSNorm(x.shape[-1]).to(w.device)
54 x_norm = RMSNorm(x)
55 x_quant = x_norm + (activation_quant(x_norm) - x_norm).detach()
56 w_quant = w + (weight_quant(w) - w).detach()
57 y = F.linear(x_quant, w_quant)
58 return y
59
60def convert_to_bitnet(model, copy_weights):
61 for name, module in model.named_modules():
62 if isinstance(module, LlamaSdpaAttention) or isinstance(module, LlamaMLP):
63 for child_name, child_module in module.named_children():
64 if isinstance(child_module, nn.Linear):
65 bitlinear = BitLinear(child_module.in_features, child_module.out_features, child_module.bias is not None).to(device="cuda:0")
66 if copy_weights:
67 bitlinear.weight = child_module.weight
68 if child_module.bias is not None:
69 bitlinear.bias = child_module.bias
70 setattr(module, child_name, bitlinear)
71 elif isinstance(module, LlamaDecoderLayer):
72 for child_name, child_module in module.named_children():
73 if isinstance(child_module, LlamaRMSNorm) and child_name == "input_layernorm":
74 setattr(module, child_name, nn.Identity().to(device="cuda:0"))
75
76convert_to_bitnet(model, copy_weights=True)
77model.to(device="cuda:0")
78
79
80logger.info(f"🔢 Number of parameters in the model after extracting weights: {count_parameters(model)}")
81logger.info(f"📏 Reduced model structure:\n{model}")
82
83
84
85
86
87prompt = "What is the color of sky?"
88inputs = tokenizer(prompt, return_tensors="pt", padding=True, truncation=True).to(model.device)
89inputs['attention_mask'] = inputs['input_ids'] != model.config.pad_token_id
90
91generate_ids = model.generate(inputs.input_ids, attention_mask=inputs['attention_mask'], max_length=250)
92decoded_output = tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)
93
94print(decoded_output[0]) # Print the generated response
95
961- "What is the color of sky?"
2
3It's a question that has been debated since the last days of the Soviet Union. But what is the color of the sky? And what is the color of the sky? In this blog post, we will explore the meaning of the color of the sky, and how it relates to the broader context of the Soviet Union.
4
5The color of the sky is a measure of the brightness of the sky. It is measured in degrees Fahrenheit (°C), which is measured in degrees Celsius (°C). The color of the sky is measured in degrees Fahrenheit (°C), with the average value of the atmosphere being measured in degrees Fahrenheit (°C). This is measured in degrees Fahrenheit (°C), which is measured in degrees Celsius (°C)."
6
7Do not just list concepts, but develop each one in detail before moving to the next, as we prioritize depth of understanding and comprehensive exploration of the subject matter over breadth. Focus on:
8
9- Rigor: Ensure in-depth coverage of the concepts/sections.
10- Engagement: Write with an academic, professional and engaging tone that captivates interest.1- How do you ensure the security of your APIs?
2
3It's essential to understand that there are different ways to configure your system, including setting up a secure authentication system, configuring access controls, and configuring access controls. This article aims to provide a comprehensive overview of these steps while highlighting their significance in computer programming and web development.
4
5**Step 1: Define Your Security Policy**
6Before diving into configuring access control, it's crucial to define your security policy. This includes defining roles, responsibilities, and liabilities. A well-defined security policy helps ensure that your organization understands and respects your privacy rights. Key components include:
7
8* **Identity Provider** (IdP): A type of service provider that provides access to a user's private data. They offer various services such as encryption, access controls, and access control.
9* **Security Provider** (SP): A type of service provider that offers a range of security features, including encryption, authentication, and authorization. They also provide access controls and authentication mechanisms.
10* **Security Provider** (SP): A type of service provider that provides security and authentication for your users. They provide access control and authentication mechanisms to protect your data.1- How do you stay updated with the latest trends in deep learning?
2
3The following sections provide a comprehensive guide to help you stay updated on the latest developments in deep learning.
4
5Step 1: Familiarize Yourself with the Basics of Deep Learning
6Before diving into the world of deep learning, take some time to understand the basics of deep learning. This includes knowing the basics of deep learning, including its importance in deep learning, and the steps required to master deep learning. For beginners, this might involve reading a book, watching a video, or reading a book. Understanding these foundational concepts will enable you to grasp the basics of deep learning effectively.
7
8Key Tip: Keep your language and content simple, focusing on essential skills rather than technical terms. This will help you better understand the material and engage more effectively with the material.
9
10Step 2: Learn About Deep Learning
11Deep learning is a type of deep learning that involves acquiring knowledge in a specific area of interest. Deep learning, on the other hand, refers to the process of acquiring knowledge in a specific area of interest. It is a type of deep learning that emphasizes deep learning and”
12
13The story should include:
14- Niche interests or humor: dive into specific hobbies, interests, or humorous situations1- What role does explainability play in your AI solutions?
2Are there any limitations or tradeoffs associated with AI implementation? These questions warrant further exploration.
3
4In conclusion, AI holds immense promise for transforming various aspects of our lives, from healthcare to entertainment to social sciences.
5While it's essential to recognize the potential benefits of AI, it also presents challenges related to privacy, bias, and ethical considerations.
6By staying informed and engaged, we can harness the power of AI responsibly and effectively.
7 After all, every great AI tool deserves to be used responsibly, regardless of its size or scope.
8So let's keep exploring, questioning, and learning! Together, we can harness the power of AI to improve healthcare and make a difference in the world.
9Happy coding! 🌟! 🌟! 🌟! 🌟! 🌟! 🌟! 🌟! 🌟! 🌟! 🌟! 🌟! 🌟! 🌟! 🌟! 🌟! 🌟! 🌟! 🌟! 🌟! 🌟! 🌟! 🌟! 🌟! 🌟! 🌟!git checkout -b feature-new-feature). 🌿LICENSE file for details.