This project transforms a Llama3 model from 8B parameters to a BitNet architecture with 2B parameters, applying BitLinear layers. Additionally, the model is trained with a predefined dataset and uploaded to Hugging Face for future use.

Model Description

This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.

Developed by: ejbejaranos@gmail.com
Funded by [optional]: ITCL
Shared by [optional]: [More Information Needed]
Model type: LLama3 8B Tramsformed to Bitnet
Language(s) (NLP): Bitnet
License: [More Information Needed]
Finetuned from model [optional]: [More Information Needed]

Model Sources [optional]

Repository: ejbejaranos/Bitnet-Llama3-from8BM-now2B

📄 Description

This repository includes scripts to:

🎯 Transform a Llama3 model to a BitNet architecture.
💻 Train the model using Hugging Face and Weights & Biases.
🚀 Upload the transformed and trained model to Hugging Face for inference and future use.

⚙️ Requirements

Python 3.8+
Pytorch 1.10+
Transformers 4.0+
Hugging Face Hub API
Weights & Biases

🧰 Installation

Make sure you have all required dependencies installed:

pip install torch transformers datasets wandb huggingface_hub

💥 How to Use

Using the trained model for inference

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from utils.bitnet_transformation import replace_linears_in_hf

# Load the BitNet model
model = "ejbejaranos/Bitnet-Llama3-from8BM-now2B"
model = AutoModelForCausalLM.from_pretrained(
    model,
    use_auth_token="YOUR_HF_TOKEN"
)

# Replace BitNet layers for inference
replace_linears_in_hf(model)
tokenizer = AutoTokenizer.from_pretrained("ejbejaranos/Bitnet-Llama3-from8BM-now2B")

# Set up for inference
model.to(device="cuda:0")
prompt = "What is Machine Learning?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
generate_ids = model.generate(inputs.input_ids, max_length=50)
output = tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]

print(output)

🧑‍🔬 Metrics

During training, the following metrics will be logged to Weights & Biases:

final_loss: 1.4.
final_perplexity: 4.2.

🎯 Future Goals

Implement additional quantization layers for inference.
Test the model on different datasets and contexts.

📢 Contact

If you have questions, suggestions, or improvements, feel free to open an Issue or contact us through Hugging Face.

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

Hardware Type: [More Information Needed]
Hours used: [More Information Needed]
Cloud Provider: [More Information Needed]
Compute Region: [More Information Needed]
Carbon Emitted: [More Information Needed]

💡 Acknowledgments

Thanks to Hugging Face and Weights & Biases for providing support and tools.

RichardErkhov/ejbejaranos_-_Bitnet-Llama3-from8BM-now2B-8bits

Original model description:

🚀 BitNet-Llama3 (from 8B to 2B) Transformation & Training