L-Mul Optimized: mistralai/Mistral-7B-Instruct-v0.2
This is a modified version of Mistral AI's
Mistral-7B-Instruct-v0.2 model. The modification consists of replacing the standard attention mechanism with one that uses a custom, approximate matrix multiplication algorithm termed "L-Mul".
This work was performed as part of a research project to evaluate the performance and accuracy trade-offs of algorithmic substitutions in transformer architectures.
This model is intended strictly for educational and scientific purposes.
Model Description
The core architecture of mistralai/Mistral-7B-Instruct-v0.2 is preserved. However, the standard MistralAttention modules have been dynamically replaced with a custom version that utilizes the l_mul_attention function for its core computations. This function is defined in the lmul.py file included in this repository.
- Base Model: mistralai/Mistral-7B-Instruct-v0.2
- Modification: Replacement of standard attention with L-Mul approximate attention.
- Primary Use-Case: Research and educational analysis of algorithmic impact on LLMs.
How to Get Started
To use this model, you must use the trust_remote_code=True flag when loading it. This is required to execute the custom lmul.py file that defines the new attention mechanism.
You can load the model directly from this repository using the transformers library:
1from transformers import AutoTokenizer, AutoModelForCausalLM
2import torch
3
4# Define the repository ID for the specific model
5repo_id = "Peacemann/mistralai_Mistral-7B-Instruct-v0.2-lmul-attention" # Replace with the correct repo ID if different
6
7# Load the tokenizer and model, trusting the remote code to load lmul.py
8tokenizer = AutoTokenizer.from_pretrained(repo_id)
9model = AutoModelForCausalLM.from_pretrained(
10 repo_id,
11 trust_remote_code=True,
12 torch_dtype=torch.bfloat16,
13 device_map="auto",
14)
15
16# Example usage
17prompt = "The L-Mul algorithm is an experimental method for..."
18inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
19outputs = model.generate(**inputs, max_new_tokens=50)
20
21print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Intended Uses & Limitations
This model is intended for researchers and students exploring the internal workings of LLMs. It is a tool for visualizing and analyzing the effects of fundamental algorithmic changes.
This model is NOT intended for any commercial or production application.
The modification is experimental. The impact on the model's performance, safety alignment, accuracy, and potential for generating biased or harmful content is unknown and untested.
Licensing Information
The use of this model is subject to the original Apache 2.0 License. By using this model, you agree to the terms outlined in the license.
