L-Mul Optimized: Qwen/Qwen2-7B-Instruct
This is a modified version of Alibaba Cloud's
Qwen2-7B-Instruct model. The modification consists of replacing the standard attention mechanism with one that uses a custom, approximate matrix multiplication algorithm termed "L-Mul".
This work was performed as part of a research project to evaluate the performance and accuracy trade-offs of algorithmic substitutions in transformer architectures.
This model is intended strictly for educational and scientific purposes.
Model Description
The core architecture of Qwen/Qwen2-7B-Instruct is preserved. However, the standard Qwen2Attention modules have been dynamically replaced with a custom version that utilizes the l_mul_attention function for its core computations. This function is defined in the lmul.py file included in this repository.
- Base Model: Qwen/Qwen2-7B-Instruct
- Modification: Replacement of standard attention with L-Mul approximate attention.
- Primary Use-Case: Research and educational analysis of algorithmic impact on LLMs.
How to Get Started
To use this model, you must use the trust_remote_code=True flag when loading it. This is required to execute the custom lmul.py file that defines the new attention mechanism.
You can load the model directly from this repository using the transformers library:
1from transformers import AutoTokenizer, AutoModelForCausalLM
2import torch
3
4# Define the repository ID for the specific model
5repo_id = "Peacemann/Qwen_Qwen2-7B-Instruct-lmul-attention" # Replace with the correct repo ID if different
6
7# Load the tokenizer and model, trusting the remote code to load lmul.py
8tokenizer = AutoTokenizer.from_pretrained(repo_id)
9model = AutoModelForCausalLM.from_pretrained(
10 repo_id,
11 trust_remote_code=True,
12 torch_dtype=torch.bfloat16,
13 device_map="auto",
14)
15
16# Example usage
17prompt = "The L-Mul algorithm is an experimental method for..."
18inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
19outputs = model.generate(**inputs, max_new_tokens=50)
20
21print(tokenizer.decode(outputs[0], skip_special_tokens=True))
For high-throughput inference, you can use vLLM:
1from vllm import LLM
2
3repo_id = "Peacemann/Qwen_Qwen2-7B-Instruct-lmul-attention" # Replace with the correct repo ID
4llm = LLM(model=repo_id, trust_remote_code=True)
Intended Uses & Limitations
This model is intended for researchers and students exploring the internal workings of LLMs. It is a tool for visualizing and analyzing the effects of fundamental algorithmic changes.
This model is NOT intended for any commercial or production application.
The modification is experimental. The impact on the model's performance, safety alignment, accuracy, and potential for generating biased or harmful content is unknown and untested. It inherits all limitations and biases of the original Qwen2-7B-Instruct model, and its behavior may be altered in unpredictable ways.
Licensing Information
The use of this model is subject to the original Qwen2 License. By using this model, you agree to the terms outlined in the license. The license can be found on the base model's Hugging Face page.
