More specifically, the attention layers are quantized to 4-bit and the experts are quantized to 2-bit.
This model should perform a lot better compared to the all 2-bit model for a slight increase in model size (18.2GB vs. 18GB).
This idea was suggest by Artem Eliseev (@lavawolfiee) and Denis Mazur (@dvmazur) in this Github discussion.
Basic Usage
To run the model, install the HQQ library:
#This model is deprecated and requires older versions
pip install hqq==0.1.8
pip install transformers==4.46.0
and use it as follows:
Python
1model_id = 'mobiuslabsgmbh/Mixtral-8x7B-Instruct-v0.1-hf-attn-4bit-moe-2bit-HQQ'
2#Load the model
3from hqq.engine.hf import HQQModelForCausalLM, AutoTokenizer
4tokenizer = AutoTokenizer.from_pretrained(model_id)
5model = HQQModelForCausalLM.from_quantized(model_id)
6#Optional
7from hqq.core.quantize import *
8HQQLinear.set_backend(HQQBackend.PYTORCH_COMPILE)
9#Text Generation
10prompt = "<s> [INST] How do I build a car? [/INST] "
11inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
12outputs = model.generate(**(inputs.to('cuda')), max_new_tokens=1000)
13print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Output:
Design the Car:
Determine the type of car you want to build (e.g., sedan, SUV, sports car) and its specifications (e.g., size, weight, horsepower, fuel efficiency).
Create detailed sketches and 3D models of the car's exterior and interior.
Design the car's technical components, including the engine, transmission, brakes, and suspension system.
Acquire Necessary Materials and Parts:
Purchase or manufacture the necessary materials, such as steel, aluminum, and plastics.
Obtain or manufacture the required parts, such as the engine, transmission, brakes, suspension system, and electrical components.
Set Up a Production Facility:
Establish a manufacturing facility with the necessary equipment, such as assembly lines, paint booths, and welding machines.
Hire a skilled workforce to oversee production and ensure quality control.
Manufacture the Car:
Follow the design specifications to assemble the car's components.
Perform rigorous testing to ensure the car meets safety and performance standards.
Market and Sell the Car:
Develop a marketing strategy to promote the car to potential buyers.
Establish a distribution network to sell the car through dealerships or online platforms.
Provide After-Sales Support:
Offer maintenance and repair services to ensure customer satisfaction and loyalty.
Continuously improve the car's design and performance based on customer feedback and market trends.
Please note, building a car requires significant expertise, resources, and adherence to strict safety and regulatory standards. It is not a project that can be undertaken without extensive knowledge and experience in automotive engineering, manufacturing, and business management.
Quantization
You can reproduce the model using the following quant configs:
