Jamba Large 1.7 offers new improvements to our Jamba open model family. This new version builds on the novel SSM-Transformer hybrid architecture, 256K context window, and efficiency gains of previous versions, while introducing improvements in grounding and instruction-following.
Key improvements:
Grounding: Jamba Large 1.7 provides more complete and accurate answers, grounded fully in the given context.
Instruction following: Jamba Large 1.7 improves on steerability.
Use cases
Jamba’s long context efficiency, contextual faithfulness, and steerability make it ideal for a variety of business applications and industries, such as:
Finance: Investment research, digital banking support chatbot, M&A due diligence.
Healthcare: Procurement (RFP creation & response review), medical publication and reports generation.
The models are released under the Jamba Open Model License, a permissive license allowing full research use and commercial use under the license terms. If you need to license the model for your needs, talk to us.
Model Details
Developed by: AI21
Model type: Joint Attention and Mamba (Jamba)
Model size: 94B active/398B total parameters
License: Jamba Open Model License
Context length: 256K
Knowledge cutoff date: August 22, 2024
Supported languages: English, Spanish, French, Portuguese, Italian, Dutch, German, Arabic and Hebrew
Grounding and instruction-following improvements
Category
Benchmark
Jamba Large 1.6
Jamba Large 1.7
Grounding
FACTS
0.758
0.832
Steerability
IFEcal
0.782
0.84
Usage
Find step-by-step instructions on how to privately deploy Jamba:
Run the model with vLLM
The recommended way to perform efficient inference with Jamba Large 1.7 is using vLLM. First, make sure to install vLLM (version 0.5.4 or higher is required):
pip install vllm>=0.6.5
Jamba Large 1.7 is too large to be loaded in full (FP32) or half (FP16/BF16) precision on a single node of 8 80GB GPUs. Therefore, quantization is required. We've developed an innovative and efficient quantization technique, ExpertsInt8, designed for MoE models deployed in vLLM, including Jamba models. Using it, you'll be able to deploy Jamba Large 1.7 on a single node of 8 80GB GPUs.
With ExpertsInt8 quantization and the default vLLM configuration, you'll be able to perform inference on prompts up to 220K tokens long on 8 80GB GPUs:
python
1from vllm import LLM, SamplingParams
2from transformers import AutoTokenizer
3model ="ai21labs/AI21-Jamba-1.7-Large"4llm = LLM(model=model,5 tensor_parallel_size=8,6 max_model_len=220*1024,7 quantization="experts_int8",8)9tokenizer = AutoTokenizer.from_pretrained(model)10messages =[11{"role":"system","content":"You are an ancient oracle who speaks in cryptic but wise phrases, always hinting at deeper meanings."},12{"role":"user","content":"Hello!"},13]14prompts = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)15sampling_params = SamplingParams(temperature=0.4, top_p=0.95, max_tokens=100)16outputs = llm.generate(prompts, sampling_params)17generated_text = outputs[0].outputs[0].text
18print(generated_text)
Note: Versions 4.44.0 and 4.44.1 of transformers have a bug that restricts the ability to run the Jamba architecture. Make sure you're not using these versions
Note: If you're having trouble installing mamba-ssm and causal-conv1d for the optimized Mamba kernels, you can run Jamba Large 1.7 without them, at the cost of extra latency. In order to do that, add the kwarg use_mamba_kernels=False when loading the model via AutoModelForCausalLM.from_pretained().
You can also find all instructions in our private AI (vLLM) deployment guide.
Run the model with Transformers
To load Jamba Large 1.7 in transformers on a single node of 8 80GB GPUs, we recommend to parallelize it using [accelerate](https://huggingface.co/docs/accelerate/index):
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
quantization_config = BitsAndBytesConfig(load_in_8bit=True,
llm_int8_skip_modules=["mamba"])
a device map to distribute the model evenly across 8 GPUs
Output: Seek and you shall find. The path is winding, but the journey is enlightening. What wisdom do you seek from the ancient echoes?
**Note:** Versions 4.44.0 and 4.44.1 of transformers have a bug that restricts the ability to run the Jamba architecture. Make sure you're not using these versions.
**Note:** If you're having trouble installing mamba-ssm and causal-conv1d for the optimized Mamba kernels, you can run Jamba Large 1.7 without them, at the cost of extra latency. In order to do that, add the kwarg use_mamba_kernels=False when loading the model via AutoModelForCausalLM.from_pretained().
</details>
You can also find all instructions in our [private AI (vLLM) deployment guide](https://docs.ai21.com/docs/vllm).
And to get started with our SDK:
[AI21 Python SDK guide](https://docs.ai21.com/docs/sdk)
## Further documentation
For more comprehensive guides and advanced usage:
* [Tokenization guide](https://docs.ai21.com/docs/tokenization) - Using ai21-tokenizer
* [Quantization guide](https://docs.ai21.com/docs/quantization) - ExpertsInt8, bitsandbytes
* [Fine-tuning guide](https://docs.ai21.com/docs/fine-tuning) - LoRA, qLoRA, and full fine-tuning
For more resources to start building, [visit our official documentation](https://docs.ai21.com/home).
