Quantization made by Richard Erkhov.
The model is designed to respond to coding related instructions over long-conext input up to 128K length and can be used to build coding assistants.
1import torch
2from transformers import AutoModelForCausalLM, AutoTokenizer
3device = "cuda" # or "cpu"
4model_path = "ibm-granite/granite-8B-Code-instruct-128k"
5tokenizer = AutoTokenizer.from_pretrained(model_path)
6# drop device_map if running on CPU
7model = AutoModelForCausalLM.from_pretrained(model_path, device_map=device)
8model.eval()
9# change input text as desired
10chat = [
11 { "role": "user", "content": "Write a code to find the maximum value in a list of numbers." },
12]
13chat = tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True)
14# tokenize the text
15input_tokens = tokenizer(chat, return_tensors="pt")
16# transfer tokenized inputs to the device
17for i in input_tokens:
18 input_tokens[i] = input_tokens[i].to(device)
19# generate output tokens
20output = model.generate(**input_tokens, max_new_tokens=100)
21# decode output tokens into text
22output = tokenizer.batch_decode(output)
23# loop over the batch to print, in this example the batch size is 1
24for i in output:
25 print(i)
Granite Code Instruct models are trained on a mix of short and long context data as follows.
We train the Granite Code models using two of IBM's super computing clusters, namely Vela and Blue Vela, both outfitted with NVIDIA A100 and H100 GPUs respectively. These clusters provide a scalable and efficient infrastructure for training our models over thousands of GPUs.
Granite code instruct models are primarily finetuned using instruction-response pairs across a specific set of programming languages. Thus, their performance may be limited with out-of-domain programming languages. In this situation, it is beneficial providing few-shot examples to steer the model's output. Moreover, developers should perform safety testing and target-specific tuning before deploying these models on critical applications. The model also inherits ethical considerations and limitations from its base model. For more information, please refer to
Granite-8B-Code-Base-128K model card.
