Views
No views yet
torch.bfloat16 for quantization tuning.Granite 3.2 2b instruct has been quantized for inference on Intel CPU, Intel iGPU (Arc) via intel-extension-for-pytorch, Intel NPU (AI Boost on Core Ultra series) via OpenVINO.1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3repo = "fbaldassarri/ibm-granite_granite-3.2-2b-instruct-auto_round-int8-gs64-sym"
4model = AutoModelForCausalLM.from_pretrained(repo, device_map="auto")
5tokenizer = AutoTokenizer.from_pretrained(repo)
6
7messages = [{"role": "user", "content": "Hello!"}]
8inputs = tokenizer.apply_chat_template(
9 messages, return_tensors="pt", add_generation_prompt=True
10).to(model.device)
11outputs = model.generate(inputs, max_new_tokens=128)
12print(tokenizer.decode(outputs[0], skip_special_tokens=True))setup.sh. Any of the install modes below produces an environment that can reproduce this quantization; pick the one that matches your goals:git clone https://git.epicdynamic.com/auto-round-pipeline
cd auto-round-pipeline
# Pinned PyPI wheel (fastest; matches what this pipeline used by default):
bash setup.sh --pip-version 0.13.1
# Or build from intel/auto-round at the same tag (byte-identical reproducibility):
bash setup.sh --source-tag v0.13.1
# Intel Arc iGPU acceleration (e.g. Core Ultra 185H) — append to either of the above:
# ... --intel-xpu
# NVIDIA / AMD opt-in: --cuda / --rocmauto-round-pipeline-v0.13.1[-src][-xpu|-cuda|-rocm]) at the end.setup.sh created, then invoke the runner with the same job filters that produced this artifact:conda activate <env-name-printed-by-setup.sh>
python runner.py \
--model 'ibm-granite/granite-3.2-2b-instruct' \
--quant 'INT8-gs64' \
--format auto_round \
--no-upload # drop this to also push to HuggingFace Hub1import torch
2from transformers import AutoModelForCausalLM, AutoTokenizer
3from auto_round import AutoRound
4
5model_name = "ibm-granite/granite-3.2-2b-instruct"
6model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16)
7tokenizer = AutoTokenizer.from_pretrained(model_name)
8
9bits, group_size, sym = 8, 64, True
10autoround = AutoRound(
11 model, tokenizer,
12 bits=bits, group_size=group_size, sym=sym,
13 device_map="cpu",
14 nsamples=128, iters=200, seqlen=512, batch_size=4,
15)
16autoround.quantize_and_save("./AutoRound/ibm-granite_granite-3.2-2b-instruct-auto_round-int8-gs64-sym", format="auto_round")| Field | Value |
|---|---|
| Intel auto-round version | 0.13.1 |
| transformers version | 4.55.3 |
| torch version | 2.12.1+cpu |
| torch_dtype (load) | torch.bfloat16 |
| calibration device | cpu |
| calibration samples | 128 |
| tuning iterations | 200 |
| calibration seq len | 512 |
| calibration batch size | 4 |
| quantization duration | 19007.5s (316.8 min) |
| completed at (UTC) | 2026-07-01T04:02:15.728857+00:00 |