Views
No views yet
pip install "optimum-intel[openvino]" openvino transformers1from optimum.intel.openvino import OVModelForCausalLM
2from transformers import AutoTokenizer
3
4model_id = "Irfanuruchi/qwen2.5-1.5b-buildeng-openvino-int8"
5
6tokenizer = AutoTokenizer.from_pretrained(model_id)
7model = OVModelForCausalLM.from_pretrained(model_id, device="CPU", compile=True)
8
9messages = [
10 {"role": "system", "content": "You are BuildEng. Give concise, inspection-first building engineering guidance."},
11 {"role": "user", "content": "A homeowner reports diagonal cracks near a window corner. What should be inspected first?"}
12]
13
14prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
15inputs = tokenizer(prompt, return_tensors="pt")
16
17output = model.generate(**inputs, max_new_tokens=160, do_sample=False)
18new_tokens = output[0][inputs["input_ids"].shape[-1]:]
19
20print(tokenizer.decode(new_tokens, skip_special_tokens=True))| Format | Repository |
|---|---|
| Hugging Face | Irfanuruchi/qwen2.5-1.5b-buildeng |
| GGUF | Irfanuruchi/qwen2.5-1.5b-buildeng-GGUF-Q4_K_M |
| MLX 4-bit | Irfanuruchi/qwen2.5-1.5b-buildeng-mlx-4bit |
| MLX 8-bit | Irfanuruchi/qwen2.5-1.5b-buildeng-mlx-8bit |
| OpenVINO INT4 | Irfanuruchi/qwen2.5-1.5b-buildeng-openvino-int4 |