from openai import OpenAI

client = OpenAI(base_url="https://<your-endpoint>.endpoints.huggingface.cloud/v1/", api_key="hf_...")

resp = client.chat.completions.create(
    model="tgi",                       # vLLM: use the served model name
    messages=[{"role": "user", "content": "Write a Python function that ..."}],
    max_tokens=1024,                   # hard ceiling — it will use all of it otherwise
    temperature=0.2,
    stop=["\n```\n", "\n```", "<|im_end|>", "<|endoftext|>"],
)

eos_token_id is [151645, 151643] (<|im_end|>, <|endoftext|>) so the server halts on either if the model emits one — but do not rely on that alone, hence the stop list above.

Local `transformers`

Stop at the end of the first code block:

python

from transformers import StoppingCriteria, StoppingCriteriaList
class StopAfterCodeBlock(StoppingCriteria):
    def __init__(self, tok, n): self.tok, self.n = tok, n
    def __call__(self, ids, s, **k):
        t = self.tok.decode(ids[0][self.n:], skip_special_tokens=True)
        i = t.find("```"); nl = t.find("\n", i) if i>=0 else -1
        return i>=0 and nl>=0 and "```" in t[nl+1:]
# model.generate(**enc, max_new_tokens=1024,
#   stopping_criteria=StoppingCriteriaList([StopAfterCodeBlock(tok, enc.input_ids.shape[1])]))

Serving notes

Prompt format: ChatML (<|im_start|>role\n...<|im_end|>). The chat template ships both inline in tokenizer_config.json (for TGI / vLLM / the HF inference toolkit) and as chat_template.jinja (for transformers 5.x).
Precision: bf16, 15.2 GB of weights. Needs a >16 GB GPU (T4 is out). KV cache is ~57 KB/token (28 layers × 4 KV heads × 128 dim × 2 × 2 bytes), i.e. ~1.9 GB for a full 32k sequence — so L4 / A10G (24 GB) serves 32k at low concurrency, and L40S (48 GB) gives room for real batching.
Context: 32768 tokens, RoPE theta 1e6.
The config carries both the transformers 4.x keys (torch_dtype, top-level rope_theta) and the 5.x keys (dtype, rope_parameters), so it loads correctly on either. Do not drop the 4.x keys — every current serving stack reads those, and without rope_theta they silently fall back to 10000.0 (wrong RoPE base → degraded output).

jmurray10/qwen25coder-7b-p2

qwen25coder-7b-p2

IMPORTANT — this model does not reliably stop on its own

Served behind an endpoint (TGI / vLLM / Inference Endpoints)

Local transformers

Serving notes

jmurray10/
qwen25coder-7b-p2

Local `transformers`