Views
No views yet
yunmorning/broken-model.
The original repository could not serve a working /chat/completions endpoint.
This README documents exactly which files were modified, what the change was, and why it was necessary.| File | Change | Reason |
|---|---|---|
tokenizer_config.json | Added the chat_template field (canonical Qwen3 Jinja template). | /chat/completions cannot format role-based messages without a chat template; the inference server (vLLM, SGLang, Friendli Engine) errors out or falls back to a wrong default. This is the root cause. |
README.md | Changed base_model: meta-llama/Meta-Llama-3.1-8B to Qwen/Qwen3-8B. | The weights, architecture, tokenizer, and vocabulary all belong to Qwen3-8B, not Llama-3.1. The wrong metadata misleads consumers and can break downstream tooling that inspects base_model. |
config.json, generation_config.json, model.safetensors.index.json, the weight shards, tokenizer.json, vocab.json, and merges.txt are identical to the original because they were already correct.base_model: meta-llama/Meta-Llama-3.1-8B, but the artifacts on disk tell a different story:| Signal | Value in repo | What it means |
|---|---|---|
config.json -> architectures | ["Qwen3ForCausalLM"] | Loaded by the Qwen3 model class. Llama would be LlamaForCausalLM. |
config.json -> model_type | "qwen3" | Routes to the Qwen3 model registry in transformers. |
config.json -> vocab_size | 151936 | Qwen tokenizer vocab. Llama-3.1 vocab is 128256. |
tokenizer_config.json -> tokenizer_class | "Qwen2Tokenizer" | BPE tokenizer with vocab.json and merges.txt (Qwen family). |
added_tokens_decoder | <|im_start|>, <|im_end|>, <tool_call>, <think>, etc. | ChatML-style markers used by Qwen, not by Llama. |
model.safetensors.index.json | Contains model.layers.N.self_attn.k_norm.weight and q_norm.weight | Qwen3-specific per-head Q/K RMSNorm. Llama-3.x has no such layers. |
total_size | 16.38 GB in BF16, roughly 8.19 B params | Matches Qwen3-8B (around 8.2 B). Llama-3.1-8B is around 8.03 B and has a smaller intermediate size of 14336, not 12288. |
config.json -> head_dim, num_hidden_layers, intermediate_size | 128 / 36 / 12288 | Exactly Qwen3-8B. Llama-3.1-8B is 128 / 32 / 14336. |
base_model tag in the original card was simply wrong. This is metadata only and does not affect inference; we correct it for hygiene, but did not consider it a "bug" in the inference sense./chat/completions was broken/chat/completions endpoint (whether served by vLLM, SGLang, TGI, or Friendli Engine) follows the OpenAI Chat Completions contract. It receives a list of {"role": ..., "content": ...} messages and must convert them to a single token stream before sending it to the model. That conversion is performed by the Jinja template stored as tokenizer.chat_template, exposed in tokenizer_config.json.tokenizer_config.json did not contain a chat_template field at all. When the chat template is missing:ValueError: As of transformers v4.44, default chat template is no longer allowed, so you must provide a chat template … and the request fails with HTTP 400.<|im_start|> / <|im_end|> markers Qwen3 was trained on, producing nonsense output.chat_template for its OpenAI-compatible route.generate) would still work because they bypass templating, which is consistent with a "completions doesn't work" report rather than a "model doesn't load" report.chat_template insertionQwen/Qwen3-8B) is the minimal change that resolves the failure. Every other field in tokenizer_config.json is already correct for Qwen3: eos_token is <|im_end|>, pad_token is <|endoftext|>, special tokens 151643 through 151668 (including <think> and </think>) are all properly registered.<|im_start|>role\ncontent<|im_end|>).tools=[...] parameter for function calling, embedding tool definitions in the system message and wrapping calls in <tool_call>…</tool_call> (the same tokens already declared in added_tokens_decoder).enable_thinking boolean used by Qwen3 to gate <think>…</think> blocks (which feeds directly into Problem (b)).add_generation_prompt so streaming endpoints know where to start the assistant turn.config.json: max_position_embeddings = 40960 vs. tokenizer_config.json: model_max_length = 131072. This looks like a contradiction but it matches the upstream Qwen/Qwen3-8B repo exactly. 40960 is the native pre-trained context; 131072 is the YaRN-extended cap once rope_scaling is enabled. Not an inference bug.bos_token = null, add_bos_token = false. Correct for the Qwen family; the model was not trained with a BOS token.vocab.json plus merges.txt and tokenizer.json present. Redundant but harmless; the fast tokenizer.json is preferred, the legacy files exist for backward compatibility. Not a bug.eos_token_id = 151645 in config.json but [151645, 151643] in generation_config.json. Intentional. Stopping on either <|im_end|> (turn boundary) or <|endoftext|> (document boundary) is the standard Qwen3 behavior.1vllm serve g1nie/broken-model-fixed \
2 --served-model-name broken-model
3
4curl http://localhost:8000/v1/chat/completions \
5 -H "Content-Type: application/json" \
6 -d '{
7 "model": "broken-model",
8 "messages": [{"role":"user","content":"Hello"}]
9 }'