⚠️ Experimental early checkpoint. This is an intermediate training checkpoint (step 20500) from a tool-calling supervised fine-tuning (SFT) run on a Qwen3.5 ~4B base model. It is not a finished, fully-evaluated release. Behavior, quality, and the prompt/tool-call format may change in later checkpoints. Use for research and experimentation only.

Model Overview

This checkpoint is a vision-language, tool-calling model based on the Qwen3.5 architecture (Qwen3_5ForConditionalGeneration). It was produced by an internal NVIDIA Nemotron "edge" experiment (codename steep_jellyfish) that applies tool-calling / function-calling SFT on top of a Qwen3.5 4B-class base model. It accepts interleaved text and images (and video frames) as input and generates text output, including structured <tool_call> blocks when tools are supplied.

Developer: NVIDIA (Nemotron edge experiments)
Base architecture: Qwen3.5 (model_type: qwen3_5)
Fine-tuning objective: Tool-calling / function-calling SFT
Checkpoint: step20500 (intermediate)
Modality: Image + Text → Text (multimodal)
Language: English

Model Architecture

Property	Value
Architecture class	`Qwen3_5ForConditionalGeneration`
Model type	`qwen3_5` (text: `qwen3_5_text`)
Hidden size	2560
Hidden layers	32
Attention pattern	Hybrid: 3× `linear_attention` then 1× `full_attention` (full-attention every 4th layer)
Attention heads	16 (4 KV heads, GQA)
Head dim	256
Linear-attention heads	16 key / 32 value (key & value head dim 128, conv kernel dim 4)
Intermediate size	9216 (SiLU MLP)
Vocab size	248,320
Max position embeddings	262,144 (≈262K context)
RoPE	mRoPE interleaved, θ = 10,000,000, partial rotary factor 0.25
Tied embeddings	Yes
Multi-token prediction	1 MTP layer
Dtype	bfloat16
Vision encoder	24-layer ViT, hidden 1024, patch size 16, spatial merge 2 → out hidden 2560
Total parameters (weights)	≈ 9.3 GB on disk across 2 safetensors shards

The vision tower uses a Qwen2-VL-style image processor (Qwen2VLImageProcessorFast, Qwen3VLProcessor) with image mean/std of 0.5.

Input / Output

Input types: Text; optionally images and video.
Input format: Chat messages (OpenAI/Qwen style) or a plain string.
Output type: Text (free-form, or <tool_call> blocks when tools are provided).
Context length: up to 262K tokens.

Special tokens

Role	Token
EOS	`<
Pad	`<
Vision start / end	`<
Image / video pad	`<

Tool Calling Format

When tools are passed, the chat template injects a system section describing the available functions and instructs the model to reply using nested XML:

<tool_call>
<function=example_function_name>
<parameter=example_parameter_1>
value_1
</parameter>
</function>
</tool_call>

Notes enforced by the template:

Function calls must be wrapped in <tool_call>...</tool_call> with an inner <function=...>...</function> block.
Required parameters must be specified.
Optional natural-language reasoning may appear before (never after) the call.

Usage

With Transformers

python

from transformers import AutoProcessor, AutoModelForImageTextToText
import torch

model_id = "nvidia/nemotron_edge_exp-steep_jellyfish-tool_calling_sft-qwen3_5_4b_base-step20500"

processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    model_id, dtype=torch.bfloat16, device_map="auto", trust_remote_code=True
)

messages = [{"role": "user", "content": "What is the capital of France?"}]
prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[prompt], return_tensors="pt").to(model.device)

out = model.generate(**inputs, max_new_tokens=256)
print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])

Tool calling

python

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the current weather for a city.",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]

messages = [{"role": "user", "content": "What's the weather in Paris right now?"}]
prompt = processor.apply_chat_template(
    messages, tools=tools, tokenize=False, add_generation_prompt=True
)

Serving with vLLM (OpenAI-compatible API)

This repository ships a generation_config.json so the model stops correctly at the end of each chat turn out of the box. The chat template terminates assistant turns with <|im_end|> (token 248046), so the generation config sets eos_token_id: [248046, 248044] (<|im_end|> and <|endoftext|>). Without this, a server would keep generating past the end of the turn.

Launch an OpenAI-compatible server:

bash

vllm serve nvidia/nemotron_edge_exp-steep_jellyfish-tool_calling_sft-qwen3_5_4b_base-step20500 \
    --trust-remote-code \
    --served-model-name nemotron-edge-jellyfish \
    --max-model-len 32768

The server then exposes the standard OpenAI routes, e.g. POST /v1/chat/completions and POST /v1/completions:

bash

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "nemotron-edge-jellyfish",
      "messages": [{"role": "user", "content": "List three uses for a paperclip."}],
      "max_tokens": 256
    }'

Use it from the OpenAI Python client by pointing base_url at the server:

python

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
    model="nemotron-edge-jellyfish",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(resp.choices[0].message.content)

Multimodal (image) requests use the OpenAI image_url content parts, which vLLM maps onto the model's vision tokens:

bash

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "nemotron-edge-jellyfish",
      "messages": [{"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": "https://example.com/cat.jpg"}},
        {"type": "text", "text": "Describe this image."}
      ]}],
      "max_tokens": 256
    }'

Tool calling with vLLM

To expose tool calls through the OpenAI tools field, start the server with auto tool choice enabled:

bash

vllm serve nvidia/nemotron_edge_exp-steep_jellyfish-tool_calling_sft-qwen3_5_4b_base-step20500 \
    --trust-remote-code \
    --served-model-name nemotron-edge-jellyfish \
    --enable-auto-tool-choice \
    --tool-call-parser hermes

Note: This checkpoint emits a custom XML tool-call format (<tool_call><function=...><parameter=...>...</parameter></function></tool_call>) rather than the JSON Hermes format. The built-in parsers may not parse it perfectly; you can still read the raw <tool_call> blocks from the message content, or supply a matching custom --tool-call-parser plugin. Verify parsing against your prompts before relying on structured tool_calls output.

Version requirement: The qwen3_5 architecture is new. Use a vLLM build recent enough to include Qwen3_5ForConditionalGeneration support (install from main if a released version does not yet recognize model_type: qwen3_5).

Deploying on Hugging Face Inference Endpoints

This repository ships a custom handler.py implementing the EndpointHandler interface and a requirements.txt, so it can be deployed directly as a Custom Inference Endpoint (see the Inference Toolkit docs).

Example request body:

json

{
  "inputs": [
    {"role": "user", "content": "List three uses for a paperclip."}
  ],
  "parameters": {"max_new_tokens": 256, "do_sample": false}
}

Multimodal request (image by URL or base64):

json

{
  "inputs": [
    {"role": "user", "content": [
      {"type": "image", "image": "https://example.com/cat.jpg"},
      {"type": "text", "text": "Describe this image."}
    ]}
  ],
  "parameters": {"max_new_tokens": 256}
}

Tool-calling request:

json

{
  "inputs": [{"role": "user", "content": "Weather in Paris?"}],
  "tools": [
    {"type": "function", "function": {
      "name": "get_weather",
      "description": "Get the current weather for a city.",
      "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}
    }}
  ],
  "parameters": {"max_new_tokens": 256}
}

The handler returns:

[{"generated_text": "..."}]

Software Integration

Runtime engines: vLLM (OpenAI-compatible server, recommended for serving) and Hugging Face Transformers (custom handler / Inference Toolkit).
Recommended hardware: NVIDIA GPU with ≥ 16 GB VRAM (bf16). CPU works but is slow.
Operating system: Linux.

The qwen3_5 architecture requires a recent Transformers build (transformers>=4.57.0, or install from source). If model loading fails with an unknown model_type: qwen3_5, upgrade Transformers from GitHub main.

Limitations & Responsible Use

This is an early, intermediate SFT checkpoint and has not undergone full safety, bias, or capability evaluation. Outputs may be inaccurate, incomplete, or unsafe.
Tool-call formatting and instruction-following may be inconsistent at this training step.
No benchmark results are published for this checkpoint.
Use in accordance with the governing license and NVIDIA's Trustworthy AI terms. Do not remove or circumvent safety guardrails without an appropriate substitute for your use case.

License

Governed by the NVIDIA Open Model License. Confirm the exact license terms applicable to this experimental checkpoint with the model owner before any production or commercial use.

Model Version

Checkpoint: step20500
Experiment: nemotron_edge_exp · steep_jellyfish · tool_calling_sft · qwen3_5_4b_base

wdrones/nemo-qwen3_5_4b_base_jellyfish

nemotron-edge-exp · steep_jellyfish · tool_calling_sft · qwen3_5_4b_base · step 20500

Model Overview

Model Architecture

Input / Output

Special tokens

Tool Calling Format

Usage

With Transformers

Tool calling

Serving with vLLM (OpenAI-compatible API)

Tool calling with vLLM

Deploying on Hugging Face Inference Endpoints

Software Integration

Limitations & Responsible Use

License

Model Version

wdrones/
nemo-qwen3_5_4b_base_jellyfish