llama.cpp exports of a Qwen3.5-4B vision-language model fine-tuned for a fixed Cityscapes structured JSON task. The release includes five text-model precisions and three multimodal projector precisions.

This is a full language-and-vision LoRA merge. Do not add the original adapter with --lora: its language and vision changes are already in these files.

llama.cpp quick start

Example using the recommended Q4_K_M text model and BF16 mmproj:

bash

llama-server \
  -m Cityscapes-Qwen3.5-4B-Q4_K_M.gguf \
  --mmproj mmproj-Cityscapes-Qwen3.5-4B-BF16.gguf \
  --ctx-size 4096 \
  --parallel 1 \
  -ngl 99 \
  --jinja \
  --no-webui \
  --host 127.0.0.1 \
  --port 8080

For the exact precision used in the reported benchmark, replace the text model with Cityscapes-Qwen3.5-4B-BF16.gguf.

The server accepts OpenAI-compatible multimodal requests at /v1/chat/completions. Put the image before the text and use a schema-explicit prompt:

python

import base64
import mimetypes
import requests

image_path = "frankfurt_000000_000294_leftImg8bit.png"
mime = mimetypes.guess_type(image_path)[0] or "image/png"
with open(image_path, "rb") as handle:
    image_url = f"data:{mime};base64,{base64.b64encode(handle.read()).decode()}"

system_prompt = """You extract structured Cityscapes facts from an image.
Return raw JSON only with exactly the top-level keys surfaces, objects, relations.
surfaces contains exactly road and sidewalk with present and zones.
objects contains exactly person, rider, car, truck, bus, train, motorcycle,
bicycle, traffic_light, traffic_sign, pole with present, count, zones.
Zones use a 3x3 grid: upper|middle|lower combined with left|center|right.
Counts are 0, 1, 2-3, 4-7, 8+, or unknown. An absent object uses false, 0, [].
Relations contain subject, relation=on, and object=road|sidewalk."""

response = requests.post(
    "http://127.0.0.1:8080/v1/chat/completions",
    json={
        "messages": [
            {"role": "system", "content": system_prompt},
            {
                "role": "user",
                "content": [
                    {"type": "image_url", "image_url": {"url": image_url}},
                    {
                        "type": "text",
                        "text": (
                            "Analyze this urban road image and return only one "
                            "completed JSON object in the required schema."
                        ),
                    },
                ],
            },
        ],
        "temperature": 0,
        "seed": 3407,
        "max_tokens": 1024,
        "chat_template_kwargs": {"enable_thinking": False},
    },
    timeout=600,
)
response.raise_for_status()
print(response.json()["choices"][0]["message"]["content"])

What was trained

Base: unsloth/Qwen3.5-4B, revision 3764fa359b9082ea5a1e4a5e3ac3aaf6e9671636.
Data: 2,975 Cityscapes training images and 500 validation images.
Objective: image-first supervised generation of a strict JSON structure.
Classes: 2 surfaces and 11 objects.
Predictions: presence, object count bucket, 3×3 zones, and conservative instance on surface relations.
Training: one epoch / 372 optimizer steps, BF16 16-bit LoRA SFT, assistant tokens only.
Effective train/eval batch: 8/4.
LoRA: rank 16, alpha 16, dropout 0, language and vision attention/MLP layers.
Optimizer: AdamW Torch, learning rate 1e-4, cosine schedule, 5% warmup, weight decay 0.001.
Context: 4,096; seed: 3,407.
Hardware: NVIDIA RTX 6000 Ada Generation 48 GB.
Final training loss: 0.06588; final validation loss: 0.04468.

Cityscapes source images and annotations are not included in this repository.

BF16 base-vs-tuned evaluation

The formal quality comparison used BF16 text + BF16 mmproj for both the untouched base and this tuned model. All 500 validation images used identical llama.cpp requests:

explicit schema system prompt plus one text-only format example;
image-first messages;
temperature=0, seed=3407, max_tokens=1024;
thinking disabled;
no grammar and no response_format;
10,000 paired bootstrap resamples.

Strict all-sample results:

Metric	Base BF16	Tuned BF16	Delta
JSON valid rate	0.292	1.000	+0.708
Strict schema valid rate	0.214	1.000	+0.786
Task score	0.1840	0.8054	+0.6214

task_score equally weights presence macro F1, per-class count accuracy, zone micro F1, and relation micro F1.

A formatting-controlled semantic diagnostic removed only a single whole-response Markdown JSON fence and selected the 402/500 rows where both models passed the strict schema:

Metric	Base BF16	Tuned BF16	Delta	Paired-bootstrap 95% CI
Presence macro F1	0.8110	0.8989	+0.0879	[0.0619, 0.1156]
Count macro accuracy	0.7280	0.8211	+0.0932	[0.0821, 0.1043]
Zone micro F1	0.3495	0.8447	+0.4953	[0.4857, 0.5046]
Relation micro F1	0.5711	0.6513	+0.0802	[0.0606, 0.1001]
Task score	0.6149	0.8040	+0.1891	[0.1790, 0.2000]

The predefined paired comparison gate passed. The largest semantic improvement was zone localization, so the measured benefit was not only JSON formatting.

The exact evaluation prompt SHA-256 was b7b80a3b18578acb8406af229392f9d5585563ecbace3aac724145f2d8187a3f.

Artifact validation

Every GGUF in this directory was checked for:

GGUF V3 magic and readable metadata;
expected architecture (qwen35 text or clip mmproj);
expected component type (model or mmproj);
expected tensor count (441 text, 298 mmproj);
actual tensor types matching the named precision;
complete SHA-256 checksum.

The BF16 pair was used successfully by llama.cpp for the full 500-image evaluation. The quantized text files and alternate mmproj precisions passed format, metadata, and load checks, but were not each rerun over all 500 images. Do not assume Q2_K has the same task score as BF16.

Limitations

The result is task-specific and does not establish a general VLM gain.
Quantization can reduce task quality, especially at Q2_K.
Count output is bucketed and relations cover only direct road/sidewalk support.
Small or occluded objects can be missed.
Use of these files is restricted to non-commercial purposes, and citing the Cityscapes Dataset is a condition of that use. See License and Citation.

License

The effective terms for these GGUF files are the intersection of two licenses, and that intersection is non-commercial.

The base model unsloth/Qwen3.5-4B is Apache-2.0.
The Cityscapes Dataset License also applies, because these fine-tuned weights are a derivative work of the dataset. It states that you may not use the dataset or any derivative work for commercial purposes, such as licensing or selling the data, or using the data with a purpose to procure a commercial gain.

This repository is therefore not labelled Apache-2.0. Apache-2.0 on its own would grant commercial rights that the Cityscapes terms withhold. If you need commercial use, take that up with the Cityscapes authors rather than relying on the base model's license.

The Cityscapes license does permit distributing abstract derivative works such as trained models, provided they do not allow the dataset to be recovered, which is what makes publishing these exports possible. It does not permit redistributing the dataset itself, so no Cityscapes images or annotations are included here.

Citation

Referencing the Cityscapes Dataset in any work that uses these files is a condition of the dataset license, not a courtesy.

bibtex

@inproceedings{Cordts2016Cityscapes,
  title     = {The Cityscapes Dataset for Semantic Urban Scene Understanding},
  author    = {Cordts, Marius and Omran, Mohamed and Ramos, Sebastian and
               Rehfeld, Timo and Enzweiler, Markus and Benenson, Rodrigo and
               Franke, Uwe and Roth, Stefan and Schiele, Bernt},
  booktitle = {Proc. of the IEEE Conference on Computer Vision and Pattern
               Recognition (CVPR)},
  year      = {2016}
}

Singularity87/Cityscapes-Qwen3.5-4B-GGUF

Cityscapes Qwen3.5 4B GGUF