You can deploy this model and start earning money today!

flavianv/
deepoutfit-qwen17b-gpt41-150-sft80

Name: deepoutfit-qwen17b-gpt41-150-sft80
Availability: InStock

Views

No views yet

DeepOutfit Qwen3 1.7B GPT-4.1 SFT80

This repository contains the sft80_from150 DeepOutfit checkpoint: a Qwen3 1.7B supervised fine-tune trained for a JSON-action outfit recommendation harness.

The model is not intended to be used as a plain chat model. It was trained to operate inside a structured agent harness that sends event-style messages, exposes a catalog-search tool, accumulates product candidates, and validates a structured final outfit report.

Model Details

Hub repo: flavianv/deepoutfit-qwen17b-gpt41-150-sft80
Local eval alias: sft80_from150
Base model: Qwen/Qwen3-1.7B
Architecture: Qwen3ForCausalLM
Training method: supervised fine-tuning
Teacher source: GPT-4.1 JSON-action traces from the DeepOutfit harness
Training query source: first 150 prompts from the local OUTFIT500-derived outfit set
Filtering: GPT-4.1 judged trace quality >=80
Training rows: 133 filtered traces
Target style: final-only assistant targets from JSON-action traces
Source checkpoint path: /home/criteo/reco-rl-json-action/outputs/models/qwen3-1.7b-json-action-outfit-gpt41-150-ge80-sft_20260530_152442_finalonly_20260530_152848

Selected training configuration recovered from the saved checkpoint:

bf16 training
batch size 1 per device
gradient accumulation 16
learning rate 5e-6
linear scheduler
max sequence length 16384
assistant-only loss
optimizer adamw_torch_fused

Harness Dependency

This model's behavior is tied to the DeepOutfit harness. The harness is part of the task definition, not just an evaluation wrapper.

Important harness pieces:

structured event messages rather than a single natural-language user prompt
planning enabled before catalog search
tool names and action schema such as todo_writer, search_products, and finalize_report
catalog-backed product search over the Clothing catalog
top_k=5 product search results
maximum 5 tool calls
final report validator requiring valid JSON, exactly 5 items, unique product IDs, and product IDs sourced from tool results
candidate accumulation and finalization logic in the local DeepOutfit batch-eval harness

The model expects the harness to provide messages like task-start events, planning instructions, tool results, and candidate updates. It is not expected to solve the complete outfit task from a naked chat prompt.

Prompt And Judge Caveat

Results for this model are not comparable to older DeepOutfit reports unless the full harness is held fixed.

Across the local experiments, several pieces changed:

the system prompt / task-start event wording
the planning instruction and JSON schema
the toolset and tool result presentation
candidate accumulation and finalization behavior
maximum context / completion lengths
decoding settings
the judge model and judge prompt
the quality rubric and failure flags
error handling for missing reports, broken reports, invalid JSON, and missing API keys

In particular, newer evaluations use a GPT-4.1 outfit judge with an updated rubric. Earlier RL/GRPO experiments used different reward prompts and sometimes GPT-4.1-mini. Treat the reported numbers as internal harness metrics, not as a public benchmark.

JSON-Action Protocol Sketch

The harness starts with structured task information, for example:

json

{
  "event": "outfit_task_start",
  "task_type": "outfit",
  "user_query": "men's Ibiza nightlife outfit for clubbing and rooftop drinks, stylish breathable and comfortable, not costume-like",
  "target_k": 5,
  "max_tool_calls": 5
}

The model should produce a planning action:

json

{
  "action": "todo_writer",
  "look": "stylish breathable men's Ibiza nightlife outfit with polished warm-weather separates",
  "searches": [
    "men breathable nightlife shirt",
    "men tailored lightweight trousers",
    "men stylish loafers",
    "men lightweight evening jacket",
    "men minimalist watch"
  ]
}

The harness then executes catalog searches and returns tool-result events. The model eventually emits:

json

{
  "action": "finalize_report",
  "results": [
    {
      "rank": 1,
      "product_id": "...",
      "category": "Clothing",
      "reasoning": "..."
    }
  ]
}

Evaluation

Latest Low-Temperature First-50 Holdout

This is the most recent first-50 holdout result for sft80_from150 from the local batch_eval_outfit_models.py harness.

Settings:

query file: queries/Clothing/OUTFIT500.json
query limit: 50
rollouts per query: 1
category: Clothing
task type: outfit
planning: on
max tool calls: 5
top-k: 5
generation temperature: 0.2
generation top-p: 0.9
judge model: gpt-4.1
judge temperature: 0

Result:

Metric	Value
Rows	50
Overall score mean	72.5391
Quality score mean	71.6867
Quality score median	71.3000
Quality `>=70`	50%
Quality `>=75`	46%
Quality `>=85`	32%
Correctness score mean	97.6
Valid JSON rate	100%
Exactly top-k rate	96%
All IDs from tool results	100%
Unique product IDs	100%
Missing report rate	0%
Broken report rate	4%
Rollouts/minute	1.9623

Same-run comparison:

Model	Training source	Overall	Quality	Q >=70	Q >=85	Correctness	Broken
`sft80_from150`	GPT-4.1 SFT, first 150, score >=80	72.54	71.69	50%	32%	97.6	4%
`onpolicy_sft_85`	on-policy SFT score >=85	72.77	70.53	52%	30%	100.0	0%
`bestprompt_ge90_from_sft80`	one best score >=90 rollout/query	70.90	71.07	50%	30%	100.0	0%
`offline_sdpo_400x4_ge90ctx`	offline SDPO-style distillation	67.78	65.87	50%	22%	98.8	2%

Older Promptfix Eval Snapshot

The previous card reported a different 50-query promptfix eval. That run used a different harness snapshot and should be treated separately.

Model	Overall	Quality	>=70 Quality	Correctness	Broken Report	Rollouts/min
Qwen3 1.7B zero-shot	65.84	59.36	32%	96.4	6	4.787
Previous DeepOutfit SFT+DPO	55.77	43.73	10%	86.8	22	4.229
`sft80_from150`	73.51	72.51	56%	100.0	0	4.996

Additional probes from that older run:

Probe	Score
Easy math generalization	10 / 10
Collapse probe suite	100 / 100

Strengths

Strong JSON-action validity in the DeepOutfit harness.
Good outfit quality relative to other local SFT/RL variants tested so far.
Uses tool results rather than inventing product IDs in the evaluated harness.
Preserves basic non-outfit behavior on small local sanity probes.

Limitations

Strongly coupled to the DeepOutfit event protocol and catalog.
Product IDs and product metadata are catalog-specific.
Metrics are internal GPT-4.1 judge metrics, not human ratings.
Prompt, tool, and judge changes can move scores materially.
The model may still produce weak outfits when search results are noisy or role coverage is ambiguous.
This checkpoint should not be used as evidence that unrelated RL or SDPO variants improved quality; those require harness-matched comparisons.

Recommended Inference Settings

For deterministic evaluation, use low-temperature decoding similar to the latest first-50 run:

temperature 0.2
top-p 0.9
one rollout per query

The saved generation_config.json contains a more exploratory setting (temperature=0.6, top_p=0.95, top_k=20) used for sampling-style experiments. Use the lower-temperature settings for judge comparisons.

Minimal Loading Example

This only loads the model. It does not recreate the DeepOutfit harness.

python

from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "flavianv/deepoutfit-qwen17b-gpt41-150-sft80"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    torch_dtype="auto",
    device_map="auto",
)

To reproduce reported behavior, run it through the local DeepOutfit JSON-action harness with the same tool schema, task-start events, product catalog, and judge configuration.

Intended Use

Research checkpoint for agentic tool-use outfit recommendation. It is useful for comparing SFT/RL/SDPO variants under a fixed harness. It is not a consumer styling service and should not be treated as a general-purpose fashion advisor.