On an out-of-distribution test set of 578 Trendyol products drawn from 18 categories (swimwear, underwear, sportswear, scarves, hats, slippers, maternity, bedding, towels, bathrobes, curtains, belts, wallets, men's bags) that were never seen during GRPO training, this adapter brings the base 12B Asure model above the frontier-LLM bar with statistical significance:

model	macro F1 (OOD, 578 rows)	95% CI	Renk	Materyal	Kalıp	Desen
Asure-12B base + v3 prompt	0.305	[0.274, 0.335]	0.448	0.187	0.169	0.418
Claude Opus 4.7 v3 (frontier reference)	0.338	[0.311, 0.365]	0.409	0.176	0.145	0.624
Asure-12B + this adapter (step 250)	0.364	[0.336, 0.391]	0.456	0.195	0.161	0.644

Δ vs Claude on OOD: +0.026 macro F1, paired-bootstrap 95% CI [+0.011, +0.041], P(no improvement) = 0.001 — significant at 99.9% confidence.

The trained adapter wins on all 4 fields OOD (not just Desen). Confidence intervals are computed via paired bootstrap (N=2000 resamples, same indices across models so per-prompt difficulty cancels — see eval/bootstrap_ci.py).

In-distribution test (200 rows, held-out from training)

For the original held-out test split:

model	macro F1 (in-dist, 200 rows)	95% CI	Renk	Materyal	Kalıp	Desen
Asure-12B base + v3 prompt	0.338	[0.294, 0.383]	0.452	0.188	0.269	0.447
Claude Opus 4.7 v3 (frontier reference)	0.366	[0.325, 0.406]	0.457	0.158	0.278	0.572
Asure-12B + this adapter (step 250)	0.380	[0.338, 0.421]	0.465	0.195	0.278	0.583

In-dist Δ vs Claude: +0.014 macro F1, 95% CI [−0.008, +0.036], P=0.11 — directionally above Claude, but the 200-row sample is too small to reach 95% significance. The OOD result above is the more rigorous test.

The biggest single contributor to the gain is the Desen (pattern) field: recall jumped from 0.41 → 0.71 because the policy fully internalized the catalog-default rule (Desen → "Düz" when no explicit pattern keyword is in the title — the modal value at ~65% in-dist and ~75% OOD).

Intended use

Given a Turkish product title (and optionally a brand), produce a strict JSON object with up to four catalog attributes:

Renk (color) — canonical colors, e.g. "Beyaz", "Siyah", "Lacivert", ...
Materyal (material) — catalog adjective form, e.g. "Pamuklu" (not "Pamuk"), "Denim" (not "Jean"), ...
Kalıp (fit) — short English form without "Fit" suffix, e.g. "Slim" (not "Slim Fit"), "Regular", "Oversize", ...
Desen (pattern) — defaults to "Düz" (plain) when no pattern keyword is in the title; otherwise "Çizgili", "Çiçekli", "Baskılı", ...

Use null (literal JSON null, not the string "null") when an attribute can't be inferred.

Example output

Input: Marka: Mavi\nÜrün adı: Erkek Slim Fit %100 Pamuk Basic Tişört Beyaz

Output:

{"Renk": "Beyaz", "Materyal": "Pamuklu", "Kalıp": "Slim", "Desen": "Düz"}

Usage

This is a PEFT/LoRA adapter — you must load it on top of the base model.

With vLLM (recommended for serving)

bash

vllm serve Trendyol/Trendyol-LLM-Asure-12B \
    --enable-lora \
    --lora-modules grpo=KocKaankk/asure-12b-tr-attr-grpo \
    --max-lora-rank 16 \
    --served-model-name asure-12b-grpo \
    --dtype bfloat16 --max-model-len 4096

Then call the OpenAI-compatible endpoint with model="asure-12b-grpo". See the prompt template below for the exact system+user shape we trained on.

With transformers + PEFT (research / single-call)

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base = AutoModelForCausalLM.from_pretrained(
    "Trendyol/Trendyol-LLM-Asure-12B",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(base, "KocKaankk/asure-12b-tr-attr-grpo")
tok = AutoTokenizer.from_pretrained("KocKaankk/asure-12b-tr-attr-grpo")

# See "Prompt template" below for SYSTEM and USER strings.
messages = [
    {"role": "system", "content": SYSTEM},
    *FEW_SHOTS,
    {"role": "user", "content": USER.format(title="Erkek Slim Fit %100 Pamuk Basic Tişört Beyaz", brand_line="Marka: Mavi\n")},
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
out = model.generate(
    **tok(prompt, return_tensors="pt").to(model.device),
    max_new_tokens=128, do_sample=False,
)
print(tok.decode(out[0], skip_special_tokens=True))

Prompt template (v3, the one we trained on)

The model is sensitive to the prompt format. The system text sets per-field canonical vocabulary and the Desen → "Düz" default rule; the user template is a tight Marka:/Ürün adı:/JSON: shape with three Turkish few-shot examples between system and user.

The complete system prompt is ~280 tokens — too long to paste here. The exact text used in training is in eval/prompts.py (V3 constant). If serving in production, copy that file verbatim — there is no other supported prompt for this adapter.

Training

	value
algorithm	GRPO (Group Relative Policy Optimization) via TRL
base model	Trendyol/Trendyol-LLM-Asure-12B (12B params, Turkish base)
adapter	LoRA, rank=16, alpha=32, dropout=0.05, target_modules=`all-linear`
quantization (training)	4-bit NF4 (QLoRA, double-quant, bf16 compute)
reward function	per-row macro F1 across 4 fields (`Renk`, `Materyal`, `Kalıp`, `Desen`), strict normalized exact-match
reward parser	strict JSON; markdown fences stripped; case-canonicalized field names
rollouts	vLLM colocated, G=8 generations per prompt, T=0.8
optimizer	AdamW 8-bit
learning rate	1.5e-5, cosine schedule, 10% warmup
KL coefficient (beta)	0.02
effective batch	4 prompts × 8 generations × 2 grad-accum = 64 rollouts per gradient update
training steps	300 (saved every 50, this is checkpoint-250 — the peak by held-out F1)
training data	1,278 Turkish Trendyol product titles + Trendyol-provided gold attribute labels
held-out test	200 prompts (deterministic SHA1 split, SHA256-fingerprinted; never seen in training)
compute	1× NVIDIA L40S (48 GB), AWS `g6e.4xlarge`, ~2:38 wallclock
training cost	~$5.90 in on-demand GPU

Held-out checkpoint sweep

We saved checkpoints every 50 steps and evaluated each on the held-out test set:

step	macro F1 on test	Δ vs base
base	0.339	ref
50	0.369	+0.030
100	0.374	+0.035
150	0.375	+0.036
200	0.376	+0.037
250	0.380	+0.041 (peak — this checkpoint)
300	0.376	+0.037 (slight overshoot — cosine LR tail)

Distribution shape (best-of-8 at T=0.8 on the held-out test set)

metric	base	this adapter (step 250)	Δ
mean-of-8	0.319	0.369	+0.050 (low-reward tail lifted)
best-of-8	0.457	0.463	+0.006 (ceiling held — no sacrifice)
RL headroom (best − mean)	+0.137	+0.094	−0.043 (−31%)

Distribution narrowed by ~31% with no ceiling sacrifice — textbook successful RLVR signature.

Limitations

Domain-specific. Trained on a 1,278-title slice of 36 Trendyol categories. Out-of-distribution categories (luxury, perishables, electronics with very different attribute schemas) have not been evaluated.
4 fields only. Trendyol pages have 8-12 attributes per product; this model is good only at the 4 most common ones.
Capability narrowing. RL on a narrow reward function reduces the model's general-Turkish capabilities. This is fine for a specialized attribute-extraction model and a known cost of RLVR; do not use this adapter for free-form Turkish chat.
Prompt-locked. The model only works well with the v3 prompt above. Other prompts (especially zero-shot) will significantly underperform.
JSON output is regex-parsed downstream. The model produces well-formed JSON in 200/200 test cases, but a robust serving pipeline should still validate.
Statistical robustness. The in-distribution +0.014 over Claude (on 200 rows) is not statistically significant (95% CI [−0.008, +0.036], P=0.11). The out-of-distribution +0.026 (on 578 rows) is significant (95% CI [+0.011, +0.041], P=0.001). When citing this model, prefer the OOD number — it's both larger and more rigorous. See notes/v0.6.2-ood-eval.md in the source repo for full methodology.

Reproducibility

Full code, dataset scrapers, training script, eval pipeline, and Terraform infra: github.com/KocKaankk/tr-rl (repo link will be live once published).

The training run that produced this adapter is documented end-to-end in notes/v0.6-grpo.md of that repo.

Citation

If you use this adapter in research:

bibtex

@misc{karakas2026asuregrpo,
  author       = {Karakaş, Kaan},
  title        = {Asure-12B + GRPO: Turkish e-commerce attribute extraction},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/KocKaankk/asure-12b-tr-attr-grpo}},
}

License

CC BY-NC 4.0 — inherited from the base model (Trendyol/Trendyol-LLM-Asure-12B).

KocKaankk/asure-12b-tr-attr-grpo-v0.8-experimental

Asure-12B + GRPO — Turkish e-commerce attribute extraction

Headline result