You can deploy this model and start earning money today!

UgurI/
Qwen3.8-27B-Turkish-CPT-LoRA-Beta

Name: Qwen3.8-27B-Turkish-CPT-LoRA-Beta
Availability: InStock

peft

safetensors

turkish

continued-pretraining

Views

No views yet

Qwen3.8-27B-Turkish-CPT-LoRA-1M

A continued-pretrained Low-Rank Adaptation (LoRA) adapter for Qwen3.8-27B, trained on high-quality filtered Turkish web text from the Kumru corpus.

Model Summary

Base Model: Qwen/Qwen3.8-27B (Text-only causal LM backbone)
Adapter Type: PEFT LoRA (Rank 16, Alpha 16)
Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Training Tokens: 1,000,000 tokens (512 token packed context)
Training Precision: 4-bit NF4 Base + BF16 Compute / Gradients
Parameters: 79,691,776 trainable parameters (304 MiB safetensors)

Evaluation & Benchmarks

Metric	Base Model (Qwen3.8-27B NF4)	Turkish-CPT-1M Adapter	Improvement
Turkish Validation Perplexity	7.5838	6.9348	-8.56%
Turkish Test Perplexity	8.0534	7.3303	-8.98%
TurkishMMLU (90-Q Stratified)	60.00%	75.56%	+15.56%
Belebele Reading Comprehension (TR)	83.33%	90.00%	+6.67%
Belebele Reading Comprehension (EN)	96.67%	96.67%	100% Retained

Usage with Transformers & PEFT

python

import torch
from transformers import AutoTokenizer, BitsAndBytesConfig, Qwen3_5ForCausalLM
from peft import PeftModel

base_model_id = "Qwen/Qwen3.8-27B"
adapter_id = "UgurInanc12/Qwen3.8-27B-Turkish-CPT-LoRA-1M"

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

tokenizer = AutoTokenizer.from_pretrained(base_model_id)
base_model = Qwen3_5ForCausalLM.from_pretrained(
    base_model_id,
    quantization_config=quantization_config,
    device_map="auto",
    torch_dtype=torch.bfloat16,
)

model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()

prompt = "Türkiye'de yapay zeka araştırmalarının geleceği hakkında bir paragraf yaz."
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

with torch.inference_mode():
    outputs = model.generate(**inputs, max_new_tokens=256, do_sample=True, temperature=0.7)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Citation & Attribution

Training Corpus: VNGRS-AI (vngrs-ai/vngrs-web-corpus)
Base Model: Qwen Team (Qwen/Qwen3.8-27B)
Author: Ugur Inanc

UgurI/Qwen3.8-27B-Turkish-CPT-LoRA-Beta