Developed by: leeminwaan
Finetuned from model: unsloth/Qwen3.5-4B
Methodology: Latent Regularization (LR) via Contextual Representation Alignment.
Project Designation: ViO (Vague is Objective)

This model utilizes a non-standard training objective designed to mitigate Sequential Semantic Drift—a failure mode in smaller parameter models where autoregressive generation progressively diverges from the input context.

Methodology: Latent Regularization (LR)

The standard Cross-Entropy (CE) loss was augmented with an Auxiliary Contextual Penalty. During training, the model's internal representations were constrained using a custom objective function targeting the latent space.

Technical Implementation

Contextual Reference Vector: A reference anchor is derived by mean-pooling the hidden states of the penultimate layer of the input prompt.
Temporal Latent Smoothing: To isolate semantic signals from syntactic fluctuations, a 1D-Average Pooling filter (Window Size: 8) is applied across the sequence dimension of the generated hidden states.
Cosine Margin Constraint: A regularization loss is applied to penalize the cosine distance between the Reference Vector and the Smoothed Latent States.

Objective Function: $$\mathcal{L}{total} = \mathcal{L}{CE} + \lambda \cdot \mathbb{E} \left[ \max(0, \tau - \cos(z_{ref}, \Phi(H_{gen}))) \right]$$ Where $\Phi$ denotes the 1D-temporal filter, $\tau$ represents the similarity margin, and $z_{ref}$ is the contextual reference.

Research Findings: Semantic Compression (SC³)

During the fine-tuning process, a novel behavioral pattern emerged, termed Semantic Compression via Cosine Constraint (SC³). When subjected to latent alignment penalties, the model converged toward a specific mathematical optimum.

Stochastic Convergence to Semantic Centroids

The model demonstrates a preference for semantically central tokens (high-level abstractions) over high-variance, specific tokens. In high-dimensional latent space, specific terms exist as outliers with high directional variance. Vague tokens function as "centroids"—maintaining higher cosine similarity across a broader range of the manifold, thus minimizing the regularization penalty while satisfying the Cross-Entropy objective.

Emergent Latent Variable Creation

The model demonstrates an ability to perform Symbolic Shorthand. It frequently maps complex, multi-token structures from the prompt to abstract internal variables (e.g., utilizing "expression" or "result" as persistent semantic pointers). This reduces the cumulative "latent walk" required to maintain logical continuity, leading to:

Reduced Sequence Entropy: Highly focused reasoning chains.
Implicit Abstraction: Automatic categorization of specific data into abstract classes to maintain proximity to the similarity margin.

Precision-Stability Equilibrium

Observed training dynamics indicate a marginal increase in Cross-Entropy loss in exchange for a significant reduction in Auxiliary Loss. The model optimizes for Stability over Precision.

Usage

This model is optimized for long-form reasoning where structural consistency and contextual anchoring are prioritized.

python

from unsloth import FastLanguageModel
import torch

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "leeminwaan/qwen_3_4B_ViO_LR_lora",
    load_in_4bit = True,
)

FastLanguageModel.for_inference(model)

# Standard inference
messages = [
    {"role": "user", "content": "Analyze the following expression and solve for x: (x + 2)^2 = 0"}
]
inputs = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt").to("cuda")
outputs = model.generate(input_ids=inputs, max_new_tokens=1000)