You can deploy this model and start earning money today!

mohanadffy/
gemma3-intent-paraphraser-lora

Name: gemma3-intent-paraphraser-lora
Availability: InStock

safetensors

Views

No views yet

Gemma-3 Intent Paraphraser (LoRA Adapter)

This repository contains the LoRA adapter weights for fine-tuning unsloth/gemma-3-1b-it into a high-precision intent-preserving paraphrase generator.
The model rewrites a user query into multiple paraphrases that maintain identical intent, answer type, and semantic meaning, strictly following a JSON schema.

This LoRA is designed for research on:

intent boundary detection
semantic caching robustness (SAFE-CACHE / SIVer)
adversarial query transformation studies
controlled, conservative paraphrasing
structured JSON generation

📌 Base Model

The adapter is applied on top of the following base model:

Base: unsloth/gemma-3-1b-it
Context length: 2048
Precision: fp16
No 4-bit or 8-bit quantization during fine-tuning

The base model is not included in this repository.
PEFT automatically loads it using the field in adapter_config.json.

📦 LoRA Configuration

The model was fine-tuned using Unsloth + TRL SFTTrainer with the following LoRA parameters:

Parameter	Value
LoRA Rank (`r`)	48
LoRA Alpha	192
LoRA Dropout	0.0
Target Modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Gradient Checkpointing	Unsloth
Full Finetuning	Disabled
4-bit / 8-bit loading	Disabled
Random Seed	3407

These settings encourage tight semantic control, low drift, and conservative paraphrasing.

🧠 Training Objective

The model is trained to:

✔ Generate 4–5 high-precision paraphrases

same meaning
same intent
same entities
same answer type
JSON-only: {"examples": ["...", "..."]}
no markdown, no surrounding text

✔ Avoid semantic drift

Dataset paraphrases were tightly filtered and manually checked to ensure answer equivalence.

❌ Not trained for:

malicious adversarial queries
unsafe content
domain knowledge generation
long-form rewriting

🧪 Training Dataset

The model was trained on approximately 600 records, generated using a cluster-based augmentation pipeline with GPT-4.1.

For each query, the dataset contains two structured components:

1. Positive Intent-Preserving Paraphrases

Each q entry includes 4–5 high-precision paraphrases that maintain:

identical semantic intent
identical answer type
consistent entities
no additional context or semantic drift

These entries form the sole training targets used during supervised fine-tuning.

2. Near-Miss Intent Variations (Semantic Drift)

For each cluster, GPT-4.1 additionally generated near-miss queries—inputs that remain lexically similar but intentionally introduce subtle meaning changes, such as:

modifying dates or temporal expressions
shifting platform/device
altering specificity (“in the UK” → “in Europe”)
pluralization or scope changes
entity generalization

These near-miss samples were not used as labels during SFT, but served as negative signals during dataset verification and evaluation. They ensure that the model learns tight intent boundaries, avoiding over-generalization.

Sample Entries

Example 1 — “Truth Social” on iOS

json

{
  "q": "When will Truth Social launch on iOS?",
  "examples": [
    "When will Truth Social be available on iOS?",
    "What is the release date for Truth Social app on iPhone?",
    "When does the Truth Social app launch on iOS?",
    "Truth Social iOS release date?",
    "Can you tell me when iOS users can access Truth Social?"
  ]
}

Example 2 — “Truth Social” on Android

json

{
  "q": "When will Truth Social launch on Android?",
  "examples": [
    "What is the release date for Truth Social on Android?",
    "When is Truth Social expected to become available on Android?",
    "When can Android users access Truth Social?",
    "On which date will Truth Social be launched for Android devices?",
    "When will the Android version of Truth Social be released?"
  ]
}

The dataset is intentionally constructed to emphasize conservative, intent-preserving paraphrasing, avoiding creative rewriting and ensuring high-fidelity semantic equivalence across paraphrase clusters.

🔧 Training Script (Summary)

Training used the following libraries:

unsloth (FastModel)
TRL SFTTrainer
PEFT LoRA
HuggingFace Datasets

Key training hyperparameters:

Hyperparameter	Value
Epochs	2
Learning Rate	5e-5
Warmup Steps	5
Optimizer	adamw_8bit
Batch Size	4
Weight Decay	0.01

A strict conversation-style template was applied using:

<start_of_turn>user
<start_of_turn>model

Training was responses-only, ensuring the model learns to output only JSON examples.

🚀 Usage

🔹 Load the LoRA Adapter + Base Model (Unsloth)

python

import torch
from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    "mohanadffy/gemma3-intent-paraphraser-lora",
    max_seq_length = 2048,
    dtype = torch.float16,
    load_in_4bit = False,
)

FastLanguageModel.for_inference(model)

🔹 Run Inference

python

SYSTEM_MSG = (
    "You are an intent paraphrase generator.
"
    "Given a user query, output 5 diverse paraphrased examples expressing the SAME intent.
"
    "CRITICAL RULES:
"
    "1. Output ONLY a valid JSON object.
"
    "2. Follow EXACT schema: {\"examples\": [\"...\", \"...\"]}
"
    "3. No commentary. No markdown. No extra text.
"
)

def generate_intent_examples(query, max_tokens=300):
    prompt = tokenizer.apply_chat_template(
        [
            {"role": "system", "content": SYSTEM_MSG},
            {"role": "user",  "content": query},
        ],
        tokenize=False,
        add_generation_prompt=True
    )

    inputs = tokenizer(prompt, return_tensors="pt").to(
        "cuda" if torch.cuda.is_available() else "cpu"
    )

    output = model.generate(
        **inputs,
        max_new_tokens=max_tokens,
        do_sample=True,
        temperature=0.3,
        top_p=0.95,
    )

    text = tokenizer.decode(output[0])

    # Extract only the JSON block
    import re
    match = re.search(r"\{.*\}", text, flags=re.DOTALL)
    return match.group(0) if match else text

🔹 Example

print(generate_intent_examples("When will Truth Social launch on iOS?"))

📖 Recommended Citation

@misc{gemma_intent_paraphraser_2025,
  title        = {Gemma-3 Intent Paraphraser (LoRA Adapter)},
  author       = {Mohanad Afiffy , Mohamed Waleed Fakhr , Fahima Maghraby},
  year         = {2025},
  howpublished = {\url{https://huggingface.co/mohanadffy/gemma3-intent-paraphraser-lora}},
}

📜 License

This repository contains only LoRA adapter weights, which represent a derivative behavioral modification. Users must comply with the Gemma base model license and Unsloth usage terms when reconstructing the full model.

🔍 Notes

The model is designed for research use, not production deployment.
Always validate JSON output during inference.
Recommended generation settings:
- temperature = 0.1–0.3
- top_p = 0.9–0.95
- max_new_tokens = 200