Qwen3-8B-TMF921-Intent-QLora

A QLoRA fine-tuned Qwen3-8B that achieves 100% in-distribution schema compliance across 6 telecom standards, 8 lifecycle operations, and adversarial rejection — evaluated on 498 stratified test samples covering all 17 target layer types.

---

Key Results

Evaluation scope note: These results measure in-distribution schema compliance — the test set was generated by the same pipeline as the training data. The 100% score confirms the model has learned to produce correctly formatted, spec-compliant JSON for the template-bounded input distribution. This is analogous to SQL generation models achieving near-perfect scores on Spider test splits while requiring additional evaluation on user-generated queries. Out-of-distribution evaluation on human-written operator intents (e.g., the ORION 100-intent benchmark) is needed to assess real-world generalization and is planned as future work.

---

Evaluation Results

Per-Layer Breakdown (498 stratified samples, 50 per major layer)

KPI Field Presence (create operations, n=300)

Evaluation Methodology

• TMF921, 3GPP, CAMARA, ETSI ZSM: Direct KPI value matching with int/float tolerance
• O-RAN A1 Policy: Reliability → Packet Error Rate (PER), latency → Packet Delay Budget (pdb), throughput → GFBR/MFBR
• O1 NRM (TS 28.541): Structural element presence (rrmPolicyMemberList, operationalState, arfcnDL, bSChannelBwDL)

---

Model Details

---

Training Configuration

---

Usage

Inference

System Prompts by Target Layer

---

Supported Target Layers

6 Specification Layers

8 Lifecycle Operations

3 Adversarial Categories

6 Slice Types

---

Training Data

Training Pipeline

---

Known Limitations

1. In-distribution evaluation only: The 100% results are on a test set from the same synthetic generator. Real operator intents are more ambiguous, underspecified, and linguistically diverse. Out-of-distribution evaluation on human-authored intents is required before deployment claims.
2. Single-turn only: Handles individual intent→config translations. Multi-turn conversations (create → monitor → modify → terminate) were not evaluated.
3. English only: All training data is in English.
4. Synthetic training data: Fine-tuned on template + LLM-augmented data, not real operator intents. Real-world intents may be more ambiguous.
5. HMTC is speculative: SST=5 for HMTC is based on Rel-17 extensions; 6G slice types are not yet standardized.

---

Citation

References

• ORION: Intent-Aware Orchestration in Open RAN
• TelecomGPT
• ORANSight-2.0
• NEFMind
• QLoRA
• TMF921 Intent Management API
• 3GPP TS 28.312