A specialized set of LoRA adapters fine-tuned for improving and analyzing Air Traffic Control (ATC) communications, extracting relevant information from raw transcripts.

Model Details

Model Description

These adapters fine-tune the meta-llama/Llama-3.2-3B-Instruct model to specialize in processing Air Traffic Control communications. When applied to the base model, it can:

Improve raw ATC transcripts with proper punctuation and formatting
Identify communication intentions (pilot requests, ATC instructions, etc.)
Extract key information such as flight numbers, altitudes, headings, and other numerical data
Analyze speaker roles and communication patterns

The adapters were created using LoRA (Low-Rank Adaptation) with PEFT (Parameter-Efficient Fine-Tuning) techniques to efficiently adapt the Llama 3B model to this specialized domain.

Developed by: Sang-Buster
Model type: LoRA adapters for meta-llama/Llama-3.2-3B-Instruct
Language(s): English, specialized for ATC terminology
License: Same as the base model
Finetuned from model: meta-llama/Llama-3.2-3B-Instruct

Uses

Direct Use

These adapters are intended for:

Transcribing and formatting raw ATC communications
Training ATC communication skills
Analyzing ATC communication patterns
Extracting structured data from ATC communications
Educational purposes for those learning ATC communication protocols

Downstream Use

The model can be integrated into:

Air traffic management training systems
Communication analysis tools
ATC transcript post-processing pipelines
Aviation safety monitoring systems
Radio communication enhancement systems

Out-of-Scope Use

This model is not suitable for:

Real-time ATC operations or safety-critical decision-making
Full language translation (it's specialized for ATC terminology only)
General language processing outside the ATC domain
Any application where model errors could impact flight safety

Bias, Risks, and Limitations

The model is specialized for ATC communications and may not perform well on general text
It may have limitations with accents or non-standard ATC phraseology
Performance depends on audio transcription quality for real-world applications
Not intended for safety-critical applications without human verification
May have biases based on the training data distribution

Recommendations

Always have human verification for safety-critical applications
Use in conjunction with standard ATC protocols, not as a replacement
Provide clear domain context for optimal performance
Test thoroughly with diverse ATC communications before deployment
Consider fine-tuning further on your specific ATC subdomain if needed

How to Get Started with the Model

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load the base model
base_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-3B-Instruct",
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-3B-Instruct")

# Load the adapter
model = PeftModel.from_pretrained(base_model, "atc_llama")

# Process an ATC message
instruction = "As an ATC communication expert, improve this transcript and analyze its intentions and data."
message = "southwest five niner two turn left heading three four zero descend and maintain flight level two five zero"

prompt = f"<|begin_of_text|><|header_start|>user<|header_end|>\n\n{instruction}\n\nOriginal: {message}<|eot|><|header_start|>assistant<|header_end|>\n\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

# Generate improved transcript and analysis
outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False)
response = tokenizer.decode(outputs[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(response)

Training Details

Training Data

The model was trained on a dataset of ATC communications with:

Original raw transcripts
Properly punctuated and formatted versions
Annotated intentions (PSC, PSR, PRP, PRQ, PRB, PAC, ASC, AGI, ACR, END)
Extracted numerical data (altitudes, headings, flight numbers, etc.)
Speaker and listener information

Training Procedure

The model was fine-tuned using LoRA with the following approach:

Parameter-efficient fine-tuning using PEFT
LoRA applied to key attention layers (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj)
Optimized with Unsloth for efficiency

Training Hyperparameters

Base model: meta-llama/Llama-3.2-3B-Instruct
LoRA rank: 16
LoRA alpha: 16
Learning rate: 0.0002
Batch size: 4
Gradient accumulation steps: 4
Epochs: 3
Warmup ratio: 0.03
Max sequence length: 2048
Training regime: BF16 mixed precision where available, FP16 otherwise
Optimizer: AdamW 8-bit

Evaluation

Testing

The adapters should be tested on diverse ATC communications, including:

Clearances and instructions
Pilot requests and reports
Emergency communications
Different accents and speaking patterns

Technical Specifications

Model Architecture and Objective

Base architecture: meta-llama/Llama-3.2-3B-Instruct
Fine-tuning method: LoRA with PEFT
Optimization library: Unsloth
Training objective: Improving and analyzing ATC communications

Compute Infrastructure

Framework versions:
- PEFT (compatible with the base model)
- Unsloth (for efficient LoRA fine-tuning)
- Transformers (compatible with the base model)
- PyTorch (with BF16 support where available)

Sang-Buster/atc-llama-adapters

ATC Communication Expert Model (LoRA Adapters)