Negative-Aware Training (NAT) Dataset for Fix-Git Task

What is NAT?

1. Positive examples - Correct trajectories that solve the task
2. Negative examples - Incorrect trajectories that demonstrate common mistakes

How NAT Works

1. Explicit Differentiation

<training_mode>
You are being shown a CORRECT example of how to solve this task.
Generate tool calls that follow the exact format shown, with only the required parameters.
Do NOT add extra parameters like message_title, message_description, or type.
When the task is complete, respond with a brief summary (NO tool call).
</training_mode>

<training_mode>
WARNING: The following is an INCORRECT example that demonstrates common mistakes.
This trajectory contains errors such as:
- Adding unnecessary parameters (message_title, message_description)
- Looping on the same commands after task completion
- Using wrong command formats
Learn to AVOID these patterns.
</training_mode>

2. Training Process

• Both positive and negative examples are mixed in the training data
• The model learns to associate the positive prompt with correct behavior
• The model learns to associate the negative prompt with incorrect behavior
• At inference time, only the positive prompt is used

3. Inference

• What correct tool calls look like
• What mistakes to avoid

Dataset Composition

Example Comparisons

Positive Example (Correct)

Negative Example 1: Hallucinated Arguments ❌

Negative Example 2: Looping Behavior ❌

Negative Example 3: Wrong Command Format ❌

Why NAT Works

• +8.74% improvement with 2k positive + 10k negative examples
• Models implicitly learn from negative samples when combined with positive ones
• The improvement is more substantial when there are fewer positive examples (data-scarce scenarios)
• High-quality negative examples are crucial - low-quality negatives can hurt performance

Files

tokenized_fix_git_v7_nat/
├── train.parquet          # Training data (tokenized)
├── val.parquet            # Validation data (tokenized)
├── README.md              # This file
└── examples/              # Human-readable examples
    ├── positive_messages.json
    ├── negative_hallucinated.json
    ├── negative_looping.json
    ├── negative_wrong_cmd.json
    ├── positive_text.txt
    └── negative_hallucinated_text.txt

Training Command

References

• Wang et al. (2024). "Learning From Failure: Integrating Negative Examples when Fine-tuning Large Language Models as Agents". NAACL 2025. arXiv:2402.11651
• GitHub: Reason-Wang/NAT