Model Card: PatentSBERTa Fine-Tuned on Green Patent Claims (Assignment 3)
Model Summary
This model is a fine-tuned version of
AI-Growth-Lab/PatentSBERTa for binary classification of patent claims as green technology (Y02) or not. It was developed as part of Assignment 3 in the Applied Deep Learning and AI course at Aalborg University. Compared to Assignment 2, this model uses a more advanced Multi-Agent System (MAS) to generate higher-quality gold labels for the 100 high-risk claims before fine-tuning PatentSBERTa.
Model Details
- Developed by: Anders Sønderbý (as58zr@student.aau.dk)
- Model type: Sentence Transformer with classification head (binary)
- Base model: AI-Growth-Lab/PatentSBERTa
- Language: English
- License: MIT
- Task: Binary text classification — Green Technology (Y02) vs. Not Green
What This Model Does
Given the text of a patent claim, the model predicts whether the claim relates to green technology as defined by the CPC Y02 classification system. The output is a binary label:
1 — Green technology (Y02)0 — Not green technology
Key Difference from Assignment 2
In Assignment 2, a single generic LLM was used to suggest labels for the 100 high-risk claims before human review. In Assignment 3, a Multi-Agent System (MAS) using CrewAI was used instead, where three specialised agents debated each claim before producing a final label. The hypothesis is that adversarial debate between agents produces higher-quality gold labels, which in turn produces a better fine-tuned PatentSBERTa model.
Training Pipeline Overview
Stage 1 & 2 — Setup (Same as Assignment 2)
The same patents_50k_green.parquet balanced 50k dataset was used. Uncertainty scores were recomputed from the Assignment 2 baseline model and the same top 100 high-risk claims (hitl_green_100.csv) were selected for labeling.
Stage 3 — Multi-Agent Labeling (CrewAI)
Three agents debated each of the 100 high-risk patent claims:
| Agent | Role | Objective |
|---|
| Advocate | Green Patent Expert | Argue why the claim qualifies as Y02 green technology |
| Skeptic | Greenwashing Analyst | Challenge the Y02 classification and identify greenwashing |
| Judge | Senior Patent Examiner | Weigh both arguments and produce a final JSON label + rationale |
The Judge output for each claim: {"label": 0 or 1, "rationale": "2-3 sentence explanation"}
The LLM used for all three agents was groq/meta-llama/llama-4-scout-17b-16e-instruct via the Groq API.
Stage 4 — Human Review (HITL)
A human reviewer assessed all 100 claims using the agent arguments and Judge rationale as context. The final gold label (is_green_gold) reflects the human decision, with the AI rationale available as supporting context.
Stage 5 — Fine-Tuning PatentSBERTa
PatentSBERTa was fine-tuned for binary classification using the combined train_silver + gold_100 dataset, where gold labels override silver labels for the 100 HITL-reviewed claims.
Training Data
- Dataset: Derived from AI-Growth-Lab/patents_claims_1.5m_traim_test
- Working file: patents_50k_green.parquet — a balanced 50k sample (25,000 green, 25,000 not green)
- Silver label source: CPC Y02* classification codes (is_green_silver)
- Gold labels: 100 human-reviewed claims labeled via MAS debate (is_green_gold)
Dataset Splits
| Split | Size | Description |
|---|
| train_silver | ~40,000 | Silver-labeled training set (CPC-derived) |
| eval_silver | ~5,000 | Silver-labeled evaluation set |
| pool_unlabeled | ~5,000 | Unlabeled pool used for uncertainty sampling |
| gold_100 | 100 | Human-reviewed high-uncertainty claims (MAS-assisted) |
Training Hyperparameters
| Parameter | Value |
|---|
| Base model | AI-Growth-Lab/PatentSBERTa |
| Max sequence length | 256 |
| Epochs | 1 |
| Learning rate | 2e-5 |
| Training set size | ~40,100 (train_silver + gold_100) |
Evaluation Results
| Evaluation Set | F1 Score | Notes |
|---|
| eval_silver (5,000) | 0.824 | Primary evaluation metric |
| gold_100 (100) | 0.667 | Human-reviewed high-uncertainty claims |
Comparative Analysis
| Model Version | Training Data Source | F1 Score |
|---|
| 1. Baseline | Frozen Embeddings (No Fine-tuning) | 0.780 |
| 2. Assignment 2 Model | Fine-tuned on Silver + Gold (Simple LLM) | 0.818 |
| 3. Assignment 3 Model | Fine-tuned on Silver + Gold (MAS - CrewAI) | 0.824 |
The MAS approach produced a modest improvement in F1 score (+0.006) over the simple LLM approach from Assignment 2. While the improvement is small, the adversarial debate structure between Advocate and Skeptic agents likely produced more nuanced and reliable gold labels for the high-risk claims, particularly for borderline cases where a single LLM might have been overconfident. The added engineering complexity of the MAS is partially justified by the quality improvement, though the marginal gain suggests that the bottleneck may lie in the size of the gold label set (100 claims) rather than label quality alone.
HITL Agreement Reporting
Human-AI agreement was tracked for both Assignment 2 and Assignment 3:
| Assignment | Labeling Method | Human-AI Agreement |
|---|
| Assignment 2 | Simple generic LLM | [97 %] |
| Assignment 3 | Multi-Agent System (MAS) | [94 %] |
Intended Use
- Primary use: Academic research and coursework in patent classification
- Intended users: Course instructors and students at Aalborg University
- Out-of-scope: Production patent classification systems, legal patent assessment, or any commercial use
Limitations
- Trained on a balanced 50k sample — performance may differ on the full unbalanced patent corpus
- Silver labels are derived from CPC codes, which may contain noise
- Gold labels are based on 100 claims only — a larger gold set would likely improve downstream performance more significantly
- The MAS agents occasionally showed bias, with the Advocate tending to over-generalise green characteristics and the Judge sometimes deferring too strongly to one agent's argument
Repository
The full code, notebooks, and data files for this assignment are available in the course GitHub repository.
