Active Learning for Patent Classification: Green Technology Identification
This repository contains an Active Learning pipeline designed to classify patent claims into "Green" (Climate Change Mitigation) and "Not Green" categories based on the CPC Y02 taxonomy.
Project Overview
This project implements a Human-in-the-Loop (HITL) workflow to refine a patent classifier. By combining automated silver labeling with expert human validation of high-uncertainty samples, we improve the model's ability to identify complex environmental technologies.
Part A: Baseline Model Evaluation
The baseline model utilized PatentSBERTa for feature extraction and a Logistic Regression classifier trained on 2,000 silver-labeled samples.
Baseline Performance Metrics:
- Macro F1-Score: 0.7488
- Macro Precision: 0.7489
- Macro Recall: 0.7488
The baseline demonstrated reasonable accuracy but lacked nuance in identifying technologies outside of primary renewables like wind and solar.
Part B: Uncertainty Sampling
Using Uncertainty Sampling, 100 high-risk claims (where predicted probability $p \approx 0.5$) were extracted for human review. The formula used for uncertainty was $u = 1 - 2 \cdot |p - 0.5|$, where a score of 1.0 represents maximum uncertainty.
Note on Part B: Due to a kernel restart during the workflow, the script generates a reproduction file named hitl_green_100_REPRODUCED.csv. This ensures that the original labeling progress is protected while still providing the required metrics for the uncertainty scores (Min: 0.9959, Max: 1.0000).
Part C: Human-in-the-Loop (HITL) Reporting
An LLM provided initial suggestions and rationales, which were then validated or overridden by a human expert. The LLM used was GPT-OSS-20B which was locally run via LM studio on a 9070 XT 16Gb Vram AMD card the Vulkan software. Temperature set to 0.1.
Labeling Statistics:
- Total Claims Labeled: 100
- Human Overrides: 3 - detected mistakes where LLM failed to classify a technology that was green.
- Agreement Rate: 97.0%
Examples of Human Overrides
A critical component of this project was correcting model biases. Below are representative examples where the human expert overrode the LLM suggestion:
- Nuclear Reactor Components (Doc 9064607)
- LLM Suggestion: 0
- Human Final: 1
- Rationale: Nuclear Energy is recognized as a low-carbon energy source for climate change mitigation under CPC Y02E 30/00. The LLM displayed a bias toward renewable-only definitions.
- Wastewater Treatment Flocculants (Doc 9328007)
- LLM Suggestion: 0
- Human Final: 1
- Rationale: Technologies reducing environmental impact in waste management are categorized under Y02W. The human expert corrected the LLM's exclusion of wastewater treatment.
- Pollution Abatement Reactor (Doc 8833279)
- LLM Suggestion: 0
- Human Final: 1
- Rationale: Technologies for atmospheric pollution abatement (Y02A) are included in the green patent taxonomy. The human expert recognized this broader scope.
Part D: Active Learning Evaluation
The model was retrained using the original training set plus the 100 human-labeled "Gold" samples.
Final Performance Comparison:
| Metric | Baseline (Part A) | Active (Part D) | Improvement |
|---|
| Precision | 0.7489 | 0.7473 | -0.0015 |
| Recall | 0.7488 | 0.7467 | -0.0021 |
| F1-score | 0.7488 | 0.7465 | -0.0023 |
Analysis of Results
The Active Learning loop provided a qualitative improvement by resolving high-uncertainty samples. While the numerical F1-score showed a minor decrease of approximately 0.2%, this is expected when introducing complex, "expert" labels that challenge the simple patterns learned from silver labels. The model gained specific robustness in identifying "hidden" green categories like Nuclear and Wastewater management, effectively trading a small amount of generic accuracy for higher expert precision in contested technology areas.
Files in Repository
- assignment_2.ipynb: The comprehensive Python implementation including data processing, baseline training, uncertainty sampling, human labeling loops, and the final active learning evaluation.
- hitl_green_100.csv: The 100-sample "Gold Label" dataset generated during the human-in-the-loop phase, featuring model predictions, LLM rationales, and final human-verified labels.
- patents_50k_green.parquet: The balanced silver-standard training dataset consisting of 50,000 patent claims, which is required for the baseline and active learning model training.
