Green Patent Detection: Advanced Architectures (Multi-Agent Systems)
This repository contains an advanced NLP pipeline for the detection of "Green" (Climate Change Mitigation) patents. Following the baseline established in Assignment 2, this project investigates whether a Multi-Agent System (MAS) workflow involving a structured debate—produces higher-quality training data than a simple LLM prompt.
Project Overview
The core objective of Assignment 3 is to compare two training data generation methods:
Assignment 2 Model: Fine-tuning based on simple LLM labeling of high-uncertainty claims.
Assignment 3 Model: Implementing a Multi-Agent System (MAS) to debate the same high-uncertainty claims.
Multi-Agent Architecture (Option 1)
Implemented a Multi-Agent System using a custom orchestration framework to label 100 high-risk patent claims. The agents utilized the Qwen3 model family (4b and 8b parameters) locally run models through LM Studio through Vulkan compatible AMD software for 9070 XT 16GB card
Settings for Multi-Agent Architecture
Both qwen/qwen3-4b models (Advocate and Skeptic) are set to the same setting of a temperature of 0.3, They have been set to 0.3 to slightly increase their "Creativity", towards their arguements, as would happend in a courtcase where each part has their story.
The qwen/qwen3-8b model (Judge) has the setting temperature of 0.1, to insure more consistent evaluation towards the arguments presented from the two more "Creative" advocater and sketic
Required Agents:
Agent 1 (The Advocate): Argues for the green classification (Y02). Focuses on identifying environmental benefits or energy savings within the claim text.
Agent 2 (The Skeptic): Argues against the classification. Specifically looks for greenwashing or generic technology that lacks specific climate change mitigation attributes.
Agent 3 (The Judge): Impartially weighs the arguments and produces a final JSON label and rationale.
Part D: Human Review and Final Integration
After the MAS generated suggestions, a human expert (Human-in-the-Loop) reviewed the 100 claims and the AI rationales to create the final "is_green_gold" column based on human judgment.
As the dataset 100 samples are the exact same, this presented an oppertunity to compare my previous HITL assigments in Assignment 2 to Assignmen 3, Leading to 10 claims on the same patents being differnt, and afterwords adjusted.
Detailed Agreement Report:
Total claims analyzed: 100, on the basis of the HITL being the "truth" for comparison
Assignment 3 (Multi-Agent) Accuracy: 67.0%
Assignment 2 (Simple LLM) Accuracy: 89.0%
Percentage agreement was reported between human labels and AI suggestions for both Assignment 2 and Assignment 3.
After adjusting my 10 HITL differences between assignment 2 and 3 mentioned in the above text, the results changed to:
Assignment 3 (Multi-Agent) Accuracy: 64.0%
Assignment 2 (Simple LLM) Accuracy: 92.0%
comparison_plot
Making a difference compared to the defined "truth" of HITL definitions and changes therein, of -3% change for the multi agents. and +3% for the single LLM
As Assigment 2 used signle model gpt-oss-20b locally hosted model, this model alone outperforms the combined locally hosted 2X qwen/qwen3-4b + qwen/qwen3-8b model, based on the HITL "truth".
The final comparisom after adjustments shows based of the selected models, performance in comparison to the HITL "truth" is -28% (after HITL adjustments) for the multi model setup compared to the signle LLM model, despite filling up the same amount of VRAM.
Looking at the above table it also shows that the mullti LLM has 34% False positives whereas the multi agent setup actually outperformed the single llm in regards to false negatives by 6%.
Analysis of Performance Differences: MAS vs. Single LLM
The performance gap between the Multi-Agent System (MAS) and the single LLM approach (GPT-OSS-20B) highlights an interesting trade-off between architectural complexity and model scale.
1. The "Advocate Bias" and False Positives
The most striking difference is the high number of False Positives 34% in the Multi-Agent setup compared to 0 in the single LLM approach.
Forced Advocacy: The Advocate agent was explicitly instructed to argue for the green classification, pushing it to find environmental benefits in even generic patent claims.
Creative Reach: With a temperature setting of 0.3, the Advocate was encouraged to be "creative" with its arguments, often over-interpreting generic technical features as green mitigation strategies.
Judge's Dilemma: Even with a conservative Judge (Temp 0.1), the system tended to default to the Advocate's logic if the Skeptic's counter-argument wasn't sufficiently robust.
2. Superior Recall and the Reduction of False Negatives
While the MAS struggled with precision, it outperformed the single LLM in False Negatives, missing only 2% green patents compared to the single LLM's 8%.
Deep Mining: The debate format ensures that a claim is analyzed from multiple angles. The Advocate's persistence ensures that nuanced "green" features—which a single-pass model might overlook as "generic"—are brought to the Judge's attention.
Bias Correction: The MAS debate forces the system to consider broader Y02 categories, such as wastewater treatment or pollution abatement, which single models often overlook in favor of solar or wind energy.
3. Model Scale vs. Workflow Complexity
The results suggest that the raw reasoning power of a larger single model (GPT-OSS-20B) currently outweighs the benefit of a multi-agent debate using smaller, more "creative" models (Qwen3-4B and Qwen3-8B).
Inherent Knowledge: Larger parameter models typically possess a deeper "latent knowledge" of technical patent terminology.
Architectural Trade-off: While the MAS improved human-in-the-loop interpretability through detailed rationales, the single-pass 20B model provided a more accurate "truth" filter for this specific dataset.
Conclusion
The Multi-Agent setup is a powerful tool for high-recall tasks (ensuring no green patent is missed), while the single, larger LLM remains superior for precision. For future iterations, a more aggressive "Skeptic" agent or a more skeptical "Judge" would be required to filter the noise introduced by the "Advocate" role.
Part E: Comparative Analysis
The final Assignment 3 Model was retrained using the combined dataset (Silver Training + 100 Gold High-Risk) to fine-tune the original PatentSBERTa model.
Model Version
Training Data Source
F1 Score (Eval Set)
1. Baseline
Frozen Embeddings (No Fine-tuning)
0.7488
2. Assignment 2 Model
Fine-tuned on Silver + Gold (GPT-OS 20B)(pre adjusting the 10 HITL, from Assignment 2)
0.7465
3. Assignment 3 Model
Fine-tuned on Silver + Gold (2 X qwen/qwen3-4B(Advocate and Skeptic)+ qwen/qwen3-8B (Judge))
0.7467
Reflection
While the Multi-Agent System provided more rigorous rationales and helped identify nuanced greenwashing, it did not lead to notibly better downstream model performance compared to the baseline in this specific test or the other single LLM from assignment 2. This indicates that while agents improve human-in-the-loop interpretability, the raw classification logic of the PatentSBERTa embeddings remains the best factor for predictive performance on this dataset.
Content
debate_transcripts/: A folder containing the complete logs of the multi-agent debates for each of the high-risk patent claims. Each transcript captures the argumentative exchange between the Advocate and Skeptic, along with the Judge’s final JSON verdict and rationale.
Assignment 3.ipynb: The full Jupyter Notebook
assignment3_model.joblib: The final fine-tuned PatentSBERTa classifier. This model was trained on the integrated dataset of 2,000 silver-labeled claims and the 100 MAS-labeled/human-verified "gold" samples.
hitl_green_100.csv: The finalized dataset for this project. It contains the 100 high-risk patent claims, the debate-driven AI suggestions, rationales, and the final human-verified is_green_gold labels.
hitl_green_100_old.csv: The original labeling output from Assignment 2 (Single LLM approach). This is provided to enable the mandatory comparative analysis of agreement rates and model performance.
