Readme for Intent Detection in Conversational Systems Using Few-shot Multi-class Classification
Introduction
This repository contains the source code and resources for the report on "Intent Detection in Conversational Systems Using Few-shot Multi-class Classification" by Shivam Dwivedi and Chehak Malhotra. The research focuses on intent recognition within conversational systems, framing it as a few-shot multi-class classification problem.
How to run the code ??
Description
This project involves a multi-phase machine learning workflow, with the development (dev) phase to be executed locally in a terminal or system environment, and the final phase to be run in Google Colab. The project leverages transfer learning, requiring specific setup for BERT model and tokenizer files.
Getting Started
Prerequisites
- Python 3.x
- Terminal or command-line interface for local development
- Google Colab for final phase execution
- Google Drive account for storing BERT model and tokenizer files
- Hugging Face account (if you need to clone the repository)
Installation
-
Clone the repository from Hugging Face to your local machine.
1git clone <huggingface-repository-url>
2cd <repository-directory>
-
Install required packages. It's recommended to use a virtual environment.
1python3 -m venv env
2source env/bin/activate
3pip install -r requirements.txt
All code for the dev phase should be run locally. Make sure you are in the project's main directory.
1python script1.py
2python script2.py
Final Phase
The final phase code is intended to be run in Google Colab.
- Upload the
filename.ipynb notebook to your Google Colab.
- Download the
datafile.zip from the Hugging Face repository and upload it to Google Colab.
Transfer Learning Setup
For the transfer learning part of the project, you will need to download the BERT model and BERT tokenizer files and upload them to your Google Drive.
- Download BERT model and BERT tokenizer files from the official source or any other trusted provider.
- Upload the downloaded files to a folder in your Google Drive.
- In your Google Colab notebook, mount your Google Drive to access the uploaded BERT files.
1from google.colab import drive
2drive.mount('/content/drive')
Navigate to the directory containing the BERT files in your notebook and set up the transfer learning as required by the project.
NOTE - > The final two codes are named final 1 and final 2
Support
If you encounter any issues or have any questions, please open an issue in the Hugging Face repository, and we will get back to you as soon as possible. Reach out to
shivam21352@iiitd.ac.in for a faster response.
REPORT
Intent Detection in Conversational Systems Using Few-shot Multi-class Classification
Shivam Dwivedi and Chehak Malhotra
Abstract
The field of natural language processing (NLP) has witnessed substantial advancements in recent years, with intent detection emerging as a critical component in conversational systems. This report delves into a comprehensive study conducted as part of Datathon@IndoML, focusing on the challenging task of intent recognition framed as a few-shot multi-class classification problem. The competition consisted of two main phases, with the final phase being the central subject of this report.
Feature Selection / Method Description
Note -> Validation scores are based on a train-test split of the given training data, whereas test scores are obtained after submission to the Codalab platform, using the competition's blind test data.
Development Phase
Initially, the development phase of the competition involved a similar intent recognition task but encompassed multiple languages and was based on a larger database. During this phase, various models such as BERT, XLNet, RoBERTa, and T5 were experimented with. However, T5 failed to produce satisfactory results. The accuracy scores achieved were as follows:
- BERT: 83.42%
- XLNet: 80.15%
- RoBERTa: 82.17%
Upon finalizing BERT as the model of choice, further fine-tuning led to the following results on the test set:
- Accuracy: 81.37%
- F1 Score: 77.83%
- Precision: 79.09%
- Recall: 77.22%
Final Phase
Initial Testing
In the final phase, basic models such as BERT, RoBERTa, DeBERTa, BART, XLNet, and T5 were revisited. The code for these experiments is housed in 'code1_initial_tests.py'. These models yielded accuracy scores ranging from 83% to 87% during validation testing. The insights drawn were as follows:
- RoBERTa outperformed BERT by approximately 2.4%.
- DeBERTa outperformed BERT by approximately 6.5%.
- BART and XLNet underperformed compared to BERT by about 2.6% and 4.2%, respectively.
- T5 failed to yield satisfactory results, similar to the development phase.
Ensemble Modeling
After the initial round of experiments, ensemble models were trained using combinations of BERT, RoBERTa, DeBERTa, and BART. These models were fine-tuned, and their performance was evaluated on the validation set. DeBERTa, when used in ensembles, consistently showed superior performance. The best ensemble model comprised BERT, RoBERTa, DeBERTa, and BART, achieving an accuracy of 92.59%, an F1 score of 92.55%, and a precision of 94.08%.
Validation Scores
Ensemble of BERT:
- Accuracy: 0.8919
- F1-Score: 0.8896
- Precision: 0.9206
Ensemble of RoBERTa:
- Accuracy: 0.9096
- F1-Score: 0.9090
- Precision: 0.9306
Ensemble of DeBERTa:
- Accuracy: 0.9304
- F1-Score: 0.9306
- Precision: 0.9448
Ensemble of BART:
- Accuracy: 0.9111
- F1-Score: 0.9092
- Precision: 0.9245
Ensemble of BERT, RoBERTa:
- Accuracy: 0.8978
- F1-Score: 0.8942
- Precision: 0.9234
Ensemble of BERT, DeBERTa:
- Accuracy: 0.9141
- F1-Score: 0.9134
- Precision: 0.9350
Ensemble of BERT, BART:
- Accuracy: 0.9052
- F1-Score: 0.9017
- Precision: 0.9252
Ensemble of RoBERTa, DeBERTa:
- Accuracy: 0.9170
- F1-Score: 0.9166
- Precision: 0.9386
Ensemble of RoBERTa, BART:
- Accuracy: 0.9141
- F1-Score: 0.9122
- Precision: 0.9331
Ensemble of DeBERTa, BART:
- Accuracy: 0.9185
- F1-Score: 0.9163
- Precision: 0.9334
Ensemble of BERT, RoBERTa, DeBERTa:
- Accuracy: 0.9230
- F1-Score: 0.9233
- Precision: 0.9406
Ensemble of BERT, RoBERTa, BART:
- Accuracy: 0.9156
- F1-Score: 0.9149
- Precision: 0.9316
Ensemble of BERT, DeBERTa, BART:
- Accuracy: 0.9259
- F1-Score: 0.9254
- Precision: 0.9399
Ensemble of RoBERTa, DeBERTa, BART:
- Accuracy: 0.9215
- F1-Score: 0.9206
- Precision: 0.9346
Ensemble of BERT, RoBERTa, DeBERTa, BART:
- Accuracy: 0.9259
- F1-Score: 0.9255
- Precision: 0.9408
Despite the promising results on the validation set, the performance on the actual test set (obtained after submitting predictions on the competition portal) was slightly lower, with the ensemble of all four models achieving 88.3% and the ensemble of RoBERTa and DeBERTa achieving 89.88%. Now this was close to the target we wanted to reach, but we still wanted to explore multiple other possibilities.
Comparison with GPT Models
Following the experiments with transfer learning models, we sought to evaluate the performance of GPT models on the same intent recognition task. We utilized GPT-2, GPT-3.5, and GPT-4 for this purpose. The respective code for each model is saved in "Code 6," "Code 7," and "GPT4 Folder".
The accuracy scores achieved by the GPT models on the validation set are as follows:
- GPT-2: Negligible (close to 0%), failed to perform
- GPT-3.5: 63.3%
- GPT-4: 83.5%
From these results, it was evident that the GPT models, particularly GPT-2, were not as effective for this specific intent recognition task as the other models we had experimented with. GPT-4 showed a competitive performance, but it still lagged behind the best ensemble models we had developed. This led us to conclude that for this particular application, the transfer learning models, and especially the ensemble approaches, were more optimal than the GPT-based models.
Utilizing Hugging Face's Pre-trained Models
Hugging Face’s repository is a treasure trove of pre-trained models, developed and fine-tuned by the global AI community on a plethora of datasets and for diverse tasks. We meticulously selected seven pre-trained models from the repository for our experiments. The details and the code related to these experiments are meticulously documented and can be found in the ‘Code3_testing-massive-optimised-libraries’ directory.
For the sake of maintaining the anonymity of the model creators and to uphold the integrity of our report, we have decided to refer to these models numerically, from 1 to 7. The results from our experiments, based on the validation set (train-test split), rounded off to the nearest integer, are as follows:
- Model 1: 72%
- Model 2: 84%
- Model 3: 81%
- Model 4: 76%
- Model 5: 78%
- Model 6: 80%
- Model 7: 81%
Despite the diversity in the models and the robustness of transfer learning as a strategy, the accuracies obtained were consistently lower than our previous attempts with other architectures and methodologies. This was a pivotal moment in our experimentation phase, prompting a reevaluation of our approach.
Transfer Learning Experiments - Utilising Dev-Phase Model - Final Model 1
In the journey to optimize our model's performance for the intent recognition task, we revisited the model we had trained during the development phase. This phase was marked by a multitude of experiments with various architectures, eventually leading us to fine-tune a BERT model. The details of the experiments and the codes are preserved in 'code_final1-devmodel++', 'bert_model', and 'bert_tokenizer' directories.
Utilizing the model from the development phase, we employed basic transfer learning techniques to adapt the model to the current task. Further, in an attempt to boost performance, we experimented with ensembling this transfer model with previously tried-and-tested models, namely BERT and RoBERTa.
Test Results:
- Transfer Model alone: 0.8623
- Ensemble with Transfer, BERT, and RoBERTa: 0.8968
While these results were promising, they did not entirely meet our expectations. The primary challenge stemmed from the lower accuracy of the model in the development phase itself. Despite this, we have chosen to include this approach in our final report, as it represents a comprehensive and theoretically sound strategy, embodying our most ideal approach to date.
Final Model 2: Leveraging RoBERTa Large and DeBERTa V2 Large
Motivated by our incessant quest for improvement, we proceeded to explore additional models and strategies. The code for this part of the experiment can be found in 'Code_final2_rob+deb2.ipynb'.
Model Selection: In this iteration, we opted to use 'RoBERTa Large' and 'DeBERTa V2 Large', given their state-of-the-art performance in various NLP tasks.
Test Results: The results from our final evaluation on the test set were as follows:
| Metric | Value |
|---|
| Accuracy | 0.9030 |
| F1 Score | 0.9016 |
| Precision | 0.9074 |
| Recall | 0.9030 |
These results underscored the efficacy of using state-of-the-art models like RoBERTa Large and DeBERTa V2 Large, showcasing a significant improvement in performance across all metrics.
Throughout our extensive experimentation and rigorous testing, we conducted numerous other tests and explored various models and strategies. While not all of them made it to this final report due to their relevance and impact on the final results, they were instrumental in shaping our understanding and approach. All the code files and notebooks pertaining to these experiments are diligently stored and documented in our repository for future reference and transparency.
In conclusion, our journey from the development phase to the final implementation was marked by trials, errors, and invaluable learning. The results from "Final Model 1' and 'Final Model 2' represent our best efforts and the pinnacle of our performance in this intent recognition task, showcasing the power of state-of-the-art models and the significance of a well-thought-out strategy in the realm of Natural Language Processing.
Experimental Results
In this section, we will delve deeper into the results obtained from various experiments conducted throughout the final phase of Datathon@IndoML, focusing on the task of intent recognition framed as a few-shot multi-class classification problem.
Initial Testing and Model Comparison
In the outset of the final phase, we revisited basic models including BERT, RoBERTa, DeBERTa, BART, XLNet, and T5. The aim was to establish a baseline performance and understand how these models fare individually in the intent recognition task. RoBERTa and DeBERTa emerged as the top performers during the validation phase, outperforming BERT by approximately 2.4% and 6.5% respectively. On the contrary, BART and XLNet showcased underwhelming performance, while T5 once again failed to yield satisfactory results.
This initial round of experimentation set the stage for subsequent trials, providing valuable insights into the models' capabilities and limitations. The discrepancy in performance between the validation scores and the test scores obtained post-submission prompted us to delve deeper, exploring ensemble models and advanced transformer models to bridge this gap.
Ensemble Modeling and Advanced Transformers
Moving forward, we experimented with ensemble models, leveraging combinations of BERT, RoBERTa, DeBERTa, and BART. The ensemble of BERT, RoBERTa, DeBERTa, and BART demonstrated promising results on the validation set, achieving an accuracy of 92.59%, an F1 score of 92.55%, and a precision of 94.08%. However, the performance on the test set post-submission was slightly lower, indicating the need for further exploration and optimization.
In parallel, we explored GPT-based models, specifically GPT-2, GPT-3.5, and GPT-4. Despite GPT-4's potential, it did not surpass the performance of our ensemble models, thereby affirming our decision to focus on ensemble strategies and transfer learning models for this particular task. The GPT models showcased varied results, with GPT-4 being the only model that demonstrated competitive performance, albeit still lagging behind our best ensemble models.
Transfer Learning and Hugging Face's Pre-trained Models
Our final leg of experimentation involved transfer learning, utilizing both a BERT model pre-trained during the development phase and several massive pre-trained models from the Hugging Face repository. The BERT model from the development phase, when adapted to the current task, showcased promising results, with the ensemble of Transfer Model, BERT, and RoBERTa achieving an accuracy of 89.68%.
However, the pre-trained models from the Hugging Face repository yielded lower accuracies compared to our previous experiments. Despite the extensive range of models explored, and the robustness of transfer learning as a strategy, the performance did not meet our expectations, underscoring the challenges and intricacies of the few-shot multi-class classification problem at hand.
Figures and Tables
Table: Test results of "Final Model 1"
| Model | Accuracy |
|---|
| Final Model 1 | 0.8623 |
Table: Test results of "Final Model 1 (Ensemble)"
| Model | Accuracy |
|---|
| Final Model 1 (Ensemble) | 0.8968 |
Table: Test results of "Final Model 2"
| Metric | Value |
|---|
| Accuracy | 0.9030 |
| F1 Score | 0.9016 |
| Precision | 0.9074 |
| Recall | 0.9030 |
Novelty
The contribution of this study lies in its exploration of intent detection in conversational systems through the lens of few-shot multi-class classification. The approach is distinctive in several ways:
- Extensive Model Exploration: A variety of models, including BERT, XLNet, RoBERTa, T5, and various GPT variants, were meticulously explored and tested. This broad spectrum of experimentation provides a comprehensive understanding of how different architectures perform in the context of few-shot multi-class classification for intent detection.
- Ensemble Techniques: The study innovatively combined multiple models through ensemble techniques, resulting in performance enhancement. The approach of using a combination of BERT, RoBERTa, DeBERTa, and BART in an ensemble model was particularly novel, yielding promising results on the validation set.
- Few-shot Learning: The focus on few-shot learning in the context of intent detection is another novel aspect of this study. Few-shot learning is crucial for practical applications where acquiring large amounts of labeled data is expensive or infeasible.
- Transfer Learning with Pre-trained Models: The study extended beyond conventional models, exploring transfer learning with massive pre-trained models from the Hugging Face repository, and using a pre-trained BERT model from the developmental phase. This approach is novel in its application to intent detection and showcases the study’s commitment to pushing the boundaries of what is possible in NLP.
- Comprehensive Evaluation: The evaluation of models was comprehensive, considering not just accuracy, but also F1 score, precision, and recall. Additionally, the distinction between validation scores and test scores was rigorously maintained, providing a clear and honest assessment of the models’ performance.
Conclusion
This report presents a thorough investigation into the application of few-shot multi-class classification for intent detection in conversational systems, conducted as part of Datathon@IndoML. The study traversed a wide array of models, from basic transformers to advanced GPT variants, and innovative ensemble techniques.
The developmental phase pinpointed BERT as a promising model, achieving an accuracy of 81.37% on the test set. However, the realization that further refinements could potentially boost performance laid the groundwork for the extensive experiments conducted in the final phase.
The final phase saw a range of experiments, from revisiting basic models and exploring ensemble models to delving into advanced transformer models and transfer learning. The ensemble models, particularly those based on DeBERTa, exhibited exceptional performance on the validation set. However, the discrepancy in performance on the test set underscored the need for further exploration.
GPT-based models provided varied results, with GPT-4 showing potential, though it did not surpass the ensemble models. The transfer learning experiments, utilizing pre-trained models and a model from the developmental phase, provided insights, though they did not achieve the highest performance.
In essence, this study provides a comprehensive analysis of intent recognition in few-shot multi-class classification settings, offering valuable insights and directions for future research in this domain. The methodologies employed, the range of models explored, and the rigorous evaluation conducted collectively contribute to the broader understanding of intent detection and few-shot learning in NLP.
Our experiments and results pave the way for future research, emphasizing the importance of ensemble techniques, the potential of transfer learning, and the need for further exploration and optimization in few-shot learning settings. The challenges encountered and lessons learned throughout this study underscore the intricacies of intent detection in conversational systems, illuminating paths for future advancements and innovations in this exciting field of NLP.
