Fine-Tuning GPT-2 and Using Transformer-Explainer for Analysis
Project Overview
This project involves fine-tuning the GPT-2 model with the assigned language and analyzing its behavior using the Transformer-Explainer package. The objective is to observe how the fine-tuned model processes language inputs and generates outputs, gaining insights into its inner workings.
Objectives
- Fine-tune GPT-2 with a specific language dataset.
- Use the Transformer-Explainer package to analyze the behavior of the custom-trained GPT-2 model.
- Generate 10 examples and observe the model’s performance using the explainer.
Steps to Complete the Project
- Fine-Tuning GPT-2
Used HuggingFace Transformers to fine-tune the GPT-2 model on finnish language.
Prepared the dataset ensuring it represents the linguistic characteristics and diversity of the assigned language.
- Running Transformer-Explainer
Clone the Transformer-Explainer GitHub repository.
Integrated the custom GPT-2 model with the explainer package.
- Observed and analyzed the output on:
Attention patterns.
Token relationships.
Resources
- Transformer-Explainer GitHub Repository
- HuggingFace Transformers Documentation
- Dataset in the assigned language
