Sentiment Analysis NLP Pipeline (Rotten Tomatoes)
This project builds a complete NLP data processing pipeline for sentiment analysis using the Rotten Tomatoes dataset.
The workflow focuses on:
- Data exploration
- Data cleaning
- Feature engineering
- Tokenization and model preparation
No model training has been performed yet. This project prepares the dataset for training transformer-based models such as DistilBERT.
Dataset
I used the Rotten Tomatoes dataset from Hugging Face Datasets library.
It contains movie reviews labeled as:
- 0 → Negative
- 1 → Positive
Project Steps
1. Data Loading
The dataset is loaded using the Hugging Face datasets library and inspected for structure, splits, and features.
2. Data Exploration
The dataset was analyzed to:
- View sample examples
- Inspect schema (features)
- Check label distribution across splits
3. Label Processing
Numeric labels were mapped to text labels:
- 0 → negative
- 1 → positive
A new column label_text was added.
4. Train/Test Split
The training data was split into:
This ensures reproducibility using a fixed random seed.
5. Data Cleaning
Text data was cleaned using:
- Lowercasing
- Removing special characters using regex
- Removing extra whitespace
Additional features were created:
- clean_text
- char_length
- word_count
6. Data Filtering
Very short texts (less than 3 words) were removed to improve data quality.
7. Tokenization
The dataset was tokenized using:
- distilbert-base-uncased tokenizer
- max_length = 96
- padding = max_length
- truncation enabled
8. Model Preparation
The dataset was formatted for PyTorch:
- input_ids
- attention_mask
- labels
Unused columns were removed to prepare for training.
9. Dataset Subsets
For experimentation:
- 300 training samples were selected
- 100 test samples were selected
Output Summary
The final dataset is fully prepared for training transformer-based models using Hugging Face Trainer or PyTorch.
Model Status
No model training or evaluation has been performed yet.
This project focuses on building a clean and structured NLP preprocessing pipeline.
Tools Used
- Hugging Face Datasets
- Hugging Face Transformers
- PyTorch
- Pandas
- Regex
Future Work
- Fine-tune DistilBERT on the prepared dataset
- Add evaluation metrics (accuracy, F1-score)
- Deploy model using Hugging Face Spaces
