Unlike standard pre-trained models (like BERT), this architecture was built from scratch to demonstrate the implementation of Self-Attention and Positional Encodings in PyTorch.
During development, the model was benchmarked against a Bidirectional LSTM. The Transformer architecture achieved a ~5% improvement in accuracy, demonstrating its superior ability to capture long-range dependencies in product reviews.
1from transformers import DistilBertTokenizer
2import torch
3
4# 1. Initialize Tokenizer
5tokenizer = DistilBertTokenizer.from_pretrained('Nefflymicn/amazon-sentiment-transformer')
6
7# 2. Load Model (Architecture must match)
8model = TransformerSentimentModel(
9 vocab_size=tokenizer.vocab_size,
10 embed_dim=128,
11 num_heads=8,
12 ff_dim=512,
13 num_layers=4,
14 output_dim=2
15 )
16model.load_state_dict(torch.load("pytorch_model.bin", map_location='cpu'))
17model.eval()
