A Switch Transformer (Mixture-of-Experts T5) model fine-tuned for abstractive text summarization. The model uses sparse expert routing to scale model capacity without a proportional increase in compute per token.
Switch Transformers replace the dense feed-forward sublayers in standard T5 with Mixture-of-Experts (MoE) layers. Each token is routed to one of num_experts=8 expert feed-forward networks by a learned routing function, allowing the model to specialize different experts for different types of input. This repo contains a fine-tuned variant configured for summarization.
1from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
2
3tokenizer = AutoTokenizer.from_pretrained("YashNagraj75/SwitchTransformers-Summarization/switch-transformer-tokenizer")
4model = AutoModelForSeq2SeqLM.from_pretrained("YashNagraj75/SwitchTransformers-Summarization/switch-transformer")
5
6text = "Your long document or conversation here..."
7inputs = tokenizer("summarize: " + text, return_tensors="pt", truncation=True)
8outputs = model.generate(
9 **inputs,
10 max_length=200,
11 min_length=30,
12 num_beams=4,
13 length_penalty=2.0,
14 no_repeat_ngram_size=3,
15 early_stopping=True,
16)
17print(tokenizer.decode(outputs[0], skip_special_tokens=True))
