This repository provides a TensorFlow Lite (TFLite) version of the DistilGPT2 language model for on-device inference. It enables running DistilGPT2 on mobile and embedded devices with reduced size and latency while preserving output quality.
Model Overview
DistilGPT2 is a distilled version of GPT-2 offering a smaller footprint and faster inference. This TFLite conversion reduces memory usage and enables offline NLP applications such as text generation, completion, and chatbots directly on edge devices.
Base model:distilgpt2
Original framework: PyTorch → TensorFlow → TFLite
Quantization: Float32 (default), INT8 (optional)
Model size: ~240 MB (Float32), ~120 MB (INT8 quantized)
Features
Lightweight and optimized for on-device AI
Supports offline text generation without internet connectivity
Compatible with Android, iOS, and embedded systems via TFLite
Optional full-integer (INT8) quantization for smaller footprint
Example integrations for Python and Flutter
Use Cases
This model can be used in scenarios where cloud-based inference is not possible or desirable, such as:
Mobile assistants – lightweight chatbots that run entirely offline
Smart IoT devices – conversational AI for smart speakers or appliances
Privacy-focused apps – local inference without sending data to servers
Low-connectivity environments – edge devices in rural areas or airplanes
Education apps – offline writing assistants, summarizers, or quiz generators
Performance Matrix
Metric
DistilGPT2 (HF PyTorch)
DistilGPT2 (TFLite FP32)
DistilGPT2 (TFLite INT8)
Model Size
~320 MB
~240 MB
~120 MB
Avg. Latency (per token, CPU)
~120 ms
~80 ms
~55 ms
Memory Footprint (RAM)
~700 MB
~450 MB
~280 MB
Accuracy (compared to FP32)
100%
~99%
~96–97%
Best Use Case
Research / Training
Mobile apps, prototyping
Resource-constrained IoT
(Benchmarks tested on Snapdragon 870 and Intel i5 CPU; numbers may vary depending on device.)
Quick Start Guide
1. Installation
Make sure you have TensorFlow Lite runtime installed:
pip install tflite-runtime tensorFlow
2. Load and Run Inference (Python)
python
1import numpy as np
2import tensorflow as tf
34# Load TFLite model5interpreter = tf.lite.Interpreter(model_path="distilgpt2.tflite")6interpreter.allocate_tensors()78# Get input and output details9input_details = interpreter.get_input_details()10output_details = interpreter.get_output_details()1112# Example input (token IDs from tokenizer)13# NOTE: Use Hugging Face's GPT2 tokenizer to prepare inputs14from transformers import GPT2Tokenizer
15tokenizer = GPT2Tokenizer.from_pretrained("distilgpt2")1617input_ids = tokenizer.encode("The future of AI is", return_tensors="np")1819# Prepare input tensor20interpreter.set_tensor(input_details[0]['index'], input_ids.astype(np.int32))2122# Run inference23interpreter.invoke()2425# Extract output26output = interpreter.get_tensor(output_details[0]['index'])27decoded_text = tokenizer.decode(output[0])28print(decoded_text)
3. Integration with Flutter
dart
1import'package:tflite_flutter/tflite_flutter.dart';23final interpreter =awaitInterpreter.fromAsset('distilgpt2.tflite');45// Prepare input tensor and run inference6// (tokenization/decoding handled separately)7// Generate answer based on query
Contributing
Found an issue or want to improve the model? Feel free to:
Open an issue for bugs or suggestions
Submit pull requests for improvements
Share your use cases and results
License
This model follows the same license terms as the original distilgpt2 model.
Made with ❤️ for the mobile AI community
Enabling powerful text understanding directly on your device
