A robust and feature-rich bigram language model implementation in Python with advanced text preprocessing, smoothing techniques, and model persistence capabilities.
🌟 Features
Advanced Text Preprocessing: Automatic case normalization, punctuation handling, and tokenization
Add-k Smoothing: Prevents zero probabilities and improves generation quality
Sentence Boundary Handling: Proper start/end tokens for natural text generation
Model Persistence: Save and load trained models in JSON format
File Loading: Train on external text corpora
Comprehensive Analysis: Probability analysis, model statistics, and evaluation metrics
Error Handling: Robust error handling and graceful degradation
Backward Compatibility: Legacy function support for existing code
📋 Requirements
Python 3.7+
Standard library modules: random, re, json, collections, typing
1# Load training data from file2sentences = model.load_data_from_file('your_corpus.txt')3model.train(sentences)45# Save trained model6model.save_model('my_bigram_model.json')78# Load model later9new_model = BigramLanguageModel()10new_model.load_model('my_bigram_model.json')1112# Generate with different options13text = new_model.generate_text(14 start_word=None,# Random start15 max_words=20,16 end_on_sentence=True17)
Input Text → Preprocessing → Bigram Counting → Probability Calculation → Text Generation
📊 Example Output
Training sentences:
1. The cat sat on the mat
2. The dog barked at the cat
3. The bird sang a beautiful song
...
Model trained on 10 sentences
Vocabulary size: 48
Generated text examples:
Starting with "the": the cat sat on a beautiful song
Starting with "a": a quick brown fox jumps over the lazy dog
🧪 Testing
Run the main script to see the model in action:
python bigram.py
This will:
Train on default sentences
Show model statistics
Generate example text with different starting words
Display bigram probabilities
Demonstrate save/load functionality
📈 Performance Characteristics
Training Time: O(n × m) where n = number of sentences, m = average sentence length
Memory Usage: O(|V|²) for bigram storage
Generation Speed: O(k × |V|) where k = number of words to generate
Vocabulary Scalability: Efficiently handles vocabularies up to 100K+ words
🔄 Legacy Compatibility
The original functional interface is still supported:
python
1# Original functions still work2sentences = prepare_Data()3bigram_probs = build_bigram_model(sentences)4text = generate_text(bigram_probs,"the",10)
🚧 Limitations
Context Window: Only considers the immediately preceding word
