AgGPT-14 is a lightweight, Python-based AI model designed for conversational tasks with context-aware responses. It combines n-gram style Markov chains with a similarity-driven context selection mechanism, providing coherent and human-like responses based on a training corpus.
Features
Deterministic Context Matching
Uses an aggressive TF-IDF inspired similarity scoring combined with Longest Common Subsequence (LCS) detection to find the best matching user query from the training corpus.
Ensures responses are relevant to the user's input.
Ensures responses are relevant to the user's input.
World Model Integration
Generates simple "world model" text to enhance conversational depth when enabled.
Can prepend or combine world model outputs with AI responses.
N-Gram AI Response Generation
Generates responses using an n-gram Markov model (configurable order) built from AI responses in the training corpus.
Supports temperature-based sampling and top-k filtering for diverse outputs.
Text Normalization
Expands common contractions.
Tokenizes text into clean, lowercase tokens.
Detokenizes output with proper punctuation and capitalization.
IDF-Weighted Matching
Emphasizes rare words in similarity scoring to capture nuanced user queries.
Debugging Support
Provides detailed debug information about tokenization, similarity scores, and context selection for each user query.
Notes
Designed for offline usage; no external API calls required.
Lightweight and fast; ideal for experimentation and educational purposes.
Can be easily extended with more advanced NLP techniques for higher-quality responses.
We noted that this model does not perform as well as traditional transformer-based models like GPT-3.5 or GPT-4, but it is designed to be lightweight. We also noted that this model is not as scalable, so further research and development is needed to improve its performance and scalability to match AgGPT-9 and AgGPT-10 performance, which is inherently more scalable but also more complex and resource-intensive, as it is a full transformer model. It is my goal to create a model that is lightweight, fast, and easy to use, while still providing high-quality responses, but also to make the model not be a black box, like most GPT models, so that it can be easily understood and modified by developers and researchers.
License
This project is licensed under the MIT License - see the LICENSE file for details.
