Hypothesis: KAN implementations can potentially achieve a smaller parameter count compared to MLPs, especially when leveraging efficient analog circuits for approximating KAN activation functions (sine, cosine, ln, log).
Motivation:
Reducing the parameter count of large language models like GPT can lead to significant memory savings and potentially faster inference.
Analog circuits, particularly those specialized for functions like sine and cosine (e.g., Mythic and IBM analog MatMul circuits), offer a promising avenue for low-power computation of KAN activations.
Roadmap:
Reverse KAN-to-MLP Algorithm: Develop a script that analyzes the weights and biases of a pretrained MLP and approximates them using a combination of KAN activation functions. This process can be seen as analogous to Fourier Transform Analysis, where a signal is decomposed into its constituent frequencies (represented by sine and cosine waves in this context).
Analog Circuit Integration: Design and implement analog/mixed-signal circuits to efficiently compute the KAN activation functions (sine, cosine, ln, log) from the input values. This step aims to leverage the inherent energy efficiency of analog computation.
Layer-by-Layer Substitution: Replace MLPs within the pretrained GPT model with the equivalent KAN networks, starting with one layer at a time.
Evaluation and Fine-tuning: Thoroughly evaluate the performance of the compressed GPT model and fine-tune it [or use adapters or low-rank parameter-efficient fine tuning PEFT] to recover any potential performance loss due to the approximation.
Preliminary Work:
Initial Python scripts have been developed to demonstrate the conversion between KAN and MLP layers:
[KAN is just MLP.ipynb](KAN is just MLP.ipynb - Colab.pdf)
[Python script converting KAN to MLP](code snippet for converting KAN to MLP)
Future Directions:
Develop the reverse KAN-to-MLP algorithm and implement it in Python.
Explore and compare various analog circuit designs for efficient KAN activation computation.
Conduct extensive experiments to measure the compression ratio, performance, and energy efficiency of KAN-based GPT models.
Contribution: Google Gemini 1.5 Pro has been instrumental in developing the initial draft Python scripts.
[My hand-typed-Draft, with prompt" "Rewrite and correct this model card in huggingface:" ]
Based on the paper at https://colab.research.google.com/drive/1v3AHz5J3gk-vu4biESubJdOsUheycJNz#scrollTo=-KtMq2NYAy57
and the discussion at https://www.reddit.com/r/MachineLearning/comments/1clcu5i/d_kolmogorovarnold_network_is_just_an_mlp/
I propose further comrpessing GPTs (such as https://huggingface.co/microsoft/phi-1_5 ) by converting their pretrained MLPs (multilevel pereceptrons) into equivalent KAN networks
based on an algorthym that reads the weights and biases in the pretrained MLP and approximates the MLP by use of analog functions
within a KAN network (the eplacement layer). Supposedly, and intuitively, a KAN implementation can have fewer variable parameters than a MLP.
But, this assumes that there is an effecient manner of generating the output of the KAN activation functions (sine, cosine, ln, Log).
I will assume that the KAN functions can be selected based on their being implemented using analog circuits. (see e.g., Mythic and IBBM analog MatMul circuits).
Here is a roadmap for further compressing pretrained GPTs including MLPs (multilevel perceptrons):
Write a reverse KAN-to-MLP algorthym script. (This will be similar to a Fourier Transform Analysis: Given a MLP having a set of weights and Bias, derive a set of functions (e.g., sine waves, cosine waves, known to KAN networks) that represents the original MLP parameters.
Use analog/mixed circuits to convert level inputs to the corresponding (KAN) sine(x) and (Cosine(x) values, to reduce power consumption.
Input the resulting values to the next GPT level, and repeat MLP to KAN approximation.
Google Gemini 1.5 Pro helped me develope the following draft python scripts to perform this layer-by-layer substitution/compression:
KAN is just MLP.ipynb - Colab.pdf
805 Tokens
Write a python script that converts KAN to MLPs based on this tutorial
Model
class KANtoMLP(nn.Module):
def init(self, in_features, hidden_features, out_features, grid_size):
super(KANtoMLP, self).init()
# Input expansion is implicitly handled in the first KANLayer
self.layers = nn.ModuleList([
KANLayer(in_features, hidden_features, grid_size),
KANLayer(hidden_features, out_features, grid_size)
])
def forward(self, x):
for layer in self.layers:
x = layer(x)
return x
KANLayer Class:
This class implements a single layer of the KAN network, converting it to an equivalent MLP structure.
init: Initializes the linear layer (self.linear) and the shifts (self.shifts) for the piecewise linear function.
forward:
Expands the input (repeated) and adds shifts (shifted).
Applies ReLU to shifted values (except the first one) and concatenates with the first shifted value.
Flattens the intermediate result and passes it through the linear layer.
KANtoMLP Class:
This class represents the overall network, stacking multiple KANLayer instances.
init: Initializes a list of KANLayer objects (self.layers).
forward: Iterates through the layers, applying each one to the input sequentially.
Example Usage:
Creates an instance of KANtoMLP with specified input, hidden, output features, and grid size.
Generates random input data.
Performs a forward pass through the network.
Prints the shape of the output tensor.
This script demonstrates the conversion of a two-layer KAN network to its MLP equivalent. The number of layers and their configurations can be adjusted to match the desired architecture.
User
Now, write a Python Script to convert the weights and biases in a MLP in one level of a pretrained GPT model into an equivalent KAN layer,
for functioning substitution of the KAN layer into the pretrained GPT model.
Model
import torch
import torch.nn as nn
def mlp_to_kan_layer(mlp_layer, grid_size):
"""
Converts a single MLP layer (Linear-ReLU-Linear) to an equivalent KAN layer.
Args:
mlp_layer: The MLP layer to convert (nn.Sequential containing Linear-ReLU-Linear).
grid_size: The desired grid size for the piecewise linear function in the KAN layer.
Returns:
A KANLayer object with weights and biases initialized from the MLP layer.
"""
linear1, relu, linear2 = mlp_layer[0], mlp_layer[1], mlp_layer[2]
in_features = linear1.in_features
out_features = linear2.out_features
# Initialize KAN layer
kan_layer = KANLayer(in_features, out_features, grid_size)
# Transfer weights and biases
with torch.no_grad():
# Linear1 weights are split across the grid points in KAN
kan_layer.linear.weight.data[:, :in_features] = linear1.weight.data
for i in range(1, grid_size):
kan_layer.linear.weight.data[:, i * in_features:(i + 1) * in_features] = linear1.weight.data
# Linear2 weights are directly transferred to KAN
kan_layer.linear.bias.data = linear2.bias.data
return kan_layer
class KANLayer(nn.Module):
# ... (Same as in previous response)
Example Usage:
Assuming 'gpt_model' is your pretrained GPT model
Select the MLP layer to convert (e.g., from a specific transformer block)
mlp_layer = gpt_model.transformer.h[0].mlp # Replace with the actual path
mlp_to_kan_layer Function:
Takes an mlp_layer (assumed to be a nn.Sequential with Linear-ReLU-Linear) and grid_size as input.
Extracts the individual linear layers and ReLU activation from mlp_layer.
Creates a KANLayer instance with the appropriate input and output dimensions.
Weight Transfer:
The weights of the first linear layer (linear1) in the MLP are split and copied to the kan_layer.linear.weight across the grid points.
The weights of the second linear layer (linear2) are directly copied to the kan_layer.linear.weight.
Bias Transfer:
The bias of the second linear layer (linear2) is copied to kan_layer.linear.bias.
Returns the initialized kan_layer.
Example Usage:
Assumes you have a pretrained GPT model (gpt_model).
You need to identify the specific MLP layer you want to replace. The example shows a placeholder path; you should replace it with the actual path to your target MLP layer within the GPT model.
The mlp_to_kan_layer function is called to convert the chosen MLP layer to a KAN layer.
Finally, the original MLP layer within the GPT model is replaced with the newly created kan_layer.
Important Considerations:
Grid Size: The choice of grid_size will influence the representational capacity of the KAN layer.
Model Behavior: After replacing the MLP layer with a KAN layer, you should carefully evaluate the performance of the modified GPT model to ensure that the substitution does not negatively impact its behavior. You may need to fine-tune the model after the modification.
########### EXAMPLE IMPLEMENTATION FOR GPT-2 ##################
Based on Quantization-Aware pretraining methods, write a python script that performs updating of wights and bias in a GPT-2 124M model
with backpropagation with the additional constraint that weights and bias updates are steered not only based on gradient descent but
are also steered in a manner that tend to reduce the MLP-to-KAN conversion error. Thus, perform KAN-aware pretraining, and simultaneously generate a parameter-efficient PeFT adapter to correct the MLP-to-KAN error.
Model
import torch
import torch.nn as nn
from transformers import GPT2LMHeadModel, GPT2Tokenizer
Assuming you have the mlp_to_kan_layer function defined
from previous responses
Load pretrained GPT-2 model and tokenizer
model_name = 'gpt2'
model = GPT2LMHeadModel.from_pretrained(model_name)
tokenizer = GPT2Tokenizer.from_pretrained(model_name)
Define KAN grid size
grid_size = 5
Create a list to store PeFT adapters for each layer
peft_adapters = []
Modify GPT-2 layers for KAN-aware pretraining
for i, layer in enumerate(model.transformer.h):
# Convert MLP to KAN
kan_layer = mlp_to_kan_layer(layer.mlp, grid_size)
layer.mlp = kan_layer
# Create a PeFT adapter for this layer
adapter = nn.Sequential(
nn.Linear(kan_layer.out_features, kan_layer.out_features),
nn.ReLU()
)
peft_adapters.append(adapter)
# Add adapter to the model (you might need to adjust
# how you integrate it into the forward pass)
layer.mlp = nn.Sequential(kan_layer, adapter)
def train(model, optimizer, loss_fn, epochs=10):
for epoch in range(epochs):
model.train()
for batch in train_dataloader: # Assuming you have a train_dataloader
input_ids = batch['input_ids'].to(device)
attention_mask = batch['attention_mask'].to(device)
labels = batch['labels'].to(device)
optimizer.zero_grad()
# Forward pass through the modified GPT-2
outputs = model(input_ids=input_ids, attention_mask=attention_mask, labels=labels)
loss = outputs.loss
# Backward pass and calculate gradients
loss.backward()
# Custom gradient modification for KAN-awareness:
# 1. Calculate MLP-to-KAN conversion error (You need to define this function)
# conversion_error = calculate_mlp_to_kan_error(kan_layer, original_mlp_layer)
# 2. Add a term to the gradients of the KAN layer weights
# that minimizes conversion_error
# Update model parameters
optimizer.step()
Explanation:
Model Loading and Modification:
Loads a pretrained GPT-2 model.
Iterates through the transformer layers, converting each MLP to a KAN layer using the mlp_to_kan_layer function.
Creates a PeFT adapter (a small neural network) for each layer to correct for potential MLP-to-KAN conversion errors.
Integrates the KAN layers and adapters into the GPT-2 model.
Training Loop:
Implements a standard training loop with a defined loss function and optimizer.
Key Addition: In the backward pass, you need to calculate the MLP-to-KAN conversion error. This will involve comparing the output of the original MLP layer with the output of the KAN layer using the same input.
Modify the gradients of the KAN layer weights by adding a term that encourages minimizing the conversion error. The exact implementation of this gradient modification will depend on how you define the conversion error and the desired balance between gradient descent and error minimization.
PeFT Adapter Training:
The PeFT adapters are trained simultaneously with the KAN layers during the backpropagation process. Their role is to learn and correct any discrepancies between the original MLP behavior and the KAN layer's output.
Saving the Model:
Saves the fine-tuned model, which now incorporates the KAN layers and PeFT adapters, allowing for efficient and accurate inference.
Crucial Implementation Details:
calculate_mlp_to_kan_error: You need to define a function to measure the difference between the outputs of the original MLP layer and the KAN layer for a given input. The choice of error metric (e.g., MSE, KL-divergence) will impact the training process.
Gradient Modification: The exact way you modify the gradients to incorporate the KAN conversion error will be crucial to the success of the KAN-aware pretraining. You need to balance the influence of the original gradient descent with the pressure to minimize conversion error.
This script outlines the general structure and key considerations for KAN-aware pretraining. The specific implementations of error calculation and gradient modification will require careful design and experimentation to achieve optimal results.
license: Experimental and Academic Implementations are allowed. Commercial implementations and uses are subject to Royalty and Patent Rights of the Inventor Martial Terran.
License and Restrictions on Use
This project and the associated code, algorithms, and concepts are subject to the following terms and conditions:
Experimental and Academic Use:
You are granted a non-exclusive, royalty-free license to use, modify, and distribute this project solely for experimental and academic purposes. This includes, but is not limited to:
Research and development.
Educational activities.
Personal, non-commercial projects.
Commercial Use:
Any commercial use of this project, including but not limited to:
Incorporation into commercial products or services.
Distribution or sale for profit.
Use in any revenue-generating activity.
is strictly prohibited without the express written permission of the inventor, Martial Terran, and is subject to royalty payments and the enforcement of all applicable patent rights held by Martial Terran.
Patent Rights:
Martial Terran retains all patent rights, including but not limited to any patents issued or pending, relating to the concepts, algorithms, and implementations described in this project.
No Warranty:
This project is provided "as is" without warranty of any kind, express or implied, including but not limited to the warranties of merchantability, fitness for a particular purpose, and non-infringement. In no event shall the inventor be liable for any claim, damages, or other liability arising from, out of, or in connection with this project.
Contact for Commercial Licensing:
To inquire about commercial licensing opportunities, please contact Martial Terran at [mrferran1970@gmail.com].
Attribution:
All use of this project must include appropriate attribution to the inventor, Martial Terran.
Governing Law:
This agreement shall be governed by and construed in accordance with the laws of [New York].
