You can deploy this model and start earning money today!

parthvpatil18/
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab

Name: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab
Availability: InStock

pytorch

autoencoder

model-embedding

parameter-compression

Views

No views yet

ModelAutoEncoder: Pre-trained Model Parameter Embeddings

Model Description

ModelAutoEncoder is a specialized autoencoder designed to compress high-dimensional pre-trained model parameter vectors (8192-dim) into compact 512-dimensional embeddings that preserve essential architectural characteristics. This model is part of a research project on pre-trained model recommendation for specific hardware using transformers.

Key Features

18M parameter autoencoder with symmetric encoder-decoder architecture
Compresses model parameter statistics from 8192 dimensions to 512 dimensions (16x reduction)
GELU activations with dropout regularization for robust representations
Trained with Smooth L1 loss + L1 sparsity penalty for stability
Achieves 0.246 validation loss (Smooth L1 metric)
Fast inference: ~0.1ms per sample on V100 GPU
Throughput: 100,000 samples/second

Use Cases

This autoencoder enables:

Efficient Model Comparison: 512-dim embeddings enable fast similarity computation for model recommendation
Feature Learning: The bottleneck learns meaningful representations of model architecture patterns
Dimensionality Reduction: Reduces computational burden for downstream transformer models that predict model-dataset compatibility

Model Architecture

Input [batch, 8192]
  ↓ (z-score normalization)
  ↓ Linear(8192 → 1024) + bias
  ↓ GELU activation
  ↓ Dropout(p=0.05)
  ↓ Linear(1024 → 512) + bias  ← Bottleneck (encoded output)
  ↓ Linear(512 → 1024) + bias
  ↓ GELU activation
  ↓ Dropout(p=0.05)
  ↓ Linear(1024 → 8192) + bias  ← Reconstruction output

Architecture Details

Encoder: 8192 → 1024 → 512 (2 linear layers, 8.91M params)
Decoder: 512 → 1024 → 8192 (2 linear layers, 8.92M params)
Total Parameters: 17,836,544 (~17.84 million)
Activation Function: GELU (Gaussian Error Linear Unit)
Regularization: Dropout (p=0.05) + L1 penalty on latent code (0.001)
Loss Function: Smooth L1 Loss (Huber loss, β=0.5)

Performance Metrics

Training Performance

Metric	Value
Best Validation Loss	0.246083
Best Training Loss	0.262606
Best Epoch	4
Training Epochs	5 (early convergence)
Batch Size	16
Learning Rate	0.001 → 0.00001 (ReduceLROnPlateau)

Training Curve

Epoch 1: train_loss=0.439, val_loss=0.432
Epoch 2: train_loss=0.440, val_loss=0.363
Epoch 3: train_loss=0.359, val_loss=0.299
Epoch 4: train_loss=0.300, val_loss=0.246 ← Best
Epoch 5: train_loss=0.263, val_loss=0.253

The model converged rapidly with stable training dynamics, achieving best validation performance at epoch 4.

Computational Performance

Metric	Value
Model Size (FP32)	68.04 MB
Model Size (FP16)	34.02 MB
Inference Latency (single)	~0.1ms on V100 GPU
Inference Latency (batch=256)	~2ms on V100 GPU
Throughput	100,000 samples/second
Memory (inference)	~273 MB (model + optimizer states)

Usage

Installation

pip install torch transformers

Loading the Model

python

import torch
from transformers import AutoModel

# Load the model from HuggingFace Hub
model = AutoModel.from_pretrained("parthvpatil18/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab", trust_remote_code=True)
model.eval()

# Move to GPU if available
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device)

Basic Inference

python

import torch
import numpy as np

# Example: 8192-dim model parameter vector (normalized)
# IMPORTANT: Apply z-score normalization with training dataset statistics
model_params = torch.randn(1, 8192).to(device)  # Replace with your normalized data

# Extract 512-dim embedding
with torch.no_grad():
    encoded, reconstructed = model(model_params)

print(f"Input shape: {model_params.shape}")          # [1, 8192]
print(f"Embedding shape: {encoded.shape}")            # [1, 512]
print(f"Reconstruction shape: {reconstructed.shape}") # [1, 8192]

Batch Processing

python

# Process multiple models at once
batch_params = torch.randn(32, 8192).to(device)  # Batch of 32 models

with torch.no_grad():
    embeddings, reconstructions = model(batch_params)

# Use embeddings for downstream tasks (similarity search, clustering, etc.)

Integration with Full Pipeline

For a complete example of using this model in the pre-trained model recommendation pipeline, see the GitHub repository.

python

# Extract model parameters using specialized extractors
from extractors import HuggingFacePipelineExtractor

extractor = HuggingFacePipelineExtractor()
raw_params = extractor.extract("your-model-name")  # Returns 8192-dim vector

# Apply normalization (CRITICAL STEP)
# You must use mean/std from the training dataset
normalized_params = (raw_params - train_mean) / train_std

# Generate embedding
with torch.no_grad():
    embedding, _ = model(torch.tensor(normalized_params).unsqueeze(0).to(device))

# Use embedding for model recommendation
print(f"Model embedding: {embedding.shape}")  # [1, 512]

Input Requirements

IMPORTANT: Normalization is Required

The model requires z-score normalization of input data using statistics computed from the training dataset:

normalized_input = (raw_input - training_mean) / training_std

Failure to normalize inputs will result in poor performance or invalid embeddings.

Input Specification

Shape: [batch_size, 8192]
Data Type: torch.float32
Range: Unbounded (after normalization)
Source: Flattened model parameter statistics from specialized extractors
Preprocessing: Per-feature z-score normalization (critical)

Output Specification

The model returns a tuple (encoded, reconstructed):

encoded: Shape [batch_size, 512], dtype torch.float32
- Compact latent representation (the embedding)
- Unbounded activations, suitable for downstream similarity computation
reconstructed: Shape [batch_size, 8192], dtype torch.float32
- Reconstructed parameter vector (matches input distribution)
- Can be used to validate encoding quality

Training Details

Training Configuration

Optimizer: Adam (β₁=0.9, β₂=0.999)
Learning Rate: 0.001 initial, reduced to 0.00001 via ReduceLROnPlateau
Batch Size: 16
Epochs: 1500 (configured), early stopping at epoch 5
Weight Decay: 0.0001
Gradient Clipping: max_norm=1.0 (global norm)
Early Stopping: patience=200, min_delta=0.00001

Loss Function

Smooth L1 Loss (Huber Loss) + L1 Regularization:

total_loss = SmoothL1(reconstructed, input, β=0.5) + 0.001 × |encoded|.mean()

Smooth L1: Robust to outliers, prevents gradient explosion
L1 Penalty: Encourages sparsity in latent code (prevents trivial identity mapping)
β=0.5: Transition point between L1 and L2 regimes

Training Stability

The model exhibits stable gradient flow due to:

GELU activations (smooth gradients, no dead neurons)
Symmetric encoder-decoder architecture (balanced gradient magnitudes)
Smooth L1 loss (bounded gradients for large errors)
Gradient clipping (safety net against spikes)
Per-feature normalization (prevents feature domination)

Limitations and Constraints

1. Fixed Input Dimension

Encoder expects exactly 8192 features
If extractor output changes, model must be retrained from scratch
No support for variable-length inputs

2. Normalization Dependency

Critical: Must apply same normalization at inference as during training
Training dataset statistics (mean/std) must be stored and reused
Using raw (unnormalized) inputs produces invalid embeddings

3. Training Data Scarcity

Typical training set: 50-200 pre-trained models (small for 18M parameters)
High risk of overfitting without proper regularization
Best suited for model families seen during training

4. Architecture Coupling

Trained on parameter statistics from specific model families (ResNet, DenseNet, VGG, SigLIP)
Embeddings may not generalize to radically different architectures (e.g., transformers, graph neural networks)
Recommendation: Retrain or fine-tune when expanding to new model types

5. No Uncertainty Quantification

Produces point estimates (single 512-dim embedding)
Cannot detect out-of-distribution inputs (novel architectures)
Enhancement: Consider variational autoencoder (VAE) for probabilistic embeddings

Citation

If you use this model in your research, please cite:

bibtex

@misc{modelautoencoder2024,
  title={ModelAutoEncoder: Pre-trained Model Parameter Embeddings for Recommendation Systems},
  author={Parth Patil},
  year={2024},
  url={https://github.com/Parth1811/ptm-recommendation-with-transformers},
  howpublished={HuggingFace Hub: \url{https://huggingface.co/parthvpatil18/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab}}
}

Code Repository

Full source code, training scripts, and evaluation tools are available at:

https://github.com/Parth1811/ptm-recommendation-with-transformers

Repository Contents

model/model_encoder.py: AutoEncoder implementation
train/model_autoencoder_trainer.py: Training pipeline
dataloader/model_npz_dataset.py: Dataset loader for model parameters
extractors/: Tools for extracting parameter statistics from pre-trained models
config.ini: Full configuration file with hyperparameters

Running Training

bash

# Clone the repository
git clone https://github.com/Parth1811/ptm-recommendation-with-transformers.git
cd ptm-recommendation-with-transformers

# Install dependencies
pip install -r requirements.txt

# Train the autoencoder
python train.py ModelAutoEncoderTrainer

Extracting Model Embeddings

bash

# Extract embeddings for all models in a directory
python evaluate_model_autoencoder.py

Model Card Authors

Developed by: Parth Patil
Model Type: Autoencoder for model parameter embedding
Language(s): Python, PyTorch
License: MIT
Research Project: Pre-trained Model Recommendation with Transformers

Acknowledgments

This model is part of a research thesis project on pre-trained model recommendation for specific hardware using transformers. The system learns to recommend which pre-trained models will perform best on given datasets by extracting compact embeddings from model parameters and dataset features.

Contact

For questions, issues, or collaboration inquiries:

GitHub Issues: https://github.com/Parth1811/ptm-recommendation-with-transformers/issues
HuggingFace Model: parthvpatil18/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab

Last updated: November 2024