You can deploy this model and start earning money today!

garetht/
identity-transformer

pytorch

identity-transformer

custom_code

Views

No views yet

Identity Transformer

A lightweight transformer implementation designed for testing tensor shapes and experimenting with transformer architecture.

Description

This model implements a minimal transformer that maintains input-output identity. It's useful for:

Understanding transformer tensor flows
Testing integration with other components
Educational purposes to see how transformers work
Debugging pipelines with a simple model

Model Details

Vocabulary size: 256 (compatible with the byte-level tokenizer)
Hidden size: 64
Number of layers: 1
Number of attention heads: 2
Intermediate size: 128
Activation function: GELU

Usage

python

from transformers import AutoTokenizer, AutoModel

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained("garetht/byte-tokenizer")
model = AutoModel.from_pretrained("garetht/identity-transformer")

# Encode text
text = "Hello, world!"
inputs = tokenizer(text, return_tensors="pt")

# Forward pass
outputs = model(**inputs)

# Process outputs
# outputs.last_hidden_state contains the hidden states

License

This project is open source and available under the MIT License.