A 124M parameter transformer that challenges a 56-year-old assumption in neural network design.
WiggleGPT Architecture
What Makes It Different?
Since Minsky and Papert's Perceptrons (1969), neural networks have relied on monotonic activation functions (Sigmoid, ReLU, GELU) — requiring multiple hidden layers to solve non-linearly separable problems like XOR.
WiggleGPT replaces monotonic activations with learnable oscillating functions, enabling single neurons to create multiple decision boundaries:
f(x) = sin(ωx + φ) · tanh(x) + baseline
Where ω (frequency) and φ (phase) are learnable per-neuron parameters.
Results
Model
Parameters
Val Loss
Notes
WiggleGPT
124M
3.1621
Oscillating activation
GPT-2
124M
~3.12
Standard GELU baseline
Within 1.3% of GPT-2 performance — proving oscillating activations are a functional drop-in replacement at scale.
The Model Actually Learned to Oscillate
Parameter
Init
After Training
Change
ω mean
1.0
1.096
+9.6%
ω std
0.1
0.602
6× increase
ω range
[0.8, 1.2]
[-0.19, 5.17]
Massive expansion
95% of neurons retained active oscillation (ω > 0.1)
Some neurons learned frequencies up to ω = 5.17 (five oscillations per unit input)
Full phase coverage [-π, +π] after training
Checkpoints
File
Description
ckpt_pretrain.pt
Base model trained on OpenWebText (~600k iterations)
ckpt_finetune.pt
Fine-tuned on SmolTalk2 (instruction following)
Architecture
Component
Specification
Parameters
123,697,920
Layers
12
Attention Heads
12
Embedding Dimension
768
Oscillating Neurons
36,864 (each with learnable ω, φ, baseline)
Normalization
RMSNorm
Position Encoding
RoPE (Rotary)
Attention
Flash Attention (when available)
Usage
See the GitHub repository for full training, inference, and chat scripts.
python
1# Quick inference example2import torch
3from model_bio import GPT, GPTConfig
45# Load checkpoint6checkpoint = torch.load('ckpt_pretrain.pt', map_location='cuda')7config = GPTConfig(**checkpoint['config'])8model = GPT(config)9model.load_state_dict(checkpoint['model'])10model.eval()1112# Generate text (see sample_bio.py for full implementation)
