TxCat-Nano
A 15,434-parameter transformer for transaction categorization, trained from scratch in raw PyTorch. No HF Trainer, no pretrained weights, no transfer learning.
Built as the smallest honest transformer experiment: how far can a stripped-down transformer go on a small transaction dataset, and what does the accuracy-versus-memory frontier actually look like?
Trained on a Kaggle P100 (torch 2.4.0+cu121), 25 epochs, about 8 seconds.
Model
| Param | Value |
|---|
| Parameters | 15,434 |
| Layers / heads / d_model | 1 / 2 / 32 |
| FF dim | 64 |
| Context length | 32 tokens |
| Vocabulary | 45 (char-level, built from train split only) |
| Amount encoding | log-scaled into 64 buckets |
Inputs: merchant/description text (char-level) plus transaction amount. Token and amount embeddings are injected into a CLS token, fed through one nn.TransformerEncoder layer, then a linear head over 10 classes.
Results (1,000 held-out rows, stratified 80/20, seed 42)
| Metric | Value |
|---|
| Accuracy | 0.9610 |
| Macro-F1 | 0.9609 |
| Weighted-F1 | 0.9609 |
| ECE (calibration) | 0.0276 |
| Log-loss | 0.1487 |
| Brier | 0.0065 |
| Top-2 accuracy | 0.9880 |
| Inference | 1.06 ms/sample, single CPU core |
The model is slightly under-confident (mean confidence 0.934 vs accuracy 0.961), which is the opposite of the usual small-model failure mode.
Baselines and the frontier
| Model | Params | Size | Held-out acc |
|---|
| TxCat-Nano (this) | 15,434 | 60 KB | 0.9610 |
| TxCat-Micro | 80,714 | 315 KB | 0.9960 |
| TxCat-Mini | 292,490 | 1.1 MB | 0.9970 |
| TF-IDF + Logistic Regression | — | 15,939 features | 1.0000 |
| Majority baseline | — | — | 0.1000 |
On this dataset a linear model on TF-IDF features beats every transformer. The transformers' result is the size frontier: Nano is 28,185x smaller than DeBERTa-large (1,659 MB) at 96% of its accuracy, with better calibration. The comparison is reported because it is the honest one.
Usage
1import torch, json
2from huggingface_hub import hf_hub_download
3
4bin_path = hf_hub_download("vivekkopthsd/txcat-tiny-transformer-nano", "pytorch_model.bin")
5cfg = json.load(open(hf_hub_download("vivekkopthsd/txcat-tiny-transformer-nano", "config.json")))
6sd = torch.load(bin_path, map_location="cpu")
7
8# Build the TinyTransformer (class definition in the companion Kaggle kernel),
9# load_state_dict(sd), tokenize merchant text + amount, forward, argmax.
Files
| File | Contents |
|---|
pytorch_model.bin | State dict (68 KB) |
config.json | Architecture, tokenizer, training config, all metrics above |
vocab.json | 45-char vocab |
Reproducibility
- Data:
bipin2/transaction-categorization on Kaggle (10 classes, 5,000 samples; 4,000 train / 1,000 val).
- Training kernel:
vivekkopthsd/txcat-tiny-transformer on Kaggle. Includes micro/mini variants, TF-IDF baseline, majority baseline, and the frontier plot.
- Weight persistence kernel:
vivekkopthsd/txcat-nano-persist. The RNG stream was replicated from the original kernel so the persisted weights reproduce the exact 0.9610 run (a first attempt with a different init stream gave 0.9490).
- Hardware: P100, torch 2.4.0+cu121. The Kaggle default image ships cu128 builds with no Pascal kernels; the pin is required.
History
Three failures were debugged during development, each documented in the training kernel log:
- cu128 torch build on P100:
CUDA error: no kernel image is available for execution on the device. cuda.is_available() returned True; the error surfaced at the first operation. Fixed by pinning torch 2.4.0+cu121 and adding a CUDA smoke test.
- Initial pip install without internet access failed at DNS.
- A cosmetic cell divided by the TF-IDF baseline's size (0 KB) and crashed the report step; baselines were excluded from the "smallest trained transformer" logic.
Known limitation
The dataset reuses merchant names across rows, so every validation merchant also appears in training. Part of the 0.961 accuracy reflects merchant-name memorization rather than pure text understanding. A merchant-disjoint split would report a lower, more honest generalization number; it is the planned next step.
License
Weights trained from scratch on the public Kaggle dataset bipin2/transaction-categorization (see its dataset page for terms). Educational use.
