A schema-aware tabular transformer pretrained on a large multi-source corpus
of real and synthetic tabular datasets.
Loss objective: multi-task MSE on target prediction from mixed numeric/categorical features,
normalized per-column (z-score). Each batch samples from a fixed-width (64-feature)
schema where unused slots are masked with NaN.
-
PMLB: fully exhausted. All 422 of 423 known datasets successfully processed
(1 download failure: chess). No new PMLB datasets can be added without an
upstream PMLB library update.
-
OpenML: largely exhausted. 4,886 unique datasets attempted. 2,949 passed
the pipeline. The 1,900 schema_fail entries are almost entirely datasets with
only 1 output column and too few rows/features to be useful (e.g. too small: (53, 1)).
These are unrecoverable without lowering quality thresholds. There may be a small
tail of undiscovered OpenML datasets not yet paginated.
-
HuggingFace tabular: 67 attempted from curated catalog. All failed due to
schema mismatches, missing splits, or download timeouts. Catalog needs expansion
with manually vetted datasets.
1import torch
2from tabula.models.transformer import TabularTransformer
3from tabula.config import ModelConfig
4
5# Load checkpoint
6ckpt = torch.load("best.pt", map_location="cpu", weights_only=False)
7cfg = ckpt["config"].model
8
9# Reconstruct model
10model = TabularTransformer(
11 d_model=cfg.d_model, n_heads=cfg.n_heads, n_layers=cfg.n_layers,
12 d_ff=cfg.d_ff, dropout=cfg.dropout,
13 num_numeric=64, num_categorical=0, num_text=0,
14 output_dim=1,
15 numeric_embedding=cfg.numeric_embedding,
16 numeric_periodic_features=cfg.numeric_periodic_features,
17 ffn_activation=cfg.ffn_activation, norm=cfg.norm, pooling=cfg.pooling,
18)
19model.load_state_dict(ckpt["model_state_dict"])
20model.eval()
The model uses a fixed-width schema (64 numeric slots) regardless of original
dataset width. Narrower datasets are zero-padded with NaN masks. This forces the
model to learn position-invariant feature representations compatible with arbitrary
tabular schemas.
Synthetic data fills gaps when real corpus buffer is empty, providing 100M+ rows
per session of controlled variation in feature distributions, missingness patterns,
and task types.
