SwipeALot Base Model
[!IMPORTANT]
This model is currently in beta status and is subject to change.
Last updated 2025-12-19
Multimodal, multi-objective transformer for swipe keyboard prediction.
Trained on the
futo-org/swipe.futo.org dataset.
This model is trained with the following objectives:
- Masked character prediction (MLM)
- Masked path prediction
- Text length prediction (CLS token)
- Path/text embedding (SEP token, contrastive + Matryoshka@ 64, 128, 384, 768)
Quick Start (Length Prediction)
1from datasets import load_dataset
2from transformers import AutoModel, AutoProcessor
3
4
5model = AutoModel.from_pretrained("dleemiller/SwipeALot-base", trust_remote_code=True)
6model.eval()
7processor = AutoProcessor.from_pretrained("dleemiller/SwipeALot-base", trust_remote_code=True)
8
9# Load a sample row from the dataset.
10ds = load_dataset("futo-org/swipe.futo.org", split="test[:50]")
11row = ds[0] # "Brahmas"
12
13# Length-only inference:
14# `encode_path(...)` preprocesses the swipe path to fixed-length motion features and sets text attention to 0.
15inputs = processor.encode_path(row["data"], return_tensors="pt")
16outputs = model(**inputs, return_dict=True)
17
18# Length prediction is a regression scalar (float); round it for an integer length.
19pred_len = float(outputs.length_logits.item())
20pred_len_rounded = max(0, int(round(pred_len)))
21true_len = sum(1 for c in row["word"].lower() if c.isalpha() or c.isdigit())
22
23print(f'Word: "{row["word"]}"')
24print(f"Length (true): {true_len}")
25print(f"Length (pred): {pred_len:.3f}")
26print(f"Length (pred rounded):{pred_len_rounded}")
1Word: "Brahmas"
2Length (true): 7
3Length (pred): 7.483
4Length (pred rounded):7
Model Details
- Architecture: Transformer encoder (768-dim, 12 layers, 12 heads)
- Parameters: 87M
- Training Data: futo-org/swipe.futo.org dataset
- Max Path Length: 128 points (paths are interpolated down or padded up to this length)
- Max Word Length: 48 characters (words are truncated or padded to this length)
- Vocab Size: 43 (a-z, 0-9, special tokens)
Input Constraints:
- Path coordinates must be normalized to [0, 1] range for x, y
- Timestamps must be normalized to [0, 1] range
- Paths longer than 128 points are downsampled using linear interpolation
- Text longer than 48 characters is truncated with EOS preserved
Capabilities
1. Character Prediction
Predict characters from swipe paths with partial text context.
Trained via masked language modeling with a sophisticated pairwise masking strategy that creates two augmented views of each input for contrastive learning. Training uses focal loss to focus on hard-to-predict characters and frequency-based weighting to handle character imbalance (rare letters like 'z' vs common letters like 'e').
Pairwise Masking Strategy:
-
Inverted Mode (80%): Asymmetric augmentation pairs
- Query view: Heavy masking (50-70% of path points and characters randomly masked) with gradients
- Key view: Light masking (10-20% of path points and characters randomly masked) with stop gradient
- Teaches robust representations invariant to noise and occlusion
-
Modality Mode (20%): Cross-modal alignment pairs
- Query view: Text fully masked, path visible (teaches path → semantic representation) with gradients
- Key view: Path fully masked, text visible (provides alignment target) with stop gradient
- Teaches correspondence between path geometry and text meaning
2. Length Prediction
Predict word length from swipe path alone.
Trained as an auxiliary task where the CLS token aggregates path information to predict word length (0-48 characters). This helps the model learn geometric properties of swipe gestures that correlate with word length, such as path extent and complexity.
Length supervision occurs only during modality mode when text attention is fully zeroed (10% of training batches: 20% modality mode × 50% zero-attention probability). This trains the model to predict length from path geometry alone without any text length cues. Uses 10% of the total loss weight to encourage learning without dominating the primary objectives.
3. Path Reconstruction
Reconstruct missing path coordinates.
Trained via masked path prediction as part of the pairwise masking strategy. During inverted mode (80% of batches), path points are randomly masked at 50-70% for heavy augmentation and 10-20% for light augmentation. During modality mode (20% of batches), either all path points are masked (key view) or none are masked (query view). The model learns to reconstruct spatial-temporal structure from partial path information and text context, teaching it the geometric and temporal patterns of swipe gestures. Uses 50% of the character prediction loss weight, making it a significant secondary objective.
4. Embedding Extraction
Extract fixed-size embeddings for similarity search.
Dimension: 768
Trained via contrastive learning where the SEP token produces fixed-size embeddings for path-text pairs. The pairwise masking strategy is central to embedding training:
- Inverted mode (80%): Pulls embeddings of heavily-masked and lightly-masked versions of the same input close together, teaching invariance to noise and occlusion
- Modality mode (20%): Pulls embeddings of path-only and text-only views of the same word close together, teaching cross-modal alignment between gesture geometry and semantic meaning
The contrastive loss (10-20% weight, temperature 0.07) pulls matching pairs together in embedding space while pushing non-matches apart. Uses Matryoshka embeddings to create nested representations at multiple dimensions (64, 128, 384, 768), with stronger weight on lower-dimensional representations (2.0×, 1.5×, 1.0×, 1.0×) to ensure the first 64 dimensions are highly informative on their own.
More Usage Examples
Embedding Similarity
Modality attention masking adds a similar task to CLIP-style models.
The model can output vector representations of words or paths (or both), with high similarity.
1# Continuing from above (reuses `model` and `processor`):
2#
3# Goal: show matching via embeddings.
4# - "word-only" embedding: `processor.encode_text(...)` (equivalent to `text=...` + `path_coords=None`)
5# -> path attention is all zeros.
6# - "path-only" embedding: `processor.encode_path(...)` (equivalent to `path_coords=...` + `text=None`)
7# -> text attention is all zeros.
8#
9# We then compare cosine similarity:
10# sim(path(row0), word(row0)) should be higher than sim(path(row0), word(row1)).
11
12import numpy as np
13
14ds = load_dataset("futo-org/swipe.futo.org", split="test[:200]")
15row0 = ds[0] # "Brahmas"
16row1 = ds[7] # "central"
17
18word_inputs = processor.encode_text([row0["word"], row1["word"]], return_tensors="pt")
19word_out = model(**word_inputs, return_dict=True)
20word_emb = word_out.pooler_output.detach().cpu().numpy() # shape: [2, d_model]
21
22path_inputs = processor.encode_path(row0["data"], return_tensors="pt")
23path_out = model(**path_inputs, return_dict=True)
24path_emb0 = path_out.pooler_output.detach().cpu().numpy()[0] # shape: [d_model]
25
26def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
27 return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))
28
29sim_pos = cosine_similarity(path_emb0, word_emb[0])
30sim_neg = cosine_similarity(path_emb0, word_emb[1])
31
32print(f'Row0 word: "{row0["word"]}"')
33print(f'Row1 word: "{row1["word"]}"')
34print(f"Cosine similarity [positive]: {sim_pos:.4f}")
35print(f"Cosine similarity [negative]: {sim_neg:.4f}")
1Row0 word: "Brahmas"
2Row1 word: "central"
3Cosine similarity [positive]: 0.7927
4Cosine similarity [negative]: -0.0117
Word Reconstruction "Blind Reconstruction"
Here's how you can do a 2-step prediction, first predicting the word length (to get # of masks),
and then using mask prediction to fill the word.
1# Continuing from above (reuses `model` and `processor`):
2#
3# Word reconstruction (unknown length):
4# 1) Run path-only inference to predict the length.
5# 2) Create a text segment of `[MASK] * predicted_length + [EOS]`, enable text attention for it,
6# then reconstruct the characters from the path.
7
8tokenizer = processor.tokenizer
9
10ds = load_dataset("futo-org/swipe.futo.org", split="test[:50]")
11row = ds[0] # "Brahmas"
12
13inputs = processor.encode_path(row["data"], return_tensors="pt")
14
15pad_id = int(tokenizer.pad_token_id)
16mask_id = int(tokenizer.mask_token_id)
17eos_id = int(getattr(tokenizer, "eos_token_id", -1))
18
19pred_len = float(model(**inputs, return_dict=True).length_logits.item())
20pred_len_rounded = max(0, int(round(pred_len)))
21pred_len_rounded = min(pred_len_rounded, int(processor.max_char_len) - 1) # reserve 1 for EOS
22
23# Overwrite the padded text segment from `encode_path(...)` with `[MASK]... [EOS]`.
24inputs["input_ids"].fill_(pad_id)
25inputs["input_ids"][:, :pred_len_rounded].fill_(mask_id)
26inputs["input_ids"][:, pred_len_rounded].fill_(eos_id)
27
28# Enable text attention up to and including EOS.
29char_start = 1 + int(processor.max_path_len) + 1 # [CLS] + path + [SEP]
30inputs["attention_mask"][:, char_start:].fill_(0)
31inputs["attention_mask"][:, char_start : char_start + pred_len_rounded + 1].fill_(1)
32
33outputs = model(
34 **inputs,
35 return_dict=True,
36)
37
38pred_ids = outputs.char_logits.argmax(dim=-1)[0].detach().cpu().tolist()
39pred_word = tokenizer.decode(pred_ids[:pred_len_rounded]).strip().lower()
40
41print(f'Word: "{row["word"]}"')
42print(f'Reconstructed word: "{pred_word}"')
1Word: "Brahmas"
2Reconstructed word: "brahmas"
Performance Metrics
Evaluated on 49,970 test samples:
| Task | Metric | Score |
|---|
| Masked Prediction (30%) | Character Accuracy | 96.7% |
| Top-3 Accuracy | 99.7% |
| Word Accuracy | 94.4% |
| Full Reconstruction (100%) | Character Accuracy | 92.4% |
| Word Accuracy | 81.7% |
| Blind Reconstruction (2-pass) | Character Accuracy | 87.7% |
| Word Accuracy | 78.0% |
| Length Prediction | Exact Accuracy | 88.9% |
| Within ±1 | 99.2% |
| Within ±2 | 99.9% |
| Path Reconstruction | MSE (masked; dims=x/y) | 0.000033 |
Model Outputs
1outputs = model(**inputs)
2
3# Available outputs:
4outputs.char_logits # [batch, char_len, vocab_size] - Character predictions
5outputs.length_logits # [batch, 1] - Length predictions
6outputs.path_logits # [batch, path_len, 3] - Path coordinate predictions
7outputs.pooler_output # [batch, d_model] - SEP token embeddings for similarity
8outputs.last_hidden_state # [batch, seq_len, d_model] - Hidden representations
1char_len = 48
2path_len = 128
3seq_len = 178
4d_model = 768
Citation
1@software{swipealot2025,
2 title={SwipeALot: Multimodal Swipe Keyboard Transformer},
3 author={Lee Miller},
4 year={2025},
5 url={https://huggingface.co/dleemiller/SwipeALot-base}
6}
License
Apache 2.0
