IdioleX-ES is a sentence encoder trained under the IDIOLEX framework for idiolectal representation learning — capturing how text is expressed rather than what it says. Embeddings encode stylistic and dialectal variation across 17 Spanish varieties, decoupled from semantic content.

Kantharuban et al., IDIOLEX: Unified and Continuous Representations for Idiolectal and Stylistic Variation (preprint, under review). Code: github.com/AnjaliRuban/IdioleX

Architecture

input_ids → BERTIN (RoBERTa-base) → layer-wise attention → mean pool
          → mean centering → L2 normalize → embedding

Component	Detail
Base encoder	`bertin-project/bertin-roberta-base-spanish`
Pooling	Learnable layer-wise attention over all 13 hidden states (embedding + 12 transformer layers), then mean pool
Centering	Running-mean subtraction estimated over the Spanish training corpus
Output	L2-normalized vector, 768-dimensional

The scalar-mix weights are learned jointly with the encoder, following Rei et al. (2020).

Training

Data

Training data consists of Reddit comments from 17 regional Spanish-language subreddits, collected via the Pushshift archive through December 2024 and filtered for language and quality. Pre-training uses ~557k authors and ~49.5M sentences. Feature-supervised training uses 200 authors per dialect with LLM-annotated linguistic features.

Variety	Subreddit	Variety	Subreddit
Argentinian	r/argentina	Mexican	r/mexico
Bolivian	r/BOLIVIA	Panamanian	r/Panama
Chilean	r/chile	Paraguayan	r/Paraguay
Colombian	r/Colombia	Peruvian	r/PERU
Cuban	r/cuba	Peninsular	r/spain
Dominican	r/Dominican	Uruguayan	r/uruguay
Ecuadorian	r/ecuador	Venezuelan	r/vzla
El Salvadorian	r/ElSalvador
Guatemalan	r/guatemala
Honduran	r/Honduras

Linguistic Features

41 binary dialectal features are extracted sentence-by-sentence using GPT-5-mini, covering: subject pronoun use (yo, tú, usted, vos, vosotros, ustedes, etc.), verbal morphology (voseo present/imperative suffixes, 2pl forms -áis/-éis/-ís), diminutive suffixes (-ito/a, -ico/a, -illo/a, -ino/a, -ete, etc.), clitic patterns (DOM a, accusative/dative doubling, preverbal clitics, clitic sequences), and orthographic markers (inverted punctuation, all-caps, repeated punctuation).

Objectives

Training proceeds in two stages:

Stage 1 — Ranking pre-training (full dataset): A margin ranking loss encourages sentences with higher hierarchical proximity to be closer in embedding space. Each batch of 16 is structured so every sentence has exactly one same-comment neighbor (r=3), two same-author neighbors (r=2), four same-dialect neighbors (r=1), and eight cross-dialect neighbors (r=0). The margin λ warms up linearly from 0 to 0.5 over 25k steps.

Stage 2 — Feature-aware training (annotated subset, α=0.5):

Loss	Weight	Purpose
Margin ranking loss	1 − α = 0.5	Proximity-based ranking
Feature prediction BCE	0.25 × α = 0.125	Predict 41 linguistic features
Supervised contrastive (Jaccard-weighted)	α = 0.5	Feature-similarity alignment

A VICReg regularizer (weight 0.25) enforces variance ≥ 1 per dimension and decorrelates embedding dimensions to prevent anisotropy.

Hyperparameters

Parameter	Value
Base model	`bertin-project/bertin-roberta-base-spanish`
Hidden size	768
Transformer layers	12
Max sequence length	512
Batch size	32
Ranking group size	16
Feature vector dimension	41
Feature loss weight (α)	0.5
Contrastive temperature (τ)	0.07
Jaccard top-k	5
Learning rate	1 × 10⁻⁵
LR warmup	25k steps
Optimizer	Adam
Margin (λ)	0 → 0.5 (linear warmup)
Training GPUs	4
Max training time	≤ 48 hrs

Performance

Dialect Identification — DSL-ML 2024 (multi-label)

Model	F1	Exact Match
IdioleX-ES	0.85	0.62
Finetuned IdioleX-ES	0.84	0.63
Finetuned IdioleX-ES + Lexical	0.85	0.63
Finetuned BERT (baseline)	0.80	0.59
Centroid Clustering w/ BERT	0.77	0.57
Saleva & Palen-Michel, 2024 (shared task winner)	0.82	0.50

Authorship Attribution — PAN 2019 (open-set, cross-domain)

Model	Accuracy
IdioleX-ES	31%
Finetuned IdioleX-ES	36%
Finetuned IdioleX-ES + Lexical	38%
Finetuned BERT (baseline)	28%
Centroid Clustering w/ BERT	16%
Centroid Clustering w/ E5	27%

PAN 2019 is a cross-domain benchmark (test domain not seen during training), so these results demonstrate transfer of stylistic features beyond topical content.

Semantic Decoupling

Pearson correlation between IdioleX-ES idiolectal similarity scores and Multilingual-E5 semantic similarity scores on withheld Reddit test pairs: ρ = 0.09. Stylistic and semantic similarity are largely independent.

Usage

python

import torch
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained(
    "your-username/idiolex-bertin-spanish",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("your-username/idiolex-bertin-spanish")

sentences = [
    "Che, ¿me tirás una mano con esto?",   # Argentinian
    "Oye, ¿me echas una mano con esto?",   # Peninsular
]

inputs = tokenizer(
    sentences,
    return_tensors="pt",
    truncation=True,
    padding=True,
    max_length=model.config.model_len,  # 512
)

with torch.no_grad():
    embeddings = model(**inputs)   # [2, 768], L2-normalized

# Cosine similarity (embeddings are L2-normalized, so dot product = cosine sim)
similarity = embeddings @ embeddings.T
print(similarity)

Config

Parameter	Value
`base_model`	`bertin-project/bertin-roberta-base-spanish`
`embedding_dim`	`768`
`layerwise_pooling`	`True`
`num_layers`	`13` (12 transformer layers + embedding layer)
`layer_norm`	`False`
`layer_dropout`	`None`
`mean_center`	`True`
`model_len`	`512`

Custom files

This model uses trust_remote_code=True. The following files are hosted in this repo:

File	Purpose
`configuration_idiolex.py`	`IdioleXConfig`
`modeling_idiolex.py`	`IdioleXModel`
`centering.py`	`MeanCenterer` — distributed running-mean buffer
`layer_pool.py`	`LayerwiseAttention` — scalar-mix of transformer layers
`pooling_utils.py`	`last_token_pool`, `average_pool`

Citation

bibtex

@article{kantharuban2025idiolex,
  title   = {IDIOLEX: Unified and Continuous Representations for Idiolectal and Stylistic Variation},
  author  = {Kantharuban, Anjali and Srivastava, Aarohi and Faisal, Fahim and Ahia, Orevaoghene
             and Anastasopoulos, Antonios and Chiang, David and Tsvetkov, Yulia and Neubig, Graham},
  year    = {2025},
  note    = {Preprint, under review}
}

AnjaliRuban/idiolex-bertin-es

IdioleX-ES — Style-Aware Spanish Sentence Embeddings