Factorized Log-linear Models trained to recover lexical content (keywords) from
SONAR sentence embeddings, as
described in:
Trained on Mozilla Common Voice v15 (English), vocabulary of 100 000 unigrams.
FactLoLM scores every vocabulary word \(w\) for an input sentence embedding \(\mathbf{s} \in \mathbb{R}^{1024}\):
where \(\mathbf{E}_1 \in \mathbb{R}^{V \times r}\) and \(\mathbf{E}_2 \in \mathbb{R}^{r \times 1024}\) form a
rank- \(r\) factorization of the full projection matrix. Keywords are the top- \(n\) words by (logit) score.
1import json
2import torch
3from safetensors.torch import load_file
4
5# Load
6tensors = load_file("mcv15/rank-512/model.safetensors") # after hf download
7E1, E2, b = tensors["E1"], tensors["E2"], tensors["b"] # CPU tensors
8
9with open("mcv15/rank-512/vocab.json") as f:
10 vocab = json.load(f) # {word: index}
11int2word = {v: k for k, v in vocab.items()}
12
13def extract_keywords(embedding: torch.Tensor, topn: int = 10):
14 """embedding: (1024,) SONAR sentence embedding (speech or text)."""
15 scores = (E1 @ E2 @ embedding.unsqueeze(-1)).squeeze(-1) + b.squeeze(-1)
16 top_ids = scores.topk(topn).indices.tolist()
17 return [int2word[i] for i in top_ids]
1python scripts/evaluate.py \
2 --sdict mcv15/rank-512/model.safetensors \
3 --vocab mcv15/rank-512/vocab.json \
4 --data_yaml configs/datasets.yaml \
5 --dataset mcv_15_en_test \
6 --topn 10 --metrics all
1@misc{kesiraju2026flip,
2 title = {{FLiP}: Towards understanding and interpreting multimodal multilingual sentence embeddings},
3 author = {Kesiraju, Santosh and Yusuf, Bolaji and Sedl{\'{a}}{\v{c}}ek, {\v{S}}imon and Plchot, Old{\v{r}}ich and Schwarz, Petr},
4 year = {2026},
5 eprint = {2604.18109},
6 archivePrefix = {arXiv},
7 primaryClass = {cs.CL},
8 url = {https://arxiv.org/abs/2604.18109},
9}
