Views
No views yet
[model size]-[objective]-[number of steps].
Example: 610m-clm-42k denotes a 610M-parameter model trained with CLM for 42,000 steps.[model size]-[objective #1]-[steps #1]-[objective #2]-[total steps].
Example: 610m-clm-10k-mlm40-42k indicates a 610M model trained first with CLM for 10k steps, then continued with MLM (40% masking ratio) for 32k more steps, totaling 42k steps.610m-clm-dec42k-mlm40-64k refers to a 610M model pretrained with CLM for 42k steps (with weight decay), then further trained with MLM (40% masking) for 22k additional steps, totaling 64k.610m-mlm40-42k-1000 corresponds to step 1,000 during the MLM training phase of a 610M model trained for 42k steps.transformers library.1from transformers import AutoTokenizer, AutoModel
2import torch
3
4# Replace with the actual model ID if different, e.g., "AhmedAliHassan/MLMvsCLM-Biphasic-210M"
5# This placeholder assumes the current repository is the model you want to load.
6model_name = "<YOUR_MODEL_ID_HERE>"
7
8# Load the tokenizer and model, ensuring trust_remote_code for custom architectures
9tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
10model = AutoModel.from_pretrained(model_name, trust_remote_code=True)
11
12text = "This is an example sentence to extract features from."
13
14inputs = tokenizer(text, return_tensors="pt")
15
16with torch.no_grad():
17 outputs = model(**inputs)
18
19# The last hidden state contains the token embeddings (features)
20last_hidden_state = outputs.last_hidden_state
21print(f"Shape of last hidden state: {last_hidden_state.shape}")
22
23# For sentence-level embeddings, common approaches include:
24# 1. Averaging the token embeddings (excluding special tokens)
25# 2. Using the embedding of the [CLS] token (if applicable for the model's architecture)
26# Example: Mean pooling (simple average over non-padding tokens)
27attention_mask = inputs["attention_mask"]
28input_mask_expanded = attention_mask.unsqueeze(-1).expand(last_hidden_state.size()).float()
29sum_embeddings = torch.sum(last_hidden_state * input_mask_expanded, 1)
30sum_mask = torch.clamp(input_mask_expanded.sum(1), min=1e-9)
31mean_pooled_embedding = sum_embeddings / sum_mask
32print(f"Shape of mean pooled embedding: {mean_pooled_embedding.shape}")1@misc{gisserotboukhlef2025pretrainencodersmaskedlanguage,
2 title={Should We Still Pretrain Encoders with Masked Language Modeling?},
3 author={Hippolyte Gisserot-Boukhlef and Nicolas Boizard and Manuel Faysse and Duarte M. Alves and Emmanuel Malherbe and André F. T. Martins and Céline Hudelot and Pierre Colombo},
4 year={2025},
5 eprint={2507.00994},
6 archivePrefix={arXiv},
7 primaryClass={cs.CL},
8 url={https://arxiv.org/abs/2507.00994},
9}