Views
No views yet
⚠️ Pretraining degeneracy (audit 2026-05-18): empirical inspection shows this checkpoint's encoder is largely collapsed: pair-wise within-sequence hidden-state cosines hover at ≈ 0.999 and the MLM head returns nearly the same top-k tokens regardless of context. The model nominally achieved a low MLM eval_loss but appears to have settled on a degenerate "predict the most frequent token" strategy. Root cause traced to an under-sized BERT pretrain corpus (training_ready_hf_dataset≈ 4k rows vs ≈ 3.3M available inarrow_splits/). Not recommended for downstream use as-is; consider re-training fromarrow_splits/instead. (Note: the matching-largevariant exhibits an even more severe collapse and was therefore not uploaded.)
🚫 Inference is currently unusable on this checkpoint (audit 2026-06-22). Two compounding issues:
- Tokenizer / model id-space mismatch — the model was trained with a legacy hash-based
MinimalTokenizer(vocab_size ≈ 50000, padded to 50008 inconfig.json), but the tokenizer files saved alongside the checkpoint are the currentMoleculeNatLangTokenizerwrapping a standard GPT-2 BPE (vocab_size = 50257). Token ids produced by the saved tokenizer do not map to the same semantics the model was trained on, so anyAutoModel.forward(...)returns meaningless outputs.mask_token = None— the savedtokenizer_config.jsondoes not register a mask token, soAutoModelForMaskedLMinference that follows the conventionaltokenizer.mask_tokenworkflow raises immediately.The matching-mediumvariant does not have the id-space mismatch (its 50264 model vocab is the padded version of GPT-2's 50257). A retrained replacement for-smallis in the pipeline and will be pushed in a subsequent release. Do not use this checkpoint for inference or downstream fine-tuning in its current state.
-small checkpoint was originally trained with a legacy hash-based MinimalTokenizer (model embedding vocab_size=50008); the GPT-2 BPE tokenizer files currently bundled with the checkpoint do not match the model's id-space — see the warning above for the practical impact.1from transformers import AutoModelForMaskedLM, AutoTokenizer
2import torch
3
4model = AutoModelForMaskedLM.from_pretrained("kojima-lab/molcrawl-molecule-nat-lang-bert-small")
5tokenizer = AutoTokenizer.from_pretrained("kojima-lab/molcrawl-molecule-nat-lang-bert-small")
6
7# Predict masked token
8# Use tokenizer.mask_token instead of hardcoded "[MASK]":
9# BERT-style tokenizers vary ("[MASK]", "<mask>", etc.)
10if tokenizer.mask_token is None:
11 raise ValueError("This tokenizer has no mask_token; masked LM inference is not supported.")
12prompt = "your input {MASK} sequence".replace("{MASK}", tokenizer.mask_token)
13inputs = tokenizer(prompt, return_tensors="pt")
14mask_index = (inputs["input_ids"] == tokenizer.mask_token_id).nonzero(as_tuple=True)[1]
15
16with torch.no_grad():
17 outputs = model(**inputs)
18logits = outputs.logits
19
20predicted_token_id = logits[0, mask_index].argmax(dim=-1)
21predicted_token = tokenizer.decode(predicted_token_id)
22result = prompt.replace(tokenizer.mask_token, predicted_token)
23print(f"Predicted: {result}")
241@misc{molcrawl_molecule_nat_lang_bert_small,
2 title={molcrawl-molecule-nat-lang-bert-small},
3 author={{RIKEN}},
4 year={2026},
5 publisher={{Hugging Face}},
6 url={{https://huggingface.co/kojima-lab/molcrawl-molecule-nat-lang-bert-small}}
7}