You can deploy this model and start earning money today!

GoshKolotyan/
perovskite-formula-tokenizer

Name: perovskite-formula-tokenizer
Availability: InStock

Views

No views yet

Perovskite Chemical Formula Tokenizer

A BPE tokenizer specialized for perovskite chemical formulas with chemical-aware preprocessing that preserves element boundaries and handles fractional compositions.

Features

Chemical-aware tokenization with element boundary preservation
Support for 44 chemical elements including organic cations (MA, FA, DMA)
Handles fractional compositions and complex bracket notation
Special token conversion for brackets and decimal points

Usage

python

from tokenizers import Tokenizer

# Load tokenizer
tokenizer = Tokenizer.from_file("tokenizer.json")

# Simple formula
encoding = tokenizer.encode("(MA0.1FA0.9)PbCl3")
print(encoding.tokens)
# ['[LB]', 'MA', '0', '[DOT]', '1', 'FA', '0', '[DOT]', '9', '[RB]', 'Pb', 'Cl', '3']
# With special tokens (if enabled)
# ['[CLS]', '[LB]', 'MA', '0', '[DOT]', '1', 'FA', '0', '[DOT]', '9', '[RB]', 'Pb', 'Cl', '3', '[SEP]']

# Complex formula with fractional composition
encoding = tokenizer.encode("(DMA0.1FA0.9)Pb(Cl0.1Br0.9)3")
print(encoding.tokens)
#['[LB]', 'DMA', '0', '[DOT]', '1', 'FA', '0', '[DOT]', '9', '[RB]', 'Pb', '[LB]', 'Cl', '0', '[DOT]', '1', 'Br', '0', '[DOT]', '9', '[RB]', '3']

Model Specifications

Model Type: BPE (Byte-Pair Encoding)
Special Tokens: [PAD], [UNK], [CLS], [SEP], [MASK], [DOT], [LB], [RB]
Pre-tokenizer: Whitespace
Normalizer: NFD

Supported Elements (44)

Organic: MA, FA, DMA
Alkali: Li, Na, K, Rb, Cs
Alkaline Earth: Mg, Sr, Ba
Transition: Ti, Mn, Fe, Co, Ni, Cu, Zn, Y, Nb, Pd, Ag, Cd, La, Yb, Tb, Au, Hg
Post-transition: Ga, In, Sn, Tl, Pb, Bi
Metalloid: Ge, Sb, Te
Halogen: F, Cl, Br, I
Other: P, S, Se

Preprocessing Pipeline

Element identification and separation
Decimal splitting: 0.5 → 0 [DOT] 5
Bracket conversion: () → [LB] ... [RB]
BPE tokenization with special tokens

Known Limitations

Only recognizes pre-defined 44 elements
No support for charge states or isotope notation

License

MIT

GoshKolotyan/perovskite-formula-tokenizer