A BERT-style transformer encoder (42.1M params, 8 layers, 512 hidden, trained from a custom BPE
tokenizer) pretrained with Masked Language Modeling and fine-tuned with SimCSE for dense
retrieval. Built for a retrieval-augmented operator chatbot — no LangChain or LlamaIndex.
Architecture, pretraining, and full RAG pipeline code:
uzbtrust/uzbek-operator-rag on GitHub.
Dense embeddings from this encoder are fused with TF-IDF sparse retrieval (score fusion, 0.5/0.5)
before generation. See the GitHub repo for the retriever and generator code.
The encoder class and BPE tokenizer live in the GitHub repo, not in this repository:
1git clone https://github.com/uzbtrust/uzbek-operator-rag
2cd uzbek-operator-rag
1import torch
2from model.transformer import Encoder # see repo for exact constructor args
3
4ckpt = torch.load("model.pt", map_location="cpu")
5model = Encoder(**ckpt["config"]["model"])
6model.load_state_dict(ckpt["model"])
7model.eval()
The tokenizer is trained with
tokenizer/train_tokenizer.py
in the GitHub repo; it is not bundled in this model repository.
MIT, matching the GitHub repository.
