import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

repo_id = "Bochkov/llm-fix-min-baseline-learned-input-table-model-classic"

tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True)
model.eval()

prompt = "Question: What is the capital of United Kingdom?\nAnswer:"
input_ids = torch.tensor([tokenizer.encode(prompt)], dtype=torch.long)

with torch.no_grad():
    output_ids = model.generate(input_ids, max_new_tokens=3, do_sample=False)

print(tokenizer.decode(output_ids[0].tolist()))

Limitations

This is a small research language model trained for architectural comparison. It is not instruction-tuned for safe deployment and should not be used as a production system.

Training data

The model was trained on the same FineWeb-Edu + Cosmopedia mixture used for the matched comparisons in the paper. Dataset terms and licenses are those of the original datasets.

🧑‍🔬 Citation & Concept

If you use this model or the underlying concepts in your research, please cite our work:

@misc{bochkov2026languagemodelstrainableinput,
      title={Language Models Without a Trainable Input Embedding Table: Learning from Fixed Minimal Binary Token Codes}, 
      author={A. Bochkov},
      year={2026},
      eprint={2605.09751},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2605.09751}, 
}