Views
No views yet
| Parameter | Value |
|---|---|
| Vocab Size | 151,646 |
| Hidden Size | 512 |
| Intermediate Size | 2,048 |
| Number of Layers | 6 |
| Attention Heads | 16 |
| Key-Value Heads | 8 |
| Max Sequence Length | 512 |
| Number of Experts | 4 |
| Top-k Experts | 2 |
1git clone https://github.com/modemneko/loftey.git
2cd loftey
3python -m venv venv
4source venv/bin/activate # On Windows use venv\Scripts\activate
5pip install -r requirements.txt| Model | Parameters | ModelScope | HuggingFace |
|---|---|---|---|
| Loftey Mini | 235M |  |  |
pip install modelscope1from modelscope import snapshot_download
2
3model_dir = snapshot_download('LightNeko/loftey-mini-235m')pip install huggingface_hub1from huggingface_hub import snapshot_download
2
3model_dir = snapshot_download(repo_id='Zeppesey/loftey-mini-235m')models/ directory of the project, or specify the correct model path in your code.1from transformers import AutoTokenizer
2from model import LofteyForCausalLM
3
4# Load model and tokenizer
5model = LofteyForCausalLM.from_pretrained('models')
6tokenizer = AutoTokenizer.from_pretrained('models')
7
8# Prepare input
9prompt = "<|im_start|>user\nHello, how are you?<|im_end|>\n<|im_start|>assistant\n"
10input_ids = tokenizer.encode(prompt, return_tensors='pt')
11
12# Generate response
13output = model.generate(
14 input_ids,
15 max_new_tokens=100,
16 temperature=0.7,
17 top_k=50,
18 top_p=0.95,
19 eos_token_id=tokenizer.eos_token_id
20)
21
22# Decode output
23response = tokenizer.decode(output[0], skip_special_tokens=True)
24print(response)python infer.py1from model import LofteyForCausalLM, Config
2
3# Load from directory
4model = LofteyForCausalLM.from_pretrained('path/to/model', device='cuda')
5
6# Or create with custom config
7config = Config(
8 hidden_size=768,
9 num_layers=12,
10 num_attention_heads=12,
11 use_moe=True,
12 expert_num=8
13)
14model = LofteyForCausalLM(config)1@software{loftey2026,
2 title={Loftey: A Lightweight MoE Language Model},
3 author={modemneko},
4 year={2026},
5 url={https://github.com/modemneko/loftey}
6}