You can deploy this model and start earning money today!

siddharth797/
fenrir-MoE-v1

Name: fenrir-MoE-v1
Availability: InStock

Views

No views yet

Fenrir‑MoE‑v1 (small)

A compact Mixture-of-Experts (MoE) language model based on the DeepSeek‑MoE architecture, fully compatible with the Hugging Face transformers library.

Model Details

Architecture: Transformer + MoE (shared + routed experts)
Hidden size: d_model = 512
Number of layers: 6
Attention heads: 8
Total routed experts: 16
Shared experts: 2
Top‑K routing: 4
Expert intermediate dimension: 512
Vocabulary size: 50257 (GPT‑2 tokenizer)
Context length: 256 tokens
Total parameters: ≈ 112 million (active per token ≈ 86 million)

This model is intended for research, fine‑tuning, or lightweight inference on CPU/GPU.

How to Use

Make sure you have transformers installed:

bash

pip install transformers torch


from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("siddharth797/fenrir-MoE-v1")
model = AutoModel.from_pretrained(
    "siddharth797/fenrir-MoE-v1",
    trust_remote_code=True
).to("cuda")



## Text Generation
### The repository includes a generation.py helper. Use it as follows:

from generation import generate_response, format_alpaca_prompt
formatted_prompt = format_alpaca_prompt(case["instruction"], case["input"])

response = generate_response(
    prompt_text=formatted_prompt,
    model=model,
    tokenizer=tokenizer,
    max_new_tokens=96,
    temperature=0.7
)





@article{deepseekmoe2024,
  title={DeepSeek-MoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models},
  author={DeepSeek-AI},
  year={2024}
}