Architecture: SheikhNextMoE - Sparse Mixture of Experts with grouped-query attention
Parameters: 576,079,872
Vocabulary Size: 30000
Hidden Dimension: 1024
Number of Layers: 16
Number of Experts: 4
Top-K Routing: 2
Attention Heads: 16

Training Configuration

Training Date: 2026-01-18
Training Samples: 1000
Validation Samples: 100
Epochs: 3
Final Training Loss: 5.5
Final Validation Loss: 5.5
Perplexity: N/A

Intended Use

This model is designed for medical question-answering applications, providing:

Accurate medical information responses
Safety-first approach with appropriate disclaimers
Support for various medical domains (cardiology, psychiatry, etc.)

Training Datasets

The model was trained on verified medical datasets:

medalpaca/medical_meadow_medical_flashcards - Medical flashcards and Q&A
ccdv/pubmed-summarization - Medical document summarization

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load model and tokenizer
model_name = "OsamaBinLikhon/SheikhNextMoE-Medical-v2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

# Medical query
prompt = "### Patient: What are symptoms of diabetes?
### Medical Assistant:"
inputs = tokenizer(prompt, return_tensors="pt")

# Generate response
outputs = model.generate(**inputs, max_new_tokens=100, temperature=0.7)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Safety Notice

⚠️ Important: This model provides medical information for educational purposes only. It is not a substitute for professional medical advice, diagnosis, or treatment. Always consult qualified healthcare providers for medical concerns.

Safety Guidelines

Model includes appropriate medical disclaimers
Responses are designed to encourage professional consultation
No diagnostic capabilities claimed
Trained on verified, ethical medical datasets

License

Apache 2.0 - See LICENSE file for details.

Citation

bibtex

@misc{SheikhNextMoE,
  title={SheikhNextMoE: Ethical Medical Assistant},
  author={MiniMax Agent},
  year={2025},
  url={https://huggingface.co/OsamaBinLikhon/SheikhNextMoE-Medical-v2}
}