Views
No views yet
1language: en
2license: apache-2.0
3tags:
4- moe
5- mixture-of-experts
6- transformer
7- untrained
8- base-model
9- llm
10- language-model
11- text-generation
12- 293M
13- houndtid
14- wesamoyo
15base_model: false
16inference: false
17pipeline_tag: text-generation
18company: Houndtid Labs
19author: Houndtid Labs AI Research Team
20contact: houndtidai@gmail.com
21---
22
23# Wesamoyo-293M-MoE
24
25**293 Million Parameter MoE Transformer Architecture**
26*Efficient Mixture-of-Experts Foundation Model*
27
28## Overview
29
30Wesamoyo-293M-MoE is a 293-million parameter Mixture-of-Experts transformer architecture. The model ships with **initialized weights** - ready for training from scratch or fine-tuning.
31
32### Key Architecture Features
33- **Mixture-of-Experts**: 64 total experts, 6 activated per token
34- **MLA Attention**: Multi-head latent attention
35- **Extended Context**: 16,384 token sequence length
36- **BF16 Precision**: Optimized for training
37- **Custom Architecture**: Proprietary transformer design
38
39---
40
41## 🚀 Official SDK Installation
42
43Install the official Wesamoyo SDK for seamless model loading:
44
45```bash
46pip install wesamoyo1import wesamoyo
2
3# One-line loading with the official SDK
4model = wesamoyo.load_wesamoyo_293m()
5
6print(f"✅ SDK Loaded: {sum(p.numel() for p in model.parameters()):,} parameters")1from wesamoyo import WesamoyoForCausalLM, WesamoyoConfig
2
3# Create config for 293M model
4config = WesamoyoConfig(
5 vocab_size=16384,
6 dim=512,
7 n_layers=6,
8 n_heads=8,
9 n_routed_experts=64,
10 max_seq_len=16384
11)
12
13# Load model with SDK
14model = WesamoyoForCausalLM(config)
15print("✅ Model loaded and ready for training")1# The SDK automatically registers with Hugging Face
2from transformers import AutoModelForCausalLM
3
4model = AutoModelForCausalLM.from_pretrained("HoundtidLabs/wesamoyo-293M-MoE")
5print("✅ Loaded via Hugging Face with SDK integration")| Component | Specification | Description |
|---|---|---|
| Model Type | MoE Transformer | Mixture-of-Experts design |
| Total Parameters | 293 Million | Architecture capacity |
| Experts | 64 total, 6 active | MoE routing configuration |
| Context Window | 16,384 tokens | Extended sequence processing |
| Transformer Blocks | 6 layers | Network depth |
| Hidden Dimension | 512 | Feature representation size |
| Attention Heads | 8 | Parallel attention computation |
| Vocabulary Size | 16,384 tokens | Token dictionary |
| Precision Support | BF16 | Training optimization |
| File Size | 591 MB | Complete weights file |
| SDK Version | wesamoyo 1.0.4 | Official loading package |
1import wesamoyo
2import torch
3import torch.nn.functional as F
4
5# Load model using official SDK
6model = wesamoyo.load_wesamoyo_293m()
7model = model.cuda()
8
9# Setup training
10optimizer = torch.optim.AdamW(model.parameters(), lr=1e-4)
11
12# Training loop
13for epoch in range(3):
14 for batch in dataloader:
15 inputs = batch["input_ids"].cuda()
16 targets = batch["labels"].cuda()
17
18 # Forward pass with SDK-loaded model
19 outputs = model(inputs)
20 loss = F.cross_entropy(
21 outputs.logits.view(-1, model.config.vocab_size),
22 targets.view(-1)
23 )
24
25 # Backward pass
26 loss.backward()
27 optimizer.step()
28 optimizer.zero_grad()
29
30 print(f"Epoch {epoch}, Loss: {loss.item():.4f}")
31
32# Save trained model
33torch.save(model.state_dict(), "trained-wesamoyo-293M.pth")pip install wesamoyo # Get the official loading packagewesamoyo SDK for seamless loading and training experience.