You can deploy this model and start earning money today!

Minute.MoE.NSFW-1.3B

Name: Minute.MoE.NSFW-1.3B
Availability: InStock

WithinUsAI Logo

A True Dense Mixture of Experts (MoE) custom architecture combining 8 specialized distilgpt2 fine-tuned experts. This model utilizes a dynamic routing layer to evaluate all 8 experts simultaneously for every single token, functioning as a high-density reasoning node with an interpolated 256k context window.

Creator: Guy Edward DuGan II (GODsStrongestSoldier)
Organization: WithinUsAI

Model Summary

Feature	Description
Architecture	Custom True Dense MoE (Modified GPT-2)
Base Model	DistilGPT2
Number of Experts	8 Distinct Fine-tuned MLPs
Total Parameters	~0.8B
Context Window	262,144 tokens (256k via Positional Embedding Interpolation)
Routing Mechanism	Learned Linear Projection (Softmax Gate applied dynamically to all experts)
Language	English
License	Apache 2.0

Expert Models

The routing network balances outputs from 8 specialized experts:

Opus4.7 Expert - Claude Opus distillation, Chain-of-Thought, coding
Qwen3.6 Expert - Qwen reasoning traces, creative problem solving
DeepSeek-V4 Expert - DeepSeek math/logic, R1 reasoning
GLM5.1 Expert - GLM-4/5 reasoning, multiturn CoT
Gemini3.2 Expert - Gemini high-reasoning, creative writing
Grok4.4 Expert - Grok hyper-creative, logic datasets
GPT5.5 Expert - GPT-5 distillation, ultra-high reasoning
Supernatural-Occult Expert - Occult, supernatural, magic, coding

How It Works

Unlike standard sparse MoE architectures that route tokens to a limited subset of experts (e.g., 2 out of 8), this True Dense MoE utilizes all experts simultaneously. A learned gating mechanism calculates the probability distribution for every token and computes a mathematically weighted sum across the entire expert array.

🖥️ Usage 1: Holographic GUI Command Center (Recommended)

Because this model uses a custom PyTorch architecture, it cannot be loaded in standard local runners like LM Studio. To provide the best experience, this repository includes a custom PyQt6 Holographic Interface (minute_moe_gui.py) engineered specifically for this architecture.

Features:

Native True Dense MoE architecture integration.
Real-time telemetry (VRAM, System RAM, Tokens/Second).
Reactive holographic CRT visualizations.
Persona selector (e.g., DeepSeek Logic, High Priest, Opus Coder).
Automatic CPU fallback for systems pushing the 256k context window limits.

How to Run Locally:

Clone or download this repository to your local machine.
Open your terminal in the downloaded folder and install the required dependencies:
pip install torch transformers PyQt6 markdown psutil
Run the GUI launcher:
python minute_moe_gui.py

💻 Usage 2: Raw Python / Jupyter Notebook

If you prefer to integrate the model directly into your own Python scripts, you must inject the custom DenseMoEMLP routing layer class into the GPT-2 skeleton before loading the raw tensor weights.

python

import torch
import torch.nn as nn
from transformers import AutoConfig, AutoModelForCausalLM, GPT2Tokenizer
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file

class DenseMoEMLP(nn.Module):
    def __init__(self, config, num_experts=8):
        super().__init__()
        self.num_experts = num_experts
        self.router = nn.Linear(config.n_embd, self.num_experts, bias=False)
        from transformers.models.gpt2.modeling_gpt2 import GPT2MLP
        self.experts = nn.ModuleList([GPT2MLP(4 * config.n_embd, config) for _ in range(self.num_experts)])

    def forward(self, hidden_states):
        routing_weights = torch.nn.functional.softmax(self.router(hidden_states), dim=-1)
        final_output = torch.zeros_like(hidden_states)
        for i, expert in enumerate(self.experts):
            final_output += routing_weights[..., i, None] * expert(hidden_states)
        return final_output

# 1. Setup Repo and Configuration
repo_id = "GODsStrongestSoldier/Minute-MoE-TrueDense.NSFW-0.8B"
config = AutoConfig.from_pretrained(repo_id)
tokenizer = GPT2Tokenizer.from_pretrained(repo_id)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

# 2. Build base model, rip out standard MLPs, inject True Dense MoE layers
model = AutoModelForCausalLM.from_config(config)
for i in range(config.n_layer):
    model.transformer.h[i].mlp = DenseMoEMLP(config, num_experts=8)

# 3. Download and map the custom weights
weights_path = hf_hub_download(repo_id=repo_id, filename="pytorch_model.bin")
state_dict = torch.load(weights_path, map_location='cpu')

model.load_state_dict(state_dict)

# Push to appropriate hardware (Use 'cpu' if system RAM is needed for full 256k context)
device = 'cuda' if torch.cuda.is_available() else 'cpu'
model.to(torch.float16).to(device)
model.eval()

# Generate
input_text = "Write a highly optimized function to calculate the Fibonacci sequence:"
inputs = tokenizer(input_text, return_tensors="pt").to(device)
with torch.no_grad():
    outputs = model.generate(**inputs, max_length=200, do_sample=True, temperature=0.7)
print(tokenizer.decode(outputs[0]))

Datasets Used

Common Datasets (All Experts)

WithinUsAI/GOD_Coder_Complete_DataSet
acheong08/nsfw_reddit

Opus4.7 Expert

WithinUsAI/Opus4.7_thinking_max_distill_god_seed_25k

Qwen3.6 Expert

WithinUsAI/gods_universe_codex_distill_god_seed_25k

DeepSeek-V4 Expert

GLM5.1 Expert

Gemini3.2 Expert

WithinUsAI/GeminiPro3.2_max_distill_god_seed_25k

Grok4.4 Expert

WithinUsAI/Grok4.4_heavy_max_distill_god_seed_25k

GPT5.5 Expert

WithinUsAI/GPT5.5_thinking_max_distill_god_seed_25K

Supernatural-Occult Expert

WithinUsAI/Supernatural_25k
WithinUsAI/high_priest_supernatural_magic_FACT_BASED_1M

Training Details

Original Expert Training

All experts were trained via full fine-tuning (no LoRA):

Epochs: 1
Batch Size: 4-8 per device
Gradient Accumulation: 8-16 steps
Global Batch Size: 128
Learning Rate: 3e-04 (5e-05 for Supernatural-Occult)
Scheduler: Cosine with 5% warmup
Precision: fp16
Hardware: Dual NVIDIA T4

Limitations

NSFW Content: Trained on mature datasets - use appropriate filtering.
English Only: Not designed for multilingual use.
Computational Cost: True Dense MoE evaluates all 8 experts per token, requiring significantly higher compute/VRAM than standard sparse MoE architectures.
Context Window: The 256k position embeddings were extended via mathematical linear interpolation. Hardware limitations may prevent utilizing the full 256k context window unless deployed in CPU environments with substantial System RAM.

License

Apache 2.0 - Free to use, modify, and distribute with attribution.

Base Model: distilgpt2
Expert Models: Fine-tuned by GODsStrongestSoldier / WithinUsAI


THE "Minute-MoE-TrueDense.NSFW-0.8B" Minute experts are
GODsStrongestSoldier/Distil-Qwen3.6-Minute.Codex.NSFW-0.1B
GODsStrongestSoldier/Distil-Deepseek.V4.Pro.Minute.Codex.NSFW-0.1B
GODsStrongestSoldier/Distil-GLM5.1-Minute.Codex.NSFW-0.1B
GODsStrongestSoldier/Distil-Opus4.7-Minute.Codex.NSFW-0.1B
GODsStrongestSoldier/Distil-Gemini3.2-Pro.Minute.Codex.NSFW-0.1B
GODsStrongestSoldier/Distil-Grok4.4-Minute.Codex.NSFW-0.1B
GODsStrongestSoldier/Distil-GPT5.5-Minute.Codex.NSFW-0.1B
GODsStrongestSoldier/distilgpt2-supernatural-occult-coder

11-47/Minute.MoE.NSFW-1.3B