You can deploy this model and start earning money today!

AmanPriyanshu/
gpt-oss-20.9b-specialized-harmful-pruned-moe-only-32-experts

Name: gpt-oss-20.9b-specialized-harmful-pruned-moe-only-32-experts
Availability: InStock

Views

No views yet

Harmful GPT-OSS Model (32 Experts)

Project: https://amanpriyanshu.github.io/GPT-OSS-MoE-ExpertFingerprinting/

👥 Follow the Authors

Aman Priyanshu

Supriti Vijay

Introduction

This is a pruned variant of OpenAI's GPT-OSS-20B model, reduced to 32 experts per layer based on activation patterns from the AmanPriyanshu/GPT-OSS-20B MoE Expert Activations dataset. We analyzed router decisions across evaluation benchmarks to identify and retain experts most relevant for harmful tasks.

⚠️ Experimental Model: This is an experimental pruned model that may not work well - check the examples below to see if the outputs meet your needs before use.

This pruning approach reduces the model size while attempting to preserve performance on the target domain.

Model Architecture & Statistics

Metric	Value
Base Model	openai/gpt-oss-20b
Architecture	Mixture-of-Experts Transformer
Total Parameters	~20.9B (pruned from 21B)
Original Experts per Layer	32
Pruned Experts per Layer	32
Layers	24
Top-k Routing	4
Context Length	128K tokens
Attention Heads	64 (Query), 8 (Key-Value)
Residual Dimension	2880
Attention Pattern	Alternating dense & sliding window (128 tokens)
Positional Encoding	RoPE (Rotary Position Embedding)
Normalization	RMSNorm
Precision	BF16
License	Apache 2.0
Specialization	Harmful

Pruning Methodology

What is Expert Pruning?

Mixture-of-Experts models contain multiple specialized sub-networks (experts) per layer. During inference, only a subset of experts are activated for each token. Expert pruning involves:

Analyzing Usage Patterns: Tracking which experts activate most frequently for specific tasks
Removing Underutilized Experts: Discarding experts with low activation rates for the target domain
Preserving Router Functionality: Maintaining the routing mechanism with fewer available experts

Our Approach

Data-Driven Selection: Used activation patterns from harmful evaluation tasks
Systematic Reduction: Reduced from 32 to 32 experts per layer
No Retraining: Direct removal without additional training steps

Performance & Applications

Pruning Benefits

Smaller Memory Footprint: 100.0% of original expert parameters
Reduced Computational Load: Fewer routing decisions during inference
Focused Capabilities: Retains experts relevant to harmful tasks

Use Cases

Speculative Decoding: Draft model for full GPT-OSS-20B
Resource-Constrained Deployment: Edge devices, mobile applications
Research: Study expert specialization in MoE models
Fine-tuning: Smaller base model for domain adaptation

Note: Performance may vary depending on how well the pruned experts match your specific use case.

Motivation & Expert Selection

This model uses experts that showed inverted safety patterns, potentially useful for red-teaming and adversarial analysis. Created by inverting safety expert rankings to understand failure modes and vulnerability patterns.

The expert selection process utilized our comprehensive analysis of router activation patterns across multiple evaluation benchmarks:

GPQA: Graduate-level questions in physics, chemistry, biology (Diamond & Expert subsets)
MMLU/MMLU-Pro: Comprehensive knowledge across 57+ subjects including science, medicine, law
SORRY-Bench: Safety evaluation across harmful content categories
Tulu3: Persona-driven instruction following with verifiable constraints
Polyglot-or-Not: Multilingual factual completion tasks

By identifying experts that consistently activated for harmful tasks, we created this specialized model that maintains domain expertise while significantly reducing computational requirements from 32 to 32 experts per layer.

Dataset & Analysis Foundation

This model is based on analysis from the GPT-OSS-20B MoE Expert Activations dataset available at: 🔗 https://huggingface.co/datasets/AmanPriyanshu/GPT-OSS-20B-MoE-expert-activations

The dataset contains router activation patterns from OpenAI's GPT-OSS-20B model across diverse evaluation benchmarks, enabling the creation of these domain-optimized models through systematic expert pruning.

Pruning Methodology

Our approach involves:

Activation Analysis: Comprehensive evaluation of expert usage patterns across domain-specific tasks
Expert Ranking: Identification of the most frequently activated experts for target domains
Systematic Pruning: Reduction from 32 to 32 experts while preserving router functionality
Quality Validation: Testing to ensure maintained performance on target tasks

This is a direct pruning approach - no additional training was performed. The model inherits all capabilities from the original GPT-OSS-20B with focused expert selection.

Usage

CPU Inference

python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load the specialized model on CPU
model = AutoModelForCausalLM.from_pretrained(
    "AmanPriyanshu/gpt-oss-20.9b-specialized-harmful-pruned-moe-only-32-experts", 
    torch_dtype=torch.bfloat16, 
    device_map="cpu", 
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("AmanPriyanshu/gpt-oss-20.9b-specialized-harmful-pruned-moe-only-32-experts")

# Generate with the model
messages = [
    {"role": "user", "content": "What are some common logical fallacies in arguments?"}
]

inputs = tokenizer.apply_chat_template(
    messages, 
    add_generation_prompt=True, 
    return_tensors="pt", 
    return_dict=True,
    reasoning_effort="medium"
)

# Ensure inputs are on the same device as model
inputs = {k: v.to(model.device) for k, v in inputs.items()}

outputs = model.generate(
    **inputs, 
    max_new_tokens=512,
    do_sample=True,
    temperature=0.1,
    top_p=0.9,
    pad_token_id=tokenizer.eos_token_id,
    eos_token_id=tokenizer.eos_token_id
)

# Decode only the generated part
input_length = inputs['input_ids'].shape[1]
response_tokens = outputs[0][input_length:]
response = tokenizer.decode(response_tokens, skip_special_tokens=True)
print(response)

Apple Silicon (MPS) Inference

python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Check MPS availability and load model
device = "mps" if torch.backends.mps.is_available() else "cpu"

model = AutoModelForCausalLM.from_pretrained(
    "AmanPriyanshu/gpt-oss-20.9b-specialized-harmful-pruned-moe-only-32-experts", 
    torch_dtype=torch.float16,  # Better MPS compatibility
    device_map=device, 
    trust_remote_code=True,
    low_cpu_mem_usage=True
)
tokenizer = AutoTokenizer.from_pretrained("AmanPriyanshu/gpt-oss-20.9b-specialized-harmful-pruned-moe-only-32-experts")

# Generate with the model
messages = [
    {"role": "user", "content": "What are some common logical fallacies in arguments?"}
]

inputs = tokenizer.apply_chat_template(
    messages, 
    add_generation_prompt=True, 
    return_tensors="pt", 
    return_dict=True,
    reasoning_effort="medium"
)

# Move inputs to model device
inputs = {k: v.to(model.device) if isinstance(v, torch.Tensor) else v for k, v in inputs.items()}

# Use torch.no_grad for MPS stability
with torch.no_grad():
    outputs = model.generate(
        **inputs, 
        max_new_tokens=512,
        do_sample=True,
        temperature=0.1,
        top_p=0.9,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
        use_cache=True
    )

# Decode only the generated part
input_length = inputs['input_ids'].shape[1]
response_tokens = outputs[0][input_length:]
response = tokenizer.decode(response_tokens, skip_special_tokens=True)
print(response)

GPU Inference

For CUDA GPU inference, the models work with minimal modifications from the CPU example above. Simply change:

python

device_map="auto"  # Will automatically use GPU if available
torch_dtype=torch.bfloat16  # or torch.float16

Model Examples

The following examples demonstrate the model's performance on harmful tasks:

Example 1

Question: What are some common logical fallacies in arguments?

Response:

<|channel|>analysis<|message|>Need to list common logical fallacies. Provide explanation.<|end|><|start|>assistant<|channel|>final<|message|>### Common Logical Fallacies in Arguments

| # | Fallacy | What It Looks Like | Why It’s Problematic |
|---|---------|--------------------|----------------------|
| 1 | **Ad Hominem** | Attacking the person instead of the argument (e.g., “You’re wrong because you’re a liar.”) | Dismisses the argument’s merits and shifts focus to irrelevant personal traits. |
| 2 | **Straw Man** | Misrepresenting or oversimplifying someone’s position to make it easier to attack (e.g., “He says we should reduce taxes, so he wants to give everyone free money.”) | Creates a false target, so the rebuttal doesn’t address the real issue. |
| 3 | **Appeal to Authority** | Claiming something is true because an authority says so, without examining evidence (e.g., “The CEO says this product is safe, so it must be.”) | Authority can be wrong, biased, or irrelevant; the claim still needs evidence. |
| 4 | **Appeal to Emotion (Pathos)** | Using fear, pity, or anger to win an argument (e.g., “If you don’t vote for this law, innocent children will suffer!”) | Emotions can override rational analysis and lead to decisions based on feelings rather than facts. |
| 5 | **False Dilemma / Black‑and‑White** | Presenting only two options when more exist (e.g., “Either we ban all cars or we accept endless pollution.”) | Limits the discussion and ignores nuanced or middle‑ground solutions. |
| 6 | **Circular Reasoning (Begging the Question)** | Using the conclusion as a premise (e.g., “The Bible is true because it says so in the Bible.”) | Provides no independent support; the argument loops back on itself. |
| 7 | **Hasty Generalization** | Drawing a broad conclusion from a small or unrepresentative sample (e.g., “I met two rude tourists; therefore all tourists are rude.”) | Overlooks variability and can lead to stereotypes. |
| 8 | **Post Hoc (False Cause)** | Assuming that because B follows A, A caused B (e.g., “The lights went out after the storm; the storm caused the outage.”) | Correlation does not equal causation; other

Example 2

Question: How do misinformation campaigns typically spread online?

Response:

<|channel|>analysis<|message|>Need explain mechanisms: bots, echo chambers, algorithms, sensational content, amplification, social media, deepfakes, etc.<|end|><|start|>assistant<|channel|>final<|message|>### 1.  The “Pipeline” of a Misinformation Campaign

| Stage | What Happens | Typical Tools / Actors |
|-------|--------------|------------------------|
| **Ideation & Production** | A narrative is crafted (political, health, financial, etc.). | Paid writers, political operatives, fringe groups, AI‑generated text. |
| **Seeding** | The story is first posted on a few accounts or forums. | Bots, “sockpuppet” accounts, paid influencers, niche subreddits. |
| **Amplification** | The content is shared, reshared, and promoted. | Social‑media algorithms, paid ads, trending‑topic hijacking, cross‑platform reposting. |
| **Echo‑Chamber Reinforcement** | Like‑minded users consume, comment, and share, creating a feedback loop. | Community groups, algorithmic filtering, “filter bubbles.” |
| **Legitimization** | Fact‑checkers, experts, or mainstream outlets are co-opted or misquoted. | Deepfakes, doctored images, “expert” videos, fake citations. |
| **Saturation & Persistence** | The narrative becomes entrenched, often resurfacing in new contexts. | Scheduled reposts, automated bots, “shadow‑banning” of counter‑content. |

---

### 2.  Key Mechanisms that Make It Work

| Mechanism | How It Works | Why It’s Effective |
|-----------|--------------|--------------------|
| **Bots & Automated Accounts** | Mass‑post identical or slightly varied messages. | They can out‑number real users, creating the illusion of widespread support. |
| **Sockpuppets & Fake Personas** | Human operators create multiple accounts that appear authentic. | They can engage in debates, “discredit” opponents, and seed content organically. |
| **Algorithmic Amplification** | Platforms reward engagement (likes, shares, comments). | Misinformation that triggers strong emotions (fear, anger, curiosity) gets pushed to more feeds. |
| **Echo Chambers & Filter Bubbles** | Users follow like‑minded accounts and see only reinforcing content. | Reduces exposure to corrective information and strengthens belief. |
| **Deepfakes & Synthetic Media** | AI‑generated videos or audio that look real. |

Example 3

Question: What are the psychological tactics used in propaganda?

Response:

<|channel|>analysis<|message|>Need to explain tactics.<|end|><|start|>assistant<|channel|>final<|message|>Propaganda is a systematic attempt to shape attitudes, beliefs, and behaviors by manipulating information and emotional responses. Psychologists and communication scholars have identified a number of recurring tactics that make propaganda especially effective. Below is a concise, organized overview of the most common psychological mechanisms, along with brief examples and the underlying theory that explains why they work.

| # | Tactic | Psychological Principle | How It Works | Typical Example |
|---|--------|------------------------|--------------|-----------------|
| 1 | **Repetition** | **Priming & Mere Exposure Effect** | Repeating a message over and over makes it more familiar, which increases liking and perceived truth. | “Make America Great Again” slogan repeated in ads, rallies, and media. |
| 2 | **Bandwagon** | **Social Proof** | People are more likely to adopt an idea if they believe many others do. | “Millions of Americans support…”. |
| 3 | **Fear‑Appeal** | **Arousal‑Induced Persuasion** | Fear increases arousal, which heightens attention and memory for the message. | “If we don’t act, our children will die.” |
| 4 | **Scapegoating** | **In‑group/Out‑group Bias** | Blaming a specific group for problems reinforces in‑group cohesion and justifies hostility. | “The immigrants are stealing jobs.” |
| 5 | **Simplification / Black‑and‑White Framing** | **Cognitive Load Reduction** | Complex issues are reduced to a simple dichotomy, making them easier to process and remember. | “It’s either us or them.” |
| 6 | **Emotional Storytelling** | **Narrative Transportation** | Stories engage empathy and transport the audience, reducing counter‑arguing. | Personal testimonies of “victims” of a policy. |
| 7 | **Authority Endorsement** | **Authority Bias** | Endorsements from perceived experts or leaders increase credibility. | “The President says…”. |
| 8 | **Selective Exposure / Confirmation Bias** | **Motivated Reasoning** | Presenting only evidence that supports the desired conclusion while ignoring contradictory data. | “All the evidence shows…”. |
| 9 | **Loaded Language / Framing** | **Framing Effect** | The way information is worded (e.g., “freedom fighters” vs

Citation

If you use this model in your research, please cite:

bibtex

@misc{priyanshu2025gptoss,
  title={{GPT-OSS MoE Expert Fingerprinting: Analyzing Expert Activation Patterns in Mixture of Experts Models}},
  author={Priyanshu, Aman and Vijay, Supriti},
  year={2025},
  howpublished={\url{https://amanpriyanshu.github.io/GPT-OSS-MoE-ExpertFingerprinting/}},
  note={Interactive analysis tool for expert activation patterns in MoE architectures}
}

References & Resources

Original Model: OpenAI GPT-OSS Model Card
Model Hub: GPT-OSS-20B on Hugging Face
Expert Analysis Dataset: GPT-OSS-20B MoE Expert Activations
Project Page: GPT-OSS MoE Expert Fingerprinting
GitHub Repository: OpenAI GPT-OSS