You can deploy this model and start earning money today!

yu505948/
mpc-router-mrpc

Name: mpc-router-mrpc
Availability: InStock

Views

No views yet

MPC-Router MRPC checkpoint

A trained GatedBertPool (cost-aware Mixture-of-BERT-Experts) checkpoint for the GLUE MRPC paraphrase classification task.

Architecture

Router stem: first 2 layers of prajjwal1/bert-tiny
MLP router: 128 → 256 → 3
Expert pool (heterogeneous, fine-tuned on MRPC):
- M-FAC/bert-tiny-finetuned-mrpc (cost = 2)
- howey/bert-base-uncased-mrpc (cost = 7)
- yoshitomo-matsubara/bert-large-uncased-mrpc (cost = 13)

Headline numbers (GLUE MRPC validation, 408 samples)

Metric	Value
Soft-gating accuracy	87.50%
Soft-gating F1	90.36%
Avg router prob (tiny / base / large)	12.4% / 28.7% / 58.9%
Average expected cost	9.91 / 13
Compute saving vs always-large	24%

File contents

This checkpoint is a slim PyTorch state-dict (no optimizer state):

python

ckpt = torch.load('mrpc_best_model.pt', map_location='cpu', weights_only=False)
ckpt.keys()                # ['model_state_dict', 'config']

Usage

python

from huggingface_hub import hf_hub_download
from mpc_router import GatedBertPool
import torch

ckpt_path = hf_hub_download('yu505948/mpc-router-mrpc', 'mrpc_best_model.pt')
ckpt   = torch.load(ckpt_path, map_location='cpu', weights_only=False)
config = ckpt['config']
model  = GatedBertPool(
    router_model_name  = config['model']['router_model_name'],
    router_stem_layers = config['model']['router_stem_layers'],
    expert_model_names = config['model']['expert_model_names'],
    num_labels         = config['task']['num_labels'],
    task_type          = config['task']['task_type'],
    router_hidden_size = config['model']['router_hidden_size'],
    router_dropout     = config['model']['router_dropout'],
)
model.load_state_dict(ckpt['model_state_dict'])

See tutorials/Demo.ipynb in the source repo for an end-to-end Colab demo.

Source

Repo: https://github.com/wocaishiniliu/MPC_Router_with_LLM_pool
Trained with mpc-router-train --config configs/task_mrpc.yaml (batch=16, epochs=9, lr=2e-6, alpha=0.01, beta=0.03)