You can deploy this model and start earning money today!

FAIRC/
mowe-dense-768-19l

Name: mowe-dense-768-19l
Availability: InStock

Views

No views yet

Dense 768, 19 Layers

Inference checkpoint for the dense baseline released with Mixture of Wide Experts.

Property	Value
Parameters	103.46M
Transformer layers	19
Model width	768
Attention heads	12
FFN hidden width	455
Context length	1,024
Tokenizer	GPT-2

The checkpoint is an inference-only safetensors state dictionary. It contains no optimizer state. This is a base language model and is not instruction-tuned.

Loading

Install the companion MoWE extension for OpenLanguageModel, then load this repository directory:

python

import torch

from mowe import load_checkpoint

model = load_checkpoint(
    "/path/to/dense-768-19l",
    device="cuda",
    dtype=torch.bfloat16,
).eval()

Limitations

This is a small research model. It may produce incorrect, biased, unsafe, or memorized text and should not be used as a production assistant without separate evaluation and safeguards.

License

Released under the Apache License 2.0. Please cite the accompanying MoWE paper.

FAIRC/mowe-dense-768-19l

Dense 768, 19 Layers

Loading

Limitations

License

FAIRC/
mowe-dense-768-19l