This model is "Built with Llama".
It is based on
meta-llama/Meta-Llama-3.1-8B-Instruct and
was created with the help of
mergekit.
This is the mergekit configuration we used:
mergekit_moe_config.ymlIt should be noted that this model is the raw model after merging.
It still has randomly initialized router networks and will not be better than a single one of its expert models.
This model requires further training before use.
This model has a total of 47.5B params, which is slightly more than the
Mixtral 8x7b
with its 46.7B params.
This model is licensed under the Llama 3.1 Community License, Copyright (c) 2024
Philip May,
Deutsche Telekom AG
Llama 3.1 is licensed under the Llama 3.1 Community License, Copyright (c) Meta Platforms, Inc. All Rights Reserved.
