How-To-Set-and-Manage-MOE-Mix-of-Experts-Model-Activation-of-Experts
This document discusses how to set/change the Mixture of Experts in various LLM/AI apps and includes links
to additional MOE Models, and other helpful resources.
LINKS:
Mixture Of Expert Models - including Reasoning/Thinking - GGUF:
All Models Source Code - For GGUFs, AWQ, HQQ, GPTQ, EXL2 and direct usage - including MOEs:
Additional:
#1 All Reasoning/Thinking Models - including MOEs - (collection) (GGUF):
#2 All Reasoning/Thinking Models - including MOES - (collection) (Source Code to generation GGUF, EXL2, AWQ, GPTQ, HQQ, etc etc and direct usage):
#3 All Adapters (collection) - Turn a "regular" model into a "thinking/reasoning" model:
These collections will update over time. Newest items are usually at the bottom of each collection.
Main Document - Setting Mixture Of Experts in LLM/AI apps
Experts Activation / Models used to build this model:
The mixture of experts can be set at 1, 2, 4, 8 ... or more experts, but you can use 1, 2, 3, or 4.
This "team" has a Captain (first listed model), and then all the team members contribute to the to "token"
choice billions of times per second. Note the Captain also contributes too.
Think of 2, 3 or 4 (or more) master chefs in the kitchen all competing to make the best dish for you.
This results in higher quality generation.
This also results in many cases in higher quality instruction following too.
That means the power of every model is available during instruction and output generation.
NOTE:
You can use one "expert" too ; however this means the model will randomly select an expert to use EACH TIME, resulting
in very different generation for each prompt / regen of a prompt.
CHANGING THE NUMBER OF EXPERTS:
You can set the number of experts in LMStudio (
https://lmstudio.ai) at the "load" screen and via other apps/llm apps by setting "Experts" or "Number of Experts".
For Text-Generation-Webui (
https://github.com/oobabooga/text-generation-webui) you set the number of experts at the loading screen page.
For KolboldCPP (
https://github.com/LostRuins/koboldcpp) Version 1.8+ , on the load screen, click on "TOKENS",
you can set experts on this page, and the launch the model.
For server.exe / Llama-server.exe (Llamacpp -
https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md )
add the following to the command line to start the "llamacpp server" (CLI):
"--override-kv llama.expert_used_count=int:3"
(no quotes, where "3" is the number of experts to use)
When using "API", you set the "num_experts_used" in the JSON payload (this maybe different for different back ends).
SUGGESTION:
The MOE models at my repo:
Contain various examples, including example generation(s) showing 2, 4, and 8 experts.
This will give you a better idea of what changes to expect when adjusting the number of experts
and the effect on generation.
Special Thanks:
Special thanks to all the following, and many more...
All the model makers, fine tuners, mergers, and tweakers:
- Provides the raw "DNA" for almost all my models.
- Sources of model(s) can be found on the repo pages, especially the "source" repos with link(s) to the model creator(s).
- The place to store, merge, and tune models endlessly.
- THE reason we have an open source community.
- The ability to compress and run models on GPU(s), CPU(s) and almost all devices.
- Imatrix, Quantization, and other tools to tune the quants and the models.
- Llama-Server : A cli based direct interface to run GGUF models.
- The only tool I use to quant models.
Quant-Masters: Team Mradermacher, Bartowski, and many others:
- Quant models day and night for us all to use.
- They are the lifeblood of open source access.
- The universal online/offline tool to merge models together and forge something new.
- Over 20 methods to almost instantly merge model, pull them apart and put them together again.
- The tool I have used to create over 1500 models.
- The go to tool to test and run models in GGUF format.
- The Tool I use to test/refine and evaluate new models.
- LMStudio forum on discord; endless info and community for open source.
Text Generation Webui // KolboldCPP // SillyTavern:
- Excellent tools to run GGUF models with - [ https://github.com/oobabooga/text-generation-webui ] [ https://github.com/LostRuins/koboldcpp ] .
- Sillytavern [ https://github.com/SillyTavern/SillyTavern ] can be used with LMSTudio [ https://lmstudio.ai/ ] , TextGen [ https://github.com/oobabooga/text-generation-webui ], Kolboldcpp [ https://github.com/LostRuins/koboldcpp ], Llama-Server [part of LLAMAcpp] as a off the scale front end control system and interface to work with models.
