Instructions on how to use this model with Ollama (Windows 11 Example)

1. You'll need to download both shards of this file in a separate folder, which had to be split due to the 50GB HuggingFace file upload limit.
2. Then, use the latest binaries from the llama.cpp release page. For example, at the moment of this writing the latest release is b4202, and for a Windows PC using Nvidia CUDA-enabled GPUs, the binary we could download might be cudart-llama-bin-win-cu12.4-x64.zip.
3. Extract the zip file with the binaries into the same folder where you downloaded the quantized model shards.
4. Open up the Windows PowerShell, I recommend you use PowerShell 7, preferably installed via WinGet, if you aren't already using it.
5. Change directory into the folder where you have your model shards and the llama.cpp binaries located and run the following command:
   .\llama-gguf-split.exe --merge Qwen-2.5-72B-Instruct-abliterated-v2-Q6_K-00001-of-00002.gguf Qwen2.5-72B-Instruct-abliterated-Q6_K-2.gguf
   You should now have an approximately 62.8 GB sized file named Qwen2.5-72B-Instruct-abliterated-Q6_K-2.gguf in your directory.
6. Next, use the PowerShell to create an empty Modelfile item
   New-Item -Path "Modelfile" -ItemType File
7. Open the Modelfile with Notepad and add the following, keeping in mind that you can alter the context length to what you like (here it's been set to 16K):
   text

   1FROM Qwen-2.5-72B-Instruct-abliterated-v2-Q6_K.gguf 2# Set context length to 16K 3PARAMETER num_ctx 16384
8. Create the model in Ollama using the following command:
   ollama create Qwen2.5-72B-Instruct-abliterated-Q6_K -f Modelfile
   It will take a few moments to transfer the model data into Ollama.
9. You can use ollama list to check to see if the model has been loaded, and you should see console output similar to this:
   powershell

   1PS C:\AI\Q25-72B-Instruct-Abliterated-Q6_K-GGUF> ollama list 2NAME ID SIZE MODIFIED 3Qwen2.5-72B-Instruct-abliterated-Q6_K:latest 1c78aabf213f 64 GB 5 seconds ago