You can deploy this model and start earning money today!

paragon-of-brah/
Nex-N2-Pro-397B-A17B-GGUF

Name: Nex-N2-Pro-397B-A17B-GGUF
Availability: InStock

Views

No views yet

Quants of Nex-N2-Pro, a fine tune built on Qwen 3.5 397B A17B. Basically the Qwen 3.6 397B that we never got. Comes with mmproj for vision, but isn't shipped with MTP.

All quants target 16/24/32GB GPUs, with varying amounts of RAM depending on the quant.

Specific quant details:

IQ5_KS - ik fork only

Only works on ik_llama.cpp, targets a 256GB RAM system + nvidia GPU 24/32GB.

Will eat 20822MB of VRAM and 214GB of RAM with this config (needs a strong CPU, like 9950x3d, or PP will be slower):

./build/bin/llama-server
    -m pmodels/Nex-397B-A17B-IQ5_KS.gguf
    --mmproj pmodels/Nex-397B-A17B-BF16-mmproj.gguf
    --no-mmproj-offload
    -a NexQ8
    --slot-save-path slots
    --context-shift off
    -ot "blk\.(?:[0-9]|[1-5][0-9])\.ffn.*_exps.*=CPU"
    -ot "token_embd\.weight=CPU"
    -c 196608
    --ctx-checkpoints 12
    --ctx-checkpoints-interval 512
    --ctx-checkpoints-tolerance 4
    --parallel 1
    -cram 0
    -b 4096 -ub 4096
    -wgt 1
    -ctk q8_0 -ctv q8_0
    -khad -vhad
    -mqkv
    --threads 7 --threads-batch 8 -ngl 100
    -cuda fusion=1,offload-batch-size=1000,mmq-id-size=0,fa-offset=0
    --host 127.0.0.1
    --port 8080
    --webui none
    --jinja

Will eat 23500MB of VRAM and 214GB of RAM with this config (increases PP speed for weaker CPUs at the cost of more VRAM usage):

./build/bin/llama-server
    -m pmodels/Nex-397B-A17B-IQ5_KS.gguf
    --mmproj pmodels/Nex-397B-A17B-BF16-mmproj.gguf
    --no-mmproj-offload
    -a NexQ8
    --slot-save-path slots
    --context-shift off
    -ot "blk\.(?:[0-9]|[1-5][0-9])\.ffn.*_exps.*=CPU"
    -ot "token_embd\.weight=CPU"
    -c 196608
    --ctx-checkpoints 12
    --ctx-checkpoints-interval 512
    --ctx-checkpoints-tolerance 4
    --parallel 1
    -cram 0
    -b 4096 -ub 4096
    -wgt 1
    -ctk q8_0 -ctv q8_0
    -khad -vhad
    -mqkv
    --threads 7 --threads-batch 8 -ngl 100
    -cuda fusion=1,offload-batch-size=16,mmq-id-size=0,fa-offset=0
    --host 127.0.0.1
    --port 8080
    --webui none
    --jinja

Details:

## Gated Attention/Delta Net [Blended 0-59]
blk\..*\.attn_gate\.weight=bf16
blk\..*\.attn_qkv\.weight=bf16
blk\..*\.ssm_alpha\.weight=bf16
blk\..*\.ssm_beta\.weight=bf16
blk\..*\.ssm_out\.weight=bf16

# Normal attention
blk\..*\.attn_output\.weight=q8_0
blk\..*\.attn_q\.weight=q8_0
blk\..*\.attn_k\.weight=q8_0
blk\..*\.attn_v\.weight=q8_0

# Shared Expert Layers [0-59]
blk\..*\.ffn_down_shexp\.weight=q8_0
blk\..*\.ffn_(gate|up)_shexp\.weight=q8_0

# Routed Experts Layers [0-59]
blk\..*\.ffn_down_exps\.weight=IQ5_KS
blk\..*\.ffn_(gate|up)_exps\.weight=IQ4_KS

# Non-Repeating Layers
token_embd\.weight=q8_0
output\.weight=q8_0

IQ4_KSS - ik fork only

Works with ik only, targets a 192GB RAM system + any GPU 24GB.

Will eat 19450MB of VRAM and 182GB of RAM with standard config:

./build/bin/llama-server
    -m pmodels/Nex-397B-A17B-IQ4_KSS.gguf
    --mmproj pmodels/Nex-397B-A17B-BF16-mmproj.gguf
    --no-mmproj-offload
    -a NexQ8
    --slot-save-path slots
    --context-shift off
    -ot "blk\.(?:[0-9]|[1-5][0-9])\.ffn.*_exps.*=CPU"
    -ot "token_embd\.weight=CPU"
    -c 196608
    --ctx-checkpoints 12
    --ctx-checkpoints-interval 512
    --ctx-checkpoints-tolerance 4
    --parallel 1
    -cram 0
    -b 4096 -ub 4096
    -wgt 1
    -ctk q8_0 -ctv q8_0
    -khad -vhad
    -mqkv
    --threads 7 --threads-batch 8 -ngl 100
    -cuda fusion=1,offload-batch-size=16,mmq-id-size=0,fa-offset=0
    --host 127.0.0.1
    --port 8080
    --webui none
    --jinja

Details:

## Gated Attention/Delta Net [Blended 0-59]
blk\..*\.attn_gate\.weight=q8_0
blk\..*\.attn_qkv\.weight=q8_0
blk\..*\.ssm_alpha\.weight=bf16
blk\..*\.ssm_beta\.weight=bf16
blk\..*\.ssm_out\.weight=bf16

# Normal attention
blk\..*\.attn_output\.weight=q8_0
blk\..*\.attn_q\.weight=q8_0
blk\..*\.attn_k\.weight=q8_0
blk\..*\.attn_v\.weight=q8_0

# Shared Expert Layers [0-59]
blk\..*\.ffn_down_shexp\.weight=q8_0
blk\..*\.ffn_(gate|up)_shexp\.weight=q8_0

# Routed Experts Layers [0-59]
blk\..*\.ffn_down_exps\.weight=iq4_kss
blk\..*\.ffn_(gate|up)_exps\.weight=iq4_kss

# Non-Repeating Layers
token_embd\.weight=q8_0
output\.weight=q8_0

IQ3_M - mainline compatible

Works with mainline and ik, targets a 196GB RAM system + any GPU 24GB.

Will eat 19600MB of VRAM and 180GB of RAM with standard config:

./build/bin/llama-server
    -m pmodels/Nex-397B-A17B-IQ3_M.gguf
    --mmproj pmodels/Nex-397B-A17B-BF16-mmproj.gguf
    --no-mmproj-offload
    -a NexQ8
    --slot-save-path slots
    --context-shift off
    -ot "blk\.(?:[0-9]|[1-5][0-9])\.ffn.*_exps.*=CPU"
    -ot "token_embd\.weight=CPU"
    -c 196608
    --ctx-checkpoints 12
    --ctx-checkpoints-interval 512
    --ctx-checkpoints-tolerance 4
    --parallel 1
    -cram 0
    -b 4096 -ub 4096
    -wgt 1
    -ctk q8_0 -ctv q8_0
    -khad -vhad
    -mqkv
    --threads 7 --threads-batch 8 -ngl 100
    -cuda fusion=1,offload-batch-size=16,mmq-id-size=0,fa-offset=0
    --host 127.0.0.1
    --port 8080
    --webui none
    --jinja

Details:

## Gated Attention/Delta Net [Blended 0-59]
blk\..*\.attn_gate\.weight=q8_0
blk\..*\.attn_qkv\.weight=q8_0
blk\..*\.ssm_alpha\.weight=q8_0
blk\..*\.ssm_beta\.weight=q8_0
blk\..*\.ssm_out\.weight=q8_0

# Normal attention
blk\..*\.attn_output\.weight=q8_0
blk\..*\.attn_q\.weight=q8_0
blk\..*\.attn_k\.weight=q8_0
blk\..*\.attn_v\.weight=q8_0

# Shared Expert Layers [0-59]
blk\..*\.ffn_down_shexp\.weight=q8_0
blk\..*\.ffn_(gate|up)_shexp\.weight=q8_0

# Routed Experts Layers [0-59]
blk\..*\.ffn_down_exps\.weight=IQ4_XS
blk\..*\.ffn_(gate|up)_exps\.weight=IQ3_S

# Non-Repeating Layers
token_embd\.weight=q8_0
output\.weight=q8_0

IQ3_XXS - mainline compatible

Works with mainline and ik, targets a 196GB RAM system + any GPU 24GB.

Will eat 18930MB of VRAM and 151GB of RAM with standard config:

./build/bin/llama-server
    -m pmodels/Nex-397B-A17B-IQ3_XXS.gguf
    --mmproj pmodels/Nex-397B-A17B-BF16-mmproj.gguf
    --no-mmproj-offload
    -a NexQ8
    --slot-save-path slots
    --context-shift off
    -ot "blk\.(?:[0-9]|[1-5][0-9])\.ffn.*_exps.*=CPU"
    -ot "token_embd\.weight=CPU"
    -c 196608
    --ctx-checkpoints 12
    --ctx-checkpoints-interval 512
    --ctx-checkpoints-tolerance 4
    --parallel 1
    -cram 0
    -b 4096 -ub 4096
    -wgt 1
    -ctk q8_0 -ctv q8_0
    -khad -vhad
    -mqkv
    --threads 7 --threads-batch 8 -ngl 100
    -cuda fusion=1,offload-batch-size=16,mmq-id-size=0,fa-offset=0
    --host 127.0.0.1
    --port 8080
    --webui none
    --jinja

Details:

## Gated Attention/Delta Net [Blended 0-59]
blk\..*\.attn_gate\.weight=q8_0
blk\..*\.attn_qkv\.weight=q8_0
blk\..*\.ssm_alpha\.weight=q8_0
blk\..*\.ssm_beta\.weight=q8_0
blk\..*\.ssm_out\.weight=q8_0

# Normal attention
blk\..*\.attn_output\.weight=q8_0
blk\..*\.attn_q\.weight=q8_0
blk\..*\.attn_k\.weight=q8_0
blk\..*\.attn_v\.weight=q8_0

# Shared Expert Layers [0-59]
blk\..*\.ffn_down_shexp\.weight=Q6_K
blk\..*\.ffn_(gate|up)_shexp\.weight=Q6_K

# Routed Experts Layers [0-59]
blk\..*\.ffn_down_exps\.weight=IQ3_XXS
blk\..*\.ffn_(gate|up)_exps\.weight=IQ3_XXS

# Non-Repeating Layers
token_embd\.weight=q6_k
output\.weight=q6_k

IQ2_M - mainline compatible

Works with mainline and ik, targets a 196GB RAM system + any GPU 24GB.

Will eat 19050MB of VRAM and 138GB of RAM with standard config:

./build/bin/llama-server
    -m pmodels/Nex-397B-A17B-IQ2_M.gguf
    --mmproj pmodels/Nex-397B-A17B-BF16-mmproj.gguf
    --no-mmproj-offload
    -a NexQ8
    --slot-save-path slots
    --context-shift off
    -ot "blk\.(?:[0-9]|[1-5][0-9])\.ffn.*_exps.*=CPU"
    -ot "token_embd\.weight=CPU"
    -c 196608
    --ctx-checkpoints 12
    --ctx-checkpoints-interval 512
    --ctx-checkpoints-tolerance 4
    --parallel 1
    -cram 0
    -b 4096 -ub 4096
    -wgt 1
    -ctk q8_0 -ctv q8_0
    -khad -vhad
    -mqkv
    --threads 7 --threads-batch 8 -ngl 100
    -cuda fusion=1,offload-batch-size=16,mmq-id-size=0,fa-offset=0
    --host 127.0.0.1
    --port 8080
    --webui none
    --jinja

Details:

## Gated Attention/Delta Net [Blended 0-59]
blk\..*\.attn_gate\.weight=q8_0
blk\..*\.attn_qkv\.weight=q8_0
blk\..*\.ssm_alpha\.weight=q8_0
blk\..*\.ssm_beta\.weight=q8_0
blk\..*\.ssm_out\.weight=q8_0

# Normal attention
blk\..*\.attn_output\.weight=q8_0
blk\..*\.attn_q\.weight=q8_0
blk\..*\.attn_k\.weight=q8_0
blk\..*\.attn_v\.weight=q8_0

# Shared Expert Layers [0-59]
blk\..*\.ffn_down_shexp\.weight=q8_0
blk\..*\.ffn_(gate|up)_shexp\.weight=q8_0

# Routed Experts Layers [0-59]
blk\..*\.ffn_down_exps\.weight=IQ3_XXS
blk\..*\.ffn_(gate|up)_exps\.weight=IQ2_S

# Non-Repeating Layers
token_embd\.weight=q8_0
output\.weight=q8_0

IQ1_M - mainline compatible

Works with mainline and ik, targets a 128GB RAM system + any GPU 16GB+.

Will eat 14210MB of VRAM and 94GB of RAM with standard config:

./build/bin/llama-server
    -m pmodels/Nex-397B-A17B-IQ1_M.gguf
    --mmproj pmodels/Nex-397B-A17B-BF16-mmproj.gguf
    --no-mmproj-offload
    -a NexQ8
    --slot-save-path slots
    --context-shift off
    -ot "blk\.(?:[0-9]|[1-5][0-9])\.ffn.*_exps.*=CPU"
    -ot "token_embd\.weight=CPU"
    -c 196608
    --ctx-checkpoints 12
    --ctx-checkpoints-interval 512
    --ctx-checkpoints-tolerance 4
    --parallel 1
    -cram 0
    -b 4096 -ub 4096
    -wgt 1
    -ctk q8_0 -ctv q8_0
    -khad -vhad
    -mqkv
    --threads 7 --threads-batch 8 -ngl 100
    -cuda fusion=1,offload-batch-size=16,mmq-id-size=0,fa-offset=0
    --host 127.0.0.1
    --port 8080
    --webui none
    --jinja

Details:

## Gated Attention/Delta Net [Blended 0-59]
blk\..*\.attn_gate\.weight=IQ4_XS
blk\..*\.attn_qkv\.weight=IQ4_XS
blk\..*\.ssm_alpha\.weight=q8_0
blk\..*\.ssm_beta\.weight=q8_0
blk\..*\.ssm_out\.weight=q8_0

# Normal attention
blk\..*\.attn_output\.weight=IQ4_XS
blk\..*\.attn_q\.weight=IQ4_XS
blk\..*\.attn_k\.weight=IQ4_XS
blk\..*\.attn_v\.weight=IQ4_XS

# Shared Expert Layers [0-59]
blk\..*\.ffn_down_shexp\.weight=IQ4_XS
blk\..*\.ffn_(gate|up)_shexp\.weight=IQ4_XS

# Routed Experts Layers [0-59]
blk\..*\.ffn_down_exps\.weight=IQ2_XXS
blk\..*\.ffn_(gate|up)_exps\.weight=IQ1_M

# Non-Repeating Layers
token_embd\.weight=Q6_K
output\.weight=Q6_K

Every additional 65536 tokens of context window require one additional GB of VRAM at Q8 KV cache.

The model was natively trained on a 262144 ctx window, so if you want to go beyond 262144 you need to use the additional YARN commands (both for ik and mainline):

  --rope-scaling yarn
  --rope-scale N
  --yarn-orig-ctx 262144

Where N is the context ceiling multiplier (2 for 524288, 4 for 1M). Close to no quality loss at scale 2, some quality loss at scale 4.

paragon-of-brah/Nex-N2-Pro-397B-A17B-GGUF

paragon-of-brah/
Nex-N2-Pro-397B-A17B-GGUF