You can deploy this model and start earning money today!

whaoyang/
gemma-3-4b-novision-quant-rk3588-1.2.1

Name: gemma-3-4b-novision-quant-rk3588-1.2.1
Availability: InStock

gghfez/gemma-3-4b-novision

finetune

gemma

autotrain_compatible

text-generation-inference

endpoints_compatible

Views

No views yet

This version of gemma-3-4b-novision has been converted to run on the RK3588 NPU using w8a8 quantization and with the quant_dataset.json file from this repo as the value for the dataset param of the rkllm.build() command.

This converted rkllm model differs from my other gemma-3-4b-novision-rk3588-1.2.1 model in that rkllm.build() was run with param dataset=quant_dataset.json instead of param dataset=None in the other model.

The quant_dataset.json file was generated with Rockchip's generate_data_quant.py script with the following arguments:

max_new_tokens = 448
top_k = 64
temperature = 0.7
repetition_penalty = 1.0
apply_chat_template = True

This model has been optimized with the following LoRA: NA

This model supports a max context length of 16384.

Compatible with RKLLM version: 1.2.1

Recommended `rkllm` parameters

This model runs well in limited testing with the following rkllm library paremeters:

n_keep = -1
top_k = 64
top_p = 0.95
temperature = 0.7
repeat_penalty = 1.0
frequency_penalty = 1.0
presence_penalty = 0.0
mirostat = 0
mirostat_tau = 5.0
mirostat_eta = 0.1

It is recommended to also apply a specific chat template using the following Python methods(that hook the rkllm library):

# System prompt taken from https://docs.unsloth.ai/basics/gemma-3-how-to-run-and-fine-tune#official-recommended-inference-settings
system_prompt = "<bos><start_of_turn>user\nHello!<end_of_turn>\n<start_of_turn>model\nHey there!<end_of_turn>\n<start_of_turn>user\nWhat is 1+1?<end_of_turn>\n<start_of_turn>model\n"
prompt_prefix = "<start_of_turn>user\n"
prompt_postfix = "<end_of_turn>\n<start_of_turn>model\n"
rkllm_lib.rkllm_set_chat_template(
  llm_handle, 
  ctypes.c_char_p(system_prompt.encode('utf-8')), 
  ctypes.c_char_p(prompt_prefix.encode('utf-8')), 
  ctypes.c_char_p(prompt_postfix.encode('utf-8')))

Useful links:

Official RKLLM GitHub

RockhipNPU Reddit

EZRKNN-LLM

Pretty much anything by these folks: marty1885 and happyme531

Converted using https://github.com/c0zaut/ez-er-rkllm-toolkit

whaoyang/gemma-3-4b-novision-quant-rk3588-1.2.1

Recommended rkllm parameters

Useful links:

whaoyang/
gemma-3-4b-novision-quant-rk3588-1.2.1

Recommended `rkllm` parameters