You can deploy this model and start earning money today!

radeon6970/
qwen3-vl-embedding-2b_w8a8_rk3588

Name: qwen3-vl-embedding-2b_w8a8_rk3588
Availability: InStock

Views

No views yet

Slopped primarily with Gemini 3.5 Flash harnessed with Antigravity. It appears to work properly, please don't ask me tricky questions

Qwen3-VL-Embedding-2B (W8A8 Quantized for RK3588 NPU)

1. Model Specifications & Quantization Details

Model Name: Qwen3-VL-Embedding-2B
Text Transformer Precision: W8A8 (8-bit weights and 8-bit activations). Compiled using Rockchip's RKLLM toolkit (v1.2.3).
Vision Encoder Precision: FP16 (16-bit half precision) to maintain visual representation fidelity and prevent quantization loss in early feature maps. Compiled using rknn-toolkit2 (v2.2.0).
Quantization Calibration Size: Calibrated using 200 representative MS-COCO image/caption pairs.

2. Alignment Performance (vs. PyTorch FP32 CPU Reference)

Evaluated over $100$ independent random MS-COCO validation samples, the calibrated W8A8 on-device model achieves reasonable similarity to the unquantized PyTorch FP32 baseline:

Embedding Type	Mean Cosine Similarity	Standard Deviation
Text Embeddings	0.9591	0.0081
Multimodal Image Embeddings	0.9338	0.0244

3. Vision Encoder Mechanics

Input Dimension: [1, 448, 448, 3] (NHWC representation, uint8).
Output Tensors: 4 distinct outputs corresponding to deepstacked visual layers. Each output tensor yields a dimension of [1, 196, 2048] floats (where 196 represents the visual patch token length).
Total Visual Tokens: Interleaved output yields a combined $784$ visual tokens of size $2048$ floats ($196 \text{ tokens} \times 4 \text{ layers}$).

4. On-Device Usage Guide (Barebone C++ Implementation)

To load and extract embeddings on-device, compile a C++ binary linking librkllmrt.so (LLM runtime) and librknnrt.so (Vision RKNN runtime). The source file qwen3_embed.cpp must implement the following key elements:

A. Last-Token Pooling Callback (`llm_cb`)

Configure the LLM parameters to output the last hidden layer and copy only the final sequence token representation inside the callback:

cpp

#include "rkllm.h"
#include <cstdio>
#include <cstring>

static float* g_output_buffer = nullptr;
static int g_embedding_dim = 2048;

int llm_cb(RKLLMResult *result, void *userdata, LLMCallState state) {
    if (state == RKLLM_RUN_NORMAL) {
        const RKLLMResultLastHiddenLayer &lhl = result->last_hidden_layer;
        int tokens = lhl.num_tokens;
        int dim = lhl.embd_size / sizeof(float);
        
        if (lhl.hidden_states != nullptr && tokens > 0) {
            // Manual last-token pooling
            int last_token_idx = tokens - 1;
            float* last_token_emb = (float*)lhl.hidden_states + last_token_idx * dim;
            
            // Save the pooled embedding vector
            memcpy(g_output_buffer, last_token_emb, dim * sizeof(float));
        }
    }
    return 0;
}

B. Vision Token Interleaving Loop

Since the vision RKNN model returns 4 deepstack layers separately, they must be interleaved token-by-token (not sequentially concatenated) before sending them to the LLM:

cpp

#include "rknn_api.h"
#include <vector>

bool interleave_vision_outputs(rknn_context ctx, int n_outputs, std::vector<rknn_output> &outputs, float* interleaved_dest) {
    int model_image_tokens = 196;   // Visual patches
    int embed_size = 2048;          // Vector dimension
    
    for (int i = 0; i < model_image_tokens; i++) {
        for (int j = 0; j < n_outputs; j++) {
            float* source_ptr = (float*)(outputs[j].buf) + i * embed_size;
            float* dest_ptr = interleaved_dest + (i * n_outputs * embed_size) + (j * embed_size);
            memcpy(dest_ptr, source_ptr, embed_size * sizeof(float));
        }
    }
    return true;
}

C. EOS Token Template Registration

Register the chat template to force the model to evaluate the <|endoftext|> token (ID 151643) at the end of the user instruction sequence:

cpp

void setup_model_and_template(LLMHandle &handle, const char* model_path) {
    RKLLMParam p = rkllm_createDefaultParam();
    p.model_path = model_path;
    p.max_context_len = 512;
    p.max_new_tokens = 1; // Unused during embedding extraction
    
    // Define visual markers
    p.img_start = "<|vision_start|>";
    p.img_end   = "<|vision_end|>";
    p.img_content = "<|image_pad|>";
    
    rkllm_init(&handle, &p, llm_cb);
    
    // Explicitly enforce EOS pooling boundary
    rkllm_set_chat_template(
        handle,
        "Represent the user's input.", // System instruction
        "<|im_start|>user\n",          // Prefix
        "<|im_end|>\n<|im_start|>assistant\n<|endoftext|>" // Postfix containing EOS
    );
}

D. Executing Embeddings

cpp

// For Text Embeddings:
RKLLMInput inp;
memset(&inp, 0, sizeof(inp));
inp.input_type = RKLLM_INPUT_PROMPT;
inp.role = "user";
inp.prompt_input = "Your text here";

RKLLMInferParam ip;
memset(&ip, 0, sizeof(ip));
ip.mode = RKLLM_INFER_GET_LAST_HIDDEN_LAYER;

rkllm_run(handle, &inp, &ip, nullptr);

// For Multimodal Image Embeddings:
RKLLMInput inp_img;
memset(&inp_img, 0, sizeof(inp_img));
inp_img.input_type = RKLLM_INPUT_MULTIMODAL;
inp_img.role = "user";
inp_img.multimodal_input.prompt = "<image>";
inp_img.multimodal_input.image_embed = interleaved_dest; // Interleaved tokens buffer
inp_img.multimodal_input.n_image_tokens = 196;
inp_img.multimodal_input.n_image = 1;
inp_img.multimodal_input.image_width = 448;
inp_img.multimodal_input.image_height = 448;

rkllm_run(handle, &inp_img, &ip, nullptr);

radeon6970/qwen3-vl-embedding-2b_w8a8_rk3588