You can deploy this model and start earning money today!

JDONE-Research/
AIOne-Agent-52B-A36B-it-NVFP4A16

Name: AIOne-Agent-52B-A36B-it-NVFP4A16
Availability: InStock

Views

No views yet

AIOne-Agent

AIOne-Agent-52B-A36B-it-NVFP4A16

NVFP4A16 weight-only quantization of AIOne-Agent-52B-A36B-it for memory-efficient deployment on NVIDIA Blackwell GPUs with vLLM.

Model Description

AIOne-Agent-52B-A36B-it-NVFP4A16 is a 4-bit weight-only quantization of JDONE-Research/AIOne-Agent-52B-A36B-it, a Korean multimodal Mixture-of-Experts model based on Gemma 4 31B IT (text backbone ~52 B, ~36 B active per token). The text-decoder dense Linear weights are compressed to NVFP4 (4-bit floating point), while activations stay in BF16 — so the model preserves the language and reasoning quality of the source model while drastically reducing VRAM footprint.

Vision tower preserved. The SigLIP-style vision tower, the multimodal embedding projection, and the LM head are kept in BF16 to keep image and video understanding bit-exact with the unquantized model.

The name follows the Gemma 4 convention (google/gemma-4-26B-A4B-it) with an -NVFP4A16 suffix to denote 4-bit weight / 16-bit activation quantization: 52B is the text-backbone parameter count, A36B is the per-token active parameter count, and the vision encoder (0.57 B) is reported separately.

Key Capabilities

Korean reasoning and instruction following.
Image understanding (caption, VQA, document understanding).
Video understanding (frame-by-frame reasoning).
Long-context document QA in Korean.
Bilingual: Korean (primary) + English.

All capabilities are inherited from AIOne-Agent-52B-A36B-it; only the storage precision of the dense text-decoder weights changes.

Quick Start

Transformers

Note. When loading the compressed model with transformers you may want to set TORCHDYNAMO_DISABLE=1 (or raise torch._dynamo.config.cache_size_limit) because the FP4 unpack kernel is torch.compile-decorated and Gemma 4's many distinct Linear shapes can otherwise trip the default recompile limit. vLLM does not need this — it uses native NVFP4 kernels.

python

import os
os.environ["TORCHDYNAMO_DISABLE"] = "1"

import torch
from transformers import AutoProcessor, Gemma4ForConditionalGeneration

MODEL_ID = "JDONE-Research/AIOne-Agent-52B-A36B-it-NVFP4A16"

model = Gemma4ForConditionalGeneration.from_pretrained(
    MODEL_ID,
    torch_dtype="auto",
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(MODEL_ID)

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "file:///path/to/image.jpg"},
            {"type": "text", "text": "이 사진에 무엇이 보이나요? 한국어로 답해주세요."},
        ],
    },
]

inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_tensors="pt",
    return_dict=True,
).to(model.device)

generated = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(
    processor.tokenizer.decode(
        generated[0, inputs.input_ids.shape[1]:], skip_special_tokens=True
    )
)

vLLM (recommended for serving)

bash

vllm serve JDONE-Research/AIOne-Agent-52B-A36B-it-NVFP4A16 \
    --quantization compressed-tensors \
    --kv-cache-dtype fp8_e4m3 \
    --max-model-len 32768 \
    --tensor-parallel-size 1

yaml

model: JDONE-Research/AIOne-Agent-52B-A36B-it-NVFP4A16
quantization: compressed-tensors  # NVFP4A16 weights
kv_cache_dtype: fp8_e4m3          # FP8 KV cache (optional)
gpu_memory_utilization: 0.9
max_model_len: 32768
tensor_parallel_size: 1
enable_prefix_caching: true
enable_chunked_prefill: true

Quantization Recipe

This model was quantized with llm-compressor using the compressed-tensors nvfp4-pack-quantized format.

yaml

default_stage:
  default_modifiers:
    GPTQModifier:
      targets: [Linear]
      ignore:
        - lm_head
        - "re:.*\\.experts\\..*"
        - "re:.*\\.router\\..*"
        - "re:.*vision_tower.*"
        - "re:.*embed_vision.*"
        - "re:.*multi_modal_projector.*"
      scheme: NVFP4A16
      block_size: 128
      dampening_frac: 0.01
      actorder: static
      offload_hessians: false
      sequential_targets: [Gemma4TextDecoderLayer]

Setting	Value
Quantization tool	`llm-compressor` 0.10.x
Format	`compressed-tensors` (`nvfp4-pack-quantized`)
Weight precision	NVFP4 (4-bit FP), block size 128
Activation precision	BF16 (weight-only quantization)
Algorithm	GPTQ (sequential per-layer Hessian fitting, `dampening_frac=0.01`, `actorder=static`)
Calibration data	256 chat-formatted samples × 2,048 tokens drawn from `JDONE-Research/Ko-OpenThoughts3-1.2M`

What is quantized vs. preserved

Component	Precision	Notes
Text-decoder dense MLPs (`mlp.{gate,up,down}_proj`)	NVFP4A16	Quantized via GPTQ
Text-decoder attention projections (`self_attn.{q,k,v,o}_proj`)	NVFP4A16	Quantized via GPTQ
MoE expert weights (`experts.gate_up_proj`, `experts.down_proj`)	BF16	Stored as 3-D `nn.Parameter`; left BF16 for stability
MoE router (`router.proj`, `router.scale`, `router.per_expert_scale`)	BF16	Routing-critical, kept dense
LM head (`lm_head`)	BF16	Tied with `embed_tokens`
Vision tower (`vision_tower.*`)	BF16	Preserves multimodal quality
Vision embedding (`embed_vision.*`)	BF16	Preserves multimodal quality
Multimodal projector (`multi_modal_projector.*`)	BF16	Preserves multimodal quality
RMSNorms / scalars	BF16	Not Linear; not a quantization target

Model Specs

Field	Value
Architecture	`Gemma4ForConditionalGeneration`
Source model	`JDONE-Research/AIOne-Agent-52B-A36B-it`
Base model	`google/gemma-4-31B-it`
Text backbone parameters	51.51 B → 52 B (in name)
Active parameters per token (text)	35.90 B → A36B (in name) (dense MLP always on + top-2 of 8 experts + attention)
Vision tower	0.57 B (BF16, unchanged)
MM projector	0.01 B (BF16, unchanged)
MoE config	`num_experts=8`, `top_k=2`, `moe_intermediate_size=2688`
Modality	Text + Image + Video → Text
Weight precision (text-decoder dense Linear)	NVFP4 (4-bit FP)
Weight precision (everything else)	BF16
Activation precision	BF16
Context length	256K (inherited from base)
Languages	Korean (primary), English

Hardware Recommendations

NVFP4 weight-only quantization is designed for NVIDIA Blackwell-class GPUs (sm120, e.g. RTX PRO 6000, B100/B200) and is supported by recent vLLM builds with --quantization compressed-tensors. The model also runs on Hopper (H100/H200) at BF16 dequantized speed.

GPU	VRAM	Tensor parallel	Notes
1 × H200 (141 GB)	OK	TP=1	Largest single-GPU option (recommended)
2 × H100 (80 GB)	OK	TP=2	Splits experts across cards
1 × RTX PRO 6000 (96 GB)	OK	TP=1	Native NVFP4 inference (Blackwell)

Files in This Repository

File	Description
`model-*.safetensors`	Quantized weights in compressed-tensors NVFP4 format
`model.safetensors.index.json`	Sharded weight index
`config.json`	Model config including `quantization_config`
`tokenizer.json`, `tokenizer_config.json`	Tokenizer
`chat_template.jinja`	Gemma 4 chat template
`generation_config.json`	Default generation parameters
`processor_config.json`	Multimodal processor config
`recipe.yaml`	Quantization recipe

Intended Use

Korean enterprise agent backend (long-context tool use, RAG, multi-turn reasoning).
Image and video understanding with Korean output.
Document QA in Korean.

Out-of-Scope Use

Sole-source decision-making with legal consequences.
Automated use of force or coercive control based purely on this model's output.
Any media analysis that infringes on personal privacy, image rights, or applicable data-protection laws.

License

This model is released under the Apache License 2.0 license.

Commercial use, redistribution, and modification are permitted, subject to the conditions of the Apache 2.0 License (including preservation of copyright notices).
Provided "as is" without warranties or conditions of any kind.

Citation

bibtex

@misc{aione_agent_52b_a36b_it_nvfp4,
  title        = {AIOne-Agent-52B-A36B-it-NVFP4A16: NVFP4 Weight-Only Quantization of a Korean Sparse-MoE Multimodal Model},
  author       = {JDONE Research},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/JDONE-Research/AIOne-Agent-52B-A36B-it-NVFP4A16}},
  note         = {Quantized with llm-compressor}
}