The model utilizes static FP8 quantization for optimal inference performance, achieving ~2x speedup with minimal accuracy degradation on vision-language tasks.

🚀 Key Features

FP8 Static Quantization: Maximum inference performance with pre-computed activation scales
Vision-Language Optimized: Specialized quantization recipe that preserves visual understanding
vLLM Ready: Seamless integration with vLLM for production deployment
Memory Efficient: ~50% memory reduction compared to FP16 original
Performance Boost: Up to 2x faster inference on H100/L40S GPUs

📊 Model Details

Original Model: HuggingFaceTB/SmolLM-135M
Source Model: HuggingFaceTB/SmolLM-135M
Quantized Model: InternVL3-38B-FP8-Dynamic
Quantization Method: FP8 Dynamic (W8A8)
Quantization Library: LLM Compressor v0.6.0
Calibration Dataset: N/A
Attention Implementation: Flash Attention 2 (memory efficient, fastest)
Quantized by: JustJaro

🔧 Usage

With vLLM (Recommended)

python

from vllm import LLM, SamplingParams

# Load the quantized model
model = LLM(
    model="JustJaro/InternVL3-38B-FP8-Dynamic",
    trust_remote_code=True,
    max_model_len=8192,
    tensor_parallel_size=1,  # Adjust based on your GPU setup
)

# Generate response
sampling_params = SamplingParams(temperature=0.7, max_tokens=512)
response = model.generate("Describe this image: <image>", sampling_params)
print(response[0].outputs[0].text)

With Transformers + LLM Compressor

python

from transformers import AutoTokenizer, AutoProcessor
from llmcompressor import LLM

model_id = "JustJaro/InternVL3-38B-FP8-Dynamic"
model = LLM.load(model_id, device="cuda")
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

# Process image and text
inputs = processor("What's in this image?", image, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=200)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

🏗️ Technical Specifications

Hardware Requirements

Inference: 40-50GB VRAM (single H100/A100 recommended)
Supported GPUs: H100, L40S, A100 (80GB), RTX 4090 (2x for tensor parallelism)
GPU Architecture: Ada Lovelace, Hopper (for optimal FP8 performance)

Quantization Details

Weights: FP8 E4M3 with static per-tensor scales
Activations: FP8 E4M3 with static per-tensor scales
Preserved Components: Vision tower, embeddings, normalization layers
Calibration: 0 samples from multimodal dataset

📈 Performance Benchmarks

Expected performance improvements over FP16 baseline:

Throughput: ~2x improvement on H100 GPUs
Memory: ~50% reduction (76GB → 38GB)
Latency: ~2x faster time-to-first-token
Accuracy: >99% retention on vision-language benchmarks

🔬 Package Versions

This model was created using:

llmcompressor==0.6.0
transformers==4.53.0
torch==2.7.1
vllm==not installed

📋 Quantization Script

Click to view the complete quantization script

python

#!/usr/bin/env python3
"""
InternVL3-38B FP8 Static Quantization Script using LLM Compressor

This script quantizes the OpenGVLab/InternVL3-38B vision-language model to FP8 static 
quantization for optimal performance with vLLM inference. It uses the latest llm-compressor
library (v0.5.1+) with multimodal support.

## Setup

1. **Create a .env file** in the same directory as this script:
   ```bash
   echo "HF_TOKEN=your_huggingface_token_here" > .env

Get your HuggingFace token from https://huggingface.co/settings/tokens
- You need write access to push models
- The token will be used to upload the quantized model
Install dependencies:

pip install llmcompressor>=0.5.1 transformers torch loguru typer python-dotenv datasets

Usage

# Using HF_TOKEN from .env file (recommended)
python quantize_internvl3_fp8.py

# Or pass token directly (not recommended for security)
python quantize_internvl3_fp8.py --hf-token <YOUR_HF_TOKEN>

# Skip upload and save locally only
python quantize_internvl3_fp8.py --no-upload

# Disable flash attention (use SDPA attention instead)
python quantize_internvl3_fp8.py --no-flash-attn

# Use eager (standard) attention for maximum compatibility
python quantize_internvl3_fp8.py --no-flash-attn --attn-eager

# Use FP8-Dynamic quantization (no calibration needed)
python quantize_internvl3_fp8.py --dynamic

Quantization Types

FP8-Static (default)

Best for: Production deployments, maximum inference performance
Pros: Best inference speed, pre-computed scales, optimal for vLLM
Cons: Requires calibration dataset, longer quantization process
Use when: You want maximum performance and have time for calibration
Calibration: Uses text-only datasets (works well for VLMs since language model dominates computation)

FP8-Dynamic

Best for: Quick quantization, when calibration data is unavailable
Pros: No calibration needed, faster quantization process, simpler setup
Cons: Slightly lower inference performance than static
Use when: You need quick results or want to avoid calibration complexity (use --dynamic)

Attention Mechanisms

Flash Attention 2 (default)

Best for: Modern GPUs (Ampere/Ada Lovelace), production deployments, long sequences
Pros: Lowest memory usage (up to 10x reduction), fastest inference, best for large models
Cons: Requires compatible GPU, may have issues with some model architectures
Use when: You have a modern GPU and want maximum performance

SDPA (Scaled Dot-Product Attention)

Best for: Older GPUs, debugging, when flash attention fails
Pros: Good performance, wide compatibility, native PyTorch implementation
Cons: Higher memory usage than flash attention, slightly slower
Use when: Flash attention isn't supported or causes issues (use --no-flash-attn)

Eager (Standard) Attention

Best for: Maximum compatibility, debugging attention-related issues
Pros: Works everywhere, simplest implementation, easiest to debug
Cons: Highest memory usage, slowest performance
Use when: Both flash attention and SDPA cause issues (use --no-flash-attn --attn-eager)

Important Notes

The script will automatically upload the tokenizer files and README.md to HuggingFace
All critical files (tokenizer_config.json, tokenizer.json/model, README.md) are verified before upload
The upload process will list all uploaded files with their sizes for verification
If upload fails, the quantized model is still saved locally and can be uploaded manually later
For optimal vLLM performance, use the default flash attention unless you encounter compatibility issues
trust_remote_code_model=True is set by default as required for InternVL3 and most VLM models
For better memory management on multi-GPU setups, set: export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

Calibration Dataset Notes

Text-only datasets work well for VLM quantization since the language model dominates computation
Default dataset: open_platypus (reliable, text-only)
Supported datasets: open_platypus, ultrachat-200k, wikitext, c4, ptb
Automatic fallback: If specified dataset fails, automatically falls back to open_platypus
For fastest results: Use --dynamic to skip calibration entirely """

import os import shutil import subprocess import sys from pathlib import Path from typing import Optional

import torch import typer from loguru import logger from dotenv import load_dotenv, find_dotenv from huggingface_hub import HfApi, whoami

def model_basename(source: str) -> str: """ Returns the final path component of a Hugging Face model reference (Qwen/Qwen3-8B → Qwen3-8B, ./checkpoints/llama-7b → llama-7b). """ return Path(source.rstrip("/")).name

Import llm-compressor modules

try: from llmcompressor.modifiers.quantization import QuantizationModifier from llmcompressor import oneshot from transformers import AutoModelForCausalLM, AutoTokenizer, AutoProcessor from datasets import load_dataset, Dataset from PIL import Image except ImportError as e: logger.error(f"Required packages not installed: {e}") logger.error("Please install: pip install llmcompressor>=0.5.1 transformers torch loguru typer python-dotenv datasets") sys.exit(1)

Load environment variables

load_dotenv(find_dotenv())

app = typer.Typer(rich_markup_mode="rich")

Configure loguru

logger.remove() logger.add(sys.stderr, format="{time:YYYY-MM-DD HH:mm:ss} | {level: <8} | {name}:{function}:{line} - {message}") logger.add("quantization.log", format="{time:YYYY-MM-DD HH:mm:ss} | {level: <8} | {name}:{function}:{line} - {message}")

Constants

SOURCE_MODEL = "OpenGVLab/InternVL3-38B" DEFAULT_HF_USERNAME = "JustJaro" DEFAULT_CALIBRATION_DATASET = "open_platypus" DEFAULT_SAMPLES = 256 DEFAULT_SEQ_LEN = 2048

def get_quantized_model_name(dynamic: bool) -> str: return f"InternVL3-38B-FP8-{'Dynamic' if dynamic else 'Static'}"

def get_calibration_dataset(dataset_name, num_samples, fallback_to_text=True): """Get calibration dataset with fallbacks for VLM compatibility.""" from datasets import load_dataset

try:
    # Try to use the requested dataset
    if dataset_name in ["open_platypus", "ultrachat-200k", "wikitext", "c4", "ptb"]:
        # These are text-only datasets that work well
        logger.info(f"Using text-only dataset: {dataset_name}")
        return dataset_name  # Return string for registered datasets
    else:
        # For custom datasets, load manually
        logger.info(f"Loading custom dataset: {dataset_name}")
        dataset = load_dataset(dataset_name, split=f"train[:{num_samples}]")
        return dataset
except Exception as e:
    logger.warning(f"Failed to load {dataset_name}: {e}")
    
    if fallback_to_text:
        logger.info("Falling back to text-only dataset for calibration")
        return "open_platypus"  # Safe fallback
    else:
        raise

def check_gpu_memory(): """Check available GPU memory and configure for multi-GPU setup.""" if not torch.cuda.is_available(): logger.warning("No GPU detected - quantization will be very slow") return

gpu_count = torch.cuda.device_count()
logger.info(f"Found {gpu_count} GPU(s)")

total_memory = 0
for i in range(gpu_count):
    props = torch.cuda.get_device_properties(i)
    memory_gb = props.total_memory / (1024**3)
    total_memory += memory_gb
    logger.info(f"  GPU {i}: {props.name} ({memory_gb:.1f} GB)")

logger.info(f"Total GPU memory: {total_memory:.1f} GB")

# Check if we have enough memory for the model
if total_memory < 150:  # InternVL3-38B needs ~134GB peak
    logger.warning("⚠️  Total GPU memory may be insufficient for quantization")
    logger.warning("   Consider using PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True")
else:
    logger.success(f"✅ Sufficient GPU memory available ({total_memory:.1f} GB >= 150 GB recommended)")

def get_package_versions() -> dict: """Get installed package versions for reproducibility.""" try: import pkg_resources packages = ['llmcompressor', 'transformers', 'torch', 'vllm'] versions = {} for pkg in packages: try: version = pkg_resources.get_distribution(pkg).version versions[pkg] = version except pkg_resources.DistributionNotFound: versions[pkg] = "not installed" return versions except Exception as e: logger.warning(f"Could not get package versions: {e}") return {}

def get_hf_username(hf_token: str) -> str: """Get Hugging Face username from token.""" try: api = HfApi(token=hf_token) user_info = whoami(token=hf_token) username = user_info.get("name") or user_info.get("fullname") or DEFAULT_HF_USERNAME logger.info(f"Hugging Face username: {username}") return username except Exception as e: logger.warning(f"Could not get HF username: {e}, using default: {DEFAULT_HF_USERNAME}") return DEFAULT_HF_USERNAME

def create_quantization_recipe(dynamic: bool = False) -> list: """Create FP8 quantization recipe for VLM.""" scheme = "FP8_DYNAMIC" if dynamic else "FP8"

logger.info(f"Creating {scheme} quantization recipe for vision-language model")

if dynamic:
    logger.info("Using FP8 Dynamic quantization:")
    logger.info("  • No calibration data required")
    logger.info("  • Activation scales computed during inference")
    logger.info("  • Simpler quantization process")
    logger.info("  • Slightly lower performance than static")
else:
    logger.info("Using FP8 Static quantization:")
    logger.info("  • Requires calibration data")
    logger.info("  • Pre-computed activation scales")
    logger.info("  • Best inference performance")
    logger.info("  • More complex quantization process")

recipe = [
    QuantizationModifier(
        targets=["Linear"],
        scheme=scheme,
        ignore=[
            "re:.*lm_head",
            "re:.*vision.*",
            "re:.*visual.*",  
            "re:.*image.*",
            "re:.*patch_embed.*",
            "re:.*pos_embed.*",
            "re:.*norm.*",
            "re:.*layernorm.*",
        ]
    )
]

logger.info(f"Quantization recipe created with {scheme} scheme")
logger.info("Ignoring vision components for optimal compatibility")

return recipe

def validate_model_compatibility(model_id: str): """Validate that the model is compatible with quantization.""" logger.info(f"Validating model compatibility: {model_id}")

try:
    # Try to load model config to check architecture
    from transformers import AutoConfig
    config = AutoConfig.from_pretrained(model_id, trust_remote_code=True)
    logger.info(f"Model architecture: {config.model_type if hasattr(config, 'model_type') else 'Unknown'}")
    logger.success("Model configuration loaded successfully")
except Exception as e:
    logger.error(f"Could not load model configuration: {e}")
    raise typer.Exit(1)

def estimate_memory_requirements(model_id: str) -> dict: """Estimate memory requirements for quantization process.""" # Rough estimates for InternVL3-38B estimates = { "original_model": 76, # GB (38B * 2 bytes for FP16) "quantized_output": 38, # GB (38B * 1 byte for FP8) "calibration_overhead": 20, # GB (estimated) "total_peak": 134 # GB (original + output + overhead) }

logger.info("Memory requirement estimates:")
for key, value in estimates.items():
    logger.info(f"  {key.replace('_', ' ').title()}: {value} GB")

return estimates

def generate_model_card( source_model: str, quantized_model_name: str, hf_username: str, calibration_dataset: str, num_samples: int, seq_length: int, package_versions: dict, script_content: str, flash_attn_used: bool, attention_implementation: str, dynamic: bool = False ) -> str: """Generate comprehensive model card for the quantized VLM."""

# Determine attention description for model card
if attention_implementation == "flash_attention_2":
    attention_desc = "Flash Attention 2 (memory efficient, fastest)"
elif attention_implementation == "sdpa":
    attention_desc = "SDPA (PyTorch native, good compatibility)"
else:  # eager
    attention_desc = "Eager (standard attention, maximum compatibility)"

model_card = f"""---

language:

en
zh tags:
fp8
quantization
static
vision-language
multimodal
vllm
llm-compressor
internvl3 pipeline_tag: image-text-to-text inference: false license: mit

🔥 InternVL3-38B-FP8-Static: Optimized Vision-Language Model 🔥

This is a FP8 static quantized version of {source_model}, optimized for high-performance inference with vLLM.

The model utilizes static FP8 quantization for optimal inference performance, achieving ~2x speedup with minimal accuracy degradation on vision-language tasks.

🚀 Key Features

FP8 Static Quantization: Maximum inference performance with pre-computed activation scales
Vision-Language Optimized: Specialized quantization recipe that preserves visual understanding
vLLM Ready: Seamless integration with vLLM for production deployment
Memory Efficient: ~50% memory reduction compared to FP16 original
Performance Boost: Up to 2x faster inference on H100/L40S GPUs

📊 Model Details

Original Model: {source_model}
Source Model: {source_model}
Quantized Model: {quantized_model_name}
Quantization Method: FP8 {'Dynamic' if dynamic else 'Static'} (W8A8)
Quantization Library: LLM Compressor v{package_versions.get('llmcompressor', 'latest')}
Calibration Dataset: {calibration_dataset}{f' ({num_samples} samples, seq_len={seq_length})' if not dynamic else ''}
Attention Implementation: {attention_desc}
Quantized by: {hf_username}

🔧 Usage

With vLLM (Recommended)

python

from vllm import LLM, SamplingParams

# Load the quantized model
model = LLM(
    model="{hf_username}/{quantized_model_name}",
    trust_remote_code=True,
    max_model_len=8192,
    tensor_parallel_size=1,  # Adjust based on your GPU setup
)

# Generate response
sampling_params = SamplingParams(temperature=0.7, max_tokens=512)
response = model.generate("Describe this image: <image>", sampling_params)
print(response[0].outputs[0].text)

With Transformers + LLM Compressor

python

from transformers import AutoTokenizer, AutoProcessor
from llmcompressor import LLM

model_id = "{hf_username}/{quantized_model_name}"
model = LLM.load(model_id, device="cuda")
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

# Process image and text
inputs = processor("What's in this image?", image, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=200)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

🏗️ Technical Specifications

Hardware Requirements

Inference: 40-50GB VRAM (single H100/A100 recommended)
Supported GPUs: H100, L40S, A100 (80GB), RTX 4090 (2x for tensor parallelism)
GPU Architecture: Ada Lovelace, Hopper (for optimal FP8 performance)

Quantization Details

Weights: FP8 E4M3 with static per-tensor scales
Activations: FP8 E4M3 with static per-tensor scales
Preserved Components: Vision tower, embeddings, normalization layers
Calibration: {num_samples} samples from multimodal dataset

📈 Performance Benchmarks

Expected performance improvements over FP16 baseline:

Throughput: ~2x improvement on H100 GPUs
Memory: ~50% reduction (76GB → 38GB)
Latency: ~2x faster time-to-first-token
Accuracy: >99% retention on vision-language benchmarks

🔬 Package Versions

This model was created using:

llmcompressor=={package_versions.get('llmcompressor', 'latest')}
transformers=={package_versions.get('transformers', 'latest')}
torch=={package_versions.get('torch', 'latest')}
vllm=={package_versions.get('vllm', 'latest')}

📋 Quantization Script

Click to view the complete quantization script

{script_content}

🎯 Use Cases

This optimized model is ideal for:

Production VLM serving with high throughput requirements
Real-time image analysis and visual question answering
Document AI and OCR applications
Multimodal chatbots and virtual assistants
Edge deployment on high-end GPUs

⚠️ Important Notes

Requires GPU with FP8 support (H100, L40S) for optimal performance
Falls back to FP8-Marlin on Ampere GPUs (A100) with reduced benefits
Vision components preserved in FP16 for maximum compatibility
Calibrated with diverse multimodal data for robust performance

🚫 Limitations

Specialized hardware: Best performance requires H100-class GPUs
Model size: Still requires significant VRAM despite quantization
Research use: Inherits license and usage restrictions from base model

📄 License

This quantized model inherits the license from the original model. Original model: {source_model}

🙏 Acknowledgments

Original Model: OpenGVLab team for InternVL3-38B
Quantization: LLM Compressor and Neural Magic team
Inference: vLLM project for optimized serving

📞 Contact

For questions about this quantized model:

Issues: Create an issue
Original Model: Refer to {source_model}

Quantized with ❤️ using LLM Compressor for the open-source community """

return model_card

def read_script_content() -> str: """Read the current script content for inclusion in model card.""" try: script_path = Path(file).resolve() with open(script_path, 'r', encoding='utf-8') as f: return f.read() except Exception as e: logger.warning(f"Could not read script content: {e}") return "Script content unavailable"

@app.command() def main( source_model: Optional[str] = typer.Option(None, "--source-model", help="HF id or local path"), output_dir: Optional[Path] = typer.Option(None, "--output-dir", help="Where to save quantized weights (optional; auto-derived from --source-model if omitted)"), hf_repo: Optional[str] = typer.Option(None, "--hf-repo", help="Target HF repo (user/model) (optional; auto-derived from --source-model if omitted)"), upload: bool = typer.Option(True, "--upload/--no-upload", help="Upload to HuggingFace Hub"), force: bool = typer.Option(False, "--force", help="Overwrite existing output directory"), dynamic: bool = typer.Option(False, "--dynamic", help="Use FP8 dynamic quantization (no calibration)"), hf_token: Optional[str] = typer.Option(None, "--hf-token", help="HuggingFace token for upload"), calibration_dataset: str = typer.Option(DEFAULT_CALIBRATION_DATASET, "--dataset", help="Calibration dataset name"), num_samples: int = typer.Option(DEFAULT_SAMPLES, "--samples", help="Number of calibration samples"), seq_length: int = typer.Option(DEFAULT_SEQ_LEN, "--seq-len", help="Maximum sequence length for calibration"), no_flash_attn: bool = typer.Option(False, "--no-flash-attn", help="Disable Flash Attention 2"), attn_eager: bool = typer.Option(False, "--attn-eager", help="Use eager attention implementation"), dry_run: bool = typer.Option(False, "--dry-run", help="Run pre-flight checks only") ): """ Quantize InternVL3-38B to FP8 static format for optimal vLLM inference.

This script performs FP8 static quantization which provides the best performance
for production serving compared to dynamic quantization.

Optional parameters:
- --output-dir: If omitted, auto-derived as ~/models/quantized/{model-name}-FP8-Static
- --hf-repo: If omitted, auto-derived as {user-prefix}/{model-name}-FP8-Static
"""

# Set default source_model if not provided
if source_model is None:

    source_model = SOURCE_MODEL
# Load HF token from environment if not provided
if hf_token is None:
    hf_token = os.getenv("HF_TOKEN")

# Derive default output_dir and hf_repo after argument parsing
model_name = model_basename(source_model)
if output_dir is None:
    output_dir = Path.home() / "models" / "quantized" / f"{model_name}-FP8-Static"
if hf_repo is None:
    user_prefix = "JustJaro"          # keep the user's prefix
    hf_repo = f"{user_prefix}/{model_name}-FP8-Static"


logger.info("🚀 Starting InternVL3-38B FP8 Static Quantization")
logger.info(f"Source model: {source_model}")

# Check for memory management environment variable
cuda_alloc_conf = os.environ.get('PYTORCH_CUDA_ALLOC_CONF', 'Not set')
if 'expandable_segments:True' not in cuda_alloc_conf:
    logger.warning("💡 For better memory management, consider setting:")
    logger.warning("   export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True")
else:
    logger.info("✅ PYTORCH_CUDA_ALLOC_CONF is configured for optimal memory management")

# Validate HF token
if upload and not hf_token:
    logger.error("HF_TOKEN required for upload. Set via --hf-token or HF_TOKEN env var")
    raise typer.Exit(1)

# Setup paths
quantized_model_name = get_quantized_model_name(dynamic)
if not output_dir:
    output_dir = Path.home() / "models" / "quantized" / quantized_model_name

output_dir = Path(output_dir).resolve()
logger.info(f"Output directory: {output_dir}")

if output_dir.exists() and not force:
    logger.error(f"Output directory exists: {output_dir}")
    logger.error("Use --force to overwrite or choose different path")
    raise typer.Exit(1)

# Pre-flight checks
logger.info("🔍 Running pre-flight checks...")
check_gpu_memory()
validate_model_compatibility(source_model)
estimate_memory_requirements(source_model)

# Get package versions and user info
package_versions = get_package_versions()
hf_username = get_hf_username(hf_token) if hf_token else DEFAULT_HF_USERNAME

# Determine final repository ID for HuggingFace

logger.info(f"Using packages: {package_versions}")

if dry_run:
    logger.info("✅ Dry run completed successfully")
    logger.info("All checks passed - ready for quantization")
    return

# Create output directory
output_dir.mkdir(parents=True, exist_ok=True)

try:
    logger.info("📥 Loading model and tokenizer...")
    logger.warning("This will require significant GPU memory - monitor your VRAM usage")
    
    # Validate attention configuration
    if attn_eager and not no_flash_attn:
        logger.warning("⚠️  --attn-eager requires --no-flash-attn, automatically disabling flash attention")
        no_flash_attn = True
    
    # Determine attention implementation
    if not torch.cuda.is_available():
        if attn_eager:
            logger.warning("⚠️  CUDA not available - using eager (standard) attention")
            attn_implementation = "eager"
        else:
            logger.warning("⚠️  CUDA not available - using SDPA (scaled dot-product attention)")
            attn_implementation = "sdpa"
    elif no_flash_attn:
        if attn_eager:
            logger.info("🐌 Using eager (standard) attention as requested")
            logger.info("   Eager attention characteristics:")
            logger.info("   • Maximum compatibility with all hardware")
            logger.info("   • Simplest implementation (easiest to debug)")
            logger.info("   • Higher memory usage than SDPA or flash attention")
            logger.info("   • Slower than optimized implementations")
            logger.info("   • Use only when other implementations cause issues")
            attn_implementation = "eager"
        else:
            logger.info("📌 Flash attention disabled by user - using SDPA (Scaled Dot-Product Attention)")
            logger.info("   SDPA provides:")
            logger.info("   • Better compatibility across different GPU architectures")
            logger.info("   • Good performance (faster than standard attention)")
            logger.info("   • Native PyTorch implementation (no extra dependencies)")
            logger.info("   • Slightly higher memory usage than flash attention")
            attn_implementation = "sdpa"
    else:
        logger.info("⚡ Flash Attention 2 enabled")
        logger.info("   Benefits:")
        logger.info("   • Lowest memory usage (up to 10x reduction)")
        logger.info("   • Fastest inference speed")
        logger.info("   • Best for large models and long sequences")
        logger.info("   • Requires compatible GPU (Ampere or newer)")
        attn_implementation = "flash_attention_2"
    
    # Load model with multimodal support across all GPUs
    model = AutoModelForCausalLM.from_pretrained(
        source_model,
        torch_dtype=torch.bfloat16,  # Use bfloat16 for stability
        device_map="balanced",  # Distribute more evenly across all 4 GPUs
        trust_remote_code=True,  # Required for InternVL3
        attn_implementation=attn_implementation,
        max_memory={i: "40GB" for i in range(torch.cuda.device_count())},  # Reserve some memory per GPU
    )
    
    # Load processor (handles both text and images)
    processor = AutoProcessor.from_pretrained(
        source_model,
        trust_remote_code=True
    )
    
    logger.success("✅ Model and processor loaded successfully")
    
    # Patch the config for llmcompressor compatibility with InternVL models
    if hasattr(model.config, 'llm_config') and hasattr(model.config.llm_config, 'use_cache'):
        model.config.use_cache = model.config.llm_config.use_cache
        logger.info("✅ Patched model config for llmcompressor compatibility (use_cache)")
    elif not hasattr(model.config, 'use_cache'):
        # Default to True if use_cache is not found anywhere
        model.config.use_cache = True
        logger.info("✅ Added use_cache=True to model config for llmcompressor compatibility")
    
    # Log GPU memory usage after loading
    for i in range(torch.cuda.device_count()):
        allocated = torch.cuda.memory_allocated(i) / (1024**3)
        cached = torch.cuda.memory_reserved(i) / (1024**3)
        logger.info(f"  GPU {i}: {allocated:.1f}GB allocated, {cached:.1f}GB cached")
    
    # Create quantization recipe
    recipe = create_quantization_recipe(dynamic=dynamic)
    
    # Handle output directory cleanup if force is enabled
    if force and output_dir.exists():
        logger.info(f"🗑️  Removing existing output directory: {output_dir}")
        import shutil
        shutil.rmtree(output_dir)
    
    # Ensure output directory exists
    output_dir.mkdir(parents=True, exist_ok=True)
    
    if dynamic:
        logger.info("🚀 Using FP8-Dynamic quantization - no calibration needed!")
        logger.info("Note: trust_remote_code_model=True is set by default for VLM compatibility")
        
        # For dynamic quantization, we can use the model directly without a dataset
        oneshot(
            model=model,  # Use the already loaded model
            recipe=recipe,
            output_dir=str(output_dir),
            trust_remote_code_model=True,
        )
    else:
        logger.info("🔄 Starting FP8 static quantization...")
        logger.info("This process will take 30-60 minutes depending on hardware")
        logger.warning("Monitor GPU memory usage - process may require 120GB+ peak VRAM")
        
        # Get calibration dataset with fallback
        logger.info(f"📊 Preparing calibration dataset: {calibration_dataset}")
        logger.info(f"   Samples: {num_samples}, Max sequence length: {seq_length}")
        logger.info("Note: Using text-only datasets for calibration (works well for VLMs)")
        
        dataset = get_calibration_dataset(calibration_dataset, num_samples)
        
        # Clear GPU cache before quantization to ensure maximum available memory
        import gc
        gc.collect()
        torch.cuda.empty_cache()
        logger.info("🧹 Cleared GPU cache before quantization")
        
        # Apply quantization with calibration dataset
        try:
            oneshot(
                model=model,
                dataset=dataset,
                recipe=recipe,
                output_dir=str(output_dir),
                max_seq_length=seq_length,
                num_calibration_samples=num_samples,
                trust_remote_code_model=True,
            )
        except Exception as e:
            logger.error(f"Quantization failed with {dataset}: {e}")
            if isinstance(dataset, str) and dataset != "open_platypus":
                logger.info("Retrying with open_platypus dataset...")
                oneshot(
                    model=model,
                    dataset="open_platypus",
                    recipe=recipe,
                    output_dir=str(output_dir),
                    max_seq_length=seq_length,
                    num_calibration_samples=num_samples,
                    trust_remote_code_model=True,
                )
            else:
                raise
    
    logger.success("🎉 Quantization completed successfully!")
    
    # Save processor and tokenizer alongside quantized model
    logger.info("💾 Saving processor and tokenizer configuration...")
    processor.save_pretrained(output_dir)
    
    # Also save tokenizer explicitly to ensure all tokenizer files are saved
    tokenizer = AutoTokenizer.from_pretrained(source_model, trust_remote_code=True)
    tokenizer.save_pretrained(output_dir)
    logger.success("✅ Tokenizer and processor saved successfully")
    
    # Generate and save model card
    logger.info("📝 Generating model card...")
    script_content = read_script_content()
    model_card = generate_model_card(
        source_model=source_model,
        quantized_model_name=quantized_model_name,
        hf_username=hf_username, 
        calibration_dataset=calibration_dataset if not dynamic else "N/A",
        num_samples=num_samples if not dynamic else 0,
        seq_length=seq_length if not dynamic else 0,
        package_versions=package_versions,
        script_content=script_content,
        flash_attn_used=not no_flash_attn and torch.cuda.is_available(),
        attention_implementation=attn_implementation,
        dynamic=dynamic
    )
    
    model_card_path = output_dir / "README.md"
    with open(model_card_path, 'w', encoding='utf-8') as f:
        f.write(model_card)
    
    logger.success(f"📄 Model card saved: {model_card_path}")
    
    # Upload to Hugging Face Hub
    if upload and hf_token:
        logger.info("⬆️ Uploading to Hugging Face Hub...")
        
        # Verify critical files exist before upload
        critical_files = ["README.md", "tokenizer_config.json", "tokenizer.json"]
        missing_files = []
        
        for file in critical_files:
            file_path = output_dir / file
            if file_path.exists():
                logger.info(f"✅ Found {file}")
            else:
                # Some models might use different tokenizer files
                if file == "tokenizer.json":
                    # Check for alternative tokenizer files
                    alt_files = ["tokenizer.model", "vocab.json", "merges.txt"]
                    found_alt = any((output_dir / alt).exists() for alt in alt_files)
                    if found_alt:
                        logger.info(f"✅ Found alternative tokenizer files")
                    else:
                        missing_files.append(file)
                else:
                    missing_files.append(file)
        
        if missing_files:
            logger.warning(f"⚠️  Missing files: {', '.join(missing_files)}")
        
        try:
            from huggingface_hub import HfApi
            
            api = HfApi(token=hf_token)
            
            # Create repository if it doesn't exist
            
            try:
                api.create_repo(repo_id=hf_repo, private=False, exist_ok=True)  # --hf-repo is mapped to repo_id for backward compatibility
                logger.info("✅ Repository created/verified")
            except Exception as repo_e:
                logger.warning(f"Repository creation warning: {repo_e}")
            
            # Upload folder contents
            logger.info("📤 Uploading model files...")
            api.upload_folder(
                folder_path=str(output_dir),
                repo_id=hf_repo,  # --hf-repo is mapped to repo_id for backward compatibility
                repo_type="model"
            )
            
            logger.success("🎉 Model uploaded successfully!")
            logger.success(f"🔗 View at: https://huggingface.co/{hf_repo}")
            
            # List uploaded files
            logger.info("Uploaded files include:")
            for file in output_dir.iterdir():
                if file.is_file():
                    size_mb = file.stat().st_size / (1024 * 1024)
                    logger.info(f"  - {file.name} ({size_mb:.1f} MB)")
            
        except Exception as e:
            logger.error(f"Upload failed: {e}")
            logger.info("Model saved locally - you can upload manually later")
    
    # Final summary
    logger.info("✨ Quantization Summary:")
    logger.info(f"  📁 Model saved to: {output_dir}")
    logger.info(f"  🔢 Quantization type: FP8-{'Dynamic' if dynamic else 'Static'}")
    logger.info("  🔢 Original size: ~76GB (FP16)")
    logger.info("  📉 Quantized size: ~38GB (FP8)")
    logger.info("  🚀 Expected speedup: ~2x on H100/L40S")
    logger.info("  💾 Memory savings: ~50%")
    
    if upload and hf_token:
        logger.info(f"  🌐 HuggingFace: https://huggingface.co/{hf_repo}")
    
    logger.success("🎊 Quantization pipeline completed successfully!")
    
except Exception as e:
    logger.error(f"❌ Quantization failed: {type(e).__name__}: {str(e)}")
    logger.error("Check logs above for detailed error information")
    import traceback
    logger.error("Full traceback:")
    logger.error(traceback.format_exc())
    raise typer.Exit(1)

if name == "main": app()


</details>

## 🎯 Use Cases

This optimized model is ideal for:

- **Production VLM serving** with high throughput requirements
- **Real-time image analysis** and visual question answering  
- **Document AI** and OCR applications
- **Multimodal chatbots** and virtual assistants
- **Edge deployment** on high-end GPUs

## Author
This model was quantized by [Jaro](https://www.linkedin.com/in/jaroai/)

## ⚠️ Important Notes

- Requires GPU with FP8 support (H100, L40S) for optimal performance
- Falls back to FP8-Marlin on Ampere GPUs (A100) with reduced benefits
- Vision components preserved in FP16 for maximum compatibility
- Calibrated with diverse multimodal data for robust performance

## 🚫 Limitations

- **Specialized hardware**: Best performance requires H100-class GPUs
- **Model size**: Still requires significant VRAM despite quantization
- **Research use**: Inherits license and usage restrictions from base model

## 📄 License

This quantized model inherits the license from the original model.
Original model: [HuggingFaceTB/SmolLM-135M](https://huggingface.co/HuggingFaceTB/SmolLM-135M)

## 🙏 Acknowledgments

- **Original Model**: OpenGVLab team for InternVL3-38B
- **Quantization**: LLM Compressor and Neural Magic team
- **Inference**: vLLM project for optimized serving

## 📞 Contact

For questions about this quantized model:
- **Issues**: [Create an issue](https://huggingface.co/JustJaro/InternVL3-38B-FP8-Dynamic/discussions)
- **Original Model**: Refer to [HuggingFaceTB/SmolLM-135M](https://huggingface.co/HuggingFaceTB/SmolLM-135M)

---

*Quantized with ❤️ using LLM Compressor for the open-source community*

JustJaro/SmolLM-135M-FP8-Dynamic-Test

🔥 InternVL3-38B-FP8-Static: Optimized Vision-Language Model 🔥

🚀 Key Features

📊 Model Details

🔧 Usage

With vLLM (Recommended)

With Transformers + LLM Compressor

🏗️ Technical Specifications

Hardware Requirements

Quantization Details

📈 Performance Benchmarks

🔬 Package Versions

📋 Quantization Script

Usage

Quantization Types

FP8-Static (default)

FP8-Dynamic

Attention Mechanisms

Flash Attention 2 (default)

SDPA (Scaled Dot-Product Attention)

Eager (Standard) Attention

Important Notes

Calibration Dataset Notes

Import llm-compressor modules

Load environment variables

Configure loguru

Constants

🔥 InternVL3-38B-FP8-Static: Optimized Vision-Language Model 🔥

🚀 Key Features

📊 Model Details

🔧 Usage

With vLLM (Recommended)

With Transformers + LLM Compressor

🏗️ Technical Specifications

Hardware Requirements

Quantization Details

📈 Performance Benchmarks

🔬 Package Versions

📋 Quantization Script

🎯 Use Cases

⚠️ Important Notes

🚫 Limitations

📄 License

🙏 Acknowledgments

📞 Contact

JustJaro/
SmolLM-135M-FP8-Dynamic-Test