You can deploy this model and start earning money today!

sinhashubham/
gemma-3-270m-it-onnx-fp16

Name: gemma-3-270m-it-onnx-fp16
Availability: InStock

onnx

gemma3_text

Views

No views yet

Gemma-3 270M Instruction-Tuned ONNX Model (FP16)

This is a quantized FP16 ONNX version of Google's Gemma-3 270M Instruction-Tuned model, optimized for browser deployment with ONNX Runtime Web.

Model Details

Base Model: google/gemma-3-270m-it
Format: ONNX
Precision: FP16 (half precision)
Model Size: ~833 MB (49.9% reduction from FP32)
Sequence Length: 8192 tokens
Vocabulary Size: 262,144 tokens

Performance Benefits

50% smaller than FP32 version
WebGPU compatible for hardware acceleration
Minimal quality degradation compared to original model
Faster inference on compatible hardware

Browser Compatibility

This model is designed for browser deployment using ONNX Runtime Web with:

WebGPU backend (recommended for performance)
WASM backend (fallback for broader compatibility)

Usage

JavaScript/TypeScript (Browser)

javascript

import { InferenceSession, Tensor } from 'onnxruntime-web';

// Load the model
const session = await InferenceSession.create('./onnx/model.onnx', {
  executionProviders: ['webgpu', 'wasm']
});

// Run inference
const inputIds = new Tensor('int64', [2, 818, 5279, 529, 7001, 563], [1, 6]);
const attentionMask = new Tensor('int64', [1, 1, 1, 1, 1, 1], [1, 6]);

const results = await session.run({
  input_ids: inputIds,
  attention_mask: attentionMask
});

const logits = results.logits; // Shape: [1, 6, 262144]

Python (Testing)

python

import onnxruntime as ort
import numpy as np

# Load the model
session = ort.InferenceSession('onnx/model.onnx')

# Prepare inputs
input_ids = np.array([[2, 818, 5279, 529, 7001, 563]], dtype=np.int64)
attention_mask = np.array([[1, 1, 1, 1, 1, 1]], dtype=np.int64)

# Run inference
outputs = session.run(None, {
    'input_ids': input_ids,
    'attention_mask': attention_mask
})

logits = outputs[0]  # Shape: (1, 6, 262144)

Model Architecture

Type: Decoder-only transformer
Layers: 18
Hidden Size: 2048
Attention Heads: 8
Key-Value Heads: 1 (Multi-Query Attention)
Intermediate Size: 16384
Max Position Embeddings: 8192

Files

onnx/model.onnx - FP16 ONNX model file (symlink to model_fp32_fp16.onnx)
onnx/model_fp32_fp16.onnx - FP16 ONNX model file
onnx/model_fp32.onnx - Original FP32 ONNX model file (for comparison)
config.json - Model configuration
tokenizer.json - Fast tokenizer
tokenizer.model - SentencePiece tokenizer
generation_config.json - Generation parameters

Conversion Details

This model was converted using:

PyTorch → ONNX: Hugging Face Optimum
FP32 → FP16: onnxconverter-common
Validation: Comprehensive testing for accuracy and compatibility

License

This model inherits the license from the original Gemma-3 270M model. Please refer to the original model card for license details.

Citation

If you use this model, please cite the original work:

bibtex

@misc{gemma3,
  title={Gemma 3: Advancing Open Language Models},
  author={Google DeepMind},
  year={2024},
  url={https://huggingface.co/google/gemma-3-270m-it}
}

sinhashubham/gemma-3-270m-it-onnx-fp16