You can deploy this model and start earning money today!

PhatcatDK/
t5gemma-2-4b-4b-encoder-only

Name: t5gemma-2-4b-4b-encoder-only
Availability: InStock

google/t5gemma-2-4b-4b

finetune

gemma

endpoints_compatible

Views

No views yet

Encoder-only extraction for google/t5gemma-2-4b-4b

This repo contains the extracted encoder-only weights including visioning for google/t5gemma-2-4b-4b. Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms.

Reference table

This extraction is part of a complete T5Gemma suite (270m, 1b, 2b, 4b, 9b) created to provide a unified environment for LLM-SDXL adapter research.

Model	T5Gemma Series	Architecture	Model Size (GB)	Saved VRAM	Note
270m-270m	V2 (Gemma-3)	Multimodal	1.37	~13%	Vision tower takes up most of the full model
1b-1b	V2 (Gemma-3)	Multimodal	2.84	~33%
2b-2b	V1 (Gemma-2)	Text-only	5.23	~53%	Most VRAM saved of V1
4b-4b	V2 (Gemma-3)	Multimodal	8.60	~43%	Most VRAM saved of V2
9b-2b	V1 (Gemma-2)	Text-only	18.46	~25%	Encoder takes up most of the full model

Why?

Standard T5-Gemma checkpoints are designed for seq2seq tasks, carrying heavy decoder overhead. This extraction isolates the V2 multimodal encoder (Gemma-3 based), stripping the decoder layers to reduce VRAM consumption by ~43%.

How to load the model

python

from transformers import T5Gemma2Model

model_path = "PhatcatDK/t5gemma-2-4b-4b-encoder-only"

# Load the weights
model = T5Gemma2Model.from_pretrained(
    model_path,
    torch_dtype="auto", # Recommended: bfloat16
    is_encoder_decoder=False, # CRITICAL: Tells Transformers there is no decoder
    trust_remote_code=True
)

Acknowledgements & Inspiration

This work was directly inspired by the pioneering research of Minthy.

Minthy was the first to demonstrate the power of the t5gemma-2b-2b-ul2 encoder within this ecosystem. This specific repository was created to round out a standardized suite (270m through 9b) with a "clean-config" setup, ensuring consistent architectural parity for cross-attention testing.

I am incredibly grateful for the groundwork Minthy laid; this suite exists to support and verify the technical infrastructure he built.