Views
No views yet
Gemma4VisionModel) and its patch embedder/poolerGemma4AudioModel) and its conformer-style encoder layersGemma4MultimodalEmbedder) that project soft tokens into the
language model's embedding spaceGemma4Model.forwardembed_tokens,
embed_tokens_per_layer) - PLE is a text-decoder feature, not a vision/audio one18 layers), exactly as in the original text backbone
(model.language_model in the original checkpoint -> model here)lm_head, tied to embed_tokens since tie_word_embeddings=True)generate() decodes were verified to match token-for-token, and logits matched with a max
absolute difference of 0.00e+00 (bf16 numerical noise floor).gemma4_text / Gemma4ForCausalLM) is natively supported in
transformers>=5.9. No trust_remote_code is required:1import torch
2from transformers import AutoModelForCausalLM, AutoTokenizer
3
4model = AutoModelForCausalLM.from_pretrained("tuandunghcmut/gemma-4-E4B-it-text-only", dtype=torch.bfloat16, device_map="auto")
5tok = AutoTokenizer.from_pretrained("tuandunghcmut/gemma-4-E4B-it-text-only")
6
7inputs = tok("The capital of France is", return_tensors="pt").to(model.device)
8out = model.generate(**inputs, max_new_tokens=30)
9print(tok.decode(out[0], skip_special_tokens=True))modeling_gemma4_text.py + configuration_gemma4_text.py,
verified bit-for-bit identical to the native transformers implementation on random weights, including
the PLE and KV-sharing code paths actually exercised by this checkpoint) is included in this repo for
transparency/portability. It is not required for loading - transformers already ships this
architecture natively - but documents exactly what the text-only forward pass does.