Gemma Causal Language Model (GemmaCausalLM)

Model Overview

1. Core Architecture

Backbone (GemmaBackbone):

• Vocabulary Size: 256,000 tokens.
• Model Depth: 26 layers.
• Attention Configuration:
  • Query Heads: 8
  • Key-Value Heads: 4
  • Head Dimension: 256
  • Sliding Window Attention: Enabled (window size: 4096).
• Dimensions:
  • Hidden Dimension: 2,304
  • Intermediate Dimension: 18,432
• Normalization and Regularization:
  • Layer Normalization (Epsilon: 1e-6).
  • Post-feedforward and post-attention normalization enabled.
• Soft Caps:
  • Final Logit Soft Cap: 30.0
  • Attention Logit Soft Cap: 50.0
• Dropout: Disabled.

Preprocessor (GemmaCausalLMPreprocessor):

• Tokenizer (GemmaTokenizer):
  • Configuration File: tokenizer.json.
  • Adds BOS (Beginning of Sequence) and EOS (End of Sequence) tokens.
• Sequence Length: 512.
• Data Type:
  • Float32 for preprocessor computations.
  • Int32 for tokenized inputs.

Metadata

• Keras Version: 3.5.0
• Keras Hub Version: 0.17.0
• Parameter Count: 2,617,270,528 (2.6 billion parameters).
• Date Saved: 2024-11-18@13:59:51

Applications

• Text Generation.
• Dialogue Systems.
• Autoregressive NLP tasks.

Model Files

• Backbone Configuration: The core architecture details for GemmaBackbone.
• Preprocessor Configuration: Tokenization and sequence preprocessing setup.
• Tokenizer File: tokenizer.json.
• Preprocessor File: preprocessor.json.

Setup and Usage

1. Dependencies: Ensure the following libraries are installed:
   pip install keras keras_hub
2. Model Loading: The model can be loaded as follows:
   python

   1from keras_hub.src.models.gemma.gemma_causal_lm import GemmaCausalLM 2 3model = GemmaCausalLM.from_config(config_file="path/to/config.json")
3. Inference: Use the preprocessor to tokenize input text and generate predictions with the model.
   python

   1preprocessor = model.get_preprocessor() 2inputs = preprocessor.tokenize("Your input text here.") 3outputs = model.predict(inputs) 4print(outputs)

Contributions

License