Model Card for Model ID
This Gemma3-1B-it model had its base model frozen and then had some of its MLP layers modified (global attention layers) to support In-Place Test Time Training.
This was trained on 2 epochs of 12k samples of the LongAlpaca-12k dataset.
The code can be found in
this repo.
Model Details
Model Description
This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
- Developed by: Chang Min Bark and Hung Ngo
- Funded by [optional]:
- Shared by [optional]:
- Model type: Dense
- Language(s) (NLP): English
- License: Apache-2.0
- Finetuned from model [optional]: google/gemma-3-1b-it
Model Sources [optional]
Uses
Direct Use
This was developed for the final project at AI with Neural Networks class at Bucknell University.
Downstream Use [optional]
[More Information Needed]
Out-of-Scope Use
This model cannot be used for any use cases outlined for google/gemma-3-1b-it.
Bias, Risks, and Limitations
This model carries all of the same biases, risks, and limitations carried by google/gemma-3-1b-it.
Recommendations
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
How to Get Started with the Model
Check the GitHub repo for more information.
Training Details
Training Data
This was trained using
LongAlpaca.
Training Procedure
Preprocessing [optional]
[More Information Needed]
Training Hyperparameters
- Training regime:
| Hyperparameter | TinyStories | LongAlpaca | Flag |
| ---------------------- | ----------- | ---------- | ------------------- |
|
max_length | 1024 | 4096 | --max-length |
| epochs | 1.0 | 2.0 | --epochs |
| batch_size | 8 | 1 | --batch-size |
| grad_accum | 8 | 16 | --grad-accum |
| effective batch | 64 | 16 | — |
| lr | 1e-4 | 5e-5 | --lr |
| weight_decay | 0.1 | 0.1 | --weight-decay |
| warmup_steps | 500 | 100 | --warmup-steps |
| max_grad_norm | 1.0 | 1.0 | --max-grad-norm |
Shared optimizer / schedule (not per-dataset): AdamW with (β1, β2) = (0.9, 0.95), cosine LR decay, bf16 (fp16 fallback on non-bf16 GPUs), gradient checkpointing off.
TTT-module knobs (defaults match the In-Place TTT paper):
| Hyperparameter | Default | Flag |
|---|
ttt_layers | [0, 6, 12, 18, 24] | --ttt-layers |
ttt_chunk | 2048 | --ttt-chunk |
ttt_lr (η) | 0.3 | --ttt-lr |
ttt_proj | on | --no-ttt-proj |
ttt_target | hidden_states | --ttt-target |
Speeds, Sizes, Times [optional]
Evaluation
Testing Data, Factors & Metrics
Testing Data
Tested using synthetic
NVIDIA RULER.
Factors
Metrics
Results
Summary
The vanilla base google/gemma-3-1b-it performed better than the In-Place TTT augmented model.
Model Examination [optional]
[More Information Needed]
Environmental Impact
Carbon emissions can be estimated using the
Machine Learning Impact calculator presented in
Lacoste et al. (2019).
- Hardware Type: [More Information Needed]
- Hours used: [More Information Needed]
- Cloud Provider: [More Information Needed]
- Compute Region: [More Information Needed]
- Carbon Emitted: [More Information Needed]
Technical Specifications [optional]
Model Architecture and Objective
Based on google/gemma-3-1b-it.
Compute Infrastructure
Run using Google Colab.
Hardware
Run using NVIDIA A100 GPU in Google Colab.
Software
Run using Google Colab.
Citation [optional]
BibTeX:
[More Information Needed]
APA:
[More Information Needed]
Glossary [optional]
[More Information Needed]
More Information [optional]
[More Information Needed]
Model Card Authors [optional]
[More Information Needed]
Model Card Contact
[More Information Needed]
