We present Tri-7B-Base, a foundation language model that serves as the pre-trained base for our Tri-7B model family. This model represents our commitment to efficient training while establishing a strong foundation for downstream fine-tuning and adaptation.
Key Features
Foundation Architecture: State-of-the-art transformer architecture optimized for efficiency
Multi-lingual Foundation: Pre-trained on diverse data in Korean, English, and Japanese
Efficient Training: Optimized training methodology for computational efficiency
Model Specifications
Tri-7B-Base
Type: Causal Language Model
Training Stage: Pre-training
Architecture: Transformer Decoder with RoPE, SwiGLU, RMSNorm
Number of Parameters: 7.76B
Number of Layers: 32
Number of Attention Heads: 32
Context Length: 4,096
Vocab Size: 128,128
Use Cases
As a base model, Tri-7B-Base is designed to serve as a foundation for various downstream applications:
Fine-tuning: Adapt to specific domains or tasks
Instruction Tuning: Create chat or assistant models
Domain Specialization: Customize for specific industries or use cases
Research: Explore model behaviors and capabilities
Language Generation: General text completion and generation tasks
Limitations
Base Model Nature: This is a pre-trained base model without instruction tuning or alignment. For chat or assistant capabilities, consider fine-tuned variants.
Language Support: The model is optimized for English, Korean, and Japanese. Usage with other languages may result in degraded performance.
Knowledge Cutoff: The model's information is limited to data available up to February, 2025.
Generation Quality: As a base model, outputs may require post-processing or filtering for production use cases.
License
This model is licensed under the Apache License 2.0.
