Architecture: transformers with RoPE, SwiGLU, RMSNorm, Attention QKV bias and tied word embeddings
Number of Parameters: 0.49B
Number of Paramaters (Non-Embedding): 0.36B
Number of Layers: 24
Number of Attention Heads (GQA): 14 for Q and 2 for KV
Context Length: Full 32,768 tokens
They do not recommend using base language models for conversations. Instead, you can apply post-training, e.g., SFT, RLHF, continued pretraining, etc., on this model.
