Model Summary Granite-SWASH-3B-a600M is a 3B parameter decoder-only sparse language model with per-layer Sliding Window Attention (SWA) and learnable attention sinks (LSE-based). Trained on a mix of open source and proprietary data, it is a broad, general-purpose, English-language base model that serves as an early exploration and small-scale preview for upcoming Granite series model releases.

Granite-SWASH-3B-a600M is based on a decoder-only mixture-of-experts transformer architecture. Core components of this architecture are: GQA, RoPE, MLP with SwiGLU, RMSNorm, MoE with shared experts, shared input/output embeddings, Sliding Window Attention, and Attention Sinks.

Attention Pattern: 8 full-attention layers (indices 0, 3, 7, 11, 15, 19, 23, 27) and 20 sliding-window layers (window=128). All layers have learnable per-head attention sinks.

MuP Scaling: embedding_multiplier=12, residual_multiplier=0.26, logits_scaling=5, attention_multiplier=0.015625

Sink Mechanism: this model uses post-attention LSE scaling in the eager implementation:

sink_scale = sigmoid(lse - sinks)
attn_output = attn_output * sink_scale

Where lse is the log-sum-exp of attention scores, and sinks is a learnable per-head parameter. Flash Attention 3 and 4 implementations natively incorporate and compute the sink token under the hood. Note that SDPA does not express sink tokens or sink_scale, and so is not a supported backend.

Requires transformers version > 5.8.0

The use of Large Language Models involves risks and ethical considerations people must be aware of, including but not limited to: bias and fairness, misinformation, and autonomous decision-making. Granite-SWASH-3B-a600M is not an exception in this regard. Even though this model is suited for multiple generative AI tasks, it has not undergone any safety alignment and it may produce problematic outputs. Additionally, it remains uncertain whether smaller models might exhibit increased susceptibility to hallucination in generation scenarios by copying text verbatim from the training dataset due to their reduced sizes and memorization capacities. This aspect is currently an active area of research, and we anticipate more rigorous exploration, comprehension, and mitigations in this domain. Regarding ethics, a latent risk associated with all Large Language Models is their malicious utilization. We urge the community to use Granite-SWASH-3B-a600M model with ethical intentions and in a responsible way. To enhance safety in enterprise deployments, we recommend using Granite Language models alongside Granite Guardian, a model designed to detect and flag risks in inputs and outputs across key dimensions outlined in the IBM AI Risk Atlas.