stable-code-3b is a 2.7B billion parameter decoder-only language model pre-trained on 1.3 trillion tokens of diverse textual and code datasets. stable-code-3b is trained on 18 programming languages (selected based on the 2023 StackOverflow Developer Survey) and demonstrates state-of-the-art performance (compared to models of similar size) on the MultiPL-E metrics across multiple programming languages tested using BigCode's Evaluation Harness.
spiderchart
Model
Size
Python
C++
Javascript
Java
PHP
Rust
Stable Code
3B
32.4%
30.9%
32.1%
32.1%
24.2%
23.0%
CodeLLama
7B
30.0%
28.2%
32.5%
31.1%
25.7%
26.3%
Deepseek Coder
1.3B
28.6%
29.2%
28.7%
29.0%
23.6%
18.5%
Wizard Coder
3B
31.6%
25.6%
26.2%
25.8%
25.3%
20.4%
StarCoder
3B
21.6%
19.8%
21.5%
20.5%
19.0%
16.9%
Replit Code V1.5
3B
23.0%
25.9%
26.2%
23.6%
23.2%
21.5%
Deci Coder
1B
19.1%
6.8%
18.4%
16.7%
2.1%
1.7%
Key Features
Fill in Middle Capability (FIM)
Supports Long Context, trained with Sequences upto 16,384
Usage
Get started generating text with stable-code-3b by using the following code snippet:
Contact: For questions and comments about the model, please email lm@stability.ai
Model Architecture
The model is a decoder-only transformer similar to the LLaMA (Touvron et al., 2023) architecture with the following modifications:
Parameters
Hidden Size
Layers
Heads
Sequence Length
2,796,431,360
2560
32
32
16384
Position Embeddings: Rotary Position Embeddings (Su et al., 2021) applied to the first 25% of head embedding dimensions for improved throughput following Black et al. (2022).
Tokenizer: We use a modified version of the GPTNeoX Tokenizer.NeoX. We add special tokens to train for Fill in the Middle (FIM) capabilities like <FIM_PREFIX> and <FIM_SUFFIX> along with other special tokens.
The model is pre-trained on the aforementioned datasets in bfloat16 precision, optimized with AdamW.
Training Infrastructure
Hardware: stable-code-3b was trained on the Stability AI cluster across 256 NVIDIA A100 40GB GPUs (AWS P4d instances).
Software: We use a fork of gpt-neox (EleutherAI, 2021), train under 2D parallelism (Data and Tensor Parallel) with ZeRO-1 (Rajbhandari et al., 2019), and rely on flash-attention as well as SwiGLU and Rotary Embedding kernels from FlashAttention-2 (Dao et al., 2023)
Use and Limitations
Intended Use
The model is intended to be used as a foundational base model for application-specific fine-tuning. Developers must evaluate and fine-tune the model for safe performance in downstream applications. For commercial use, please refer to https://stability.ai/membership.
Limitations and Bias
As a base model, this model may exhibit unreliable, unsafe, or other undesirable behaviors that must be corrected through evaluation and fine-tuning prior to deployment. The pre-training dataset may have contained offensive or inappropriate content, even after applying data cleansing filters, which can be reflected in the model-generated text. We recommend that users exercise caution when using these models in production systems. Do not use the models if they are unsuitable for your application, or for any applications that may cause deliberate or unintentional harm to others.
How to Cite
bibtex
1@misc{stable-code-3b,
2 url={[https://huggingface.co/stabilityai/stable-code-3b](https://huggingface.co/stabilityai/stable-code-3b)},
3 title={Stable Code 3B},
4 author={Pinnaparaju, Nikhil and Adithyan, Reshinth and Phung, Duy and Tow, Jonathan and Baicoianu, James and Cooper, Nathan}
5}
