Bark is a transformer-based text-to-audio model created by Suno.
Bark can generate highly realistic, multilingual speech as well as other audio - including music,
background noise and simple sound effects. The model can also produce nonverbal
communications like laughing, sighing and crying. To support the research community,
we are providing access to pretrained model checkpoints ready for inference.
The original github repo and model card can be found here.
This model is meant for research purposes only.
The model output is not censored and the authors do not endorse the opinions in the generated content.
Use at your own risk.
Run the following Python code to generate speech samples:
python
1from transformers import AutoProcessor, AutoModel
234processor = AutoProcessor.from_pretrained("suno/bark-small")5model = AutoModel.from_pretrained("suno/bark-small")67inputs = processor(8 text=["Hello, my name is Suno. And, uh — and I like pizza. [laughs] But I also have other interests such as playing tic tac toe."],9 return_tensors="pt",10)1112speech_values = model.generate(**inputs, do_sample=True)
Listen to the speech samples either in an ipynb notebook:
1from bark import SAMPLE_RATE, generate_audio, preload_models
2from IPython.display import Audio
34# download and load all models5preload_models()67# generate audio from text8text_prompt ="""
9 Hello, my name is Suno. And, uh — and I like pizza. [laughs]
10 But I also have other interests such as playing tic tac toe.
11"""12speech_array = generate_audio(text_prompt)1314# play text in notebook15Audio(speech_array, rate=SAMPLE_RATE)
Output: semantic tokens that encode the audio to be generated
Semantic to coarse tokens
Input: semantic tokens
Output: tokens from the first two codebooks of the EnCodec Codec from facebook
Coarse to fine tokens
Input: the first two codebooks from EnCodec
Output: 8 codebooks from EnCodec
Architecture
Model
Parameters
Attention
Output Vocab size
Text to semantic tokens
80/300 M
Causal
10,000
Semantic to coarse tokens
80/300 M
Causal
2x 1,024
Coarse to fine tokens
80/300 M
Non-causal
6x 1,024
Release date
April 2023
Broader Implications
We anticipate that this model's text to audio capabilities can be used to improve accessbility tools in a variety of languages.
While we hope that this release will enable users to express their creativity and build applications that are a force
for good, we acknowledge that any text to audio model has the potential for dual use. While it is not straightforward
to voice clone known people with Bark, it can still be used for nefarious purposes. To further reduce the chances of unintended use of Bark,
we also release a simple classifier to detect Bark-generated audio with high accuracy (see notebooks section of the main repository).
License
Bark is licensed under the MIT License, meaning it's available for commercial use.
