NatureLM-audio is the first audio-language foundation model specifically designed for bioacoustics. It is trained on a diverse dataset of text-audio pairs spanning bioacoustics, speech, and music, enabling it to perform tasks such as species classification, detection, captioning, and lifestage classification. The model demonstrates strong generalization to unseen taxa and tasks, setting a new state-of-the-art on several bioacoustics benchmarks.
Model Details
Model Description
NatureLM-audio is an audio-language model designed to address bioacoustic tasks such as species classification, detection, and captioning. It leverages a combination of bioacoustic, speech, and music data to learn robust representations that generalize across domains.
Developed by: David Robinson, Marius Miron, Masato Hagiwara, Milad Alizadeh, Gagan Narula, Sara Keen, Benno Weck, Matthieu Geist, Olivier Pietquin (Earth Species Project)
NatureLM-audio can be used directly for bioacoustic tasks such as species classification, detection, and captioning. It is particularly useful for biodiversity monitoring, conservation, and animal behavior studies.
Example prompts:
Prompt: What is the common name for the focal species in the audio?
Answer: Humpback Whale
Prompt: Which of these, if any, are present in the audio recording? Single pulse gibbon call, Multiple pulse gibbon call, Gibbon duet, None.
Answer: Gibbon duet
Prompt: What is the common name for the focal species in the audio?
Answer: Spectacled Tetraka
Prompt: What is the life stage of the focal species in the audio?
Answer: Juvenile
Prompt: What type of vocalization is heard from the focal species in the audio?
Answer with either 'call' or 'song'.
Prompt: Caption the audio, using the common name for any animal species.
Downstream Use
The model can be used to structure audio for ethology research, be integrated into larger ecological monitoring systems, or be fine-tuned for specific bioacoustic tasks.
Out-of-Scope Use
The model is not designed for tasks outside of bioacoustics. It was not tested for tasks such as individual-id, and call-type and lifestage classification tasks have only been tested on birds Tasks beyond those evaluated in the paper may require in-context learning or fine-tuning. The model does not currently perform fine-grained detection with exact time stamps.
Bias, Risks, and Limitations
Bias: The model may exhibit biases towards bird vocalizations due to the overrepresentation of bird datasets in the training data. This could limit its effectiveness for other taxa. Further, the model may inherit biases from the parent Llama model.
Risks: The model’s ability to detect and classify endangered species could be misused for illegal activities such as poaching.
Limitations: The model’s performance may be limited for under-represented taxa.
Red-teaming results: We ran a red-teaming assessment by first defining 16 risk categories adapted for AI safety in the context of animals, ecosystems, and the environment, such as Wildlife Exploitation, Non-Compliance with Environmental Laws, and Biodiversity Loss. Then, we used an LLM to generate adversarial prompts that could potentially lead to harmful output, and the responses were then evaluated to determine their safety. While the majority of responses from NatureLM-audio were safe, often providing no content for problematic prompts, we identified several scenarios where the model's responses were potentially harmful, including cases where the model failed to discourage unethical actions related to wildlife exploitation and environmental harm.
Recommendations
Users should be aware of the risks, biases, and limitations of the model. It is recommended to use the model in conjunction with other ecological monitoring tools and to validate its predictions in real-world settings.
How to Get Started with the Model
Instantiating the model:
python
1from NatureLM.models import NatureLM
23# Download the model from HuggingFace4model = NatureLM.from_pretrained("EarthSpeciesProject/NatureLM-audio")5model = model.eval().to("cuda")
Using the model:
python
1from NatureLM.infer import Pipeline
23audio_paths =["assets/nri-GreenTreeFrogEvergladesNP.mp3"]4queries =["What is the common name for the focal species in the audio? Answer:"]56pipeline = Pipeline(model=model)78# Run the model over the audio in sliding windows of 10 seconds with a hop length of 10 seconds9results = pipeline(audio_paths, queries, window_length_seconds=10.0, hop_length_seconds=10.0)1011print(results)12# ['#0.00s - 10.00s#: Green Treefrog\n']
The model is trained on a diverse dataset of text-audio pairs, including bioacoustic recordings, general audio, speech, and music datasets. The training data includes datasets such as Xeno-canto, iNaturalist, and Watkins. We have released the training dataset on Hugging Face.
Training Procedure
The model is trained in two stages:
Perception Pretraining on species classification.
Generalization Fine-tuning on a variety of bioacoustic tasks.
Training Hyperparameters
Learning rate: 9.0e-5 (peak), 2.0e-5 (end)
Batch size: 128
Training steps: 5.0e5 (Stage 1), 1.6e6 (Stage 2)
For the full list of hyperparameters consult the NatureLM-audio repository.
Evaluation
Testing Data, Factors & Metrics
Testing Data
The model is evaluated on the BEANS-Zero benchmark, which includes tasks such as species classification, detection, and captioning.
Metrics
Accuracy for classification
F1 for detection
SPIDEr for captioning
Results
The model achieves state-of-the-art performance on several bioacoustics tasks, including zero-shot classification of unseen species.
The model uses a BEATs audio encoder, Q-Former for connecting audio embeddings to the LLM, and Llama-3.1-8B-Instruct as the text generator.
Compute Infrastructure
Hardware: 8xH100
Software: Pytorch
Citation
BibTeX:
@inproceedings{naturelm-audio,
title={NatureLM-audio: An Audio-Language Foundation Model for Bioacoustics},
author={Robinson, David and Miron, Marius and Hagiwara, Masato and Pietquin, Olivier},
booktitle={Proceedings of the International Conference on Learning Representations},
year={2025}
}
APA:
Robinson, D., Miron, M., Hagiwara, M., & Pietquin, O. (2025). NatureLM-audio: An Audio-Language Foundation Model for Bioacoustics. ICLR 2025
Glossary
Bioacoustics: The study of sound production and reception in animals.
Zero-shot learning: The ability of a model to perform tasks it has not explicitly been trained on.
Taxa: A group of organisms, such as species, genus, or family.
More Information
For more information, please visit the project page.
