SAM-Audio: Segment Anything Model for Audio
SAM-Audio is a model for isolating any sound in audio using text, visual, or temporal prompts. It can separate specific sounds from complex audio mixtures based on natural language descriptions, visual cues from video, or time spans.
Authentication
Before using SAM-Audio, you need to:
- Request access to the checkpoints on the SAM-Audio Hugging Face repo
- Authenticate with Hugging Face:
huggingface-cli login
Usage
SAM-Audio supports three types of prompting: text, visual, and span. Each method allows you to specify which sounds to isolate in different ways.
1. Text Prompting
Use natural language descriptions to isolate sounds.
1import torch
2import torchaudio
3from sam_audio import SAMAudio, SAMAudioProcessor
4
5# Load model and processor
6device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
7model = SAMAudio.from_pretrained("facebook/sam-audio-large").to(device).eval()
8processor = SAMAudioProcessor.from_pretrained("facebook/sam-audio-large")
9
10# Load audio file
11audio_file = "path/to/audio.wav"
12
13# Describe the sound you want to isolate
14description = "A man speaking"
15
16# Process and separate
17inputs = processor(audios=[audio_file], descriptions=[description]).to(device)
18with torch.inference_mode():
19 result = model.separate(inputs)
20
21# Save results
22torchaudio.save("target.wav", result.target[0].unsqueeze(0).cpu(), processor.audio_sampling_rate)
23torchaudio.save("residual.wav", result.residual[0].unsqueeze(0).cpu(), processor.audio_sampling_rate)
Examples of text descriptions:
- "A person coughing"
- "Raindrops are falling heavily, splashing on the ground"
- "A dog barking"
- "Piano playing a melody"
- "Car engine revving"
2. Visual Prompting
Isolate sounds associated with specific visual objects in a video using masked video frames.
1import torch
2import numpy as np
3from sam_audio import SAMAudio, SAMAudioProcessor
4from torchcodec.decoders import VideoDecoder
5
6# NOTE: Requires SAM3 for creating masks
7# pip install git+https://github.com/facebookresearch/sam3.git
8from sam3.model_builder import build_sam3_video_predictor
9
10# Load SAM-Audio model
11device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
12model = SAMAudio.from_pretrained("facebook/sam-audio-large").to(device).eval()
13processor = SAMAudioProcessor.from_pretrained("facebook/sam-audio-large")
14
15# Load video
16video_file = "path/to/video.mp4"
17decoder = VideoDecoder(video_file)
18frames = decoder[:]
19
20# Create mask using SAM3 (example with text prompt)
21video_predictor = build_sam3_video_predictor()
22response = video_predictor.handle_request({
23 "type": "start_session",
24 "resource_path": video_file,
25})
26session_id = response["session_id"]
27
28masks = []
29for frame_index in range(len(decoder)):
30 response = video_predictor.handle_request({
31 "type": "add_prompt",
32 "session_id": session_id,
33 "frame_index": frame_index,
34 "text": "The person on the left", # Visual object to isolate
35 })
36 mask = response["outputs"]["out_binary_masks"]
37 if mask.shape[0] == 0:
38 mask = np.zeros_like(frames[0, [0]], dtype=bool)
39 masks.append(mask[:1])
40
41mask = torch.from_numpy(np.concatenate(masks)).unsqueeze(1)
42
43# Process with visual prompting
44inputs = processor(
45 audios=[video_file],
46 descriptions=[""],
47 masked_videos=processor.mask_videos([frames], [mask]),
48).to(device)
49
50with torch.inference_mode():
51 result = model.separate(inputs)
3. Span Prompting (Temporal Anchors)
Specify time ranges where the target sound occurs or doesn't occur. This provides a specific example to the model of what to isolate
1import torch
2import torchaudio
3from sam_audio import SAMAudio, SAMAudioProcessor
4
5# Load model and processor
6device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
7model = SAMAudio.from_pretrained("facebook/sam-audio-large").to(device).eval()
8processor = SAMAudioProcessor.from_pretrained("facebook/sam-audio-large")
9
10# Define anchors: [type, start_time, end_time]
11# "+" means the sound IS present in this time range
12# "-" means the sound is NOT present in this time range
13anchors = [
14 ["+", 6.3, 7.0], # Sound occurs between 6.3 and 7.0 seconds
15]
16
17# Process with span prompting
18inputs = processor(
19 audios=[audio_file],
20 descriptions=["A horn honking"],
21 anchors=[anchors],
22).to(device)
23
24with torch.inference_mode():
25 result = model.separate(inputs)
Example with multiple anchors:
1anchors = [
2 ["+", 2.0, 3.5], # Sound present from 2.0 to 3.5 seconds
3 ["+", 8.0, 9.0], # Sound present from 8.0 to 9.0 seconds
4 ["-", 0.0, 1.0], # Sound NOT present from 0.0 to 1.0 seconds
5]
Output Format
The model.separate() method returns a result object with:
result.target: The isolated sound (what you asked for)result.residual: Everything else (the remainder)
Both are list[torch.Tensor] where each tensor is a 1D waveform
Citation
If you use SAM-Audio in your research, please cite:
1@article{sam-audio,
2 title={SAM-Audio: Segment Anything in Audio},
3 author={Bowen Shi, Andros Tjandra, John Hoffman, Helin Wang, Yi-Chiao Wu, Luya Gao, Julius Richter, Matt Le, Apoorv Vyas, Sanyuan Chen, Christoph Feichtenhofer, Piotr Dollár, Wei-Ning Hsu, Ann Lee},
4 year={2025}
5 url={arxiv link coming soon}
6}
License
This project is licensed under the SAM License. See the
LICENSE file for details.
