WavJEPANat, a waveform-based version of the Joint-Embedding Predictive Architecture. WavJEPANat leverages high-level semantic representation learning to tackle the shortcomings of representation learning at the speech unit or token level. We show that this approach substantially outperforms state-of-the-art time-domain audio foundation models across a wide variety of downstream benchmark tasks, while requiring considerably fewer computational resources. Additionally, WavJEPA-Nat overcomes the performance drop that time-domain models typically exhibit in noisy and reverberant real-world acoustic environments

The WavJEPANat framework comprises a waveform encoder, context encoder, target encoder and a predictor. WavJEPANat’s objective is to predict latent representation of various targets blocks based on a single context block extracted from the same sound wave. As waveform encoder, we use the feature encoder of Wav2Vec 2.0, which is composed of stacked temporal convolution layers (Baevski et al., 2020). Similar to the original I-JEPA architecture (Assran et al., 2023), a Vision Transformer (ViT) (Dosovitskiy et al., 2021) is used for the target encoder, context encoder and predictor.

WavJEPANat is the first framework applying semantic learning to general-purpose audio representations in the time domain, surpassing state-of-the-art time-domain approaches on the HEAR (Turian et al., 2022) benchmark suite while requiring only a fraction of the computational resources. WavJEPANat leverages high-level semantic representation learning to tackle the shortcomings of representation learning at the speech unit or token level. We show that this approach substantially outperforms state-of-the-art time-domain audio foundation models across a wide variety of downstream benchmark tasks, while requiring considerably fewer computational resources. Additionally, we address the degraded performance of time-domain models in real-world sound scenes with WavJEPANat, a multi-channel extension of the WavJEPA framework trained on simulated real-world sound scenes. Evaluation on Nat-HEAR (Yuksel et al., 2025), a naturalistic version of the HEAR benchmark suite, demonstrates that WavJEPA-Nat exceeds the robustness of other time-domain foundation models to noise and reverberation.

WavJEPANat can be used as a powerful feature extractor for downstream tasks such as enviromental sound classification, speech recognition, speaker counting etc on adverse conditions. Later, training a linear head on top of these extracted features would yield a fine-tuned audio scene analysis model.

We train WavJEPANat on the unbalanced training set of AudioSet, which consists of 1.74 million 10-second sound clips scraped from YouTube (Gemmeke et al., 2017), and 70,000 naturalistic scenes (corresponding to 70 Matterport3D houses). We used the 70,000 naturalistic scenes in the train set to generate naturalistic scenes for all audio clips in the unbalanced training set of AudioSet (10-second sound tracks of 1.74 million YouTube videos (Gemmeke et al., 2017)). Specifically, during training we randomly paired an AudioSet clip with a noise sound clip from the WHAMR! background noise database (Maciejewski et al., 2020). WHAMR! noise clips longer than 10 s were trimmed to 10 s duration and a linear fade-in/fade-out of 200 ms was applied to every noise clip prior to mixing of the sound scene. To create a naturalistic sound scene, we then convolved the AudioSet clip with RIR(s, r, θ).

Each sound clip was resampled to 16 kHz and mean centered to enforce equal loudness across sound clips. We then randomly sampled 8 sections of 2 s from each sound clip, effectively increasing the batch size by a factor of 8 in a computationally efficient manner. Finally, each instance is instance normalized (Ulyanov et al., 2017). The waveform encoder converts each 2 s instance into an embedding w 200×768, effectively resampling the audio to 100 Hz with a stride of 10 ms and a receptive field size of 12.5 ms

We sampled starting indices for the context block with p = 0.065 and for target blocks with p = 0.025. We set M to 10 for both context block and target block . To update the target encoder parameters ∆, we linearly increased τ from τ0 = 0.999 to τe = 0.99999 over the first 100,000 steps, after which τ was kept constant. We used K = 8 for the top K averaging. We trained WavJEPANat for 375,000 steps using a batch size of 16 on two NVIDIA H100 94 GB GPUs. Given our in-batch sampling factor of 8, we boost our effective batch size to 256. We use the AdamW optimizer (Loshchilov & Hutter, 2019) with a weight decay coefficient λw = 0.04. The learning rate schedule follows a cosine decay with linear warm-up over 100,000 steps, reaching a peak learning rate of 2 × 10−4 before decaying to zero

RMS Normalization was applied to audio clips to get all of them in the same loudness levels, and later instance normalization is applied.

We evaluate WavJEPANat and other state-of-the-art models on the HEAR task suite, which presents a wide range of tasks to evaluate the downstream performance of audio representation models (Turian et al., 2022).

HEAR: The aim of the HEAR benchmark is to develop a general-purpose audio representation that provides a strong basis for learning in a wide variety of tasks and scenarios. HEAR evaluates audio representations using a benchmark suite across a variety of domains, including speech, environmental sound, and music. HEAR was launched as a NeurIPS 2021 shared challenge. It still remains an open question whether one single general-purpose audio representation can perform as holistically as the human ear.

We presented WavJEPANat, a state-of-the-art audio foundation model that leverages self-supervised semantic learning to obtain robust general-purpose audio representations from raw waveforms. WavJEPANat’s results highlight the superior performance of semantic audio representation learning in comparison with representation learning at the speech unit or token level, as is common in existing time-domain speech representation learning approaches.

Goksenin Yuksel; goksenin.yuksel@ru.nl