You can deploy this model and start earning money today!

5roop/
Wav2Vec2BertProsodicUnitsFrameClassifier

transformers

safetensors

wav2vec2-bert

audio-frame-classification

facebook/w2v-bert-2.0

finetune

endpoints_compatible

Views

No views yet

Wav2VecBert2 Audio frame classifier for prosodic unit detection

This model predicts prosodic units on speech. For each 20ms frame the model predicts 1 or 0, indicating whether there is a prosodic unit in this frame or not.

This frame-level output can be grouped into events with the frames_to_intervals function provided in the code snippets below.

It is known that the model is unreliable if the audio starts or ends within a prosodic unit. This can be somewhat circumvented by 1) using the largest possible chunks that will fit your machine and 2) use overlapping chunks and combining results smartly.

Model Details

Model Description

Developed by: Peter Rupnik, Nikola Ljubešić, Darinka Verdonik, Simona Majhenič
Funded by: MEZZANINE project
Model type: Wav2VecBert2 for Audio Frame Classification
Language(s) (NLP): Trained and tested on Slovenian
Finetuned from model: facebook/w2v-bert-2.0

The model was trained on ROG-Art dataset, on train split only.

Model performance

We evaluate the model indirectly, and only care about the positive class:

first prosodic units (intervals with start and end times, e.g. [0.123, 5.546]) are extracted from data and model outputs
if a predicted prosodic unit has an overlapping counterpart in true prosodic units, we count it as a True Positive. If there is no overlapping true counterpart, we count it as a False Positive, and if we have a true prosodic unit without a counterpart in predictions, we count that as a False Negative.
Based on the TP, FN, FP numbers recall, precision, and F1 score is calculated.

In this fashion we obtain the following metrics:

Precision: 0.9464
Recall: 0.8260
F_1 score: 0.8821

A gif illustrating correspondance between true and predicted prosodic
units — A gif illustrating correspondance between true and predicted prosodic units

As seen in the gif image above, we observe generally good correspondence between true (blue) and predicted (orange) prosodic units, but there are cases where the grouping is incorrect: the model will annotate only a single prosodic unit where a human annotator would annotate two or more.

Known limitations

Edge cases: if the input audio starts or ends within a prosodic unit, there is a high chance of not detecting the ending or starting prosodic unit.
Unknown behaviour on non-speech audio: as of the time of writing, no tests were performed to check what happens in cases of music, noise, pure sine, ...

Uses

Simple use (short files)

For shorter audios that fit on your GPU the classifier can be used directly.

python

import numpy as np

from datasets import Audio, Dataset
from transformers import AutoFeatureExtractor, Wav2Vec2BertForAudioFrameClassification
import torch
import numpy as np

if torch.cuda.is_available():
    device = torch.device("cuda")
else:
    device = torch.device("cpu")

model_name = "classla/wav2vecbert2-prosodicUnit"
feature_extractor = AutoFeatureExtractor.from_pretrained(model_name)
model = Wav2Vec2BertForAudioFrameClassification.from_pretrained(model_name).to(device)
f = "data/Rog-Art-N-G6007-P600702_181.070_211.070.wav"


def frames_to_intervals(frames: list) -> list[tuple]:
    from itertools import pairwise
    import pandas as pd

    results = []
    ndf = pd.DataFrame(
        data={
            "time_s": [0.020 * i for i in range(len(frames))],
            "frames": frames,
        }
    )
    ndf = ndf.dropna()
    indices_of_change = ndf.frames.diff()[ndf.frames.diff() != 0].index.values
    for si, ei in pairwise(indices_of_change):
        if ndf.loc[si : ei - 1, "frames"].mode()[0] == 0:
            pass
        else:
            results.append(
                (round(ndf.loc[si, "time_s"], 3), round(ndf.loc[ei - 1, "time_s"], 3))
            )
    return results


def evaluator(chunks):
    sampling_rate = chunks["audio"][0]["sampling_rate"]
    with torch.no_grad():
        inputs = feature_extractor(
            [i["array"] for i in chunks["audio"]],
            return_tensors="pt",
            sampling_rate=sampling_rate,
        ).to(device)
        logits = model(**inputs).logits
    y_pred_raw = np.array(logits.cpu())
    y_pred = y_pred_raw.argmax(axis=-1)
    prosodic_units = [frames_to_intervals(i) for i in y_pred]
    return {
        "y_pred": y_pred,
        "y_pred_logits": y_pred_raw,
        "prosodic_units": prosodic_units,
    }

# Create a dataset with a single instance and map our evaluator function on it:
ds = Dataset.from_dict({"audio": [f]}).cast_column("audio", Audio(16000, mono=True))
ds = ds.map(evaluator, batched=True, batch_size=1) # Adjust batch size according to your hardware specs
print(ds["y_pred"][0])
# Outputs: [0, 0, 1, 1, 1, 1, 1, ...]
print(ds["y_pred_logits"][0])
# Outputs:
# [[ 0.89419061, -0.77746612],
#  [ 0.44213724, -0.34862748],
#  [-0.08605709,  0.13012762],
# ....
print(ds["prosodic_units"][0])
# Outputs: [[0.04, 2.4], [3.52, 6.6], ....

Inference on longer files

If the file is too big for straight-forward inference, some chunking needs to be performed in order to process it. We know that for starts and ends of chunks the probability of false negatives increases, so it is best to process the file with some overlap between chunks or split it on silence. We illustrate the former approach here: