You can deploy this model and start earning money today!

keras/
whisper_large_multi

Name: whisper_large_multi
Rating: 2.5 (10 reviews)

keras-hub

speech-recognition

keras

automatic-speech-recognition

2212.04356

mit

Views

No views yet

Model Overview

⚠️ Whisper is currently only available via the keras-hub-nightly package. Use pip install keras-hub-nightly to try this model.

A Whisper encoder-decoder network for speech.

This class implements a Transformer-based encoder-decoder model as described in "Robust Speech Recognition via Large-Scale Weak Supervision". It includes the embedding lookups and transformer layers, but not the head for predicting the next token.

The default constructor gives a fully customizable, randomly initialized Whisper model with any number of layers, heads, and embedding dimensions. To load preset architectures and weights, use the from_preset() constructor.

Disclaimer: Pre-trained models are provided on an "as is" basis, without warranties or conditions of any kind. The underlying model is provided by a third party and subject to a separate license, available here.

Installation

Keras and KerasHub can be installed with:

pip install -U -q keras-hub
pip install -U -q keras

Jax, TensorFlow, and Torch come preinstalled in Kaggle Notebooks. For instructions on installing them in another environment see the Keras Getting Started page.

Arguments

vocabulary_size: int. The size of the token vocabulary.
num_layers: int. The number of transformer encoder layers and transformer decoder layers.
num_heads: int. The number of attention heads for each transformer. The hidden size must be divisible by the number of attention heads.
hidden_dim: int. The size of the transformer encoding and pooler layers.
intermediate_dim: int. The output dimension of the first Dense layer in a two-layer feedforward network for each transformer.
num_mels: int. The number of mel-frequency filters. Defaults to 80.
dropout: float. Dropout probability for the Transformer encoder.
max_encoder_sequence_length: int. The maximum sequence length that the audio encoder can consume. Since the second convolutional layer in the encoder reduces the sequence length by half (stride of 2), we use max_encoder_sequence_length // 2 as the sequence length for the positional embedding layer.
max_decoder_sequence_length: int. The maximum sequence length that the text decoder can consume.

Example Usage

python

import keras_hub
import keras_core as keras
import numpy as np

python

input_data = {
    "encoder_features": np.ones(shape=(1, 12, 80), dtype="int32"),
    "decoder_token_ids": np.ones(shape=(1, 12), dtype="int32"),
    "decoder_padding_mask": np.array(
        [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0]]
    ),
}

# Randomly initialized Whisper encoder-decoder model with a custom config.
model = keras_hub.models.WhisperBackbone(
    vocabulary_size=51864,
    num_layers=4,
    num_heads=4,
    hidden_dim=256,
    intermediate_dim=512,
    max_encoder_sequence_length=128,
    max_decoder_sequence_length=128,
)
model(input_data)

Example Usage with Hugging Face URI

python

import keras_hub
import keras_core as keras
import numpy as np

python

input_data = {
    "encoder_features": np.ones(shape=(1, 12, 80), dtype="int32"),
    "decoder_token_ids": np.ones(shape=(1, 12), dtype="int32"),
    "decoder_padding_mask": np.array(
        [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0]]
    ),
}

# Randomly initialized Whisper encoder-decoder model with a custom config.
model = keras_hub.models.WhisperBackbone(
    vocabulary_size=51864,
    num_layers=4,
    num_heads=4,
    hidden_dim=256,
    intermediate_dim=512,
    max_encoder_sequence_length=128,
    max_decoder_sequence_length=128,
)
model(input_data)

keras/whisper_large_multi

Model Overview

Links

Installation

Example Usage

Example Usage with Hugging Face URI

keras/
whisper_large_multi