Transformer-based language model for text generation.
Description
RoBERTa builds on BERT’s language masking strategy and modifies key hyperparameters in BERT, including removing BERT’s next-sentence pretraining objective, and training with much larger mini-batches and learning rates. RoBERTa was also trained on an order of magnitude more data than BERT, for a longer amount of time. This allows RoBERTa representations to generalize even better to downstream tasks compared to BERT.
Tutorial for running inference for RoBERTa-SequenceClassification model using onnxruntime can be found in the inference notebook.
Input
input_ids: Indices of input tokens in the vocabulary. It's a int64 tensor of dynamic shape (batch_size, sequence_length). Text tokenized by RobertaTokenizer.
For RoBERTa-BASE model:
Input is a sequence of words as a string. Example: "Text to encode: Hello, World"
For RoBERTa-SequenceClassification model:
Input is a sequence of words as a string including sentiment. Example: "This film is so good"
Preprocessing
For RoBERTa-BASE and RoBERTa-SequenceClassification model use tokenizer.encode() to encode the input text:
python
1import torch
2import numpy as np
3from simpletransformers.model import TransformerModel
4from transformers import RobertaForSequenceClassification, RobertaTokenizer
56text ="This film is so good"7tokenizer = RobertaTokenizer.from_pretrained('roberta-base')8input_ids = torch.tensor(tokenizer.encode(text, add_special_tokens=True)).unsqueeze(0)# Batch size 1
Output
For RoBERTa-BASE model:
Output of this model is a float32 tensors [batch_size,seq_len,768] and [batch_size,768]
For RoBERTa-SequenceClassification model:
Output of this model is a float32 tensor [batch_size, 2]
Postprocessing
For RoBERTa-BASE model:
last_hidden_states = ort_out[0]
For RoBERTa-SequenceClassification model:
Print sentiment prediction
