Sequence of words as a string. Example: "Here is some text to encode : Hello World", tokenized by Byte-Pair-Encoding. input_ids: Indices of input tokens in the vocabulary. It's a long tensor of dynamic shape (batch_size, sequence_length).

Preprocessing steps

Use tokenizer.encode() to encode the input text:

python

text = "Here is some text to encode : Hello World"
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
tokens_tensor = torch.tensor([torch.tensor(tokenizer.encode(text))])

Output of model

For GPT-2 model:

last_hidden_state: Sequence of hidden-states at the last layer of the model. It's a float tensor of size (batch_size, sequence_length, hidden_size). past: pre-computed hidden-states. It's a list of tensors (key and values in the attention blocks) of size (batch_size, num_heads, sequence_length, sequence_length), one per each layer.

Output of this model is the tuple (last_hidden_state, past)

For GPT-2-LM-HEAD model:

prediction_scores: Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). It's a float tensor of size (batch_size, sequence_length, vocab_size). past: pre-computed hidden-states. It's a list of tensors (key and values in the attention blocks) of size (batch_size, num_heads, sequence_length, sequence_length), one per each layer.

Output of this model is the tuple (prediction_scores, past)

Note that output_hidden_states=False and output_attentions=False in the PretrainedConfig configs.

Postprocessing steps

For GPT-2 model:

python

outputs = model(input_ids)
last_hidden_states = outputs[0]

For GPT-2-LM-HEAD model, to generate next 10 words:

import numpy as np
import torch
import torch.nn.functional as F
from transformers import GPT2Tokenizer

batch_size = 1
length = 10
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')

text = "Here is some text to encode : Hello World!"
tokens = np.array(tokenizer.encode(text))
context = torch.tensor(tokens, device=device, dtype=torch.long).unsqueeze(0).repeat(batch_size, 1)
prev = context
output = context

for i in range(length):
outputs = model(prev)
logits = outputs[0]
logits = logits[:, -1, :]
log_probs = F.softmax(logits, dim=-1)
_, prev = torch.topk(log_probs, k=1, dim=-1)
output = torch.cat((output, prev), dim=1)

output = output[:, len(tokens):].tolist()
generated = 0
for i in range(batch_size):
generated += 1
text = tokenizer.decode(output[i])
print(text)

Dataset (Train and validation)

The original model from OpenAI is pretrained on a dataset of 8 million web pages. The pretrained model is referenced in huggingface/transformers repository as a causal (unidirectional) transformer pre-trained using language modeling on a very large corpus of ~40 GB of text data. https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-pytorch_model.bin

Validation accuracy

Metric and benchmarking details are provided by HuggingFace in this post.

Publication/Attribution

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, andIlya Sutskever. Language Models are Unsupervised Multitask Learners. 2019.

References

This model is converted directly from huggingface/transformers.

Contributors

Negin Raoof Joddiy Zhang

License

Apache 2.0 License

onnxmodelzoo/gpt2-lm-head-10

GPT-2

Use-cases

Description

Model

Source

Inference

Input to model