Architecture: GPT-2 (decoder-only transformer)
Parameters: 218,040,320 trainable unique parameters (weight-tying between token embedding and output projection; zero biases added for Hugging Face compatibility)
Layers: 16
Attention heads: 16
Embedding dimension: 1024
Vocabulary: 16,000 tokens (SentencePiece BPE)
Sequence length: 256 tokens
Activation: GELU
Dropout: 0.1

Note on parameter count: During export, the weight-tying between the token embedding and the language model head is preserved, and zero-initialized bias tensors are added for compatibility with Hugging Face's GPT2LMHeadModel. These biases are not part of the original trained model and do not affect behavior.

Datasets

Portuguese Wikipedia corpus — used for autoregressive pretraining.
Alpaca PT-BR — Portuguese instruction-following dataset derived from Stanford Alpaca (Taori et al., 2023). Used for the first SFT stage.
Canarim-Instruct-PTBR — Portuguese instruction-following dataset by Maicon Domingues (Domingues, 2023). Used for the second SFT stage.

Tokenizer

Type: SentencePiece BPE
Vocabulary: 16,000 tokens
Special tokens: <unk> = 0, <bos> = 1, <eos> = 2, <pad> = 3
Pre-tokenizer: Metaspace (SentencePiece native)
Model max length: 256 tokens

The tokenizer was trained from scratch on the Portuguese Wikipedia corpus.

Training details

Type: sft_response_only
Best validation loss: 2.1538288593292236
Training steps: 1050

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "augustoafleal/gpt2-ptbr-218m"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

prompt = "A inteligência artificial é"
inputs = tokenizer(prompt, return_tensors="pt")

output = model.generate(
    **inputs,
    max_new_tokens=100,
    do_sample=True,
    temperature=0.7,
    top_k=40,
    pad_token_id=tokenizer.pad_token_id,
    eos_token_id=tokenizer.eos_token_id,
)

print(tokenizer.decode(output[0], skip_special_tokens=True))

Limitations

The model may generate factual errors.
The model may repeat phrases.
The model may fail to follow instructions exactly.
The context length is limited to 256 tokens.
The model was trained on limited data compared to modern LLMs.
This model is not suitable for high-stakes use without human validation.

Dataset citations

Stanford Alpaca: Taori et al., 2023. https://github.com/tatsu-lab/stanford_alpaca
Canarim-Instruct-PTBR: Domingues, 2023. https://huggingface.co/datasets/dominguesm/Canarim-Instruct-PTBR

Citation

If you use this model, please cite:

bibtex

@misc{gpt2ptbr218m,
  title        = {gpt2-ptbr-218m: A Portuguese GPT-2-like Autoregressive Language Model},
  author       = {Augusto Antônio Fontanive Leal},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/augustoafleal/gpt2-ptbr-218m}}
}