This model was developed to support teaching and hands-on practice in the AIO course, while also serving as a general Vietnamese pretrained language model for experimentation and downstream adaptation.

Model Details

Model name: ViGPT2 AIO
Architecture: GPT-2
Task: Causal language modeling / text generation
Language: Vietnamese
Library: Transformers
Weights format: Safetensors

Training Data

The model was pretrained on a mixture of Vietnamese news and Vietnamese Wikipedia text.

Data Sources

BKAINewsCorpus from bkai-foundation-models/BKAINewsCorpus
Vietnamese Wikipedia collected through a custom crawling pipeline, then cleaned from raw wikitext into plain text

Data Processing

Before pretraining, the corpora were cleaned and deduplicated.

The tokenizer was trained on the raw corpora:
- bkai_train.parquet
- vi_wiki_articles_clean.parquet
The language model was pretrained on deduplicated versions of these corpora.
In the final training mixture, the Vietnamese Wikipedia corpus was upweighted relative to the news corpus.

Training Mixture

The pretraining mixture used:

bkai_train.parquet with weight 1
vi_wiki_articles_clean.parquet with weight 3

Training Objective

The model was pretrained with a standard causal language modeling objective, where the model learns to predict the next token in a sequence.

Limitations

The model may generate incorrect, nonsensical, or fabricated content.
Outputs can reflect biases or artifacts present in the pretraining data.
The model is not a verified factual source and should not be used without human validation in high-stakes settings.

Usage

python

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

repo_id = "VLAI-AIVN/vigpt2-aio"

tokenizer = AutoTokenizer.from_pretrained(repo_id)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(repo_id)
model.config.pad_token_id = tokenizer.pad_token_id

device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device).eval()

prompt = "Việt Nam là một đất nước"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=80,
        do_sample=True,
        temperature=0.8,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.pad_token_id,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))