You can deploy this model and start earning money today!

linhanhvlog123/
phobert-viquad2.0-qa

Name: phobert-viquad2.0-qa
Availability: InStock

Views

No views yet

🇻🇳 PhoBERT for Vietnamese Question Answering (ViQuAD 2.0)

This repository contains a PhoBERT-based Question Answering (QA) model fine-tuned on the UIT-ViQuAD 2.0 dataset – a Vietnamese Machine Reading Comprehension benchmark with both answerable and unanswerable questions.

The model is trained and evaluated following the SQuAD v2.0-style extractive QA formulation, where the system predicts either:

a text span from the context, or
no answer if the question is unanswerable.

📚 Dataset: UIT-ViQuAD 2.0

UIT-ViQuAD 2.0 is a large-scale Vietnamese MRC dataset introduced at VLSP 2021, designed to address the limitation of earlier Vietnamese QA datasets that only contain answerable questions.

Dataset statistics

Split	# Questions
Train	28,457
Dev	3,821
Private Test	3,712

Each sample consists of:

context: a Vietnamese paragraph
question: a natural language question
answers: answer spans (empty for unanswerable questions)

🧩 Data preprocessing pipeline

To ensure compatibility with both Hugging Face Trainer and official SQuAD evaluation, the data is processed in two stages:

1️⃣ Convert original ViQuAD → SQuAD v2.0 format

Used for:

official evaluation with evaluate-v2.0.py
preserving original dataset structure

json

{
  "data": [
    {
      "paragraphs": [
        {
          "context": "...",
          "qas": [
            {
              "id": "...",
              "question": "...",
              "answers": [
                {
                  "text": "...",
                  "answer_start": 123
                }
              ],
              "is_impossible": false
            }
          ]
        }
      ]
    }
  ]
}

2️⃣ Convert SQuAD format → Hugging Face QA format

Used for training & inference with run_qa.py.

json

{
  "id": "...",
  "context": "...",
  "question": "...",
  "answers": {
    "text": ["..."],
    "answer_start": [123]
  }
}

🔁 This two-step preprocessing ensures fair evaluation and full compatibility with Hugging Face QA pipelines.

🤖 Model: PhoBERT-base

Base model: vinai/phobert-base
Architecture: RoBERTa-style Transformer
Hidden size (d_model): 768
Number of layers: 12
Attention heads: 12

⚠️ Important technical notes (PhoBERT-specific)

🔹 Fast tokenizer workaround

PhoBERT does not officially provide a fast tokenizer compatible with Hugging Face QA scripts. To resolve this:

We force usage of RobertaTokenizerFast
Load PhoBERT vocabulary
Resize model token embeddings to match PhoBERT’s vocab size

python

tokenizer = RobertaTokenizerFast.from_pretrained("vinai/phobert-base", use_fast=True)
model.resize_token_embeddings(len(tokenizer))

This workaround allows PhoBERT to work seamlessly with run_qa.py.

🔹 Maximum sequence length constraint

PhoBERT has:

"max_position_embeddings": 258

Therefore:

Maximum usable max_seq_length is 256
Values like 384 or 512 will cause runtime errors

The extra 2 positions are reserved for special tokens (e.g. <s>, </s>), following RoBERTa-style encoding.

⚙️ Training configuration

bash

python run_qa.py \
  --model_name_or_path vinai/phobert-base \
  --train_file train_hf.json \
  --validation_file dev_hf.json \
  --do_train \
  --do_eval \
  --version_2_with_negative \
  --learning_rate 2e-5 \
  --num_train_epochs 3 \
  --per_device_train_batch_size 16 \
  --per_device_eval_batch_size 8 \
  --max_seq_length 256 \
  --doc_stride 128 \
  --seed 42 \
  --fp16 \
  --output_dir phobert_viquad

ℹ️ A fixed hyperparameter setting is used across models (PhoBERT, mBERT, XLM-R) to ensure fair comparison. Model-specific optimization is left for future experiments.

📊 Evaluation results (Private Test Set)

Official evaluation using evaluate-v2.0.py (SQuAD v2.0 script):

json

{
  "exact": 45.64,
  "f1": 55.83,
  "HasAns_exact": 42.41,
  "HasAns_f1": 56.99,
  "NoAns_exact": 53.14,
  "NoAns_f1": 53.14,
  "total": 3712
}

Interpretation

The model shows balanced performance on both answerable and unanswerable questions.
Lower overall scores compared to XLM-R are expected due to:
- shorter maximum context length (256 tokens)
- no multilingual pretraining advantage

🔍 Comparison note

Model	Max Seq Length	EM	F1
mBERT	512	49.33	60.36
XLM-R	512	47.25	56.84
PhoBERT	256	45.64	55.83

PhoBERT trades Vietnamese-specific pretraining for shorter context capacity.

📌 Summary

✅ PhoBERT can be successfully fine-tuned for Vietnamese QA
⚠️ Requires fast-tokenizer workaround
⚠️ Strict max sequence length = 256
📉 Performance impacted by limited positional embeddings
📈 Still competitive given architectural constraints

linhanhvlog123/phobert-viquad2.0-qa

🇻🇳 PhoBERT for Vietnamese Question Answering (ViQuAD 2.0)

📚 Dataset: UIT-ViQuAD 2.0

Dataset statistics

🧩 Data preprocessing pipeline

1️⃣ Convert original ViQuAD → SQuAD v2.0 format

2️⃣ Convert SQuAD format → Hugging Face QA format

🤖 Model: PhoBERT-base

⚠️ Important technical notes (PhoBERT-specific)

🔹 Fast tokenizer workaround

🔹 Maximum sequence length constraint

⚙️ Training configuration

📊 Evaluation results (Private Test Set)

Interpretation

🔍 Comparison note

📌 Summary

linhanhvlog123/
phobert-viquad2.0-qa