BanglaSenti XLM-RoBERTa Experiment Models

Model Subdirectories

• ex-baseline-4-8-qv/ — Baseline LoRA (rank=4, alpha=8, query/value)
• ex-16-32-qv/ — LoRA (rank=16, alpha=32, query/value)
• ex-32-64-qv/ — LoRA (rank=32, alpha=64, query/value)
• ex-32-64-tm-qkv/ — LoRA (rank=32, alpha=64, query/key/value)
• ex-32-64-tm-all/ — LoRA (rank=32, alpha=64, query/key/value/dense) — Main SOTA
• ex-xlm-roberta-base/ — Baseline full fine-tuned XLM-RoBERTa (no LoRA)

• checkpoints/ — All model weights, adapter weights, tokenizer files, and info
  • banglasenti-lora-xlmr/ or banglasenti-xlmr/ (baseline)
    • lora_adapter_state_dict.pt (LoRA only)
    • lora_xlmr_weights.pt, final_lora_xlmr_weights.pt, or final_xlmr_weights.pt
    • model_info.txt (metadata for each checkpoint)
    • lora_adapter_weights/ or final_lora_adapter_weights/ (contains adapter_config.json, config.json)
    • lora_xlmr_tokenizer/, final_tokenizer/, or xlmr_tokenizer/ (contains tokenizer.json, tokenizer_config.json, special_tokens_map.json, sentencepiece.bpe.model)
    • final_state/ — Contains the final checkpoint after all training epochs, with the same structure as above
• configs/ — YAML config files for training and evaluation (train.yaml, eval.yaml, eval-xlm.yaml)
• logs/ — All logs for training and evaluation runs (train_banglasenti.log, train_banglasenti_main.log, eval_run.log, eval_run_xlm.log)

Checkpoint Types

• During-training: Intermediate checkpoints such as lora_adapter_state_dict.pt, lora_xlmr_weights.pt, and model_info.txt
• Final: Last checkpoint after training, located in the final_state/ subfolder, such as final_lora_adapter_state_dict.pt, final_lora_xlmr_weights.pt, final_xlmr_weights.pt, and model_info.txt

File Types

• Model weights: Files with the .pt extension (lora_adapter_state_dict.pt, lora_xlmr_weights.pt, final_lora_adapter_state_dict.pt, final_lora_xlmr_weights.pt, final_xlmr_weights.pt)
• Configs: config.json, adapter_config.json (with peft_type for LoRA)
• Tokenizer files: tokenizer.json, tokenizer_config.json, special_tokens_map.json, sentencepiece.bpe.model (the sentencepiece.bpe.model file is optional for LoRA adapters; if you face issues, see the main project documentation)
• Info: model_info.txt
• Training and evaluation configs: train.yaml, eval.yaml, eval-xlm.yaml
• Logs: train_banglasenti.log, eval_run.log, train_banglasenti_main.log, eval_run_xlm.log

Usage

• Load models and tokenizers using Hugging Face Transformers or PEFT, strictly from local files.
• For LoRA: Use PEFT/LoRA config and weights; the peft_type field must be present in the config.
• For baseline: Use standard Hugging Face model loading from the provided checkpoint and config.
• No external Hugging Face Hub calls are required for any operation.

Results Reference

• Project repository: LORA-FINETUNING-BANGLASENTI-XLMR-GOOGLE-TPU
• Full experiment report: experiment_report.md

License

Acknowledgement

• This research was supported by the Google Research TPU Research Cloud (TRC) program. Special thanks to the TRC team at Google Research for providing free access to Google Cloud TPUs, which made this work possible.
• The BanglaSenti dataset is from the open-source banglasenti-dataset-prep project.
• The base model xlm-roberta-base is provided by Facebook AI.
• This project builds on the Hugging Face Transformers and PEFT libraries.
• Thanks to the open-source community and all contributors to the code, data, and research.

Citation

• docs/dataset_citations.md (main project)
• CITATIONS.md from banglasenti-dataset-prep