This model is based on the GPT-2 architecture and was further pre-trained (domain-adapted) using the text in AG-News dataset, excluding its test split.

Developed by: Cesar Gonzalez-Gutierrez
Funded by: ERC
Architecture: GPT-2
Language: English
License: MIT
Base model: GPT-2

Checkpoints

Intermediate checkpoints from the pre-training process are available and can be accessed using specific tags, which correspond to training epochs and steps:

Epoch	Step	Tags
1	1125	epoch-1	step-1125
5	5625	epoch-5	step-5625
10	11250	epoch-10	step-11250
20	22500	epoch-20	step-22500
30	33750	epoch-30	step-33750
40	45000	epoch-40	step-45000
50	56250	epoch-50	step-56250
60	67500	epoch-60	step-67500
70	78750	epoch-70	step-78750
80	90000	epoch-80	step-90000
90	101250	epoch-90	step-101250
100	112500	epoch-100	step-112500

To load a model from a specific intermediate checkpoint, use the revision parameter with the corresponding tag:

python

from transformers import AutoModelForCausalLM

model = AutoModelForMaskedLM.from_pretrained("<model-name>", revision="<checkpoint-tag>")

Sources

Paper: [Information pending]

Training Details

For more details on the training procedure, please refer to the base model's documentation: Training procedure.

Training Data

All texts from AG-News dataset, excluding the test partition.

Preprocessing [optional]

[More Information Needed]

Training Hyperparameters

Precision: fp16
Batch size: 8
Gradient accumulation steps: 12

Uses

For typical use cases and limitations, please refer to the base model's guidance: Inteded uses & limitations.

Bias, Risks, and Limitations

This model inherits potential risks and limitations from the base model. Refer to: Limitations and bias.

Environmental Impact

Hardware Type: NVIDIA A100 PCIE 40GB
Hours used: 14
Cluster Provider: Artemisa
Compute Region: EU
Carbon Emitted: 1.51 kg CO2 eq.

Citation

BibTeX:

[More Information Needed]

cglez/gpt2-dapt-ag_news

Model Card: GPT-2-DAPT-AG-News

Model Details

Description