The model won 1st place in the ValueEval'24 competition on detecting the human values behind texts.
Model Details
Model Description
To address the multi-lingual nature of the problem and the differences in annotations between each language, a custom ensemble model was constructed. The architecture, as seen in Figure 2, leverages a pre-trained Transformer language model as its foundation. On top of this, nine custom Transformer heads were added, each tailored to a specific language: English, Greek, Dutch, Turkish, French, Bulgarian, Hebrew, Italian, and German.
Each custom Transformer head comprises the following components:
Three Transformer Layers which incorporate:
a) Self-Attention Mechanism: Allows the model to focus on different parts of the input se-
quence.
b) Layer Normalization: Stabilizes and accelerates the training process.
c) Feed-Forward Neural Network: Introduces non-linearity and complexity.
d) Residual Connection: Helps in mitigating the vanishing gradient problem and allows deeper networks.
e) Dropout: Prevents overfitting by randomly dropping units during training.
Classification Process:
a) The [CLS] token from the last Transformer layer (Transformer Layer 3) is passed through a dropout layer followed by a linear layer.
b) Finally, the output of the previous linear layer is passed through a Tanh activation function and then subjected to a dropout and a linear layer. The last linear layer produces logits corresponding to the number of classes.
Regarding the model training workflow, during each training iteration:
The input batch is fed into the pre-trained base model (Transformer).
The output of the pre-trained model is passed through the language splitter which splits it according to the language identifiers within the batch. Each split tensor is directed to the corresponding custom Transformer head based on its language for further processing.
The logits produced by each custom Transformer head are concatenated into a single batch through the language combiner.
The concatenated logits batch is passed through the loss function to compute the training loss.
Model performs backpropagation.
This approach allows the model to handle multiple languages effectively by utilizing specialized components tailored to the linguistic features and annotation styles of each language.
docker run --publish 8000:8000 -v $(pwd):/repository -it --rm johanneskiesel/value-detector-hierocles-of-alexandria
Citation
BibTeX:
{bibtex}
1@InProceedings{legkas:2024,
2 author = {Sotirios Legkas and Christina Christodoulou and Matthaios Zidianakis and Dimitrios Koutrintzes and Maria Dagioglou and Georgios Petasis},
3 booktitle = {Working Notes Papers of the CLEF 2024 Evaluation Labs},
4 editor = {Guglielmo Faggioli and Nicola Ferro and Petra Galuscakova and Alba Garc{\'i}a Seco Herrera},
5 month = sep,
6 pages = {3419--3432},
7 series = {CEUR Workshop Proceedings},
8 site = {Grenoble, France},
9 title = {Hierocles of Alexandria at Touch{\'e}: Multi-task \& Multi-head Custom Architecture with Transformer-based Models for Human Value Detection},
10 url = {http://ceur-ws.org/Vol-3740/paper-330.pdf},
11 volume = 3740,
12 year = 2024
13}
APA:
Legkas, S., Christodoulou, C., Zidianakis, M., Koutrintzes, D., Dagioglou, M., & Petasis, G. (2024, September). Hierocles of Alexandria at Touché: Multi-task & multi-head custom architecture with transformer-based models for human value detection. In G. Faggioli, N. Ferro, P. Galuscakova, & A. G. S. Herrera (Eds.), Working Notes Papers of the CLEF 2024 Evaluation Labs (Vol. 3740, pp. 3419–3432). CEUR Workshop Proceedings.
