This model is a fine-tuned version of LLama3. The research involves a multi-step summarization approach to long, legal documents. Many decisions in the renewables energy space are heavily dependent on regulations. But these regulations are often long and complicated. The proposed architecture first uses one or more extractive summarization steps to compress the source text, before the final summary is created by the abstractive summarization model. This fine-tuned abstractive model has been trained on a dataset, pre-processed through extractive summarization by RoBERTa with dependent ratio. The research has used multiple extractive-abstractive model combinations, which can be found on
https://huggingface.co/MikaSie. To obtain optimal results, feed the model an extractive summary as input as it was designed this way!
The dataset used by this model is the
EUR-lex-sum dataset. The evaluation metrics can be found in the metadata of this model card.
This paper was introduced by the master thesis of Mika Sie at the University Utrecht in collaboration with Power2x. More information can be found in PAPER_LINK.
This model can be directly used for summarizing long, legal documents. However, it is recommended to first use an extractive summarization tool, such as RoBERTa, to compress the source text before feeding it to this model. This model has been specifically designed to work with extractive summaries.
An example using the Huggingface pipeline could be:
1pip install bert-extractive-summarizer
2
3from summarizer import Summarizer
4from transformers import pipeline
5
6extractive_model = Summarizer()
7
8text = 'Original document text to be summarized'
9
10extractive_summary = Summarizer(text)
11
12abstractive_model = pipeline('summarization', model = 'MikaSie/RoBERTa_LLama3_dependent_V1', tokenizer = 'MikaSie/RoBERTa_LLama3_dependent_V1')
13
14result = pipeline(extractive_summary)
But more information of implementation can be found in the Thesis report.
Using this model without an extractive summarization step may not yield optimal results. It is recommended to follow the proposed multi-step summarization approach outlined in the model description for best performance.
As with any language model, this model may inherit biases present in the training data. It is important to be aware of potential biases in the source text and to critically evaluate the generated summaries.
