This model has been obtained by fine-tuning
araft_trained_sft with DPO. The trajectories from the
Araft dataset were used to adapt the model to make a novel query at every step, instead of repeating the query from the previous one.
This model has been generated in the context of the
Araft project. The Araft project consists in fine-tuning a Llama2-7B model to enable the use of the
ReAct pattern for Wikipedia-augmented question-answering. This model is the product of the second and final training step: DPO training.
In the DPO training step, the trajectories from the
Araft dataset have been used to fine-tune the model. Each step was used as a desired output for the previous part of the trajectory, whereas the repetition of the previous step was used as undesired output. The model achieves a 26% performace (f1 score) on the
HotpotQA dataset.
For further information, please see the
Araft github repo.
