LlaMa-DUSFT is a custom variant of the LLaMA-2-7B model created using the DUS (Dynamic Update Strategy) methodology. The original LLaMA-2-7B model consists of 32 layers, and this variant introduces a novel approach to optimize performance by reconfiguring and expanding the layer architecture to 40 layers.
-
The original 32 layers of LLaMA-2-7B were duplicated.
-
In one variant, the last 12 layers were removed.
-
In another variant, the first 12 layers were removed.
This architectural modification was designed to test whether the DUS approach with an expanded layer count improves performance compared to the standard LLaMA-2 architecture.
Data preprocessing followed the guidelines for LLaMA-2 models, ensuring tokenization and alignment were consistent with the original architecture.
-
Due to the experimental nature of this model, specific evaluation metrics are currently limited.
-
Initial results indicate improved adaptability in specific downstream tasks from the OpenOrca dataset.
-
The DUS layer modification shows potential for enhancing model depth without significant degradation of performance.
-
Further evaluation with larger datasets and varied tasks is required to confirm generalizability.
