Model Card for Phi-4 LoRA Fine-tuned Model
Model Details
Model Description
This model is a fine-tuned version of Microsoft’s Phi-4, a large language model, adapted using Low-Rank Adaptation (LoRA) for efficient parameter tuning. LoRA enables fine-tuning with significantly fewer trainable parameters, reducing memory usage and training time, making it feasible to run on mid-range hardware without sacrificing performance. This model supports prompt-based tuning for fast adaptation to various NLP tasks.
- Developed by: Christinathattil
- Funded by: N/A
- Shared by: Christinathattil
- Model type: Transformer-based large language model (LLM) fine-tuned with LoRA
- Language(s) (NLP): English
Model Sources
Uses
Direct Use
This model can be used for natural language understanding and generation tasks in English. Due to efficient LoRA fine-tuning, it is suitable for developers and researchers who want to quickly adapt a large language model to specific domains or tasks without needing extensive compute resources.
Downstream Use
Can be further fine-tuned or integrated into applications requiring text generation, summarization, question answering, or other NLP capabilities, especially where compute or memory constraints are a concern.
Out-of-Scope Use
This model is not designed for real-time inference on very low-resource devices, nor for applications requiring multi-language support beyond English. It may also have limitations typical of LLMs, such as biases or generating incorrect/inappropriate outputs.
Bias, Risks, and Limitations
Like all large language models, this model may reflect biases present in training data. Users should be aware of potential ethical and societal impacts, including but not limited to misinformation, biased language generation, and fairness concerns.
Recommendations
Careful evaluation and monitoring are recommended when deploying this model in sensitive contexts. Users should implement safeguards such as content filtering and human oversight.
How to Get Started with the Model
Example usage to load and fine-tune with PEFT:
1from transformers import AutoModelForCausalLM, AutoTokenizer
2from peft import PeftModel
3
4base_model = "unsloth/phi-4-unsloth-bnb-4bit"
5tokenizer = AutoTokenizer.from_pretrained(base_model)
6model = AutoModelForCausalLM.from_pretrained(base_model)
7
8lora_model = PeftModel.from_pretrained(model, "your-lora-finetuned-checkpoint")
9
10# Use the lora_model for inference or further fine-tuning
Training Details
Training Data
Fine-tuned on mlabonne/FineTome-100k, a diverse dataset curated for fine-tuning language models with varied English text.
Training Procedure
- Precision: Mixed precision training (bf16)
- Framework: Hugging Face Transformers + PEFT
- LoRA settings: [Specify rank, alpha, dropout if possible]
- Compute: Mid-range GPU (e.g., NVIDIA RTX 3090 or equivalent)
- Duration: [Add training time here]
Technical Specifications
Model Architecture and Objective
Based on Phi-4 transformer architecture, with LoRA applied to select weight matrices to enable efficient parameter tuning.
Compute Infrastructure
- Hardware: NVIDIA RTX 3090 (or equivalent)
- Software: Hugging Face Transformers, PEFT 0.15.2, PyTorch
Citation
If you use this model, please cite:
1@misc{phi4-lora-finetuning,
2 author = {Christina Thattil},
3 title = {Fine-tuning Phi-4 Model using LoRA for efficient adaptation},
4 howpublished = {\url{https://github.com/Christinathattil/Fine-tuning-Phi-4-Model-using-LoRA-for-efficient-adaptation}},
5 year = {2025}
6}
Glossary
- LoRA: Low-Rank Adaptation, a parameter-efficient fine-tuning technique
- Phi-4: A large language model developed by Microsoft
- PEFT: Parameter-Efficient Fine-Tuning
Model Card Authors
Christina Thattil
Model Card Contact
[Add contact info or leave blank]
Framework versions
- PEFT 0.15.2
- Hugging Face Transformers [version]
- PyTorch [version]
