Great, thanks for sharing the code. Let's start filling in the model card template based on the details provided.
Model Card for vishals9711/vishals9711_ft
Model Details
Model Description
Developed by: [Your Name or Team Name]
Funded by [optional]: [More Information Needed]
Shared by [optional]: [More Information Needed]
Model type: Causal Language Model (Fine-tuned with LoRA)
Language(s) (NLP): English (or other applicable languages)
License: [License Type]
Finetuned from model [optional]:TheBloke/TinyLlama-1.1B-Chat-v1.0-GPTQ
Model Sources [optional]
Repository: [Model Repository URL]
Paper [optional]: [Link to any relevant paper]
Demo [optional]: [Demo URL]
Uses
Direct Use
This model can be directly used for text generation tasks, including chatbot applications, headline generation, and similar tasks where causal language modeling is beneficial.
Downstream Use [optional]
The model can be further fine-tuned on specific datasets for tasks like sentiment analysis, summarization, etc.
Out-of-Scope Use
The model is not suitable for real-time applications without further optimization. It may also produce biased or harmful text if used without proper constraints and filtering.
Bias, Risks, and Limitations
This model, like all large language models, is subject to biases present in the training data. Users should be cautious when deploying this model in sensitive or high-stakes applications.
Recommendations
Users (both direct and downstream) should be made aware of the risks, biases, and limitations of the model. It is recommended to use safety mechanisms such as content filtering when deploying in production environments.
The model was fine-tuned on the vishals9711/tiny_llama_headline dataset, which includes various headlines for training purposes. Preprocessing involved tokenization with truncation to a maximum length of 2048 tokens.
Training Procedure
Preprocessing: The dataset was tokenized using AutoTokenizer with truncation on the left.
Training regime: The model was trained using 8-bit quantization, with gradient checkpointing enabled for memory efficiency. Training was performed using the LoRA technique with specific target modules for parameter-efficient tuning.
Training Hyperparameters:
Learning Rate: 2e-4
Batch Size: 4
Number of Epochs: 10
Weight Decay: 0.01
Gradient Accumulation Steps: 4
Warmup Steps: 2
Mixed Precision: FP16
Speeds, Sizes, Times [optional]
[More Information Needed]
Evaluation
Testing Data, Factors & Metrics
Testing Data
Evaluation was performed on a split of the vishals9711/tiny_llama_headline dataset.
Factors
[More Information Needed]
Metrics
Evaluation metrics included perplexity and loss on the evaluation dataset.
Results
[More Information Needed]
Environmental Impact
Hardware Type: [More Information Needed]
Hours used: [More Information Needed]
Cloud Provider: [More Information Needed]
Compute Region: [More Information Needed]
Carbon Emitted: [More Information Needed]
Technical Specifications [optional]
Model Architecture and Objective
The model architecture is based on the TinyLlama variant, optimized with LoRA for efficient fine-tuning.
Compute Infrastructure
[More Information Needed]
Hardware
[More Information Needed]
Software
Transformers library
PEFT library for LoRA
CUDA-enabled GPU for training
Citation [optional]
BibTeX:
[More Information Needed]
Feel free to provide more details or edit sections where necessary.
