You can deploy this model and start earning money today!

henriceriocain/
HenriAI

Name: HenriAI
Availability: InStock

Views

No views yet

HenriAI - QLoRA-tuned GPT-J

This model is a QLoRA-tuned version of GPT-J-6B, trained on a diverse dataset including commonsense reasoning, academic essays, and natural conversations. It uses 4-bit quantization for efficient deployment while maintaining performance.

Model Details

Base Model: EleutherAI/gpt-j-6b
Training Technique: QLoRA (Quantized Low-Rank Adaptation)
Language: English
License: Same as base model (Apache 2.0)

Training Configuration

LoRA Parameters

Rank (r): 72
Alpha: 144
Dropout: 0.1
Target Modules: q_proj, v_proj, k_proj, out_proj, fc_in, fc_out
Bias: none

Quantization Settings

4-bit quantization (NF4)
FP16 compute dtype
Double quantization enabled

Training Process

Batch Size: 28
Learning Rate: 3e-4
Maximum Sequence Length: 512
Number of Epochs: 5
Gradient Accumulation Steps: 1
Optimizer: AdamW with Cosine Annealing LR

Training Data

The model was trained on multiple datasets including:

Commonsense reasoning data
Academic essays
Instruction sets
Natural language conversations
Introduction and conclusion examples

Usage

To use this model, format your inputs as:

prompt = f"Question: {your_question}\nAnswer:"

Recommended Generation Settings

python

generation_config = {
    "max_length": 512,
    "temperature": 0.7,
    "do_sample": True,
    "use_cache": True,
    "num_return_sequences": 1
}

Limitations

Inherits base model biases from GPT-J-6B
Optimized for Q&A format conversations
May require 4-bit quantization support for inference

Technical Requirements

transformers>=4.31.0
peft
bitsandbytes>=0.39.0
torch>=2.0.0
accelerate

Citation & Contact

If you use this model or want to learn more about it, please contact me at hceriocain@gmail.com

henriceriocain/HenriAI