adapter: lora
base_model: mistralai/Mistral-7B-Instruct-v0.3
model_type: MistralForCausalLM
tokenizer_type: AutoTokenizer
bf16: true
dataset_processes: 32
datasets:
  - path: bytess/zrah-personal-ai
    type: alpaca
gradient_accumulation_steps: 4
gradient_checkpointing: false
learning_rate: 0.0002
lora_alpha: 32
lora_dropout: 0.05
lora_r: 16           # Or try 8 for smaller size later
lora_target_modules:
  - q_proj
  - v_proj
  - k_proj
  - o_proj
  - gate_proj
  - down_proj
  - up_proj
loraplus_lr_embedding: 1.0e-06
lr_scheduler: cosine
max_prompt_len: 512
micro_batch_size: 4  # Increase from 2 if GPU allows
num_epochs: 3
optimizer: adamw_torch
output_dir: ./outputs/zrah_model
pretrain_multipack_attn: true
sample_packing_bin_size: 200
sample_packing_group_size: 100000
save_only_model: true
save_safetensors: true
sequence_len: 2048
shuffle_merged_datasets: true
train_on_inputs: false
trl:
  use_vllm: false
val_set_size: 0.0
weight_decay: 0.0

Zrah Model 1.0

This model is a fine-tuned version of mistralai/Mistral-7B-Instruct-v0.3, trained using Axolotl on a RunPod instance with H100 PCIe GPU. The training was conducted for personal/self-aligned instruction tuning using a custom dataset: bytess/zrah-personal-ai.

⚠️ This model is intended only for demonstration and trial purposes as part of a personal portfolio. Use is allowed, but modification, fine-tuning, or redistribution is prohibited due to the private nature of the data.

Model Description

This model uses LoRA adapters to fine-tune Mistral-7B on a dataset of personal prompts and completions designed to align the model to the author's style, tone, and preferences. The model is intended for self-use in chat, writing assistance, and experimentation with instruction-following behavior.

Intended Uses & Limitations

Intended Uses:

Personal assistant/chatbot
Writing/code helper tailored to author's style
Experimentation with LoRA fine-tuning and self-alignment

Limitations:

Trained only on a small personal dataset; generalization to broader tasks is limited
Not evaluated on standardized benchmarks
May reflect idiosyncratic styles or biases present in the training data

Training and Evaluation Data

The model was trained on the bytess/zrah-personal-ai dataset. This dataset is structured in Alpaca-style (instruction, input, output) and consists of hand-curated personal instructions and completions.

Number of examples: 130 rows of data
Validation set: None used (val_set_size: 0.0)
Data shuffled and packed for efficiency using Axolotl's sample packing

Training Procedure

Hyperparameters

Base model: mistralai/Mistral-7B-v0.1
Adapter: LoRA (q_proj, k_proj, v_proj, o_proj, gate_proj, down_proj, up_proj)
LoRA Rank: 16, Alpha: 32, Dropout: 0.05
Batch size: 2 (micro) × 4 (gradient accumulation) = 8 total
Learning rate: 2e-4
Optimizer: AdamW
Scheduler: Cosine
Sequence length: 2048
Epochs: 3
Train on inputs: False (only outputs used for loss) Hardware
Trained on RunPod using an H100 PCIe GPU

Framework Versions

PEFT: 0.14.0
Transformers: 4.49.0
PyTorch: 2.5.1+cu124
Datasets: 3.2.0
Tokenizers: 0.21.0
Axolotl: 0.8.0.dev0

License

This model is provided under a custom license:

✅ Use allowed for testing and evaluation

❌ No fine-tuning

❌ No redistribution

❌ No use in commercial or production applications

Contact

For questions or portfolio purposes, contact project@ezrahernowo.com.

Model Card Authors

Ezra H

bytess/zrah-model-1.0