You can deploy this model and start earning money today!

sparsh35/
edly

Name: edly
Rating: 2.5 (9 reviews)

qwen2

Views

No views yet

BaseTrainer

🚀 Trained With EasyDeL

EasyDeL is an open-source framework designed to enhance and streamline the training process of machine learning models. With a primary focus on Jax, EasyDeL aims to provide convenient and effective solutions for training Flax/Jax models on TPU/GPU, for both serving and training purposes.

📦 Installation & Usage

python

from easydel import AutoEasyDeLModelForCausalLM
from jax import numpy as jnp, lax

model = AutoEasyDeLModelForCausalLM.from_pretrained(
    f"REPO_ID/BaseTrainer",
    dtype=...,
    param_dtype=...,
    precision=lax.Precision("fastest"),
    auto_shard_model=True,
)

🔧 Training Configuration

Model Details

Architecture: qwen2
Platform: TPU
Number of Devices: 16

Training Parameters

Learning Rate: 5e-05 → 5e-06
Optimizer: adamw
Scheduler: cosine
Warmup Steps: 160
Weight Decay: 0.02
Loss Config: LossConfig( ignore_index: -100 label_smoothing: 0.0 z_loss: 0.0 loss_normalizing_factor: 'NUM_REAL_TARGET_TOKENS' num_labels: None problem_type: None divide_weight_sum: False shift_tokens: True break_on_nan: True reduction: None num_classification_labels: None classification_problem_type: None )

Training Setup

Epochs: 5
Batch Size: 16
Sequence Length: 4096
Dtype: <class 'jax.numpy.bfloat16'>
Params Dtype: <class 'jax.numpy.bfloat16'>

Advanced Configuration

Gradient Checkpointing:
Gradient Accumulation Steps: 1
Max Training Steps: None
Max Evaluation Steps: None
Training Duration: 7H

Sharding Configuration