QuantFactory/CabraMistral-v3-7b-32k-GGUF

Model Description

Detalhes do Modelo

Modelo: Mistral 7b Instruct 0.3

• Grouped-Query Attention
• Sliding-Window Attention
• Byte-fallback BPE tokenizer

dataset: Cabra 10k

Exemplo

Paramentros de trainamento

- learning_rate: 1e-05
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- distributed_type: multi-GPU
- num_devices: 2
- gradient_accumulation_steps: 8
- total_train_batch_size: 64
- total_eval_batch_size: 8
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.01
- num_epochs: 3

Framework

• Transformers 4.39.0.dev0
• Pytorch 2.1.2+cu118
• Datasets 2.14.6
• Tokenizers 0.15.2

Evals

Open Portuguese LLM Leaderboard Evaluation Results