Views
No views yet
transformers library on a machine with GPUs.1import torch
2from transformers import pipeline
3generate_text = pipeline(model="PSanni/Deer-3b", torch_dtype=torch.bfloat16, trust_remote_code=True, device_map="auto")1res = generate_text("Explain to me the difference between nuclear fission and fusion.")
2print(res[0]["generated_text"])| Metric | Value |
|---|---|
| Avg. | 32.01 |
| ARC (25-shot) | 38.48 |
| HellaSwag (10-shot) | 57.41 |
| MMLU (5-shot) | 25.64 |
| TruthfulQA (0-shot) | 39.98 |
| Winogrande (5-shot) | 57.46 |
| GSM8K (5-shot) | 0.3 |
| DROP (3-shot) | 4.83 |