Views
No views yet
| Hyperparameter | Value |
|---|---|
| n_parameters | 1.3B |
| n_layers | 24 |
| n_heads | 16 |
| d_model | 2048 |
| vocab size | 50432 |
| sequence length | 2048 |
1import torch
2from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline
3
4tokenizer = AutoTokenizer.from_pretrained("team-lucid/mptk-1b")
5model = AutoModelForCausalLM.from_pretrained("team-lucid/mptk-1b")
6
7pipe = pipeline('text-generation', model=model, tokenizer=tokenizer, device='cuda:0')
8
9with torch.autocast('cuda', dtype=torch.bfloat16):
10 print(
11 pipe(
12 '대한민국의 수도는',
13 max_new_tokens=100,
14 do_sample=True,
15 )
16 )
17| Hyperparameter | Value |
|---|---|
| Precision | bfloat16 |
| Optimizer | Lion |
| Learning rate | 2e-4 |
| Batch size | 1024 |