Views
No views yet
El primer modelo de lenguaje de Pedus Company
| Especificacion | Valor |
|---|---|
| Parametros | 5.5M |
| Arquitectura | Causal LM (GPT-style) |
| Contexto | 128 tokens |
| Capas | 6 |
| Cabezas de atencion | 6 |
| Dimension | 96 |
| Vocabulario | 50,257 tokens |
| Precision | FP32 |
| Training Steps | 10,000 |
| Dataset | Wikipedia ES (50M tokens) |
| Hardware | CPU (GitHub Codespaces Pro) |
| Framework | PyTorch puro |
1import torch
2from model import PedusAINano
3from config import *
4import tiktoken
5
6model = PedusAINano()
7checkpoint = torch.load(
8 'checkpoints/pedusai_nano_best.pt',
9 map_location='cpu',
10 weights_only=False
11)
12model.load_state_dict(checkpoint['model_state_dict'])
13model.eval()
14
15tokenizer = tiktoken.get_encoding('gpt2')
16prompt = '<|user|>Hola<|bot|>'
17tokens = tokenizer.encode(prompt)
18input_ids = torch.tensor([tokens], dtype=torch.long)
19
20with torch.no_grad():
21 output = model.generate(
22 input_ids,
23 max_new_tokens=100,
24 temperature=0.7,
25 top_k=40
26 )
27
28response = tokenizer.decode(output[0].tolist()[len(tokens):])
29print(response)
30PedusAI Nano 1 DONE 5.5M params (Este modelo)
PedusAI Nano 2 WIP 30M params (Ryzen 5600G)
PedusAI 1 TODO 1.1B params (RTX 5060 + LoRA)
PedusAI 2 TODO 7B params (RTX 5060 + QLoRA)