Views
No views yet
ignore_index = -1) and 1-token autoregressive shifting.unitary/toxic-bert) for input and output safety.F.scaled_dot_product_attention), and tied embedding weights.GPTConfig and TrainConfig.tiktoken BPE wrapper (50,257 vocabulary size).| Parameter | Value |
|---|---|
| Model Name | Apex-64M |
| Total Parameters | 63,823,360 (~63.82M) |
Embedding Dimension (d_model) | 512 |
Transformer Blocks (n_layer) | 12 |
Attention Heads (n_head) | 8 (head_dim = 64) |
Feed-Forward Dimension (d_ffn) | 2048 (4 x d_model) |
| Context Length | 512 tokens |
| Vocabulary Size | 50,257 (tiktoken GPT-2 / r50k_base) |
| Attention Mechanism | Fused FlashAttention (F.scaled_dot_product_attention) |
| Weight Sharing | Token Embedding <-> LM Output Head (wte.weight) |
| Pre-Training Loss | ~4.02 on C4 English (~1.28B tokens) |
| SFT Loss | ~3.17 on Databricks Dolly 15k (3 Epochs) |
1git clone https://github.com/kapilverse/SMALL_MODEL.git
2cd SMALL_MODEL
3pip install torch transformers datasets tiktoken gradio1import torch
2from config import GPTConfig
3from model import SmallGPT
4from tokenizer import get_tokenizer
5
6device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
7tokenizer = get_tokenizer()
8eot_id = tokenizer.eot_token
9
10config = GPTConfig()
11model = SmallGPT(config)
12
13# Load SFT Weights
14state_dict = torch.load("sft_model.pt", map_location=device)
15model.load_state_dict(state_dict)
16model = model.to(device).eval()
17
18def ask(question: str, temperature: float = 0.7, top_p: float = 0.9):
19 prompt = f"### Instruction:\n{question}\n\n### Response:\n"
20 tokens = tokenizer.encode(prompt)
21 idx = torch.tensor(tokens, dtype=torch.long, device=device).unsqueeze(0)
22
23 with torch.no_grad():
24 for _ in range(150):
25 idx_cond = idx[:, -config.context_length:]
26 logits, _ = model(idx_cond)
27 logits = logits[:, -1, :] / max(temperature, 1e-5)
28
29 # Top-P sampling
30 sorted_logits, sorted_indices = torch.sort(logits, descending=True)
31 cumulative_probs = torch.cumsum(torch.softmax(sorted_logits, dim=-1), dim=-1)
32 sorted_indices_to_remove = cumulative_probs > top_p
33 sorted_indices_to_remove[..., 1:] = sorted_indices_to_remove[..., :-1].clone()
34 sorted_indices_to_remove[..., 0] = 0
35 indices_to_remove = sorted_indices[sorted_indices_to_remove]
36 logits[:, indices_to_remove] = -float("Inf")
37
38 probs = torch.softmax(logits, dim=-1)
39 next_token = torch.multinomial(probs, num_samples=1)
40
41 if next_token.item() == eot_id:
42 break
43
44 idx = torch.cat((idx, next_token), dim=1)
45
46 # Early stop on new instruction header
47 current_text = tokenizer.decode(idx[0].tolist())
48 if "### Instruction:" in current_text.split("### Response:\n")[-1]:
49 break
50
51 full_text = tokenizer.decode(idx[0].tolist())
52 answer = full_text.split("### Response:\n")[-1].split("### Instruction:")[0].strip()
53 return answer.replace("<|endoftext|>", "")
54
55print("Apex:", ask("What is the difference between a stack and a queue?"))python app.py Stage 1: Pre-Training
---------------------
[C4 English ~1.28B Tokens] ---> [Apex-64M Base Foundation Model]
| (best_model.pt, val_loss ~4.02)
v
Stage 2: Supervised Fine-Tuning
-------------------------------
[Databricks Dolly 15k] ---> [Prompt Loss Masking & 1-Token Shift]
| (sft_model.pt, loss ~3.17)
v
Stage 3: Deployment
-------------------
[Apex-64M Assistant]
- Multi-Turn Memory Buffer
- Toxic-BERT Safety Guardrails
- Interactive Gradio Web UI