Views
No views yet
Note:Vektor_v0is a simplified MLA prototype (low-rank Q + separate low-rank K/V). Made as an experiment for testing.
| Attribute | Value |
|---|---|
| Architecture | Vektor_v0 — LLaMA-style transformer with MLA low-rank Q/KV |
| Parameters (non-embedding) | 177.3M |
| Parameters (total) | 228.8M |
| Hidden size | 1024 |
| Layers | 12 |
| Attention heads | 16 |
| Head dimension | 64 |
| Q/KV low-rank dimension | 192 |
| FFN intermediate size | 2752 (SwiGLU) |
| Max position embeddings | 16384 |
| RoPE base | 500,000 |
| Vocab size | 50,257 |
| Tokenizer | GPT-2 BPE |
| Precision | BF16 |
| License | Apache 2.0 |
EleutherAI/SmolLM2-135M-10BEleutherAI/SmolLM2-135M-10B validation: 37.91trust_remote_code=True:1from transformers import AutoModelForCausalLM, AutoTokenizer
2import torch
3
4model_id = "BlivionIaG/vektor-v0-170m"
5
6tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
7model = AutoModelForCausalLM.from_pretrained(
8 model_id,
9 trust_remote_code=True,
10 dtype=torch.bfloat16,
11).to("cuda")
12
13prompt = "The future of AI is"
14inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
15with torch.no_grad():
16 output = model.generate(
17 **inputs,
18 max_new_tokens=100,
19 temperature=0.7,
20 do_sample=True,
21 top_p=0.9,
22 eos_token_id=tokenizer.eos_token_id,
23 pad_token_id=tokenizer.pad_token_id,
24 )
25print(tokenizer.decode(output[0], skip_special_tokens=True))1@misc{vektor2026,
2 title = {Vektor Training Pipeline},
3 author = {BlivionIaG},
4 year = {2026},
5 howpublished = {\url{https://github.com/BlivionIaG/vektor-training}}
6}
7
8@article{deepseekv2,
9 title={DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model},
10 author={DeepSeek-AI},
11 journal={arXiv preprint arXiv:2405.04434},
12 year={2024}
13}
14
15@article{deepseekv3,
16 title={DeepSeek-V3: Technical Report},
17 author={DeepSeek-AI},
18 journal={arXiv preprint arXiv:2412.19437},
19 year={2024}
20}