Views
No views yet
| Parameter | Value |
|---|---|
| Sequence Length | 2048 |
| Number of Layers | 24 |
| Embedding Size | 2,560 |
| FFN Hidden Size | 10,240 |
| Number of Heads | 20 |
| Number of KV Heads | 5 |
| Activation Function | SiLU |
| Position Encodings | RoPE (Θ=500,000) |
| Layer Norm | RMSNorm (ε=10⁻⁵) |
| Tied Embeddings | No |
1from transformers import AutoTokenizer, AutoModelForCausalLM, TextStreamer
2
3device = "cuda"
4model_id = "faur-ai/LLMic_v2"
5prompt = "Capitala României este"
6
7model = AutoModelForCausalLM.from_pretrained(model_id).to(device)
8tokenizer = AutoTokenizer.from_pretrained(model_id)
9streamer = TextStreamer(tokenizer)
10
11inputs = tokenizer.encode(
12 prompt,
13 add_special_tokens=False,
14 return_tensors='pt',
15).to(device)
16
17outputs = model.generate(
18 streamer=streamer,
19 input_ids=inputs,
20 temperature=0.8,
21 do_sample=True
22)@misc{bădoiu2025llmicromanianfoundationlanguage,
title={LLMic: Romanian Foundation Language Model},
author={Vlad-Andrei Bădoiu and Mihai-Valentin Dumitru and Alexandru M. Gherghescu and Alexandru Agache and Costin Raiciu},
year={2025},
eprint={2501.07721},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2501.07721},
}