Views
No views yet
██╗ ██╗ ██╗███╗ ███╗██╗ ██████╗██╗ ██╗ █████╗ ████████╗███████╗
██║ ██║ ██║████╗ ████║██║██╔════╝██║ ██║██╔══██╗╚══██╔══╝██╔════╝
██║ ██║ ██║██╔████╔██║██║██║ ███████║███████║ ██║ ███████╗
██║ ██║ ██║██║╚██╔╝██║██║██║ ██╔══██║██╔══██║ ██║ ╚════██║
███████╗╚██████╔╝██║ ╚═╝ ██║██║╚██████╗██║ ██║██║ ██║ ██║ ███████║
╚══════╝ ╚═════╝ ╚═╝ ╚═╝╚═╝ ╚═════╝╚═╝ ╚═╝╚═╝ ╚═╝ ╚═╝ ╚══════╝⚠️ Critical Naming Clarification: This model is 14.7 Billion parameters. The name "Phi-4" refers to Microsoft's 4th generation Phi series — the "4" is a version number, NOT a parameter count. Do not confuse with Phi-3-mini (3.8B). This is a full 14B model, confirmed by the training output:Trainable parameters = 65,536,000 of 14,725,043,200.
| Capability | 🔲 Base Phi-4 (microsoft/phi-4) | 🔶 LumiChats-Instruct-14B-LoRA |
|---|---|---|
| Multi-turn conversation | Generic, not optimized | ✅ Specifically fine-tuned for dialogue |
| Instruction following | Moderate pretrain behavior | ✅ Reinforced via response-only training |
| Chat template | Requires manual configuration | ✅ Phi-4 template pre-applied & verified |
| Training data | 9.8T token web/book crawl | ✅ 99,990 curated conversational samples |
| Training objective | Predict every token equally | ✅ Only trains on assistant responses — no prompt memorization |
| Response structure | Unstructured completions | ✅ Consistent, well-formed reply format |
| Deployment readiness | Raw — needs post-processing | ✅ Drop-in ready for chat applications |
| Base knowledge preserved | N/A | ✅ 99.55% of weights untouched — full Phi-4 intelligence retained |
🔲 Base Phi-4 = A brilliant PhD graduate who has read everything but never held a conversation in a structured job.🔶 LumiChats-Instruct-14B-LoRA = That same PhD, now trained specifically to hold clear, helpful, structured conversations — same intelligence, purpose-built delivery.
| Property | Details |
|---|---|
| Model Name | LumiChats-Instruct-14B-LoRA |
| Developed By | LumiChats |
| Base Model | microsoft/phi-4 |
| Total Parameters | 14,725,043,200 (14.7 Billion) |
| Architecture | Dense Decoder-only Transformer — 40 layers |
| Fine-tuning Method | LoRA (Low-Rank Adaptation) via PEFT |
| Trainable Parameters | 65,536,000 (0.45% of 14.7B) |
| LoRA Rank | r = 16, alpha = 16 |
| Target Modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Quantization | 4-bit NF4 (bitsandbytes) |
| Context Length | 2048 tokens |
| Chat Format | Phi-4 (<|im_start|> / <|im_sep|> / <|im_end|>) |
| Language | English |
| License | MIT |
| Property | Value |
|---|---|
| Dataset | FineTome-100k by Maxime Labonne |
| Total Samples | 100,000 → 99,990 after quality filtering |
| Format | ShareGPT → HuggingFace (role, content) multi-turn |
| Topics Covered | Reasoning, Science, Mathematics, Coding, Logic, General Q&A |
| Training Objective | Response-only masking — loss computed on assistant turns only |
1# LoRA
2r = 16, lora_alpha = 16, lora_dropout = 0
3target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
4 "gate_proj", "up_proj", "down_proj"]
5
6# Optimizer
7optim = "adamw_8bit"
8learning_rate = 2e-4
9lr_scheduler_type = "linear"
10warmup_steps = 5
11weight_decay = 0.001
12
13# Batch
14per_device_train_batch_size = 2
15gradient_accumulation_steps = 4 # effective batch = 8
16max_steps = 30
17seed = 3407GPU Tesla T4 — 14.563 GB VRAM
Training Duration 19.28 minutes (30 steps)
Peak VRAM Used 13.242 GB (90.9% of T4)
LoRA Training Overhead 2.515 GB only
Trainable % of Model 0.45%
Framework Unsloth 2026.2.1 + TRL 0.22.2
Platform Google Colab| Benchmark | What It Tests | Phi-4 14B | GPT-4o-mini | Llama-3.3 70B | GPT-4o |
|---|---|---|---|---|---|
| MMLU | General Knowledge | 84.8 | 81.8 | 86.3 | 88.1 |
| GPQA | Graduate Science | 56.1 🏆 | 40.9 | 49.1 | 50.6 |
| MATH | Competition Math | 80.4 | 73.0 | 66.3 | 74.6 |
| HumanEval | Code Generation | 82.6 | 86.2 | 78.9 | 90.6 |
| MGSM | Multilingual Math | 80.6 | 86.5 | 89.1 | 90.4 |
| DROP | Reasoning | 75.5 | 79.3 | 90.2 | 80.9 |
🔶 Phi-4 outperforms GPT-4o on Graduate-level Science (GPQA) despite being ~14x smaller in parameter count.
pip install unsloth transformers bitsandbytes accelerate peft1from unsloth import FastLanguageModel
2from transformers import TextStreamer
3
4model, tokenizer = FastLanguageModel.from_pretrained(
5 model_name = "adityakum667388/LumiChats-Instruct-14B_lora",
6 max_seq_length = 2048,
7 load_in_4bit = True,
8)
9FastLanguageModel.for_inference(model)
10
11messages = [
12 {"role": "user", "content": "Explain quantum entanglement in simple terms."}
13]
14
15inputs = tokenizer.apply_chat_template(
16 messages,
17 tokenize = True,
18 add_generation_prompt = True,
19 return_tensors = "pt",
20).to("cuda")
21
22streamer = TextStreamer(tokenizer, skip_prompt=True)
23_ = model.generate(
24 input_ids = inputs,
25 streamer = streamer,
26 max_new_tokens = 256,
27 temperature = 1.5,
28 min_p = 0.1,
29 use_cache = True,
30)<|im_start|>system<|im_sep|>
You are a helpful AI assistant built by LumiChats.<|im_end|>
<|im_start|>user<|im_sep|>
Your question here<|im_end|>
<|im_start|>assistant<|im_sep|>| Status | Item |
|---|---|
| ✅ | LoRA adapter — public release |
| 🔜 | Full epoch production fine-tune |
| 🔜 | GGUF quantized versions (Q4_K_M, Q8_0) for local deployment |
| 🔜 | Domain-specific fine-tunes (education, enterprise, coding) |
| 🔜 | Extended 16K context version |
| 🔜 | Multilingual conversational variants |
1@misc{lumichats-instruct-14b-lora-2026,
2 author = {LumiChats},
3 title = {LumiChats-Instruct-14B-LoRA: Fine-tuned Microsoft Phi-4 for Conversational AI},
4 year = {2026},
5 publisher = {Hugging Face},
6 url = {https://huggingface.co/adityakum667388/LumiChats-Instruct-14B_lora}
7}