AhiskaAI 25M Base v0.1 (Pre-trained From Scratch)
AhiskaAI 25M Base v0.1 is an ultra-lightweight, foundational causal language model trained entirely from scratch for the Turkish language ecosystem.
With only ~25 Million active parameters, this model was initialized from absolute zero to explore the lower limits of grammatical compression, syntax mapping, and localized semantic density within highly constrained computational environments. It serves as the raw, unaligned backbone for the AhıskaAI research pipeline.
🧠 Training Profile & Dataset
Despite its miniature footprint, the model captures core Turkish morphology and factual token paths due to a highly strategic, clean data mixture of a 5.3 GB base corpus:
- CulturaX (Turkish Split): Utilized for deep web-scale text distributions, teaching the network core token connectivity, basic sentence boundaries, and general Turkish vocabulary.
- Custom Filtered Wikipedia (75 MB): A highly curated, hand-filtered subset of Turkish Wikipedia explicitly processed by AhıskaAI to target rich historical timelines, cultural identity milestones, and factual knowledge. This dense sub-matrix is the primary driver behind the model's factual recall capabilities despite its size.
💻 Hardware & Infrastructure
- Hardware: NVIDIA GeForce RTX 4050 Laptop GPU (6GB VRAM)
- Training Depth: Trained under strict local VRAM constraints with an indie "Build in Public" ethos.
🛠️ Quickstart & Inference
You can easily load and run text generation using the Hugging Face transformers library.
1from transformers import GPT2LMHeadModel, AutoTokenizer
2import torch
3
4model_name = "AhiskaAI/AhiskaAI-25m-Base-v0.1"
5
6# Load model and custom Turkish tokenizer
7model = GPT2LMHeadModel.from_pretrained(model_name)
8tokenizer = AutoTokenizer.from_pretrained(model_name)
9
10# Configure pad token for GPT-2 architecture
11tokenizer.pad_token = tokenizer.eos_token
12
13# Sample generation
14prompt = "Ahıska Türkleri,"
15inputs = tokenizer(prompt, return_tensors="pt")
16
17with torch.no_grad():
18 outputs = model.generate(
19 **inputs,
20 max_length=100,
21 do_sample=True,
22 top_k=50,
23 top_p=0.95,
24 temperature=0.7,
25 no_repeat_ngram_size=2
26 )
27
28print(tokenizer.decode(outputs[0], skip_special_tokens=True))
⚠️ Model Limitations & Intended Use
Pure Base Model: This is a raw text completer and has not been aligned using Instruction Fine-Tuning (SFT) or RLHF. It will not act as a conversational chatbot out of the box and might loop sequences if not sampled correctly.
Intended Use: This model is highly receptive to immediate downstream SFT fine-tuning, vocabulary adaptations, or continuation of pre-training checkpoints.
For the aligned chat version, please check out: AhiskaAI-25m-Chat-v0.1-Experimental
