AhıskaAI-10M-Experimental-v0.1 is an ultra-compact ~10 million parameter Small Language Model (SLM) trained from scratch. As part of the AhıskaAI experimental series, this micro-model is designed to test the absolute lower limits of Turkish language representations, syntax learning, and synthetic task understanding at minimal computational costs.

Experimental Scope & Setup

Objective: Evaluate how much linguistic structure and reasoning a 10M parameter micro-model can capture when trained on dense synthetic data.
Training Epochs: Pre-trained for 1 full epoch across a ~3.5 GB synthetic Turkish corpus.
Vocabulary Efficiency: Uses an optimized 8,000 token vocabulary specifically suited for micro-scale models.

Model Details

Architecture: LlamaForCausalLM (Micro MHA)
Parameters: ~10M
Hidden Size: 288
Intermediate Size: 768
Attention Heads: 8 (Query) / 8 (Key/Value)
Layers: 8
Context Length: 512 tokens
Vocabulary Size: 8,000
Precision: float32

Pre-training Dataset Mixture

The model was pre-trained on a synthetic Turkish corpus (~3.5 GB total):

Data Source	Approx. Size	Primary Focus	Link
BILGEM AI Synthetic Web	~2.0 GB	General Web, Knowledge & Grammar	BILGEM-AI/BILGE-Synthetic-Web
BILGEM AI Synthetic Math	~750 MB	Mathematical Reasoning & Logic	BILGEM-AI/BILGE-Synthetic-Math
BILGEM AI Synthetic Stories	~750 MB	Narrative Flow & Text Generation	BILGEM-AI/BILGE-Synthetic-Stories

Usage

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "AhıskaAI/AhıskaAI-10M-Experimental-v0.1-Base"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32,
    device_map="auto"
)

prompt = "Türkiye"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=50,
    temperature=0.7,
    top_p=0.9,
    do_sample=True
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

About AhıskaAI

AhıskaAI is an independent initiative dedicated to developing efficient, high-performance Small Language Models (SLMs) tailored for the Turkish language ecosystem.

AhiskaAI/AhiskaAI-10M-Experimental-v0.1-Base

AhıskaAI-10M-Experimental-v0.1

Experimental Scope & Setup

Model Details

Pre-training Dataset Mixture

Usage

About AhıskaAI

AhiskaAI/
AhiskaAI-10M-Experimental-v0.1-Base