⚠️ Important Notice: Running the Model Locally (Ollama / LM Studio)
Why isn't this model supported in Ollama or LM Studio?
Unlike standard models (such as Llama, Mistral, or Qwen), The Architect is built on a completely custom, from-scratch PyTorch architecture. It features experimental mathematical layers that do not exist in standard Transformers, specifically:
Holographic State Memory (HSM)
Neuro-Symbolic Gating (NSRG)
Because these custom operations are not part of the standard Transformer vocabulary, C++ based inference engines (like llama.cpp, which powers Ollama and LM Studio) cannot natively understand or execute them. Converting this model to GGUF format for Ollama would require writing custom C++ kernels for the HSM and NSRG layers, which is a complex engineering task planned for the future roadmap.
How can you test and use this model?
To ensure everyone can easily run and test this model locally without dealing with complex code, we provide a one-click Python Web UI. It creates a ChatGPT-like interface right in your browser.
Download the app.py and run.bat (Windows) or run.sh (Linux/Mac) scripts provided in this repository.
Ensure you have Python installed on your system.
Double-click the run.bat file. It will automatically install the required libraries (torch, gradio, tokenizers) and launch a local web server.
Your browser will automatically open a clean, interactive UI where you can chat with the model immediately.
Like this
Terminal:
@user cd Luck-spark-100m-base
~/Luck-spark-100m-base
@user python3 app.py
Model indiriliyor ve yükleniyor, lütfen bekleyin...
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
Model hazır!
To create a public link, set share=True in launch().
No C++ compilation or terminal knowledge required!
A Boutique, From-Scratch Turkish Language Model
THE ARCHITECT is an independently developed, approximately 90M-parameter experimental language model designed and trained entirely from scratch with PyTorch.
The project explores a simple but ambitious research question:
How much language modeling capability can be obtained from a relatively small model through architectural experimentation, aggressive training, and carefully selected Turkish data?
Rather than attempting to compete directly with billion-parameter commercial or open-weight foundation models, THE ARCHITECT is intended as a research-oriented boutique language model: compact, experimental, transparent, and built to investigate alternative architectural ideas.
🧭 Project Philosophy
Modern language models increasingly rely on extremely large parameter counts and enormous computational budgets.
THE ARCHITECT takes a different approach.
The project investigates whether a comparatively small model can obtain useful language representations through:
substantial pre-training relative to its parameter count,
a Turkish-focused corpus,
a custom tokenizer,
architectural experimentation,
parameter sharing,
and experimental memory/gating mechanisms.
This project should not be interpreted as evidence that small models universally outperform large models. Instead, the goal is to investigate the efficiency frontier of language modeling at a small scale.
The model is therefore best understood as an independent research experiment, rather than a commercial competitor to systems such as GPT-class or Llama-class large language models.
🧠 Architecture
THE ARCHITECT is implemented directly in PyTorch rather than being assembled from a standard pretrained Transformer architecture.
The overall structure follows a decoder-only autoregressive language-modeling paradigm, while introducing experimental components inside each block.
THE ARCHITECT uses causal multi-head self-attention.
A triangular causal mask prevents a token from attending to future tokens during autoregressive training.
Formally, positions satisfy:
[
M_{ij} =
\begin{cases}
0 & j \leq i
-\infty & j > i
\end{cases}
]
This ensures that prediction at position i cannot directly access tokens occurring after position i.
Configuration
16 attention heads
64 dimensions per head
768-dimensional hidden representation
128-token context
2. SwiGLU Feed-Forward Network
The standard ReLU-style feed-forward layer is replaced by SwiGLU.
The implementation uses three linear projections:
[
\text{SwiGLU}(x)
W_2
\left(
\text{SiLU}(W_1x)
\odot
W_3x
\right)
]
The intermediate dimension is:
2048
SwiGLU-style feed-forward networks are widely used in modern language-model architectures and provide an efficient nonlinear transformation.
🧬 3. Holographic State Memory (HSM)
Holographic State Memory (HSM) is one of the experimental components of THE ARCHITECT.
HSM attempts to introduce a learned state representation that interacts with the current hidden representation.
Conceptually:
text
1Current Representation
2 │
3 ├──────────────┐
4 │ │
5 ▼ ▼
6 Current Learned
7 State State
8 │ │
9 └──────┬───────┘
10 ▼
11 State Binder
12 │
13 ▼
14 New Representation
The current implementation should be interpreted carefully.
Despite its name, HSM does not yet constitute a fully recurrent sequence-level memory mechanism. The current implementation behaves more like an experimental dynamic state/bias transformation inside each Architect block.
Layer normalization is used to stabilize its interaction with the main representation.
This distinction is intentional: the project documents what the mechanism currently does, rather than claiming capabilities that have not yet been demonstrated.
🧩 4. Neuro-Symbolic Gating (NSRG)
Neuro-Symbolic Gating (NSRG) is an experimental gating mechanism designed around four learned information-handling operations:
Gate
Concept
Preserve
Preserve existing information
Erase
Suppress existing information
Bind
Incorporate new information
Contradiction
Suppress contradictory information
The mechanism produces four normalized gate values through a softmax operation.
NSRG is experimental and should not be interpreted as a formally verified symbolic reasoning system.
The terminology describes the intended information-routing behavior of the mechanism, not a claim that the model possesses human-like symbolic reasoning.
🔗 Weight Tying
THE ARCHITECT shares the weights of:
text
1Token Embedding
2 ↕
3LM Head
This reduces parameter redundancy and improves parameter efficiency.
The embedding matrix is therefore reused when projecting hidden representations back into vocabulary logits.
📐 Model Configuration
Property
Value
Model name
THE ARCHITECT v0.1
Model type
Decoder-only autoregressive LM
Parameters
~90.72M
Vocabulary
32,000
Hidden dimension
768
Layers
8
Attention heads
16
Head dimension
64
FFN intermediate dimension
2,048
Context length
128 tokens
Tokenizer
Custom BPE
Framework
PyTorch
Training precision
Mixed Precision / AMP
Weight tying
Yes
Positional encoding
Not currently implemented
KV Cache
Not currently implemented
🗣️ Training Data
THE ARCHITECT was designed primarily around Turkish-language modeling.
The training corpus contains approximately:
~850 MB of text
~200–250M tokens
700K+ Turkish Wikipedia articles after filtering
conversational data from HuggingFaceH4/no_robots
Turkish instruction-oriented data
The corpus combines informational, conversational, and instruction-style text.
However, the current model is a base/pretrained language model, not an instruction-tuned assistant.
🔤 Tokenization
THE ARCHITECT uses a custom 32,000-token BPE tokenizer trained with the Hugging Face tokenizers library.
A TemplateProcessing post-processor was incorporated to improve handling of tokenization boundaries and special-token formatting.
The tokenizer was developed specifically for the project's Turkish-focused corpus rather than adopting the tokenizer of an existing large language model.
⚙️ Pre-Training
Training was performed using Kaggle GPU infrastructure, primarily Tesla T4 GPUs.
Optimization
Parameter
Configuration
Optimizer
AdamW
β₁
0.9
β₂
0.95
Weight decay
0.1
Initial learning rate
3e-4
Warmup
500 steps
Scheduler
Cosine decay
Batch size
16
Sequence length
128
Precision
PyTorch AMP
Gradient clipping
Enabled
The training system uses mixed precision through PyTorch AMP to reduce memory consumption and improve throughput.
📈 Training Progress
The initial training objective consisted of approximately 570,000 steps, corresponding to one planned epoch over the prepared training configuration.
During training, the observed loss decreased substantially from its initial values.
At the time of this model-card revision, more than 420,000 training steps had been completed.
Important: Training loss alone is not sufficient to establish model quality. A proper evaluation suite and held-out validation set are required before making quantitative claims about linguistic performance or reasoning capability.
💾 Checkpoint & MLOps System
The training pipeline was designed to survive temporary GPU environments such as Kaggle sessions.
Every 10,000 training steps, the system creates a checkpoint containing model and training state.
Training checkpoints are uploaded automatically during training.
After training, the model is additionally converted into a standard Hugging Face-compatible distribution containing the required configuration, model implementation and serialized weights.
The long-term goal is to make THE ARCHITECT usable through standard Hugging Face tooling rather than requiring the original training notebook.
🧪 Current Capabilities
THE ARCHITECT is currently a base language model.
It can perform autoregressive text continuation based on patterns learned during pre-training.
The model has demonstrated learning of Turkish linguistic patterns during training.
However, it should not currently be treated as a general-purpose conversational assistant.
For example, a prompt such as:
Türkiye'nin başkenti
may cause the model to continue the text in an encyclopedic or corpus-like style rather than responding conversationally.
This is expected behavior for a base model.
⚠️ Known Limitations
This section is intentionally explicit.
1. Base Model, Not a Chat Model
THE ARCHITECT has not yet undergone a dedicated instruction-tuning or preference-optimization stage.
It should therefore not be expected to behave like ChatGPT or other instruction-following assistants.
Planned:
text
1Pretraining
2 ↓
3SFT
4 ↓
5Instruction-following model
2. No Positional Encoding
The current architecture does not yet implement RoPE or another explicit positional embedding mechanism.
This is a significant architectural limitation.
Positional information is an important component of Transformer language models, and its absence may limit sequence modeling capability.
Adding a positional mechanism is therefore a major item on the roadmap.
3. Short Context
The current context length is:
128 tokens
This is extremely short compared with modern language models.
Long documents, conversations, and multi-paragraph reasoning cannot currently be represented effectively within a single context window.
4. No KV Cache
Inference currently recomputes attention over the sequence for each generated token.
A KV cache has not yet been implemented.
Consequently, autoregressive generation is substantially less efficient than it could be.
5. Experimental HSM
The name "Holographic State Memory" should not be interpreted as evidence of a sophisticated long-term memory system.
The current mechanism does not yet provide a fully persistent sequence-level memory.
It is an experimental architectural component whose usefulness requires further controlled evaluation.
6. Experimental NSRG
NSRG is an architectural hypothesis rather than a proven neuro-symbolic reasoning system.
The four gates are learned neural operations.
The terms:
Preserve
Erase
Bind
Contradiction
describe the intended information-routing semantics.
They do not imply formal symbolic reasoning, logical theorem proving, or guaranteed contradiction detection.
7. No Comprehensive Benchmark Yet
The project currently does not provide a comprehensive benchmark suite comparing THE ARCHITECT against established Turkish language models.
Therefore, claims about:
intelligence,
reasoning,
factual knowledge,
Turkish superiority,
parameter efficiency,
or general language-model performance
should be considered research hypotheses rather than established results.
The goal is to transform the base language model into a model capable of following natural-language instructions.
Phase 4 — Evaluation
A dedicated evaluation suite is planned, including:
Turkish language understanding
Turkish text generation
reading comprehension
instruction following
factual knowledge
basic reasoning
perplexity
held-out validation loss
comparison with similarly sized Turkish/open models
A key goal is to measure whether the architectural experiments provide measurable benefits rather than relying solely on qualitative impressions.
🧪 Research Questions
THE ARCHITECT is ultimately an investigation into several questions:
Can a small model learn substantial Turkish linguistic structure?
Can aggressive training compensate for some of the limitations imposed by a small parameter budget?
Can experimental state mechanisms improve representation quality?
Can information-routing gates provide useful behavior without dramatically increasing parameter count?
How much architectural experimentation is worthwhile at the ~100M parameter scale?
These questions remain open.
The project is designed to produce empirical evidence rather than assume that the proposed mechanisms work simply because they sound theoretically promising.
📊 Evaluation Philosophy
THE ARCHITECT will prioritize measured results over architectural claims.
Future model releases should ideally report:
Metric
Purpose
Validation loss
Generalization
Perplexity
Language-model quality
Turkish benchmarks
Language understanding
Generation samples
Qualitative behavior
Instruction benchmarks
Following instructions
Inference speed
Practical usability
Memory usage
Deployment efficiency
Parameter count
Model efficiency
The objective is not merely to build an unusual architecture, but to determine whether the unusual components provide measurable value.
🏗️ Why "THE ARCHITECT"?
The name reflects the central idea behind the project:
Intelligence is not only a question of how much material is used, but also how that material is organized.
The project therefore treats model architecture as an experimental object rather than assuming that scaling parameters is always the only meaningful direction.
THE ARCHITECT is an attempt to explore that space at a scale accessible to an independent researcher.
📜 Project Status
Status: Experimental / Research
Version: v0.1 — Base Model
Training: In progress
Primary language: Turkish
Model size: ~90M parameters
Instruction tuning: Not yet completed
Production readiness: ❌ No
Research/experimentation: ✅ Yes
⚖️ Intended Use
THE ARCHITECT is intended for:
educational research
language-model experimentation
Turkish NLP research
architecture experimentation
small-model research
reproducibility experiments
Hugging Face ecosystem experimentation
It is not currently recommended for:
production conversational AI
high-stakes decision making
factual information retrieval without verification
medical or legal applications
autonomous systems
safety-critical applications
🧑🔬 Development
THE ARCHITECT is an independently developed project.
It is intentionally developed without the infrastructure or computational resources of a large AI organization.
The project therefore emphasizes:
transparency,
reproducibility,
experimentation,
documenting failures,
and distinguishing hypotheses from demonstrated results.
Architectural components that do not work as expected are considered part of the research process rather than hidden from the model documentation.
🙏 Acknowledgements
The project makes use of the broader open-source machine-learning ecosystem, including:
PyTorch
Hugging Face Hub
Hugging Face Tokenizers
Kaggle GPU infrastructure
publicly available Turkish-language datasets
HuggingFaceH4/no_robots
This project would not be practical without the open-source ecosystem surrounding modern machine learning.
📚 Citation
If you use THE ARCHITECT in research or experimentation, please cite the project as:
bibtex
1@misc{the_architect_2026,
2 title = {THE ARCHITECT v0.1: A Boutique From-Scratch Turkish Language Model},
3 author = {Independent AI Researcher},
4 year = {2026},
5 publisher = {Hugging Face},
6 note = {Experimental 90M-parameter Turkish language model}
7}
⚠️ Final Note
THE ARCHITECT is deliberately presented as an experiment, not a finished claim about the future of language modeling.
A 90M-parameter model is not expected to match the capabilities of modern multi-billion-parameter foundation models.
The interesting question is narrower:
How far can a carefully designed, aggressively trained, Turkish-focused small model be pushed—and which architectural ideas actually help?
That question is what THE ARCHITECT is being built to investigate.
