It is a 34.9 million parameter decoder-only Transformer trained entirely from scratch using FlatBuild, exported to GGUF, quantized to Q4_0, and optimized for fast CPU inference with FlatRun.

The original FP32 checkpoint is approximately 135 MB, while the quantized Q4_0 GGUF model significantly reduces memory and storage requirements, making it well suited for efficient local deployment on consumer hardware.

Experimental model: Flatbot-Mini-35M is designed for research, education, and experimentation. Although considerably more capable than Flatbot-Micro-4M, it may still generate hallucinations, factual inaccuracies, or inconsistent responses.

Architecture

Component	Details
Architecture	Decoder-only Transformer
Position Encoding	RoPE
Normalization	RMSNorm
Feed Forward	SwiGLU
Attention	Grouped Query Attention (16 Query Heads / 4 KV Heads)
Weight Tying	Yes
Context Length	512
Parameters	34.9M

Model Configuration

text

vocab_size      = 1024
hidden_size     = 512
num_layers      = 12
num_heads       = 16
num_kv_heads    = 4
head_dim        = 32
ffn_dim         = 1408
context_length  = 512
rope_theta      = 10000

Training

Flatbot-Mini-35M was trained entirely from random initialization using FlatBuild without relying on pretrained weights.

Dataset

Property	Value
Conversations	~10,000
Train Split	95%
Validation Split	5%
Context Length	512 tokens

The dataset contains approximately 10,000 multi-turn conversational examples covering greetings, question answering, explanations, recommendations, coding assistance, reasoning, and general-purpose assistant interactions.

Training Configuration

Hyperparameter	Value
Optimizer	AdamW
Learning Rate	1e-3
Scheduler	Cosine
Warmup	50 steps
Epochs	10
Batch Size	16
Gradient Accumulation	2
Precision	FP32

Quantization

This release is distributed as a GGUF Q4_0 model for efficient inference.

Item	Value
Original Format	FP32 SafeTensors
Original Size	~135 MB
Quantization	GGUF Q4_0
Parameters	34.9M
Optimized For	CPU inference
Compatible Runtimes	FlatRun, llama.cpp, LM Studio, Ollama (GGUF)

Features

Trained entirely from scratch
34.9M parameter Transformer
Custom tokenizer (1,024 vocabulary)
Native chat template
RoPE positional embeddings
RMSNorm normalization
SwiGLU feed-forward network
Grouped Query Attention (GQA)
Weight-tied embeddings
SafeTensors export
GGUF export
Q4_0 quantization
Native FlatRun compatibility

Usage

FlatBuild Training

bash

pip install flatbuild

flatbuild train configs/flatbot-mini-35M.yaml

flatbuild export \
    outputs/flatbot-mini-35M/*/checkpoint/final \
    --format gguf \
    --output flatbot-mini-35M

LM Studio

lms import flatbot-mini-35M-Q4_0.gguf

FlatRun Inference

bash

pip install flatrun

flatrun chat \
    --model flatbot-mini-35M-Q4_0.gguf \
    --temp 0.2

Example:

text

Detected format: gguf
Building tokenizer from GGUF metadata...
Loaded model in 0.03 s

You: Who are you?

Assistant:
I'm Flatbot, a conversational AI assistant trained from scratch using the Flatseek ecosystem. I'm here to answer questions, explain concepts, and help with everyday tasks.

Purpose

Flatbot-Mini-35M demonstrates the complete Flatseek AI development pipeline:

Build a conversational dataset
Train a tokenizer
Configure a Transformer architecture
Train entirely from scratch
Export SafeTensors checkpoints
Convert to GGUF
Quantize to Q4_0
Run efficient local inference with FlatRun

The entire workflow is fully reproducible on consumer hardware and showcases how modern language models can be trained, exported, quantized, and deployed without relying on proprietary foundation models.

Limitations

Flatbot-Mini-35M is still a compact language model compared with modern foundation models containing billions of parameters. While it offers substantially stronger conversational ability than the earlier Flatbot-Micro-4M demonstration model, it remains limited in factual knowledge, complex reasoning, multilingual capability, and long-context understanding.

Its primary purpose is to demonstrate an end-to-end open-source training, quantization, and inference pipeline built entirely within the Flatseek ecosystem.

flatseek/flatbot-mini-35M

Flatbot-Mini-35M-Q4_0