Project Genesis is a research codebase for building a decoder-only large
language model from randomly initialized weights with Python and PyTorch.
Development proceeds in reviewed phases.
Phases 2 through 14 provide:
Safe typed YAML configuration with strict dotted overrides.
Runtime environment detection and configuration-relative paths.
Immutable dataset records, schemas, metadata, and statistics.
Deterministic local manifests, SHA-256 fingerprints, and integrity checks.
Dataset registry, cache contract, and atomic local manifest storage.
Deterministic readers for text, Markdown, JSON, JSONL, CSV, PDF, Git snapshots,
and local HTML snapshots.
Configurable normalization, filtering, exact deduplication, quality reports,
and processed-data manifests.
PyTorch token and position embeddings, custom LayerNorm, feed-forward layers,
causal multi-head self-attention, and strict residual addition.
Opt-in RoPE, RMSNorm, SwiGLU, grouped-query attention, and native scaled
dot-product attention while legacy bundles retain their original architecture.
Pre-normalization transformer blocks and a GPT-style decoder.
Deterministic next-token batching, AdamW, warmup/cosine scheduling, gradient
accumulation and clipping, mixed precision, and atomic resumable checkpoints.
Token-weighted validation, perplexity, named language and coding cases,
throughput measurements, canonical reports, and regression gates.
Cached autoregressive generation, greedy and stochastic sampling, stop/context
handling, immutable conversations, prompt assembly, and completion benchmarks.
Verified inference-only bundles, bounded generate/chat HTTP APIs,
health/readiness checks, structured request logs, and a non-root CPU container.
Required bundle provenance and semantic compatibility checks.
Fatal inference failure isolation, a dependency-free concurrent load probe,
operational recovery procedures, and checksummed attested releases.
Atomic end-to-end experiment execution from verified local sources through a
trained, evaluated, checkpointed, deployable inference bundle.
A non-allocating training preflight for source integrity, configuration
compatibility, parameter count, schedule size, and device capacity facts.
Recoverable long runs with periodic checkpoints, latest-checkpoint resume,
periodic validation, durable metrics, best-checkpoint selection, and retention.
Deterministic seeded epoch shuffling with exact microbatch replay after resume.
The opt-in modern-super.yaml profile is a
303,350,784-parameter, 24-layer decoder with 16 query heads, 4 key/value heads,
a 32,000-token vocabulary, and an 8,192-token context. Preflight estimates a
4.85 GB persistent training-state lower bound; activations and CUDA workspace
require additional memory. Select batch size and training sequence length only
after measuring the target GPU. Existing model configs and trained bundles
continue to use the legacy architecture.
Run a complete experiment with a dataset configuration that declares both
training and validation sources:
console
1genesis-preflight \
2 --dataset-config configs/dataset/experiment.yaml \
3 --device cuda
This 20-step run verifies the system. It is not enough data or training to
produce a useful coding assistant.
The CodeSearchNet Git repository contributes its MIT-licensed tooling source,
not the separately hosted multi-gigabyte CodeSearchNet corpus. Full-corpus use
requires a later sharded data pipeline.
CodeSearchNet Python CPU training
Materialize a deterministic 12,000-example training split and 1,000-example
validation split from the preserved CodeSearchNet Python archive:
uv run --locked python scripts/prepare_codesearchnet_instructions.py
The script downloads the 897 MiB archive once, verifies Zenodo's published size
and MD5, streams compressed JSONL without extracting it, filters oversized
functions, and formats each documentation/function pair with the same role
contract used by chat inference. The upstream functions carry mixed licenses;
the generated manifest retains source URLs for review and attribution. Do not
redistribute the generated sample without performing that review.
Validate and train the 4,935,680-parameter CPU profile:
This profile schedules 5.12 million training tokens. It is a meaningful
from-scratch CPU experiment, not a production coding assistant; much larger
models, broader licensed corpora, and GPU training are required for that.
Small English conversation CPU run
Download and deterministically split the pinned, human-written Databricks
Dolly-15k corpus (CC BY-SA 3.0):
py -m uv run --locked python scripts/prepare_english_conversations.py
The conversation run deliberately reuses the existing 4,935,680-parameter CPU
model, tokenizer, training, and evaluation profiles:
This is a bounded from-scratch experiment for short English replies, not a
general-purpose assistant. Keep the generated dataset attribution when sharing
adaptations, as required by CC BY-SA 3.0.
For a broader second CPU run, add the pinned Apache-2.0 OpenAssistant English
subset and train it together with Dolly rather than replacing the earlier data:
With genesis-serve running, open http://127.0.0.1:8000/ for the local chat
interface. The OpenAPI testing page remains available at
http://127.0.0.1:8000/docs.
Verify
console
1uv run --locked ruff format --check .
2uv run --locked ruff check .
3uv run --locked mypy src
4uv run --locked pytest
5uv build
