Orchestrated Recursive Code Hierarchy — a 272M-parameter decoder-only transformer trained from scratch on synthetic code data, designed to generate complete multi-file projects on a single consumer GPU.
Custom PyTorch (.pt) — not Hugging Face Transformers
License
Apache 2.0
Why this model exists
Most code-generation LLMs in the public conversation are either:
huge (70B+) and require a multi-GPU rig to even load,
proprietary (Copilot, Claude, ChatGPT) with per-token billing,
or fine-tuned from an existing base, which inherits everything the base learned.
ORCH Fusion goes the other direction: train a small model from scratch, with a custom tokenizer sized for code generation, on a single 12GB consumer GPU. The thesis is that for narrow tasks (generating well-structured multi-file projects), you don't need a frontier model — you need a model whose entire training signal is the task you care about.
Architecture
LLaMA-style decoder-only transformer, built from scratch:
Architectural features pulled directly from the LLaMA family: RoPE (rotary position embeddings), GQA (grouped-query attention with 4 KV heads — cheaper inference), SwiGLU activation, and RMSNorm. No pretrained weights from any other source.
Training
Data: synthetic code data (project-level multi-file examples), tokenized with a custom 2,103-vocab tokenizer fitted to the domain
Framework: custom PyTorch implementation (not the Hugging Face Trainer)
Hardware: NVIDIA RTX 3060 12GB (consumer)
Precision: mixed
Final benchmark (ORCH-ProjectBench, custom):
Overall score: 76.6
Code parse rate: 95.3 (% of generated code that parses cleanly)
Format correctness: 93.3 (% with correct project structure)
Usage
Because this model is in custom PyTorch format (not Hugging Face Transformers), you need the ORCH inference code.
python
1import torch
2from tokenizers import Tokenizer
3from orch import OrchForCausalLM # from github.com/raihan-js/orch45# Load model + tokenizer6model = OrchForCausalLM.from_pretrained("raihan-js/orch-fusion", subfolder="350m-project")7tokenizer = Tokenizer.from_file("orch-tokenizer.json")89prompt ="Create a React dashboard with authentication and dark mode"10ids = tokenizer.encode(prompt).ids
11input_ids = torch.tensor([ids])12output = model.generate(input_ids, max_new_tokens=2048, temperature=0.7)13print(tokenizer.decode(output[0].tolist()))
Output is a multi-file project structure rather than a single snippet.
