The largest model in the from-scratch ORCH lineup. Designed for full-stack Next.js generation: not just snippets, but complete project structures — TypeScript components, App Router pages, server actions, Prisma schemas, Tailwind utilities, and configuration files.
This is not a fine-tune of any pretrained model. Architecture and weights are trained end-to-end on curated Next.js repositories.
Standard LLaMA-style ingredients (RoPE, GQA, SwiGLU, RMSNorm) at scale. The 16,384-token context length allows the model to keep a meaningful portion of a project in context during generation.
Training
Data: curated Next.js repositories from GitHub
Hardware: single NVIDIA A40 48GB (RunPod)
Duration: ~2 hours
Epochs: 3 (~29,000 steps)
Batch size: 1 with gradient accumulation = 16
Sequence length: 512 (training) — the 16K context length applies at inference
Precision: BFloat16
Generated artifacts
The model is trained to produce complete project structures:
Custom PyTorch format — use the ORCH inference code:
python
1import torch
2from orch.model.config import OrchConfig
3from orch.model.transformer import OrchForCausalLM
45model = OrchForCausalLM.from_pretrained("raihan-js/orch-nextjs-3b")6# ... use the same tokenizer.json from the repo
Intended use
Full Next.js project bootstrapping from a natural language description
Research into scaling SLMs trained from scratch on domain-specific data
A from-scratch baseline to compare against fine-tuned models like ORCH-7B
Limitations
Training data scale: 3 epochs on curated Next.js repos. Don't expect the world knowledge of a 7B+ general-purpose model.
Sequence length during training (512): capable of using long context at inference but may show degradation outside the training distribution.
No safety alignment.
Custom format: requires the ORCH inference code, not loadable with AutoModelForCausalLM.
