A Qwen3-style decoder-only transformer, implemented from scratch in PyTorch. It loads as a stock Qwen3ForCausalLM, so no trust_remote_code is required (needs transformers>=4.51).

Component	Choice
Normalisation	RMSNorm (pre-norm)
Positions	RoPE, theta = 1000000
Attention	Grouped-Query Attention (GQA)
QK-Norm	Yes (RMSNorm on Q and K, before RoPE)
Feed-forward	SwiGLU
Biases	None
Embeddings	Tied (input = output)

Size


Parameters	30.7M
Layers	8
Hidden size	512
Attention heads	8 (KV heads: 2)
Head dim	64
FFN size	1408
Context length	1024 (trained on 512-token windows)
Vocab	16000 — byte-level BPE trained on PHP
Weights	`model.safetensors` (float32)

The tokenizer was trained on this PHP corpus rather than reused, so it packs PHP efficiently (~4.1 characters per token) and can represent any byte sequence.

Training


Data	~24.5M tokens of PHP (29,844 files from 41 open-source projects)
Objective	Next-token prediction (causal LM)
Steps	4000
Validation loss	1.1826 (perplexity ~3.3)
Optimiser	AdamW, cosine LR decay with warmup
Hardware	Single free-tier GPU

What it does well

Continues PHP you have started: class bodies, method signatures, property declarations, docblocks.
Produces syntactically valid, idiomatic modern PHP — typed signatures (: string, ?int), constructor property promotion, fluent return $this;, PSR-style formatting.
Knows conventions of major frameworks it trained on (Symfony, Laravel, Doctrine, PHPUnit style).
Fast: roughly 0.3 s for a short completion on a CPU — usable as live editor autocomplete.

What it does NOT do

Please read this before using it — it is a small model and these limits are real:

It is not a chatbot. It does not follow instructions. Prompting it with "write me a function that sorts users" will not work — give it the start of code and it continues.
It does not understand your program's logic. It writes code that looks right more reliably than code that is right. Example: asked to complete add(Money $other), it may return 0 instead of summing.
It repeats itself. It sometimes emits the same method twice. Use a repetition penalty (~1.15) and keep max_new_tokens modest.
No fill-in-the-middle (FIM). It only sees code before the cursor. Do not send a suffix; use prefix-only completion.
It does not know your codebase, private APIs, or anything outside its training data.
Not for security-sensitive code. Always review output. It can produce insecure or non-functional code.
PHP only. Other languages will be poor.

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("AnandRago/PHPko-31M")
tok = AutoTokenizer.from_pretrained("AnandRago/PHPko-31M")

prompt = "<?php\n\nclass Invoice\n{\n    private array $lines = [];\n\n    public function getTotal(): float\n    {\n        return "
ids = tok(prompt, return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=64, do_sample=True,
                     temperature=0.2, top_p=0.9, repetition_penalty=1.15)
print(tok.decode(out[0], skip_special_tokens=True))

Recommended settings: temperature 0.2-0.5, top_p 0.9, repetition_penalty 1.15, max_new_tokens 48-96. Very low temperature (<0.1) makes it loop.

Training data & provenance

Trained only on permissively licensed (MIT / BSD-3) open-source PHP. GPL projects (e.g. WordPress, Drupal) were deliberately excluded so the corpus carries no copyleft obligations.

Files were filtered to remove vendor directories, tests, generated code, minified files, translation/lookup tables, and exact duplicates. License header comments were stripped.

Sources (each retains its original license)

Corpus sources. Each retains its original license.

Carbon                   MIT      https://github.com/briannesbitt/Carbon.git
FastRoute                BSD-3    https://github.com/nikic/FastRoute.git
PHP-Parser               BSD-3    https://github.com/nikic/PHP-Parser.git
Slim                     MIT      https://github.com/slimphp/Slim.git
Sylius                   MIT      https://github.com/Sylius/Sylius.git
Twig                     BSD-3    https://github.com/twigphp/Twig.git
bagisto                  MIT      https://github.com/bagisto/bagisto.git
cakephp                  MIT      https://github.com/cakephp/cakephp.git
collections              MIT      https://github.com/doctrine/collections.git
collision                MIT      https://github.com/nunomaduro/collision.git
commonmark               BSD-3    https://github.com/thephpleague/commonmark.git
composer                 MIT      https://github.com/composer/composer.git
core                     MIT      https://github.com/api-platform/core.git
csv                      MIT      https://github.com/thephpleague/csv.git
dbal                     MIT      https://github.com/doctrine/dbal.git
filament                 MIT      https://github.com/filamentphp/filament.git
flysystem                MIT      https://github.com/thephpleague/flysystem.git
framework                MIT      https://github.com/laravel/framework.git
guzzle                   MIT      https://github.com/guzzle/guzzle.git
horizon                  MIT      https://github.com/laravel/horizon.git
laminas-mvc              BSD-3    https://github.com/laminas/laminas-mvc.git
laravel                  MIT      https://github.com/laravel/laravel.git
laravel-medialibrary     MIT      https://github.com/spatie/laravel-medialibrary.git
laravel-permission       MIT      https://github.com/spatie/laravel-permission.git
livewire                 MIT      https://github.com/livewire/livewire.git
log                      MIT      https://github.com/php-fig/log.git
migrations               MIT      https://github.com/doctrine/migrations.git
monolog                  MIT      https://github.com/Seldaek/monolog.git
oauth2-server            MIT      https://github.com/thephpleague/oauth2-server.git
orm                      MIT      https://github.com/doctrine/orm.git
phpdotenv                BSD-3    https://github.com/vlucas/phpdotenv.git
phpstan-src              MIT      https://github.com/phpstan/phpstan-src.git
phpunit                  BSD-3    https://github.com/sebastianbergmann/phpunit.git
promises                 MIT      https://github.com/guzzle/promises.git
psr7                     MIT      https://github.com/guzzle/psr7.git
rector-src               MIT      https://github.com/rectorphp/rector-src.git
symfony                  MIT      https://github.com/symfony/symfony.git
symplify                 MIT      https://github.com/symplify/symplify.git
telescope                MIT      https://github.com/laravel/telescope.git
uuid                     MIT      https://github.com/ramsey/uuid.git
yii2                     BSD-3    https://github.com/yiisoft/yii2.git

Files kept per source repository

Repo	Files
symfony	7,498
Sylius	3,600
rector-src	3,077
filament	2,458
phpstan-src	1,884
framework	1,619
bagisto	1,433
core	1,357
phpunit	1,131
cakephp	788
livewire	596
Carbon	470
orm	459
yii2	459
dbal	435
Twig	335
composer	311
commonmark	298
PHP-Parser	271
flysystem	181
migrations	164
horizon	140
csv	125
monolog	123
uuid	114
laravel-medialibrary	93
telescope	86
oauth2-server	85
laminas-mvc	81
Slim	72
guzzle	47
laravel-permission	42
psr7	42
phpdotenv	41
FastRoute	35
collision	35
laravel	19
promises	18
collections	14
log	7
symplify	1

Total: 30,044 files, ~25.6M BPE tokens.

License

Apache-2.0 for the model weights. Each training source retains its own license (listed above); please respect them.