A local Tileman.io-style multiplayer simulator, live AI spectator, and
compute-efficient reinforcement-learning environment. The simulation core is
written in Rust, the training interface uses Python/Gymnasium, and the
spectator runs in the browser.
Quick start
Requirements
Linux or macOS
Python 3.11 or newer
Node.js 18 or newer
Rust toolchain (only required for the fast native simulator)
Optional: an NVIDIA GPU with a working CUDA build of PyTorch
Clone the code and create an isolated Python environment:
Open http://127.0.0.1:4173. Choose a world size, simulation speed, AI
policies, and the number of instances of each policy, then press Start.
The selector includes the three main policies:
first: the original conservative capture policy
champion: the tournament-selected multiplayer champion
killer: the fresh-from-scratch combat policy
The world builder lets you choose:
A preset or custom world size and simulation speed
One to three trained AI policies
The exact number of instances controlled by each policy, up to 48 agents
Starting a simulation loads the selected policies on CUDA and opens the live
world. Use the Exit button in the spectator header to stop that simulation
and return to the world builder.
If CUDA is unavailable, change the device to cpu in direct Python commands.
You can verify GPU support with:
If maturin develop cannot find Rust, install it from
https://rustup.rs, restart the shell, and rerun the command.
If the Rust extension is not built, the environments still run using the
slower reference Python implementation.
If hf is not found, reactivate .venv or run
python -m pip install huggingface_hub.
The first CUDA launch may take a few seconds while PyTorch initializes and
loads the policies.
Trained multi-agent spectator
Run 32 clones of the best trained PPO policy on one shared 256×180 world:
npm run spectate -- --device cuda
Open http://127.0.0.1:4174. The first launch can take a little while while PyTorch initializes CUDA and loads the model.
Spectator controls:
Click any name in the left leaderboard to smoothly follow that AI.
Scroll anywhere over the world to zoom without releasing the followed AI.
Use WASD, the arrow keys, or pointer dragging to release follow mode and explore freely.
Use Fit world to detach the camera and frame the complete map.
Click another leaderboard name at any time to resume smooth follow mode.
The server batches every visible clone through runs/ppo-conservative/best/best_model.zip; these are instances of the trained policy, not scripted browser bots. World size, population, model, and tick rate are configurable:
To watch the Killer policy, select its checkpoint explicitly:
bash
1npm run spectate -- \2 --model runs/ppo-killer-v1-cuda/best/best_model.zip \3 --device cuda
The spectator detects whether a model uses the original four observation channels or the multiplayer seven-channel view, so both generations of checkpoints remain usable.
To place half of the old agents and half of the new agents on the same map:
bash
1npm run spectate -- \2 --model runs/ppo-conservative/best/best_model.zip \3 --model-b runs/ppo-champion/best_model.zip \4 --agents 32\5 --device cuda
The leaderboard labels the two groups OLD and NEW. With an odd population, the extra agent is assigned to the new model.
The same champion match has a shortcut:
npm run spectate:champion
To compare all three generations with 11 agents each:
npm run spectate:all
This runs the original model (OLD), the fresh-from-scratch combat policy
(KILLER), and the tournament-selected champion (CHAMP) on one shared map.
Policy assignments are interleaved by agent ID to reduce movement-order bias.
Current champion
runs/ppo-champion/best_model.zip was selected by paired-seed tournaments, not training reward. Every world seed is replayed with model assignments swapped to cancel spawn and agent-ID advantages. In the final 32-agent, 256×180, 1,000-tick validation, the champion controlled first place for 56.4% of ticks versus 43.7% for the old model, with more captures and kills and slightly fewer deaths.
The champion is an evolved variant of the old capture policy with a small deterministic action-logit adjustment, discovered after PPO league branches were rejected whenever they failed the same tournament. Reproduce a comparison with:
Actions are up, right, down, and left. An observation contains a flat Uint8Array grid (0 empty, 1 owned, 2 trail), grid shape, player position/direction, trail length, area, and step count.
The browser reward now mirrors the conservative training reward: captures are positive, dying is -1, and exposed trails longer than 12 cells become progressively more expensive.
Reinforcement-learning environment
The Python environment uses:
A configurable 128×128 world by default
A rotated 43×25 semantic viewport centred on the player
Three relative actions: straight, left, and right
Compact statistics: trail length, time outside, direction to home, and coverage
No browser or rendering work during training
The reward is:
text
1+0.01 × newly captured tiles
2-0.0001 per step
3-0.002 per step outside owned territory
4-0.001 × every trail cell beyond 12
5-1.0 on death
The excess-trail cost is applied on every exposed step. Long trails therefore become increasingly unattractive, but remain possible when a large capture justifies the risk.
Run all Python contract and rule tests:
python3 -m unittest discover -s tests -v
Rust simulator
The single-agent and multiplayer training environments automatically use a
native Rust core when it is installed. The multiplayer training core keeps the
world, scripted opponent plans, trail kills, capture flood fill, respawns,
rewards, and observation buffers in Rust, crossing into Python once per Gym
step. The general shared-world class also has a compatibility-oriented Rust
backend while preserving its mutable NumPy grids for spectators and policy
leagues.
If the native module has not been built, both environments safely fall back to
their reference Python implementations. Pass backend="python" to
MultiAgentWorld or MultiplayerTilemanEnv for differential debugging;
PythonTilemanTrainingEnv remains available for single-agent parity testing.
Training begins with 20,000 actions from the conservative rectangle expert and four balanced behavior-cloning epochs. This prevents PPO from discovering the safe-but-useless strategy of circling inside its spawn forever. The expert bootstrap can be changed or disabled:
bash
1# Faster development probe2python3 -m tileman_rl.train --timesteps 10000 --expert-steps 5000 --bc-epochs 334# PPO from scratch, mainly for comparison5python3 -m tileman_rl.train --timesteps 100000 --expert-steps 0
Once python3 -c 'import torch; print(torch.cuda.is_available())' returns True, use the GPU:
Evaluation reports area, captures, survival, peak trail length, reward, and death causes. TensorBoard is optional; install it separately if training charts are wanted.
Multiplayer fine-tuning
The multiplayer environment puts one learning agent and six opponents in a shared world. Its seven-channel local observation adds enemy territory, trails, and heads to the original four channels. Existing capture models are widened from four to seven channels while preserving all learned weights.
Training uses two stages:
A short combat curriculum exposes vulnerable enemy trails and gives a temporary small kill reward so combat is no longer an extremely rare learning event.
Outcome training removes the kill reward and rewards survival, final rank, and held territory while some opponents actively hunt exposed trails.
Run the complete curriculum from the original capture model:
