Welcome to the Tau LLM Unity ML Agents Project repository! This project focuses on training reinforcement learning agents using Unity ML-Agents and the PPO algorithm. Our goal is to optimize the performance of the agents through various configurations and training runs.
Project Overview
This repository contains the code and configurations for training agents in a Unity environment using the Proximal Policy Optimization (PPO) algorithm. The agents are designed to learn and adapt to their environment, improving their performance over time.
Key Features
Reinforcement Learning: Utilizes the PPO algorithm for training agents.
Unity ML-Agents: Integrates with Unity ML-Agents for a seamless training experience.
Custom Reward Functions: Implements gradient-based reward functions for nuanced feedback.
Memory Networks: Incorporates memory networks to handle temporal dependencies.
TensorBoard Integration: Monitors training progress and performance using TensorBoard.
Configuration
Below is the configuration used for training the agents:
yaml
1behaviors:2TauAgent:3trainer_type: ppo
4hyperparameters:5batch_size:2566buffer_size:40967learning_rate:0.000038beta:0.0059epsilon:0.210lambd:0.9511num_epoch:1012learning_rate_schedule: linear
13network_settings:14normalize:true15hidden_units:25616num_layers:417vis_encode_type: simple
18memory:19memory_size:25620sequence_length:25621num_layers:422reward_signals:23extrinsic:24gamma:0.9925strength:1.026curiosity:27gamma:0.99528strength:0.129network_settings:30normalize:true31hidden_units:25632num_layers:433learning_rate:0.0000334keep_checkpoints:1035checkpoint_interval:10000036threaded:true37max_steps:300000038time_horizon:25639summary_freq:10000
Model Naming Convention
The models in this repository follow the naming convention Tau_<series>_<max_steps>. This helps in easily identifying the series and the number of training steps for each model.
To start training the agent, run the following command:
mlagents-learn .\config\tau_agent_ppo_c.yaml --run-id=tau_agent_ppo_A0 --env .\Build --torch-device cuda --timeout-wait 300 --force
Note: The preferred way to run a build is by creating a new build into the Build directory which is referenced by the above command.
Monitoring Training
You can monitor the training progress using TensorBoard:
tensorboard --logdir results
Results
The training results, including the average reward and cumulative reward, can be visualized using TensorBoard. The graphs below show the performance of the agent over time:
Average Reward
Average Reward
Average Reward
Citation
If you use this project in your research, please cite it as follows:
