Market features are normalized using a StandardScaler fit only on the training-period data. The history is a rolling window of the last 30 observations (chronologically ordered).
Reward Function
Risk-sensitive reward with multiple penalty terms:
PnL reward: Bounded log-equity return with a 1% starting-cash floor, so rare binary-market jackpot payouts do not dominate the critic target
Drawdown penalty: Penalizes increasing drawdown (soft: linear + quadratic)
CVaR penalty: Amplifies losses during drawdown periods (tail risk) — subtracted from reward to penalize negative PnL more severely when already underwater
Fee Model
All trading operations (buy, liquidate, reduce) charge a taker fee based on the Crypto/BTC prediction market schedule:
Quadratic fee shape: Highest at price ≈ 0.50 (max uncertainty), zero at price = 0 or 1. This matches how prediction market fees scale with outcome uncertainty.
The fee is applied consistently in both:
BTCTradingEnv — live policy replay, deducts from cash on buys (added to cost) and from proceeds on sells/liquidation
dataset.py counterfactual simulator — offline training rewards include fee drag so the learned Q-values and policy account for trading costs
To disable fees (e.g. ablation): --taker-fee-rate 0.0 or TAKER_FEE_RATE = 0.0 in constants.py.
IQL Algorithm (v2 — Fixed)
What was wrong in v1
Component
v1 (Buggy)
v2 (Fixed)
Q-network
(s, one_hot(a)) → scalar
s → [Q(s,a₀), ..., Q(s,a₇)]
V-update
Q_target(next_s, one_hot(a))
min(Q₁_target(s), Q₂_target(s)) gathered at dataset actions
Evaluation
return np.mean(rewards) (no policy)
Action agreement + entropy diagnostics
Dataset
Greedy best-action (98.8% HOLD) / random softmax churn
Conservative edge-gated behavioral policy with epsilon exploration
Moderate upper expectile — balances optimism with robustness
temperature
3.0
3.0
Low β → policy stays closer to behavioral (safer for financial data)
gamma
0.99
0.99
Standard discount for episodic tasks
tau
0.005
0.005
Slow target network update for stability
lr
3e-4
3e-4
Standard Adam LR
batch_size
512
256
Large batches for stable gradients (cloud); smaller for local memory
hidden_dim
256
256
2-layer MLP
dropout
0.1
0.0
Light regularization (cloud only)
policy_update_freq
2
2
TD3-style delayed policy updates
early_stopping_patience
20 evals
20 evals
Stops if eval reward doesn't improve
behavioral_policy_mode
conservative
conservative
Avoids fee-heavy uniform random trading
min_trade_edge
0.005
0.005
Directional action must beat HOLD/FLAT by this reward edge
behavioral_epsilon
0.03
0.03
Keeps limited action support for IQL without dominating the dataset
Dataset Building
Process
Load parquet → filter to obs_pos ∈ {0,1,2,3,4}, sort by time
Fill NaNs → defaults per column (0.0 for most, 1.0 for long_short_ratio)
Temporal split → last 20% of calendar days held out for test, with a purge/embargo gap of episode_span_days on each side to prevent overlapping windows from leaking information
Fit StandardScaler → normalize market features using train-period data only
Build episodes → sliding windows (span=30d, stride=15d) within each split
Counterfactual simulation → for each step, simulate ALL 8 actions and compute rewards
Settlement guard → obs_pos == 4 can settle existing inventory but cannot open new exposure, preventing same-row outcome leakage
Behavioral policy → pick the best of HOLD/FLAT unless the best directional trade clears min_trade_edge; use small epsilon exploration for action support
Shard metadata → stores sampled action distribution and, by default, all_action_rewards for counterfactual supervised training
Data Leakage Prevention
The dataset builder enforces a clean temporal separation:
[--- train days ---][<-- embargo -->][<-- embargo -->][--- test days ---]
scaler fit here gap gap held out
The StandardScaler is fit only on training-period rows.
An embargo buffer (default: one full episode_span_days, i.e. 30 days) is removed from both sides of the split boundary so that no 30-day sliding window can straddle the train/test line.
Episodes are built independently within each split's day list.
Scale (full config)
~750 episodes from 2263 days of 5m data (before embargo removal)
State dim: 1149 (30 × 38 + 9)
Action distribution: recorded in shard metadata; reject runs where fee-paying actions are near-uniform without a clear edge
Evaluation
What the trainer provides
The IQL trainer computes diagnostics at evaluation checkpoints:
Metric
What it measures
What it does NOT measure
Agreement with optimal Q-action
How often the policy picks the action with the highest Q-value
Actual trading PnL or Sharpe
Action entropy
Diversity of the policy's action distribution
Out-of-sample performance
These are diagnostics of training convergence, not trading performance metrics. Agreement and entropy tell you whether the policy has learned to distinguish actions, but they do not substitute for a true policy replay on held-out market data.
Policy replay
train_local_sharded.py and train_counterfactual_local.py both run held-out replay through BTCTradingEnv when --replay-episodes > 0. The replay metrics (mean_pnl, fees, drawdown, action counts) are the primary acceptance criteria.
Counterfactual Q path
Because the simulator computes rewards for every action at each state, train_counterfactual_local.py can train a direct action-value model on all_action_rewards. This avoids throwing away supervision by sampling one behavioral action per row. Deployment remains edge-gated: choose HOLD/FLAT unless the best directional action beats the no-new-risk baseline by min_trade_edge.
Current State
What works ✅
IQL trainer with correct discrete-action architecture
Behavioral dataset with diverse actions
Counterfactual Q trainer using all-action reward labels
Leakage-free temporal split with embargo gap
Scaler fit on training data only
LR scheduling, early stopping, best model checkpointing
Cloud training scripts with Trackio monitoring
HF Hub upload/download
Known Challenges ⚠️
Sparse positive rewards — most 5m windows don't have strong signals. The model needs many epochs to learn meaningful patterns.
High-dimensional state (1149 dims) — the 30-step history window creates a large input.
Counterfactual simulation is slow — dataset building takes time for full config (30-day episodes).
Sparse or no deployment trades — after removing settlement-row leakage, held-out replay may correctly select no-trade if directional actions do not clear fees and drawdown risk.
How to Run
Local training (small config, CPU/MPS)
bash
1cd /path/to/doug-data
2python -m rl_btc_v4.train
Default local config: batch_size=256, dropout=0.0, epochs=100.
Cloud training (GPU, full config)
python rl_btc_v4/train_cloud_v2.py
Cloud config uses batch_size=512, dropout=0.1.
Via HF Jobs
python
1from huggingface_hub import HfApi
23# Upload code to HF Hub first4api = HfApi()5api.upload_folder(6 folder_path="rl_btc_v4",7 repo_id="fbzu/rl_btc_v4_iql",8 repo_type="model",9)1011# Then run train_hf_job.py via hf_jobs with GPU hardware
Artifacts
After training, the following are saved to https://huggingface.co/fbzu/rl_btc_v4_iql:
