This repository contains a PyTorch implementation of UD7 of the paper:
Provable Generalization of Clipped Double Q-Learning for Variance Reduction and Sample Efficiency
Jangwon Kim, Jiseok Jeong, Soohee Han Neurocomputing, Volume 673, 7 April 2026, 132772
UD7 is an off-policy actor–critic algorithm that builds on a TD7-style training pipeline, while replacing the critic target formulation with UBOC.
1) Background: Clipped Double Q-Learning (CDQ)
Clipped double Q-learning is a widely-used bias correction in actor-critic methods (e.g., TD3). It maintains two critics and uses the minimum of the two as the TD target:
Effective overestimation control: taking a minimum is conservative, often preventing exploding Q-values.
Robust baseline behavior: works well across many continuous-control tasks.
Limitations
High variance: when critics are poorly learned early on, the min operator can yield high-variance TD targets, destabilizing TD learning and reducing sample efficiency.
UBOC is motivated by a concrete question:
Can we obtain the same expected target value as CDQ, but with smaller variance?
UBOC views the critic outputs as a distribution of Q estimates (because function approximation is noisy).
Instead of using min(Q1, Q2), UBOC uses N critics to estimate:
a mean (m),
an (unbiased) standard deviation \(\hat{s}\),
and then forms a corrected value:
$$
Q_{\text{corrected}} = m - x\cdot \hat s
$$
where (x>0) controls conservativeness.
2.1 Expectation equivalence to clipped double-Q
Under the assumption that critic estimates behave like i.i.d. samples from a normal distribution, we can derive:
