Official pretrained model weights for DR.Q, presented at the Forty-third International Conference on Machine Learning (ICML 2026).
Authors: Jiafei Lyu, Zichuan Lin, Scott Fujimoto, Kai Yang, Yangkun Chen, Saiyong Yang, Zongqing Lu, Deheng Ye
🔍 Overview
The framework of DR.Q is shown below:
DR.Q Framework
Model Description
DR.Q is a model-free reinforcement learning algorithm that achieves strong sample efficiency in continuous control by learning debiased model-based representations. The key insight is that naively training model-based representations can introduce representation bias that hurts downstream policy learning. DR.Q debiases two sources of biases:
Representation bias — mitigated by adding InfoNCE loss besides the MSE loss
Sampling bias — mitigated by introducing the faded prioritized experience replay (Faded PER)
DR.Q builds upon and substantially extends the MR.Q codebase (Facebook Research).
1# Gym / MuJoCo (1M steps)2python main.py --env Gym-HalfCheetah-v4
3python main.py --env Gym-Humanoid-v4
45# DeepMind Control Suite — proprioceptive (500K steps)6python main.py --env Dmc-cheetah-run
7python main.py --env Dmc-quadruped-walk
89# DeepMind Control Suite — pixel observations10python main.py --env Dmc-visual-dog-run
11python main.py --env Dmc-visual-walker-walk
1213# HumanoidBench (requires separate installation)14python main.py --env HBench-h1-run-v0
Loading Pretrained Weights
Pretrained model weights for all reported tasks are hosted here on HuggingFace.
Training Details
Evaluated Benchmark Suites
Suite
Obs. Type
Steps
Tasks
Gym MuJoCo (Gymnasium)
Proprioceptive
1M
5 tasks
DeepMind Control (DMC) — Easy
Proprioceptive
500K
21 tasks
DeepMind Control (DMC) — Hard
Proprioceptive
500K
7 tasks
DeepMind Control (DMC) — Visual
Pixel (84×84)
500K
12 tasks
HumanoidBench (w/o hands)
Proprioceptive
500K
14 tasks
HumanoidBench (w/ hands)
Proprioceptive
500K
14 tasks
Training Infrastructure
Framework: PyTorch ≥ 2.3.0
Python: 3.11 (compatible with 3.9–3.12)
Hardware: CUDA GPU (CPU also supported)
Seeds: Results averaged over 10 random seeds with 95% bootstrap confidence intervals
Evaluation Results
All results report the final average return at the end of training. Aggregate metrics (IQM, Median, Mean) are computed over the task-specific normalized score. Values in [brackets] denote 95% bootstrap confidence intervals.
Gym MuJoCo Tasks (1M environment steps)
Full comparison against domain-specific and general model-free / model-based RL algorithms. Aggregate metrics are computed over the TD3-normalized score.
Pixel-based observations at 84×84 resolution. Aggregate metrics computed over the success normalized score.
Task
DrQ-v2
PPO
TDMPC2
DreamerV3
MR.Q
DR.Q
acrobot-swingup
168 [127, 219]
2 [1, 4]
197 [179, 217]
121 [106, 145]
287 [254, 316]
324 [283, 365]
dog-run
10 [9, 12]
11 [9, 14]
14 [10, 18]
9 [6, 14]
60 [44, 80]
118 [104, 132]
dog-stand
43 [37, 49]
51 [48, 56]
117 [72, 148]
61 [30, 92]
216 [201, 232]
700 [660, 740]
dog-trot
14 [11, 18]
13 [12, 15]
20 [14, 25]
14 [13, 16]
65 [55, 79]
113 [98, 128]
dog-walk
22 [18, 29]
16 [14, 18]
22 [17, 28]
11 [11, 12]
77 [71, 83]
201 [146, 256]
hopper-hop
224 [170, 278]
0 [0, 0]
187 [119, 238]
205 [125, 287]
270 [230, 315]
330 [283, 377]
hopper-stand
917 [903, 931]
1 [0, 2]
582 [321, 794]
888 [875, 900]
852 [703, 930]
937 [930, 944]
humanoid-run
1 [1, 1]
1 [1, 1]
0 [1, 1]
1 [1, 1]
1 [1, 2]
1 [1, 1]
quadruped-run
459 [412, 507]
118 [98, 139]
262 [184, 330]
328 [255, 397]
498 [476, 522]
655 [573, 737]
quadruped-walk
750 [699, 796]
149 [113, 184]
246 [179, 310]
316 [260, 379]
833 [797, 867]
927 [914, 941]
reacher-hard
705 [580, 831]
10 [0, 30]
911 [867, 946]
338 [227, 461]
965 [945, 977]
954 [930, 979]
walker-run
546 [475, 612]
39 [35, 44]
665 [566, 719]
669 [615, 708]
615 [571, 655]
746 [713, 778]
IQM
0.241 [0.214, 0.271]
0.016 [0.013, 0.018]
0.154 [0.113, 0.224]
0.168 [0.152, 0.184]
0.322 [0.239, 0.423]
0.494 [0.395, 0.604]
Median
0.191 [0.172, 0.211]
0.013 [0.012, 0.013]
0.295 [0.198, 0.339]
0.134 [0.124, 0.198]
0.398 [0.320, 0.466]
0.500 [0.427, 0.576]
Mean
0.321 [0.303, 0.340]
0.034 [0.031, 0.037]
0.269 [0.214, 0.326]
0.247 [0.231, 0.262]
0.395 [0.335, 0.457]
0.501 [0.439, 0.564]
HumanoidBench — Without Dexterous Hands (500K steps / 1M env steps with action repeat 2)
Aggregate metrics computed over the success normalized score.
Task
Simba
SimbaV2
MR.Q
FoG
DR.Q
h1-pole-v0
716 [667, 765]
791 [785, 797]
578 [534, 623]
893 [846, 940]
887 [853, 921]
h1-slide-v0
277 [252, 303]
487 [404, 571]
303 [270, 337]
674 [562, 785]
355 [324, 386]
h1-stair-v0
269 [153, 385]
493 [467, 518]
235 [213, 257]
466 [383, 548]
401 [328, 475]
h1-balance-hard-v0
75 [71, 80]
143 [128, 157]
69 [67, 72]
81 [71, 91]
92 [87, 97]
h1-balance-simple-v0
337 [193, 482]
723 [651, 795]
135 [110, 160]
616 [536, 696]
205 [166, 244]
h1-sit-hard-v0
512 [354, 670]
679 [548, 811]
553 [421, 686]
770 [738, 802]
843 [747, 939]
h1-sit-simple-v0
833 [814, 853]
875 [870, 880]
850 [819, 882]
828 [800, 856]
931 [924, 938]
h1-maze-v0
354 [342, 366]
313 [287, 340]
344 [340, 347]
331 [310, 353]
354 [349, 359]
h1-crawl-v0
923 [904, 942]
946 [933, 959]
932 [919, 945]
971 [969, 973]
973 [972, 974]
h1-hurdle-v0
175 [150, 201]
202 [167, 236]
131 [108, 155]
114 [100, 129]
344 [245, 443]
h1-reach-v0
3874 [3220, 4527]
3850 [3272, 4427]
4902 [4390, 5414]
2434 [2083, 2785]
8101 [7640, 8563]
h1-run-v0
232 [185, 279]
415 [307, 524]
278 [192, 364]
749 [666, 832]
820 [815, 824]
h1-stand-v0
772 [701, 843]
814 [770, 857]
800 [754, 846]
671 [516, 825]
856 [815, 897]
h1-walk-v0
550 [391, 709]
845 [840, 850]
716 [657, 775]
866 [859, 872]
850 [830, 869]
IQM
0.521 [0.413, 0.633]
0.799 [0.686, 0.908]
0.519 [0.417, 0.630]
0.846 [0.713, 0.969]
0.864 [0.735, 0.976]
Median
0.598 [0.514, 0.692]
0.781 [0.693, 0.865]
0.602 [0.516, 0.687]
0.794 [0.705, 0.899]
0.823 [0.733, 0.920]
Mean
0.606 [0.536, 0.678]
0.776 [0.705, 0.849]
0.604 [0.531, 0.677]
0.802 [0.721, 0.883]
0.825 [0.748, 0.902]
HumanoidBench — With Dexterous Hands (500K steps / 1M env steps with action repeat 2)
Aggregate metrics computed over the success normalized score.
Task
DreamerV3
TDMPC2
SimBa
SimbaV2
MR.Q
FoG
DR.Q
h1hand-door-v0
10 [7, 13]
134 [23, 246]
206 [169, 244]
310 [302, 318]
293 [280, 305]
244 [227, 261]
320 [308, 333]
h1hand-slide-v0
21 [19, 23]
79 [68, 90]
67 [55, 79]
136 [97, 175]
146 [131, 161]
201 [173, 228]
285 [258, 312]
h1hand-stair-v0
16 [8, 25]
43 [35, 51]
61 [44, 78]
120 [89, 151]
127 [104, 150]
135 [126, 144]
288 [193, 382]
h1hand-bookshelf-simple-v0
45 [41, 50]
97 [59, 134]
487 [315, 660]
838 [834, 843]
691 [599, 783]
610 [523, 697]
709 [572, 846]
h1hand-bookshelf-hard-v0
27 [24, 30]
34 [19, 50]
490 [447, 533]
496 [417, 575]
332 [240, 425]
577 [548, 605]
349 [262, 435]
h1hand-sit-simple-v0
48 [42, 54]
607 [268, 947]
643 [580, 705]
927 [904, 951]
653 [568, 737]
631 [528, 735]
942 [926, 958]
h1hand-sit-hard-v0
15 [11, 20]
139 [86, 193]
649 [500, 797]
724 [609, 838]
487 [353, 621]
179 [128, 229]
891 [841, 941]
h1hand-basketball-v0
13 [12, 13]
47 [21, 73]
54 [25, 83]
56 [34, 78]
53 [34, 72]
182 [131, 232]
75 [45, 105]
h1hand-pole-v0
48 [36, 60]
99 [87, 111]
224 [195, 254]
493 [426, 559]
237 [202, 273]
257 [237, 277]
424 [299, 549]
h1hand-crawl-v0
256 [244, 268]
897 [858, 935]
779 [748, 809]
640 [549, 732]
807 [783, 831]
794 [721, 866]
526 [477, 574]
h1hand-reach-v0
864 [578, 1150]
3610 [2912, 4309]
3185 [2664, 3707]
3223 [2703, 3744]
4101 [3540, 4662]
2877 [2487, 3267]
4950 [4280, 5619]
h1hand-run-v0
6 [4, 8]
29 [27, 30]
31 [24, 37]
30 [22, 38]
35 [29, 41]
22 [19, 25]
129 [77, 181]
h1hand-stand-v0
41 [38, 44]
193 [147, 238]
127 [72, 181]
103 [81, 126]
300 [194, 405]
79 [66, 91]
491 [344, 638]
h1hand-walk-v0
19 [12, 27]
234 [125, 343]
94 [79, 109]
64 [52, 76]
95 [77, 112]
75 [63, 87]
512 [371, 652]
IQM
0.019 [0.013, 0.026]
0.150 [0.091, 0.224]
0.219 [0.179, 0.267]
0.298 [0.241, 0.374]
0.286 [0.245, 0.333]
0.254 [0.222, 0.285]
0.452 [0.400, 0.512]
Median
0.021 [0.010, 0.030]
0.298 [0.147, 0.433]
0.356 [0.269, 0.413]
0.420 [0.338, 0.491]
0.388 [0.313, 0.449]
0.342 [0.268, 0.395]
0.529 [0.455, 0.607]
Mean
0.020 [0.011, 0.028]
0.282 [0.169, 0.413]
0.345 [0.286, 0.406]
0.417 [0.356, 0.482]
0.385 [0.329, 0.443]
0.336 [0.285, 0.393]
0.534 [0.473, 0.595]
Citation
bibtex
1@inproceedings{lyu2026debiased,
2 title={Debiased Model-based Representations for Sample-efficient Continuous Control},
3 author={Jiafei Lyu and Zichuan Lin and Scott Fujimoto and Kai Yang and Yangkun Chen and Saiyong Yang and Zongqing Lu and Deheng Ye},
4 booktitle={Forty-third International Conference on Machine Learning},
5 year={2026},
6 url={https://openreview.net/forum?id=ZP1p8k106p}
7}
Acknowledgements
DR.Q builds upon the MR.Q codebase by Facebook Research. We thank the authors of TD7, TDMPC2, MR.Q, FoG, SimBa, SimbaV2, DrQ-v2, DreamerV3, and PPO for their open-source implementations used as baselines.
