Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BugFix,Feature] Allow non-tensor data in envs #1944

Open
wants to merge 10 commits into
base: main
Choose a base branch
from
Open

[BugFix,Feature] Allow non-tensor data in envs #1944

wants to merge 10 commits into from

Conversation

vmoens
Copy link
Contributor

@vmoens vmoens commented Feb 21, 2024 •
edited

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Feb 21, 2024 •
edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/1944

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 2 Unrelated Failures

As of commit b443095 with merge base f613eef (image):

NEW FAILURES - The following jobs have failed:

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Feb 21, 2024
@vmoens vmoens linked an issue Feb 21, 2024 that may be closed by this pull request
[BUG] A dataclass in "next" doesn't get copied over in step_mdp #1919
Open
3 tasks
@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 21, 2024 •
edited

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 91. Improved: $\large\color{#35bf28}20$. Worsened: $\large\color{#d91a1a}6$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_single 0.1334s 63.1755ms 15.8289 Ops/s 17.7545 Ops/s $\textbf{\color{#d91a1a}-10.85\%}$
test_sync 49.5499ms 37.1977ms 26.8834 Ops/s 31.8370 Ops/s $\textbf{\color{#d91a1a}-15.56\%}$
test_async 58.2356ms 29.7459ms 33.6181 Ops/s 33.0058 Ops/s $\color{#35bf28}+1.86\%$
test_simple 0.4929s 0.4160s 2.4037 Ops/s 2.5855 Ops/s $\textbf{\color{#d91a1a}-7.03\%}$
test_transformed 0.5606s 0.5577s 1.7931 Ops/s 1.8423 Ops/s $\color{#d91a1a}-2.67\%$
test_serial 1.4272s 1.3270s 0.7536 Ops/s 0.7569 Ops/s $\color{#d91a1a}-0.45\%$
test_parallel 1.1735s 1.1060s 0.9042 Ops/s 0.8944 Ops/s $\color{#35bf28}+1.09\%$
test_step_mdp_speed[True-True-True-True-True] 0.2018ms 22.4798μs 44.4845 KOps/s 46.0194 KOps/s $\color{#d91a1a}-3.34\%$
test_step_mdp_speed[True-True-True-True-False] 41.5180μs 13.3996μs 74.6288 KOps/s 76.1458 KOps/s $\color{#d91a1a}-1.99\%$
test_step_mdp_speed[True-True-True-False-True] 65.5830μs 13.2901μs 75.2438 KOps/s 78.5904 KOps/s $\color{#d91a1a}-4.26\%$
test_step_mdp_speed[True-True-True-False-False] 42.0190μs 7.7639μs 128.8005 KOps/s 128.8941 KOps/s $\color{#d91a1a}-0.07\%$
test_step_mdp_speed[True-True-False-True-True] 75.7420μs 23.9106μs 41.8224 KOps/s 43.6054 KOps/s $\color{#d91a1a}-4.09\%$
test_step_mdp_speed[True-True-False-True-False] 45.9460μs 14.8195μs 67.4784 KOps/s 69.0566 KOps/s $\color{#d91a1a}-2.29\%$
test_step_mdp_speed[True-True-False-False-True] 44.5530μs 14.3590μs 69.6428 KOps/s 71.7151 KOps/s $\color{#d91a1a}-2.89\%$
test_step_mdp_speed[True-True-False-False-False] 45.6960μs 9.0376μs 110.6488 KOps/s 113.2767 KOps/s $\color{#d91a1a}-2.32\%$
test_step_mdp_speed[True-False-True-True-True] 62.3170μs 25.2126μs 39.6627 KOps/s 41.2441 KOps/s $\color{#d91a1a}-3.83\%$
test_step_mdp_speed[True-False-True-True-False] 47.3780μs 15.9323μs 62.7658 KOps/s 63.2460 KOps/s $\color{#d91a1a}-0.76\%$
test_step_mdp_speed[True-False-True-False-True] 47.4390μs 14.4162μs 69.3665 KOps/s 71.2229 KOps/s $\color{#d91a1a}-2.61\%$
test_step_mdp_speed[True-False-True-False-False] 70.8730μs 9.1237μs 109.6045 KOps/s 110.6491 KOps/s $\color{#d91a1a}-0.94\%$
test_step_mdp_speed[True-False-False-True-True] 59.6020μs 26.2590μs 38.0821 KOps/s 39.4742 KOps/s $\color{#d91a1a}-3.53\%$
test_step_mdp_speed[True-False-False-True-False] 82.0240μs 17.3316μs 57.6981 KOps/s 59.8374 KOps/s $\color{#d91a1a}-3.58\%$
test_step_mdp_speed[True-False-False-False-True] 78.9300μs 15.3954μs 64.9544 KOps/s 66.2437 KOps/s $\color{#d91a1a}-1.95\%$
test_step_mdp_speed[True-False-False-False-False] 58.9310μs 10.1807μs 98.2252 KOps/s 98.4850 KOps/s $\color{#d91a1a}-0.26\%$
test_step_mdp_speed[False-True-True-True-True] 64.2610μs 24.8845μs 40.1857 KOps/s 41.1072 KOps/s $\color{#d91a1a}-2.24\%$
test_step_mdp_speed[False-True-True-True-False] 50.5950μs 16.0183μs 62.4288 KOps/s 63.2301 KOps/s $\color{#d91a1a}-1.27\%$
test_step_mdp_speed[False-True-True-False-True] 49.4420μs 16.6115μs 60.1991 KOps/s 61.1276 KOps/s $\color{#d91a1a}-1.52\%$
test_step_mdp_speed[False-True-True-False-False] 44.2340μs 10.2432μs 97.6255 KOps/s 98.3575 KOps/s $\color{#d91a1a}-0.74\%$
test_step_mdp_speed[False-True-False-True-True] 81.0420μs 26.0295μs 38.4179 KOps/s 39.2568 KOps/s $\color{#d91a1a}-2.14\%$
test_step_mdp_speed[False-True-False-True-False] 57.6280μs 17.1208μs 58.4086 KOps/s 52.7504 KOps/s $\textbf{\color{#35bf28}+10.73\%}$
test_step_mdp_speed[False-True-False-False-True] 77.5560μs 17.5911μs 56.8470 KOps/s 58.3816 KOps/s $\color{#d91a1a}-2.63\%$
test_step_mdp_speed[False-True-False-False-False] 55.9050μs 11.4476μs 87.3543 KOps/s 88.8669 KOps/s $\color{#d91a1a}-1.70\%$
test_step_mdp_speed[False-False-True-True-True] 0.3010ms 29.4976μs 33.9010 KOps/s 37.6421 KOps/s $\textbf{\color{#d91a1a}-9.94\%}$
test_step_mdp_speed[False-False-True-True-False] 98.6250μs 18.4243μs 54.2762 KOps/s 55.5057 KOps/s $\color{#d91a1a}-2.22\%$
test_step_mdp_speed[False-False-True-False-True] 58.7800μs 17.6866μs 56.5400 KOps/s 58.0094 KOps/s $\color{#d91a1a}-2.53\%$
test_step_mdp_speed[False-False-True-False-False] 58.4000μs 11.4321μs 87.4729 KOps/s 89.1259 KOps/s $\color{#d91a1a}-1.85\%$
test_step_mdp_speed[False-False-False-True-True] 49.4120μs 29.1872μs 34.2616 KOps/s 35.5962 KOps/s $\color{#d91a1a}-3.75\%$
test_step_mdp_speed[False-False-False-True-False] 66.0040μs 19.3425μs 51.6995 KOps/s 52.3628 KOps/s $\color{#d91a1a}-1.27\%$
test_step_mdp_speed[False-False-False-False-True] 60.4930μs 18.5131μs 54.0159 KOps/s 54.9820 KOps/s $\color{#d91a1a}-1.76\%$
test_step_mdp_speed[False-False-False-False-False] 59.0810μs 12.4754μs 80.1576 KOps/s 80.0078 KOps/s $\color{#35bf28}+0.19\%$
test_values[generalized_advantage_estimate-True-True] 12.4405ms 9.8427ms 101.5981 Ops/s 101.8287 Ops/s $\color{#d91a1a}-0.23\%$
test_values[vec_generalized_advantage_estimate-True-True] 39.2310ms 36.2926ms 27.5538 Ops/s 27.7783 Ops/s $\color{#d91a1a}-0.81\%$
test_values[td0_return_estimate-False-False] 0.2614ms 0.1904ms 5.2511 KOps/s 5.2531 KOps/s $\color{#d91a1a}-0.04\%$
test_values[td1_return_estimate-False-False] 25.9408ms 24.2401ms 41.2540 Ops/s 40.2563 Ops/s $\color{#35bf28}+2.48\%$
test_values[vec_td1_return_estimate-False-False] 37.2172ms 35.9726ms 27.7989 Ops/s 27.7553 Ops/s $\color{#35bf28}+0.16\%$
test_values[td_lambda_return_estimate-True-False] 37.5387ms 34.5267ms 28.9631 Ops/s 27.9799 Ops/s $\color{#35bf28}+3.51\%$
test_values[vec_td_lambda_return_estimate-True-False] 40.8590ms 36.5378ms 27.3689 Ops/s 27.6643 Ops/s $\color{#d91a1a}-1.07\%$
test_gae_speed[generalized_advantage_estimate-False-1-512] 11.7440ms 8.3620ms 119.5880 Ops/s 119.8381 Ops/s $\color{#d91a1a}-0.21\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 2.7415ms 2.0446ms 489.0853 Ops/s 482.1286 Ops/s $\color{#35bf28}+1.44\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.6705ms 0.3731ms 2.6804 KOps/s 2.7407 KOps/s $\color{#d91a1a}-2.20\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 56.3997ms 47.4837ms 21.0599 Ops/s 22.1132 Ops/s $\color{#d91a1a}-4.76\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 3.4824ms 3.1063ms 321.9261 Ops/s 307.7569 Ops/s $\color{#35bf28}+4.60\%$
test_dqn_speed 1.7723ms 1.3845ms 722.2894 Ops/s 717.5754 Ops/s $\color{#35bf28}+0.66\%$
test_ddpg_speed 3.6719ms 2.9349ms 340.7297 Ops/s 336.2748 Ops/s $\color{#35bf28}+1.32\%$
test_sac_speed 15.1839ms 9.3939ms 106.4519 Ops/s 108.6762 Ops/s $\color{#d91a1a}-2.05\%$
test_redq_speed 22.6582ms 14.7151ms 67.9575 Ops/s 68.7108 Ops/s $\color{#d91a1a}-1.10\%$
test_redq_deprec_speed 23.5015ms 15.3648ms 65.0840 Ops/s 62.0661 Ops/s $\color{#35bf28}+4.86\%$
test_td3_speed 17.9223ms 8.7135ms 114.7644 Ops/s 105.4474 Ops/s $\textbf{\color{#35bf28}+8.84\%}$
test_cql_speed 49.1468ms 37.7006ms 26.5248 Ops/s 25.2094 Ops/s $\textbf{\color{#35bf28}+5.22\%}$
test_a2c_speed 9.4231ms 7.8845ms 126.8304 Ops/s 117.9729 Ops/s $\textbf{\color{#35bf28}+7.51\%}$
test_ppo_speed 9.0976ms 8.0864ms 123.6646 Ops/s 116.0875 Ops/s $\textbf{\color{#35bf28}+6.53\%}$
test_reinforce_speed 8.1029ms 6.8995ms 144.9371 Ops/s 140.3489 Ops/s $\color{#35bf28}+3.27\%$
test_iql_speed 35.3228ms 34.1921ms 29.2465 Ops/s 27.7846 Ops/s $\textbf{\color{#35bf28}+5.26\%}$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 4.3493ms 3.8690ms 258.4627 Ops/s 245.6061 Ops/s $\textbf{\color{#35bf28}+5.23\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 1.1040ms 0.5053ms 1.9792 KOps/s 1.6387 KOps/s $\textbf{\color{#35bf28}+20.78\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.7121ms 0.4848ms 2.0625 KOps/s 1.9748 KOps/s $\color{#35bf28}+4.44\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 5.6544ms 3.7032ms 270.0350 Ops/s 228.9506 Ops/s $\textbf{\color{#35bf28}+17.94\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.7413ms 0.5053ms 1.9792 KOps/s 1.9276 KOps/s $\color{#35bf28}+2.68\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.7810ms 0.4818ms 2.0754 KOps/s 2.0272 KOps/s $\color{#35bf28}+2.38\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] 2.1142ms 1.7146ms 583.2404 Ops/s 570.3827 Ops/s $\color{#35bf28}+2.25\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] 4.5029ms 1.6285ms 614.0599 Ops/s 606.5216 Ops/s $\color{#35bf28}+1.24\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 5.4741ms 3.7778ms 264.7022 Ops/s 236.3151 Ops/s $\textbf{\color{#35bf28}+12.01\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1.1691ms 0.6357ms 1.5731 KOps/s 1.5634 KOps/s $\color{#35bf28}+0.62\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.1129s 0.6857ms 1.4583 KOps/s 1.6424 KOps/s $\textbf{\color{#d91a1a}-11.21\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 3.9892ms 3.7323ms 267.9294 Ops/s 247.3566 Ops/s $\textbf{\color{#35bf28}+8.32\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.6303ms 0.5025ms 1.9902 KOps/s 1.9000 KOps/s $\color{#35bf28}+4.74\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.7792ms 0.4889ms 2.0455 KOps/s 1.9686 KOps/s $\color{#35bf28}+3.91\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 4.0697ms 3.7587ms 266.0526 Ops/s 251.6036 Ops/s $\textbf{\color{#35bf28}+5.74\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.1130s 0.6597ms 1.5159 KOps/s 1.9015 KOps/s $\textbf{\color{#d91a1a}-20.28\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.6153ms 0.4795ms 2.0854 KOps/s 1.9758 KOps/s $\textbf{\color{#35bf28}+5.55\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 5.6243ms 3.8646ms 258.7572 Ops/s 217.0177 Ops/s $\textbf{\color{#35bf28}+19.23\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 0.8825ms 0.6296ms 1.5882 KOps/s 1.5412 KOps/s $\color{#35bf28}+3.05\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.8115ms 0.5994ms 1.6683 KOps/s 1.5795 KOps/s $\textbf{\color{#35bf28}+5.62\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 0.1149s 8.0721ms 123.8838 Ops/s 109.9464 Ops/s $\textbf{\color{#35bf28}+12.68\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 15.9182ms 12.7253ms 78.5833 Ops/s 73.6257 Ops/s $\textbf{\color{#35bf28}+6.73\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 1.7326ms 1.0715ms 933.3137 Ops/s 892.6612 Ops/s $\color{#35bf28}+4.55\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 0.1120s 5.9695ms 167.5172 Ops/s 148.2725 Ops/s $\textbf{\color{#35bf28}+12.98\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 15.0503ms 12.5623ms 79.6035 Ops/s 73.0144 Ops/s $\textbf{\color{#35bf28}+9.02\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 1.6226ms 1.0591ms 944.2286 Ops/s 916.8637 Ops/s $\color{#35bf28}+2.98\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 0.1125s 8.1748ms 122.3268 Ops/s 110.3679 Ops/s $\textbf{\color{#35bf28}+10.84\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 17.2953ms 13.0109ms 76.8586 Ops/s 73.6808 Ops/s $\color{#35bf28}+4.31\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 2.0179ms 1.2272ms 814.8706 Ops/s 801.7305 Ops/s $\color{#35bf28}+1.64\%$

@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 21, 2024 •
edited

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of GPU Benchmark Tests

Total Benchmarks: 94. Improved: $\large\color{#35bf28}6$. Worsened: $\large\color{#d91a1a}2$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_single 0.1153s 0.1150s 8.6927 Ops/s 8.4247 Ops/s $\color{#35bf28}+3.18\%$
test_sync 0.1058s 0.1044s 9.5829 Ops/s 9.4364 Ops/s $\color{#35bf28}+1.55\%$
test_async 0.1978s 96.1856ms 10.3966 Ops/s 10.3858 Ops/s $\color{#35bf28}+0.10\%$
test_single_pixels 0.1260s 0.1258s 7.9482 Ops/s 7.7537 Ops/s $\color{#35bf28}+2.51\%$
test_sync_pixels 85.4930ms 83.1413ms 12.0277 Ops/s 12.1946 Ops/s $\color{#d91a1a}-1.37\%$
test_async_pixels 0.1606s 69.9111ms 14.3039 Ops/s 14.8645 Ops/s $\color{#d91a1a}-3.77\%$
test_simple 0.7911s 0.7903s 1.2653 Ops/s 1.1863 Ops/s $\textbf{\color{#35bf28}+6.65\%}$
test_transformed 1.0443s 1.0424s 0.9593 Ops/s 0.9095 Ops/s $\textbf{\color{#35bf28}+5.48\%}$
test_serial 2.4977s 2.4389s 0.4100 Ops/s 0.3976 Ops/s $\color{#35bf28}+3.12\%$
test_parallel 2.3949s 2.3193s 0.4312 Ops/s 0.4219 Ops/s $\color{#35bf28}+2.20\%$
test_step_mdp_speed[True-True-True-True-True] 92.3710μs 34.1357μs 29.2949 KOps/s 29.3287 KOps/s $\color{#d91a1a}-0.12\%$
test_step_mdp_speed[True-True-True-True-False] 47.3700μs 19.8238μs 50.4445 KOps/s 49.7739 KOps/s $\color{#35bf28}+1.35\%$
test_step_mdp_speed[True-True-True-False-True] 89.1330μs 19.5435μs 51.1679 KOps/s 51.6569 KOps/s $\color{#d91a1a}-0.95\%$
test_step_mdp_speed[True-True-True-False-False] 56.4210μs 11.4599μs 87.2609 KOps/s 87.0955 KOps/s $\color{#35bf28}+0.19\%$
test_step_mdp_speed[True-True-False-True-True] 54.0210μs 35.7441μs 27.9766 KOps/s 27.6290 KOps/s $\color{#35bf28}+1.26\%$
test_step_mdp_speed[True-True-False-True-False] 0.2126ms 21.6116μs 46.2715 KOps/s 45.2756 KOps/s $\color{#35bf28}+2.20\%$
test_step_mdp_speed[True-True-False-False-True] 42.4600μs 21.4514μs 46.6170 KOps/s 47.1491 KOps/s $\color{#d91a1a}-1.13\%$
test_step_mdp_speed[True-True-False-False-False] 42.2710μs 13.3491μs 74.9115 KOps/s 74.8333 KOps/s $\color{#35bf28}+0.10\%$
test_step_mdp_speed[True-False-True-True-True] 72.0620μs 38.0047μs 26.3125 KOps/s 26.2429 KOps/s $\color{#35bf28}+0.27\%$
test_step_mdp_speed[True-False-True-True-False] 44.2410μs 23.5727μs 42.4220 KOps/s 41.0245 KOps/s $\color{#35bf28}+3.41\%$
test_step_mdp_speed[True-False-True-False-True] 37.5600μs 21.3933μs 46.7436 KOps/s 47.6141 KOps/s $\color{#d91a1a}-1.83\%$
test_step_mdp_speed[True-False-True-False-False] 32.1710μs 13.3746μs 74.7686 KOps/s 74.8099 KOps/s $\color{#d91a1a}-0.06\%$
test_step_mdp_speed[True-False-False-True-True] 67.3720μs 39.6426μs 25.2254 KOps/s 25.1662 KOps/s $\color{#35bf28}+0.24\%$
test_step_mdp_speed[True-False-False-True-False] 0.2170ms 25.9021μs 38.6069 KOps/s 38.4031 KOps/s $\color{#35bf28}+0.53\%$
test_step_mdp_speed[True-False-False-False-True] 0.1490ms 23.2946μs 42.9284 KOps/s 44.0519 KOps/s $\color{#d91a1a}-2.55\%$
test_step_mdp_speed[True-False-False-False-False] 83.8420μs 15.0637μs 66.3847 KOps/s 65.2287 KOps/s $\color{#35bf28}+1.77\%$
test_step_mdp_speed[False-True-True-True-True] 58.4610μs 37.9855μs 26.3258 KOps/s 26.4257 KOps/s $\color{#d91a1a}-0.38\%$
test_step_mdp_speed[False-True-True-True-False] 52.8110μs 23.5963μs 42.3796 KOps/s 40.8951 KOps/s $\color{#35bf28}+3.63\%$
test_step_mdp_speed[False-True-True-False-True] 72.6810μs 25.8058μs 38.7509 KOps/s 39.7210 KOps/s $\color{#d91a1a}-2.44\%$
test_step_mdp_speed[False-True-True-False-False] 35.0310μs 15.1751μs 65.8976 KOps/s 64.7536 KOps/s $\color{#35bf28}+1.77\%$
test_step_mdp_speed[False-True-False-True-True] 91.1720μs 39.5385μs 25.2918 KOps/s 25.2071 KOps/s $\color{#35bf28}+0.34\%$
test_step_mdp_speed[False-True-False-True-False] 42.2710μs 25.5728μs 39.1040 KOps/s 38.5626 KOps/s $\color{#35bf28}+1.40\%$
test_step_mdp_speed[False-True-False-False-True] 61.7310μs 27.3830μs 36.5191 KOps/s 36.7178 KOps/s $\color{#d91a1a}-0.54\%$
test_step_mdp_speed[False-True-False-False-False] 33.7800μs 16.9850μs 58.8756 KOps/s 57.9339 KOps/s $\color{#35bf28}+1.63\%$
test_step_mdp_speed[False-False-True-True-True] 70.7620μs 41.3210μs 24.2007 KOps/s 23.8341 KOps/s $\color{#35bf28}+1.54\%$
test_step_mdp_speed[False-False-True-True-False] 82.3120μs 27.6663μs 36.1450 KOps/s 35.2330 KOps/s $\color{#35bf28}+2.59\%$
test_step_mdp_speed[False-False-True-False-True] 50.9410μs 27.3462μs 36.5682 KOps/s 37.5435 KOps/s $\color{#d91a1a}-2.60\%$
test_step_mdp_speed[False-False-True-False-False] 38.0710μs 16.8952μs 59.1883 KOps/s 57.9677 KOps/s $\color{#35bf28}+2.11\%$
test_step_mdp_speed[False-False-False-True-True] 59.1000μs 43.2691μs 23.1112 KOps/s 23.1002 KOps/s $\color{#35bf28}+0.05\%$
test_step_mdp_speed[False-False-False-True-False] 51.3010μs 29.3804μs 34.0363 KOps/s 33.5620 KOps/s $\color{#35bf28}+1.41\%$
test_step_mdp_speed[False-False-False-False-True] 62.6710μs 28.7446μs 34.7892 KOps/s 34.8097 KOps/s $\color{#d91a1a}-0.06\%$
test_step_mdp_speed[False-False-False-False-False] 46.8800μs 18.6250μs 53.6913 KOps/s 52.5460 KOps/s $\color{#35bf28}+2.18\%$
test_values[generalized_advantage_estimate-True-True] 25.3334ms 24.8859ms 40.1833 Ops/s 39.9949 Ops/s $\color{#35bf28}+0.47\%$
test_values[vec_generalized_advantage_estimate-True-True] 88.5020ms 2.6685ms 374.7492 Ops/s 368.7491 Ops/s $\color{#35bf28}+1.63\%$
test_values[td0_return_estimate-False-False] 92.2630μs 65.3595μs 15.3000 KOps/s 14.9942 KOps/s $\color{#35bf28}+2.04\%$
test_values[td1_return_estimate-False-False] 55.6359ms 55.3713ms 18.0599 Ops/s 18.0286 Ops/s $\color{#35bf28}+0.17\%$
test_values[vec_td1_return_estimate-False-False] 1.3776ms 1.0833ms 923.0875 Ops/s 909.2437 Ops/s $\color{#35bf28}+1.52\%$
test_values[td_lambda_return_estimate-True-False] 88.0475ms 87.5611ms 11.4206 Ops/s 10.3346 Ops/s $\textbf{\color{#35bf28}+10.51\%}$
test_values[vec_td_lambda_return_estimate-True-False] 1.3927ms 1.0789ms 926.8324 Ops/s 917.2612 Ops/s $\color{#35bf28}+1.04\%$
test_gae_speed[generalized_advantage_estimate-False-1-512] 25.4850ms 25.2276ms 39.6392 Ops/s 39.0212 Ops/s $\color{#35bf28}+1.58\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 0.9390ms 0.7192ms 1.3904 KOps/s 1.3702 KOps/s $\color{#35bf28}+1.47\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.8324ms 0.6706ms 1.4912 KOps/s 1.4220 KOps/s $\color{#35bf28}+4.87\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 1.5981ms 1.4699ms 680.3278 Ops/s 659.1048 Ops/s $\color{#35bf28}+3.22\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 0.8565ms 0.6873ms 1.4549 KOps/s 1.3585 KOps/s $\textbf{\color{#35bf28}+7.10\%}$
test_dqn_speed 1.6305ms 1.4387ms 695.0882 Ops/s 670.0366 Ops/s $\color{#35bf28}+3.74\%$
test_ddpg_speed 3.1120ms 2.9602ms 337.8175 Ops/s 328.5923 Ops/s $\color{#35bf28}+2.81\%$
test_sac_speed 8.7490ms 8.5022ms 117.6169 Ops/s 114.9839 Ops/s $\color{#35bf28}+2.29\%$
test_redq_speed 11.1678ms 10.6733ms 93.6919 Ops/s 91.4720 Ops/s $\color{#35bf28}+2.43\%$
test_redq_deprec_speed 12.0207ms 11.5975ms 86.2258 Ops/s 84.4346 Ops/s $\color{#35bf28}+2.12\%$
test_td3_speed 8.4977ms 8.3824ms 119.2983 Ops/s 116.4192 Ops/s $\color{#35bf28}+2.47\%$
test_cql_speed 26.5902ms 26.0223ms 38.4285 Ops/s 37.8375 Ops/s $\color{#35bf28}+1.56\%$
test_a2c_speed 7.2228ms 5.7596ms 173.6234 Ops/s 173.3911 Ops/s $\color{#35bf28}+0.13\%$
test_ppo_speed 6.4301ms 6.0294ms 165.8527 Ops/s 164.7121 Ops/s $\color{#35bf28}+0.69\%$
test_reinforce_speed 5.4579ms 4.6856ms 213.4191 Ops/s 211.8982 Ops/s $\color{#35bf28}+0.72\%$
test_iql_speed 20.7934ms 20.2436ms 49.3983 Ops/s 46.2290 Ops/s $\textbf{\color{#35bf28}+6.86\%}$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 4.9062ms 4.7349ms 211.1973 Ops/s 203.2641 Ops/s $\color{#35bf28}+3.90\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 1.1926ms 0.6001ms 1.6665 KOps/s 1.6577 KOps/s $\color{#35bf28}+0.53\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.7311ms 0.5723ms 1.7473 KOps/s 1.7227 KOps/s $\color{#35bf28}+1.43\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 5.1456ms 4.7321ms 211.3233 Ops/s 205.9789 Ops/s $\color{#35bf28}+2.59\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 1.2713ms 0.5894ms 1.6968 KOps/s 1.6485 KOps/s $\color{#35bf28}+2.93\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.7173ms 0.5687ms 1.7585 KOps/s 1.7295 KOps/s $\color{#35bf28}+1.67\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] 4.6395ms 2.1630ms 462.3205 Ops/s 457.4426 Ops/s $\color{#35bf28}+1.07\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] 2.2159ms 2.0454ms 488.9107 Ops/s 480.7804 Ops/s $\color{#35bf28}+1.69\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 5.0487ms 4.8577ms 205.8594 Ops/s 198.5485 Ops/s $\color{#35bf28}+3.68\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 0.9183ms 0.7246ms 1.3801 KOps/s 1.3418 KOps/s $\color{#35bf28}+2.85\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 4.4491ms 0.7005ms 1.4275 KOps/s 1.4053 KOps/s $\color{#35bf28}+1.58\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 4.9007ms 4.6900ms 213.2211 Ops/s 203.9829 Ops/s $\color{#35bf28}+4.53\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.7775ms 0.5973ms 1.6741 KOps/s 1.6412 KOps/s $\color{#35bf28}+2.01\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.7427ms 0.5725ms 1.7467 KOps/s 1.7070 KOps/s $\color{#35bf28}+2.33\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 5.0384ms 4.7141ms 212.1314 Ops/s 205.5659 Ops/s $\color{#35bf28}+3.19\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.7449ms 0.5902ms 1.6944 KOps/s 1.6607 KOps/s $\color{#35bf28}+2.03\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 4.5523ms 0.5727ms 1.7463 KOps/s 1.7170 KOps/s $\color{#35bf28}+1.70\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 5.1491ms 4.8732ms 205.2027 Ops/s 198.8030 Ops/s $\color{#35bf28}+3.22\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 0.8889ms 0.7275ms 1.3746 KOps/s 1.3661 KOps/s $\color{#35bf28}+0.63\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.9099ms 0.7008ms 1.4268 KOps/s 1.3798 KOps/s $\color{#35bf28}+3.41\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 0.1140s 9.3319ms 107.1595 Ops/s 133.9695 Ops/s $\textbf{\color{#d91a1a}-20.01\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 19.3311ms 16.5328ms 60.4860 Ops/s 59.1215 Ops/s $\color{#35bf28}+2.31\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 1.4385ms 1.3160ms 759.8684 Ops/s 751.0979 Ops/s $\color{#35bf28}+1.17\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 0.1075s 7.1604ms 139.6580 Ops/s 137.4590 Ops/s $\color{#35bf28}+1.60\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 19.3489ms 16.5130ms 60.5582 Ops/s 52.8938 Ops/s $\textbf{\color{#35bf28}+14.49\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 2.4542ms 1.3315ms 751.0375 Ops/s 720.5155 Ops/s $\color{#35bf28}+4.24\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 0.1080s 9.3541ms 106.9053 Ops/s 133.9165 Ops/s $\textbf{\color{#d91a1a}-20.17\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 19.0991ms 16.7130ms 59.8336 Ops/s 57.7737 Ops/s $\color{#35bf28}+3.57\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 2.5602ms 1.4959ms 668.4904 Ops/s 646.0991 Ops/s $\color{#35bf28}+3.47\%$

@vmoens vmoens added the enhancement New feature or request label Feb 21, 2024
@vmoens vmoens linked an issue Jun 11, 2024 that may be closed by this pull request
[BUG] NonTensorData disappears from environment after step_mdp() #2171
Open
3 tasks
@vmoens
Merge remote-tracking branch 'origin/main' into fix-env-nontensor
78d910f 
# Conflicts:
#	test/mocking_classes.py
#	test/test_env.py
#	torchrl/data/tensor_specs.py
#	torchrl/envs/batched_envs.py
#	torchrl/envs/utils.py
@vmoens
Copy link
Contributor Author

vmoens commented Jun 11, 2024

@maximilianigl @Michael-C-Strobel I'm going back to this
For a single env it is easy to patch.
For batched envs there are two ways of doing this:

  • We used to rely entirely on buffers in batched envs to share data across processes. We can't do that generically with non-tensor data atm. So I was thinking to share the non-tensor data separately through the pipes and the rest on the buffers. This is awfully complex but it could work
  • The simpler idea could be to prevent users from using buffers altogether. Envs with non-tensor data will be slow to be executed in parallel envs, but that's going to be the price to pay to get them running.

I guess my question is this: how much do you guys care about parallel envs runtime when it comes to envs with non tensor data?

@vmoens
Copy link
Contributor Author

vmoens commented Jun 11, 2024

I fixed both options (with and without buffers) and all are incorporated in the tests.
There may still be some incompatibility with collectors, but I think this should belong to a separate PR.

@maximilianigl
Copy link

For my project I've hacked together an "encoder"/"decoder" step that encodes non-tensor data into ints and stores the values in a dictionary. That allowed me to use buffers and just "decode" the data in places I need it.
Regarding your question: If I still had the problem, I'd care about runtime. But not sure how much complexity that justifies.

@Michael-C-Strobel
Copy link

I think parallel envs will end up being pretty important to us. In our case our environment initialization instantiates an object from another package but neither the environment nor agent modifies it (the agent just needs access to some of its methods/attributed).

For us, a constant NonTensorSpec would probably work and I suspect wouldn’t cause any issues for the collector or replay buffer.

@vmoens
Copy link
Contributor Author

vmoens commented Jun 11, 2024

Ok so I think that now runtime should be ok-ish (at least not drastically slower)
Next steps after this PR are collectors and buffers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. enhancement New feature or request
Projects
None yet
Milestone
No milestone
4 participants
@vmoens @maximilianigl @Michael-C-Strobel @facebook-github-bot