[NOMERG] Prototyping tensorclass for losses

[vmoens]

We project on using @Tensorclass to represent losses.

The advantage of tensorclass for losses instead of tensordict is that it will help us use all the features of tensordict while preserving type annotation or even completion.

This draft shows the various steps to integrate this feature:

• Check the out_keys of the loss;
• Create a tensorclass with the respective fields;
• Type the forward as returning that class (and/or a tensordict)
• Add an argument to return the class in the constructor with the False value by default;
• Update the docstrings (not done)
• Write a little test to check that things work as expected (this test should be new and not parametrized - if we add one more parameter to the existing tests the code will be much longer and harder to follow, and the tests will run for a long time).

Hopefully, all the losses in torchrl.objectives should be patchable with this.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/1892

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (5 Unrelated Failures)

As of commit f949925 with merge base 4d52d5f ():

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• Habitat Tests on Linux / tests (3.9, 11.6) / linux-job (gh)
  RuntimeError: Command docker exec -t dcb5ef69ef7c8a4fdb328c850832e54d5192de3b6ad4c13e4daaa11f646a0f55 /exec failed with exit code 139
• Lint / python-source-and-configs / linux-job (gh)
  RuntimeError: Command docker exec -t d3f1409b0db47a13fb005111b30ab8b19da4c286eed9b4b7f186a805fb188c56 /exec failed with exit code 1
• Unit-tests on Linux / tests-gpu (3.8, 12.1) / linux-job (gh)
  test/test_rb.py::TestStorages::test_storage_dumps_loads[True-pytree-LazyTensorStorage-device_data0]
• Unit-tests on Windows / unittests-gpu / windows-job (gh)
  ##[error]The operation was canceled.

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Unit-tests on Linux / tests-optdeps (3.9, 12.1) / linux-job (gh)
  test/test_rb.py::TestStorages::test_storage_dumps_loads[True-pytree-LazyTensorStorage-device_data0]

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 89. Improved: $\large\color{#35bf28}7$. Worsened: $\large\color{#d91a1a}3$.

Expand to view detailed results

Name	Max	Mean	Ops	Ops on Repo HEAD	Change
test_single	0.1317s	66.7379ms	14.9840 Ops/s	16.1768 Ops/s	$\textbf{\color{#d91a1a}-7.37\%}$
test_sync	43.0471ms	33.8776ms	29.5180 Ops/s	29.7845 Ops/s	$\color{#d91a1a}-0.89\%$
test_async	99.9110ms	33.2546ms	30.0710 Ops/s	29.9225 Ops/s	$\color{#35bf28}+0.50\%$
test_simple	0.4970s	0.4359s	2.2942 Ops/s	2.2958 Ops/s	$\color{#d91a1a}-0.07\%$
test_transformed	0.6571s	0.6032s	1.6578 Ops/s	1.6726 Ops/s	$\color{#d91a1a}-0.88\%$
test_serial	1.5146s	1.4526s	0.6884 Ops/s	0.6947 Ops/s	$\color{#d91a1a}-0.90\%$
test_parallel	1.4499s	1.3902s	0.7193 Ops/s	0.7354 Ops/s	$\color{#d91a1a}-2.19\%$
test_step_mdp_speed[True-True-True-True-True]	88.1650μs	20.9590μs	47.7123 KOps/s	47.8348 KOps/s	$\color{#d91a1a}-0.26\%$
test_step_mdp_speed[True-True-True-True-False]	38.3820μs	12.8344μs	77.9157 KOps/s	77.7086 KOps/s	$\color{#35bf28}+0.27\%$
test_step_mdp_speed[True-True-True-False-True]	34.2040μs	12.4656μs	80.2207 KOps/s	81.7145 KOps/s	$\color{#d91a1a}-1.83\%$
test_step_mdp_speed[True-True-True-False-False]	33.1720μs	7.5053μs	133.2396 KOps/s	132.3081 KOps/s	$\color{#35bf28}+0.70\%$
test_step_mdp_speed[True-True-False-True-True]	61.9950μs	22.5489μs	44.3481 KOps/s	44.4220 KOps/s	$\color{#d91a1a}-0.17\%$
test_step_mdp_speed[True-True-False-True-False]	37.4800μs	14.1972μs	70.4362 KOps/s	70.8679 KOps/s	$\color{#d91a1a}-0.61\%$
test_step_mdp_speed[True-True-False-False-True]	39.8050μs	13.6177μs	73.4337 KOps/s	73.8477 KOps/s	$\color{#d91a1a}-0.56\%$
test_step_mdp_speed[True-True-False-False-False]	48.3770μs	8.7709μs	114.0137 KOps/s	114.2242 KOps/s	$\color{#d91a1a}-0.18\%$
test_step_mdp_speed[True-False-True-True-True]	67.6060μs	23.9648μs	41.7279 KOps/s	41.9937 KOps/s	$\color{#d91a1a}-0.63\%$
test_step_mdp_speed[True-False-True-True-False]	62.6570μs	15.5761μs	64.2011 KOps/s	64.7681 KOps/s	$\color{#d91a1a}-0.88\%$
test_step_mdp_speed[True-False-True-False-True]	68.4780μs	13.5192μs	73.9689 KOps/s	73.7232 KOps/s	$\color{#35bf28}+0.33\%$
test_step_mdp_speed[True-False-True-False-False]	32.4310μs	8.6906μs	115.0668 KOps/s	115.0521 KOps/s	$\color{#35bf28}+0.01\%$
test_step_mdp_speed[True-False-False-True-True]	66.3740μs	24.5790μs	40.6851 KOps/s	39.9482 KOps/s	$\color{#35bf28}+1.84\%$
test_step_mdp_speed[True-False-False-True-False]	44.7430μs	16.5635μs	60.3739 KOps/s	60.2639 KOps/s	$\color{#35bf28}+0.18\%$
test_step_mdp_speed[True-False-False-False-True]	62.6070μs	14.7021μs	68.0176 KOps/s	68.7703 KOps/s	$\color{#d91a1a}-1.09\%$
test_step_mdp_speed[True-False-False-False-False]	33.0310μs	9.9923μs	100.0770 KOps/s	100.3671 KOps/s	$\color{#d91a1a}-0.29\%$
test_step_mdp_speed[False-True-True-True-True]	68.3280μs	23.8000μs	42.0169 KOps/s	42.2796 KOps/s	$\color{#d91a1a}-0.62\%$
test_step_mdp_speed[False-True-True-True-False]	60.3930μs	15.4881μs	64.5659 KOps/s	64.7847 KOps/s	$\color{#d91a1a}-0.34\%$
test_step_mdp_speed[False-True-True-False-True]	44.5130μs	15.7759μs	63.3880 KOps/s	63.1991 KOps/s	$\color{#35bf28}+0.30\%$
test_step_mdp_speed[False-True-True-False-False]	53.1190μs	9.8077μs	101.9608 KOps/s	100.0898 KOps/s	$\color{#35bf28}+1.87\%$
test_step_mdp_speed[False-True-False-True-True]	50.5440μs	25.0179μs	39.9713 KOps/s	40.0240 KOps/s	$\color{#d91a1a}-0.13\%$
test_step_mdp_speed[False-True-False-True-False]	40.6360μs	16.6357μs	60.1119 KOps/s	60.2332 KOps/s	$\color{#d91a1a}-0.20\%$
test_step_mdp_speed[False-True-False-False-True]	60.5830μs	16.7080μs	59.8515 KOps/s	59.2713 KOps/s	$\color{#35bf28}+0.98\%$
test_step_mdp_speed[False-True-False-False-False]	59.0500μs	11.0780μs	90.2692 KOps/s	89.9029 KOps/s	$\color{#35bf28}+0.41\%$
test_step_mdp_speed[False-False-True-True-True]	77.5940μs	26.2853μs	38.0441 KOps/s	38.2894 KOps/s	$\color{#d91a1a}-0.64\%$
test_step_mdp_speed[False-False-True-True-False]	56.4750μs	17.8522μs	56.0155 KOps/s	55.7692 KOps/s	$\color{#35bf28}+0.44\%$
test_step_mdp_speed[False-False-True-False-True]	43.9720μs	16.8988μs	59.1757 KOps/s	59.2517 KOps/s	$\color{#d91a1a}-0.13\%$
test_step_mdp_speed[False-False-True-False-False]	56.4150μs	11.0275μs	90.6827 KOps/s	89.1656 KOps/s	$\color{#35bf28}+1.70\%$
test_step_mdp_speed[False-False-False-True-True]	56.7760μs	27.2094μs	36.7521 KOps/s	36.9291 KOps/s	$\color{#d91a1a}-0.48\%$
test_step_mdp_speed[False-False-False-True-False]	0.2886ms	19.3875μs	51.5796 KOps/s	52.7550 KOps/s	$\color{#d91a1a}-2.23\%$
test_step_mdp_speed[False-False-False-False-True]	53.0890μs	17.9729μs	55.6395 KOps/s	55.8695 KOps/s	$\color{#d91a1a}-0.41\%$
test_step_mdp_speed[False-False-False-False-False]	29.7850μs	12.1660μs	82.1961 KOps/s	82.3408 KOps/s	$\color{#d91a1a}-0.18\%$
test_values[generalized_advantage_estimate-True-True]	11.3920ms	9.5138ms	105.1107 Ops/s	106.6806 Ops/s	$\color{#d91a1a}-1.47\%$
test_values[vec_generalized_advantage_estimate-True-True]	35.4236ms	32.9150ms	30.3813 Ops/s	28.6882 Ops/s	$\textbf{\color{#35bf28}+5.90\%}$
test_values[td0_return_estimate-False-False]	0.2206ms	0.1655ms	6.0421 KOps/s	6.1572 KOps/s	$\color{#d91a1a}-1.87\%$
test_values[td1_return_estimate-False-False]	24.1455ms	23.8220ms	41.9780 Ops/s	44.1222 Ops/s	$\color{#d91a1a}-4.86\%$
test_values[vec_td1_return_estimate-False-False]	34.2651ms	33.0600ms	30.2480 Ops/s	28.6137 Ops/s	$\textbf{\color{#35bf28}+5.71\%}$
test_values[td_lambda_return_estimate-True-False]	37.4565ms	34.4619ms	29.0175 Ops/s	30.2422 Ops/s	$\color{#d91a1a}-4.05\%$
test_values[vec_td_lambda_return_estimate-True-False]	34.6997ms	32.8962ms	30.3987 Ops/s	28.6020 Ops/s	$\textbf{\color{#35bf28}+6.28\%}$
test_gae_speed[generalized_advantage_estimate-False-1-512]	11.3351ms	8.3587ms	119.6357 Ops/s	124.0618 Ops/s	$\color{#d91a1a}-3.57\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512]	2.6059ms	1.9797ms	505.1167 Ops/s	560.7918 Ops/s	$\textbf{\color{#d91a1a}-9.93\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512]	0.4586ms	0.3536ms	2.8285 KOps/s	2.9327 KOps/s	$\color{#d91a1a}-3.55\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512]	40.2049ms	38.6055ms	25.9030 Ops/s	23.5077 Ops/s	$\textbf{\color{#35bf28}+10.19\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512]	3.5874ms	3.0145ms	331.7257 Ops/s	332.1506 Ops/s	$\color{#d91a1a}-0.13\%$
test_dqn_speed	1.8557ms	1.3650ms	732.5942 Ops/s	695.3602 Ops/s	$\textbf{\color{#35bf28}+5.35\%}$
test_ddpg_speed	3.1865ms	2.7272ms	366.6722 Ops/s	370.2487 Ops/s	$\color{#d91a1a}-0.97\%$
test_sac_speed	11.5429ms	8.6226ms	115.9741 Ops/s	118.4600 Ops/s	$\color{#d91a1a}-2.10\%$
test_redq_speed	14.3070ms	13.2079ms	75.7124 Ops/s	76.5414 Ops/s	$\color{#d91a1a}-1.08\%$
test_redq_deprec_speed	15.2292ms	13.4248ms	74.4892 Ops/s	75.0865 Ops/s	$\color{#d91a1a}-0.80\%$
test_td3_speed	8.8409ms	8.6440ms	115.6869 Ops/s	116.7111 Ops/s	$\color{#d91a1a}-0.88\%$
test_cql_speed	37.8557ms	36.4538ms	27.4320 Ops/s	27.4838 Ops/s	$\color{#d91a1a}-0.19\%$
test_a2c_speed	8.0616ms	7.3534ms	135.9916 Ops/s	137.2881 Ops/s	$\color{#d91a1a}-0.94\%$
test_ppo_speed	8.2274ms	7.6060ms	131.4749 Ops/s	133.0168 Ops/s	$\color{#d91a1a}-1.16\%$
test_reinforce_speed	7.5024ms	6.5784ms	152.0129 Ops/s	152.7289 Ops/s	$\color{#d91a1a}-0.47\%$
test_iql_speed	34.3591ms	32.8578ms	30.4342 Ops/s	30.5895 Ops/s	$\color{#d91a1a}-0.51\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	2.9634ms	2.6799ms	373.1445 Ops/s	383.2968 Ops/s	$\color{#d91a1a}-2.65\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	0.7629ms	0.5125ms	1.9512 KOps/s	1.9734 KOps/s	$\color{#d91a1a}-1.13\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.9172ms	0.4836ms	2.0677 KOps/s	2.0737 KOps/s	$\color{#d91a1a}-0.29\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	3.1588ms	2.6914ms	371.5541 Ops/s	370.5536 Ops/s	$\color{#35bf28}+0.27\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	0.6782ms	0.5027ms	1.9891 KOps/s	1.9980 KOps/s	$\color{#d91a1a}-0.44\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.7713ms	0.4834ms	2.0686 KOps/s	2.0892 KOps/s	$\color{#d91a1a}-0.98\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	3.0846ms	2.7645ms	361.7321 Ops/s	363.7553 Ops/s	$\color{#d91a1a}-0.56\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	0.9741ms	0.6375ms	1.5686 KOps/s	1.6139 KOps/s	$\color{#d91a1a}-2.81\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	4.9127ms	0.6106ms	1.6377 KOps/s	1.6748 KOps/s	$\color{#d91a1a}-2.22\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	3.9597ms	2.6952ms	371.0272 Ops/s	382.1488 Ops/s	$\color{#d91a1a}-2.91\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	0.6113ms	0.5125ms	1.9513 KOps/s	1.9732 KOps/s	$\color{#d91a1a}-1.11\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.7462ms	0.4914ms	2.0350 KOps/s	2.0659 KOps/s	$\color{#d91a1a}-1.49\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	3.2014ms	2.7498ms	363.6631 Ops/s	348.2003 Ops/s	$\color{#35bf28}+4.44\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	0.6299ms	0.5099ms	1.9613 KOps/s	1.9969 KOps/s	$\color{#d91a1a}-1.78\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.7138ms	0.4864ms	2.0558 KOps/s	2.0877 KOps/s	$\color{#d91a1a}-1.53\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	3.1028ms	2.8097ms	355.9074 Ops/s	348.5543 Ops/s	$\color{#35bf28}+2.11\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	0.7653ms	0.6325ms	1.5809 KOps/s	1.6023 KOps/s	$\color{#d91a1a}-1.33\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.7537ms	0.6080ms	1.6448 KOps/s	1.6738 KOps/s	$\color{#d91a1a}-1.73\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400]	0.1095s	8.0175ms	124.7268 Ops/s	122.9759 Ops/s	$\color{#35bf28}+1.42\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400]	15.0576ms	13.1005ms	76.3330 Ops/s	77.2690 Ops/s	$\color{#d91a1a}-1.21\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400]	3.7956ms	2.5195ms	396.8996 Ops/s	395.5817 Ops/s	$\color{#35bf28}+0.33\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400]	92.2882ms	9.2998ms	107.5291 Ops/s	102.1554 Ops/s	$\textbf{\color{#35bf28}+5.26\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400]	15.6308ms	13.1799ms	75.8731 Ops/s	72.2723 Ops/s	$\color{#35bf28}+4.98\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400]	4.0738ms	2.5459ms	392.7919 Ops/s	391.8544 Ops/s	$\color{#35bf28}+0.24\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400]	0.1067s	10.0310ms	99.6913 Ops/s	125.5193 Ops/s	$\textbf{\color{#d91a1a}-20.58\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400]	15.5958ms	13.4472ms	74.3652 Ops/s	65.8234 Ops/s	$\textbf{\color{#35bf28}+12.98\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400]	4.9960ms	2.8226ms	354.2851 Ops/s	360.6660 Ops/s	$\color{#d91a1a}-1.77\%$

[github-actions]

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of GPU Benchmark Tests

Total Benchmarks: 92. Improved: $\large\color{#35bf28}2$. Worsened: $\large\color{#d91a1a}1$.

Expand to view detailed results

Name	Max	Mean	Ops	Ops on Repo HEAD	Change
test_single	0.1221s	0.1177s	8.4952 Ops/s	8.3065 Ops/s	$\color{#35bf28}+2.27\%$
test_sync	0.1718s	0.1030s	9.7131 Ops/s	9.7053 Ops/s	$\color{#35bf28}+0.08\%$
test_async	0.2541s	92.0860ms	10.8594 Ops/s	10.9334 Ops/s	$\color{#d91a1a}-0.68\%$
test_single_pixels	0.1389s	0.1380s	7.2473 Ops/s	7.2120 Ops/s	$\color{#35bf28}+0.49\%$
test_sync_pixels	83.8074ms	83.0213ms	12.0451 Ops/s	12.4081 Ops/s	$\color{#d91a1a}-2.93\%$
test_async_pixels	0.2167s	74.3765ms	13.4451 Ops/s	13.2519 Ops/s	$\color{#35bf28}+1.46\%$
test_simple	0.9064s	0.8357s	1.1966 Ops/s	1.2069 Ops/s	$\color{#d91a1a}-0.85\%$
test_transformed	1.1530s	1.0859s	0.9209 Ops/s	0.9107 Ops/s	$\color{#35bf28}+1.12\%$
test_serial	2.5123s	2.4521s	0.4078 Ops/s	0.4093 Ops/s	$\color{#d91a1a}-0.38\%$
test_parallel	2.2229s	2.1228s	0.4711 Ops/s	0.4808 Ops/s	$\color{#d91a1a}-2.02\%$
test_step_mdp_speed[True-True-True-True-True]	73.5320μs	33.1107μs	30.2017 KOps/s	30.4348 KOps/s	$\color{#d91a1a}-0.77\%$
test_step_mdp_speed[True-True-True-True-False]	46.5220μs	19.6602μs	50.8642 KOps/s	51.6482 KOps/s	$\color{#d91a1a}-1.52\%$
test_step_mdp_speed[True-True-True-False-True]	53.6510μs	18.3788μs	54.4104 KOps/s	53.7403 KOps/s	$\color{#35bf28}+1.25\%$
test_step_mdp_speed[True-True-True-False-False]	31.4900μs	10.8735μs	91.9665 KOps/s	89.5362 KOps/s	$\color{#35bf28}+2.71\%$
test_step_mdp_speed[True-True-False-True-True]	61.0810μs	33.3664μs	29.9703 KOps/s	29.7370 KOps/s	$\color{#35bf28}+0.78\%$
test_step_mdp_speed[True-True-False-True-False]	45.9110μs	20.7536μs	48.1845 KOps/s	47.5779 KOps/s	$\color{#35bf28}+1.27\%$
test_step_mdp_speed[True-True-False-False-True]	44.3510μs	20.0342μs	49.9147 KOps/s	49.0423 KOps/s	$\color{#35bf28}+1.78\%$
test_step_mdp_speed[True-True-False-False-False]	33.3800μs	12.7732μs	78.2889 KOps/s	77.3742 KOps/s	$\color{#35bf28}+1.18\%$
test_step_mdp_speed[True-False-True-True-True]	64.0510μs	35.4592μs	28.2014 KOps/s	27.9774 KOps/s	$\color{#35bf28}+0.80\%$
test_step_mdp_speed[True-False-True-True-False]	61.0220μs	22.5655μs	44.3154 KOps/s	43.3115 KOps/s	$\color{#35bf28}+2.32\%$
test_step_mdp_speed[True-False-True-False-True]	49.3110μs	20.0256μs	49.9362 KOps/s	48.9737 KOps/s	$\color{#35bf28}+1.97\%$
test_step_mdp_speed[True-False-True-False-False]	31.8210μs	12.5369μs	79.7643 KOps/s	78.9761 KOps/s	$\color{#35bf28}+1.00\%$
test_step_mdp_speed[True-False-False-True-True]	64.9620μs	36.7974μs	27.1758 KOps/s	26.8348 KOps/s	$\color{#35bf28}+1.27\%$
test_step_mdp_speed[True-False-False-True-False]	45.3510μs	24.3945μs	40.9929 KOps/s	40.4502 KOps/s	$\color{#35bf28}+1.34\%$
test_step_mdp_speed[True-False-False-False-True]	47.6620μs	21.8879μs	45.6872 KOps/s	45.7024 KOps/s	$\color{#d91a1a}-0.03\%$
test_step_mdp_speed[True-False-False-False-False]	35.5710μs	14.3344μs	69.7622 KOps/s	68.8340 KOps/s	$\color{#35bf28}+1.35\%$
test_step_mdp_speed[False-True-True-True-True]	78.1520μs	35.3893μs	28.2571 KOps/s	27.4532 KOps/s	$\color{#35bf28}+2.93\%$
test_step_mdp_speed[False-True-True-True-False]	47.1410μs	22.6284μs	44.1923 KOps/s	43.3142 KOps/s	$\color{#35bf28}+2.03\%$
test_step_mdp_speed[False-True-True-False-True]	48.4310μs	23.7057μs	42.1840 KOps/s	41.6068 KOps/s	$\color{#35bf28}+1.39\%$
test_step_mdp_speed[False-True-True-False-False]	33.6110μs	14.3729μs	69.5753 KOps/s	68.8308 KOps/s	$\color{#35bf28}+1.08\%$
test_step_mdp_speed[False-True-False-True-True]	70.8320μs	37.8034μs	26.4526 KOps/s	25.9017 KOps/s	$\color{#35bf28}+2.13\%$
test_step_mdp_speed[False-True-False-True-False]	47.2420μs	24.5801μs	40.6834 KOps/s	40.1889 KOps/s	$\color{#35bf28}+1.23\%$
test_step_mdp_speed[False-True-False-False-True]	51.3420μs	25.7088μs	38.8972 KOps/s	38.6801 KOps/s	$\color{#35bf28}+0.56\%$
test_step_mdp_speed[False-True-False-False-False]	36.7810μs	16.0933μs	62.1376 KOps/s	60.2672 KOps/s	$\color{#35bf28}+3.10\%$
test_step_mdp_speed[False-False-True-True-True]	68.2110μs	38.9283μs	25.6882 KOps/s	25.1311 KOps/s	$\color{#35bf28}+2.22\%$
test_step_mdp_speed[False-False-True-True-False]	50.2410μs	26.2073μs	38.1574 KOps/s	37.4497 KOps/s	$\color{#35bf28}+1.89\%$
test_step_mdp_speed[False-False-True-False-True]	46.9010μs	25.3140μs	39.5038 KOps/s	39.0699 KOps/s	$\color{#35bf28}+1.11\%$
test_step_mdp_speed[False-False-True-False-False]	38.7210μs	16.0684μs	62.2339 KOps/s	61.0442 KOps/s	$\color{#35bf28}+1.95\%$
test_step_mdp_speed[False-False-False-True-True]	66.6620μs	40.1442μs	24.9102 KOps/s	24.3500 KOps/s	$\color{#35bf28}+2.30\%$
test_step_mdp_speed[False-False-False-True-False]	55.1110μs	28.4240μs	35.1815 KOps/s	35.2784 KOps/s	$\color{#d91a1a}-0.27\%$
test_step_mdp_speed[False-False-False-False-True]	51.5010μs	27.4631μs	36.4125 KOps/s	37.3028 KOps/s	$\color{#d91a1a}-2.39\%$
test_step_mdp_speed[False-False-False-False-False]	43.7310μs	18.0333μs	55.4530 KOps/s	55.4494 KOps/s	$+0.01\%$
test_values[generalized_advantage_estimate-True-True]	27.2212ms	26.6797ms	37.4817 Ops/s	39.1156 Ops/s	$\color{#d91a1a}-4.18\%$
test_values[vec_generalized_advantage_estimate-True-True]	91.0395ms	3.3901ms	294.9806 Ops/s	294.8425 Ops/s	$\color{#35bf28}+0.05\%$
test_values[td0_return_estimate-False-False]	0.1018ms	62.2585μs	16.0621 KOps/s	16.3261 KOps/s	$\color{#d91a1a}-1.62\%$
test_values[td1_return_estimate-False-False]	54.5751ms	54.1384ms	18.4712 Ops/s	18.0452 Ops/s	$\color{#35bf28}+2.36\%$
test_values[vec_td1_return_estimate-False-False]	2.0880ms	1.7672ms	565.8537 Ops/s	564.7870 Ops/s	$\color{#35bf28}+0.19\%$
test_values[td_lambda_return_estimate-True-False]	92.3128ms	88.2978ms	11.3253 Ops/s	11.5403 Ops/s	$\color{#d91a1a}-1.86\%$
test_values[vec_td_lambda_return_estimate-True-False]	4.0257ms	1.8388ms	543.8289 Ops/s	552.5542 Ops/s	$\color{#d91a1a}-1.58\%$
test_gae_speed[generalized_advantage_estimate-False-1-512]	24.2837ms	24.0840ms	41.5214 Ops/s	39.6430 Ops/s	$\color{#35bf28}+4.74\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512]	0.8903ms	0.7085ms	1.4114 KOps/s	1.3992 KOps/s	$\color{#35bf28}+0.87\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512]	0.7241ms	0.6594ms	1.5166 KOps/s	1.5075 KOps/s	$\color{#35bf28}+0.61\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512]	1.5451ms	1.4602ms	684.8504 Ops/s	682.1878 Ops/s	$\color{#35bf28}+0.39\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512]	0.9416ms	0.6746ms	1.4824 KOps/s	1.4655 KOps/s	$\color{#35bf28}+1.15\%$
test_dqn_speed	8.0148ms	1.4573ms	686.2001 Ops/s	699.1882 Ops/s	$\color{#d91a1a}-1.86\%$
test_ddpg_speed	3.2274ms	2.7714ms	360.8309 Ops/s	360.3233 Ops/s	$\color{#35bf28}+0.14\%$
test_sac_speed	9.1391ms	8.5604ms	116.8169 Ops/s	116.7505 Ops/s	$\color{#35bf28}+0.06\%$
test_redq_speed	11.2879ms	10.6424ms	93.9636 Ops/s	94.5424 Ops/s	$\color{#d91a1a}-0.61\%$
test_redq_deprec_speed	12.6686ms	11.8821ms	84.1600 Ops/s	82.5704 Ops/s	$\color{#35bf28}+1.93\%$
test_td3_speed	17.6941ms	8.7953ms	113.6972 Ops/s	113.1527 Ops/s	$\color{#35bf28}+0.48\%$
test_cql_speed	27.1657ms	25.6180ms	39.0351 Ops/s	38.8628 Ops/s	$\color{#35bf28}+0.44\%$
test_a2c_speed	6.0731ms	5.3417ms	187.2046 Ops/s	182.4859 Ops/s	$\color{#35bf28}+2.59\%$
test_ppo_speed	6.7748ms	5.6514ms	176.9462 Ops/s	169.8745 Ops/s	$\color{#35bf28}+4.16\%$
test_reinforce_speed	4.8452ms	4.4293ms	225.7687 Ops/s	217.6525 Ops/s	$\color{#35bf28}+3.73\%$
test_iql_speed	20.5029ms	19.4602ms	51.3869 Ops/s	49.9941 Ops/s	$\color{#35bf28}+2.79\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	3.7086ms	3.5202ms	284.0756 Ops/s	277.8365 Ops/s	$\color{#35bf28}+2.25\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	0.6786ms	0.5511ms	1.8146 KOps/s	1.8013 KOps/s	$\color{#35bf28}+0.74\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.6953ms	0.5264ms	1.8997 KOps/s	1.8978 KOps/s	$\color{#35bf28}+0.10\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	3.6785ms	3.5491ms	281.7582 Ops/s	276.8810 Ops/s	$\color{#35bf28}+1.76\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	0.6743ms	0.5452ms	1.8343 KOps/s	1.8285 KOps/s	$\color{#35bf28}+0.31\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.6589ms	0.5206ms	1.9207 KOps/s	1.9172 KOps/s	$\color{#35bf28}+0.18\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	3.8046ms	3.6659ms	272.7873 Ops/s	266.3998 Ops/s	$\color{#35bf28}+2.40\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	0.8914ms	0.6803ms	1.4699 KOps/s	1.4708 KOps/s	$\color{#d91a1a}-0.06\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.8952ms	0.6522ms	1.5332 KOps/s	1.3442 KOps/s	$\textbf{\color{#35bf28}+14.07\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	3.7492ms	3.5399ms	282.4951 Ops/s	277.6220 Ops/s	$\color{#35bf28}+1.76\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	0.8001ms	0.5525ms	1.8099 KOps/s	1.7954 KOps/s	$\color{#35bf28}+0.81\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.6657ms	0.5263ms	1.9002 KOps/s	1.8866 KOps/s	$\color{#35bf28}+0.72\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	3.8989ms	3.5808ms	279.2633 Ops/s	274.4021 Ops/s	$\color{#35bf28}+1.77\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	0.6916ms	0.5469ms	1.8286 KOps/s	1.8169 KOps/s	$\color{#35bf28}+0.65\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.7786ms	0.5212ms	1.9187 KOps/s	1.9059 KOps/s	$\color{#35bf28}+0.67\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	3.8350ms	3.6848ms	271.3874 Ops/s	267.1395 Ops/s	$\color{#35bf28}+1.59\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	0.9254ms	0.6852ms	1.4595 KOps/s	1.4608 KOps/s	$\color{#d91a1a}-0.09\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.8058ms	0.6604ms	1.5143 KOps/s	1.5226 KOps/s	$\color{#d91a1a}-0.54\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400]	0.1252s	10.1580ms	98.4449 Ops/s	96.3714 Ops/s	$\color{#35bf28}+2.15\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400]	18.3758ms	16.0623ms	62.2577 Ops/s	61.2868 Ops/s	$\color{#35bf28}+1.58\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400]	7.1185ms	3.0545ms	327.3825 Ops/s	321.7154 Ops/s	$\color{#35bf28}+1.76\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400]	0.1224s	12.3617ms	80.8952 Ops/s	98.3563 Ops/s	$\textbf{\color{#d91a1a}-17.75\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400]	18.0948ms	16.0778ms	62.1975 Ops/s	61.2134 Ops/s	$\color{#35bf28}+1.61\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400]	7.3449ms	3.0351ms	329.4799 Ops/s	329.6164 Ops/s	$\color{#d91a1a}-0.04\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400]	0.1231s	10.4253ms	95.9202 Ops/s	78.6963 Ops/s	$\textbf{\color{#35bf28}+21.89\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400]	18.7368ms	16.3837ms	61.0362 Ops/s	60.1265 Ops/s	$\color{#35bf28}+1.51\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400]	8.2807ms	3.3474ms	298.7426 Ops/s	302.3509 Ops/s	$\color{#d91a1a}-1.19\%$