Memory leaked

[chocolate9624]

🐛 Bug

Command

pytest ./tests/modules/test_encoders.py

Environment

Collecting environment information...
PyTorch version: 1.9.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 12.3 (x86_64)
GCC version: Could not collect
Clang version: 13.1.6 (clang-1316.0.21.2.5)
CMake version: Could not collect
Libc version: N/A

Python version: 3.7 (64-bit runtime)
Python platform: Darwin-21.4.0-x86_64-i386-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.21.4
[pip3] pytorch-lightning==1.6.0.dev0
[pip3] torch==1.9.0
[pip3] torchaudio==0.9.0
[pip3] torchmetrics==0.10.0
[pip3] torchtext==0.5.0
[pip3] torchvision==0.10.0
[conda] numpy 1.21.4 pypi_0 pypi
[conda] pytorch-lightning 1.6.0.dev0 pypi_0 pypi
[conda] torch 1.9.0 pypi_0 pypi
[conda] torchaudio 0.9.0 pypi_0 pypi
[conda] torchmetrics 0.10.0 pypi_0 pypi
[conda] torchtext 0.5.0 pypi_0 pypi
[conda] torchvision 0.10.0 pypi_0 pypi

Additional context

LEAKED 35.07421875MB in tests/modules/test_encoders.py::TestEncoders::test_finetune_faster_rcnn_fpn_fc7
LEAKED 726.6328125MB in tests/modules/test_encoders.py::TestEncoders::test_multimodal_encoder_base
LEAKED 174.46875MB in tests/modules/test_encoders.py::TestEncoders::test_r2plus1d18_video_encoder
LEAKED 70.15234375MB in tests/modules/test_encoders.py::TestEncoders::test_resnet152_image_encoder
LEAKED 36.2734375MB in tests/modules/test_encoders.py::TestEncoders::test_resnet18_audio_encoder
LEAKED 177.26953125MB in tests/modules/test_encoders.py::TestEncoders::test_transformer_encoder
LEAKED 157.26953125MB in tests/modules/test_encoders.py::TestEncoders::test_transformer_encoder_forward
LEAKED 75.54296875MB in tests/modules/test_encoders.py::TestEncoders::test_vit_encoder