Sagemaker Fails to download code from S3

[uwaisiqbal]

Describe the bug
When using sagemaker in script mode, if there are files in the /opt/ml/code directory, sagemaker doesn't download the packaged code from S3.

In my Dockerfile to build the image, I copy over some files into /opt/ml/code and I would like to use this container in a way where I can execute my scripts against it with sagemaker script mode. However it seems as though the scripts which are uploaded and packaged to s3 are never actually downloaded once the instance spins up and sagemaker starts. It fails to find the script I supplied.

When I tried running it without copying my code into /opt/ml/code in the Dockerfile everything worked fine. I think there is a problematic line of code which enforces this logic to only download the code when the /opt/ml/code directory is empty.

You can find the problematic line here:
When run is called and the files are attempted to be downloaded:

sagemaker-training-toolkit/src/sagemaker_training/entry_point.py

Line 92 in 7d9883c

files.download_and_extract(uri=uri, path=environment.code_dir)

sagemaker-training-toolkit/src/sagemaker_training/files.py

Line 127 in 7d9883c

if not os.listdir(path):