Use boto3 instead of awscli to download code package

[tkislan]

Updated PR from #389
Resolves #387

[tkislan]

No idea why the build failed

Downloading https://homebrew.bintray.com/bottles/qpdf-10.3.1.catalina.bottle.tar.gz
curl: (22) The requested URL returned error: 403 Forbidden
Error: Failed to download resource "qpdf"

[savingoyal]

I launched the tests again. Hopefully, it should resolve now. The existing test suite doesn't exercise the files that the PR touches.

[savingoyal]

@tkislan Have you tested this PR? I am running into this error - boto3;: 1: boto3;: Syntax error: end of file unexpected (expecting "done")

[tkislan]

I only tested the generated command that downloads the file

Can you please provide more context?
What the whole command looks like, etc

[savingoyal]

@tkislan This parsing can happen while we are generating the get_s3_download_command locally rather than in the remote environment. That will save us precious entry point chars.

[savingoyal]

I doubt \" would work here given that we use shlex downstream for parsing the entry point.

[savingoyal]

I only tested the generated command that downloads the file

Can you please provide more context?
What the whole command looks like, etc

The issue is with incorrect quoting given that we use shlex downstream. This is what the command looks like -

"/bin/sh","-c","set -e && echo 'Setting up task environment.' && python -m pip install click requests boto3 -qqq && mkdir metaflow && cd metaflow && mkdir .metaflow && i=0; while [ $i -le 5 ]; do echo 'Downloading code package.'; python -c import","boto3;","exec(try:\\n from urlparse import urlparse\\nexcept:\\n from urllib.parse import urlparse);parsed","=","urlparse(s3://oss-dev-metaflows3bucket-13oayj2nh8n4l/metaflow/ForeachFlow2q/data/95/950443769eee3cac5d35c5fa724bf390d5093b2b);boto3.client(s3).download_file(parsed.netloc,parsed.path.lstrip(/),","job.tar); >/dev/null &&                         echo 'Code package downloaded.' && break; sleep 10;...

[tkislan]

@savingoyal Thx for the pointer to shlex .. Honestly I didn't delve into the coda that much, as I hoped that previous PR had it almost done

This is the new full command that it generates, and I was able to run it locally

['/bin/sh', '-c', "set -e && echo 'Setting up task environment.' && python -m pip install click requests boto3 -qqq && mkdir metaflow && cd metaflow && mkdir .metaflow && i=0; while [ $i -le 5 ]; do echo 'Downloading code package.'; python -c 'import sys; import boto3; boto3.client(sys.argv[1]).download_file(sys.argv[2], sys.argv[3], sys.argv[4])' s3 'def3b3-mf-test' 'ParameterFlow/data/24/243633f93d432b2431cfac4518a44f4e9eb68881' job.tar >/dev/null &&                         echo 'Code package downloaded.' && break; sleep 10; i=$((i+1)); done && if [ $i -gt 5 ]; then echo 'Failed to download code package from s3://def3b3-mf-test/ParameterFlow/data/24/243633f93d432b2431cfac4518a44f4e9eb68881 after 6 tries. Exiting...' && exit 1; fi && tar xf job.tar && echo 'Task is starting.' && sleep 10"]

I had to go for the sys.argv solution, as I wasn't able to force the correct quote escaping inside the single quoted string

[tkislan]

@savingoyal is anything still blocking this?

[tkislan]

Closing, as nobody cares ..