[SVCS-826] S3 Compatablilty #351
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cslzchen
requested changes
Jun 26, 2018
|
Yep that's true, its related to s3-compat. Replacing the calls with client library is because the signature stuff in boto2 is delusional. the signature stuff in boto3 works as intended. It is going to be synchronous as written now, but there's two options - |
NyanHelsing
changed the title
NyanHelsing
changed the title
boto2 causes lots of problems, and trying to use both is trying. Use boto3 for all s3 operations.
Upgrades boto2 to boto3. This required some implementations of operations to change because of some inflexibility inside of boto - It likes to do it the boto way, but using some of the internals, to generate signed urls for example, is difficult to call in isolation. In the interest of getting it implemented sooner than later, we use the boto way of doing some things rather than making our own requests manually.
Removes some code that was written during trying to get the provider working. This code was exploratory and does not work.
Finishes updates to the sign_url functions to allow use with boto3's `generate_presigned_url`
NyanHelsing
commented
Dec 17, 2018
waterbutler/providers/s3/provider.py
Outdated
|
|
||
| async def revisions(self, path, **kwargs): | ||
| """Get past versions of the requested key | ||
|
|
||
| :param str path: The path to a key | ||
| :rtype list: | ||
| """ | ||
| await self._check_region() | ||
|
|
||
| query_params = {'prefix': path.path, 'delimiter': '/', 'versions': ''} | ||
| url = functools.partial(self.bucket.generate_url, settings.TEMP_URL_SECS, 'GET', query_parameters=query_params) |
There was a problem hiding this comment.
This needs to be fixed
| @@ -1,6 +1,7 @@ | |||
| agent==0.1.2 | |||
| aiohttp==0.18.2 | |||
| git+https://github.com/felliott/boto.git@feature/gen-url-query-params-6#egg=boto | |||
There was a problem hiding this comment.
Need to remove old boto
| @@ -187,12 +188,16 @@ def size(self): | |||
| def at_eof(self): | |||
| return self.inner.at_eof() | |||
|
|
|||
| def tell(self): | |||
| return self.offset | |||
|
|
|||
There was a problem hiding this comment.
Not strictly necessary, but was needed for a transient implementation. Still technically correct and simply makes our streams more stream-y.
| @@ -105,15 +105,17 @@ def _async_retry(func): | |||
| )) | |||
|
|
|||
|
|
|||
| def normalize_datetime(date_string): | |||
| if date_string is None: | |||
| def normalize_datetime(date): | |||
There was a problem hiding this comment.
not always a string, so saying date_str here is a little misleading.
|
|
||
| import xmltodict | ||
|
|
||
| import xml.sax.saxutils | ||
|
|
||
| #import aioboto3 | ||
| from boto import config as boto_config | ||
| from boto.compat import BytesIO # type: ignore | ||
| from boto.utils import compute_md5 | ||
| from boto.auth import get_auth_handler | ||
| from boto.s3.connection import S3Connection | ||
| from boto.s3.connection import OrdinaryCallingFormat |
There was a problem hiding this comment.
Need to remove old boto
| :param HttpMethod: The http method to use on the generated url. By | ||
| default, the http method is whatever is used in the method's model. | ||
| :returns: The presigned url | ||
| """ |
There was a problem hiding this comment.
Add another comment here explaining the patch better
| # to supply the client with a string of the region name. First we | ||
| # make a temporary client in order to get that string. We put the | ||
| # client creation inside a lmabda so its easier to call twice. | ||
| # This must be a lambda an *not* a partial, because we want the |
There was a problem hiding this comment.
This isn't really relevant here.
| # if kwargs.get('displayName'): | ||
| # get_kwargs['ResponseContentDisposition'] = 'attachment; filename*=UTF-8\'\'{}'.format(parse.quote(kwargs['displayName'])) | ||
| # else: | ||
| # get_kwargs['ResponseContentDisposition'] = 'attachment' |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose
This PR enables users to connect any s3 compatible storage provider to their OSF account using the s3 addon.
Changes
Rewrites all s3 provider operations to use boto3 (boto2 has issues signing urls that will work properly with minio)
QA Notes
Ensure that users can connect s3-compatible providers
#Documentation
Side Effects
Users modifying an account will necessarily modify that account for any other user that also has connected this account. (This is already a behaviour experienced using the 'connect or reauthorize dialog)
Ticket
https://openscience.atlassian.net/browse/SVCS-826