HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags #6789
+899
−23
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
note the commented out bit where we considered adding options like "hive" or "spark". @HarshitGupta11 and I discussed this; for now lets go with a list of options and "*" |
virajjasani
reviewed
May 2, 2024
Comment on lines
151
to
154
| /* case "hive": | ||
| case "impala": | ||
| case "spark": | ||
| case "distcp": |
There was a problem hiding this comment.
Should we not let downstreamers decide what flags they want (after extensive testing)? And across different releases, they might need different flags to be turned on (in case of any regression)?
We can just recommend the flags (as already commented out here) but not set the flags for them. Thoughts?
There was a problem hiding this comment.
harshit and I were discussing this. i think it's best to have that option list, as app settings could be too brittle to changes
Comment on lines
76
to
78
| public boolean isDelete() { | ||
| return delete; | ||
| } |
There was a problem hiding this comment.
This one also we want to tackle as separate task (after HADOOP-19072), correct?
There was a problem hiding this comment.
yes. harshit did an experiment where he turned off all attempts at creating parent dirs after delete. fairly brittle, i think
|
I have a better design for this. changign this to draft. Proposed: we have a this makes it trivial to reuse/process. the implementation would be outside the actual Configuration class to make it easy for AbfsConfiguration to use too |
steveloughran
force-pushed
the
s3/HADOOP-19161-performance-flags
branch
from
feb384f
to
28cd50a
Compare
|
steveloughran
force-pushed
the
s3/HADOOP-19161-performance-flags
branch
from
28cd50a
to
f3571ac
Compare
|
💔 -1 overall
This message was automatically generated. |
…performance flags * A new FlagSet class in hadoop common enables this * and Configuration.getEnumSet() supports getting a set of enum values. * served up via StoreContext. Not sure about the merits of that I think it is needed so it gets down to all AbstractStoreOperation instances, * create performance is wired up. * tests which configure fs.s3a.create.performance clear fs.s3a.performance.flags in test setup. Change-Id: I52e48d19c624e7c18f22b3130943ffe72fac501f
steveloughran
force-pushed
the
s3/HADOOP-19161-performance-flags
branch
from
a564aa4
to
82974c4
Compare
|
💔 -1 overall
This message was automatically generated. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
HADOOP-19161
Initial design
For testing we need to make sure ths is unset from all cost tests.
relates to #6543; the logic to set up that operation is here...that PR would
just be the implementation.
Same for a delete optimisation where we'd skip parent dir probe.
rename could do the same for its source dir too.
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?