Remove tempStorageDirectory and rely on task dir instead #16416
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| } | ||
|
|
||
| @VisibleForTesting | ||
| protected RetryableS3OutputStream( | ||
| S3OutputConfig config, | ||
| ServerSideEncryptingAmazonS3 s3, | ||
| String s3Key, | ||
| boolean chunkValidation | ||
| boolean chunkValidation, |
Check notice
Code scanning / CodeQL
Useless parameter Note
| } | ||
|
|
||
| @Override | ||
| public StorageConnector get() | ||
| { | ||
| return new AzureStorageConnector(this, azureStorage); | ||
| final File tempDir = injector.getInstance(Key.get(File.class, Names.named(DataSourceTaskIdHolder.TMP_DIR_BINDING))); |
There was a problem hiding this comment.
Thanks so much for this change @adarshsanjeev 👍
Although, it might be worth checking if you can directly JacksonInject the File object - one way to try it could be to do :
@JacksonInject
@Named(DataSourceTaskIdHolder.TMP_DIR_BINDING)
File tempDir;
Also, could consider adding the JacksonInject in the constructor itself to keep all the relevant things there.
There was a problem hiding this comment.
Lastly, could be worth having the temp dir instance setup for other processes as well, and also for indexers
There was a problem hiding this comment.
I tried the above initially, but injecting the tempDir as a variable fails (maybe since it is a jackson inject, and that binding has not been added?).
There was a problem hiding this comment.
does it fail for processes which are not peons? if that's the case you can try also marking this Nullable
also, if you have the error available, please feel free to paste it here
There was a problem hiding this comment.
The above code fails for peons with the error
Guice configuration errors: 1) No implementation for java.io.File annotated with @com.google.inject.name.Named(value="druidTempDirectory") was bound. while locating java.io.File annotated with @com.google.inject.name.Named(value="druidTempDirectory") 1 error
...re-extensions/src/main/java/org/apache/druid/storage/azure/output/AzureStorageConnector.java
Outdated
Show resolved
Hide resolved
github-actions
bot
added
Area - Batch Ingestion
Area - MSQ
Removes the temporary directory used by durable storage and export, from a user configured value. This PR instead reuses the temporary storage configured for the task.
Pending:
This PR has: