You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We created an empty delta table with pyspark so that it can be appended with awswrangler's to_deltalake method with lambdas running in parallel. But when we look at cloudwatch there is no error, and some data is written and others are not, without there really being any error.
camposvinicius
changed the title
Data is not being written with append using awswrangler delta with multiple lambdas running in parallel
Data is not being totally written with append using awswrangler delta with multiple lambdas running in parallel
Apr 11, 2024
camposvinicius
changed the title
Data is not being totally written with append using awswrangler delta with multiple lambdas running in parallel
Data is not being totally written with append using awswrangler to_deltalake with multiple lambdas running in parallel
Apr 11, 2024
When s3_allow_unsafe_rename is set to True, consistency will not be enforced between different simultaneous write operations. In order to make use of the locking mechanism, a DynamoDB table needs to be created and passed using the lock_dynamodb_table argument. More details can be found in the to_deltalake documentation.
Describe the bug
We created an empty delta table with pyspark so that it can be appended with awswrangler's to_deltalake method with lambdas running in parallel. But when we look at cloudwatch there is no error, and some data is written and others are not, without there really being any error.
How to Reproduce
Expected behavior
No response
Your project
No response
Screenshots
No response
OS
Linux
Python version
3.8
AWS SDK for pandas version
3.6.0
Additional context
No response
The text was updated successfully, but these errors were encountered: