Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add overwrite or overwrite-delete-stale to BigQuery #17538

Closed
1 task done
soren-olympus opened this issue Apr 5, 2024 · 2 comments
Closed
1 task done

feat: Add overwrite or overwrite-delete-stale to BigQuery #17538

soren-olympus opened this issue Apr 5, 2024 · 2 comments

Comments

@soren-olympus
Copy link

Which problem is this feature request solving?

Get incremental syncs in BigQuery destination plugin.

Describe the solution you'd like

Add write or overwrite-delete-stale write modes for BigQuery destination plugin.

Pull request (optional)

  • I can submit a pull request
@hermanschaaf
Copy link
Member

@soren-olympus write and overwrite-delete-stale write modes are not a prerequisite for incremental syncs, and that should work today with append mode already.

I'm saying this partly because I don't know if it's possible to implement overwrite-delete-stale mode in BigQuery - the BigQuery streaming buffer disallows deletes, so delete-stale syncs would have to be spaced at least several hours apart, and may still run into errors or inconsistencies even then. We can do more research, but this is what I remember from my initial attempts to implement this feature.

@soren-olympus
Copy link
Author

Appreciate this note!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
Development

No branches or pull requests

3 participants