Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Support "full refresh"-type functionality for incremental tables #17344

Open
hermanschaaf opened this issue Mar 26, 2024 · 1 comment
Open
Labels
area/cli needs-rfc Needs a more detailed RFC before implementation

Comments

@hermanschaaf
Copy link
Member

At the moment, incremental tables always read their previous state and sync incrementally. Sometimes, however, it is desirable to perform a full refresh periodically, without fiddling with the cursor in the state store. For example, this could be used to remove deleted rows, or to fetch rows that were somehow added after the cursor moved on.

I'd love to hear more ideas around this. A CLI feature like --full-refresh could be good, or otherwise maybe a config setting. Either one is fine, but I'm leaning towards a CLI option because then you can switch between incremental and full syncs, maybe running one full refresh per week, without needing to use a different config.

When a full refresh is being done, the sync should also perform the delete-stale step if the write mode is overwrite-delete-stale, so that deleted rows can be removed.

@erezrokah erezrokah added area/cli needs-rfc Needs a more detailed RFC before implementation and removed needs-area labels Apr 30, 2024
@erezrokah
Copy link
Contributor

Another thought on this issue - when you do a full refresh you'd still want to persist the cursor (which is a bit different from a non incremental sync).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/cli needs-rfc Needs a more detailed RFC before implementation
Projects
Status: Ready
Development

No branches or pull requests

3 participants