You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
At the moment, incremental tables always read their previous state and sync incrementally. Sometimes, however, it is desirable to perform a full refresh periodically, without fiddling with the cursor in the state store. For example, this could be used to remove deleted rows, or to fetch rows that were somehow added after the cursor moved on.
I'd love to hear more ideas around this. A CLI feature like --full-refresh could be good, or otherwise maybe a config setting. Either one is fine, but I'm leaning towards a CLI option because then you can switch between incremental and full syncs, maybe running one full refresh per week, without needing to use a different config.
When a full refresh is being done, the sync should also perform the delete-stale step if the write mode is overwrite-delete-stale, so that deleted rows can be removed.
The text was updated successfully, but these errors were encountered:
Another thought on this issue - when you do a full refresh you'd still want to persist the cursor (which is a bit different from a non incremental sync).
At the moment, incremental tables always read their previous state and sync incrementally. Sometimes, however, it is desirable to perform a full refresh periodically, without fiddling with the cursor in the state store. For example, this could be used to remove deleted rows, or to fetch rows that were somehow added after the cursor moved on.
I'd love to hear more ideas around this. A CLI feature like
--full-refresh
could be good, or otherwise maybe a config setting. Either one is fine, but I'm leaning towards a CLI option because then you can switch between incremental and full syncs, maybe running one full refresh per week, without needing to use a different config.When a full refresh is being done, the sync should also perform the
delete-stale
step if the write mode isoverwrite-delete-stale
, so that deleted rows can be removed.The text was updated successfully, but these errors were encountered: