Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

_cq_pg_cdc_event_time for long lived processes #16888

Open
1 task
smylbb opened this issue Feb 26, 2024 · 1 comment
Open
1 task

_cq_pg_cdc_event_time for long lived processes #16888

smylbb opened this issue Feb 26, 2024 · 1 comment

Comments

@smylbb
Copy link

smylbb commented Feb 26, 2024

Which problem is this feature request solving?

Currently for long lived syncing processes like postgres CDC mode sync, the _cq_sync_time reflects the time that the process began which causes issues of duplication with destinations such as clickhouse which could have otherwise been resolved with ReplacingMergeTree at merge time.

Describe the solution you'd like

Add a dynamic sync time as a separate column in the cdc mode, something like "_cq_pg_cdc_event_time", which clickhouse will use to determine the most recent record to keep at merge time.

Pull request (optional)

  • I can submit a pull request
@bbernays
Copy link
Collaborator

I think this is a duplicate of #15690

@cq-bot cq-bot removed the needs-area label Feb 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: No status
Development

No branches or pull requests

4 participants