Destination Clickhouse - Add Typing and Deduping

[jbfbell]

We removed (PR )"normalization" from Destination Clickhouse. Normalization was a process which would extract data from the raw json format and cast the data into tables with the appropriate types. For other destinations, we implemented Typing and Deduping to replace normalization and we will bring typing and deduplication to Clickhouse if there is sufficient interest

The purpose of this discussion is to gauge interest in bringing this functionality back to Clickhouse.

Without typing and deduping, you would need to integrate another tool, such as dbt to accomplish the same result. While integrating another tool gives you greater customization over the data format, it comes with additional operational overhead.

Should Airbyte bring Typing and Deduping to Clickhouse?

[asimumba]

How can I revert to pre 1.0.0 upgrade of the Clickhouse destination?

[kvosha]

Definitely we need normalization for clickhouse! Is there any way to stay on old version of clickhouse connection until normalization will be implemented?

[alberk8]

Yes, bring Typing and Deduping ClickHouse

[Riya-gupta-8]

Yes . You can degrade to v0.2.5 version for clickhouse

[nailele]

airbyte should bring typing and deduping for clickhouse 🥹

[MaxMusing]

Strong +1 on this, we would benefit greatly from typing and deduping on ClickHouse as a destination.

[asimumba]

In support to have this feature implemented.

[machariamuguku]

FYI: The airbyte team published the connector docker images on dockerhub. You can 'revert' to an older connector version (with normalization for example) by uploading it's older image as a custom connector. See clickhouse connector images here cc @asimumba @kvosha

[asimumba]

Hi, I actually resolved this already. Thanks.

[AlexisSerneels]

I've spent hours debugging why the final tables were not created before finding this discussion.
We definitely need this feature ;)

[vietpm]

I can't understand why this feature is removed. Please bring this back in destination v2.

For anyone wants this feature, you can go back to older version 0.2.5 of clickhouse destination.
1, Settings -> Destinations -> Clickhouse. Change version to 0.2.5 and click "Change".
2, Enter your connection, choose Transformation, change to "Normalized tabular data", click "Save".
3, (optional) If you're using Incremental sync mode, "Incremental | Append + Deduped" is available now so you may be want to change.
4, Reset data, resync from scratch.

Note that you'll be switched back to destination v1.

[edwardysun]

+1 need this

[cedstrom]

Holy hell, please bring normalization back +1000.

[luishdosreis]

Pretty please bring normalisation back :) 🍰

[zowen-ch]

We need the normalization for our workflows. Please keep supporting this in Clickhouse!

[georgeemr]

Please bring it back!
Any reason why it was removed?

[jesperbagge]

A big heartfelt YES! Please bring Typing and Deduping to ClickHouse!

[Lawchak-Fadhil]

Please bring it back, I've spent countless hours trying to figure out what is wrong with my setup! We really need this for ClickHouse.

[alekseyp]

Just for the history - as of now - this is the number one issue by popularity in https://github.com/airbytehq/airbyte/discussions/categories/ideas-and-features