Support Schema Discovery/Evolution in Builder/Low-Code Connectors

[NAjustin]

Current Problem

Currently, Builder/Low-Code connectors don't support detecting schema changes once the connector is published—only static/fixed schemas specified in the connector. This leads to several issues:

• Confusion in the community, especially since the Connector Builder documentation doesn't clearly state this limitation (it's only stated in the low-code docs—and probably needs a separate docs bug).
• Non-functional UI, where the Detect and propagate schema changes setting in the connection configuration doesn't actually do anything (which only adds to the confusion noted above based on what I see in Slack and the issues here).
• Less adoption because those who don't know how to build CDK connectors can't support this use case.

Specific Use-Cases

The first is straightforward: Not having to publish a new connector version each time a field is added to an API, which creates a maintenance burden on the developer/user. This is a pain and avoiding it would be a quality-of-life improvement, but it isn't insurmountable for most organizations.

The second case is a much bigger challenge. Working in a vertical-specific (in our case non-profit/NGO) agency and SaaS platform, we end up working with a lot of vertical-specific or niche tools like CRMs, email marketing platforms, and form builders/plugins. Many of these systems represent custom fields or metadata directly to the object being represented, similar to popular platforms like Salesforce or HubSpot (which list custom values directly in the object, not in a dedicated endpoint). Not only does this mean that additions and edits to these field names cause issues with Builder/low-code, it also means that each individual customer using that system can have different fields for the same API object.

The only "solution" is to make a separate version of the connector for every client (and then have to maintain all those separate versions)—or move away from builder/low-code, then requiring someone with the skills and experience needed to re-build those connectors using the CDK.

Proposed Solution:

Discovery and evolution, like most functions supported in Builder/low-code, is highly repetitive and formulaic. Since most people using Builder are allowing it to detect the schema anyway, that same logic would make sense as a default for this feature.

I would argue that this could be enabled by default without breaking anything (maybe just by forcing a change with the next update to set the schema of existing connectors not to evolve by default or forcing a prompt asking the user):

• Discovery would just happen automatically and allow the user to decide if they want to apply those changes
• If there are conflicting features where this isn't possible, those could be checked and the Detect and propagate schema changes option could be disabled in the UI with a note for that connector until it is updated to a version that resolves those issues.

If there's a concern with rolling this out for existing connectors, here's an alternate strategy that could be employed in Builder:

1. Make the schema specified in builder/low-code function simply as a default
2. Add a Builder section and corresponding low-code config (IMHO enabled by default in new connectors) to enable schema detection/evolution
3. Optionally, this section/config could contain options to control when/how detection is run. For example . . .
   • Maybe allow passing a custom discovery handler
   • Maybe the ability to limit which streams it applies to
   • Maybe the ability to limit which evolution options are presented to the end-user in the connection config screen (e.g. suppress automatic evolution if it's likely to cause issues, force the user to manually accept the changes)
   • I could also see the need for an option to limit these fields by type (e.g. whether to expand nested objects or not)

Summary:

Ultimately I think the core issue here is that while there may be some limitations, schema detection and evolution is core to Airbyte's purpose and should not be limited by how a connector is built. It's simply too limiting and too confusing to not support or have to explain this behavior and why it's different than other connectors.

I also firmly believe that adding this functionality will broaden what can be build with Builder/Low-Code, and make more of these able to be contributed to the community connector listing because they no longer only support one organization's needs.