source-zendesk-support: ingestion fails with Could not determine airbyte type from JSON schema

[jasonhallup]

Connector Name

source-zendesk-support

Connector Version

2.4.1

What step the error happened?

During the sync

Relevant information

When running the connector to sync data, the following error was returned:

Connection check succeeded for source-zendesk-support.
Started source-zendesk-support read operation at 11:54:30...
Could not determine airbyte type from JSON schema: {'type': ['null', 'object', 'string'], 'properties': {'id': {'type': ['null', 'integer']}, 'assignee_id': {'type':
['null', 'integer']}, 'group_id': {'type': ['null', 'integer']}, 'reason_id': {'type': ['null', 'integer']}, 'requester_id': {'type': ['null', 'integer']},
'ticket_id': {'type': ['null', 'integer']}, 'updated_at': {'type': ['null', 'string'], 'format': 'date-time'}, 'created_at': {'type': ['null', 'string'], 'format':
'date-time'}, 'url': {'type': ['null', 'string']}, 'score': {'type': ['null', 'string']}, 'reason': {'type': ['null', 'string']}, 'comment': {'type': ['null',
'string']}}}
Completed source-zendesk-support read operation at 11:54:33.

Relevant log output

No response

Contribute

• Yes, I want to contribute

[marcosmarxm]

Can you test using latest version of Zendesk connector as it fixed some schema issue.
If it doesn't fix the issue, can you share the complete sync log and what streams you're trying to sync?

[jasonhallup]

Newbie question, but I can't easily find the answer online. How do I update a specific connector in pyAirbyte? Is there a line I can call in my script to make sure it's always pulling down the latest version?

…

On Tue, Apr 30, 2024 at 3:41 PM Marcos Marx ***@***.***> wrote: Can you test using latest version of Zendesk connector as it fixed some schema issue. If it doesn't fix the issue, can you share the complete sync log and what streams you're trying to sync? — Reply to this email directly, view it on GitHub <#212>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A2FJFPWEWGVOW2RXU5TWZRDY77XX3AVCNFSM6AAAAABGZGIDJ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBWHA4DCMBTGI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[aaronsteers]

Newbie question, but I can't easily find the answer online. How do I update a specific connector in pyAirbyte? Is there a line I can call in my script to make sure it's always pulling down the latest version?

Hi, @jasonhallup!

We don't have a way to check as of today in PyAirbyte. To answer your direct question though, PyAirbyte will automatically attempt to install the latest version if you do not provide a different version perference when calling get_source().

You might be able to get this with source.executor._get_installed_version(). Started a PR here to make this a supported operation:

• Feat: Add Source.connector_version property #211

Short of the above, a fresh install of PyAirbyte should do a fresh install of the source connector, using the latest version. You can check for (and delete) a folder called .venv-source-zendesk-support if you feel you are not getting the latest version.

cc @marcosmarxm

[jasonhallup]

I just tried running my script again, and source.executor._get_installed_version() returns 2.4.1, and the script does not read any data from the tickets stream: Script: result = source.check() print(source.executor._get_installed_version()) source.select_streams(["tickets"]) read_result: ab.ReadResult = source.read() tickets_df = read_result["tickets"].to_pandas() Connection check succeeded for `source-zendesk-support`. 2.4.1 Started `source-zendesk-support` read operation at *21:09:11*... Completed `source-zendesk-support` read operation at *21:09:13*. *Read Progress* Started reading at 01:09:12. Read *0* records over *1 seconds* (0.0 records / second). Finished reading at 01:09:13. Started finalizing streams at 01:09:13. Finalized *0* batches over 0 seconds. On Tue, Apr 30, 2024 at 5:46 PM aaronsteers ***@***.***> wrote:

…

Newbie question, but I can't easily find the answer online. How do I update a specific connector in pyAirbyte? Is there a line I can call in my script to make sure it's always pulling down the latest version? … <#m_-3717267742669524783_> On Tue, Apr 30, 2024 at 3:41 PM Marcos Marx *@*.*> wrote: Can you test using latest version of Zendesk connector as it fixed some schema issue. If it doesn't fix the issue, can you share the complete sync log and what streams you're trying to sync? — Reply to this email directly, view it on GitHub <#37577 (comment) <#212>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/A2FJFPWEWGVOW2RXU5TWZRDY77XX3AVCNFSM6AAAAABGZGIDJ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBWHA4DCMBTGI <https://github.com/notifications/unsubscribe-auth/A2FJFPWEWGVOW2RXU5TWZRDY77XX3AVCNFSM6AAAAABGZGIDJ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBWHA4DCMBTGI> . You are receiving this because you authored the thread.Message ID: @.*> Hi, @jasonhallup <https://github.com/jasonhallup>! We don't have a way to check as of today in PyAirbyte. To answer your direct question though, PyAirbyte will automatically attempt to install the latest version if you do not provide a different version perference when calling get_source(). You *might* be able to get this with source.executor._get_installed_version(). Started a PR here to make this a supported operation: - airbytehq/PyAirbyte#211 <#211> cc @marcosmarxm <https://github.com/marcosmarxm> — Reply to this email directly, view it on GitHub <#212>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A2FJFPXPO36R5DKHZ7U4V7LZAAGKLAVCNFSM6AAAAABGZGIDJ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBXGQ2DKNBWHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[aaronsteers]

(Migrated this to PyAirbyte repo, since it seems like it may be PyAirbyte-specific behavior.)

@jasonhallup - On re-reading the issue description, I'm noticing that the error message may actually be a warning. And the subsequent execution which generated zero rows may be due to the fact that sync operations in PyAirbyte are incremental-by-default.

What do you get if you re-run your existing code and then try something like:

display(cache["tickets"].to_pandas())

if in Jupyter/Colab, or else

print(len(cache["tickets"]))

if in terminal or native python?

---

Logged this to improve messaging in general:

• 🐛 Bug: Warnings looks like errors: Could not determine airbyte type from JSON schema #213

[jasonhallup]

Running this is showing 0 as the size/length of the cache: Connection check succeeded for `source-zendesk-support`. 2.4.1 Started `source-zendesk-support` read operation at *11:26:48*... Completed `source-zendesk-support` read operation at *11:26:51*. 0 result = source.check() print(source.executor._get_installed_version()) source.select_streams(["tickets"]) read_result: ab.ReadResult = source.read() print(len(read_result.cache["tickets"]))

…

On Tue, Apr 30, 2024 at 10:29 PM aaronsteers ***@***.***> wrote: *(Migrated this to PyAirbyte repo, since it seems like it may be PyAirbyte-specific behavior.)* @jasonhallup <https://github.com/jasonhallup> - On re-reading the issue description, I'm noticing that the error message may actually be a warning. And the subsequent execution which generated zero rows may be due to the fact that sync operations in PyAirbyte are incremental-by-default. What do you get if you re-run your existing code and then try something like: display(cache["tickets"].to_pandas()) if in Jupyter/Colab, or else print(len(cache["tickets"])) if in terminal or native python? — Reply to this email directly, view it on GitHub <#212 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A2FJFPXUURYY3RLKKPANQFDZABHO3AVCNFSM6AAAAABHBKSBQSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBXHA4DKMZTGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[aaronsteers]

Thanks for confirming. Is it possible that the dataset is empty or that the tables were inadvertently deleted/truncated?

Just to be sure, can you tell me what happens if you run with the force_full_refresh flag:

source.select_streams(["tickets"])
read_result: ab.ReadResult = source.read(force_full_refresh=True)

print(len(read_result.cache["tickets"]))

[jasonhallup]

I'll have to check on that. We use zendesk support for incident management, and I know we have lots of tickets in the system, but even with the full refresh option I'm not getting anything. I'll check with our zendesk team to see if they have any idea's what's up. Thanks for all of your assistance! I will report back what I find.

…

On Wed, May 1, 2024 at 12:53 PM aaronsteers ***@***.***> wrote: Thanks for confirming. Is it possible that the dataset is empty or that the tables were inadvertently deleted/truncated? Just to be sure, can you tell me what happens if you run with the force_full_refresh flag: source.select_streams(["tickets"]) read_result: ab.ReadResult = source.read(force_full_refresh=True) print(len(read_result.cache["tickets"])) — Reply to this email directly, view it on GitHub <#212 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A2FJFPRQDB5J7F6DGT7NYNLZAEM2DAVCNFSM6AAAAABHBKSBQSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBYG42TSNJYGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>