You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We are trying to use saveAsBigQueryTable to append data to a table where additional fields might be added to the schema. At the moment, we are unable to do so as the schema for the data we want to append to the table has additional fields not present in the table schema.
"errors" : [ {
"message" : "Provided Schema does not match Table table_name. Cannot add fields (field: field_name)",
"reason" : "invalid"
}
SOLUTION
Google offers a parameter SchemaUpdateOptions which allows you to update the schema of a destination table (docs). This parameter can be configured in two ways; ALLOW_FIELD_ADDITION and ALLOW_FIELD_RELAXATION. If we configure it to ALLOW_FIELD_ADDITION then additional nullable fields, that are not already present in the table schema, could be appended using the saveAsBigQueryTable method.
At the moment this is not offered as a configuration option with the scio wrapper. Would it be possible to get this added in?
The text was updated successfully, but these errors were encountered:
arthurenc
changed the title
Could we add SchemaUpdateOption param to BQ WriteParam?
Could we add SchemaUpdateOption param to BQ WriteParam?
Jan 15, 2024
CONTEXT
We are trying to use saveAsBigQueryTable to append data to a table where additional fields might be added to the schema. At the moment, we are unable to do so as the schema for the data we want to append to the table has additional fields not present in the table schema.
SOLUTION
Google offers a parameter
SchemaUpdateOptions
which allows you to update the schema of a destination table (docs). This parameter can be configured in two ways; ALLOW_FIELD_ADDITION and ALLOW_FIELD_RELAXATION. If we configure it to ALLOW_FIELD_ADDITION then additional nullable fields, that are not already present in the table schema, could be appended using the saveAsBigQueryTable method.At the moment this is not offered as a configuration option with the scio wrapper. Would it be possible to get this added in?
The text was updated successfully, but these errors were encountered: