-
Notifications
You must be signed in to change notification settings - Fork 474
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PS-9219: MySQL converts collation of date data type in ibd but data dictionary (trunk) #5295
Merged
venkatesh-prasad-v
merged 2 commits into
percona:release-8.4.0-1
from
venkatesh-prasad-v:PS-9219-trunk
Jun 17, 2024
Merged
PS-9219: MySQL converts collation of date data type in ibd but data dictionary (trunk) #5295
venkatesh-prasad-v
merged 2 commits into
percona:release-8.4.0-1
from
venkatesh-prasad-v:PS-9219-trunk
Jun 17, 2024
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2935de5
to
a2f53c1
Compare
eb0fe0c
to
acfeccc
Compare
dlenev
approved these changes
Jun 6, 2024
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hello Venki!
I think it is OK to push this patch to trunk since there is no differences from 8.0
patch (which I have approved earlier).
…ictionary https://perconadev.atlassian.net/browse/PS-9219 Problem ======= Import tablespace operation fails due to the incorrect table definition in the ibd file if the charset-collation was changed before the backup if the table had a column of temporal type (date, datetime, timestamp). Analysis ======== In MySQL temporal types are always stored and compared using my_charset_latin1 charset. During the execution of ALTER TABLE CONVERT TO CHARACTER SET, MySQL changes the charset and collation stored in data-dictionary and SDI for temporal columns to a different collation_id. In practice, this new collation_id is ignored when such columns are stored/compared. Corresponding Field objects are not updated to use this new collation (Actually, Field objects for temporal types are hardcoded to use my_charset_latin1). This new collation is not visible in I_S and SHOW CREATE TABLE output. It will be ignored by CREATE TABLE LIKE and rewritten by ALTER TABLE since both these statements use info from Field objects of source table to produce Create_field objects describing columns of new table/new version of the table. For example: create table a(dt datetime); $ ./bin/ibd2sdi ./var/mysqld.1/data/test/a.ibd "name": "dt", "collation_id": 8 alter table a CONVERT TO CHARACTER SET utf8mb4 collate utf8mb4_unicode_ci; $ ./bin/ibd2sdi ./var/mysqld.1/data/test/a.ibd "name": "dt", "collation_id": 224 alter table a engine = innodb; $ ./bin/ibd2sdi ./var/mysqld.1/data/test/a.ibd "name": "dt", "collation_id": 8 However, this almost invisible incorrect collation_id causes problems when we try to restore InnoDB table with such a column from the backup created using Percona XtraBackup tool. This tool uses information from DD/SDI to produce .cfg file describing InnoDB table being restored. Later information from this file is used by ALTER TABLE IMPORT TABLESPACE that imports restored table. Particularly, for new temporal types that support fractional seconds collation_id from DD/SDI affects the InnoDB "precise type" describing the column and stored in .cfg. Because of this, incorrect collation_id in DD/SDI for such columns results in incorrect "precise type" in .cfg file. As consequence we get Schema mismatch (Column a precise type mismatch.) error when ALTER TABLE IMPORT TABLESPACE compares the "precise type" from such .cfg file with "precise type" for column in table version being imported into. The latter is based on collation_id which ultimately comes from Field object and always corresponds to my_charset_latin1 for temporal types. Note that this problem do not affect scenarios when we import table with .cfg file that was generated by MySQL's FLUSH TABLE FOR EXPORT command, as the latter doesn't use DD/SDI to calculate "precise type" but gets information from Field object in table being exported instead. Solution ======== This commit changes ALTER TABLE CONVERT TO CHARACTER SET command to not alter the character set for temporal types stored in data-dictionary/SDI. In other words, we now force the server to always use my_charset_latin1 in DD/SDI for temporal types. There will be a separate fix to Percona XtraBackup tool which will change code generating .cfg to ignore collation_id stored in DD/SDI for temporal columns like it is already done for some other types with fixed collation_id.
acfeccc
to
f924c18
Compare
…ictionary https://perconadev.atlassian.net/browse/PS-9219 Merge remote-tracking branch 'venki/PS-9219-8.0' into PS-9219-trunk Problem ======= Import tablespace operation fails due to the incorrect table definition in the ibd file if the charset-collation was changed before the backup if the table had a column of temporal type (date, datetime, timestamp). Analysis ======== In MySQL temporal types are always stored and compared using my_charset_latin1 charset. During the execution of ALTER TABLE CONVERT TO CHARACTER SET, MySQL changes the charset and collation stored in data-dictionary and SDI for temporal columns to a different collation_id. In practice, this new collation_id is ignored when such columns are stored/compared. Corresponding Field objects are not updated to use this new collation (Actually, Field objects for temporal types are hardcoded to use my_charset_latin1). This new collation is not visible in I_S and SHOW CREATE TABLE output. It will be ignored by CREATE TABLE LIKE and rewritten by ALTER TABLE since both these statements use info from Field objects of source table to produce Create_field objects describing columns of new table/new version of the table. For example: create table a(dt datetime); $ ./bin/ibd2sdi ./var/mysqld.1/data/test/a.ibd "name": "dt", "collation_id": 8 alter table a CONVERT TO CHARACTER SET utf8mb4 collate utf8mb4_unicode_ci; $ ./bin/ibd2sdi ./var/mysqld.1/data/test/a.ibd "name": "dt", "collation_id": 224 alter table a engine = innodb; $ ./bin/ibd2sdi ./var/mysqld.1/data/test/a.ibd "name": "dt", "collation_id": 8 However, this almost invisible incorrect collation_id causes problems when we try to restore InnoDB table with such a column from the backup created using Percona XtraBackup tool. This tool uses information from DD/SDI to produce .cfg file describing InnoDB table being restored. Later information from this file is used by ALTER TABLE IMPORT TABLESPACE that imports restored table. Particularly, for new temporal types that support fractional seconds collation_id from DD/SDI affects the InnoDB "precise type" describing the column and stored in .cfg. Because of this, incorrect collation_id in DD/SDI for such columns results in incorrect "precise type" in .cfg file. As consequence we get Schema mismatch (Column a precise type mismatch.) error when ALTER TABLE IMPORT TABLESPACE compares the "precise type" from such .cfg file with "precise type" for column in table version being imported into. The latter is based on collation_id which ultimately comes from Field object and always corresponds to my_charset_latin1 for temporal types. Note that this problem do not affect scenarios when we import table with .cfg file that was generated by MySQL's FLUSH TABLE FOR EXPORT command, as the latter doesn't use DD/SDI to calculate "precise type" but gets information from Field object in table being exported instead. Solution ======== This commit changes ALTER TABLE CONVERT TO CHARACTER SET command to not alter the character set for temporal types stored in data-dictionary/SDI. In other words, we now force the server to always use my_charset_latin1 in DD/SDI for temporal types. There will be a separate fix to Percona XtraBackup tool which will change code generating .cfg to ignore collation_id stored in DD/SDI for temporal columns like it is already done for some other types with fixed collation_id.
f924c18
to
74568e2
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PS-9219: MySQL converts collation of date data type in ibd but data dictionary
https://perconadev.atlassian.net/browse/PS-9219
Problem
Import tablespace operation fails due to the incorrect table definition
in the ibd file if the charset-collation was changed before the backup
if the table had a column of temporal type (date, datetime, timestamp).
Analysis
In MySQL temporal types are always stored and compared using
my_charset_latin1 charset.
During the execution of ALTER TABLE CONVERT TO CHARACTER SET, MySQL
changes the charset and collation stored in data-dictionary and SDI
for temporal columns to a different collation_id.
In practice, this new collation_id is ignored when such columns
are stored/compared. Corresponding Field objects are not updated to
use this new collation (Actually, Field objects for temporal types are
hardcoded to use my_charset_latin1).
This new collation is not visible in I_S and SHOW CREATE TABLE output.
It will be ignored by CREATE TABLE LIKE and rewritten by ALTER TABLE
since both these statements use info from Field objects of source table
to produce Create_field objects describing columns of new table/new
version of the table.
For example:
create table a(dt datetime);
$ ./bin/ibd2sdi ./var/mysqld.1/data/test/a.ibd
"name": "dt",
"collation_id": 8
alter table a CONVERT TO CHARACTER SET utf8mb4 collate utf8mb4_unicode_ci;
$ ./bin/ibd2sdi ./var/mysqld.1/data/test/a.ibd
"name": "dt",
"collation_id": 224
alter table a engine = innodb;
$ ./bin/ibd2sdi ./var/mysqld.1/data/test/a.ibd
"name": "dt",
"collation_id": 8
However, this almost invisible incorrect collation_id causes problems when
we try to restore InnoDB table with such a column from the backup created
using Percona eXtraBackup tool.
This tool uses information from DD/SDI to produce .cfg file describing
InnoDB table being restored. Later information from this file is used
by ALTER TABLE IMPORT TABLESPACE that imports restored table.
Particularly, for new temporal types that support fractional seconds
collation_id from DD/SDI affects the InnoDB "precise type" describing
the column and stored in .cfg. Because of this, incorrect collation_id
in DD/SDI for such columns results in incorrect "precise type" in .cfg
file.
As consequence we get Schema mismatch (Column a precise type mismatch.)
error when ALTER TABLE IMPORT TABLESPACE compares the "precise type"
from such .cfg file with "precise type" for column in table version
being imported into. The latter is based on collation_id which ultimately
comes from Field object and always corresponds to my_charset_latin1 for
temporal types.
Note that this problem do not affect scenarios when we import table
with .cfg file that was generated by MySQL's FLUSH TABLE FOR EXPORT
command, as the latter doesn't use DD/SDI to calculate "precise type"
but gets information from Field object in table being exported instead.
Solution
This commit changes ALTER TABLE CONVERT TO CHARACTER SET command to
not alter the character set for temporal types stored in data-dictionary/SDI.
In other words, we now force the server to always use my_charset_latin1
in DD/SDI for temporal types.
There will be a separate fix to Percona eXtraBackup tool which will
change code generating .cfg to ignore collation_id stored in DD/SDI for
temporal columns like it is already done for some other types with fixed
collation_id.
PR for 8.0: #5294
Testing Done
Jenkins: https://ps80.cd.percona.com/view/8.0%20parallel%20MTR/job/percona-server-8.x-param-parallel-mtr/16/console