avoid merge errors when blocks contain different type #45269
+22
−1
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
example:
wil get
# <class 'pyarrow.lib.Table'>, <class 'pandas.core.frame.DataFrame'>
because in blockaccess, fallback maybe arise, like
["a", "b", np.nan]
cannot cover to arrow will raise exception cause fallback, and then return pandaswhen use this code
will finally cause
# <class 'pyarrow.lib.Table'>, <class 'pandas.core.frame.DataFrame'>
merge in to_pandas method, it will raise exception.In short, I solved the problem that merging two blocks of a dataset will throw an exception when there are two formats of pandas arrow in the blocks.
Related issue number
#45236
Checks
git commit -s
) in this PR.scripts/format.sh
to lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/
under thecorresponding
.rst
file.