Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incorrect dtype of object of polars into-df when numeric and null values are mixed. #12726

Open
ayax79 opened this issue May 1, 2024 · 0 comments
Labels
🐛 bug Something isn't working dataframe issues related to the dataframe implementation needs-triage An issue that hasn't had any proper look

Comments

@ayax79
Copy link
Contributor

ayax79 commented May 1, 2024

Describe the bug

Discovered by @maxim-uvarov

Converting to a dataframe with null and numeric values results in the dataframe column being of dtype object. Futhermore, when attempting to apply a schema, the column still of dtype object.

Ideally, the Value::Nothing types should be converted to NaN polars values when the rest of the table is numeric. Though, this could be problematic when inferring the schema.

Minimally, an error should be returned when a column cannot be created as the type provided by the schema.

How to reproduce

[[a b]; [6 2] [1 1] [1 4] [2 null]] | polars into-df --schema {a: i64, b: i64} | polars schema

Expected behavior

  1. An error should occur if the column cannot be converted to the specified type in the schema
  2. Convert the null values to NaN if possible.

Screenshots

Screenshot 2024-05-01 at 15 32 36

Configuration

key value
version 0.93.0
major 0
minor 93
patch 0
branch
commit_hash
build_os macos-aarch64
build_target aarch64-apple-darwin
rust_version rustc 1.77.2 (25ef9e3d8 2024-04-09)
rust_channel stable-aarch64-apple-darwin
cargo_version cargo 1.77.2 (e52e36006 2024-03-26)
build_time 2024-05-01 09:22:02 -07:00
build_rust_channel release
allocator mimalloc
features default, sqlite, system-clipboard, trash, which
installed_plugins plist, polars

Additional context

Added to the Polars roadmap backlog

@ayax79 ayax79 added the needs-triage An issue that hasn't had any proper look label May 1, 2024
@sholderbach sholderbach added 🐛 bug Something isn't working dataframe issues related to the dataframe implementation labels May 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🐛 bug Something isn't working dataframe issues related to the dataframe implementation needs-triage An issue that hasn't had any proper look
Projects
None yet
Development

No branches or pull requests

2 participants