the draft pr to demonstrate the performance of polars lazyframes on 20240516
#1404
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
numd run dataframes.md --prepend-intermid "$env.config.footer_mode = 'never'; ∙ $env.config.table.header_on_separator = false; $env.config.table.mode = 'default'; $env.config.table.abbreviated_row_count = 10000"
This reverts commit bdaf8c4.
fdncred
reviewed
May 16, 2024
book/dataframes.md
Outdated
| ``` | ||
|
|
||
| This time it took us 0.6 seconds. Not bad at all. | ||
| This time it took us 0.00053 seconds. Not bad at all. |
There was a problem hiding this comment.
where did this time come from? above it says 11 sec, which I don't understand either. I'm not sure how dataframes would be slower than nushell opening the same file.
There was a problem hiding this comment.
11 seconds is the execution time inside numd script. I executed it manually now and it shows 5 seconds which is also above expectations.
0.00053 seconds is the time it took to create a lazyframe plan. I will fix it in the next commit.
|
This is my timings from my Windows desktop that is not super fast. I didn't do the python timings. ❯ timeit {open Data7602DescendingYearOrder.csv}
5sec 244ms 662µs 400ns
❯ timeit {polars open Data7602DescendingYearOrder.csv | polars collect; null}
239ms 526µs 800ns
❯ timeit {
open 'Data7602DescendingYearOrder.csv'
| group-by year --to-table
| update items {|i|
$i.items.geo_count
| math sum
}
}
10sec 906ms 680µs 300ns
❯ cat load.nu
let df = polars open Data7602DescendingYearOrder.csv
let res = $df | polars group-by year | polars agg (polars col geo_count | polars sum)
$res | polars collect
❯ timeit {source load.nu}
155ms 678µs 500ns |
|
Just FYI - Regular nushell tables can emulate the polars output too. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
this pr is not intended to be merged.