Features/1411 implement sketched percentile #1420

mrfh92 · 2024-04-03T13:48:26Z

This solves #1411 by implementing a sketched version of percentile / median and a corresponding option for the RobustScaler in the preprocessing module.

All new features integrate into the current functionality without changes, due to choice of the current non-sketched version as default.

Performance comparison
on 14 MPI-processes (Workstation, CPU) on behalf of the "asteroids data set" (1317275 data points, 9 features), times and deviation measured over10 runs

	avg time	inaccuracy*
median main	~12.3s
median sketched (10% of data)	0.247s	0.30%
median sketched (1% of data)	0.01212s	1.3%
median sketched (0.1% of data)	0.00312s	2.7%

*relative deviation of the sketched median vs true median per feature, averaged over 10 runs; only relative deviation for the feature with maximal deviation is reported

github-actions · 2024-04-03T13:53:51Z

Thank you for the PR!

…ed test for this

github-actions · 2024-04-03T14:21:44Z

Thank you for the PR!

codecov · 2024-04-03T14:22:04Z

Codecov Report

Attention: Patch coverage is 91.66667% with 2 lines in your changes missing coverage. Please review.

Project coverage is 91.86%. Comparing base (bc5ec47) to head (d9723e9).

Files	Patch %	Lines
heat/core/statistics.py	90.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1420      +/-   ##
==========================================
- Coverage   91.87%   91.86%   -0.02%     
==========================================
  Files          80       80              
  Lines       11822    11840      +18     
==========================================
+ Hits        10862    10877      +15     
- Misses        960      963       +3

Flag	Coverage Δ
unit	`91.86% <91.66%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

github-actions · 2024-04-04T08:13:14Z

Thank you for the PR!

github-actions · 2024-04-04T08:14:36Z

Thank you for the PR!

mtar

Some minor comments on documentation

heat/core/statistics.py

heat/core/tests/test_statistics.py

followed mtar's suggestion Co-authored-by: Michael Tarnawa <[email protected]>

updated docstring according to review

for more information, see https://pre-commit.ci

github-actions · 2024-04-05T07:47:00Z

Thank you for the PR!

for more information, see https://pre-commit.ci

github-actions · 2024-04-05T07:51:04Z

Thank you for the PR!

github-actions · 2024-04-05T07:54:11Z

Thank you for the PR!

updated docstring according to review

for more information, see https://pre-commit.ci

updated docstring according to review

for more information, see https://pre-commit.ci

github-actions · 2024-04-05T07:59:21Z

Thank you for the PR!

github-actions · 2024-04-05T08:03:08Z

Thank you for the PR!

ClaudiaComito

This is a great feature to have, thanks a lot @mrfh92 ! A few comments below...

heat/core/statistics.py

github-actions · 2024-04-23T11:07:08Z

Thank you for the PR!

github-actions · 2024-05-31T14:26:27Z

Thank you for the PR!

github-actions · 2024-06-05T17:20:56Z

Thank you for the PR!

github-actions · 2024-06-17T15:56:44Z

Thank you for the PR!

github-actions · 2024-06-17T16:42:57Z

Thank you for the PR!

github-actions · 2024-06-20T13:13:27Z

Thank you for the PR!

github-actions · 2024-06-21T03:56:44Z

Thank you for the PR!

Hoppe added 3 commits April 3, 2024 14:33

added sketched version of percentile computation

60fb1de

added tests for sketched percentile

fadbcc0

added sketching to median

cdea1d0

added support for sketched median/percentile to preprocessing and add…

1db9e3d

…ed test for this

Hoppe and others added 2 commits April 4, 2024 10:08

removed extensive checks from auxiliary function _create_sketch

c36b514

Merge branch 'main' into features/1411-Implement_sketched_percentile

32da442

mrfh92 requested review from ClaudiaComito and mtar April 4, 2024 08:45

mrfh92 added the PR talk label Apr 4, 2024

mrfh92 mentioned this pull request Apr 4, 2024

Speed up ht.percentile #1389

Open

mrfh92 added the statistics label Apr 4, 2024

mtar reviewed Apr 4, 2024

View reviewed changes

heat/core/statistics.py Outdated Show resolved Hide resolved

heat/core/statistics.py Outdated Show resolved Hide resolved

heat/core/statistics.py Outdated Show resolved Hide resolved

heat/core/tests/test_statistics.py Outdated Show resolved Hide resolved

mrfh92 and others added 3 commits April 5, 2024 09:40

Update heat/core/tests/test_statistics.py

18506b7

followed mtar's suggestion Co-authored-by: Michael Tarnawa <[email protected]>

Update statistics.py

391db33

updated docstring according to review

[pre-commit.ci] auto fixes from pre-commit.com hooks

efc3293

for more information, see https://pre-commit.ci

mrfh92 and others added 2 commits April 5, 2024 09:48

Update preprocessing.py

f7f2104

[pre-commit.ci] auto fixes from pre-commit.com hooks

dde023f

for more information, see https://pre-commit.ci

mrfh92 and others added 4 commits April 5, 2024 09:54

Update statistics.py

659c7a1

updated docstring according to review

[pre-commit.ci] auto fixes from pre-commit.com hooks

a1aef0b

for more information, see https://pre-commit.ci

Update preprocessing.py

3d0b9bb

updated docstring according to review

[pre-commit.ci] auto fixes from pre-commit.com hooks

342d0f5

for more information, see https://pre-commit.ci

mrfh92 added the Review me label Apr 8, 2024

ClaudiaComito requested changes Apr 15, 2024

View reviewed changes

mrfh92 removed Review me PR talk labels Apr 22, 2024

Merge branch 'main' into features/1411-Implement_sketched_percentile

3195de6

Merge branch 'main' into features/1411-Implement_sketched_percentile

fd0c4a9

Merge branch 'main' into features/1411-Implement_sketched_percentile

bbc452e

ClaudiaComito mentioned this pull request Jun 11, 2024

Support multiple axes for ht.percentile #1510

Open

4 tasks

Hoppe added 4 commits June 13, 2024 12:35

some of review requests addressed

05a6da7

took review into account

a89be1c

addressed all change requests

872f9de

added fix for single process setting

b11f026

Merge branch 'main' into features/1411-Implement_sketched_percentile

cf72321

mrfh92 requested a review from ClaudiaComito June 19, 2024 08:38

Merge branch 'main' into features/1411-Implement_sketched_percentile

7822c87

Merge branch 'main' into features/1411-Implement_sketched_percentile

d9723e9

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Features/1411 implement sketched percentile #1420

Features/1411 implement sketched percentile #1420

mrfh92 commented Apr 3, 2024 •

edited

github-actions bot commented Apr 3, 2024

github-actions bot commented Apr 3, 2024

codecov bot commented Apr 3, 2024 •

edited

github-actions bot commented Apr 4, 2024

github-actions bot commented Apr 4, 2024

mtar left a comment

github-actions bot commented Apr 5, 2024

github-actions bot commented Apr 5, 2024

github-actions bot commented Apr 5, 2024

github-actions bot commented Apr 5, 2024

github-actions bot commented Apr 5, 2024

ClaudiaComito left a comment

github-actions bot commented Apr 23, 2024

github-actions bot commented May 31, 2024

github-actions bot commented Jun 5, 2024

github-actions bot commented Jun 17, 2024

github-actions bot commented Jun 17, 2024

github-actions bot commented Jun 20, 2024

github-actions bot commented Jun 21, 2024

Features/1411 implement sketched percentile #1420

Are you sure you want to change the base?

Features/1411 implement sketched percentile #1420

Conversation

mrfh92 commented Apr 3, 2024 • edited

github-actions bot commented Apr 3, 2024

github-actions bot commented Apr 3, 2024

codecov bot commented Apr 3, 2024 • edited

Codecov Report

github-actions bot commented Apr 4, 2024

github-actions bot commented Apr 4, 2024

mtar left a comment

Choose a reason for hiding this comment

github-actions bot commented Apr 5, 2024

github-actions bot commented Apr 5, 2024

github-actions bot commented Apr 5, 2024

github-actions bot commented Apr 5, 2024

github-actions bot commented Apr 5, 2024

ClaudiaComito left a comment

Choose a reason for hiding this comment

github-actions bot commented Apr 23, 2024

github-actions bot commented May 31, 2024

github-actions bot commented Jun 5, 2024

github-actions bot commented Jun 17, 2024

github-actions bot commented Jun 17, 2024

github-actions bot commented Jun 20, 2024

github-actions bot commented Jun 21, 2024

mrfh92 commented Apr 3, 2024 •

edited

codecov bot commented Apr 3, 2024 •

edited