Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parallel Processing: Integrate Parallel Helpers Into Functions That Loop Or Groupby-Apply #104

Open
9 of 16 tasks
mdancho84 opened this issue Oct 11, 2023 · 2 comments
Open
9 of 16 tasks

Parallel Processing: Integrate Parallel Helpers Into Functions That Loop Or Groupby-Apply #104

mdancho84 opened this issue Oct 11, 2023 · 2 comments
Assignees
@mdancho84 @samuelmacedo83
Labels
enhancement New feature or request Priority
Milestone
v0.2.0

Comments

@mdancho84
Copy link
Contributor

mdancho84 commented Oct 11, 2023 •
edited

Running list of functions that should be evaluated for parallel processing / progress bars for engine = 'pandas'.

Core

  • summarize_by_time - Not sure if we need to parallelize agg() function
  • apply_by_time
  • future_frame
  • pad_by_time - Implementing Cartesian Product (Vectorized Padding Technique)

Anomalies

  • anomalize

Feature Engineering

  • augment_lags
  • augment_leads
  • augment_rolling
  • augment_rolling_apply
  • augment_expanding
  • augment_expanding_apply
  • augment_hilbert
  • augment_wavelet
  • augment_fourier

TS Summaries

  • ts_features
  • ts_summary
@mdancho84 mdancho84 added this to the v0.2.0 milestone Oct 11, 2023
mdancho84 added a commit that referenced this issue Oct 11, 2023
@mdancho84
#104 Parallel Processing
52ef1b8
mdancho84 added a commit that referenced this issue Oct 11, 2023
@mdancho84
#104 pad by time - cartesian product
81623bf
@mdancho84 mdancho84 mentioned this issue Oct 11, 2023
Closed
2 tasks
mdancho84 added a commit that referenced this issue Oct 11, 2023
@mdancho84
#104 future_frame speed boost
4926aac
mdancho84 added a commit that referenced this issue Oct 11, 2023
@mdancho84
#104 future_frame: Add tqdm for groups
aabc1ff
mdancho84 added a commit that referenced this issue Oct 11, 2023
@mdancho84
#104 future_frame - parallel processing
a0f1ca1
@mdancho84 mdancho84 added the enhancement New feature or request label Oct 11, 2023
mdancho84 added a commit that referenced this issue Oct 11, 2023
@mdancho84
#104 - Integrate parallel processing
97406df
@mdancho84
Copy link
Contributor Author

Keeping this quote for posterity. THIS is why vectorization is important.

image

@mdancho84
Copy link
Contributor Author

Parallel Processing update - I've switched to using pathos because it handles lambda functions whereas concurrent futures ProcessPoolExecutor does not. https://github.com/uqfoundation/pathos

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mdancho84 mdancho84

@samuelmacedo83 samuelmacedo83

Labels
enhancement New feature or request Priority
Projects
Timetk Development
Status: Development
Milestone
v0.2.0
Development

No branches or pull requests
3 participants
@mdancho84 @samuelmacedo83 @rabadzhiyski