-
Notifications
You must be signed in to change notification settings - Fork 211
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to better tune peak memory usage #260
Comments
Quick initial questions - |
a. TFT 1.5.0 |
Also, I should add that from my experiments testing these analyzers using DirectRunner it's not the quantiles analyzer that consumes most of the memory, it's the |
I have some datasets and transformations that I want to run that unfortunately won't fit on n1-highmem-16 instances (which is what FlexRS requires). The features are fairly standard scalar features with
tft.quantiles
analyzer and string features withtft.vocabulary
analyzer (but there are a lot of each type of feature). Generally the analyze step will run fine up until the final combine which will typically run on a very small number of machines and cause them to repeatedly OOM.Of course I can do something like use a larger machine type or even a custom machine type, but these don't work with FlexRS and would be more expensive. I'm generally curious about whether either of the following two options would be viable solutions:
Are either of these two options viable or is there a solution that I have not considered yet?
The text was updated successfully, but these errors were encountered: