Hyperopt results improvement request #7342
Comments
Hyperopt is not using 2d spaces, but n-dimensional spaces (unless you use only 2 parameters). As such - i'm not aware how it should be possible visualize a n-dimensional space (take 6 or 7 as an example - but obviously it'll have to work for everything). It'd be possible up to 3 spaces - as that'd then be a 3d image - but I've rarely (if ever) seen strategies that only use 3 parameters. Sure it would e nice - but i don't think it's possible to visualize in a human-readable format. i'll leave this open for some time so you can maybe provide us with some links or examples on what you'd imagine (please also keep higher dimensional data in mind, so images of 2d/3d spaces will not suffice) - but for now, i doubt we'll be keeping this open as enhancement, as it's an enhancement that'd stay open forever - as there's no technical solution. |
|
Hi, Thank you in any regards (with varying 2d parameter space i meant to let the user choose 2...) |
|
i'm not sure this would provide actual benefits to the user (other than wasting their time looking at questionable information). especially with many parameters, you'll not be able to manually digest "is this good". if you assume for example "rsi was < 14 - so this is a non-robust result" - then your parameter range was wrong - as you didn't give hyperopt this additional information - which you had and are now using to determine that the result is "not robust". |
|
As far as i can see it would give me the benefit of determining or at least gives me the opportunity to try to find out how much i can trust a particular parameter combination not being an absolute random good hit.
Especially with continuous parameters I would e.g. like to know whether there may exist a certain threshold for usable results, indicating a certain range of good parameters.
With this i think it may be easier to find parameter sets which are more robust to changing input conditions saying market conditions.
I mean in a complete noisy parameter space like in weather forecasting with nonlinear differential equations you cannot find something like a threshold in the starting conditions for the simulation because of the inherent randomness.
But in our case there will be use cases and i think it will work out with many strategies using continuous Parameter spaces.
Of course i can just (try and) tell by the loss and profit values whether the parameters are good and where may be an area of optimal parameters, and refine the search space accordingly.
But this is not the same thing.
With the option to get all epochs with the corresponding probed parameters printed out to console or log file I would be really happy.
|
|
I have a rough implementation in my repo. I don't have time to clean it up and create a PR at the moment but it'd be great if you or someone else could implement it. It's been INCREDIBLY useful so far. This is what it looks like |
|
Wonderful, looks perfectly well and like what i was searching for. I doubt that I have the skills to do this, because I never did a PR, but will have a look definetly. |
|
For anyone considering this - we'll not be adding a dependency on matplotlib. we use plotly as plotting library - we won't need another one (which also doesn't provide any interactivity on top of it). |
|
I don't think it's necessary as I'm using |
|
Could you please serve a brief explanation on how this cn be used, or does it plot right away when using |
the plots look too much like matplotlib to be coincidence. Go on and try to uninstall matplotlib - skopt has an optional dependency to matplotlib - so i'm pretty sure that it'll break the moment matplotlib is no longer installed. |
It plots once per epoch when hyperopting based on the points it tried so far.
yes it uses matplotlib under the hood but it doesn't require an additional direct dependency. |
that's not true - it'll require matplotlib to be installed. |
|
Should be possible to transcript it to use plotly, only question remains who wants to put work in this and has the know how |
|
Just confirming the |
|
Getting a As your link refers to a tree which is not directly copyable by 'git clone ...' I just used the zip extraction. Is this a wrong approach? Any suggestions? |
|
@sanoj2021 this is the branch that I use, I'm not sure what you did exactly |
|
Ok, here's what i did: After setting everything in the config.json and copying my strategy file and candle data into the user-data/strategies folder: The same strategy file runs without problems on the official freqtrade version. |
|
This is my log from where the actual hyperopting should start: |
|
I think the key info I need is which parameter types you are trying to optimize and their ranges. Could you provide the section of your strategy in which they are defined? Also, try to set just a couple of them as |
|
@sanoj2021 how much parameters does your strategy have? |
|
@nartes The strategy uses 10 parameters for each pair and I tested it with 2 pairs: 2 CategoricalParameter, 6 DecimalParameter and 4 IntParameter. |
|
@sanoj2021 from statistical standpoint, it is like you randomly sample parameters, obtain backtest results versus input parameters. Then just try to correlate numeric parameters versus profit, or if its categorical one - then A/B testing. It also makes sense to examine manually some random deals, just to see what pattern the strategy has detected, and what was the decision. I've been recently working on support line indicator. Here are some examples: A sample of detected support line with some intermediate visuals.
A table with 10 samples containing parameters versus performance metrics.
|
|
@nartes You're right, I do random sampling, but isn't that the whole point while hyperopting? Generally I have fairly good understanding of how hyperopting and backtesting or bayesian optimization works.
That's what I was searching for in the first place and found in the plotting option of @italodamato 's custom branch. My current problem is solely related to getting this to work. (I may also just did something wrong with the installation of the custom branch. Someone kindly likes to have a look on my installation procedure a few posts above?)
This seems quite interesting to me, (how) can I add manual random examination in hyperopting? But most importantly I would like to know how you did get to the table with parameters vs performance parameters? ... as this would also perfectly well give me all I need. Do you possibly share your steps? |
Describe your environment
(if applicable)
python -V)pip freeze | grep ccxt)freqtrade -Vordocker-compose run --rm freqtrade -Vfor Freqtrade running in docker)Describe the enhancement
Hi,
Maybe someone considers the following a reasonable contribution...
It would be nice to get a feedback from hyperopting about the "places" of the good (and best) results in the parameter spaces. The goal of this is to make better estimates about the robustness of the found optimal parameter configuration in terms of "how robust is the solustion to small variations in the optimal parameters".
One can think of e.g. some form of 2D graph of the loss in a varying 2d-parameter space. Or, even inclusion of the particular parameters of each epoch in the live log or a log file would suffice.
In case I missed the according feature please let me know. Thanks
The text was updated successfully, but these errors were encountered: