Automatic HyperOpt run every X days

[xwurg]

Hello, first of all, many thanks for this awesome library!

I am working on 5m timeframe strategies and found out that the profitable parameters can change from one week to another. Hence, I thought it might be time saving and efficient money-wise to add an 'Auto HyperOpt' functionality, that would, if turned on :

1. automatically run an hyperopt loop for x days, on a daily/weekly basis
2. compare with current parameters
3. pick up the best parameters and reload the config/strategy

The config.json file would thus need another section :

"autohyperopt": {
    "enabled" : true,
    "run_every_x_days": 1,
    "backtest_x_days": 5,
    "epochs": 100,
    "hyper opt_loss": SortinoHyperOptLossDaily,
    "compare_with_current_parameters": false,
}

Let me know what you think and if you like part of this idea or this idea as a whole. I'm looking forward to discussing it!

Best
xwurg

[xmatthias]

well edge does something "pretty" similar - although with a different outcome / goal.

i'm not ... against it per se - but there's a few things to consider:

• if you run with this enabled on a VPS - how can you make sure it's not crashing the bot process due to OOM?
  it's an error we can neglect / accept when running hyperopt as standalone (and work around by using fewer data) - but now you're running it intra-process as part of trade mode ... so i'd hope we can somehow prevent that / capture that...
• You'll need to evaluate if a reload of config/strategy is really (like - REALLY) necessary, or if a simple "resetting" of the hyperoptable parameters would suffice (obviously while also persisting the values to disk - to handle an eventual restart). It works for hyperopt too, so i don't see why it can't work for trade mode.
• what would you expect from "compare to current parameters"? While it's mentioned above - it's pretty vague and unclear what this step should actually do.
• convince me that auto-accepting the "best" result based on hyperopt-loss is really the strategy i'd want to pick ... i often pick the 2nd or 5th best value, maybe because i like overall profit, or win/loss ratio (or amount of trades) better than for the first result.

For the parameters - i'm not sure you'll need all of them - epochs and hyperopt_loss are already (undocumented) parameters at the root level (every CLI argument translates 1:1 to a config parameter) ... so you'll only really need the remaining parameters...

[xwurg]

Thanks for your reply, Matthias. Indeed, as you said, Edge is "pretty" similar. But as you know - for instance - the crypto market sometimes is more volatile than others. Sometimes, no trade occur , some other times, too many occur... It would thus be interesting to automatically work out parameters that follow the market trend, instead of "just" seeing which pairs works and which do not.

Now addressing your points. Please forgive me if some parts of the documentation has slipped out of my mind.

1. Indeed, strong point. I personally run Freqtrade on an 8-core CPU with 64 gigs of RAM... But enough bragging :) I used to run it on a much less powerful system (4 cores, 8 gigs) and hyper opt still worked well even when trading was on. I think that if you stick to 7-10 days of loopback, it should be fine, although if this gets implemented, users should be warned. As for a way to catch that... I'm no python expert, but there should be a way to issue an error message and stop the hyperopt or to run it in a parallel/separate docker instance?

2. I may not be understanding that part too well. Changes in my strategy file only take place when I reload the config.

3. When hyperopting, it would be interesting to also run a backtest with current parameters. The bot would then compare the best result from hyperopt and the result from backtesting current parameters. Only if hyperopt found better parameters the current parameters would be replaced.

4. I agree with you on that. Maybe then it would be nice to implement another parameter in the config file that would command hyperopt to look at win/loose ratio...

... Which is the reason why I thought adding hyperopt_loss would be nice. OnlyProfitHyperoptLoss looks into the most profiting parameters... Perhaps there should be another hyperopt_loss parameter to find out the best strategy parameters in terms of win/loose ratio. As for epochs, I think it's important regarding your first point. Less epochs is less demanding in terms of memory usage.

Up to you !

[xmatthias]

1. for "trade"-mode - i think both of your systems are overpowered - and most people (including myself) run trade mode on cheap VPSes (5$ digitalocean / vultr for example).
   With 1Gig of Ram, you'll have trouble doing anything with this. Warning might be difficult - as you can go to the limit of memory - but the system decides to kill your process.
   you can also limit parallelity (-j) or other things to limit this - there's however still no guarantee it'll not kill your process.

2. let's leave that as implementation detail for now ... it's true for the moment, but there would be ways to avoid this, too.

3. the problem here is the definition of "better" - which is something vague, and difficult to put in code - as it's usually judgement we're applying, not "because of these 2 values" (which would be scientific).

4. it'll still only be "a" loss function - it won't matter what it looks for - you'll always look at a combination of "one over the other" manually - and usually make judgement calls (at least i do) on which result "feels" better.

---

i'll be adding a 5th point here - what about open trades?
Open trades will suddenly have new parameters apply (for example, it could be -7% - and new hyperopt determined stop is moved from 10% to 5% ...).
It can result in strange effects - and (maybe more importantly) in problems with accountability.
assume you run hyperopt once a week ... and you'd like to analyze your trades from 3 weeks ago (why was this a loss, what could be improved) ?
You can't look at the charts (at least not easily) - as you'll not know what parameters have had been applied during that time. While we can store the old parameter files, from a usability perspective it'll be difficult to find the "right" set of parameters for that trade.

[anakein]

you can to make this outside freqtrade.

1 RB for hyperopt
1 RB for backtesting
and 2 for trade. sync strategy files with rsync and reload by rpc

[briankudera]

Upvoting/following. I'd also prefer this over the Edge module. If we can add the --days parm to HO then I think it should be able to be automated with cron jobs.

[gaugau3000]

The market don't change that fast -> running every n days with not a representative sample will lead to overfitting. On classical market a good way to do is to do 5 years in sample data and 1 year out of sample (or 10 years in and 2 years out ) then continue to to this years after years and it will five you a walk forward analysis. Believe me or not i have a strategy in live that don't have change for 2 years and still perform the same way (modulo the upper trend biais of course of the crypto world).

[mosquitotrip]

Another approach (If I may) would be splitting freqtrade to a 'trade node' and 'opt node', once 'opt node' completes performing its tasks the 'trade node' should consider reloading optimizations settings.

That will mean a lighter FT node, mostly sleeping, that can be restricted, resource-wise.

There should be some thought put into the workflow and interaction between the node types, @xmatthias is right, performing hyperopt inside trade instance is risking the trading instance greatly.

[kyrypto]

You can currently do this with a similar cron job, of course fixing this path to your own directories and tuning to your own needs:
@daily screen -dmS hyperopt /home/<user>/freqai/.env/bin/freqtrade hyperopt --hyperopt-loss SharpeHyperOptLoss --strategy ExampleStrategy --strategy-path /home/<user>/freqai/user_data/strategies --userdir /home/<user>/freqai/user_data --config /home/<user>/freqai/config.json --spaces roi stoploss --timerange "$(date --date='-3 month' '+%Y%m%d')"-"$(date '+%Y%m%d')" -e 1000

[maksam07]

@kyrypto Came to this page by accident, but saw your comment with the code $(date --date='-3 month' '+%Y%m%d'). It helped me to improve my commands a bit, thanks :)