Hyperopt cmd line new options for forward backtesting (and new result column)

[taebow]

Describe your environment

(if applicable)

• Operating system: Archlinux
• Python Version: 3.10.5
• CCXT version: 1.91.29
• Freqtrade Version: 2022.7

Describe the enhancement

I came with the idea of having some new options on the freqtrade hypeopt cmd line, where we could specify another time range for forward backtesting. Currently, I run hyperopt multiple times within a shell loop and with the same parameters (except for the random initial state), and end up with multiple results that I further backtest with a new time range (a sort of naive cross validation) to see if the results can generalize in the future. This option could automate this process, and rerun the epoch result for this new specified time range , with the downside of slowing the process of course, so that's why I propose as well a --validation-filter to add the capability to only do that for interesting results.

Here are the proposed new options:

• --validation-timerange TIMERANGE
• --validation-filter VALIDATION_FILTERS [all, best, profitable]

I don't have a clue at the moment on which indicator(s) could be best to add to the final result table (and json objects), if this should be only the validation loss function result or the Profit percentage for example, open for discussion...

[xmatthias]

the "filter" criteria will be difficult to implement i think, as it'll be very difficult to determine properly covering all cases.

No matter which loss-function is used - you'll usually get several "best" results (improving based on the loss function).
the definition of "best" is however problematic - if you look at the results - you'll usually immediately see "there's just been 10 trades but high profits - i'd prefer 100 trades and slightly lower profits" (this would spread the risk better, and not rely on 10 outlier trades).

I guess the metrics would be either the same as for hyperopt results (the reduced, oneline entry) - or the full backtest result for the "new" period (although doing that for 100 "best" results will probably overflow the shell buffer - so you won't be able to scroll back far enough)

To be fair - i think you'd want to do this after all epochs (at the end of the hyperopt run, basically) - and only for the "selected best" epoch.
doing it "mid-epochs" will slow down the whole process - as you'll have 1 epoch per "batch" that'll take longer as it's doing more work - slowing down the generation of future points.

That would only make sense if you do it for all epochs (independently) - and somehow incorporate the result of that into the loss value - to immediately mark "bad" validation results as bad (even though the original result was good).

HOW that'd be incorporated into the different loss functions is however difficult to determine - maybe by running the loss twice - once for validation results, once for test - and then aggregate the two results somehow ...

[taebow]

On the selection on which epoch should be validated against another time range, I don't think only the 'best' should be selected, because I have the impression that sometimes, the best result is not giving the best validation (because of overfitting). That is why maybe selecting only the profitable ones, would be a good start for a filter like this, but even with that, there would be probably many of them selected, and so this would be slow (depending of the solution to the next problem).

On the "after each epoch" versus "after the whole hyperopt process", I think each of them comes with a bunch of benefits and impediments, and maybe there is a tradeoff (that I was not aware because I didn't look at the code...):

• "after each epoch" would for sure slow everything, as we would have to run two backtesting instead of one for each .
• "after the whole process": the problem with that solution is that we would not see any validation result during the execution, only at the end, which will not allow us to add anything to the result table that is generated during the execution.
• A tradeoff would be to run the validation for each "EVAL" after the self.run_optimizer_parallel() call, at this point we even know what is best or profitable, so we can select the ones we want and backtest them with the new time range, this would slow the process but less than the first solution, and with that we can make some improvement to the result table during the execution (add some new columns or lines, I don't know).

On the final result, in the table, maybe we can start small, and just include the result of the loss function for the new time range in a new column. Ideally I would like to have the same information than an epoch result line, maybe we can double the lines (having for each epoch that a validation has ran on, another one below) with the validation to compare. Of course all of this would be optional, and not specifying this (these) option(s) would let the current behavior unchanged.

[xmatthias]

if you think about how hyperopt works - i don't think the "tradeoff" (option 3) is a good approach.

Think about a process running with -j 3.
It'd look as follows (E = 3 processes, - = 1 process).

-EEE---EEE---EEE
Hyperopt will run backtesting in parallel - before returning to single mode - and calculating the next term.
This is already a huge bottleneck, which takes in unfortunate cases almost half of your hyperopt time (if backtesting is quickish and you have a lot of parameters).
The load of the system will as such swing between 3 cores at 100% to 1 core at 100%.

The "tradeoff" would worsen this, by running another backtest for 0-3 results (we used -j 3 - so we have 3 epochs per batch). Using "profitable" will (in good cases) result in all 3 results being selected, which results in 3 backtests running (now in sequence - we left the parallel scope).

Data / memory will also become a topic - if you run it "not after hyperopt" (all options that run a validation backtest mid-hyperopt) - you'll need to keep the data in memory - which will make an already memory-problematic topic worse.

The only approach against this is to really do it afterwards - after tossing the "training" data out - so enough memory is freed.

[taebow]

Well, I didn't suggest we should run them sequentially, we could run them in parallel as well, in another parallel execution.
I understand that this is a bottleneck, in term of processing, no ideas what are the impediments regarding the memory (I already crashed everything on my 16GB RAM laptop by running a too large time range with 1min timeframe). The idea though is to run the additional backtests on a smaller time range than the initial one (~30 %), and if configured as such, it doesn't appear to me that this would cause an issue with the memory, but my intuition could be wrong, I still want to try it and see for myself.

And as mentioned before, this is only an option, at the end, the user is responsible for triggering it or not, and should account for the memory usage and thus adapt it's configuration if necessary.

[xmatthias]

Assuming you can run 1 month of 1m data on your 16Gb machine (tuning the parallelism a bit).

You'd now use 1month for hyperopt - and an additional week (~30%) for the validation - then the memory requirement would increase by this 30% - either reducing your parallelism further - or having it crash because of that.

Memory is a known problem area with hyperopt - and i'm against adding anything that will make this problem worse - especially if there's other approaches (running it afterwards, for selected epochs) - which can easily mitigate that.

this could even make it an additional subcommand (maybe ontop of the builtin "running afterwards") - where you pass in X epochs - and it does this validation step for you - using the parameters hyperopt gathered for each epoch.

I also don't see adding stuff to the results table as something necessary - not if it has no influence on further epochs / parameters calculated.
If we assume to run it "somewhere in the middle" - then it should run for every epoch and should be factored into the loss function (nothing says that a losing training period can't be profitable during the test period).

[taebow]

I agree to disagree (well just a bit, you have good arguments). Another option that just come into my mind could be to add a new little sister command instead to the already existing hyperopt-list and hyperopt-show that I would call: hyperopt-validate which would work in the same way (same file loading parameters and epoch selection), but would do backtesting instead on a new time range. Maybe it's a better approach than adding some more things to the hyperopt command.

[xmatthias]

Another option that just come into my mind could be to add a new little sister command

well that's what i mean with the additional subcommand :)

i'd make it selective though - so the user can specify "--epochs 1 5 6 7 8 10" - (defaults to all) ... ;)

[taebow]

It will take me some time to implement it, but I'm definitely willing to do it. So please don't close this issue yet.

[xmatthias]

didn't intend to. it's labeled discussion (so we're discussing the best approach) - and if you look carefully through the issues - these can be years old 😆

I closed your other issue as it's a "won't fix" issue - as we'll not be reimplementing something that is (hopefully soonish) available through an upstream library.