[Feature] fetch public trades

[TheJoeSchr]

Orderflow

Summary

This PR enables to use ccxt fetch_public_trades so freqtrade can use trades data for backtesting and trading

Quickstart:

• use --dl-trades to fetch trades for timerange
• enable using trades in config.json

"exchange": {
   ...
   "use_public_trades": true,
}

• set orderflow processing configuration in config.json:

"orderflow": {
   "scale": 0.5,
   "stacked_imbalance_range": 3,
   "imbalance_volume": 1,
   "imbalance_ratio": 300
 },

TODO

• [docs] write (new?) docs about how to access this new trades data in populate_indicators etc
  • dataframe['trades']
  • dataframe['orderflow']
  • dataframe['delta']
• [docs] how to use --dl-trades to get trade data
• [docs] which configuration options are available and how to use them
• [tests] unit tests exist, but they are using too big for github test files. so I need to shrink them and rewrite tests to still make sense

from previous feedback by @xmatthias :

• Some diffs are a bit strange at first glance (especially in the exchange class), so expect some questions once i look at it more carefully (might be the webUI doing the diffs oddly if 2 functions are next to each other and are similar enough though) ...

• and i usually don't like random formatting changes (like - an unnecessary linebreak in a function that's not even touched otherwise) => fixed with commit 137ee07

• i also assume we'll want to move some things around in the end (converter seems to become quite big now) - but i can eventually help with that.

What's new?

[biiiipy]

This is awesome! Orderflow has a lot of valuable information, can't wait to start experimenting with this!

[xmatthias]

@dijvar it's an open PR. Docs will be written before merging the PR (which the PR description also mentions).
Please don't pollute the PR discussion with pointless requests, thanks.

@TheJoeSchr you may want to look at the conflicts (don't just look at the conflicting lines though ...).
trades datahandling changed right around the time (one PR number before your's 😆 ) you submitted this PR (#9065) - so i suspect some of the code you've written is now already available - or types won't align etc.

I think the diff of #9065 will be helpful to show what changed .... i don't think much else changed ... but these changes WILL impact this PR for sure.

[TheJoeSchr]

I think the diff of #9065 will be helpful to show what changed .... i don't think much else changed ... but these changes WILL impact this PR for sure.

thanks for the headsup @xmatthias . I think the overlap is minimal, because my "trades" are about L2 data aka orderflow trades and that PR is about trades the user/bot send to the exchange to execute.

but the naming overlap is unfortunate, I'm not sure how to handle it so there won't be any more confusion

[TheJoeSchr]

Please, write a short docs to use it. Thank you very much for the work :)

Thanks for showing interest, this helps keeping my motivation up to finally write the docs. did you try the quickstart on top of this PR though? If yes, what steps didn't work?

[xmatthias]

I see a few things that i don't like and make a review very difficult
For example:

• changes that don't belong here (see explicit comment)
• docstring (and signature) changes on methods that should not have been changed, causing pointless diffs (refresh_latest_ohlcv() - for example)
• reintroduction of arrow to exchange (which has been removed in Add datetime helpers, reduce arrow usage to a minimum #8661, and was replaced with internal helpers directly for datetime)
• duplicate methods in the same class (one will overwrite the other)

Due to the above, doing a proper review is currently near impossible - a the diff is unnecessarily large (especially in the exchange file).
Please do a review yourself (for example vscode's github pull requests extension can assist you with this by showing you the diff locally, allowing you to change what you don't like), comparing the differences - and identify what you actually INTENDED to change. That's something that's VERY difficult for me to identify.

Things you think should be changed but are not directly related to this pr (for example the max_calls point) - should either be removed, or extracted into individual PR's.
this becomes increasingly important with bigger PR's - small pr's combining 2-3 things - not really a problem, usually
but in a big feature like this - very problematic - as it draws focus apart - having high potential to introduce unwanted bugs due to these "non-directly" related changes.

[TheJoeSchr]

@xmatthias do you have an idea how I could mock out get_datahandler in my tests? I need to do this so there are no trade cache files are generated/overwritten when the tests run.

I tried overwriting it directly with a mock function in my test, but that doesn't work:

def test_refresh_latest_trades(mocker, default_conf, caplog, candle_type) -> None:
    # ...
    from freqtrade.data.history import get_datahandler
    get_datahandler = mock_get_datahandler
    # ...
    res = exchange.refresh_latest_trades(pairs, cache=False)

because the module gets loaded directly in refresh_latest_trades

    def refresh_latest_trades(self,
                              pair_list: ListPairsWithTimeframes,
                              *,
                              cache: bool = True,
                              ) -> Dict[PairWithTimeframe, DataFrame]:
        
        from freqtrade.data.history import get_datahandler
        data_handler = get_datahandler(
            self._config['datadir'], data_format=self._config['dataformat_trades']
        )

Maybe you have encountered something like this before or have a good idea?

[xmatthias]

Why is writing files bad? that's something that can be tested and asserted, too.

You just need to make sure it's not in the user's directory, but in a temporary space.

You can change user_data_dir to tmp_path quite easily (i was of the opinion that this is done globally, but apparently that's not the case) by assigning that in the tests (that's also done elsewhere ...).
For these particular tests - you might wanna update it in some "local" conftest file - so it applies globally to all tests where this applies.

[TheJoeSchr]

@xmatthias don't worry about my previous message I think I figured it out with mocker.patch()

[xmatthias]

This method seems unused ... intentional ?

can it be removed ?

[xmatthias]

is .apply() actually necessary here?
.apply() (without additional argument) means it'll run once for each row ... but the row is not actually used, so i'm not entirely sure if it's really necessary to do this per row, or if the np.where() could work by itself - which would greatly speed up this operation.
I also can't really see a place where df.loc[is_between,...] can return more than 1 row, so that kinda reinforces me questioning if apply() is necessary for that reason either.

---

doing a quick test for this based on the test: - with a breakpoint on the sell line - testing for buy above:

np.all(buy[0] == np.where(
                        trades_grouped_df["side"].str.contains("buy"),
                        0,
                        trades_grouped_df["amount"],
                    ))

Now doing that change (use np.where() directly to assign buy/sell) also has implications below - as the "deltas per trade" loop only returns one value now - but with both changes, all tests do still pass, so i'd assume it's good, unless doing this as loop covers some odd scenario i currently can't imagine.

[xmatthias]

Why would trades_grouped_df not be a dataframe?

[xmatthias]

I'm not entirely sure i understand why we need a shift(-1) here.
It looks like a bug from "birds view" - so if it's correct, i think it'll justify an explaining comment as to why we need it shifted.

[xmatthias]

are you sure this condition is correct?
wouldn't you want the "ask" side of the OB to be the same as "sell" (and bid the same as buy)?

np.where's syntax is np.where(condition, value_if_true, value_if_false) ... so the "asks" line would use "0" in case of a sell side trade?

[xmatthias]

i think this should raise a "freqtrade-ish" exception instead? otherwise we catch it here - but still have any random exception bubble "everywhere" ...

[TheJoeSchr]

what's a "freqtrade-ish" exception for example?

[xmatthias]

Depends what we want to accomplish i think
most likely, something around DependencyException (which doesn't completely crash the bot) should work.

ValueError can also be used - the point is though - it should be one error - not "all possible errors python has to offer" 😆

[xmatthias]

You sure "OperationalException" is the right one here?
That exception is assumed to stop the bot - so it completely restarts - though i'd not do that just because there's no ticks ... which could very well happen on random, low-volume pairs (?)

[xmatthias]

Suggested change

	logger.error(f"Refreshing TRADES data for {pair} failed")
	logger.error(e)
	logger.exception(f"Refreshing TRADES data for {pair} failed")

[xmatthias]

I think we need to re-visit what i've said before (where i actually just questioned why a "-cached" was appened - not "Actually" said to change it).
this should use a -cached in the filename to avoid overriding actual trade files.

These are slow to download (hence hard to come by) - and we shouldn't overwrite them with a trimmed, rolling version.