Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/dydx v4 perpetual derivative #6987

Open
wants to merge 45 commits into
base: development
Choose a base branch
from

Conversation

yancong001
Copy link
Contributor

@yancong001 yancong001 commented Apr 25, 2024

Before submitting this PR, please make sure:

  • Your code builds clean without any errors or warnings
  • You are using approved title ("feat/", "fix/", "docs/", "refactor/")

A description of the changes proposed in the pull request:

This PR create dydx v4 connector to the latest standards.

Tests performed by the developer:

Tips for QA testing:

Copy link

1 similar comment
Copy link

@rapcmia rapcmia linked an issue Apr 29, 2024 that may be closed by this pull request
@cardosofede
Copy link
Contributor

@yancong001 are you going to submit the test soon? so I can review everything together

@yancong001
Copy link
Contributor Author

Yes, I will submit it soon

@rapcmia
Copy link
Contributor

rapcmia commented May 15, 2024

PR update on commit 75b3aebb8cfc9ef

Setup simple v1

  • Create and cancel orders ok
  • Order filled and recorded ok
  • Creating order to close position
    • There are instances of the connector returning failed to submit order due to account sequence mismatch and incorrect account sequence. The connector retries which eventually creates order ok.

      2024-05-15 09:45:05,119 - 84113 - hummingbot.strategy.perpetual_market_making.perpetual_market_making - INFO - (LINK-USD) Creating 1  LIMIT ask orders at (Size, Price): ['1 LINK, 13.098 USD'] to CLOSE position.
      2024-05-15 09:45:06,048 - 84113 - hummingbot.strategy.perpetual_market_making.perpetual_market_making - INFO - Initiated cancelation of sell order 551455486 in favour of take profit order.
      2024-05-15 09:45:06,421 - 84113 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp": 1715737506.0, "order_id": "551455486", "exchange_order_id": "ff8658d4-ad44-580d-963c-ed638c5be7c6", "event_name": "OrderCancelledEvent", "event_source": "dydx_v4_perpetual"}
      2024-05-15 09:45:06,440 - 84113 - hummingbot.connector.client_order_tracker - INFO - Successfully canceled order 551455486.
      2024-05-15 09:45:08,316 - 84113 - hummingbot.connector.derivative.dydx_v4_perpetual.dydx_v4_perpetual_derivative.DydxV4PerpetualDerivative - WARNING - Failed to submit order 626539696 (retry 1), account sequence mismatch, expected 161, got 162: incorrect account sequence
      2024-05-15 09:45:10,696 - 84113 - hummingbot.connector.derivative.dydx_v4_perpetual.dydx_v4_perpetual_derivative.DydxV4PerpetualDerivative - WARNING - Failed to submit order 626539696 (retry 2), account sequence mismatch, expected 162, got 161: incorrect account sequence
    • https://www.loom.com/share/c02ad957a77f4c6a9eae5a33099b3a48

  • Position retrieval ok

Setup simple v2 (dmanv3)

  • Sample scenario for order 1071018602 and 1076937857
    image
  • Create limit order to open position 1071018602
    • Order filled and recorded on CSV and exchange. Timestamp and trade data matched
  • Create market order to close position 1076937857
    • Got order fill updates did not arrived on time , The complete update will be processed with incomplete information.
    • Got a order event completed for the sell order which is completely filled but not recorded on history nor CSV
      • On the screenshot above, market sell order was completed and recorded on exchange under price 12.937
      2024-05-15 10:53:18,466 - 10729 - hummingbot.strategy.script_strategy_base - INFO - Creating LINK-USD sell order: price: NaN amount: 1.
      # timestamp removed 
      2024-05-15 10:53:22,840 - 10729 - hummingbot.connector.client_order_tracker - INFO - Created MARKET SELL order 1076937857 for 1 to CLOSE a LINK-USD position.
      2024-05-15 10:53:22,853 - 10729 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp": 1715741602.0, "type": "OrderType.MARKET", "trading_pair": "LINK-USD", "amount": "1", "price": "NaN", "order_id": "1076937857", "creation_timestamp": 1715741598.0, "exchange_order_id": null, "leverage": 20, "position": "CLOSE", "event_name": "SellOrderCreatedEvent", "event_source": "dydx_v4_perpetual"}
      2024-05-15 10:53:25,004 - 10729 - hummingbot.connector.derivative.dydx_v4_perpetual.dydx_v4_perpetual_derivative.DydxV4PerpetualDerivative - WARNING - Failed to submit order 1079182838 (retry 1), account sequence mismatch, expected 184, got 185: incorrect account sequence
      # timestamp removed
      2024-05-15 10:53:29,117 - 10729 - hummingbot.connector.client_order_tracker - WARNING - The order fill updates did not arrive on time for 1076937857. The complete update will be processed with incomplete information.
      2024-05-15 10:53:29,120 - 10729 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp": 1715741608.0, "order_id": "1076937857", "base_asset": "LINK", "quote_asset": "USD", "base_asset_amount": "0", "quote_asset_amount": "0", "order_type": "OrderType.MARKET", "exchange_order_id": "17de96b3-0664-5699-a276-2a6c6b42c49a", "event_name": "SellOrderCompletedEvent", "event_source": "dydx_v4_perpetual"}
      2024-05-15 10:53:29,149 - 10729 - hummingbot.connector.client_order_tracker - INFO - SELL order 1076937857 completely filled.
      2024-05-15 10:53:32,708 - 10729 - hummingbot.connector.client_order_tracker - WARNING - The order fill updates did not arrive on time for 1076937857. The complete update will be processed with incomplete information.
      2024-05-15 10:53:33,294 - 10729 - hummingbot.connector.derivative.dydx_v4_perpetual.dydx_v4_perpetual_derivative.DydxV4PerpetualDerivative - ERROR - Unexpected error in user stream listener loop.
      Traceback (most recent call last):
        File "/Users/rapcomia/github/hummingbot/6987/hummingbot/core/data_type/in_flight_order.py", line 294, in get_exchange_order_id
          await self.exchange_order_id_update_event.wait()
        File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/asyncio/locks.py", line 214, in wait
          await fut
      asyncio.exceptions.CancelledError
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "/Users/rapcomia/github/hummingbot/6987/hummingbot/connector/derivative/dydx_v4_perpetual/dydx_v4_perpetual_derivative.py", line 424, in _user_stream_event_listener
          trade_updates = await self._process_ws_fills(data["fills"])
        File "/Users/rapcomia/github/hummingbot/6987/hummingbot/connector/derivative/dydx_v4_perpetual/dydx_v4_perpetual_derivative.py", line 518, in _process_ws_fills
          await v.get_exchange_order_id()
        File "/Users/rapcomia/github/hummingbot/6987/hummingbot/core/data_type/in_flight_order.py", line 293, in get_exchange_order_id
          async with timeout(GET_EX_ORDER_ID_TIMEOUT):
        File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/site-packages/async_timeout/__init__.py", line 141, in __aexit__
          self._do_exit(exc_type)
        File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/site-packages/async_timeout/__init__.py", line 228, in _do_exit
          raise asyncio.TimeoutError
      asyncio.exceptions.TimeoutError
      
      image
      image
      - Other instances of market orders are recorded successfully
      - It seems the connector fails to recognize the market sell order ❌
      - Crosscheck with other perpetual exchange (binance) running on dmanv3, observed expected behavior for the executor ok ✅
  • on stop command, getting unexpected error in user stream listener loop, syncio.execptions.TimeoutError as well
    2024-05-15 10:50:13,397 - 99919 - hummingbot.client.hummingbot_application - INFO - stop command initiated.
    2024-05-15 10:50:13,397 - 99919 - hummingbot.strategy.script_strategy_base - INFO - Creating LINK-USD sell order: price: NaN amount: 1.
    2024-05-15 10:50:13,399 - 99919 - hummingbot.strategy.script_strategy_base - INFO - (LINK-USD) Canceling the limit order 148399024. [clock=2024-05-15 02:50:13+00:00]
    2024-05-15 10:50:13,399 - 99919 - hummingbot.strategy.script_strategy_base - INFO - Creating LINK-USD sell order: price: NaN amount: 0.
    2024-05-15 10:50:13,430 - 99919 - hummingbot.connector.derivative.dydx_v4_perpetual.dydx_v4_perpetual_derivative.DydxV4PerpetualDerivative - ERROR - Unexpected error in user stream listener loop.
    Traceback (most recent call last):
      File "/Users/rapcomia/github/hummingbot/6987/hummingbot/core/data_type/in_flight_order.py", line 294, in get_exchange_order_id
        await self.exchange_order_id_update_event.wait()
      File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/asyncio/locks.py", line 214, in wait
        await fut
    asyncio.exceptions.CancelledError
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/Users/rapcomia/github/hummingbot/6987/hummingbot/connector/derivative/dydx_v4_perpetual/dydx_v4_perpetual_derivative.py", line 424, in _user_stream_event_listener
        trade_updates = await self._process_ws_fills(data["fills"])
      File "/Users/rapcomia/github/hummingbot/6987/hummingbot/connector/derivative/dydx_v4_perpetual/dydx_v4_perpetual_derivative.py", line 518, in _process_ws_fills
        await v.get_exchange_order_id()
      File "/Users/rapcomia/github/hummingbot/6987/hummingbot/core/data_type/in_flight_order.py", line 293, in get_exchange_order_id
        async with timeout(GET_EX_ORDER_ID_TIMEOUT):
      File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/site-packages/async_timeout/__init__.py", line 141, in __aexit__
        self._do_exit(exc_type)
      File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/site-packages/async_timeout/__init__.py", line 228, in _do_exit
        raise asyncio.TimeoutError
    asyncio.exceptions.TimeoutError
    

Steps to reproduce:

  1. Copy the controller config to conf/controller
  2. Copy the script config to conf/script
  3. Open Hummingbot client, run start --script v2_with_controllers --conf conf_dydx4.yml

05152024.zip

@yancong001
Copy link
Contributor Author

Unittest has submitted, separate passed (coverage run - m nose "test/hummingbot/connector/derivative/dydx_v4_perpetual") @cardosofede

@rapcmia
Copy link
Contributor

rapcmia commented May 20, 2024

Test commit fd51c807eb14a1f4f8b727cf484043d46d4e7ccf

  • Minor possible fix for leverage when exceeds the allowable value, it displays that connector reduced it and successfully set but not using the reduce value
    image
  • Setup bollinger_v1 config
    image
    image
    2024-05-20 16:12:07,966 - 35165 - hummingbot.strategy.script_strategy_base - INFO - Creating LINK-USD buy order: price: 16.671 amount: 0.5998440405494571411433027413.
    2024-05-20 16:12:17,889 - 35165 - hummingbot.connector.derivative.dydx_v4_perpetual.dydx_v4_perpetual_derivative.DydxV4PerpetualDerivative - ERROR - Unexpected error in user stream listener loop.
    Traceback (most recent call last):
      File "/Users/rapcomia/github/hummingbot/6987/hummingbot/connector/derivative/dydx_v4_perpetual/dydx_v4_perpetual_derivative.py", line 394, in _user_stream_event_listener
        trading_pair = await self.trading_pair_associated_to_exchange_symbol(order["market"])
    KeyError: 'market'
    2024-05-20 16:12:21,149 - 35165 - hummingbot.connector.client_order_tracker - INFO - Created MARKET BUY order 1602266450 for 0 to OPEN a LINK-USD position.
    2024-05-20 16:12:21,150 - 35165 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp": 1716192741.0, "type": "OrderType.MARKET", "trading_pair": "LINK-USD", "amount": "0", "price": "16.671", "order_id": "1602266450", "creation_timestamp": 1716192727.0, "exchange_order_id": null, "leverage": 30, "position": "OPEN", "event_name": "BuyOrderCreatedEvent", "event_source": "dydx_v4_perpetual"}
    2024-05-20 16:14:01,042 - 35165 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp": 1716192841.0, "order_id": "1602266450", "base_asset": "LINK", "quote_asset": "USD", "base_asset_amount": "0", "quote_asset_amount": "0", "order_type": "OrderType.MARKET", "exchange_order_id": "4593c0ac-d8e3-5aec-9bc7-b520cee4607c", "event_name": "BuyOrderCompletedEvent", "event_source": "dydx_v4_perpetual"}
    2024-05-20 16:14:01,042 - 35165 - hummingbot.connector.client_order_tracker - INFO - BUY order 1602266450 completely filled.
    
    • On this test, the script had opportunity to open long positions
    • Placing order on exchange return unexpected error in user stream listener
    • Market buy order created and completed
      • No record of CSV or history on client
      • exchange order if ✅
      • price of order is not available on logs
        image
  • Setup simple perpmm
    2024-05-20 15:27:38,003 - 30603 - hummingbot.strategy.perpetual_market_making.perpetual_market_making - INFO - (LINK-USD) Creating 1 LIMIT bid orders at (Size, Price): ['1 LINK, 16.549 USD'] to OPEN position.
    2024-05-20 15:27:38,004 - 30603 - hummingbot.strategy.perpetual_market_making.perpetual_market_making - INFO - (LINK-USD) Creating 1  LIMIT ask orders at (Size, Price): ['1 LINK, 16.884 USD'] to OPEN position.
    2024-05-20 15:27:39,997 - 30603 - hummingbot.connector.client_order_tracker - INFO - Created LIMIT BUY order 902512256 for 1 to OPEN a LINK-USD position.
    2024-05-20 15:27:39,998 - 30603 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp": 1716190059.0, "type": "OrderType.LIMIT", "trading_pair": "LINK-USD", "amount": "1", "price": "16.549", "order_id": "902512256", "creation_timestamp": 1716190058.0, "exchange_order_id": null, "leverage": 30, "position": "OPEN", "event_name": "BuyOrderCreatedEvent", "event_source": "dydx_v4_perpetual"}
    2024-05-20 15:27:40,395 - 30603 - hummingbot.connector.client_order_tracker - INFO - Created LIMIT SELL order 902512691 for 1 to OPEN a LINK-USD position.
    2024-05-20 15:27:40,396 - 30603 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp": 1716190060.0, "type": "OrderType.LIMIT", "trading_pair": "LINK-USD", "amount": "1", "price": "16.884", "order_id": "902512691", "creation_timestamp": 1716190058.0, "exchange_order_id": null, "leverage": 30, "position": "OPEN", "event_name": "SellOrderCreatedEvent", "event_source": "dydx_v4_perpetual"}
    2024-05-20 15:27:40,546 - 30603 - hummingbot.connector.derivative.dydx_v4_perpetual.dydx_v4_perpetual_derivative.DydxV4PerpetualDerivative - ERROR - Unexpected error in user stream listener loop.
    Traceback (most recent call last):
      File "/Users/rapcomia/github/hummingbot/6987/hummingbot/connector/derivative/dydx_v4_perpetual/dydx_v4_perpetual_derivative.py", line 394, in _user_stream_event_listener
        trading_pair = await self.trading_pair_associated_to_exchange_symbol(order["market"])
    KeyError: 'market'
    2024-05-20 15:27:41,505 - 30603 - hummingbot.connector.derivative.dydx_v4_perpetual.dydx_v4_perpetual_derivative.DydxV4PerpetualDerivative - ERROR - Unexpected error in user stream listener loop.
    Traceback (most recent call last):
      File "/Users/rapcomia/github/hummingbot/6987/hummingbot/connector/derivative/dydx_v4_perpetual/dydx_v4_perpetual_derivative.py", line 394, in _user_stream_event_listener
        trading_pair = await self.trading_pair_associated_to_exchange_symbol(order["market"])
    KeyError: 'market'
    2024-05-20 15:28:38,003 - 30603 - hummingbot.strategy.perpetual_market_making.perpetual_market_making - INFO - (LINK-USD) Canceling the limit order 902512256. [clock=2024-05-20 07:28:38+00:00]
    2024-05-20 15:28:38,004 - 30603 - hummingbot.strategy.perpetual_market_making.perpetual_market_making - INFO - Canceling active order 902512256.
    2024-05-20 15:28:38,004 - 30603 - hummingbot.strategy.perpetual_market_making.perpetual_market_making - INFO - (LINK-USD) Canceling the limit order 902512691. [clock=2024-05-20 07:28:38+00:00]
    2024-05-20 15:28:38,004 - 30603 - hummingbot.strategy.perpetual_market_making.perpetual_market_making - INFO - Canceling active order 902512691.
    2024-05-20 15:28:38,666 - 30603 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp": 1716190118.0, "order_id": "902512256", "exchange_order_id": null, "event_name": "OrderCancelledEvent", "event_source": "dydx_v4_perpetual"}
    2024-05-20 15:28:38,670 - 30603 - hummingbot.connector.client_order_tracker - INFO - Successfully canceled order 902512256.
    2024-05-20 15:28:39,002 - 30603 - hummingbot.strategy.perpetual_market_making.perpetual_market_making - INFO - Canceling active order 902512691.
    2024-05-20 15:28:39,071 - 30603 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp": 1716190119.0, "order_id": "902512691", "exchange_order_id": null, "event_name": "OrderCancelledEvent", "event_source": "dydx_v4_perpetual"}
    2024-05-20 15:28:39,075 - 30603 - hummingbot.connector.client_order_tracker - INFO - Successfully canceled order 902512691.
    
    • Same results whenever an order is created and canceled, it returns Unexpected error in user stream listener loop

Steps to reproduce:

  1. Copy the controller config to conf/controller
  2. Copy the script config to conf/script
  3. Open Hummingbot client, run start --script v2_with_controllers --conf conf_v2_with_controllers_1.yml
  4. Attached strategy yml file for perpetual-mm

https://drive.google.com/drive/folders/10KrgraekoCV6q96Z-MXgz4L2tJ3kTtQu

@yancong001
Copy link
Contributor Author

fixed

@lukebyrne
Copy link

I have pulled this branch down to assist with some QA.

At first I was getting an error with the password, that was solved by clearing out the CONFIG_PASSWORD in the docker-compose.yml

Now I have hummingbot loaded up, I am looking to actually load the V4 DyDx and play with it.

Any suggestions welcome on getting started on this.

@rapcmia
Copy link
Contributor

rapcmia commented May 24, 2024

I have pulled this branch down to assist with some QA.

At first I was getting an error with the password, that was solved by clearing out the CONFIG_PASSWORD in the docker-compose.yml

Now I have hummingbot loaded up, I am looking to actually load the V4 DyDx and play with it.

Any suggestions welcome on getting started on this.

Hi @lukebyrne

Note: both setup below share same folders for configs, logs and scripts

For source build
image

  • Clone the PR and change the settings on the screenshot above
  • Start the hummingbot client and add your API keys

For docker build

  • You need to change the files on source then build docker image of the PR
  • Setup docker compose file
    • Make sure to uncomment this part on the docker-compose.yml file
      build: # Uncomment this and comment image if you want to build it locally
      context: .
      dockerfile: Dockerfile
      
      • Run docker compose up -d and it will build local docker image for the PR
      • Once completed, run docker ps to see active containers
      • Run docker attach hummingbot to open the container
  • If all ok or you want to stop docker compose, run docker compose down

@lukebyrne
Copy link

lukebyrne commented May 24, 2024 via email

@rapcmia
Copy link
Contributor

rapcmia commented May 24, 2024

I don’t see anyway to connect to DyDx V4?

Please recheck the instructions above and change the settings.py in order to access dydxv4. Currently there is an issue with compatibility with injective connector thats why we need to modify it manually. The foundation team is in discussion on how we can fix it but for now its the way to access the dydxv4 on this PR

Please see recording below on connecting to dydxv4
https://www.loom.com/share/ca7e5c6c0b5e4398903bf7a1d08894e4

@lukebyrne
Copy link

@rapcmia thanks for the update, I needed to switch the right git branch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Under Review
Development

Successfully merging this pull request may close these issues.

dYdX - Upgrade the connector to v4
4 participants