Update tdex to v1.0.0 #754

altafan · 2023-08-28T10:31:11Z

This updates TDEX from v0.9.1 to v1.0.0.

The compose file defines a brand new oceand service required by the new version of tdexd v1.0.0. Updated also the dashboard to v1.0.1.

* Update docker-compose * Update images * Fixes * Expose dashboard port * Fixes * Update dashboard image * Update dashboard image * Get ready for being merged * Get ready for the merge

nmfretz

Thanks for this update @altafan! I have left some suggestions below for you to address before I test.

Also, are you okay if I push a commit directly to your update-tdex branch? We need to add a pre-start hook to make sure that users who already have tdex installed receive the new ocean-data directory with appropriate ownership when they update.

tdex/docker-compose.yml

tdex/hooks/pre-start

nmfretz · 2023-09-13T21:13:34Z

tdex/docker-compose.yml

+ ports:
+ - ${APP_TDEX_PORT}:${APP_TDEX_PORT}


we don't need to map ports here, as Docker's internal DNS resolution will allow the containers in this compose file to connect to each other. Unless you are trying to expose this port on the host so that it is reachable externally?

Yeah the idea is to make this service accessible from the outside world.

ah okay, so in that case we need to change ${APP_TDEX_PORT} to something other than 9090 because this port has been taken by Urbit Bitcoin Connector, meaning if a user is running both TDEX and Urbit Bitcoin Connector there will be a port clash on the host and one of the apps will break:

umbrel-apps/urbit-bitcoin-connector/umbrel-app.yml

Line 29 in 5ec41a7

port: 9090

it shouldn't be a problem at all, do you suggest any port number?

what about 9000?

Hi @altafan , port 9000 is currently occupied by portainer. To avoid conflicts, please consider using 9002, which is the closest available option. Before selecting a port, it's a good practice to search within this repository to ensure it doesn't overlap with existing port assignments.

I'm sorry! I just realized I changed this for the ease of testing and it wasn't meant to be a committed change. I reverted to port 9090 🙏

altafan · 2023-09-14T14:08:40Z

Thanks for this update @altafan! I have left some suggestions below for you to address before I test.

Also, are you okay if I push a commit directly to your update-tdex branch? We need to add a pre-start hook to make sure that users who already have tdex installed receive the new ocean-data directory with appropriate ownership when they update.

Thank you @nmfretz for the review!

I guess I have to give you access to my repo in order to directly push changes? Can you wait until all required changes are fixed?

nmfretz · 2023-09-14T15:59:40Z

I guess I have to give you access to my repo in order to directly push changes? Can you wait until all required changes are fixed?

Actually, here's what I think will be needed. You can add this above the Tor Hidden Service logic in the existing pre-start.

set -euo pipefail

APP_DATA_DIR="$(readlink -f $(dirname "${BASH_SOURCE[0]}")/..)"
OCEAN_DATA_DIR="${APP_DATA_DIR}/ocean-data"

[ ! -d "${OCEAN_DATA_DIR}" ] && mkdir -p "${OCEAN_DATA_DIR}" && chown 1000:1000 "${OCEAN_DATA_DIR}"

nmfretz · 2023-09-26T23:10:27Z

Hey @altafan can you please update the exposed daemon port to 9002: #754 (comment)

Also, what would be the best way for us to test external connection in the tdex ecosystem?

altafan · 2023-09-27T16:09:32Z

Also, what would be the best way for us to test external connection in the tdex ecosystem?

@nmfretz I made a test on a remote host, and once I installed the tdex app, I just went to <ip>:9090 to see a page similar to the following one (the url might be different):

tiero · 2023-10-04T14:30:40Z

Any news? @nmfretz

nmfretz · 2023-10-06T20:08:28Z

oh sorry, @tiero and @altafan can you guys push the quick fix noted here: #754 (comment). Port 9090 clashes with an existing app. 9092 is free for example.

Also, since this is an update, we can now add release notes to the umbrel-app.yml file. See Lightning app as an example:

umbrel-apps/lightning/umbrel-app.yml

Lines 24 to 28 in 7765871

 releaseNotes: >- 

  This release is a minor release to fix a memory leak inadvertently introduced by optimizations to the mempool scanning logic. 

  Read the full release notes for additional information and detailed changes at https://github.com/lightningnetwork/lnd/blob/master/docs/release-notes/release-notes-0.16.4.md

altafan · 2023-10-10T10:35:38Z

@nmfretz i pushed the changes. Now the port is 9092. I also followed your suggestion and added some release notes, thanks 🙏.

nmfretz · 2023-10-10T18:40:22Z

Thanks @altafan. I have made a few changes and we are nearly there:

in your recent commit 19e9840, you changed the port in the manifest to 9092. This port is actually for your dashboard UI, so should stay as 9094. I have changed that here a5bac52
updated the Caddyfile so that new installs start with the correct tdexd port (9092) 2f61879
added logic to the pre-start hook so that existing installs that are updating to this new version receive the change to the caddyfile f566598

I tested on a new install and it works well 👌. However, something breaks in the tdexd container when updating the app from the previous version. Here is the error I get from the tdexd container

$ sudo docker logs tdex_tdexd_1
INFO[0000] All 0 tables opened in 0s
INFO[0000] Discard stats nextEmptySlot: 0
INFO[0000] Set nextTxnTs to 0
INFO[0000] Deleting empty file: /home/tdex/.tdex-daemon/db/feeder/000007.vlog
INFO[0000] All 0 tables opened in 0s
INFO[0000] Discard stats nextEmptySlot: 0
INFO[0000] Set nextTxnTs to 0
INFO[0000] Deleting empty file: /home/tdex/.tdex-daemon/db/markets/000007.vlog
FATA[0000] failed to initialize grpc service             error="invalid opts: invalid app config: opening prices db: manifest has unsupported version: 7 (we support 8).\nPlease see https://github.com/dgraph-io/badger/blob/master/README.md#i-see-manifest-has-unsupported-version-x-we-support-y-error on how to fix this."

Could you please take a look into this? Once this is fixed we can go live!

altafan · 2023-11-15T16:46:42Z

@nmfretz sorry for the late reply

However, something breaks in the tdexd container when updating the app from the previous version. Here is the error I get from the tdexd container

The reason is that the database of the prev version of tdexd must be migrated in order to be compatible with the new version.
The migration tool is packed within the official docker image, do you think it's possible to setup a migration script that runs it if needed to solve the issue?

Otherwise, it could just be enough to delete the existing datadir, the daemon will create a new one at first startup and the user will restore the wallet, which seems an easier solution.

altafan · 2023-11-23T15:29:16Z

@nmfretz let's go for the easiest way and eventually remove the datadir if existing. Can you tell me if I can add this conditional operation to some hook script?

tiero · 2023-12-14T11:00:34Z

@smolgrrr any inputs here on what @altafan mentioned?

nmfretz · 2024-01-26T21:00:12Z

@tiero @altafan sorry for the delay.

Otherwise, it could just be enough to delete the existing datadir, the daemon will create a new one at first startup and the user will restore the wallet, which seems an easier solution.

For this option we'd run the risk of users clicking update without reading the release notes, so there could be a subset of users who update and don't have (or have lost) their recovery phrase, which would be pretty bad I think.

Additionally we'd have to make sure that this database deletion logic doesn't run on subsequent app starts/restarts/updates, which is totally possible but adds complexity.

The migration tool is packed within the official docker image, do you think it's possible to setup a migration script that runs it if needed to solve the issue?

Yes, we can do this with a pre-start script. The benefit here is that we users won't accidently lose access to their funds.

We could do something like this:

before starting the app, check if tdex-data dir has anything in it. If it doesn't then this is a new app install. We create a file in the data directory called something like HAS_MIGRATED and then exit the script and allow the app to start.
If tdex-data has data then check if the file called HAS_MIGRATED exists:
- If it HAS_MIGRATED exists then we exit the script and allow the app to start normally. This will ensure that subsequent app restarts/updates don't nuke the database again.
- If HAS_MIGRATED doesn't exist then we know that we need to migrate. We start the required container(s) for the migration tool and run the necessary commands to migrate. Then we create the HAS_MIGRATED file and exit the script so that the app starts up.

I can write this script on priority for you guys and test it out, I would just need guidance on the commands to run for migration. If I don't answer here, please feel free to DM me on X, Nostr, or Telegram!

altafan · 2024-02-13T17:59:13Z

@nmfretz the migration is an interactive process that requires the user to enter the password to unlock the data stored in the db. Can this be a problem for the workflow?

altafan added 3 commits August 28, 2023 12:19

Update docker-compose (#3)

82d49cb

* Update docker-compose * Update images * Fixes * Expose dashboard port * Fixes * Update dashboard image * Update dashboard image * Get ready for being merged * Get ready for the merge

Merge branch 'master' of github.com:tdex-network/umbrel-apps

145f480

Bump dashboard to v1.0.1

a0aef8c

nmfretz reviewed Sep 13, 2023

View reviewed changes

altafan added 3 commits September 15, 2023 12:46

Fix pre-start hook

53e5e06

Fix images digest & remove exported ports

c124114

Fix service port

1fc6fa0

altafan requested a review from nmfretz September 20, 2023 10:52

Revert changes to service port

6f81462

altafan added 2 commits October 10, 2023 11:48

Fix service port

cea16db

Add release notes

19e9840

nmfretz added 3 commits October 10, 2023 10:36

change dashboard port back to 9094

a5bac52

update tdexd reverse proxy in caddyfile

2f61879

add caddyfile update logic to pre-start

f566598

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update tdex to v1.0.0 #754

Update tdex to v1.0.0 #754

altafan commented Aug 28, 2023 •

edited

nmfretz left a comment

nmfretz Sep 13, 2023

altafan Sep 14, 2023

nmfretz Sep 14, 2023

nmfretz Sep 14, 2023

altafan Sep 15, 2023

altafan Sep 15, 2023

ParthJadhav Sep 25, 2023 •

edited

altafan Sep 27, 2023

altafan commented Sep 14, 2023

nmfretz commented Sep 14, 2023

nmfretz commented Sep 26, 2023

altafan commented Sep 27, 2023 •

edited

tiero commented Oct 4, 2023

nmfretz commented Oct 6, 2023

altafan commented Oct 10, 2023

nmfretz commented Oct 10, 2023

altafan commented Nov 15, 2023 •

edited

altafan commented Nov 23, 2023

tiero commented Dec 14, 2023

nmfretz commented Jan 26, 2024

altafan commented Feb 13, 2024

Update tdex to v1.0.0 #754

Are you sure you want to change the base?

Update tdex to v1.0.0 #754

Conversation

altafan commented Aug 28, 2023 • edited

nmfretz left a comment

Choose a reason for hiding this comment

nmfretz Sep 13, 2023

Choose a reason for hiding this comment

altafan Sep 14, 2023

Choose a reason for hiding this comment

nmfretz Sep 14, 2023

Choose a reason for hiding this comment

nmfretz Sep 14, 2023

Choose a reason for hiding this comment

altafan Sep 15, 2023

Choose a reason for hiding this comment

altafan Sep 15, 2023

Choose a reason for hiding this comment

ParthJadhav Sep 25, 2023 • edited

Choose a reason for hiding this comment

altafan Sep 27, 2023

Choose a reason for hiding this comment

altafan commented Sep 14, 2023

nmfretz commented Sep 14, 2023

nmfretz commented Sep 26, 2023

altafan commented Sep 27, 2023 • edited

tiero commented Oct 4, 2023

nmfretz commented Oct 6, 2023

altafan commented Oct 10, 2023

nmfretz commented Oct 10, 2023

altafan commented Nov 15, 2023 • edited

altafan commented Nov 23, 2023

tiero commented Dec 14, 2023

nmfretz commented Jan 26, 2024

altafan commented Feb 13, 2024

altafan commented Aug 28, 2023 •

edited

ParthJadhav Sep 25, 2023 •

edited

altafan commented Sep 27, 2023 •

edited

altafan commented Nov 15, 2023 •

edited