feature: implement an initial mempool.space websocket client, and library for block-events #3

oleonardolima · 2022-05-16T19:50:15Z

Description

This PR focus on solving the issues:

Refactoring the previous implementation for the competency test to become an initial library that produces a stream of BlockEvent, focusing only on new block data and deprecating the previous track-address and other features implemented.

This PR although moving some files in library structure, it's still maintained and makes it possible to use it through a CLI interface where the block events are printed to stdout.

As a result, now we have the module for mempool.space wrt to the WebSocket client, message building, and parsing (both for request and response).

Notes to the reviewers

There is no need to go through each commit, the PR could be reviewed as a whole to connect the parts and get an overview related to the logic and implementation.

There is no unit or integration test at the moment, as I'm still researching the best way to build some sort of running process with electrsd and bitcoind that replicates the mempool.space behaviour, and it's being addressed by the #6 . If you have any suggestion feel free to reach me on BDK discord 😁

It has been tested locally with nigiri for running the bitcoind and also running mempool.space through docker, connected to the local regtest RPC.

If you would like to test it, please take a look at the README file for the commands and guidelines.

Checklists

All Submissions:

I've signed all my commits
I followed the contribution guidelines
I ran cargo fmt and cargo clippy before committing

New Features:

I've added tests for the new feature
I've added docs for the new feature
I've updated CHANGELOG.md

Signed-off-by: Leonardo Lima <[email protected]>

ce and basic apis Signed-off-by: Leonardo Lima <[email protected]>

Signed-off-by: Leonardo Lima <[email protected]>

LLFourn

I've suggested some changes. Most of which I've suggested to you on our call :)

LLFourn · 2022-06-07T07:48:17Z

src/lib.rs

+ network: &Network,
+ data: &MempoolSpaceWebSocketRequestData,
+) -> anyhow::Result<impl Stream<Item = BlockEvent>> {
+ env_logger::init();


in general, you shouldn't env_logger::init inside a library. This should be done in the fn main.

Thanks! I've moved it to the main function

LLFourn · 2022-06-07T07:49:47Z

src/mempool_space/mod.rs

+) -> anyhow::Result<impl Stream<Item = BlockEvent>> {
+ let url: Url = url::Url::parse(&build_websocket_address(network)).unwrap();
+ websocket::connect_and_publish_message(url, message).await
+}


I think this function can be removed. Let the caller pass in the mempool.space .../api to use to find the websocket. For example, What if I want to use my own self-hosted mempool.space?

Yeah, agreed. I've removed this fn as updated the other fn to subscribe_to_blocks

LLFourn · 2022-06-07T08:02:18Z

src/mempool_space/mod.rs

+ message
+}
+
+fn build_websocket_address(network: &Network) -> String {


this could be renamed and made public so you could get the string and pass it in as I suggested above.

Done, this is being used by caller now :)

LLFourn · 2022-06-07T08:05:14Z

src/mempool_space/websocket.rs

+ let (mut websocket_stream, websocket_response) =
+ connect_async_tls_with_config(&url, None, None)
+ .await
+ .unwrap_or_else(|_| panic!("failed to connect with url: {}", &url));


This can just be await? I think . It's a bad idea to panic on an IO failure!

Yeah, updated

LLFourn · 2022-06-07T08:09:00Z

src/mempool_space/mod.rs

+ websocket::connect_and_publish_message(url, message).await
+}
+
+pub fn build_websocket_request_message(


This should be a private function for now.

@LLFourn Good, I've updated it to be private and moved this one to websocket.rs as it's only used by websocket, wdyt ?

LLFourn · 2022-06-07T08:23:23Z

src/mempool_space/websocket.rs

+use tokio_tungstenite::tungstenite::protocol::Message;
+use url::Url;
+
+pub async fn connect_and_publish_message(


I think this function should just be subscribe_to_blocks and take the mempool.space ../api endpoint. Later it will take an optional height and blockhash checkpoint.

@LLFourn Great, I've renamed it and updated it to expect only the URL for now 😁

…rary for block-events from #3

oleonardolima · 2022-07-07T00:20:59Z

This PR and its commits have been squashed and merged into #10, to make it easier for review and code checking at this stage.

…point and reorg handling) (#10) * feature: implement an initial mempool.space websocket client, and library for block-events from #3 * feature: add unit and integration tests to block events from #8 * feat+test: add initial http client and tests * wip(feat): add initial approach for subscribing to blocks from a starting height * wip(feat): fix fn and use a tuple as return type for prev and new block streams * wip(refactor): initial updates for fns refactor and architecture change * wip(feat+refactor): add fn to process candidate BlockHeaders, handle reorg and yield BlockEvent * wip(refactor+docs): extract fns to cache struct, and add documentation for fns, enums and structs * wip(fixes+tests): fixes, improve error handling and add new integration tests * wip(fix+test): fix common ancestor and fork branch bug, add new reorg tests * fix: disconnected and connected block events emitting order for reorg events * chore: simplify cli usage and output * chore: update docs and readme examples, use u32 instead of u64 * fix: remaining cargo.toml conflicts from other branches * feat: add full block stream api, better error handling - use TryStream instead of stream, returning result instead - use `?` operator instead of unwraping and panicking - add new mempool.space endpoints for http client - add features for wss:// and https:// usage - add features for api versioning based on mempool.space backend * refactor: do not use features expect two base_url instead, and improve error handling - refactor and update http client, url usage and methods - refactor and add creation methods for cache struct, use specific method to build initial one - add `get_block_header` method in http client, and refactor lib to use `bitcoin::BlockHeader` instead of custom mempool.space `BlockExtended` struct - refactor lib and websocket to improve error handling, update some `unwrap()` usage and match errors for messages in `WebSocketStream` - remove duplicated/unused deps, and use only necessary features * chore: add and update CHANGELOG.md file * fix(test): docs and integration tests - bump mempool/backend docker container version to v2.4.1 - update api+websocket client to handle genesis blocks without prev_blockhash - update and fix integration tests to new fns and structure

chore: update cargo.toml and move main.rs to bin.rs

2f6ebcf

oleonardolima force-pushed the feature/add-blocks-stream-as-module branch 3 times, most recently from 836c8e6 to 3191aa3 Compare May 18, 2022 16:49

oleonardolima force-pushed the feature/add-blocks-stream-as-module branch 2 times, most recently from 23d823e to 4db855e Compare June 1, 2022 03:25

oleonardolima changed the title ~~Feature/add blocks stream as module~~ [issue-4][issue-5] Implement an initial mempool.space websocket client, and library for block-events Jun 1, 2022

oleonardolima force-pushed the feature/add-blocks-stream-as-module branch 2 times, most recently from 8c0dd96 to 7bf75eb Compare June 2, 2022 14:57

oleonardolima added 16 commits June 2, 2022 11:58

wip(lib): extract ws subscription to another fn

2862e53

Signed-off-by: Leonardo Lima <[email protected]>

wip(lib): move ws connection and polling fn to lib module

4841927

Signed-off-by: Leonardo Lima <[email protected]>

wip: comment connection

2968ade

Signed-off-by: Leonardo Lima <[email protected]>

wip: add rust-bitcoin as deps

74a6119

Signed-off-by: Leonardo Lima <[email protected]>

wip(client+api): add initial ws client for mempool spa

dba6c3c

ce and basic apis Signed-off-by: Leonardo Lima <[email protected]>

wip(lib): update lib and bin to use mempool space module

f39468e

Signed-off-by: Leonardo Lima <[email protected]>

fix(logging): add env-logger deps

bf12173

Signed-off-by: Leonardo Lima <[email protected]>

chore: remove unnecessary log prefix

071055e

Signed-off-by: Leonardo Lima <[email protected]>

wip: websocket fn returns a stream of BlockExtended

85d36d0

Signed-off-by: Leonardo Lima <[email protected]>

chore(bin+lib): refactor cli and lib fns, to use specific enums

85dce36

Signed-off-by: Leonardo Lima <[email protected]>

chore: remove TODOs and keep some APIs simple

ba33217

Signed-off-by: Leonardo Lima <[email protected]>

chore(bin+api): use rust-bitcoin type for BlockHash and Address

88edad2

Signed-off-by: Leonardo Lima <[email protected]>

fix: remove solvable TODOs

8aca8c9

Signed-off-by: Leonardo Lima <[email protected]>

refactor: update lib to return stream of block events instead

868b7d5

Signed-off-by: Leonardo Lima <[email protected]>

chore: update README, remove trailing spaces, fix deps usage

181ef0e

Signed-off-by: Leonardo Lima <[email protected]>

chore: add pull request template

10e1e11

Signed-off-by: Leonardo Lima <[email protected]>

oleonardolima force-pushed the feature/add-blocks-stream-as-module branch from 7bf75eb to 10e1e11 Compare June 2, 2022 14:58

oleonardolima changed the title ~~[issue-4][issue-5] Implement an initial mempool.space websocket client, and library for block-events~~ feature: Implement an initial mempool.space websocket client, and library for block-events Jun 2, 2022

oleonardolima changed the title ~~feature: Implement an initial mempool.space websocket client, and library for block-events~~ feature: implement an initial mempool.space websocket client, and library for block-events Jun 2, 2022

oleonardolima self-assigned this Jun 2, 2022

oleonardolima added the enhancement New feature or request label Jun 2, 2022

oleonardolima added this to the milestone #1 milestone Jun 2, 2022

oleonardolima marked this pull request as ready for review June 2, 2022 15:13

oleonardolima added 2 commits June 2, 2022 12:58

fix: formatting with cargo fmt

4f01ccf

fix: apply suggestions from cargo clippy

2147092

oleonardolima force-pushed the feature/add-blocks-stream-as-module branch from e96ea41 to 2147092 Compare June 2, 2022 15:59

oleonardolima requested a review from LLFourn June 3, 2022 12:18

LLFourn requested changes Jun 7, 2022

View reviewed changes

notmandatory mentioned this pull request Jun 7, 2022

W23 BDK Library Team Call bitcoindevkit/.github#8

Closed

16 tasks

oleonardolima added 2 commits June 7, 2022 13:02

fix: apply review suggestions

612f8ac

chore: update README example

bddefcb

oleonardolima requested a review from LLFourn June 7, 2022 18:41

LLFourn and others added 2 commits June 23, 2022 23:56

refactor: Lloyd's re-engineering of API (#7)

d9ea581

chore: update bin, and update example in README

4c41c16

This was referenced Jun 24, 2022

Add integration and unit tests for mempool.space websocket client, API and fns #6

Open

feature: add unit and integration tests to block events #8

Closed

oleonardolima mentioned this pull request Jul 6, 2022

feature: add block_headers and blocks streams full APIs (w/ checkpoint and reorg handling) #10

Merged

6 tasks

oleonardolima added a commit that referenced this pull request Jul 7, 2022

feature: implement an initial mempool.space websocket client, and lib…

67e5f0a

…rary for block-events from #3

oleonardolima closed this Jul 7, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature: implement an initial mempool.space websocket client, and library for block-events #3

feature: implement an initial mempool.space websocket client, and library for block-events #3

oleonardolima commented May 16, 2022 •

edited

Loading

LLFourn left a comment

LLFourn Jun 7, 2022

oleonardolima Jun 7, 2022

LLFourn Jun 7, 2022

oleonardolima Jun 7, 2022 •

edited

Loading

LLFourn Jun 7, 2022

oleonardolima Jun 7, 2022 •

edited

Loading

LLFourn Jun 7, 2022

oleonardolima Jun 7, 2022 •

edited

Loading

LLFourn Jun 7, 2022

oleonardolima Jun 7, 2022

LLFourn Jun 7, 2022

oleonardolima Jun 7, 2022

oleonardolima commented Jul 7, 2022

feature: implement an initial mempool.space websocket client, and library for block-events #3

feature: implement an initial mempool.space websocket client, and library for block-events #3

Conversation

oleonardolima commented May 16, 2022 • edited Loading

Description

Notes to the reviewers

Checklists

All Submissions:

New Features:

LLFourn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oleonardolima Jun 7, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oleonardolima Jun 7, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oleonardolima Jun 7, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oleonardolima commented Jul 7, 2022

oleonardolima commented May 16, 2022 •

edited

Loading

oleonardolima Jun 7, 2022 •

edited

Loading

oleonardolima Jun 7, 2022 •

edited

Loading

oleonardolima Jun 7, 2022 •

edited

Loading