Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: provide first-class ssz support on api #6749

Open
wants to merge 263 commits into
base: unstable
Choose a base branch
from
Open

Conversation

nflaig
Copy link
Member

@nflaig nflaig commented May 8, 2024

Motivation

Initial discussion around this has started in #5128 to provide first-class support for SSZ in both request and response bodies but besides that, there are plenty of other shortcomings in the way we currently define routes and handle server / client responses. In a lot of cases, Lodestar was not following the spec (e.g. not setting all required headers) causing incompatibilities with other clients. The main reason for this is because our API spec tests were incomplete, headers were not checked at all or if we support SSZ on all required routes as per spec, and addressing those shortcomings requires workarounds like overriding handlers on both the client and server.

Description

This PR changes the structure of how routes are defined and makes it trivial to add SSZ support to any route. The only exception are routes with data payloads that cannot be represented as a SSZ type due to lack of JSON mapping, meaning it is not possible to translate the JSON payload into a SSZ-serialized payload.

The second major change is how we handle requests and responses on the server and client. While previously if you would request a state as SSZ from the server, the client would only be able to handle that format and if the server didn't support it, the request would fail. Now, we use content type negotiation between client and server which works based on headers and status codes to allow a more flexible and robust handling of requests and responses.

The third redesign in this PR is related to the API client, it is now possible to provide options like HTTP timeout per request and the response object is much more complete and self-contained, e.g. it's no longer needed to call ApiError.assert(...) everywhere (more examples below). This refactoring allows for further extensions in the future like a per route strategy for sending requests to fallback nodes and picking responses based on metadata, e.g. pick the most profitable block from all connected beacon nodes.

BREAKING CHANGES

While there is are no breaking changes to the validator client and beacon node, the client exported from @lodestar/api package changed significantly, and there are minor other changes to exported types but those are not as relevant for most consumers.

These are the most notable changes

Client initialization

before

const api = getClient({urls, getAbortSignal: () => opts.abortController.signal}, {config, logger});

after

const api = getClient({urls, globalInit: {signal: opts.abortController.signal}}, {config, logger});

Request with value

before

const res = await api.beacon.getGenesis();
ApiError.assert(res, "Can not fetch genesis data from beacon node");
const genesis = res.response.data;

after (.value() will assert if request was successful and throw detailed error)

const genesis = (await api.beacon.getGenesis()).value()

Request without value

before

ApiError.assert(await beaconClient.submitPoolVoluntaryExit(signedVoluntaryExit));

after (response object is self-contained, no utility like ApiError.assert is required anymore)

(await beaconClient.submitPoolVoluntaryExit({signedVoluntaryExit})).assertOk();

Passing arguments

before

api.validator.produceAttestationData(committeeIndex, slot)

after (args are passed as object, it is now possible to pass per request options, like timeout)

api.validator.produceAttestationData({committeeIndex, slot}, {timeoutMs: 4000})

TODO

  • review proofs routes, do we wanna follow spec proposal?
  • review lightclient routes, spec suggests to get fork version from attested_header slot, previous work from #4641? test against https://eth-light.xyz/, Access-Control-Expose-Headers: Eth-Consensus-Version header required?
  • review network identity route, use enr package / add ssz support?
  • move @param from method jsdoc to property
  • more documentation, update examples, update reasoning about API definitions
  • more http client tests, mostly in regard to retries
  • retry mechanism in http client if we receive a 415 we need to cache (per url x operationId) that server does not support it, resend different content type and on subsequent requests only sent supported content type
  • default per route config to use spec compliant content types, e.g. if spec does not define SSZ request body, do not use it by default. Should be configurable to always use SSZ in a Lodestar only setups
  • add option to server to disable JSON responses to avoid potential DoS vector, how to handle routes that do not support SSZ?
  • ensure Lodestar is compatible with latest checkpointz release (#164, #165)
  • final compatibility check with other clients and older Lodestar version using kurtosis

Closes #5128
Closes #5244
Closes #5826
Closes #6110
Closes #6564
Closes #6634

@nflaig nflaig added the meta-breaking-change Introduces breaking changes to DB, Validator, Beacon Node, or CLI interfaces. Handle with care! label May 8, 2024
Copy link
Contributor

github-actions bot commented May 8, 2024

⚠️ Performance Alert ⚠️

Possible performance regression was detected for some benchmarks.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold.

Benchmark suite Current: e84f23a Previous: 1831d47 Ratio
Map get x1000 7.1310 ns/op 0.82900 ns/op 8.60
Full benchmark results
Benchmark suite Current: e84f23a Previous: 1831d47 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 717.36 us/op 1.0257 ms/op 0.70
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 47.501 us/op 91.813 us/op 0.52
BLS verify - blst-native 1.1190 ms/op 1.1944 ms/op 0.94
BLS verifyMultipleSignatures 3 - blst-native 2.3795 ms/op 2.5098 ms/op 0.95
BLS verifyMultipleSignatures 8 - blst-native 5.2538 ms/op 5.5071 ms/op 0.95
BLS verifyMultipleSignatures 32 - blst-native 19.106 ms/op 20.004 ms/op 0.96
BLS verifyMultipleSignatures 64 - blst-native 37.582 ms/op 39.836 ms/op 0.94
BLS verifyMultipleSignatures 128 - blst-native 74.634 ms/op 77.788 ms/op 0.96
BLS deserializing 10000 signatures 854.11 ms/op 890.53 ms/op 0.96
BLS deserializing 100000 signatures 8.4223 s/op 8.9152 s/op 0.94
BLS verifyMultipleSignatures - same message - 3 - blst-native 1.1995 ms/op 1.2771 ms/op 0.94
BLS verifyMultipleSignatures - same message - 8 - blst-native 1.3619 ms/op 1.4502 ms/op 0.94
BLS verifyMultipleSignatures - same message - 32 - blst-native 2.1465 ms/op 2.2828 ms/op 0.94
BLS verifyMultipleSignatures - same message - 64 - blst-native 3.1780 ms/op 3.3959 ms/op 0.94
BLS verifyMultipleSignatures - same message - 128 - blst-native 5.2473 ms/op 5.6055 ms/op 0.94
BLS aggregatePubkeys 32 - blst-native 26.610 us/op 28.328 us/op 0.94
BLS aggregatePubkeys 128 - blst-native 102.23 us/op 110.16 us/op 0.93
notSeenSlots=1 numMissedVotes=1 numBadVotes=10 49.609 ms/op 63.617 ms/op 0.78
notSeenSlots=1 numMissedVotes=0 numBadVotes=4 48.017 ms/op 51.832 ms/op 0.93
notSeenSlots=2 numMissedVotes=1 numBadVotes=10 29.144 ms/op 31.658 ms/op 0.92
getSlashingsAndExits - default max 92.674 us/op 206.36 us/op 0.45
getSlashingsAndExits - 2k 288.96 us/op 526.77 us/op 0.55
proposeBlockBody type=full, size=empty 5.5493 ms/op 5.6800 ms/op 0.98
isKnown best case - 1 super set check 318.00 ns/op 469.00 ns/op 0.68
isKnown normal case - 2 super set checks 303.00 ns/op 401.00 ns/op 0.76
isKnown worse case - 16 super set checks 302.00 ns/op 403.00 ns/op 0.75
InMemoryCheckpointStateCache - add get delete 5.0440 us/op 6.4700 us/op 0.78
validate api signedAggregateAndProof - struct 2.5908 ms/op 2.5282 ms/op 1.02
validate gossip signedAggregateAndProof - struct 2.5227 ms/op 2.5318 ms/op 1.00
validate gossip attestation - vc 640000 1.2305 ms/op 1.2302 ms/op 1.00
batch validate gossip attestation - vc 640000 - chunk 32 171.48 us/op 169.82 us/op 1.01
batch validate gossip attestation - vc 640000 - chunk 64 156.19 us/op 154.43 us/op 1.01
batch validate gossip attestation - vc 640000 - chunk 128 143.80 us/op 147.94 us/op 0.97
batch validate gossip attestation - vc 640000 - chunk 256 140.89 us/op 138.31 us/op 1.02
pickEth1Vote - no votes 1.1501 ms/op 1.3061 ms/op 0.88
pickEth1Vote - max votes 9.5006 ms/op 7.5759 ms/op 1.25
pickEth1Vote - Eth1Data hashTreeRoot value x2048 20.891 ms/op 17.163 ms/op 1.22
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 22.641 ms/op 18.817 ms/op 1.20
pickEth1Vote - Eth1Data fastSerialize value x2048 540.88 us/op 629.05 us/op 0.86
pickEth1Vote - Eth1Data fastSerialize tree x2048 6.6546 ms/op 4.0735 ms/op 1.63
bytes32 toHexString 836.00 ns/op 454.00 ns/op 1.84
bytes32 Buffer.toString(hex) 271.00 ns/op 280.00 ns/op 0.97
bytes32 Buffer.toString(hex) from Uint8Array 568.00 ns/op 410.00 ns/op 1.39
bytes32 Buffer.toString(hex) + 0x 327.00 ns/op 282.00 ns/op 1.16
Object access 1 prop 0.23700 ns/op 0.16000 ns/op 1.48
Map access 1 prop 0.16100 ns/op 0.14400 ns/op 1.12
Object get x1000 6.6570 ns/op 7.3090 ns/op 0.91
Map get x1000 7.1310 ns/op 0.82900 ns/op 8.60
Object set x1000 74.578 ns/op 46.375 ns/op 1.61
Map set x1000 48.396 ns/op 26.952 ns/op 1.80
Return object 10000 times 0.43920 ns/op 0.25260 ns/op 1.74
Throw Error 10000 times 5.0318 us/op 3.5376 us/op 1.42
fastMsgIdFn sha256 / 200 bytes 3.0700 us/op 2.4420 us/op 1.26
fastMsgIdFn h32 xxhash / 200 bytes 473.00 ns/op 316.00 ns/op 1.50
fastMsgIdFn h64 xxhash / 200 bytes 368.00 ns/op 365.00 ns/op 1.01
fastMsgIdFn sha256 / 1000 bytes 8.8850 us/op 7.7160 us/op 1.15
fastMsgIdFn h32 xxhash / 1000 bytes 544.00 ns/op 452.00 ns/op 1.20
fastMsgIdFn h64 xxhash / 1000 bytes 472.00 ns/op 439.00 ns/op 1.08
fastMsgIdFn sha256 / 10000 bytes 71.496 us/op 66.973 us/op 1.07
fastMsgIdFn h32 xxhash / 10000 bytes 2.1470 us/op 1.9620 us/op 1.09
fastMsgIdFn h64 xxhash / 10000 bytes 1.3640 us/op 1.3320 us/op 1.02
send data - 1000 256B messages 20.799 ms/op 15.178 ms/op 1.37
send data - 1000 512B messages 27.668 ms/op 19.989 ms/op 1.38
send data - 1000 1024B messages 41.638 ms/op 27.936 ms/op 1.49
send data - 1000 1200B messages 48.658 ms/op 21.489 ms/op 2.26
send data - 1000 2048B messages 40.573 ms/op 38.635 ms/op 1.05
send data - 1000 4096B messages 39.037 ms/op 38.485 ms/op 1.01
send data - 1000 16384B messages 92.544 ms/op 84.417 ms/op 1.10
send data - 1000 65536B messages 266.21 ms/op 322.70 ms/op 0.82
enrSubnets - fastDeserialize 64 bits 1.5880 us/op 1.2680 us/op 1.25
enrSubnets - ssz BitVector 64 bits 562.00 ns/op 513.00 ns/op 1.10
enrSubnets - fastDeserialize 4 bits 234.00 ns/op 204.00 ns/op 1.15
enrSubnets - ssz BitVector 4 bits 507.00 ns/op 506.00 ns/op 1.00
prioritizePeers score -10:0 att 32-0.1 sync 2-0 211.57 us/op 202.45 us/op 1.05
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 253.60 us/op 257.98 us/op 0.98
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 524.07 us/op 340.46 us/op 1.54
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 554.00 us/op 519.15 us/op 1.07
prioritizePeers score 0:0 att 64-1 sync 4-1 1.2016 ms/op 620.46 us/op 1.94
array of 16000 items push then shift 1.7976 us/op 1.7221 us/op 1.04
LinkedList of 16000 items push then shift 10.576 ns/op 8.4480 ns/op 1.25
array of 16000 items push then pop 143.70 ns/op 143.64 ns/op 1.00
LinkedList of 16000 items push then pop 9.8460 ns/op 7.4310 ns/op 1.32
array of 24000 items push then shift 2.9821 us/op 2.7020 us/op 1.10
LinkedList of 24000 items push then shift 9.5390 ns/op 7.5680 ns/op 1.26
array of 24000 items push then pop 194.34 ns/op 178.29 ns/op 1.09
LinkedList of 24000 items push then pop 8.5180 ns/op 6.7180 ns/op 1.27
intersect bitArray bitLen 8 7.3090 ns/op 6.0160 ns/op 1.21
intersect array and set length 8 61.546 ns/op 57.157 ns/op 1.08
intersect bitArray bitLen 128 33.434 ns/op 36.208 ns/op 0.92
intersect array and set length 128 964.12 ns/op 811.92 ns/op 1.19
bitArray.getTrueBitIndexes() bitLen 128 2.7510 us/op 1.3180 us/op 2.09
bitArray.getTrueBitIndexes() bitLen 248 5.3300 us/op 2.0960 us/op 2.54
bitArray.getTrueBitIndexes() bitLen 512 9.4890 us/op 4.1650 us/op 2.28
Buffer.concat 32 items 1.0900 us/op 860.00 ns/op 1.27
Uint8Array.set 32 items 1.7850 us/op 1.7580 us/op 1.02
Buffer.copy 2.0500 us/op 1.9350 us/op 1.06
Uint8Array.set - with subarray 3.1800 us/op 2.4580 us/op 1.29
Uint8Array.set - without subarray 1.7340 us/op 1.7020 us/op 1.02
Set add up to 64 items then delete first 2.9597 us/op 2.4258 us/op 1.22
OrderedSet add up to 64 items then delete first 4.1268 us/op 3.3601 us/op 1.23
Set add up to 64 items then delete last 3.2680 us/op 2.5285 us/op 1.29
OrderedSet add up to 64 items then delete last 4.9786 us/op 3.7789 us/op 1.32
Set add up to 64 items then delete middle 3.8933 us/op 2.6221 us/op 1.48
OrderedSet add up to 64 items then delete middle 9.5427 us/op 4.8645 us/op 1.96
Set add up to 128 items then delete first 6.4875 us/op 5.2343 us/op 1.24
OrderedSet add up to 128 items then delete first 8.7866 us/op 8.4389 us/op 1.04
Set add up to 128 items then delete last 5.4154 us/op 6.4213 us/op 0.84
OrderedSet add up to 128 items then delete last 8.2181 us/op 11.490 us/op 0.72
Set add up to 128 items then delete middle 5.0981 us/op 7.3960 us/op 0.69
OrderedSet add up to 128 items then delete middle 16.407 us/op 17.591 us/op 0.93
Set add up to 256 items then delete first 12.682 us/op 16.320 us/op 0.78
OrderedSet add up to 256 items then delete first 16.862 us/op 25.965 us/op 0.65
Set add up to 256 items then delete last 9.9250 us/op 17.377 us/op 0.57
OrderedSet add up to 256 items then delete last 15.052 us/op 26.055 us/op 0.58
Set add up to 256 items then delete middle 10.598 us/op 12.880 us/op 0.82
OrderedSet add up to 256 items then delete middle 43.825 us/op 45.374 us/op 0.97
transfer serialized Status (84 B) 1.4910 us/op 1.9850 us/op 0.75
copy serialized Status (84 B) 1.3320 us/op 1.3290 us/op 1.00
transfer serialized SignedVoluntaryExit (112 B) 1.7110 us/op 2.0340 us/op 0.84
copy serialized SignedVoluntaryExit (112 B) 1.2720 us/op 1.6760 us/op 0.76
transfer serialized ProposerSlashing (416 B) 2.4460 us/op 2.4480 us/op 1.00
copy serialized ProposerSlashing (416 B) 2.2230 us/op 2.1600 us/op 1.03
transfer serialized Attestation (485 B) 2.0580 us/op 2.9640 us/op 0.69
copy serialized Attestation (485 B) 2.4300 us/op 1.9820 us/op 1.23
transfer serialized AttesterSlashing (33232 B) 2.4970 us/op 2.3240 us/op 1.07
copy serialized AttesterSlashing (33232 B) 5.4950 us/op 11.485 us/op 0.48
transfer serialized Small SignedBeaconBlock (128000 B) 2.6080 us/op 3.1210 us/op 0.84
copy serialized Small SignedBeaconBlock (128000 B) 17.921 us/op 36.542 us/op 0.49
transfer serialized Avg SignedBeaconBlock (200000 B) 3.0240 us/op 4.0810 us/op 0.74
copy serialized Avg SignedBeaconBlock (200000 B) 25.382 us/op 51.741 us/op 0.49
transfer serialized BlobsSidecar (524380 B) 3.0610 us/op 5.3190 us/op 0.58
copy serialized BlobsSidecar (524380 B) 141.11 us/op 170.88 us/op 0.83
transfer serialized Big SignedBeaconBlock (1000000 B) 3.3400 us/op 5.5620 us/op 0.60
copy serialized Big SignedBeaconBlock (1000000 B) 256.83 us/op 391.14 us/op 0.66
pass gossip attestations to forkchoice per slot 3.2949 ms/op 3.5228 ms/op 0.94
forkChoice updateHead vc 100000 bc 64 eq 0 496.61 us/op 577.32 us/op 0.86
forkChoice updateHead vc 600000 bc 64 eq 0 3.7673 ms/op 5.9359 ms/op 0.63
forkChoice updateHead vc 1000000 bc 64 eq 0 5.8118 ms/op 7.2292 ms/op 0.80
forkChoice updateHead vc 600000 bc 320 eq 0 3.5539 ms/op 3.9336 ms/op 0.90
forkChoice updateHead vc 600000 bc 1200 eq 0 3.7184 ms/op 4.3957 ms/op 0.85
forkChoice updateHead vc 600000 bc 7200 eq 0 4.7934 ms/op 5.9561 ms/op 0.80
forkChoice updateHead vc 600000 bc 64 eq 1000 11.143 ms/op 11.262 ms/op 0.99
forkChoice updateHead vc 600000 bc 64 eq 10000 11.165 ms/op 11.498 ms/op 0.97
forkChoice updateHead vc 600000 bc 64 eq 300000 16.353 ms/op 27.811 ms/op 0.59
computeDeltas 500000 validators 300 proto nodes 3.8173 ms/op 4.7317 ms/op 0.81
computeDeltas 500000 validators 1200 proto nodes 3.8890 ms/op 4.4011 ms/op 0.88
computeDeltas 500000 validators 7200 proto nodes 4.1847 ms/op 4.3780 ms/op 0.96
computeDeltas 750000 validators 300 proto nodes 6.0486 ms/op 5.8905 ms/op 1.03
computeDeltas 750000 validators 1200 proto nodes 6.1345 ms/op 6.2860 ms/op 0.98
computeDeltas 750000 validators 7200 proto nodes 6.0329 ms/op 5.7938 ms/op 1.04
computeDeltas 1400000 validators 300 proto nodes 11.572 ms/op 10.666 ms/op 1.08
computeDeltas 1400000 validators 1200 proto nodes 11.718 ms/op 10.422 ms/op 1.12
computeDeltas 1400000 validators 7200 proto nodes 11.506 ms/op 9.8196 ms/op 1.17
computeDeltas 2100000 validators 300 proto nodes 17.615 ms/op 14.477 ms/op 1.22
computeDeltas 2100000 validators 1200 proto nodes 18.186 ms/op 14.634 ms/op 1.24
computeDeltas 2100000 validators 7200 proto nodes 17.606 ms/op 15.341 ms/op 1.15
altair processAttestation - 250000 vs - 7PWei normalcase 2.5951 ms/op 1.8005 ms/op 1.44
altair processAttestation - 250000 vs - 7PWei worstcase 3.8259 ms/op 3.3565 ms/op 1.14
altair processAttestation - setStatus - 1/6 committees join 113.66 us/op 138.49 us/op 0.82
altair processAttestation - setStatus - 1/3 committees join 208.29 us/op 279.08 us/op 0.75
altair processAttestation - setStatus - 1/2 committees join 281.85 us/op 403.35 us/op 0.70
altair processAttestation - setStatus - 2/3 committees join 366.86 us/op 478.24 us/op 0.77
altair processAttestation - setStatus - 4/5 committees join 562.41 us/op 663.75 us/op 0.85
altair processAttestation - setStatus - 100% committees join 640.47 us/op 828.05 us/op 0.77
altair processBlock - 250000 vs - 7PWei normalcase 5.9900 ms/op 9.1828 ms/op 0.65
altair processBlock - 250000 vs - 7PWei normalcase hashState 29.565 ms/op 38.930 ms/op 0.76
altair processBlock - 250000 vs - 7PWei worstcase 44.265 ms/op 33.732 ms/op 1.31
altair processBlock - 250000 vs - 7PWei worstcase hashState 82.431 ms/op 99.842 ms/op 0.83
phase0 processBlock - 250000 vs - 7PWei normalcase 2.4844 ms/op 2.8316 ms/op 0.88
phase0 processBlock - 250000 vs - 7PWei worstcase 30.375 ms/op 31.345 ms/op 0.97
altair processEth1Data - 250000 vs - 7PWei normalcase 451.99 us/op 575.72 us/op 0.79
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:15 10.702 us/op 20.016 us/op 0.53
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:219 39.193 us/op 78.520 us/op 0.50
getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:42 12.930 us/op 24.386 us/op 0.53
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:18 8.1840 us/op 16.203 us/op 0.51
getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1020 153.31 us/op 232.44 us/op 0.66
getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11777 893.27 us/op 1.4294 ms/op 0.62
getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384 1.1086 ms/op 1.6577 ms/op 0.67
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384 982.71 us/op 1.6281 ms/op 0.60
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384 3.4565 ms/op 3.4342 ms/op 1.01
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384 1.9277 ms/op 3.0323 ms/op 0.64
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384 5.5892 ms/op 4.2436 ms/op 1.32
Tree 40 250000 create 503.44 ms/op 226.50 ms/op 2.22
Tree 40 250000 get(125000) 182.39 ns/op 151.80 ns/op 1.20
Tree 40 250000 set(125000) 1.7527 us/op 689.21 ns/op 2.54
Tree 40 250000 toArray() 25.409 ms/op 18.327 ms/op 1.39
Tree 40 250000 iterate all - toArray() + loop 26.128 ms/op 16.471 ms/op 1.59
Tree 40 250000 iterate all - get(i) 67.751 ms/op 59.284 ms/op 1.14
MutableVector 250000 create 17.643 ms/op 8.3733 ms/op 2.11
MutableVector 250000 get(125000) 6.8770 ns/op 6.8270 ns/op 1.01
MutableVector 250000 set(125000) 521.39 ns/op 235.54 ns/op 2.21
MutableVector 250000 toArray() 6.2158 ms/op 4.1303 ms/op 1.50
MutableVector 250000 iterate all - toArray() + loop 5.5484 ms/op 4.9632 ms/op 1.12
MutableVector 250000 iterate all - get(i) 1.7228 ms/op 1.7527 ms/op 0.98
Array 250000 create 4.8131 ms/op 3.9852 ms/op 1.21
Array 250000 clone - spread 3.9687 ms/op 1.4409 ms/op 2.75
Array 250000 get(125000) 0.47900 ns/op 0.99800 ns/op 0.48
Array 250000 set(125000) 0.50500 ns/op 1.2290 ns/op 0.41
Array 250000 iterate all - loop 92.459 us/op 168.16 us/op 0.55
effectiveBalanceIncrements clone Uint8Array 300000 66.749 us/op 33.614 us/op 1.99
effectiveBalanceIncrements clone MutableVector 300000 136.00 ns/op 311.00 ns/op 0.44
effectiveBalanceIncrements rw all Uint8Array 300000 215.82 us/op 202.44 us/op 1.07
effectiveBalanceIncrements rw all MutableVector 300000 143.00 ms/op 77.851 ms/op 1.84
phase0 afterProcessEpoch - 250000 vs - 7PWei 89.196 ms/op 92.217 ms/op 0.97
phase0 beforeProcessEpoch - 250000 vs - 7PWei 71.583 ms/op 47.568 ms/op 1.50
altair processEpoch - mainnet_e81889 540.05 ms/op 407.61 ms/op 1.32
mainnet_e81889 - altair beforeProcessEpoch 100.96 ms/op 75.233 ms/op 1.34
mainnet_e81889 - altair processJustificationAndFinalization 33.554 us/op 17.173 us/op 1.95
mainnet_e81889 - altair processInactivityUpdates 8.8985 ms/op 5.6221 ms/op 1.58
mainnet_e81889 - altair processRewardsAndPenalties 60.912 ms/op 40.128 ms/op 1.52
mainnet_e81889 - altair processRegistryUpdates 5.7510 us/op 2.4800 us/op 2.32
mainnet_e81889 - altair processSlashings 642.00 ns/op 477.00 ns/op 1.35
mainnet_e81889 - altair processEth1DataReset 1.0770 us/op 931.00 ns/op 1.16
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.5760 ms/op 1.4703 ms/op 1.07
mainnet_e81889 - altair processSlashingsReset 3.3890 us/op 4.8730 us/op 0.70
mainnet_e81889 - altair processRandaoMixesReset 7.7920 us/op 6.2030 us/op 1.26
mainnet_e81889 - altair processHistoricalRootsUpdate 528.00 ns/op 983.00 ns/op 0.54
mainnet_e81889 - altair processParticipationFlagUpdates 3.9460 us/op 2.2730 us/op 1.74
mainnet_e81889 - altair processSyncCommitteeUpdates 487.00 ns/op 1.0150 us/op 0.48
mainnet_e81889 - altair afterProcessEpoch 94.026 ms/op 100.33 ms/op 0.94
capella processEpoch - mainnet_e217614 1.5124 s/op 1.5443 s/op 0.98
mainnet_e217614 - capella beforeProcessEpoch 259.59 ms/op 299.30 ms/op 0.87
mainnet_e217614 - capella processJustificationAndFinalization 15.130 us/op 18.902 us/op 0.80
mainnet_e217614 - capella processInactivityUpdates 20.023 ms/op 17.245 ms/op 1.16
mainnet_e217614 - capella processRewardsAndPenalties 248.31 ms/op 250.60 ms/op 0.99
mainnet_e217614 - capella processRegistryUpdates 12.130 us/op 31.322 us/op 0.39
mainnet_e217614 - capella processSlashings 396.00 ns/op 649.00 ns/op 0.61
mainnet_e217614 - capella processEth1DataReset 303.00 ns/op 430.00 ns/op 0.70
mainnet_e217614 - capella processEffectiveBalanceUpdates 12.952 ms/op 4.3153 ms/op 3.00
mainnet_e217614 - capella processSlashingsReset 3.4120 us/op 3.8000 us/op 0.90
mainnet_e217614 - capella processRandaoMixesReset 3.9690 us/op 6.0340 us/op 0.66
mainnet_e217614 - capella processHistoricalRootsUpdate 604.00 ns/op 992.00 ns/op 0.61
mainnet_e217614 - capella processParticipationFlagUpdates 1.4860 us/op 1.9630 us/op 0.76
mainnet_e217614 - capella afterProcessEpoch 255.86 ms/op 275.25 ms/op 0.93
phase0 processEpoch - mainnet_e58758 417.70 ms/op 498.36 ms/op 0.84
mainnet_e58758 - phase0 beforeProcessEpoch 122.92 ms/op 156.95 ms/op 0.78
mainnet_e58758 - phase0 processJustificationAndFinalization 14.536 us/op 22.245 us/op 0.65
mainnet_e58758 - phase0 processRewardsAndPenalties 33.122 ms/op 22.159 ms/op 1.49
mainnet_e58758 - phase0 processRegistryUpdates 6.8440 us/op 13.971 us/op 0.49
mainnet_e58758 - phase0 processSlashings 323.00 ns/op 588.00 ns/op 0.55
mainnet_e58758 - phase0 processEth1DataReset 351.00 ns/op 432.00 ns/op 0.81
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 955.57 us/op 1.1432 ms/op 0.84
mainnet_e58758 - phase0 processSlashingsReset 3.1410 us/op 4.0090 us/op 0.78
mainnet_e58758 - phase0 processRandaoMixesReset 3.9690 us/op 4.9280 us/op 0.81
mainnet_e58758 - phase0 processHistoricalRootsUpdate 324.00 ns/op 430.00 ns/op 0.75
mainnet_e58758 - phase0 processParticipationRecordUpdates 2.3460 us/op 4.7500 us/op 0.49
mainnet_e58758 - phase0 afterProcessEpoch 70.963 ms/op 84.735 ms/op 0.84
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.1159 ms/op 1.3000 ms/op 0.86
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.9436 ms/op 5.3494 ms/op 0.36
altair processInactivityUpdates - 250000 normalcase 17.477 ms/op 17.397 ms/op 1.00
altair processInactivityUpdates - 250000 worstcase 18.417 ms/op 17.093 ms/op 1.08
phase0 processRegistryUpdates - 250000 normalcase 6.7810 us/op 12.195 us/op 0.56
phase0 processRegistryUpdates - 250000 badcase_full_deposits 267.28 us/op 374.11 us/op 0.71
phase0 processRegistryUpdates - 250000 worstcase 0.5 115.16 ms/op 144.93 ms/op 0.79
altair processRewardsAndPenalties - 250000 normalcase 40.928 ms/op 45.625 ms/op 0.90
altair processRewardsAndPenalties - 250000 worstcase 38.337 ms/op 44.668 ms/op 0.86
phase0 getAttestationDeltas - 250000 normalcase 7.0286 ms/op 8.6649 ms/op 0.81
phase0 getAttestationDeltas - 250000 worstcase 7.6395 ms/op 9.3644 ms/op 0.82
phase0 processSlashings - 250000 worstcase 80.419 us/op 90.393 us/op 0.89
altair processSyncCommitteeUpdates - 250000 115.88 ms/op 137.11 ms/op 0.85
BeaconState.hashTreeRoot - No change 278.00 ns/op 484.00 ns/op 0.57
BeaconState.hashTreeRoot - 1 full validator 96.103 us/op 139.48 us/op 0.69
BeaconState.hashTreeRoot - 32 full validator 1.4806 ms/op 1.5793 ms/op 0.94
BeaconState.hashTreeRoot - 512 full validator 11.141 ms/op 14.521 ms/op 0.77
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 136.96 us/op 154.61 us/op 0.89
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.4896 ms/op 2.1347 ms/op 0.70
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 17.160 ms/op 22.629 ms/op 0.76
BeaconState.hashTreeRoot - 1 balances 80.310 us/op 119.33 us/op 0.67
BeaconState.hashTreeRoot - 32 balances 736.24 us/op 1.3224 ms/op 0.56
BeaconState.hashTreeRoot - 512 balances 7.7171 ms/op 10.348 ms/op 0.75
BeaconState.hashTreeRoot - 250000 balances 152.93 ms/op 188.35 ms/op 0.81
aggregationBits - 2048 els - zipIndexesInBitList 25.779 us/op 26.606 us/op 0.97
byteArrayEquals 32 55.175 ns/op 76.758 ns/op 0.72
Buffer.compare 32 46.184 ns/op 52.083 ns/op 0.89
byteArrayEquals 1024 1.6338 us/op 2.0902 us/op 0.78
Buffer.compare 1024 54.922 ns/op 53.031 ns/op 1.04
byteArrayEquals 16384 26.587 us/op 33.277 us/op 0.80
Buffer.compare 16384 237.61 ns/op 240.68 ns/op 0.99
byteArrayEquals 123687377 198.71 ms/op 258.62 ms/op 0.77
Buffer.compare 123687377 6.9449 ms/op 8.5620 ms/op 0.81
byteArrayEquals 32 - diff last byte 54.125 ns/op 76.409 ns/op 0.71
Buffer.compare 32 - diff last byte 47.248 ns/op 52.859 ns/op 0.89
byteArrayEquals 1024 - diff last byte 1.6317 us/op 2.2269 us/op 0.73
Buffer.compare 1024 - diff last byte 54.850 ns/op 56.932 ns/op 0.96
byteArrayEquals 16384 - diff last byte 26.278 us/op 33.224 us/op 0.79
Buffer.compare 16384 - diff last byte 239.42 ns/op 232.68 ns/op 1.03
byteArrayEquals 123687377 - diff last byte 196.92 ms/op 251.80 ms/op 0.78
Buffer.compare 123687377 - diff last byte 6.4144 ms/op 8.0219 ms/op 0.80
byteArrayEquals 32 - random bytes 5.2140 ns/op 5.7060 ns/op 0.91
Buffer.compare 32 - random bytes 48.360 ns/op 52.205 ns/op 0.93
byteArrayEquals 1024 - random bytes 5.2630 ns/op 5.5290 ns/op 0.95
Buffer.compare 1024 - random bytes 46.467 ns/op 51.235 ns/op 0.91
byteArrayEquals 16384 - random bytes 5.2240 ns/op 5.5270 ns/op 0.95
Buffer.compare 16384 - random bytes 46.443 ns/op 51.360 ns/op 0.90
byteArrayEquals 123687377 - random bytes 6.4900 ns/op 8.3900 ns/op 0.77
Buffer.compare 123687377 - random bytes 48.070 ns/op 55.650 ns/op 0.86
regular array get 100000 times 33.754 us/op 45.339 us/op 0.74
wrappedArray get 100000 times 33.732 us/op 45.599 us/op 0.74
arrayWithProxy get 100000 times 12.954 ms/op 14.268 ms/op 0.91
ssz.Root.equals 46.767 ns/op 57.033 ns/op 0.82
byteArrayEquals 46.092 ns/op 53.686 ns/op 0.86
Buffer.compare 10.443 ns/op 10.997 ns/op 0.95
shuffle list - 16384 els 6.3062 ms/op 6.6871 ms/op 0.94
shuffle list - 250000 els 92.719 ms/op 99.323 ms/op 0.93
processSlot - 1 slots 12.435 us/op 15.135 us/op 0.82
processSlot - 32 slots 2.2957 ms/op 3.4792 ms/op 0.66
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 37.384 ms/op 47.169 ms/op 0.79
getCommitteeAssignments - req 1 vs - 250000 vc 2.0456 ms/op 2.6829 ms/op 0.76
getCommitteeAssignments - req 100 vs - 250000 vc 3.9183 ms/op 3.8575 ms/op 1.02
getCommitteeAssignments - req 1000 vs - 250000 vc 4.1941 ms/op 4.2412 ms/op 0.99
findModifiedValidators - 10000 modified validators 255.42 ms/op 311.01 ms/op 0.82
findModifiedValidators - 1000 modified validators 164.21 ms/op 220.13 ms/op 0.75
findModifiedValidators - 100 modified validators 150.43 ms/op 217.60 ms/op 0.69
findModifiedValidators - 10 modified validators 146.02 ms/op 191.80 ms/op 0.76
findModifiedValidators - 1 modified validators 145.47 ms/op 184.59 ms/op 0.79
findModifiedValidators - no difference 145.48 ms/op 194.63 ms/op 0.75
compare ViewDUs 2.9580 s/op 4.1051 s/op 0.72
compare each validator Uint8Array 1.6114 s/op 1.5573 s/op 1.03
compare ViewDU to Uint8Array 1.1471 s/op 1.2889 s/op 0.89
migrate state 1000000 validators, 24 modified, 0 new 629.86 ms/op 642.52 ms/op 0.98
migrate state 1000000 validators, 1700 modified, 1000 new 868.73 ms/op 916.66 ms/op 0.95
migrate state 1000000 validators, 3400 modified, 2000 new 1.2222 s/op 1.1687 s/op 1.05
migrate state 1500000 validators, 24 modified, 0 new 606.09 ms/op 675.94 ms/op 0.90
migrate state 1500000 validators, 1700 modified, 1000 new 890.61 ms/op 877.72 ms/op 1.01
migrate state 1500000 validators, 3400 modified, 2000 new 1.1509 s/op 1.1449 s/op 1.01
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 4.9000 ns/op 4.0300 ns/op 1.22
state getBlockRootAtSlot - 250000 vs - 7PWei 858.65 ns/op 658.45 ns/op 1.30
computeProposers - vc 250000 7.8902 ms/op 8.6700 ms/op 0.91
computeEpochShuffling - vc 250000 94.985 ms/op 97.930 ms/op 0.97
getNextSyncCommittee - vc 250000 127.08 ms/op 139.76 ms/op 0.91
computeSigningRoot for AttestationData 24.449 us/op 26.015 us/op 0.94
hash AttestationData serialized data then Buffer.toString(base64) 1.5814 us/op 1.6037 us/op 0.99
toHexString serialized data 914.13 ns/op 957.76 ns/op 0.95
Buffer.toString(base64) 199.73 ns/op 215.65 ns/op 0.93

by benchmarkbot/action

@nflaig nflaig marked this pull request as ready for review May 17, 2024 13:32
@nflaig nflaig requested a review from a team as a code owner May 17, 2024 13:32
@philknows philknows modified the milestone: v1.19.0 May 21, 2024
Copy link
Contributor

@nazarhussain nazarhussain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a lot to review further but sharing my initial review.

My major concern upto this point is the the lack of response codes from the routes definitions. Now use will never know any endpoint can response with 200, 206, 503, or 404.

We are putting alls calls in a promise box and saying it would be either fullfil or fail. But in case of failure user don't know what could be expected status codes.

This was a major information missing in earlier implementation and we added that in last refactor. I don't think loosing that context is a worthwhile.

}

export function createApiClientMethods<Es extends Record<string, Endpoint>>(
definitions: RouteDefinitions<Es>,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use either routesDefinitions or routes instead of just definitions for clarity. The word definitions does is not understandable in context this function is used.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We use the term "definitions" throughout the api package for route definitions, the function only has 1 line of code and we have the type name in the same line, should be clear enough.

Would prefer to keep as is for consistency

): ApiClientMethods<Es> {
return mapValues(definitions, (definition, operationId) => {
return createApiClientMethod(definition, client, operationId as string);
}) as unknown as ApiClientMethods<Es>;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anywhere you have to use unknown as that highlights some hidden type complexity which may lead to problem in future. As it's all new code written so better to identify and address such points right now.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree we should try to avoid using unknown as, I previously looked at this already and gave it another attempt today but I don't see what's the issue, the types look good to me, seems to be some issue how mapValues remaps the types, the type error is pretty strange Type 'Es[keyof Es]' is not comparable to type 'Es[K]'

We have the same type cast on unstable branch

}) as unknown as ApiWithExtraOpts<Api>;

My guess is there is some issue with the typing of mapValues that makes the translation incorrect but not sure how to fix it right now, maybe you could take a look and help with that. If it's related to mapValues type might be better to fix on unstable branch first

return client.getBlockV2(blockId, format);
},
};
export function getClient(config: ChainForkConfig, httpClient: IHttpClient): ApiClientMethods<Endpoints> {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The method name and the return type is not matching. We asked for a client and return type says it's returning client methods. Would be nicer to align the type names with the functions where these are used.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated return type name to better align with method name

} as ApiClientResponse<{[HttpStatusCode.OK]: Uint8Array}>;
}
return client.getState(stateId, format);
getState(args, init) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the second argument are request options or initialization options?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

second arg is RequestInit & LodestarExtras

eventSource.close();
onClose?.();
signal.removeEventListener("abort", close);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As it's one time listener, so need to remove. This way you can use this listener as inline similars to all others used in this file.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you suggestion to just remove the line

signal.removeEventListener("abort", close);

or anything else in addition to that?


export type Api = {
type AttestationList = ValueOf<typeof AttestationListType>;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This type is same as phase0.Attestation[] then why to declare with more complex patterns?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the idea is to always match the ssz type by deriving the ts type from it. In this specific case, I think it's even better if we stick to phase0.Attestation[] as for electra we will have to change it to allForks.Attestation[] and construct the ssz type dynamically based on the fork.

I am honestly in favor of changing it as well as you suggested, and remove all -List suffixed type unless those are more broadly used (outside routes file).

Those were added by @wemeetagain, let's get his thoughts first

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the idea is to always match the ssz type by deriving the ts type from it.

Yeah that was the idea. This is how its done in our types package, and my thought was that we may want to move all these types defined in the api over to our types package.

*/
getPoolProposerSlashings(): Promise<ApiClientResponse<{[HttpStatusCode.OK]: {data: phase0.ProposerSlashing[]}}>>;
getPoolProposerSlashings: Endpoint<
//
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
//

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above

submitPoolBlsToExecutionChange(
blsToExecutionChange: capella.SignedBLSToExecutionChange[]
): Promise<ApiClientResponse<{[HttpStatusCode.OK]: void}, HttpStatusCode.BAD_REQUEST>>;
submitPoolBLSToExecutionChange: Endpoint<
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should defer changing Bls to BLS from this PR and do it project wise later.

Copy link
Member Author

@nflaig nflaig May 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This rename is required to make the routes testable against the spec which is critical for this PR to ensure correctness.

I don't feel strongly about renaming this, it's just required here to match spec operationId.

We could open an issue to do a follow-up renaming of other parts in the code after this PR is merged.

req: {
writeReq: ({syncingStatus}) => ({query: {syncing_status: syncingStatus}}),
parseReq: ({query}) => ({syncingStatus: query.syncing_status}),
schema: {query: {syncing_status: Schema.Uint}},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Uint have different range but this parameter have different integer range 100-599.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the same as on unstable, Schema.Unit will currently be translated to {type: "integer", minimum: 0};, we can add more accurate / ranged-based schema validation for integer values but this is the only API as far as I know that requires it.

We currently already enforce that the range is valid in server implementation

if (syncingStatus != null && (syncingStatus < 100 || syncingStatus > 599)) {
throw new ApiError(400, `Invalid syncing status code: ${syncingStatus}`);

url: "/eth/v1/node/syncing",
method: "GET",
req: EmptyRequestCodec,
resp: JsonOnlyResponseCodec,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When we say our API have full SSZ support then it we should not have JsonOnlyResponseCodec expect the lodestar namespace.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the node namespace has several response schemas that don't cleanly map to ssz types.

Future steps would be to update the spec to better align to ssz everywhere.
If we require ALL endpoints to support SSZ before landing this feature, then we will likely never land this feature.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When we say our API have full SSZ support

This is not the claim of this PR, but it provides first-class support for ssz on the api. What we can claim though is that we have full ssz support on all performance relevant apis, that includes mostly beacon and validator required apis.

If we require ALL endpoints to support SSZ before landing this feature, then we will likely never land this feature.

I don't even think this is practically useful. A lot of apis are not performance sensitive or mostly consumed by clients that do not even supports ssz, think dappnode querying the peer count.

I am not saying we shouldn't push for more apis to support ssz but there are limitations to it's usefulness in most of those cases.

@nflaig
Copy link
Member Author

nflaig commented May 23, 2024

My major concern upto this point is the the lack of response codes from the routes definitions. Now use will never know any endpoint can response with 200, 206, 503, or 404.

We are putting alls calls in a promise box and saying it would be either fullfil or fail. But in case of failure user don't know what could be expected status codes.

This was a major information missing in earlier implementation and we added that in last refactor. I don't think loosing that context is a worthwhile.

We could add this by extending the Endpoint type but I honestly kinda happy that those are removed.

The reasons why I think those were not great are

  • not correctness checked against spec, e.g. getHeader claims it only returns 200 while the spec also has 204 defined (the spec also has no 404 while our types do...)
  • I have only found a single case in the code where we actually use the status code directly
  • we don't enforce those status codes anywhere to ensure implementation correctness, in current unstable, it's not even possible to set the status code from the server impl (although we can do that now)
  • the information communicated by the status code types is not very detailed, e.g. it says what status codes can be returned but not why those might be returned, making it hardly actionable without looking at the spec which contains more details

I personally don't think we should add status codes to each endpoint for reasons listed above but if we feel strongly about it could still be done.

@nflaig
Copy link
Member Author

nflaig commented May 24, 2024

Request times from validator client point of view stable (on the left) vs. feat1 (on the right)

Last 7 days, rate interval 1d

getAttesterDuties

image

getProposerDuties

image

prepareBeaconCommitteeSubnet

image

produceAttestationData

image

submitPoolAttestations

image

produceBlockV3

image

publishBlockV2

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
meta-breaking-change Introduces breaking changes to DB, Validator, Beacon Node, or CLI interfaces. Handle with care!
Projects
None yet
4 participants