feat: indexing of Celo transactions #9713

fedor-ivn · 2024-03-24T09:05:54Z

Motivation

This PR adds support for custom fields from Celo blockchain and implements a
custom way to index Celo transactions.

Changelog

Enhancements

Environment Variables

New value for CHAIN_TYPE variable: celo

New variable: CELO_CORE_CONTRACTS

The CELO network includes several core contracts that are central to its operation. These core contracts manage various aspects of the Celo network, including its stablecoins, governance, identity, and more.

All these contracts can be obtained by querying the Registry contract, which is deployed at the genesis block with the address 0x000000000000000000000000000000000000ce10. This contract serves as a repository for the addresses of all core contracts on the Celo network.

While it's possible to process events from the Registry contract to keep information on core contracts up to date, implementing this in the current Blockscout indexer is not trivial. Fortunately, core contracts are not updated frequently. Thus, once fetched, the addresses typically remain unchanged. To simplify, I decided to store information about core contracts under the CELO_CORE_CONTRACTS variable, which is a JSON object of a specified structure.

In case the contracts are updated, maintaining the Blockscout instance will require updating the CELO_CORE_CONTRACTS variable and manually setting blocks for refetch (all blocks produced after the core contracts' update).

The structure of the JSON value for the CELO_CORE_CONTRACTS variable is detailed in this gist.

The JSON structure in the gist contains two main sections: addresses and events.

addresses: This section maps contract names to the list of respective addresses and the block number at which they were updated. For example:
- "Accounts": [{ "address": "0xed7f51a34b4e71fbe69b3091fcf879cd14bd73a9", "updated_at_block_number": 574 }]
- "Election": [{ "address": "0x1c3edf937cfc2f6f51784d20deb1af1f9a8655fa", "updated_at_block_number": 592 }]
events: This section maps contract addresses to the events associated with them, including any parameters and the block numbers at which these events were emitted. For example:
- "EpochRewards": { "0xb10ee11244526b94879e1956745ba2e35ae2ba20": { "CarbonOffsettingFundSet": [{ "address": "0x22579ca45ee22e2e16ddf72d955d6cf4c767b0ef", "updated_at_block_number": 15049265 }] } }
- "FeeHandler": { "0x90d94229623a0a38826a4a7557a6d79acde43f76": { "BurnFractionSet": [{ "updated_at_block_number": 19732579, "value": 0.7999999999999999 }] } }

In the next PR, there is already an implemented mix task that automatically assembles this variable by indexing the entire chain from the 0 to the latest block number.

These changes are also reflected in docs, see blockscout/docs#276.

Transactions

New fields specific to the CELO network are now supported. Here is an outline of how the fields from the RPC node are mapped to the fields in the transactions table:

feeCurrency → gas_token_contract_address_hash
gatewayFeeRecipient → gas_fee_recipient_address_hash (this field is scheduled for deprecation as stated in the CELO docs)
gatewayFee → gateway_fee (this field is scheduled for deprecation as stated in the CELO docs)

The token used for gas payment (feeCurrency) is indexed within the block fetcher process.

Token-Duality Feature

Native chain asset transfers are treated as CELO ERC-20 token transfers. These token transfers are "artificial," meaning there is no actual Transfer event emitted by the CELO ERC-20 contract that corresponds to the token transfer.

⚠️ Important Note: Since there is no actual log associated with such token transfers, these transfers are saved with a negative log index in the token_transfers table.

Indexing of native chain asset transfers as CELO ERC-20 token transfers is handled by the block fetcher and internal transactions fetcher.

API Updates

The following endpoints have been extended to include a celo element in the response, offering additional information about the token used for paying gas:

/api/v2/transactions
/api/v2/transactions/{transaction_hash}
/api/v2/addresses/{address_hash}/transactions
/api/v2/main-page/transactions

Example of the celo section in the response:

"celo": {
  "gas_token": {
    "address": "0x874069Fa1Eb16D44d622F2e0Ca25eeA172369bC1",
    "circulating_market_cap": null,
    "decimals": "18",
    "exchange_rate": null,
    "holders": "3778",
    "icon_url": null,
    "name": "Celo Dollar",
    "symbol": "cUSD",
    "total_supply": "7090075332805485657875961",
    "type": "ERC-20",
    "volume_24h": null
  }
}

Checklist for your Pull Request (PR)

If I added new functionality, I added tests covering it.
If I fixed a bug, I added a regression test to prevent the bug from silently reappearing again.
I checked whether I should update the docs and did so by submitting a PR to https://github.com/blockscout/docs
If I added/changed/removed ENV var, I submitted a PR to https://github.com/blockscout/docs to update the list of env vars at https://github.com/blockscout/docs/blob/master/for-developers/information-and-settings/env-variables.md and I updated the version to master in the Version column. Changes will be reflected in this table: https://docs.blockscout.com/for-developers/information-and-settings/env-variables.
If I add new indices into DB, I checked, that they are not redundant with PGHero or other tools

apps/indexer/test/indexer/block/fetcher_test.exs

fedor-ivn · 2024-03-26T10:11:19Z

apps/indexer/test/indexer/block/fetcher_test.exs

@@ -614,6 +607,39 @@ defmodule Indexer.Block.FetcherTest do
 assert fifth_address.fetched_coin_balance_block_number == block_number

 EthereumJSONRPC.Nethermind ->


For some reason, the tests in this file just ignore the EthereumJSONRPC variant.

In my case, this config sets the variant geth:

blockscout/apps/indexer/config/runtime/test.exs

Lines 1 to 11 in 509b897

import Config

alias EthereumJSONRPC.Variant

config :indexer, Indexer.Fetcher.Beacon.Blob.Supervisor, disabled?: true

config :indexer, Indexer.Fetcher.Beacon.Blob, start_block: 0

variant = Variant.get()

Code.require_file("#{variant}.exs", "#{__DIR__}/../../../explorer/config/test")

Code.require_file("#{variant}.exs", "#{__DIR__}/../../../indexer/config/test")

So, it loads this config, setting EthereumJSONRPC.Geth variant:

blockscout/apps/indexer/config/test/geth.exs

Lines 1 to 8 in 509b897

import Config

config :indexer,

json_rpc_named_arguments: [

transport: EthereumJSONRPC.Mox,

transport_options: [],

variant: EthereumJSONRPC.Geth

]

blockscout/apps/explorer/config/test/geth.exs

Lines 1 to 13 in 509b897

import Config

config :explorer,

json_rpc_named_arguments: [

transport: EthereumJSONRPC.Mox,

transport_options: [],

variant: EthereumJSONRPC.Geth

],

subscribe_named_arguments: [

transport: EthereumJSONRPC.Mox,

transport_options: [],

variant: EthereumJSONRPC.Geth

]

However, these json_rpc_named_arguments settings are just ignored in the tests. Why it could be so?

Because of this, I had nothing to do, but put my celo-changes to EthereumJSONRPC.Nethermind case branch

fedor-ivn · 2024-04-03T09:31:05Z

apps/explorer/lib/explorer/chain/cache/celo_core_contracts.ex

~~I think hard-coded values are fine at least for v1, because they are really not a subject to change.~~

For instance, on Celo mainnet, the last transaction to Registry contract was done 4 years ago. On alfajores, Registry contract was not even updated -- e.g it returns nil-address (0x000...) for the core contract names.

The workflow regarding core contracts was updated. Please, read the new PR description

nikitosing · 2024-04-11T16:25:14Z

apps/indexer/lib/indexer/fetcher/internal_transaction.ex

+ case tokens do
+ [token] ->


Why not to case token_transfers?

Yeah, make sense. Please, check 222feba to verify if I implemented your suggestions correctly

nikitosing · 2024-04-11T16:26:22Z

apps/indexer/lib/indexer/fetcher/internal_transaction.ex

+ defp to_address_token_balances(token_transfers, celo_token) do
+ %{token_transfers: token_transfers}
+ |> Addresses.extract_addresses()
+ |> Enum.map(fn %{fetched_coin_balance_block_number: block_number, hash: hash} ->
+ with {:ok, address_hash} <- Hash.Address.cast(hash),
+ {:ok, token_contract_address_hash} <- Hash.Address.cast(celo_token.contract_address_hash) do
+ %{
+ address_hash: address_hash,
+ token_contract_address_hash: token_contract_address_hash,
+ block_number: block_number,
+ token_type: celo_token.type,
+ token_id: nil
+ }
+ else
+ error -> Logger.error("Failed to cast string to hash: #{inspect(error)}")
+ end
+ end)
+ |> MapSet.new()
+ end
+
+ defp async_import_celo_token_balances(%{token_transfers: token_transfers, tokens: tokens}) do


I would suggest to define functions in order in which they are called. I.e. firstly define async_import_celo_token_balances and then to_address_token_balances

Also fixed in 222feba

apps/explorer/lib/explorer/chain/cache/celo_core_contracts.ex

…ield

* master: (65 commits) fix: Add healthcheck endpoints for indexer-only setup (#10076) 6.6.0 fix: Rework revert_reason (#9212) fix: Eliminate from_address_hash == #{address_hash} clause for transactions query in case of smart-contracts (#9469) feat: Add optional retry of NFT metadata fetch in Indexer.Fetcher.Tok… (#10036) fix: Separate indexer setup (#10032) feat: Blueprint contracts support (#10058) chore: Update hackney pool size: add new fetchers accounting (#9941) fix: Disallow batched queries in GraphQL endpoint (#10050) feat: Clone with immutable arguments proxy pattern (#10039) fix: vyper contracts re-verificaiton (#10053) refactor: Refactor get_additional_sources/4 -> get_additional_sources/3 (#10046) chore: Bump credo from 1.7.5 to 1.7.6 (#10060) chore: Bump redix from 1.5.0 to 1.5.1 (#10059) chore: Bump ex_doc from 0.32.1 to 0.32.2 (#10061) fix: Fix Unknown UID bug at smart-contract verification (#9986) refactor: test database config (#9662) chore: remove `has_methods` from `/addresses` (#10051) feat: Improve retry NFT fetcher (#10027) chore: Add support of Blast-specific L1 OP withdrawal events (#10049) ...

fedor-ivn · 2024-05-18T17:02:10Z

apps/indexer/test/indexer/block/realtime/fetcher_test.exs

+ # In CELO network, there is a token duality feature where CELO can be used
+ # as both a native chain currency and as an ERC-20 token. Transactions
+ # that transfer CELO are also counted as token transfers, and the
+ # TokenInstance fetcher is called. However, for simplicity, we disable it
+ # in this test.
+ Application.put_env(:indexer, Indexer.Fetcher.TokenInstance.Realtime.Supervisor, disabled?: true)


What do you think about this workaround?

@vbaranov @nikitosing

* master: refactor: Remove hardcoded numResults from fetch_pending_transactions_besu (#10117) feat: Add window between balance fetch retries for missing balanceOf tokens (#10142) fix: Add missing preloads to tokens endpoints (#10072) Explicit message on token balance update error (#10129) fix: missing nil case for revert reason (#10136) fix: Hotfix for Indexer.Fetcher.Optimism.WithdrawalEvent and EthereumJSONRPC.Receipt (#10130) hide chain specific fields behind Map.get (#10131) --- (#10096) feat: indexer for cross level messages on Arbitrum (#9312) chore: Bump ecto_sql from 3.11.1 to 3.11.2 Indexer/API separated images for Redstone Update CHANGELOG Improve response of address API to return multiple implementations for Diamond proxy (#10113) Fix GA pre-release && release workflows Update CHANGELOG feat: implement fetch_first_trace for Geth (#10087) Remove custom release CI for Immutable Update CHANGELOG for 6.6.0 Fix certified flag in the search API v2 endpoint (#10094) fix: Update Vyper inner compilers list to support all compilers (#10091)

…r` to `Transform.TokenTransfers`

fedor-ivn commented Mar 26, 2024

View reviewed changes

apps/indexer/test/indexer/block/fetcher_test.exs Outdated Show resolved Hide resolved

fedor-ivn commented Mar 26, 2024

View reviewed changes

fedor-ivn force-pushed the fi-celo-transactions branch 2 times, most recently from 7c372ce to d16cca3 Compare March 26, 2024 19:25

fedor-ivn force-pushed the fi-celo-transactions branch from a06d98f to 2e8a756 Compare April 2, 2024 20:33

fedor-ivn requested a review from nikitosing April 2, 2024 20:34

fedor-ivn commented Apr 3, 2024

View reviewed changes

fedor-ivn added the ci:celo label Apr 5, 2024

fedor-ivn force-pushed the fi-celo-transactions branch 4 times, most recently from 40f044d to a0e8cc3 Compare April 10, 2024 09:22

fedor-ivn requested a review from vbaranov April 10, 2024 18:41

nikitosing reviewed Apr 11, 2024

View reviewed changes

fedor-ivn force-pushed the fi-celo-transactions branch from 2a4851c to 3b7624a Compare April 15, 2024 10:41

fedor-ivn added ci:all and removed ci:celo labels Apr 15, 2024

fedor-ivn commented Apr 15, 2024

View reviewed changes

apps/explorer/lib/explorer/chain/cache/celo_core_contracts.ex Outdated Show resolved Hide resolved

fedor-ivn added 12 commits April 16, 2024 13:48

feat(json-rpc): support celo-specific fields in transactions

ef6c53c

chore: add new chain type to matrix builder

e31566e

feat: add celo-specific fields to transactions schema

8cf18ff

chore: add "celo" to cspell.json

8ca776a

chore: fix formatting

7c04a5b

refactor: improve naming for celo fields and ignore ethCompatible f…

161e96d

…ield

chore: add block fetcher test for celo fields

e61a36b

feat: add celo core contract abis

9b3e45a

feat: add a cache for core celo contracts

3e90794

feat: treat celo native token transfers as token transfers

3d31c9f

fix: update cspell.json with celo words

8d338e4

fix(dialyzer): pattern can never match the type

1100f96

fedor-ivn force-pushed the fi-celo-transactions branch 10 times, most recently from 3615ae2 to b6f833b Compare May 18, 2024 12:58

fedor-ivn added 2 commits May 18, 2024 13:06

fix: swap arguments in put_if_present

f927d74

fedor-ivn force-pushed the fi-celo-transactions branch from 90e5394 to b095ec3 Compare May 18, 2024 15:42

fix: indexer tests

5cf8be3

fedor-ivn added ci:all and removed ci:celo labels May 18, 2024

fedor-ivn commented May 18, 2024

View reviewed changes

fedor-ivn added 2 commits May 22, 2024 21:22

refactor: always include gas_token to the response

decbd25

feat: add gas_token to all transaction-related endpoints

18d82d1

fedor-ivn force-pushed the fi-celo-transactions branch from b095ec3 to 18d82d1 Compare May 23, 2024 12:56

fedor-ivn added 4 commits May 30, 2024 12:42

feat: celo core contracts with historical data

42569b5

fix: uncles field is expected but not present in RPC node response

a782ea9

fix: credo & format

86b86f6

fedor-ivn force-pushed the fi-celo-transactions branch from cd6f72d to 86b86f6 Compare May 30, 2024 14:59

fedor-ivn added 4 commits May 30, 2024 18:02

fix: define CELO_CORE_CONTRACTS in tests

e1553c2

fix: put gas_token under celo section

780b1fb

refactor: move token transfers filtering from `TokenTotalSupplyUpdate…

2d40d26

…r` to `Transform.TokenTransfers`

feat: add workflow to publish docker image

caed937

fedor-ivn changed the title ~~feat: index celo transactions~~ feat: indexing of Celo transactions May 31, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: indexing of Celo transactions #9713

feat: indexing of Celo transactions #9713

fedor-ivn commented Mar 24, 2024 •

edited

fedor-ivn Mar 26, 2024

fedor-ivn Mar 26, 2024 •

edited

fedor-ivn Apr 3, 2024 •

edited

fedor-ivn May 31, 2024

nikitosing Apr 11, 2024

fedor-ivn Apr 12, 2024

nikitosing Apr 11, 2024

fedor-ivn Apr 12, 2024

fedor-ivn May 18, 2024

fedor-ivn May 18, 2024

		@@ -614,6 +607,39 @@ defmodule Indexer.Block.FetcherTest do
		assert fifth_address.fetched_coin_balance_block_number == block_number

		EthereumJSONRPC.Nethermind ->

	import Config

	alias EthereumJSONRPC.Variant

	config :indexer, Indexer.Fetcher.Beacon.Blob.Supervisor, disabled?: true
	config :indexer, Indexer.Fetcher.Beacon.Blob, start_block: 0

	variant = Variant.get()

	Code.require_file("#{variant}.exs", "#{__DIR__}/../../../explorer/config/test")
	Code.require_file("#{variant}.exs", "#{__DIR__}/../../../indexer/config/test")

	import Config

	config :indexer,
	json_rpc_named_arguments: [
	transport: EthereumJSONRPC.Mox,
	transport_options: [],
	variant: EthereumJSONRPC.Geth
	]

	import Config

	config :explorer,
	json_rpc_named_arguments: [
	transport: EthereumJSONRPC.Mox,
	transport_options: [],
	variant: EthereumJSONRPC.Geth
	],
	subscribe_named_arguments: [
	transport: EthereumJSONRPC.Mox,
	transport_options: [],
	variant: EthereumJSONRPC.Geth
	]

feat: indexing of Celo transactions #9713

Are you sure you want to change the base?

feat: indexing of Celo transactions #9713

Conversation

fedor-ivn commented Mar 24, 2024 • edited

Motivation

Changelog

Enhancements

Environment Variables

Transactions

Token-Duality Feature

API Updates

Checklist for your Pull Request (PR)

Choose a reason for hiding this comment

fedor-ivn Mar 26, 2024 • edited

Choose a reason for hiding this comment

fedor-ivn Apr 3, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fedor-ivn commented Mar 24, 2024 •

edited

fedor-ivn Mar 26, 2024 •

edited

fedor-ivn Apr 3, 2024 •

edited