all: Trace types

docs/CHANGELOG.md

@@ -17,6 +17,8 @@ Test fixtures for use by clients are available for each release on the [Github r
 
 ### 🔧 EVM Tools
 
+- ✨ Better trace printing by selecting relevant trace lines depending on error type ([#474](https://github.com/ethereum/execution-spec-tests/pull/474)).
+
 ### 📋 Misc
 
 - 🐞 Fix CI by using Golang 1.21 in Github Actions to build geth ([#484](https://github.com/ethereum/execution-spec-tests/pull/484)).

docs/navigation.md

@@ -13,6 +13,7 @@
  * [Types of Test](writing_tests/types_of_tests.md)
  * [Adding a New Test](writing_tests/adding_a_new_test.md)
  * [Writing a New Test](writing_tests/writing_a_new_test.md)
+ * [Debugging Tests](writing_tests/debugging_tests.md)
  * [Referencing an EIP Spec Version](writing_tests/reference_specification.md)
  * [Verifying Changes Locally](writing_tests/verifying_changes.md)
  * [Exception Tests](writing_tests/exception_tests.md)

docs/writing_tests/debugging_tests.md

@@ -0,0 +1,95 @@
+# Debugging Tests
+
+The easiest way to debug tests while writing them is to run the test with the `--traces` flag. This will output the traces of the test, which can be used to understand the behavior of the test and the client.
+
+```bash
+fill 'tests/frontier/opcodes/test_dup.py::test_dup[fork_Byzantium-state_test-DUP1]' --traces
+```
+
+Depending on the test failure type, a list of relevant traces will be printed.
+
+For example, if the test fails because an expected storage key was not set (`Storage.KeyValueMismatch`), the output will show EVM traces where the storage key was set, or the executing contract returned with error.
+
+```python
+{
+ 1: RelevantTraceContext(
+ execution_index=0,
+ transaction_index=0,
+ transaction_hash='0xf8616cc40f214a4729758189c77720a6b672f29dcb62cd206a49ebacaf00e96f',
+ traces=[
+ EVMTraceLine(
+ pc=34,
+ op=128,
+ op_name='DUP1',
+ gas=78949,
+ gas_cost=3,
+ mem_size=0,
+ stack=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16],
+ depth=1,
+ refund=0,
+ context_address='0x0000000000000000000000000000000000000100'
+ ),
+ EVMTraceLine(
+ pc=35,
+ op=96,
+ op_name='PUSH1',
+ gas=78946,
+ gas_cost=3,
+ mem_size=0,
+ stack=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 16],
+ depth=1,
+ refund=0,
+ context_address='0x0000000000000000000000000000000000000100'
+ ),
+ EVMTraceLine(
+ pc=37,
+ op=85,
+ op_name='SSTORE',
+ gas=78943,
+ gas_cost=20000,
+ mem_size=0,
+ stack=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 16, 0],
+ depth=1,
+ refund=0,
+ context_address='0x0000000000000000000000000000000000000100'
+ )
+ ],
+ description='SSTORE event on address 0x0000000000000000000000000000000000000100 and key 0x0'
+ ),
+ 2: RelevantTraceContext(
+ execution_index=0,
+ transaction_index=0,
+ transaction_hash='0xf8616cc40f214a4729758189c77720a6b672f29dcb62cd206a49ebacaf00e96f',
+ traces=[
+ EVMTraceLine(
+ pc=44,
+ op=96,
+ op_name='PUSH1',
+ gas=18937,
+ gas_cost=3,
+ mem_size=0,
+ stack=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14],
+ depth=1,
+ refund=0,
+ context_address='0x0000000000000000000000000000000000000100'
+ ),
+ EVMTraceLine(
+ pc=46,
+ op=85,
+ op_name='SSTORE',
+ gas=18934,
+ gas_cost=20000,
+ mem_size=0,
+ stack=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 3],
+ depth=1,
+ refund=0,
+ context_address='0x0000000000000000000000000000000000000100'
+ ),
+ EVMCallFrameExit(from_address='0x0000000000000000000000000000000000000100', output=b'', gas_used=79000, error='out of gas')
+ ],
+ description='Exit frame from address 0x0000000000000000000000000000000000000100 with error'
+ )
+}
+```
+
+In this example, the test failed because the storage key `0x0` was not set. The traces show that the storage key was indeed set to `0x10`, but later the contract ran out of gas.

setup.cfg

@@ -33,6 +33,8 @@ install_requires =
  coincurve>=18.0.0,<19
  trie==2.1.1
  semver==3.0.1
+ pydantic>=2.6.3
+ rich>=13.7.0,<14
  click>=8.0.0,<9
 
 [options.package_data]
@@ -61,7 +63,7 @@ test =
 
 lint =
  isort>=5.8,<6
- mypy==0.982; implementation_name == "cpython"
+ mypy==0.991; implementation_name == "cpython"
  types-requests
  black==22.3.0; implementation_name == "cpython"
  flake8-spellcheck>=0.24,<0.25

src/ethereum_test_tools/common/__init__.py

@@ -1,6 +1,7 @@
 """
 Common definitions and types.
 """
+
 from .base_types import (
  Address,
  Bloom,
@@ -41,6 +42,7 @@
  JSONEncoder,
  Removable,
  Storage,
+ TraceableException,
  Transaction,
  Withdrawal,
  alloc_to_accounts,
@@ -75,6 +77,7 @@
  "TestParameterGroup",
  "TestPrivateKey",
  "TestPrivateKey2",
+ "TraceableException",
  "Transaction",
  "Withdrawal",
  "ZeroPaddedHexNumber",