Benchmark and Profiling - alignment on the work to be done

[csgui]

1. Context

   Benchmarking and profiling work - with the Nakamoto release as focus - need be initiated. Thus, this thread has the goal to bring the needs and ideas related with that subject to discussion.

   Profiling and benchmarking are both techniques used in software development and performance analysis, but they serve different purposes and focus on different aspects of a system.

   • Profiling

     Purpose: Is the process of analyzing a program's runtime behavior to identify bottlenecks, performance issues, or areas where optimization is needed.

     Focus: Profiling provides information about the time spent in various parts of the code, including function calls, loops, and memory usage. It helps developers understand where the program spends most of its time during execution.

     Tools: Profiling tools gather data on the execution of a program, such as function call times, memory usage, and CPU utilization. This information helps developers pinpoint areas that may need optimization.

   • Benchmarking

     Purpose: Is the process of comparing the performance of one system, component, or software against another to assess relative speed, efficiency, or other characteristics.

     Focus: Benchmarking focuses on measuring and comparing the overall performance of systems or specific components under standardized conditions. It is often used to evaluate the performance of different algorithms, hardware configurations, or software implementations.

     Tools: Benchmarking tools create controlled environments to assess the performance of specific tasks or workloads. The results are used to compare different systems or configurations objectively.

2. Areas of interest

   • General Costs Limit
     • Update https://github.com/stacks-network/clarity-benchmarking
       • It should work with stacks-core clarity-wasm-next branch.
     • Costs limit:
       • 30 seconds time limit should be adjust accordingly to the Nakamoto block time generation.
       • Benchmark the existing interpreter with the purpose of updating the cost limits if epoch 3.0 initially launches with the interpreter.
     • Requirements:
       • Hardware requirements to run a Stacks node are low. For instance, a raspberry-pi should be able to run a node.
   • Clarity - current interpreter
     • Benchmark (setup and teardown processes should be excluded from the results of the cost limit execution time).
   • Clarity-Wasm VM
     • Profiling/Benchmark the VM to identify bottlenecks, especially looking at setup/teardown and host interface.
     • Profiling of simple contracts.
     • Profiling of complex contracts
       • heavy on read IO band: Planbetter, FastPool
       • Clarity corner cases: Arkadiko, Alex
       • heavy on computing band: Pyth Oracle
   • Expected performance increase
     • on read-count by at least a factor of 10
   • Miscellaneous
     • Evaluate tenure_extension transaction and its impact on the costs limit.
     • Database profiling as described here: Chainstate DB performance #3777. Lower priority but can be done in parallel.

3. Ramp-up plan

   • Fork feat/next-benchmarking from stacks-core branch next.
   • Update branch costs-3 of clarity-benchmarking and feat/next-benchmarking of stacks-core so it runs.
   • Merge feat/next-benchmarking into next.
     • We shouldn't need a dedicated branch to run benchmarking.
     • Benchmarking-specific code can be gated with #[cfg(any(test, feature = "testing"))].
   • Run benchmarks on next.
   • Merge next into feat/clarity-wasm-next.
   • Run benchmarks on feat/clarity-wasm-next.
   • Completely ignore develop.

[xoloki]

@AshtonStephens has asked me to take this on, so I'll be reviewing the current state today and getting started on making progress next week.

[jcnelson]

The branch https://github.com/stacks-network/stacks-core/tree/test/replay-block-2.4.0.1.0 adds a replay-block subcommand to stacks-inspect, which will let you re-evaluate a given Stacks block (e.g. so you can perf it).