Introduce initial benchmarking infrastructure #464
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vobst
changed the title
vobst
force-pushed
the
initial_benchmarking_infrastructure
branch
2 times, most recently
from
6cc0a14
to
f56691c
Compare
vobst
changed the title
vobst
force-pushed
the
initial_benchmarking_infrastructure
branch
from
f56691c
to
c3762d7
Compare
On stable Rust, Criterion.rs can only use and benchmark public functions. Thus, expose some functions that will be used in the benchmarks. Signed-off-by: Valentin Obst <[email protected]>
vobst
pushed a commit
to vobst/cwe_checker
that referenced
this pull request
May 15, 2024
Link: fkie-cad#464 Signed-off-by: Valentin Obst <[email protected]>
vobst
force-pushed
the
initial_benchmarking_infrastructure
branch
from
6258bc6
to
cca43fc
Compare
added 4 commits
May 15, 2024 09:10
On stable Rust, Criterion.rs can only use and benchmark public functions. Thus, expose some functions that will be used in the benchmarks. Signed-off-by: Valentin Obst <[email protected]>
To better understand the current performance characteristics and to guide future development, introduce microbenchmarks for various steps in the `cwe_checker`'s analysis. This commit adds benchmarks for the following steps: - CFG construction, - individual normalization passes, - function signatures analysis, - pointer inference, - string abstractions, - individual checkers. All benchmarks are executed on the following input programs: - ls, - netfs.ko, for the following architectures: - amd64, - arm64, - armel, - armhf, - mips64el, - mipsel, - ppc64el, - x86. Inputs are stored in the `benches/_data/` directory. It contains json-serialized pcode projects and binaries. The pcode projects are included for the following reasons: - not requiring a Ghidra installation on the benchmarking system, - avoid that changes in Ghidra version influence the benchmark results, - reduce the time it takes to run the benchmarks. The inputs are not included in the repository and have to be obtained from an external source. Since the binaries are taken from Debian packages it shouldn't be a problem to re-distribute them if we include the copyright information. This implementation is based on the Criterion.rs crate. It is added to the dev-dependencies of the `cwe_checker_lib` and the `Cargo.lock` is updated accordingly. Signed-off-by: Valentin Obst <[email protected]>
Signed-off-by: Valentin Obst <[email protected]>
Link: fkie-cad#464 Signed-off-by: Valentin Obst <[email protected]>
vobst
force-pushed
the
initial_benchmarking_infrastructure
branch
from
cca43fc
to
c97272c
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
To better understand the current performance characteristics and to
guide future development, introduce microbenchmarks for various steps in
the
cwe_checker's analysis.This PR adds benchmarks for the following steps:
All benchmarks are executed on the following input programs:
for the following architectures:
Inputs are stored in the
benches/_data/directory. It containsjson-serialized pcode projects and binaries. The pcode projects are
included for the following reasons:
The inputs are not included in the repository and have to be obtained
from an external source. Since the binaries are taken from Debian
packages it shouldn't be a problem to re-distribute them if we include
the copyright information.
This implementation is based on the Criterion.rs crate. It is added to
the dev-dependencies of the
cwe_checker_liband theCargo.lockisupdated accordingly.
Benchmark Results
Full results for the current master are available here for future reference.
Even though both full runs were performed on the same, dedicated system with minimized background activity, there are some considerable differences in some benchmarks. I need to investigate this further to get a reproducible setup.