Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Utility function for accuracy and l2 norm [JATIC-I2-IBM] #2302

Draft
wants to merge 2 commits into
base: dev_1.17.0
Choose a base branch
from

Conversation

kieranfraser
Copy link
Collaborator

Description

Adding an attack-agnostic metric function to return clean and robust accuracy as well as calculation of average perturbation added to images which successfully fool classification estimator.

Fixes #2301

Type of change

Please check all relevant options.

  • Improvement (non-breaking)
  • Bug fix (non-breaking)
  • New feature (non-breaking)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

Testing

Please describe the tests that you ran to verify your changes. Consider listing any relevant details of your test configuration.

  • Test A
  • Test B

Test Configuration:

  • OS
  • Python version
  • ART version or commit number
  • TensorFlow / Keras / PyTorch / MXNet version

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • My changes have been tested using both CPU and GPU devices

robust_acc = np.sum((y_pred == y_orig) & y_corr) / np.sum(y_corr)

idxs = y_pred != y
avg_perts = 0.0

Check warning

Code scanning / CodeQL

Variable defined multiple times Warning

This assignment to 'avg_perts' is unnecessary as it is
redefined
before this value is used.
@beat-buesser beat-buesser self-requested a review October 10, 2023 08:37
@beat-buesser beat-buesser self-assigned this Oct 10, 2023
@beat-buesser beat-buesser added the enhancement New feature or request label Oct 10, 2023
@beat-buesser beat-buesser added this to the ART 1.17.0 milestone Oct 10, 2023
@codecov-commenter
Copy link

codecov-commenter commented Oct 25, 2023

Codecov Report

Merging #2302 (2b77357) into dev_1.17.0 (337a15f) will decrease coverage by 1.29%.
The diff coverage is 13.33%.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Impacted file tree graph

@@              Coverage Diff               @@
##           dev_1.17.0    #2302      +/-   ##
==============================================
- Coverage       85.62%   84.34%   -1.29%     
==============================================
  Files             324      324              
  Lines           29323    29337      +14     
  Branches         5405     5406       +1     
==============================================
- Hits            25108    24744     -364     
- Misses           2837     3208     +371     
- Partials         1378     1385       +7     
Files Coverage Δ
art/metrics/metrics.py 82.82% <13.33%> (-7.12%) ⬇️

... and 17 files with indirect coverage changes

@beat-buesser beat-buesser modified the milestones: ART 1.17.0, ART 1.18.0 Dec 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
ART 1.18.0
Awaiting triage
Development

Successfully merging this pull request may close these issues.

None yet

3 participants