Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support a "known issue" test result type #226

Open
echeran opened this issue May 6, 2024 · 0 comments
Open

Support a "known issue" test result type #226

echeran opened this issue May 6, 2024 · 0 comments
Assignees

Comments

@echeran
Copy link
Collaborator

echeran commented May 6, 2024

We should add a "known issue" result type. It would correspond to when a test input for a platform/library matches criteria for an existing bug filed in the upstream project of the platform.

The benefit of having the result type is to better categorize / triage the results of our executor implementations, as we continue to update and improve them. Currently, we have test failures, and those failures could be the result of: the library (unknown bug), the library (known bug), or a deficiency in our executor. Sometimes, a known issue causes a systematic set of many test failures.

For the purposes of our project, it would help instill confidence in our self-assessment of correct executor implementation work to not include known bugs in the set of test failures.

@echeran echeran added this to the 2024 Q2 ⟨P1⟩ milestone May 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants