You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Mutation testing introduces bugs into a copy of your code and then runs your test suite to see how good it is at testing your code. However, it can be slow to rebuild the project every time a bug is introduced.
With mutation schemata, all bugs are introduced at the same time, but with flags that allow them to be enabled and disabled at run-time via environment variables. This allows for rapid re-runs of your tests with different bugs enabled. The more potential bugs muter can add to your project, the more time mutation schemata will save you.
Motivation
Mutation schemata is a well-known technique to improve the mutation testing performance.
Although it may sound weird at first, because this is an established term that is being used for years, we are going along with it.
The idea behind it is to encode all of the operators at once (known as program schemata) into a simple compiled binary (known as metaprogram) and use the runtime to enable/disable operators.
Proposed solution
For Swift projects (iOS/macOS) it is possible to have the build system (xcodebuild or swift) running tests without recompiling the app, enabling us to have the metaprogram.
Detailed design
Environment variables
Environment variables are user-definable values that can affect how a running process behaves and are part of the environment in which a process runs.
In the iOS/macOS world, we have an abstraction for that, the ProcessInfo's environment dictionary.
By using it, we can have an implementation that can change, just by manipulating the variables, for instance:
Say we have a Remove Side Effects operator that needs to be applied on a mutation test round:
Original source code:
func foo(){bar()}
Mutated source code:
func foo(){}
Using ProcessInfo, the mutated source code would look like this:
Now, we can turn the operator on and off at will.
For the operator_unique_id, we use the original source code line, column, and the utf8 offset values plus the operator id. This gives us the uniqueness we are looking for.
Schemata mappings
TODO:
Describe the steps and the new data structures like MutationSchema, [CodeBlockItemListSyntax: MutationSchemata] (a mapping), and the SchemataMutationMapping class.
Source compatibility
limitation to @resultBuilder due to the return statement that the if statement ProcessInfo.processInfo.environment introduces to the source code
ABI compatibility
None?
Implications on adoption
Requires to ignore SwiftUI code or any other code that uses the @resultBuilders (muter:enable, muter:disable may be helpful)
Future directions
Partial execution with schemata plan to have it run in parallel or other machines
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Mutation Schemata
Introduction
Mutation testing introduces bugs into a copy of your code and then runs your test suite to see how good it is at testing your code. However, it can be slow to rebuild the project every time a bug is introduced.
With mutation schemata, all bugs are introduced at the same time, but with flags that allow them to be enabled and disabled at run-time via environment variables. This allows for rapid re-runs of your tests with different bugs enabled. The more potential bugs muter can add to your project, the more time mutation schemata will save you.
Motivation
Mutation schemata is a well-known technique to improve the mutation testing performance.
Although it may sound weird at first, because this is an established term that is being used for years, we are going along with it.
The idea behind it is to encode all of the operators at once (known as
program schemata
) into a simple compiled binary (known asmetaprogram
) and use the runtime to enable/disable operators.Proposed solution
For Swift projects (iOS/macOS) it is possible to have the build system (
xcodebuild
orswift
) running tests without recompiling the app, enabling us to have themetaprogram
.Detailed design
Environment variables
Environment variables are user-definable values that can affect how a running process behaves and are part of the environment in which a process runs.
In the iOS/macOS world, we have an abstraction for that, the
ProcessInfo
'senvironment
dictionary.By using it, we can have an implementation that can change, just by manipulating the variables, for instance:
Say we have a
Remove Side Effects
operator that needs to be applied on a mutation test round:Original source code:
Mutated source code:
Using
ProcessInfo
, the mutated source code would look like this:Now, we can turn the operator on and off at will.
For the
operator_unique_id
, we use the original source code line, column, and the utf8 offset values plus the operator id. This gives us the uniqueness we are looking for.Schemata mappings
TODO:
Describe the steps and the new data structures like
MutationSchema
,[CodeBlockItemListSyntax: MutationSchemata]
(a mapping), and theSchemataMutationMapping
class.Source compatibility
return
statement that the if statementProcessInfo.processInfo.environment
introduces to the source codeABI compatibility
None?
Implications on adoption
SwiftUI
code or any other code that uses the@resultBuilders
(muter:enable
,muter:disable
may be helpful)Future directions
Alternatives considered
Acknowledgments
Beta Was this translation helpful? Give feedback.
All reactions