Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add logic test support #555

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Add logic test support #555

wants to merge 5 commits into from

Conversation

lthrockmorton
Copy link

@lthrockmorton lthrockmorton commented Jun 15, 2023

Adds logic test support to Bluepill, so that unit tests can be run without an app host. Resolves #531.

Leverages CoreSimulator's private spawn method to run the xctest executable on individual logic test bundles. Doing so requires adjustments to various inputs (e.g. the xctest executable expects the list of tests in a different format) as well as different error handling (based on the nuances of how errors/crashes surface from the spawn method.

Also fixes a bug where retrying tests after a crash was being dictated by the wrong config variable (onlyRetryFailed rather than retryAppCrashTests).

Tests were added to validate:

  • happy path testing
  • sad paths (failure/timeout/crashes)
  • timeouts for both individual test cases + execution timeouts
  • retries
  • swift vs objc tests (requires name transformation for swift)
  • testing multiple test bundles

Logic test execution time data was collected and available here.

Note: should be tested with consuming app (Voyager iOS) before committing, which requires a pending change in that app first. In the mean time, this change is only internally tested.

@lthrockmorton lthrockmorton requested a review from a team as a code owner June 15, 2023 00:50
@dostrander
Copy link
Contributor

Very cool! This seems like it'd satisfy #531

@lthrockmorton
Copy link
Author

Very cool! This seems like it'd satisfy #531

Yes! Very excited to support this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Proposal: Allow Bluepill to run non-app hosted tests
2 participants