Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add logic test support #567

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Conversation

lthrockmorton
Copy link

Adds logic test support to Bluepill, so that unit tests can be run without an app host. Resolves #531.

Leverages CoreSimulator's private spawn method to run the xctest executable on individual logic test bundles. Doing so requires adjustments to various inputs (e.g. the xctest executable expects the list of tests in a different format) as well as different error handling (based on the nuances of how errors/crashes surface from the spawn method.

A number of critical pieces of functionality were added to support this:

  1. Logic tests run using a different command than traditional unit or scenario tests (xcrun simctl spawn -s xctest).
  • Reverse engineering of Apple's CoreSimulator framework was required to use the private spawnAsyncWithPath: method to spawn a logic test execution.
  • Implementation details can be found in BPSimulator's executeLogicTestsWithParser method.
  1. The xctest API does not support a test skip-list, and instead only provides an opt-in list. Bluepill has long had a hacky, broken solution for identifying all tests in a test bundle.
  • The fix is to inject a dylib directly into the xctest execution to pipe out test data. Full context on the problem and solution are detailed in this document
  • The injected library is BPTestInspector, a new project in the BP workspace. It is injected during BPSimulator's collectTestSuiteInfoWithCompletion method.
  • Making this dylib available and linked to both the Bluepill and xctest executions requires some additional env setup, both in objc code (for xctest) and Bazel (for Bluepill)
  1. When running Xcode in Rosetta mode (as is the case for many consumers), there is a weird interaction with spawned executions -- they will try to run universal binaries in arm64 rather than x86.
  • As a result, we have to adapt the xctest executable to the desired architecture before running it.
  • Impl details are in BPUtils's lipoExecutableAtPath:

Also fixes a bug where retrying tests after a crash was being dictated by the wrong config variable (onlyRetryFailed rather than retryAppCrashTests).

Tests were added to validate:

  • happy path testing
  • sad paths (failure/timeout/crashes)
  • timeouts for both individual test cases + execution timeouts
  • retries
  • swift vs objc tests (requires name transformation for swift)
  • testing multiple test bundles
  • Logic test execution time data was collected and available here.

Additionally, a new logic test batch was created in a consuming app (Voyager iOS), and tests passed. Notably, the new BPTestInspector code correctly omitted hundreds of other legacy false positive test method names, and only the desired tests were treated as tests.

@lthrockmorton lthrockmorton requested a review from a team as a code owner December 21, 2023 01:29
@lthrockmorton lthrockmorton force-pushed the lthrockm/logicTestClean branch 9 times, most recently from 2e12142 to d3ec5a2 Compare December 21, 2023 06:36
Signed-off-by: lthrockmorton <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Proposal: Allow Bluepill to run non-app hosted tests
1 participant