Scan Does Not Output Tests Results

[patmalt]

New Issue Checklist

• Updated fastlane to the latest version
• I read the Contribution Guidelines
• I read docs.fastlane.tools
• I searched for existing GitHub issues

Issue Description

When running scan, I do not see the test results table explaining the number of successful/failing tests.
If a test fails, I get status code 65.

Complete output when running fastlane, including the stack trace and command used

bundle exec fastlane test_ui_dev
[09:42:46]: -----------------------------
[09:42:46]: --- Step: update_fastlane ---
[09:42:46]: -----------------------------
[09:42:46]: Looking for updates for fastlane...
[09:42:49]: Nothing to update ✅
[09:42:49]:
[09:42:49]: Please help us test early releases of fastlane by opting into nightly builds 🌃
[09:42:49]: Just replace your `update_fastlane` call with
[09:42:49]:
[09:42:49]: ▸ update_fastlane(nightly: true)
[09:42:49]:
[09:42:49]: Nightly builds are reviewed and tested just like the public releases 🚂
[09:42:49]:
[09:42:49]: -------------------------------------------------
[09:42:49]: --- Step: Verifying required fastlane version ---
[09:42:49]: -------------------------------------------------
[09:42:49]: Your fastlane version 2.61.0 matches the minimum requirement of 2.60.0  ✅
[09:42:49]: ------------------------------
[09:42:49]: --- Step: default_platform ---
[09:42:49]: ------------------------------
[09:42:49]: Driving the lane 'ios test_ui_dev' 🚀
[09:42:49]: ------------------
[09:42:49]: --- Step: scan ---
[09:42:49]: ------------------
[09:42:49]: $ xcodebuild -list -project ./App.xcodeproj
[09:42:51]: $ xcodebuild -showBuildSettings -scheme App_UITests -project ./App.xcodeproj
[09:42:52]: $ xcodebuild -list -project ./App.xcodeproj
[09:42:53]: $ xcodebuild -showBuildSettings -scheme App_UITests -project ./App.xcodeproj

+------------------------+--------------------------------------------------+
|                          Summary for scan 2.61.0                          |
+------------------------+--------------------------------------------------+
| clean                  | true                                             |
| device                 | iPhone 6s                                        |
| scheme                 | App_UITests                             |
| project                | ./App.xcodeproj                         |
| derived_data_path      | /Users/patmalt/Library/Developer/Xcode/DerivedD  |
|                        | ata/App-hlgbzztiheiiaqgdasgtubtpaeff    |
| skip_build             | false                                            |
| output_directory       | ./fastlane/test_output                           |
| output_types           | html,junit                                       |
| buildlog_path          | ~/Library/Logs/scan                              |
| include_simulator_logs | false                                            |
| open_report            | false                                            |
| skip_slack             | false                                            |
| slack_only_on_failure  | false                                            |
| use_clang_report_name  | false                                            |
| fail_build             | true                                             |
| xcode_path             | /Applications/Xcode.app                          |
+------------------------+--------------------------------------------------+

[09:42:54]: $ set -o pipefail && env NSUnbufferedIO=YES xcodebuild -scheme App_UITests -project ./App.xcodeproj -destination 'platform=iOS Simulator,id=94F7EF96-C4A0-4897-A215-86A28D87B972' -derivedDataPath '/Users/patmalt/Library/Developer/Xcode/DerivedData/App-hlgbzztiheiiaqgdasgtubtpaeff' clean build test | tee '/Users/patmalt/Library/Logs/scan/App-App_UITests.log' | xcpretty  --report html --output '/Users/patmalt/Dev/mobile-app/fastlane/test_output/report.html' --report junit --output '/Users/patmalt/Dev/mobile-app/fastlane/test_output/report.junit' --report junit --output '/var/folders/72/qdk6z07d59z7ch4683qwnzcm0000gn/T/junit_report20171010-51393-xr1z6h'
[09:42:54]: ▸ Loading...
[09:42:55]: ▸ Cleaning App/AppTodayExtension [Debug DevEnv]
[09:42:55]: ▸ Check Dependencies
[09:42:56]: ▸ Cleaning App/App [Debug DevEnv]
[09:42:56]: ▸ Check Dependencies
[09:42:56]: ▸ Clean Succeeded
[09:42:56]: ▸ Building App/AppTodayExtension [Debug DevEnv]
[09:42:56]: ▸ Check Dependencies
...
[09:43:05]: Running Tests: ▸ Touching AppTodayExtension.appex
[09:43:05]: ▸ Signing /Users/patmalt/Library/Developer/Xcode/DerivedData/App-hlgbzztiheiiaqgdasgtubtpaeff/Build/Products/Debug\ DevEnv-iphonesimulator/AppTodayExtension.appex
[09:43:05]: ▸ Building App/App [Debug DevEnv]
[09:43:05]: ▸ Check Dependencies
...
▸ Touching AppTodayExtension.appex
▸ Signing /Users/patmalt/Library/Developer/Xcode/DerivedData/App-hlgbzztiheiiaqgdasgtubtpaeff/Build/Products/Debug\ DevEnv-iphonesimulator/AppTodayExtension.appex
▸ Building App/App [Debug DevEnv]
▸ Checking Dependencies
...
▸ Processing Info.plist
▸ Running script 'Setup Firebase'
▸ Copying /Users/patmalt/Library/Developer/Xcode/DerivedData/App-hlgbzztiheiiaqgdasgtubtpaeff/Build/Products/Debug\ DevEnv-iphonesimulator/AppTodayExtension.appex
▸ Touching App.app
▸ Signing /Users/patmalt/Library/Developer/Xcode/DerivedData/App-hlgbzztiheiiaqgdasgtubtpaeff/Build/Products/Debug\ DevEnv-iphonesimulator/App.app
▸ Build Succeeded
2017-10-10 09:45:22.161 xcodebuild[51426:422277]  IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/var/folders/72/qdk6z07d59z7ch4683qwnzcm0000gn/T/com.apple.dt.XCTest/IDETestRunSession-80F77F26-6E9B-4075-A430-D40CADADD82D/AppUITests-CA429EF8-281C-4221-9AE9-532B51D1CC52/Session-AppUITests-2017-10-10_094522-Oi4OAs.log
2017-10-10 09:45:22.162 xcodebuild[51426:422273] [MT] IDETestOperationsObserverDebug: (4F22A563-AF79-4C72-8FC5-E8A9A49B8A40) Beginning test session AppUITests-4F22A563-AF79-4C72-8FC5-E8A9A49B8A40 at 2017-10-10 09:45:22.162 with Xcode 9A235 on target <DVTiPhoneSimulator: 0x7fcc3bf301a0> {
SimDevice: iPhone 6s (94F7EF96-C4A0-4897-A215-86A28D87B972, iOS 11.0, Booted)
} (11.0 (15A372))
▸ Building App/AppTodayExtension [Debug DevEnv]
▸ Check Dependencies
▸ Building App/App [Debug DevEnv]
▸ Check Dependencies
▸ Running script 'Copy Carthage Frameworks'
▸ Running script 'Setup Firebase'
▸ Building App/AppUITests [Debug DevEnv]
▸ Check Dependencies
▸ Running script 'Copy Carthage Frameworks'
All tests
AppUITests-Runner.app
CCIAgendaDetails
✗ agendaDetailsAreDisplayedOnTheAgendaDetailsScreen, failed - Timed out waiting for condition
** TEST FAILED **
[09:46:33]: Exit status: 65
+----------------------------+--------------------------------------------------+
|                                 Lane Context                                  |
+----------------------------+--------------------------------------------------+
| DEFAULT_PLATFORM           | ios                                              |
| PLATFORM_NAME              | ios                                              |
| LANE_NAME                  | ios test_ui_dev                                  |
| SCAN_DERIVED_DATA_PATH     | /Users/patmalt/Library/Developer/Xcode/DerivedD  |
|                            | ata/App-hlgbzztiheiiaqgdasgtubtpaeff    |
| SCAN_GENERATED_PLIST_FILES | ["/Users/patmalt/Library/Developer/Xcode/Derive  |
|                            | dData/App-hlgbzztiheiiaqgdasgtubtpaeff  |
|                            | /Logs/Test/AC6DD6CB-20FB-4A7D-AB27-10B2047C4787  |
|                            | _TestSummaries.plist"]                           |
| SCAN_GENERATED_PLIST_FILE  | /Users/patmalt/Library/Developer/Xcode/DerivedD  |
|                            | ata/App-hlgbzztiheiiaqgdasgtubtpaeff/L  |
|                            | ogs/Test/AC6DD6CB-20FB-4A7D-AB27-10B2047C4787_T  |
|                            | estSummaries.plist                               |
+----------------------------+--------------------------------------------------+
[09:46:33]: Error building/testing the application - see the log above

+------+------------------------------+-------------+
|                 fastlane summary                  |
+------+------------------------------+-------------+
| Step | Action                       | Time (in s) |
+------+------------------------------+-------------+
| 1    | update_fastlane              | 3           |
| 2    | Verifying required fastlane  | 0           |
|      | version                      |             |
| 3    | default_platform             | 0           |
| 💥   | scan                         | 223         |
+------+------------------------------+-------------+

[09:46:33]: fastlane finished with errors

[!] Error building/testing the application - see the log above

```ruby
/Library/Ruby/Gems/2.0.0/gems/fastlane-2.61.0/fastlane/lib/fastlane/actions/actions_helper.rb:48:in `execute_action'
/Library/Ruby/Gems/2.0.0/gems/fastlane-2.61.0/fastlane/lib/fastlane/runner.rb:230:in `block in execute_action'
/Library/Ruby/Gems/2.0.0/gems/fastlane-2.61.0/fastlane/lib/fastlane/runner.rb:226:in `chdir'
/Library/Ruby/Gems/2.0.0/gems/fastlane-2.61.0/fastlane/lib/fastlane/runner.rb:226:in `execute_action'
/Library/Ruby/Gems/2.0.0/gems/fastlane-2.61.0/fastlane/lib/fastlane/runner.rb:148:in `trigger_action_by_name'
/Library/Ruby/Gems/2.0.0/gems/fastlane-2.61.0/fastlane/lib/fastlane/fast_file.rb:146:in `method_missing'
Fastfile:49:in `block (2 levels) in parsing_binding'
/Library/Ruby/Gems/2.0.0/gems/fastlane-2.61.0/fastlane/lib/fastlane/lane.rb:33:in `call'
/Library/Ruby/Gems/2.0.0/gems/fastlane-2.61.0/fastlane/lib/fastlane/lane.rb:33:in `call'
/Library/Ruby/Gems/2.0.0/gems/fastlane-2.61.0/fastlane/lib/fastlane/runner.rb:49:in `block in execute'
/Library/Ruby/Gems/2.0.0/gems/fastlane-2.61.0/fastlane/lib/fastlane/runner.rb:45:in `chdir'
/Library/Ruby/Gems/2.0.0/gems/fastlane-2.61.0/fastlane/lib/fastlane/runner.rb:45:in `execute'
/Library/Ruby/Gems/2.0.0/gems/fastlane-2.61.0/fastlane/lib/fastlane/lane_manager.rb:52:in `cruise_lane'
/Library/Ruby/Gems/2.0.0/gems/fastlane-2.61.0/fastlane/lib/fastlane/command_line_handler.rb:30:in `handle'
/Library/Ruby/Gems/2.0.0/gems/fastlane-2.61.0/fastlane/lib/fastlane/commands_generator.rb:104:in `block (2 levels) in run'
/Library/Ruby/Gems/2.0.0/gems/commander-fastlane-4.4.5/lib/commander/command.rb:178:in `call'
/Library/Ruby/Gems/2.0.0/gems/commander-fastlane-4.4.5/lib/commander/command.rb:178:in `call'
/Library/Ruby/Gems/2.0.0/gems/commander-fastlane-4.4.5/lib/commander/command.rb:153:in `run'
/Library/Ruby/Gems/2.0.0/gems/commander-fastlane-4.4.5/lib/commander/runner.rb:476:in `run_active_command'
/Library/Ruby/Gems/2.0.0/gems/fastlane-2.61.0/fastlane_core/lib/fastlane_core/ui/fastlane_runner.rb:64:in `run!'
/Library/Ruby/Gems/2.0.0/gems/commander-fastlane-4.4.5/lib/commander/delegates.rb:15:in `run!'
/Library/Ruby/Gems/2.0.0/gems/fastlane-2.61.0/fastlane/lib/fastlane/commands_generator.rb:303:in `run'
/Library/Ruby/Gems/2.0.0/gems/fastlane-2.61.0/fastlane/lib/fastlane/commands_generator.rb:42:in `start'
/Library/Ruby/Gems/2.0.0/gems/fastlane-2.61.0/fastlane/lib/fastlane/cli_tools_distributor.rb:66:in `take_off'
/Library/Ruby/Gems/2.0.0/gems/fastlane-2.61.0/bin/fastlane:20:in `<top (required)>'
/usr/local/bin/fastlane:23:in `load'
/usr/local/bin/fastlane:23:in `<main>'

Environment

✅ fastlane environment ✅

Stack

Key	Value
OS	10.13
Ruby	2.4.1
Bundler?	true
Git	git version 2.13.5 (Apple Git-94)
Installation Source	~/.rvm/gems/ruby-2.4.1/bin/fastlane
Host	Mac OS X 10.13 (17A405)
Ruby Lib Dir	~/.rvm/rubies/ruby-2.4.1/lib
OpenSSL Version	OpenSSL 1.1.0f 25 May 2017
Is contained	false
Is homebrew	false
Is installed via Fabric.app	false
Xcode Path	/Applications/Xcode.app/Contents/Developer/
Xcode Version	9.0

System Locale

Variable	Value
LANG	en_US.UTF-8	✅
LC_ALL
LANGUAGE

fastlane files:

`./fastlane/Fastfile`

# Customise this file, documentation can be found here:
# https://github.com/fastlane/fastlane/tree/master/fastlane/docs
# All available actions: https://docs.fastlane.tools/actions
# can also be listed using the `fastlane actions` command

# Change the syntax highlighting to Ruby
# All lines starting with a # are ignored when running `fastlane`

# If you want to automatically update fastlane if a new version is available:
update_fastlane

# This is the minimum version number required.
# Update this, if you use features of a newer version
fastlane_version "2.60.0"

default_platform :ios

platform :ios do

  desc "Run the UI Tests in the DEV environment"
  lane :test_ui_dev do
    scan(
      clean: true,
      device: 'iPhone 6s',
      scheme: 'App_UITests'
    )
  end
end

### fastlane gems

| Gem      | Version | Update-Status |
| -------- | ------- | ------------- |
| fastlane | 2.61.0  | ✅ Up-To-Date  |


### Loaded fastlane plugins:

**No plugins Loaded**


<details><summary><b>Loaded gems</b></summary>

| Gem                       | Version      |
| ------------------------- | ------------ |
| did_you_mean              | 1.1.0        |
| io-console                | 0.4.6        |
| CFPropertyList            | 2.3.5        |
| public_suffix             | 2.0.5        |
| addressable               | 2.5.2        |
| babosa                    | 1.0.2        |
| bundler                   | 1.15.4       |
| claide                    | 1.0.2        |
| colored                   | 1.2          |
| colored2                  | 3.1.2        |
| highline                  | 1.7.8        |
| commander-fastlane        | 4.4.5        |
| declarative               | 0.0.10       |
| declarative-option        | 0.1.0        |
| unf_ext                   | 0.0.7.4      |
| unf                       | 0.1.4        |
| domain_name               | 0.5.20170404 |
| dotenv                    | 2.2.1        |
| excon                     | 0.59.0       |
| multipart-post            | 2.0.0        |
| faraday                   | 0.13.1       |
| http-cookie               | 1.0.3        |
| faraday-cookie_jar        | 0.0.6        |
| faraday_middleware        | 0.12.2       |
| fastimage                 | 2.1.0        |
| gh_inspector              | 1.0.3        |
| jwt                       | 1.5.6        |
| little-plugger            | 1.1.4        |
| multi_json                | 1.12.2       |
| logging                   | 2.2.2        |
| memoist                   | 0.16.0       |
| os                        | 0.9.6        |
| signet                    | 0.7.3        |
| googleauth                | 0.5.3        |
| httpclient                | 2.8.3        |
| mime-types-data           | 3.2016.0521  |
| mime-types                | 3.1          |
| uber                      | 0.1.0        |
| representable             | 3.0.4        |
| retriable                 | 3.1.1        |
| google-api-client         | 0.13.6       |
| json                      | 2.1.0        |
| mini_magick               | 4.5.1        |
| multi_xml                 | 0.6.0        |
| plist                     | 3.3.0        |
| rubyzip                   | 1.2.1        |
| security                  | 0.1.3        |
| slack-notifier            | 1.5.1        |
| terminal-notifier         | 1.8.0        |
| unicode-display_width     | 1.3.0        |
| terminal-table            | 1.8.0        |
| tty-screen                | 0.5.0        |
| word_wrap                 | 1.0.0        |
| nanaimo                   | 0.2.3        |
| xcodeproj                 | 1.5.2        |
| rouge                     | 2.0.7        |
| xcpretty                  | 0.2.8        |
| xcpretty-travis-formatter | 0.0.4        |
</details>


*generated on:* **2017-10-10**
</details>

[fastlane-bot]

It seems like this issue might be related to code signing 🚫

Have you seen our new Code Signing Troubleshooting Guide? It will help you resolve the most common code signing issues 👍

[patmalt]

This issue is not related to code signing.
Looks like there is a missing method being called somewhere in fastlane

[SeanRobinson159]

I am having the same issue. If all of the tests succeed, it still doesn't list the tests.

[alexandreblin]

I'm having the same issue, no tests are reported in report.html and report.junit. The shell output used to display a table with the results, and now it doesn't.

[ohayon]

Hey everyone- does this reliably happen every time a test fails?

[SeanRobinson159]

Yes. I don't get reports anymore. Whether the tests succeed or fail. Just like @alexandreblin said, there used to be a nice table displayed in the shell output, but not anymore.

[ohayon]

Hmmm yeah there should be but Error 65 is different from a failing test case so fastlane just crashes there. If feels like the simulator is timing out given that that is usually what causes error 65's and this line Timed out waiting for condition in your output. Is it possible that you could comment out that failing test and give it another shot? I'd also be curious if the same thing happens if you have a test that is made to fail like assert(false).

[SeanRobinson159]

This is the result from a test with assert(false) (In my project)

[09:10:19]	▸ Check Dependencies
[09:10:19]	▸ Running script 'Run Script'
[09:10:19]	Failing tests:
[09:10:19]	ALoginUITests.testForgotPassword()
[09:10:19]	** TEST FAILED **
[09:10:19]	[09:10:19]: Exit status: 65
[09:10:25]	[09:10:25]: -------------------
[09:10:25]	[09:10:25]: --- Step: slack ---
[09:10:25]	[09:10:25]: -------------------
[09:10:27]	[09:10:27]: Successfully sent Slack notification

[SeanRobinson159]

Sometimes I get this result. I am not sure why there are different results.

[20:56:32]	▸ Check Dependencies
[20:56:32]	▸ Running script 'Run Script'
[20:56:32]	** TEST FAILED **
[20:56:32]	[20:56:32]: Exit status: 65
[20:56:37]	[20:56:37]: -------------------
[20:56:37]	[20:56:37]: --- Step: slack ---
[20:56:37]	[20:56:37]: -------------------
[20:56:38]	[20:56:38]: Successfully sent Slack notification

[ohayon]

Hmmm, hey @SeanRobinson159 - could you try resetting your simulators and giving this another shot? Error 65 has always been a bit allusive, but generally it has something to do with something going wrong in the simulator.

[SeanRobinson159]

Funny you say that, I just tried that this morning. Same results.
If I run the tests using a code they all pass (same simulators).

[ohayon]

Could you just elaborate what you mean by "If I run the tests using a code they all pass"?

[SeanRobinson159]

Sorry, autocorrect. It was supposed to say: “...using Xcode they all pass.”

[ohayon]

Hmmm interesting. And have you tried clearing your derived data directory at all?

[SeanRobinson159]

This is the result when all tests pass. I have 28 UITests, but it still says 0/0.

[10:23:56]	[10:23:56]: ▸ Test Succeeded
[10:24:00]	+--------------------+---+
[10:24:00]	| Test Results |
[10:24:00]	+--------------------+---+
[10:24:00]	| Number of tests | 0 |
[10:24:00]	| Number of failures | 0 |
[10:24:00]	+--------------------+---+

[SeanRobinson159]

Clearing the derived data directory did not change the results at all.

[SeanRobinson159]

Any update? I am still getting the same results on v2.64.1

[mpirri]

@SeanRobinson159 How long do your tests take to run from Xcode? If you were to disable all of the tests (like comment them out, or just remove them from your test target temporarily), and add a simple test like an assert(true) does it start working? We are still wondering if the problem is somehow with the simulator/Xcode itself, rather than the actual tests, so if we can remove the tests from the equation, maybe that will be enlightening?

[karpelcevs]

It looks like an issue from xcpretty not recognising latest Xcode output: xcpretty/xcpretty#295

[SeanRobinson159]

I started seeing this issue as soon as I updated to Xcode 9. So that would make sense.

All of my UI tests take approx. 17 minutes to complete.

With just a single test case, with assert(true), I get the same results as above (when all tests pass)

[07:19:49]: ▸ Test Succeeded
+--------------------+---+
|      Test Results      |
+--------------------+---+
| Number of tests    | 0 |
| Number of failures | 0 |
+--------------------+---+

I have also tried running these tests through Fastlane on multiple computers, with the same results. This only started happening when I updated to Xcode 9. I am running the tests on 2 different simulators concurrently. I don't know if that helps you or not.

[fastlane-bot]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.

Please make sure to update to the latest fastlane version and check if that solves the issue. Let us know if that works for you by adding a comment 👍

[karpelcevs]

We ended up using https://github.com/KrauseFx/trainer as a fastlane plugin to at least get JUnit reports in Jenkins while xcpretty isn't fixed.

[fastlane-bot]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.

Please make sure to update to the latest fastlane version and check if that solves the issue. Let us know if that works for you by adding a comment 👍

[foxware00]

latest fastlane doesn't seem to resolve this issue, tested with fastlane-2.84.0

[karpelcevs]

Until xcpretty/xcpretty#295 is resolved, this won't get changed.

[Shadester]

Hopefully something can be done for this.

[fastlane-bot]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.

Please make sure to update to the latest fastlane version and check if that solves the issue. Let us know if that works for you by adding a comment 👍

[fastlane-bot]

This issue will be auto-closed because there hasn't been any activity for a few months. Feel free to open a new one if you still experience this problem 👍