Any ETA on this issue? Is this being worked on?

Seems like just need to update the regex in lib/xcpretty/parser.rb to handle the new xcodebuild output. This works for me locally.

diff --git a/lib/xcpretty/parser.rb b/lib/xcpretty/parser.rb
index 103e36f..f298010 100644
--- a/lib/xcpretty/parser.rb
+++ b/lib/xcpretty/parser.rb
@@ -123,24 +123,24 @@ module XCPretty
     # $1 = suite
     # $2 = test_case
     # $3 = time
-    TEST_CASE_PASSED_MATCHER = /^\s*Test Case\s'-\[(.*)\s(.*)\]'\spassed\s\((\d*\.\d{3})\sseconds\)/
+    TEST_CASE_PASSED_MATCHER = /^\s*Test [Cc]ase\s'-\[(\w*)\s(\w*)\]'\spassed\so?n?\s?'?\w*\s?\w*'?\s?\((\d*\.\d{3})\sseconds\)/


     # @regex Captured groups
     # $1 = suite
     # $2 = test_case
-    TEST_CASE_STARTED_MATCHER = /^Test Case '-\[(.*) (.*)\]' started.$/
+    TEST_CASE_STARTED_MATCHER = /^Test [Cc]ase '-\[(.*) (.*)\]' started.$/

     # @regex Captured groups
     # $1 = suite
     # $2 = test_case
-    TEST_CASE_PENDING_MATCHER = /^Test Case\s'-\[(.*)\s(.*)PENDING\]'\spassed/
+    TEST_CASE_PENDING_MATCHER = /^Test [Cc]ase\s'-\[(.*)\s(.*)PENDING\]'\spassed/

     # @regex Captured groups
     # $1 = suite
     # $2 = test_case
     # $3 = time
-    TEST_CASE_MEASURED_MATCHER = /^[^:]*:[^:]*:\sTest Case\s'-\[(.*)\s(.*)\]'\smeasured\s\[Time,\sseconds\]\saverage:\s(\d*\.\d{3}),/
+    TEST_CASE_MEASURED_MATCHER = /^[^:]*:[^:]*:\sTest [Cc]ase\s'-\[(.*)\s(.*)\]'\smeasured\s\[Time,\sseconds\]\saverage:\s(\d*\.\d{3}),/

     PHASE_SUCCESS_MATCHER = /^\*\*\s(.*)\sSUCCEEDED\s\*\*/

@@ -176,7 +176,7 @@ module XCPretty
     # @regex Captured groups
     # $1 = suite
     # $2 = time
-    TEST_SUITE_STARTED_MATCHER = /^\s*Test Suite '(?:.*\/)?(.*[ox]ctest.*)' started at(.*)/
+    TEST_SUITE_STARTED_MATCHER = /^\s*Test [Ss]uite '(?:.*\/)?(.*[ox]ctest.*)' started at(.*)/

I'd love to send PR but I'm not familiar with Ruby testing (Cucumber, RSpec).

@nicnocquee this does not work for me locally. I will take a look at making this work and open a PR. Update, I am working on this and it should be ready for review in the next day (or 2 at the most).

@marcelofabri @nicnocquee @eneko I am using logs that have been pre-processed by bluepill and I want to make sure the work I am doing will work with regular logs produced by Xcode 9 running multiple simulators. Can you post a sample log of the output of such logs?

Please see this PR

@lyndsey-ferguson

Testing started on 'iPhone 7 Plus'
Test suite 'Selected tests' started on 'iPhone 7 Plus'
Test suite 'UangkuTests.xctest' started on 'iPhone 7 Plus'
Test suite 'TopupMethodViewControllerTests' started on 'iPhone 7 Plus'
Test case '-[TopupMethodViewControllerTests testTopUpCashView]' passed on 'iPhone 7 Plus' (2.343 seconds)
Test case '-[TopupMethodViewControllerTests testTopUpTransferView]' passed on 'iPhone 7 Plus' (4.197 seconds)
Testing passed on 'iPhone 7 Plus'
Testing started on 'iPhone 7'
Test suite 'Selected tests' started on 'iPhone 7'
Test suite 'UangkuTests.xctest' started on 'iPhone 7'
Test suite 'TopupMethodViewControllerTests' started on 'iPhone 7'
Test case '-[TopupMethodViewControllerTests testTopUpCashView]' passed on 'iPhone 7' (2.213 seconds)
Test case '-[TopupMethodViewControllerTests testTopUpTransferView]' passed on 'iPhone 7' (4.133 seconds)
Testing passed on 'iPhone 7'
Testing started on 'iPhone 5s'
Test suite 'Selected tests' started on 'iPhone 5s'
Test suite 'UangkuTests.xctest' started on 'iPhone 5s'
Test suite 'TopupMethodViewControllerTests' started on 'iPhone 5s'
Test case '-[TopupMethodViewControllerTests testTopUpCashView]' passed on 'iPhone 5s' (2.224 seconds)
Test case '-[TopupMethodViewControllerTests testTopUpTransferView]' passed on 'iPhone 5s' (4.096 seconds)
Testing passed on 'iPhone 5s'
Testing started on 'iPhone X'
Test suite 'Selected tests' started on 'iPhone X'
Test suite 'UangkuTests.xctest' started on 'iPhone X'
Test suite 'TopupMethodViewControllerTests' started on 'iPhone X'
Test case '-[TopupMethodViewControllerTests testTopUpCashView]' passed on 'iPhone X' (2.305 seconds)
Test case '-[TopupMethodViewControllerTests testTopUpTransferView]' passed on 'iPhone X' (4.143 seconds)
Testing passed on 'iPhone X'
** TEST SUCCEEDED **

@nicnocquee Those tests look pretty sequential to me, how are you specifying the multiple destinations? They should start at the same time and run in parallel.

What's interesting is that running on a single destination vs. using Xcode 9's new support of multiple destination (parallel testing) seems to format logs completely differently; I'm not seeing the same logs as @nicnocquee. With a single destination, logs are formatted in Objective-C, but when targeting multiple destinations, logs are formatted in Swift and less verbose (no performance metrics, no test summary, etc.).

One Destination

set -o pipefail && xcodebuild -scheme MemoryHog -configuration Debug -destination "OS=latest,name=iPhone 6" test

Test Suite 'All tests' started at 2017-10-04 15:26:18.344
Test Suite 'MemoryHogTests.xctest' started at 2017-10-04 15:26:18.345
Test Suite 'MemoryHogTests' started at 2017-10-04 15:26:18.345
Test Case '-[MemoryHogTests.MemoryHogTests testOne]' started.
Test Case '-[MemoryHogTests.MemoryHogTests testOne]' passed (0.001 seconds).
Test Case '-[MemoryHogTests.MemoryHogTests testPerformance]' started.
/Users/john.hammerlund/Library/Github/MemoryHog/MemoryHogTests/MemoryHogTests.swift:28: Test Case '-[MemoryHogTests.MemoryHogTests testPerformance]' measured [Time, seconds] average: 0.000, relative standard deviation: 134.165%, values: [0.000004, 0.000001, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[MemoryHogTests.MemoryHogTests testPerformance]' passed (0.337 seconds).
Test Case '-[MemoryHogTests.MemoryHogTests testThree]' started.
Test Case '-[MemoryHogTests.MemoryHogTests testThree]' passed (0.001 seconds).
Test Case '-[MemoryHogTests.MemoryHogTests testTwo]' started.
Test Case '-[MemoryHogTests.MemoryHogTests testTwo]' passed (0.000 seconds).
Test Suite 'MemoryHogTests' passed at 2017-10-04 15:26:18.686.
	 Executed 4 tests, with 0 failures (0 unexpected) in 0.339 (0.341) seconds
Test Suite 'MemoryHogTests.xctest' passed at 2017-10-04 15:26:18.687.
	 Executed 4 tests, with 0 failures (0 unexpected) in 0.339 (0.342) seconds
Test Suite 'All tests' passed at 2017-10-04 15:26:18.687.
	 Executed 4 tests, with 0 failures (0 unexpected) in 0.339 (0.343) seconds
** TEST SUCCEEDED **

Multiple Destinations

set -o pipefail && xcodebuild -scheme MemoryHog -configuration Debug -destination "OS=latest,name=iPhone 6" -destination "OS=latest,name=iPhone 5S" test

Testing started on 'iPhone 6'
Testing started on 'iPhone 5s'
Test suite 'All tests' started on 'iPhone 6'
Test suite 'MemoryHogTests.xctest' started on 'iPhone 6'
Test suite 'MemoryHogTests' started on 'iPhone 6'
Test case 'MemoryHogTests.testOne()' passed on 'iPhone 6' (0.003 seconds)
Test suite 'All tests' started on 'iPhone 5s'
Test suite 'MemoryHogTests.xctest' started on 'iPhone 5s'
Test suite 'MemoryHogTests' started on 'iPhone 5s'
Test case 'MemoryHogTests.testOne()' passed on 'iPhone 5s' (0.001 seconds)
Test case 'MemoryHogTests.testPerformance()' passed on 'iPhone 6' (0.462 seconds)
Test case 'MemoryHogTests.testThree()' passed on 'iPhone 6' (0.001 seconds)
Test case 'MemoryHogTests.testTwo()' passed on 'iPhone 6' (0.000 seconds)
Test case 'MemoryHogTests.testPerformance()' passed on 'iPhone 5s' (0.340 seconds)
Test case 'MemoryHogTests.testThree()' passed on 'iPhone 5s' (0.000 seconds)
Test case 'MemoryHogTests.testTwo()' passed on 'iPhone 5s' (0.000 seconds)
Testing passed on 'iPhone 6'
Testing passed on 'iPhone 5s'
** TEST SUCCEEDED **

I was able to stitch some of the pieces back together to match previous behavior by updating the few of the matchers, though they're far more convoluted than the ones mentioned above; for example, for passing tests:

# @regex Captured groups
# $1 = suite
# $2 = test_case
# $3 = device (added)
# $4 = time
TEST_CASE_PASSED_MATCHER = /^\s*Test [Cc]ase\s'(?:-\[(?:.*)\.)?(.*)(?:\s|\.)(.*)(?:\]|\(\))'\spassed\s(?:on\s'(.*)'\s)?\((\d*\.\d{3})\sseconds\)/

I'd be willing to keep testing/moving forward if this still hasn't gotten much traction, but behavior will inevitably be different for parallel testing.

@eneko sorry forgot to mention I added -jobs 1 -maximum-concurrent-test-simulator-destinations 1 when running xcodebuild. Had problem with my tests running parallel.

xcodebuild -workspace Uangku.xcworkspace 
-scheme UangkuDemo 
-configuration Debug 
-sdk 'iphonesimulator11.0' 
-destination 'platform=iOS Simulator,id=33ACF59B-E864-42F5-B05E-90E2EF139B36' 
-destination 'platform=iOS Simulator,id=5C71F407-1930-4119-87AD-038388F1F02E' 
-destination 'platform=iOS Simulator,id=FA2064AC-5A8C-47A6-A131-77E80195D1D4' 
-destination 'platform=iOS Simulator,id=4CD5897F-F46B-4279-B6F2-A8BF89A4CF9C' 
-derivedDataPath '/Users/nico/Library/Developer/Xcode/DerivedData/Uangku-fwiimgqssqptgdbrmdznmhdkisea' 
-enableCodeCoverage NO 
-jobs 1 
-maximum-concurrent-test-simulator-destinations 1 
GCC_PREPROCESSOR_DEFINITIONS="POD_CONFIGURATION_DEBUG=1 COCOAPODS=1 DEBUG=1 GPB_USE_PROTOBUF_FRAMEWORK_IMPORTS=1" 
-only-testing:UangkuTests/TopupMethodViewControllerTests 
-skip-testing:UangkuDemoUITests
clean test

I'm having similar issues, not only with xcpretty but with xcodebuild itself. If I use tee to pipe the output to a file, there are no lines about test results. I think that now that there is the support for multiple simulators, the log outputs are not from xcodebuild, but from the simulator themselves, so using pipe you don't catch them.

I haven't seen much activity on this, so I've been prototyping some changes to formatting/reporting: https://github.com/Hammertime38/xcpretty/tree/feature/parallel-testing-support

@alessandroorru I've been seeing the same issues; it seems that xcodebuild I/O buffering is different when ran on multiple destinations...that might be a separate issue.

Any updates on this issue?

Hey, Any update on this? Big blocker for me.

@astasiak93 You can always use @johnhammerlund's PR : #299. It's been ok for our use (parallel testing on iOS11 simulators and real devices).

gem "xcpretty", :git => "https://github.com/technology-ebay-de/xcpretty", :ref => "2b681274dbdef611374d70118cdf4170fae3d55b"

(ebay's version 2b68127 removes somes deprecation warnings from the logs)

@marcc-orange Hey, thank you for the snippet - it works for successful test sessions but for failures it is only displayed : Build error 1 on the scan slack notification message.
Report in ./fastlane/test_output/ directory contains info about failed and passed tests.

How is this still an issue lol. I feel so many people use fastlane which then uses scan and then uses xcpretty by default.

Does scan support running multiple simulators in parallel?

Yes, you can pass a list of devices and it works except the output doesn’t show correctly due to xcpretty not supporting it.

You can get some output directly from junit report file using:

$(sed '2q;d' ./fastlane/test_output/report.junit | tr "<""/"">" ' ')

Then you will get output like:

"testsuites name="something.xctest' tests='4' failures='0'"

I am using for now:
gem "xcpretty", :git => "https://github.com/technology-ebay-de/xcpretty", :ref => "2b681274dbdef611374d70118cdf4170fae3d55b"

Any update on that issue?

Having this issue too - would be nice not to have to use that specific gem workaround

I would strongly advise everyone facing this issue to start using https://github.com/TitouanVanBelle/XCTestHTMLReport since there has been no update on this issue. The report that i am talking about has support for screenshots and also supports reporting when running tests in parallel on multiple devices or simulators.
Go and show this guy some support, he clearly has done a fabulous job!!

Are there any plans to update this for Xcode 10?

Any updates on this ?

Any updates on this? 🤔

+1

We've migrated to https://github.com/KrauseFx/trainer for ingestion. keeping xcpretty for formatting for now.

https://github.com/TitouanVanBelle/XCTestHTMLReport can also produce JUnit output and a nice HTML report output. It handles parallel simulators with no problems

For anyone comes across this, please give xcbeautify a try.

Alternatively, for a clean build output, -quiet parameter can be passed to xcodebuild. We've been doing this to reduce our CI log output.

Any update?

Without using XCPretty in fastlane am able to run the tests but since fastlane is heavily dependent on XCPretty, it is frustrating Parallel Testing doesn't work with XCPretty.

Would this issue be even fixed?

@ghost I've been using the unofficial gem suggested a few posts above by @tomaszmnich for the same reason and it works fine, have you tried it?
edit: may have to create your own fork these days given fastlane has requirements over xcpretty's minimum version

I think xcpretty shouldn't be producing any reports (junit or html) and just focus on making the output pretty. Creating reports is best left to other tools that read the results bundle that Xcode produces at the end.

If using Fastlane and you want to use parallel testing set these options in scan:

    fail_build: false, # If tests fail the build will be failed later based on the result_bundle
    output_types: '', # Don't generate reports, they be generated from the result_bundle
    output_style: 'raw', # Disable xcpretty
    xcargs: '-quiet', # Prevent xcodebuild from spewing output (only shows warnings and errors) now that xcpretty is disabled.
    result_bundle: true # Needed so that a following tool can generate reports/fail the build

Tbh at this point almost might as well just use xcodebuild directly. It would be nice if Fastlane scan was more modular/allowed more control..

I just came across this issue when I was looking to file this myself.

Any update on this 😬?

@thii xcbeautify works like a charm when running UI Tests in parallel with -parallel-testing-enabled YES! Thanks for making it. It would be nice if xcpretty worked for this case though 😞.

How has this not been fixed yet?

Btw it is still relevant 🙃

-parallel-testing-enabled YES

any code snippet somewhere that can help to get started soon?

This can probably be closed since the latest release of fastlane removes xcpretty: https://github.com/fastlane/fastlane/releases/tag/2.201.0

Any updates on this issue?

hi @li-arezkisidali , I think this can be closed like @gmoraleda said (cc @joshdholtz) -- fastlane users just have to use xcbeautify as the xcodebuild_formatter given the issue here was with xcpretty, the former default formatter

just make sure xcbeautify is installed on your build machine, then fastlane should use it by default 👍