[WIP] Parallel device testing support

[chrisballinger]

Resolves #295.

Since it looks like progress on #299 has stalled, I'm opening up a new PR based on the original that contains additional fixes from the technology-ebay-de fork, as well as some fixes of my own. Unfortunately after merging the latest upstream/master the cucumber tests no longer pass, with the following error:

  Scenario: Showing tests with one having a swift fatal error # features/junit_report.feature:46
    Given I have a swift fatal error in a test in my suite    # features/steps/formatting_steps.rb:34
    When I pipe to xcpretty with "--report junit"             # features/steps/formatting_steps.rb:179
    Then I should see a failed test node in my report         # features/steps/junit_steps.rb:1
      expected: not nil
           got: nil (RSpec::Expectations::ExpectationNotMetError)
      features/junit_report.feature:49:in `Then I should see a failed test node in my report'

I believe this is caused by the recent change in 87dd9f8. I'm not very familiar with how cucumber tests work, so any help there would be appreciated. Perhaps it has something to do with how parallel test output appears as Swift instead of ObjC?

TODO:

• rake spec passes
• rake lint passes
• rake cucumber fails: Scenario: Showing tests with one having a swift fatal error
• Time is missing from failed tests
• Show device name when xcodebuild -disable-concurrent-testing is enabled
• Show device iOS version in addition to device name

[houndci-bot]

Some files could not be reviewed due to errors:

.rubocop.yml: Lint/DefEndAlignment has the wrong namespace - should be Layout

.rubocop.yml: Lint/DefEndAlignment has the wrong namespace - should be Layout
.rubocop.yml: Lint/EndAlignment has the wrong namespace - should be Layout
.rubocop.yml: Lint/Eval has the wrong namespace - should be Security
.rubocop.yml: Style/AlignHash has the wrong namespace - should be Layout
.rubocop.yml: Style/AlignParameters has the wrong namespace - should be Layout
.rubocop.yml: Style/CaseIndentation has the wrong namespace - should be Layout
.rubocop.yml: Style/ClosingParenthesisIndentation has the wrong namespace - should be Layout
.rubocop.yml: Style/CommentIndentation has the wrong namespace - should be Layout
.rubocop.yml: Style/EmptyLineBetweenDefs has the wrong namespace - should be Layout
.rubocop.yml: Style/EmptyLines has the wrong namespace - should be Layout
.rubocop.yml: Style/EmptyLinesAroundBlockBody has the wrong namespace - should be Layout
.rubocop.yml: Style/EmptyLinesAroundClassBody has the wrong namespace - should be Layout
.rubocop.yml: Style/EmptyLinesAroundModuleBody has the wrong namespace - should be Layout
.rubocop.yml: Style/ExtraSpacing has the wrong namespace - should be Layout
.rubocop.yml: Style/FirstParameterIndentation has the wrong namespace - should be Layout
.rubocop.yml: Style/IndentationWidth has the wrong namespace - should be Layout
.rubocop.yml: Style/LeadingCommentSpace has the wrong namespace - should be Layout
.rubocop.yml: Style/MultilineOperationIndentation has the wrong namespace - should be Layout
.rubocop.yml: Style/SpaceAfterComma has the wrong namespace - should be Layout
.rubocop.yml: Style/SpaceAroundEqualsInParameterDefault has the wrong namespace - should be Layout
.rubocop.yml: Style/SpaceAroundOperators has the wrong namespace - should be Layout
.rubocop.yml: Style/SpaceInsideBlockBraces has the wrong namespace - should be Layout
.rubocop.yml: Style/SpaceInsideHashLiteralBraces has the wrong namespace - should be Layout
.rubocop.yml: Style/SpaceInsideParens has the wrong namespace - should be Layout
.rubocop.yml: Style/TrailingBlankLines has the wrong namespace - should be Layout
.rubocop.yml: Style/TrailingWhitespace has the wrong namespace - should be Layout
Error: The `Style/TrailingComma` cop no longer exists. Please use `Style/TrailingCommaInArguments`, `Style/TrailingCommaInArrayLiteral`, and/or `Style/TrailingCommaInHashLiteral` instead.
(obsolete configuration found in .rubocop.yml, please update it)

[chrisballinger]

Another issue I found is that when xcodebuild is run with multiple destinations with -disable-concurrent-testing, the device names are not present in the report.

[thii]

Any update on this?

[chrisballinger]

@thii Nope! Feel free to finish it up. I've since switched over to using
XCTestHTMLReport

[thii]

@chrisballinger Thanks. I ended up porting xcpretty to Swift: https://github.com/thii/xcbeautify.

[chrisballinger]

@thii Hahaha awesome, that's the spirit!

[ecerney]

@chrisballinger / @thii I'm assuming you're not using xcpretty through fastlane then? Since xcpretty seems to be stalled on fixing this, and since fastlane uses xcpretty, using alternatives like xcbeautify and XCTestHTMLReport don't seem to be an option if using fastlane's scan function to run our parallel tests

[thii]

@ecerney Yeah you would have to use xcodebuild directly.