Bump xunit to 2.8.0 #9457

drewnoakes · 2024-05-01T04:43:55Z

Increases package versions. I'm hopeful this will make unit test discovery quicker in the solution.

Also fixes compile issues due to changes in xUnit's APIs, and addresses all diagnostics logged via new capabilities in xUnit's analyzers.

Microsoft Reviewers: Open in CodeFlow

drewnoakes · 2024-05-02T05:44:50Z

/azp run

azure-pipelines · 2024-05-02T05:44:59Z

Azure Pipelines successfully started running 1 pipeline(s).

drewnoakes · 2024-05-20T01:22:07Z

/azp run

azure-pipelines · 2024-05-20T01:22:16Z

Azure Pipelines successfully started running 1 pipeline(s).

drewnoakes · 2024-05-20T02:31:24Z

The build is timing out while running tests. Here you can see the last thing it did before being cancelled.

  5:24>Project "D:\a\_work\1\s\tests\Microsoft.VisualStudio.ProjectSystem.Managed.UnitTests\Microsoft.VisualStudio.ProjectSystem.Managed.UnitTests.csproj" (5:24) is building "D:\a\_work\1\s\tests\Microsoft.VisualStudio.ProjectSystem.Managed.UnitTests\Microsoft.VisualStudio.ProjectSystem.Managed.UnitTests.csproj" (5:25) on node 3 (Test target(s)).
     5>TestCore:
         Microsoft.VisualStudio.ProjectSystem.Managed.UnitTests -> Running tests...
         dotnet test --verbosity quiet --nologo --logger "trx;logfilename=D:\a\_work\1\s\artifacts\Debug\TestResults\net472\Microsoft.VisualStudio.ProjectSystem.Managed.UnitTests.dll.trx" --logger "html;logfilename=D:\a\_work\1\s\artifacts\Debug\TestResults\net472\Microsoft.VisualStudio.ProjectSystem.Managed.UnitTests.dll.html" D:\a\_work\1\s\artifacts\Debug\bin\UnitTests\Microsoft.VisualStudio.ProjectSystem.Managed.UnitTests\net472\Microsoft.VisualStudio.ProjectSystem.Managed.UnitTests.dll
         Starting test execution, please wait...
         A total of 1 test files matched the specified pattern.
##[error]The Operation will be canceled. The next steps may not contain expected logs.
Attempting to cancel the build...

Locally, these net472 tests in Microsoft.VisualStudio.ProjectSystem.Managed.UnitTests pass.

Locally, net8.0 tests fail with the below, but that might be unrelated as the CI failures are on net472.

StreamJsonRpc.RemoteInvocationException: Value cannot be null.
Parameter name: versionRange
   at Microsoft.ServiceHub.Framework.ServiceJsonRpcDescriptor.LocalProxyGeneration.<>c__DisplayClass40_0`1.<ReturnedTaskOfTHelperAsync>b__0(Task`1 _)
   at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.TestWindow.Controller.NuGetService.<GetReferencedNuGetPackagesAsync>d__5.MoveNext()
RPC server exception:
System.ArgumentNullException: Value cannot be null.
Parameter name: versionRange
      at NuGet.PackageManagement.VisualStudio.Utility.ProjectInstalledPackage..ctor(VersionRange versionRange, PackageIdentity installedPackage)
      at NuGet.PackageManagement.VisualStudio.Utility.GetPackageReferenceUtility.UpdateResolvedVersion(LibraryDependency projectLibrary, NuGetFramework targetFramework, IList`1 targets, Dictionary`2 installedPackages)
      at NuGet.PackageManagement.VisualStudio.PackageReferenceProject`2.<>c__DisplayClass48_0.<GetPackageReferences>b__1(LibraryDependency library)
      at System.Linq.Enumerable.WhereSelectListIterator`2.MoveNext()
      at System.Linq.Enumerable.<SelectManyIterator>d__17`2.MoveNext()
      at System.Linq.Lookup`2.Create[TSource](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer)
      at System.Linq.GroupedEnumerable`3.GetEnumerator()
      at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
      at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
      at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
      at NuGet.PackageManagement.VisualStudio.PackageReferenceProject`2.<GetInstalledAndTransitivePackagesAsync>d__43.MoveNext()
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at NuGet.PackageManagement.VisualStudio.PackageReferenceProject`2.<GetInstalledAndTransitivePackagesAsync>d__42.MoveNext()
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at NuGet.VisualStudio.Implementation.Extensibility.NuGetProjectService.<GetInstalledPackagesAsync>d__6.MoveNext()
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at NuGet.VisualStudio.Implementation.Extensibility.NuGetProjectService.<GetInstalledPackagesAsync>d__5.MoveNext()
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at NuGet.VisualStudio.Implementation.Extensibility.NuGetProjectService.<GetInstalledPackagesAsync>d__5.MoveNext()

drewnoakes · 2024-05-20T03:08:57Z

I'm not sure how to progress this. The test run hangs, so is killed, and no results are published. We only know which assembly is seeing the hang.

@smitpatel @haileymck do either of you know whether there's a way to have .trx files published even if the run is cancelled? Or is there some other diagnostic data we might be able to get? For example, perhaps the code's stuck on a Debug.Fail popup or similar (though during tests there's usually a trace listener that converts these dialogs into exceptions).

drewnoakes · 2024-05-20T03:34:37Z

I've pushed a change that'll increase the log level from quiet to diagnostic.

drewnoakes · 2024-05-20T21:39:10Z

That didn't give much more of value:

2024-05-20T04:34:05.1327807Z   5:24>Project "D:\a\_work\1\s\tests\Microsoft.VisualStudio.ProjectSystem.Managed.UnitTests\Microsoft.VisualStudio.ProjectSystem.Managed.UnitTests.csproj" (5:24) is building "D:\a\_work\1\s\tests\Microsoft.VisualStudio.ProjectSystem.Managed.UnitTests\Microsoft.VisualStudio.ProjectSystem.Managed.UnitTests.csproj" (5:25) on node 2 (Test target(s)).
2024-05-20T04:34:05.1329661Z      5>TestCore:
2024-05-20T04:34:05.1330489Z          Microsoft.VisualStudio.ProjectSystem.Managed.UnitTests -> Running tests...
2024-05-20T04:34:05.1331806Z          dotnet test --verbosity diagnostic --nologo --logger "trx;logfilename=D:\a\_work\1\s\artifacts\Debug\TestResults\net472\Microsoft.VisualStudio.ProjectSystem.Managed.UnitTests.dll.trx" --logger "html;logfilename=D:\a\_work\1\s\artifacts\Debug\TestResults\net472\Microsoft.VisualStudio.ProjectSystem.Managed.UnitTests.dll.html" D:\a\_work\1\s\artifacts\Debug\bin\UnitTests\Microsoft.VisualStudio.ProjectSystem.Managed.UnitTests\net472\Microsoft.VisualStudio.ProjectSystem.Managed.UnitTests.dll
2024-05-20T04:34:06.5864159Z      4>TestCore:
2024-05-20T04:34:06.5864499Z          Starting test execution, please wait...
2024-05-20T04:34:06.5864714Z      5>TestCore:
2024-05-20T04:34:06.5865741Z          Starting test execution, please wait...
2024-05-20T04:34:07.1162673Z      4>TestCore:
2024-05-20T04:34:07.1163661Z          A total of 1 test files matched the specified pattern.
2024-05-20T04:34:07.1164646Z      5>TestCore:
2024-05-20T04:34:07.1165095Z          A total of 1 test files matched the specified pattern.
2024-05-20T04:34:10.2894527Z      4>TestCore:
2024-05-20T04:34:10.2894801Z          
2024-05-20T04:34:10.2951879Z          Passed!  - Failed:     0, Passed:   168, Skipped:     0, Total:   168, Duration: 281 ms - Microsoft.VisualStudio.ProjectSystem.Managed.TestServices.UnitTests.dll (net472)
2024-05-20T04:34:10.3982872Z          Results File: D:\a\_work\1\s\artifacts\Debug\TestResults\net472\Microsoft.VisualStudio.ProjectSystem.Managed.TestServices.UnitTests.dll.trx
2024-05-20T04:34:10.5592436Z          Html test results file : D:\a\_work\1\s\artifacts\Debug\TestResults\net472\Microsoft.VisualStudio.ProjectSystem.Managed.TestServices.UnitTests.dll.html
2024-05-20T04:34:10.5892500Z          Microsoft.VisualStudio.ProjectSystem.Managed.TestServices.UnitTests -> Tests succeeded
2024-05-20T04:34:10.5902013Z      4>Done Building Project "D:\a\_work\1\s\tests\Microsoft.VisualStudio.ProjectSystem.Managed.TestServices.UnitTests\Microsoft.VisualStudio.ProjectSystem.Managed.TestServices.UnitTests.csproj" (Test target(s)).
2024-05-20T04:34:11.2382585Z      5>TestCore:
2024-05-20T04:34:11.2387935Z          [xUnit.net 00:00:02.43]     WhenAddingAnItem_TheReturnedItemHasAllTheExpectedMetadata [SKIP]
2024-05-20T04:34:11.2701932Z            Skipped WhenAddingAnItem_TheReturnedItemHasAllTheExpectedMetadata [1 ms]
2024-05-20T04:34:15.2955066Z          [xUnit.net 00:00:06.49]     IsCycleDetectedAsync(sequence: "ABCABCABC", isCycleDetected: True) [SKIP]
2024-05-20T04:34:15.3156186Z            Skipped IsCycleDetectedAsync(sequence: "ABCABCABC", isCycleDetected: True) [1 ms]

~15 min pause...

2024-05-20T04:49:52.7298386Z ##[error]The Operation will be canceled. The next steps may not contain expected logs.
2024-05-20T04:49:52.7421616Z Attempting to cancel the build...
2024-05-20T04:49:52.7565817Z      1>Done Building Project "D:\a\_work\1\s\eng\Build.proj" (default targets) -- FAILED.
2024-05-20T04:49:52.7566405Z      2>Done Building Project "D:\a\_work\1\s\artifacts\Build.sln" (Restore;Build;Test;Pack target(s)) -- FAILED.
2024-05-20T04:49:52.7899205Z      5>Done Building Project "D:\a\_work\1\s\tests\Microsoft.VisualStudio.ProjectSystem.Managed.UnitTests\Microsoft.VisualStudio.ProjectSystem.Managed.UnitTests.csproj" (Test target(s)) -- FAILED.
2024-05-20T04:49:52.7922982Z      5>D:\a\_work\1\s\eng\imports\UnitTests.targets(46,5): warning MSB5021: Terminating the task executable "cmd" and its child processes because the build was canceled. [D:\a\_work\1\s\tests\Microsoft.VisualStudio.ProjectSystem.Managed.UnitTests\Microsoft.VisualStudio.ProjectSystem.Managed.UnitTests.csproj::TargetFramework=net472]
2024-05-20T04:49:52.8432494Z      5>Done Building Project "D:\a\_work\1\s\tests\Microsoft.VisualStudio.ProjectSystem.Managed.UnitTests\Microsoft.VisualStudio.ProjectSystem.Managed.UnitTests.csproj" (Test target(s)) -- FAILED.
2024-05-20T04:49:53.0104792Z 
2024-05-20T04:49:53.0105559Z Build FAILED.

Splitting that out by project:

2024-05-20T04:34:05.1290221Z      2>Project "D:\a\_work\1\s\artifacts\Build.sln" (2) is building "D:\a\_work\1\s\tests\Microsoft.VisualStudio.ProjectSystem.Managed.TestServices.UnitTests\Microsoft.VisualStudio.ProjectSystem.Managed.TestServices.UnitTests.csproj" (4:7) on node 3 (Test target(s)).
2024-05-20T04:34:05.1291485Z      4>TestCore:
2024-05-20T04:34:05.1292401Z          Microsoft.VisualStudio.ProjectSystem.Managed.TestServices.UnitTests -> Running tests...
2024-05-20T04:34:05.1292751Z          Creating directory "D:\a\_work\1\s\artifacts\Debug\TestResults\net472\".
2024-05-20T04:34:05.1295397Z          dotnet test --verbosity diagnostic --nologo --logger "trx;logfilename=D:\a\_work\1\s\artifacts\Debug\TestResults\net472\Microsoft.VisualStudio.ProjectSystem.Managed.TestServices.UnitTests.dll.trx" --logger "html;logfilename=D:\a\_work\1\s\artifacts\Debug\TestResults\net472\Microsoft.VisualStudio.ProjectSystem.Managed.TestServices.UnitTests.dll.html" D:\a\_work\1\s\artifacts\Debug\bin\UnitTests\Microsoft.VisualStudio.ProjectSystem.Managed.TestServices.UnitTests\net472\Microsoft.VisualStudio.ProjectSystem.Managed.TestServices.UnitTests.dll
2024-05-20T04:34:06.5864159Z      4>TestCore:
2024-05-20T04:34:06.5864499Z          Starting test execution, please wait...
2024-05-20T04:34:07.1162673Z      4>TestCore:
2024-05-20T04:34:07.1163661Z          A total of 1 test files matched the specified pattern.
2024-05-20T04:34:10.2894527Z      4>TestCore:
2024-05-20T04:34:10.2894801Z          
2024-05-20T04:34:10.2951879Z          Passed!  - Failed:     0, Passed:   168, Skipped:     0, Total:   168, Duration: 281 ms - Microsoft.VisualStudio.ProjectSystem.Managed.TestServices.UnitTests.dll (net472)
2024-05-20T04:34:10.3982872Z          Results File: D:\a\_work\1\s\artifacts\Debug\TestResults\net472\Microsoft.VisualStudio.ProjectSystem.Managed.TestServices.UnitTests.dll.trx
2024-05-20T04:34:10.5592436Z          Html test results file : D:\a\_work\1\s\artifacts\Debug\TestResults\net472\Microsoft.VisualStudio.ProjectSystem.Managed.TestServices.UnitTests.dll.html
2024-05-20T04:34:10.5892500Z          Microsoft.VisualStudio.ProjectSystem.Managed.TestServices.UnitTests -> Tests succeeded
2024-05-20T04:34:10.5902013Z      4>Done Building Project "D:\a\_work\1\s\tests\Microsoft.VisualStudio.ProjectSystem.Managed.TestServices.UnitTests\Microsoft.VisualStudio.ProjectSystem.Managed.TestServices.UnitTests.csproj" (Test target(s)).

Those tests all passed in about five seconds. We can see what a full run is supposed to resemble.

The following test hangs at the end of this log:

2024-05-20T04:34:05.1327807Z   5:24>Project "D:\a\_work\1\s\tests\Microsoft.VisualStudio.ProjectSystem.Managed.UnitTests\Microsoft.VisualStudio.ProjectSystem.Managed.UnitTests.csproj" (5:24) is building "D:\a\_work\1\s\tests\Microsoft.VisualStudio.ProjectSystem.Managed.UnitTests\Microsoft.VisualStudio.ProjectSystem.Managed.UnitTests.csproj" (5:25) on node 2 (Test target(s)).
2024-05-20T04:34:05.1329661Z      5>TestCore:
2024-05-20T04:34:05.1330489Z          Microsoft.VisualStudio.ProjectSystem.Managed.UnitTests -> Running tests...
2024-05-20T04:34:05.1331806Z          dotnet test --verbosity diagnostic --nologo --logger "trx;logfilename=D:\a\_work\1\s\artifacts\Debug\TestResults\net472\Microsoft.VisualStudio.ProjectSystem.Managed.UnitTests.dll.trx" --logger "html;logfilename=D:\a\_work\1\s\artifacts\Debug\TestResults\net472\Microsoft.VisualStudio.ProjectSystem.Managed.UnitTests.dll.html" D:\a\_work\1\s\artifacts\Debug\bin\UnitTests\Microsoft.VisualStudio.ProjectSystem.Managed.UnitTests\net472\Microsoft.VisualStudio.ProjectSystem.Managed.UnitTests.dll
2024-05-20T04:34:06.5864714Z      5>TestCore:
2024-05-20T04:34:06.5865741Z          Starting test execution, please wait...
2024-05-20T04:34:07.1164646Z      5>TestCore:
2024-05-20T04:34:07.1165095Z          A total of 1 test files matched the specified pattern.
2024-05-20T04:34:11.2382585Z      5>TestCore:
2024-05-20T04:34:11.2387935Z          [xUnit.net 00:00:02.43]     WhenAddingAnItem_TheReturnedItemHasAllTheExpectedMetadata [SKIP]
2024-05-20T04:34:11.2701932Z            Skipped WhenAddingAnItem_TheReturnedItemHasAllTheExpectedMetadata [1 ms]
2024-05-20T04:34:15.2955066Z          [xUnit.net 00:00:06.49]     IsCycleDetectedAsync(sequence: "ABCABCABC", isCycleDetected: True) [SKIP]
2024-05-20T04:34:15.3156186Z            Skipped IsCycleDetectedAsync(sequence: "ABCABCABC", isCycleDetected: True) [1 ms]

smitpatel · 2024-05-20T21:50:53Z

I don't think trx will be generated if the test run is terminated.

drewnoakes · 2024-05-20T22:41:51Z

@smitpatel. Makes sense. The log says it generated a binlog, but the run says there are no artifacts. Is it available somewhere? If not, can we make a change to capture the binlog even in case of failure like this?

smitpatel · 2024-05-20T22:46:49Z

Our current pipeline is configured to capture logs even when a step fails. It publishes all test results, artifacts. Issue here is that whole job is terminated on timeout so azdo doesn't allow to run any steps after that at all. I am not sure if that is any direct way to achieve it.

drewnoakes requested a review from a team as a code owner May 1, 2024 04:43

haileymck approved these changes May 1, 2024

View reviewed changes

tmeschter approved these changes May 1, 2024

View reviewed changes

drewnoakes force-pushed the xunit-2.8.0 branch from 30ef502 to 80e3b47 Compare May 20, 2024 01:46

drewnoakes added 2 commits May 20, 2024 13:00

Bump xunit to 2.8.0

e10978f

Fix xUnit diagnostics

f135b81

Increase test log level

7dea492

drewnoakes force-pushed the xunit-2.8.0 branch from 80e3b47 to 7dea492 Compare May 20, 2024 03:34

Increase test log level further

c2494c8

Try skipping a test to see if it makes the build pass

85525df

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bump xunit to 2.8.0 #9457

Bump xunit to 2.8.0 #9457

drewnoakes commented May 1, 2024 •

edited by dotnet-policy-service bot

drewnoakes commented May 2, 2024

azure-pipelines bot commented May 2, 2024

drewnoakes commented May 20, 2024

azure-pipelines bot commented May 20, 2024

drewnoakes commented May 20, 2024

drewnoakes commented May 20, 2024

drewnoakes commented May 20, 2024 •

edited

drewnoakes commented May 20, 2024

smitpatel commented May 20, 2024

drewnoakes commented May 20, 2024 •

edited

smitpatel commented May 20, 2024

Bump xunit to 2.8.0 #9457

Are you sure you want to change the base?

Bump xunit to 2.8.0 #9457

Conversation

drewnoakes commented May 1, 2024 • edited by dotnet-policy-service bot

Microsoft Reviewers: Open in CodeFlow

drewnoakes commented May 2, 2024

azure-pipelines bot commented May 2, 2024

drewnoakes commented May 20, 2024

azure-pipelines bot commented May 20, 2024

drewnoakes commented May 20, 2024

drewnoakes commented May 20, 2024

drewnoakes commented May 20, 2024 • edited

drewnoakes commented May 20, 2024

smitpatel commented May 20, 2024

drewnoakes commented May 20, 2024 • edited

smitpatel commented May 20, 2024

drewnoakes commented May 1, 2024 •

edited by dotnet-policy-service bot

drewnoakes commented May 20, 2024 •

edited

drewnoakes commented May 20, 2024 •

edited