Skip to content

Latest commit

 

History

History
411 lines (270 loc) · 20.4 KB

readme.source.md

File metadata and controls

411 lines (270 loc) · 20.4 KB
Raw

Verify

Discussions Build status NuGet Status NuGet Status NuGet Status NuGet Status NuGet Status NuGet Status

Verify is a snapshot tool that simplifies the assertion of complex data models and documents.

Verify is called on the test result during the assertion phase. It serializes that result and stores it in a file that matches the test name. On the next test execution, the result is again serialized and compared to the existing file. The test will fail if the two snapshots do not match: either the change is unexpected, or the reference snapshot needs to be updated to the new result.

See Milestones for release notes.

Sponsors

A HUGE Thank-You to AWS for sponsoring this project in September 2023 as part of the AWS Open Source Software Fund.

Thanks to DataDog for the generous monthly sponsorship.

Requirements

  • Supported runtimes: net472, net48, net481, net6, net8, and net9.
  • Supported SDK: 8.0.300 and up

Getting started wizard

Get customized instructions for the specific combination of Operating System, IDE, Test Framework, and Build Server.

Start wizard.

NuGet packages

Snapshot management

Accepting or declining a snapshot file is part of the core workflow of Verify. There are several ways to do this and the approach(s) selected is a personal preference.

Usage

ImplicitUsings

include: implicit-usings

Class being tested

Given a class to be tested:

snippet: ClassBeingTested

NUnit

Support for NUnit

snippet: SampleTestNUnit

xUnit

Support for xUnit

snippet: SampleTestXunit

Fixie

Support for Fixie

snippet: SampleTestFixie

Expecto

Support for Expecto

snippet: SampleTestExpecto

Caveats

Due to the nature of the Expecto implementation, the following APIs in Verify are not supported.

  • settings.UseParameters()
  • settings.UseTextForParameters()

MSTest

Support for MSTest

snippet: SampleTestMSTest

Marking tests as 'Using Verify'

include: mstest-marker

Initial Verification

No existing .verified. file.

graph TD
run(Run test and<br/>create Received file)
failTest(Fail Test<br/>and show Diff)
closeDiff(Close Diff)
run-->failTest
shouldAccept{Accept ?}
failTest-->shouldAccept
accept(Move Received<br/>to Verified)
shouldAccept-- Yes -->accept
discard(Discard<br/>Received)
shouldAccept-- No -->discard
accept-->closeDiff
discard-->closeDiff

When the test is initially run will fail. If a Diff Tool is detected it will display the diff.

InitialDiff

To verify the result:

Verified result

This will result in the Sample.Test.verified.txt being created:

snippet: Verify.Xunit.Tests/Snippets/Sample.Test.verified.txt

Subsequent Verification

Existing .verified. file.

graph TD
run(Run test and<br/>create Received file)
closeDiff(Close Diff)
failTest(Fail Test<br/>and show Diff)
run-->isSame
shouldAccept{Accept ?}
failTest-->shouldAccept
accept(Move Received<br/>to Verified)
shouldAccept-- Yes -->accept
discard(Discard<br/>Received)
shouldAccept-- No -->discard

isSame{Compare<br/>Verified +<br/>Received}
passTest(Pass Test and<br/>discard Received)
isSame-- Same --> passTest
isSame-- Different --> failTest
accept-->closeDiff
discard-->closeDiff

If the implementation of ClassBeingTested changes:

snippet: ClassBeingTestedChanged

And the test is re run it will fail.

The Diff Tool will display the diff:

SecondDiff

The same approach can be used to verify the results and the change to Sample.Test.verified.txt is committed to source control along with the change to ClassBeingTested.

Async

Verify() has overloads that accept Task<T>, ValueTask<T>, and IAsyncEnumerable<T>. These are awaited before verification.

There is also an overload that accepts Func<Task<T>>, which works well with async lambda expressions:

snippet: VerifyFuncOfTaskOfT

VerifyJson

VerifyJson performs the following actions

  • Convert to JToken (if necessary).
  • Apply ignore member by name for keys.
  • PrettyPrint the resulting text.

snippet: VerifyJson

Results in a .txt file:

snippet: JsonTests.VerifyJsonString.verified.txt

Source control: Received and Verified files

Includes/Excludes

include: include-exclude

Text file settings

include: text-file-settings

Static settings

Most settings are available at the both global level and at the instance level.

When modifying settings at the both global level it should be done using a Module Initializer:

snippet: StaticSettings.cs

VerifyResult

In some scenarios it can be helpful to get access to the resulting *.verified.* files after a successful run. For example to do an explicit check for contains or not-contains in the resulting text. To allow this all Verify methods return a VerifyResult.

snippet: VerifyResult

If using Verifier.Throws, the resulting Exception will also be accessible

snippet: ExceptionResult

CurrentFile

Utility for finding paths based on the current file.

snippet: CurrentFile.cs

Versioning

Verify follows Semantic Versioning. The same applies for extensions to Verify. Small changes in the resulting snapshot files may be deployed in a minor version. As such nuget updates to Verify.* should be done as follows:

  • Updates all Verify.*packages in isolation
  • Re-run all tests.
  • If there are changes, ensure they look correct given the release notes. If the changes do not look correct, raise an issue.
  • Accept those changes.

Snapshot changes do not trigger a major version change to avoid causing Diamond dependency issues for downstream extensions.

Unit testing inside virtualized environment

Unit tests referencing Verify (including unit tests within this repository as well as any other code referencing Verify) can be run and debugged on a local virtualized environment supported by Visual Studio Remote Testing. Initial configurations have been added for WSL and net 7.0 linux docker via testenvironments.json (for third party code, the file needs to be copied or recreated next to the .sln solution file for solution to leverage the functionality).

Upon opening the Tests Explorer the advanced environments are available in the GUI:

TestExplorerEnvironments

This readme will not discuss definitive list of details for proper setup of the environments instead refer the following information sources and warn about particular gotchas:

Media

Extensions

More Documentation

include: doc-index

Icon

Helmet designed by Leonidas Ikonomou from The Noun Project.