=head1 NAME

Test::X1 - A lightweight test manager

=head1 SYNOPSIS

  use Test::X1;
  use Test::More;
  
  test {
      my $c = shift;
      ok 1;
      ok 2;
      $c->done;
  };
  
  test {
      my $c = shift;
      my $timer; $timer = AnyEvent->timer(
          after => 2,
          cb => sub {
              test {
                  ok 2;
                  is 3, 0;
                  undef $timer;
                  $c->done;
                  undef $c;
              } $c;
          },
      );
      ok 1;
  } n => 3, name => ['anyevent', 'callback'];

  test {
      my $c = shift;
      ok 1;
      $c->done;
  } n => 1;
  
  run_tests;

  ok 1 - [1] - [1]
  ok 2 - [1] - [2]
  ok 3 - [2] anyevent.callback - [1]
  ok 4 - [3] - [1]
  ok 5 - [2] anyevent.callback - [2]
  not ok 6 - [2] anyevent.callback - [3]
  #   Failed test '[2] anyevent.callback - [3]'
  #   at lib/Test/X1.pod line 24.
  #          got: '3'
  #     expected: '0'
  # [2] anyevent.callback: 1 test failed
  1..6
  # Looks like you failed 1 test of 6.

=head1 DESCRIPTION

The C<Test::X1> module defines a simple, lightweight, and
L<AnyEvent>-compatible test management framework for
L<Test::Builder>-based tests.  It supports automatic naming of tests,
partial execution, concurrent execution, and other useful features.

=head1 USAGE

By C<use>ing the C<Test::X1> module, functions for defining and
running tests are exported to the caller package.  Note that it only
exports test management functions.  You have to import or define your
favorite test assertion sets such as L<Test::More> or
L<Test::Differences> by yourself.  The C<Test::X1> test framework only
assumes that you are writing a L<Test::Builder>-based test script.  As
long as test assertion functions are build upon L<Test::Builder>'s
framework, you can choose any test module.

Note that some test modules are incompatible with L<AnyEvent>-based
asynchronous invocation of callbacks.  For example, C<subtest>s of
L<Test::More> does not work well.

=head2 Tests

A C<Test::X1>-based test script consists of one or more tests.  A
I<test> is a group of I<subtests>, which are L<Test::Builder>-based
assertion functions such as I<is> and I<ok>.  A test can be defined by
enclosing subtests and related codes by the C<test{}> function:

  test {
      my $c = shift;
      
      is 2 * 4, 8;
      is 40 - 30, 10;
      
      $c->done;
  };

This usage of the C<test{}> function is sometimes referred to as
I<test definition>.  The code block in the test definition is invoked
with an argument, the context object for the test, I<$c>.  The C<<
$c->done >> method has to be called when all subtests in the test are
done.  (See L</"Context objects"> for more information on I<$c>.)

You can specify the number of subtests in a test by specifying the
C<n> option to the test definition.  If the C<n> option is specified,
it is verified that the expected number of subtests are done before
the C<< $c->done >> metohd is invoked.  It is considered as good
practice to specify the number of subtests, especially when there are
loops in the test, such that it can be confirmed that all expected
subtests are really executed.  Since the number of subtests can be
frequently changed in some cases, or it could even be unknown a
priori, the C<n> option is not required.

The test can be named by specifying the C<name> option to the test
definition.  The name of the test is C<[n]>, where I<n> is the
sequential number of the test, followed by space and C<name> value if
any, e.g. C<[1] hoge> or C<[2]>.

The name of the test is used to construct the name of the subtests in
the test.

The C<TEST_METHOD> environment variable can be used to specify the
regular expression used to filter the tests to run by their names.  If
the environment variable is not set, all tests are executed.
Otherwise, only the tests whose name matches the pattern are executed.

The C<TEST_METHOD_EXCLUDED> environment variable can be used to
specify the regular expression used to filter the tests to I<not> run
by their names.  If the environment variable is not set, no test is
excluded.

See also L</Naming> for how names are handled and used by the module.

For example, consider the following test script:

  # t/foo.t
  use Test::X1;
  use Test::More;
  
  my $x = 10;
  
  test {
      my $c = shift;
      is $x, 10;
      $c->done;
  } name => 'abc2';
  
  test {
      my $c = shift;
      is $x * 2, 20;
      $c->done;
  };
  
  run_tests;

The first test is named as C<[1] abc2>, while the second test is named
as C<[2]>.  If the script is executed with no C<TEST_METHOD>, both of
them are executed.  If the script is executed with C<TEST_METHOD=abc>,
only the first test is executed.  If the script is executed with
C<TEST_METHOD=2>, both of them are executed.

The C<run_tests> function runs the tests defined by C<test{}>
functions.  It must be invoked exactly once in the test script, after
all tests are defined.  You can run any preparation code before the
invocation, or cleanup code after the invocation.  The C<run_tests>
function does not return until all tests are done.

=head2 Context objects

The code block in the test definition is invoked with an argument,
i.e. the context object, in the C<@_> array.  The context object
provides several utility methods for the test and is created
specifically for the test.  In other word, different context objects
are created for different test definitions.  In this document, the
context object is sometimes referred to as C<$c>.

The most significant method of the test context object is the C<done>
method.  The C<< $c->done >> method must be invoked when and only when
all the subtests in the test has been done, successfully or not.  Once
the method has been invoked, no subtest can be performed in the test.
If the method is not invoked in the test, an error would be reported.

The most basic usage would be invoking the C<< $c->done >> method just
before the end of the test code:

  test {
      my $c = shift;
      
      ... tests ...
      
      $c->done;
  };

If there are callbacks, the C<< $c->done >> method should be invoked
at the end of the last callback (the one invoked finally):

  test {
      my $c = shift;
      my $timer = AE::timer 10, 0, sub {
          test {
              ...;
              $c->done;
              undef $c;
          } $c;
      };
  };

The inner C<test{}> statement is a test block; See L</"Test blocks">
for details.  Also note that C<undef $c> is executed after C<<
$c->done >> is invoked.  Though this is not required for this
particular case, it might be a good practice to C<undef> the context
object reference when it is C<done>'ed within a callback function as
it would delete a possible loop reference when the context object has
the reference to some object and the object then contains the
reference to the callback function which contains the reference to the
context object.

More complex example:

  test {
      my $c = shift;
      
      my $cv = AE::cv;
      
      # This callback will be executed after all of
      # following callbacks are invoked.
      $cv->begin (sub { test { $c->done } $c });
      
      $cv->begin;
      my $timer1 = AE::timer 10, 0, sub { $cv->end };
      
      $cv->begin;
      my $timer2 = AE::timer 4, 0, sub { $cv->end };
      
      ...
      
      $cv->end;
  };

Application test framework built on top of the module might define
additional methods to context objects.  See L</"Subclassing"> on the
guideline for extending the context object interface.

=head2 Test blocks

Another usage of the C<test{}> function is defining a B<test block>.
They are different from the test definitions in that they are used
within some test definition and takes the test context object I<$c> as
the first argument.

Test blocks are typically used within callback functions in a test; in
fact there has to be a test block within an asynchronously invoked
callback function:

  test {
      my $c = shift;
      AE::io *STDIN, 0, sub {
          test {
              is scalar <STDIN>, "hoge";
              $c->done;
          } $c;
      };
  };

A test block gives the test context as encapsulated by I<$c> to the
subtests within the block.  You have to enclose subtests within
callback functions by yourself, unfortunately, otherwise the test
manager losts the association of subtests and their "parent" test, due
to the asynchronousness of the invocation of the callback.

Test block can be named by the C<name> option:

  test {
      test {
          ok 1, 'Test X';
      } $c, name => 'hoge';
  };
  
  # ok 1 - [1] - [1] hoge Test X

The C<TEST_BLOCK_SKIP> environment variable can be used to specify the
regular expression to skip the test blocks with names matching to the
pattern.  For example, C<TEST_BLOCK_SKIP=og.$> would prevent the C<ok>
statement in the example above from executed.  See also L</Naming>.

=head2 Naming

Tests, test blocks, and subtests can be named.  Instead of a scalar
value, representing the literal string, an array reference containing
string components can be specified.

  test {
      my $c = shift;
      test {
          is $hoge, $fuga, 'Subtest 1';
      } $c, name => 'Test block 1';
      $c->done;
  } name => 'Test 1';

  test {
      my $c = shift;
      test {
          is $hoge, $fuga, ['Subtest', 2];
      } $c, name => ['Test block', 2];
      $c->done;
  } name => ['Test', 2];

Naming by array reference would be particularly useful when defining
multiple tests by iteration:

  for my $value (1, 2, 30, 120) {
      test {
          my $c = shift;
          like $c, qr{^\d+$};
          $c->done;
      } name => ['Test', $value];
  }

If the name is represented as an array reference, its items are joined
by C<.> (period) before actually used to output results, or filter
tests by environment variables, i.e. C<TEST_METHOD>,
C<TEST_METHOD_EXCLUDED>, and C<TEST_BLOCK_SKIP>.  Any empty string is
replaced by C<(empty)> and any C<undef> value is replaced by
C<(undef)>.

=head2 Waiting for a condvar

An L<AnyEvent> condvar can be specified as the C<wait> parameter to a
test definition (I<not> test block!) to wait for the condvar to
receive a value.  The received value can be accessed from the C<<
$c->received_data >> method of the context object.

  my $cv = AE::cv;
  test {
      my $c = shift;
      is $c->received_data, 123;
      $c->done;
  } wait => $cv;

Instead of a codevar object, a code reference which, when invoked,
returns a condvar object or the C<undef> value can be specified as the
C<wait> value.  This practice is rather recommended as it would
prevend the condvar object from being instantiated when the target
tests are filtered by C<TEST_METHOD> environment variable.  For
example,

  test {
      my $c = shift;
      is $c->received_data, 123;
      $c->done;
  } wait => sub { start_server_and_return_cv () };

... does not start the server for the test when the test is excluded
from the execution.

The default C<wait> value, used when no C<wait> parameter is
explicitly specified to test definitions, can be provided by
subclassing (see L</"Subclassing"> for details) and defining C<<
$tm->default_test_wait_cv >> method returning a condvar (or C<undef>)
in the test manager subclass.  In this case, by explicitly setting
C<undef> value for the C<wait> parameter of test definitions, this
default can be cleared.  (See C<t/cv-wait-default.t> test script for
examples.)

The C<wait> value can also be a hash reference (or a code reference
which returns a hash reference).  The C<cv> key of the hash reference
can have the condvar object as the value.  The C<destroy_as_cv> key
can contain the code reference, which will be invoked after relevant
tests have been run.  The code should be useful to stop the server
started by the C<cv> condvar's preparation, for example.  The code
must return a condvar object, whose callback will be invoked after the
destroy process has been done.  The same C<destroy_as_cv> code is
invoked only once.  If the code is specified as part of the C<wait>
value of multiple tests, it is only invoked after all of them has been
executed.  Example:

  test {
      ...
  } wait => {cv => sub {
      return $server->start_as_cv;
  }, destroy_as_cv => sub {
      return $server->stop_as_cv;
  }};

Additionally, the C<timeout> value can also be specified in the
C<wait> hash reference.  Its default value is 60.  After the seconds
of the timeout elapse, if the C<wait> condvar's callback is not
invoked, the associated test fails.  The timeout value is also applied
to the C<context_begin> and C<context_end> methods of the C<<
$c->received_data >> object; These methods have to invoke the callback
before the timeout.  Please note that they should not block the entire
script, otherwise the timeout will not work.

=head2 Concurrent execution of tests

Thanks to L<AnyEvent> framework, tests (as defined by outermost
C<test{}> functions) can be concurrenrly executed when they are
written in non-blocking way using L<AnyEvent>.

Consider the following test script fragment:

  test {
      my $c = shift;
      ok 'Subtest #1.1';
      AnyEvent::Example->something(cb => sub {
          test {
              is $_[0], 'hoge', 'Subtest #1.2';
              $c->done;
          } $c;
      });
  };
  
  test {
      my $c = shift;
      ok 'Subtest #2.1';
      AnyEvent::Example->something(cb => sub {
          test {
              is $_[0], 'hoge', 'Subtest #2.2';
              $c->done;
              undef $c;
          } $c;
      });
  };
  
  run_tests;

In this case, execution order of Subtests #1.2 and #2.2 is unclear at
all, depending on how long C<< AnyEvent::Example->something >> defers
the execution of the callbacks.  (Please also note that, although in
the current implementation Subtest #2.1 is always executed after
Subtest #1.1, as that test is defined by C<test{}> after the other
test, this is not guaranteed and you should not reply on this exact
order.  Future version of the module could introduce shuffling
execution mode, for instance.)

Anyway, we can describe this situation that multiple tests are
concurrently executed.  By default, at most five tests are
concurrently executed by C<Test::X1>.  Setting a number to the
C<TEST_MAX_CONCUR> environment variable can override this default, if
desired.  C<TEST_MAX_CONCUR=1> disables this concurrency, which will
be useful for debugging purposes in particular.

=head2 More test option

The C<timeout> option of the test definition (not a test block)
specifies the timeout in seconds.  The test must end within the
seconds after the test is started (not inlcuding any C<wait>,
C<context_before>, and C<context_after> processing).  The default
value is 60.  Note that the timeout might be applied as intended if
the test blocks the script, by, e.g., blocking I/O access, C<system>,
or C<sleep>.

=head2 Subclassing

Test scripts often need application-specific factory functions and/or
utility functions to create expected precondition or to manage states
of tested environment.  For example, a test for Web application would
need to start a Web server before any test and stop the server after
tests.  A test for database operation would want to insert a number of
records into the database at some points in the test code.  Although
they can be implemented orthogonally from the C<Test::X1>'s framework,
subclassing of the L<Test::X1> class should be a good candidate if you
want to control the lifetime of such temporary objects by relating
them with lexical scopes of tests.

If you'd like to extend C<Test::X1> for your application C<My>, the
subclass module, C<My/Test/X1.pm>, would look like:

  package My::Test::X1;
  use Test::X1 ();
  Test::X1::define_functions(__PACKAGE__);
  
  package My::Test::X1::Manager;
  
  sub my_create_database { ... }
  sub my_drop_database { ... }
  
  sub stop_test_manager { shift->my_drop_database }
  
  package My::Test::X1::Context;
  
  sub my_insert_data { ... }
  
  1;

The C<Test::X1> module is C<use>d without importing any function, then
the C<Test::X1::define_functions> function is invoked with the
package, i.e. C<My::Test::X1>.  Additional methods are defined in
subclasses of test manager and context objects.  Then, in your test
script, instead of directly C<use>ing C<Test::X1> module, load your
module:

  use My::Test::X1;
  my $tm = get_test_manager;
  $tm->my_create_database;
  
  test {
      my $c = shift;
      $c->my_insert_data;
      ...
  };
  
  run_tests;

It is considered as good practice to prepend a short prefix taken from
the subclass name (C<my_> in this example) to the method names defined
by subclasses such that future additions to base classes will not
conflict with them.

Both test manager and context objects are blessed hash references.
Subclasses can use hashes to save their data associated with objects.
Such hash keys should be prefixed by subclass names as well.

=head1 EXPORTED FUNCTIONS

By C<use>ing the C<Test::X1> module, your test script imports
following functions:

=over 4

=item $tm = get_test_manager

Obtain the instance of the test manager for the test script.  The test
manager object is singleton; the function always returns the same
object.

=item test { CODE } NAME => VALUE, ...; (Test definition)

Define a test.  It must be invoked outside of any other C<test{}>
function call.  It must be invoked before C<run_tests> function call.

The code block is expected to run one or more subtests.  The number of
subtests is expected to be equal to the C<n> parameter value, if
specified.  The code is expected to not throw any exception.

The code block, when invoked, receives the text context object I<$c>
for the test as the argument.  The C<< $c->done >> method is expected
to be invoked after all subtests are run.

See also L</Tests> for usage.

After the code block, zero or more name/value pairs can be specified.
Following name/value pairs are supported:

=over 4

=item n => non-negative integer

Specify the expected number of substests in the test.  If the
parameter is not specified, number of subtests are not known a priori.

=item name => string or array reference of strings

Name the test.  See L</Naming> for details.

=item wait => anyevent-condvar

Specify a L<AnyEvent::CondVar> object to wait before the execution of
the test.  See also L</"Waiting for a condvar">.

=back

=item test { CODE } $c, NAME => VALUE, ...; (Test block)

Define a subpart of test (or a block of subtests).  It must be invoked
within the code part of a test definition.

See also L</"Test blocks">.

The context object for the current test must be specified as the
argument next to the code block.  Additionally, zero or more
name/value pairs can be specified.

=over 4

=item name => string or array reference of strings

Name the test block.  See L</Naming> for details.

=back

=item run_tests

Run the defined tests.  The function returns after all the tests has
been executed and done.  This function must be invoked exactly once in
the test script.  After the function call, the C<test{}> function (for
defining a test) must not be invoked.

=back

=head1 TEST MANAGER OBJECT

The test manager object is the object created for the test script,
holding references to tests in the test script and monitoring their
results.  The test manager object is singleton; there is at most one
test manager object at one time.

The C<Text::X1> class (and its subclasses) exports C<get_test_manager>
function, which takes no argument, returning the current test manager
object.

If you are writing simple test scripts, you don't have to directly
access test manager usually.  The exported functions explained in the
previous section are in fact invoking appropriate methods of the test
manager object.

Following methods can be invoked or defined when you are subclassing
the test manager object:

=over 4

=item $cv = $tm->default_test_wait_cv

This method can be overridden by subclasses, if desired.  This method
is expected to return an L<AnyEvent> condvar or C<undef>.  The value
returned by this method is used as the C<wait> parameter value of test
definitions, when it is not explicitly specified.

=item $hashref = $tm->context_args

This method can be overridden by subclasses, if desired.  This method
is invoked when test context objects are created.  It is expected to
return a hash reference containing name/value pairs passed as
arguments to the C<new> method of the test context class.  By default
it returns an empty hash reference.

Name/value pairs specified here can be accessed from the test context
object's blessed hash reference.  See L</"Subclassing"> for their
usage.

=item $tm->stop_test_manager

This method can be overridden by subclasses, if desired.  This method
is invoked before the test manager object is destructed.  It is
expected to be used to close anything opened by the test manager, if
necessary.  This method can be invoked more than once for an test
manager object.  The C<Test::X1> module does it's best effort to
invoke the method for the test manager object before Perl goes into
the global destruction phase.

=item $tm->diag($color, $message)

=item $tm->note($color, $message)

Print a diagnostic message or a note, through C<Test::Builder>'s
C<diag> or C<note> method.  For their usage, see L<Test::Builder> and
L<Test::More> documentations.

The first argument must be a color specification for
L<Term::ANSIColor>, e.g. C<"red"> or C<"">.

The second argument must be a diagnostic or note text, possibly
utf8-flagged.

=back

=head1 CONTEXT OBJECT

The context object is created for each test.  It provides several
information on the test, which can be used within test for debugging
purpose.  The context object is passed to the test as the first
argument.

=over 4

=item $name = $c->test_name

The compound name of the test, in a single character string.  However
the name is specified (or not), the method returns the single string
as used in TAP test name part.  See L</Naming>.

=item $data = $c->received_data

Return the data received from the L<AnyEvent> condvar specified to the
C<wait> parameter of the test definition for the current test.  See
also L</"Waiting for a condvar">.

The data can be any value, including the C<undef> value.  If the data
is an object which has C<context_begin> and C<context_end> methods,
they are invoked just after and just before the data is associated
with a context object.  (Please note that the data can be associated
with multiple context objects when the condvar is specified for
multiple tests.)  Both methods will receive a code reference as an
argument.  The methods are expected to invoke the code once the object
is ready for start or termination of the test.  Typical use case of
these methods is preparation and termination of a server process used
within the test.

Example of class for such an object:

  package My::Data;
  
  sub context_begin {
    my ($self, $code) = @_;
    $self->{rc}++;
    $code->();
  }
  
  sub context_end {
    my ($self, $code) = @_;
    $self->{rc}--;
    $self->stop_server unless $self->{rc};
    $code->();
  }

=item $c->diag($color, $message)

Print a diagnostic message, through C<Test::Builder>'s C<diag> method.
For their usage, see L<Test::Builder> and L<Test::More>
documentations.

The first argument must be a color specification for
L<Term::ANSIColor>, e.g. C<"red"> or C<"">.  (Please note that this
argument is ignored in this version.)

The second argument must be a diagnostic text, possibly utf8-flagged.

=item $c->done

Notify that the substests in the test is done.  This method must be
invoked exactly once for a test.  See also L</"Context objects">.

=back

=head1 LIMITATIONS

Subtests as implemented by L<Test::More> / L<Test::Builder> cannot be
used in the context of this module as they are globally stateful such
that concurrent execution of multiple different test introduced by
this module is incompatibile with them.  Please note that this
module's concept of subtests is different from those subtests.

=head1 EXAMPLES

See C<t/*.t> for more examples.

=head1 DEPENDENCY

This module requires Perl 5.10 or later.  In addition to core modules,
this module depends on L<Exporter::Lite> and L<AnyEvent>.

=head1 AUTHOR

Wakaba <[email protected]>.

=head1 HISTORY

This module is inspired by following modules: C<Test::Builder>,
C<Test::Class>, C<Test::More>.

This repository was located at
<https://github.com/wakaba/perl-test-x1> until 19 April 2023, then
transferred to <https://github.com/pawjy/perl-test-x1>.

=head1 LICENSE

Copyright 2012-2013 Hatena <https://www.hatena.ne.jp/>.

Copyright 2012-2017 Wakaba <[email protected]>.

This library is free software; you can redistribute it and/or modify
it under the same terms as Perl itself.

=cut