0.8.1: pytest based test suite is failing

[kloczek]

Just normal build, install and test cycle used on building package from non-root account:

• "setup.py build"
• "setup.py install --root </install/prefix>"
• "pytest with PYTHONPATH pointing to setearch and sitelib inside </install/prefix>

+ PYTHONDONTWRITEBYTECODE=1
+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-betamax-0.8.1-14.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-betamax-0.8.1-14.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra
...................F..F.FF.........................................................FF....................................................                            [100%]
================================================================================= FAILURES =================================================================================
____________________________________________________________ TestRecordOnce.test_replays_response_from_cassette ____________________________________________________________

self = <tests.integration.test_record_modes.TestRecordOnce testMethod=test_replays_response_from_cassette>

    def test_replays_response_from_cassette(self):
        s = self.session
        with Betamax(s).use_cassette('test_replays_response') as betamax:
            self.cassette_path = betamax.current_cassette.cassette_path
            assert betamax.current_cassette.is_empty() is True
            r0 = s.get('http://httpbin.org/get')
            assert r0.status_code == 200
            assert betamax.current_cassette.interactions != []
            assert len(betamax.current_cassette.interactions) == 1
            r1 = s.get('http://httpbin.org/get')
            assert len(betamax.current_cassette.interactions) == 2
            assert r1.status_code == 200
            r0_headers = r0.headers.copy()
            r0_headers.pop('Date')
            r0_headers.pop('Age', None)
            r0_headers.pop('X-Processed-Time', None)
            r1_headers = r1.headers.copy()
            r1_headers.pop('Date')
            r1_headers.pop('Age', None)
            r1_headers.pop('X-Processed-Time', None)
            # NOTE(sigmavirus24): This fails if the second request is
            # technically a second later. Ignoring the Date headers allows
            # this test to succeed.
            # NOTE(hroncok): httpbin.org added X-Processed-Time header that
            # can possibly differ (and often does)
            assert r0_headers == r1_headers
>           assert r0.content == r1.content
E           AssertionError: assert b'{\n  "args"...org/get"\n}\n' == b'{\n  "args"...org/get"\n}\n'
E             At index 208 diff: b'5' != b'2'
E             Full diff:
E               (
E                b'{\n  "args": {}, \n  "headers": {\n    "Accept": "*/*", \n    "Accept-Encodi'
E                b'ng": "gzip, deflate", \n    "Host": "httpbin.org", \n    "User-Agent": "py'
E             -  b'thon-requests/2.25.1", \n    "X-Amzn-Trace-Id": "Root=1-60ee1640-2b89e4dc'
E             ?                                                                     ^^^^^^ ^...
E
E             ...Full output truncated (9 lines hidden), use '-vv' to show

tests/integration/test_record_modes.py:43: AssertionError
___________________________________________________ TestRecordNewEpisodes.test_records_new_events_with_existing_cassette ___________________________________________________

self = <tests.integration.test_record_modes.TestRecordNewEpisodes testMethod=test_records_new_events_with_existing_cassette>

    def test_records_new_events_with_existing_cassette(self):
        s = self.session
        opts = {'record': 'new_episodes'}
        with Betamax(s).use_cassette('test_record_new', **opts) as betamax:
            cassette = betamax.current_cassette
            self.cassette_path = cassette.cassette_path
            assert cassette.interactions != []
>           assert len(cassette.interactions) == 4
E           AssertionError: assert 2 == 4
E            +  where 2 = len([<betamax.cassette.interaction.Interaction object at 0x7f8024dd9e80>, <betamax.cassette.interaction.Interaction object at 0x7f8024efbcd0>])
E            +    where [<betamax.cassette.interaction.Interaction object at 0x7f8024dd9e80>, <betamax.cassette.interaction.Interaction object at 0x7f8024efbcd0>] = <betamax.cassette.cassette.Cassette object at 0x7f8024dd98b0>.interactions

tests/integration/test_record_modes.py:77: AssertionError
_______________________________________________________________ TestRecordAll.test_records_new_interactions ________________________________________________________________

self = <tests.integration.test_record_modes.TestRecordAll testMethod=test_records_new_interactions>

    def test_records_new_interactions(self):
        s = self.session
        opts = {'record': 'all'}
        with Betamax(s).use_cassette('test_record_all', **opts) as betamax:
            cassette = betamax.current_cassette
            self.cassette_path = cassette.cassette_path
            assert cassette.interactions != []
>           assert len(cassette.interactions) == 5
E           AssertionError: assert 3 == 5
E            +  where 3 = len([<betamax.cassette.interaction.Interaction object at 0x7f8024e04b20>, <betamax.cassette.interaction.Interaction object at 0x7f8024e04040>, <betamax.cassette.interaction.Interaction object at 0x7f8024e04160>])
E            +    where [<betamax.cassette.interaction.Interaction object at 0x7f8024e04b20>, <betamax.cassette.interaction.Interaction object at 0x7f8024e04040>, <betamax.cassette.interaction.Interaction object at 0x7f8024e04160>] = <betamax.cassette.cassette.Cassette object at 0x7f8024e040a0>.interactions

tests/integration/test_record_modes.py:114: AssertionError
_______________________________________________________________ TestRecordAll.test_replaces_old_interactions _______________________________________________________________

self = <tests.integration.test_record_modes.TestRecordAll testMethod=test_replaces_old_interactions>

    def test_replaces_old_interactions(self):
        s = self.session
        opts = {'record': 'all'}
        with Betamax(s).use_cassette('test_record_all', **opts) as betamax:
            cassette = betamax.current_cassette
            self.cassette_path = cassette.cassette_path
            assert cassette.interactions != []
>           assert len(cassette.interactions) == 5
E           AssertionError: assert 3 == 5
E            +  where 3 = len([<betamax.cassette.interaction.Interaction object at 0x7f8024e23640>, <betamax.cassette.interaction.Interaction object at 0x7f80249b7f10>, <betamax.cassette.interaction.Interaction object at 0x7f8024e38130>])
E            +    where [<betamax.cassette.interaction.Interaction object at 0x7f8024e23640>, <betamax.cassette.interaction.Interaction object at 0x7f80249b7f10>, <betamax.cassette.interaction.Interaction object at 0x7f8024e38130>] = <betamax.cassette.cassette.Cassette object at 0x7f8024e23a90>.interactions

tests/integration/test_record_modes.py:129: AssertionError
_____________________________________________________________ TestPyTestFixture.test_adds_stop_as_a_finalizer ______________________________________________________________
Fixture "betamax_recorder" called directly. Fixtures are not meant to be called directly,
but are created automatically when test functions request them as parameters.
See https://docs.pytest.org/en/stable/fixture.html for more information about fixtures, and
https://docs.pytest.org/en/stable/deprecations.html#calling-fixtures-directly about how to update your code.
_____________________________________________________________ TestPyTestFixture.test_auto_starts_the_recorder ______________________________________________________________
Fixture "betamax_recorder" called directly. Fixtures are not meant to be called directly,
but are created automatically when test functions request them as parameters.
See https://docs.pytest.org/en/stable/fixture.html for more information about fixtures, and
https://docs.pytest.org/en/stable/deprecations.html#calling-fixtures-directly about how to update your code.
============================================================================= warnings summary =============================================================================
tests/integration/test_fixtures.py::TestPyTestParametrizedFixtures::test_pytest_fixture[https://httpbin.org/get]
tests/integration/test_fixtures.py::test_pytest_parametrize_with_filesystem_problematic_chars[aaa\\bbb]
tests/integration/test_fixtures.py::test_pytest_parametrize_with_filesystem_problematic_chars[ccc:ddd]
tests/integration/test_fixtures.py::test_pytest_parametrize_with_filesystem_problematic_chars[eee*fff]
  /home/tkloczko/rpmbuild/BUILDROOT/python-betamax-0.8.1-14.fc35.x86_64/usr/lib/python3.8/site-packages/betamax/fixtures/pytest.py:128: DeprecationWarning: betamax_parametrized_recorder and betamax_parametrized_session will be removed in betamax 1.0. Their behavior will be the default.
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/warnings.html
========================================================================= short test summary info ==========================================================================
FAILED tests/integration/test_record_modes.py::TestRecordOnce::test_replays_response_from_cassette - AssertionError: assert b'{\n  "args"...org/get"\n}\n' == b'{\n  "arg...
FAILED tests/integration/test_record_modes.py::TestRecordNewEpisodes::test_records_new_events_with_existing_cassette - AssertionError: assert 2 == 4
FAILED tests/integration/test_record_modes.py::TestRecordAll::test_records_new_interactions - AssertionError: assert 3 == 5
FAILED tests/integration/test_record_modes.py::TestRecordAll::test_replaces_old_interactions - AssertionError: assert 3 == 5
FAILED tests/unit/test_fixtures.py::TestPyTestFixture::test_adds_stop_as_a_finalizer
FAILED tests/unit/test_fixtures.py::TestPyTestFixture::test_auto_starts_the_recorder
6 failed, 131 passed, 4 warnings in 11.16s

[threexc]

Seeing the same issue when running betamax tests inside Yocto Project images. Anyone have any insight?

[sigmavirus24]

The tests should probably use pytest-httpbin instead of relying on httpbin.org which has been sold and re-sold several times

[mtelka]

See also #184.