Tests failing on openSUSE Tumbleweed, but succeeding on Leap/SLES

[johanneskastl]

Hi all,

just as a warning, the ansible-sign package started failing its build recently, but only on openSUSE Tumbleweed. The build for openSUSE Leap / SLES is still fine, so this looks like some incompatibility introduced by newer versions of $SOMETHING in Tumbleweed.

In case you need more details, please reach out.

Kind Regards,
Johannes

[   22s] ------------------------------ Captured log call -------------------------------
[   22s] [33mWARNING [0m gnupg:gnupg.py:1087 gpg returned a non-zero error code: 2
[   22s] 
[   22s] ---------- coverage: platform linux, python 3.11.5-final-0 -----------
[   22s] Name                                                   Stmts   Miss Branch BrPart  Cover   Missing
[   22s] --------------------------------------------------------------------------------------------------
[   22s] src/ansible_sign/__init__.py                               6      0      0      0   100%
[   22s] src/ansible_sign/checksum/__init__.py                      2      0      0      0   100%
[   22s] src/ansible_sign/checksum/base.py                         81      0     36      1    99%   165->171
[   22s] src/ansible_sign/checksum/differ/__init__.py               1      0      0      0   100%
[   22s] src/ansible_sign/checksum/differ/base.py                  28      2     12      1    92%   41, 61
[   22s] src/ansible_sign/checksum/differ/distlib_manifest.py      25      0     12      0   100%
[   22s] src/ansible_sign/cli.py                                  189     31     54     14    80%   172, 175->177, 180, 190, 202, 207-210, 229-231, 236-238, 241-242, 244->256, 249-254, 257, 305-306, 311-312, 336, 351-353, 374
[   22s] src/ansible_sign/signing/__init__.py                       1      0      0      0   100%
[   22s] src/ansible_sign/signing/base.py                          18      0      0      0   100%
[   22s] src/ansible_sign/signing/gpg/__init__.py                   2      0      0      0   100%
[   22s] src/ansible_sign/signing/gpg/signer.py                    26      0     10      1    97%   57->56
[   22s] src/ansible_sign/signing/gpg/verifier.py                  33      3     10      3    86%   24, 28, 36
[   22s] --------------------------------------------------------------------------------------------------
[   22s] TOTAL                                                    412     36    134     20    89%
[   22s] 
[   22s] [36m[1m=========================== short test summary info ============================[0m
[   22s] [31mFAILED[0m tests/test_cli.py::[1mtest_gpg_verify_manifest_scenario[valid checksum file and signature][0m - AssertionError: assert 'GPG signature verification succeeded' in '[\x1b[91m...
[   22s] [31mFAILED[0m tests/test_cli.py::[1mtest_gpg_verify_manifest_scenario[valid signature but broken checksum file][0m - AssertionError: assert 'Invalid line encountered in checksum manifest' in '...
[   22s] [31mFAILED[0m tests/test_cli.py::[1mtest_gpg_verify_manifest_scenario[checksum file with wrong hashes][0m - AssertionError: assert 'Checksum validation failed.' in '[\x1b[91mERROR\x1b...
[   22s] [31mFAILED[0m tests/test_cli.py::[1mtest_gpg_verify_manifest_scenario[broken MANIFEST.in after signing][0m - assert "An error was encountered while parsing MANIFEST.in: unknown action ...
[   22s] [31m================= [31m[1m4 failed[0m, [32m35 passed[0m, [33m1 deselected[0m[31m in 11.82s[0m[31m ==================[0m
[   22s] error: Bad exit status from /var/tmp/rpm-tmp.6Y6sEU (%check)

Full error output:

[   22s] =================================== FAILURES ===================================
[   22s] [31m[1m_____ test_gpg_verify_manifest_scenario[valid checksum file and signature] _____[0m
[   22s] 
[   22s] capsys = <_pytest.capture.CaptureFixture object at 0x7f391b578990>
[   22s] request = <FixtureRequest for <Function test_gpg_verify_manifest_scenario[valid checksum file and signature]>>
[   22s] project_fixture = 'signed_project_and_gpg'
[   22s] exp_stdout_substr = 'GPG signature verification succeeded'
[   22s] exp_stderr_substr = '', exp_rc = 0
[   22s] 
[   22s]     @pytest.mark.parametrize(
[   22s]         "project_fixture, exp_stdout_substr, exp_stderr_substr, exp_rc",
[   22s]         [
[   22s]             ("signed_project_and_gpg", "GPG signature verification succeeded", "", 0),
[   22s]             ("signed_project_broken_manifest", "Invalid line encountered in checksum manifest", "", 1),
[   22s]             ("signed_project_missing_manifest", "Checksum manifest file does not exist:", "", 1),
[   22s]             ("signed_project_modified_manifest", "Checksum validation failed.", "", 2),
[   22s]             ("signed_project_with_different_gpg_home", "Re-run with the global --debug flag", "", 3),
[   22s]             ("signed_project_broken_manifest_in", "An error was encountered while parsing MANIFEST.in: unknown action 'invalid-directive'", "", 1),
[   22s]         ],
[   22s]         ids=[
[   22s]             "valid checksum file and signature",
[   22s]             "valid signature but broken checksum file",
[   22s]             "missing checksum file entirely",
[   22s]             "checksum file with wrong hashes",
[   22s]             "matching pubkey does not exist in gpg home",
[   22s]             "broken MANIFEST.in after signing",
[   22s]         ],
[   22s]     )
[   22s]     def test_gpg_verify_manifest_scenario(capsys, request, project_fixture, exp_stdout_substr, exp_stderr_substr, exp_rc):
[   22s]         """
[   22s]         Test `ansible-sign project gpg-verify` given different project directory
[   22s]         scenarios (fixtures).
[   22s]         """
[   22s]         (project_root, gpg_home) = request.getfixturevalue(project_fixture)
[   22s]         keyring = os.path.join(gpg_home, "pubring.kbx")
[   22s]         args = [
[   22s]             "project",
[   22s]             "gpg-verify",
[   22s]             f"--keyring={keyring}",
[   22s]             str(project_root),
[   22s]         ]
[   22s]         rc = main(args)
[   22s]         captured = capsys.readouterr()
[   22s] >       assert exp_stdout_substr in captured.out
[   22s] [1m[31mE       AssertionError: assert 'GPG signature verification succeeded' in '[\x1b[91mERROR\x1b[0m] GPG signature verification failed.\n[\x1b[94mNOTE \x1b[0m] Re-run with the global --debug flag for more information.\n'[0m
[   22s] [1m[31mE        +  where '[\x1b[91mERROR\x1b[0m] GPG signature verification failed.\n[\x1b[94mNOTE \x1b[0m] Re-run with the global --debug flag for more information.\n' = CaptureResult(out='[\x1b[91mERROR\x1b[0m] GPG signature verification failed.\n[\x1b[94mNOTE \x1b[0m] Re-run with the global --debug flag for more information.\n', err='').out[0m
[   22s] 
[   22s] [1m[31mtests/test_cli.py[0m:163: AssertionError
[   22s] ------------------------------ Captured log call -------------------------------
[   22s] [33mWARNING [0m gnupg:gnupg.py:1087 gpg returned a non-zero error code: 2
[   22s] [31m[1m_ test_gpg_verify_manifest_scenario[valid signature but broken checksum file] __[0m
[   22s] 
[   22s] capsys = <_pytest.capture.CaptureFixture object at 0x7f391b3fbfd0>
[   22s] request = <FixtureRequest for <Function test_gpg_verify_manifest_scenario[valid signature but broken checksum file]>>
[   22s] project_fixture = 'signed_project_broken_manifest'
[   22s] exp_stdout_substr = 'Invalid line encountered in checksum manifest'
[   22s] exp_stderr_substr = '', exp_rc = 1
[   22s] 
[   22s]     @pytest.mark.parametrize(
[   22s]         "project_fixture, exp_stdout_substr, exp_stderr_substr, exp_rc",
[   22s]         [
[   22s]             ("signed_project_and_gpg", "GPG signature verification succeeded", "", 0),
[   22s]             ("signed_project_broken_manifest", "Invalid line encountered in checksum manifest", "", 1),
[   22s]             ("signed_project_missing_manifest", "Checksum manifest file does not exist:", "", 1),
[   22s]             ("signed_project_modified_manifest", "Checksum validation failed.", "", 2),
[   22s]             ("signed_project_with_different_gpg_home", "Re-run with the global --debug flag", "", 3),
[   22s]             ("signed_project_broken_manifest_in", "An error was encountered while parsing MANIFEST.in: unknown action 'invalid-directive'", "", 1),
[   22s]         ],
[   22s]         ids=[
[   22s]             "valid checksum file and signature",
[   22s]             "valid signature but broken checksum file",
[   22s]             "missing checksum file entirely",
[   22s]             "checksum file with wrong hashes",
[   22s]             "matching pubkey does not exist in gpg home",
[   22s]             "broken MANIFEST.in after signing",
[   22s]         ],
[   22s]     )
[   22s]     def test_gpg_verify_manifest_scenario(capsys, request, project_fixture, exp_stdout_substr, exp_stderr_substr, exp_rc):
[   22s]         """
[   22s]         Test `ansible-sign project gpg-verify` given different project directory
[   22s]         scenarios (fixtures).
[   22s]         """
[   22s]         (project_root, gpg_home) = request.getfixturevalue(project_fixture)
[   22s]         keyring = os.path.join(gpg_home, "pubring.kbx")
[   22s]         args = [
[   22s]             "project",
[   22s]             "gpg-verify",
[   22s]             f"--keyring={keyring}",
[   22s]             str(project_root),
[   22s]         ]
[   22s]         rc = main(args)
[   22s]         captured = capsys.readouterr()
[   22s] >       assert exp_stdout_substr in captured.out
[   22s] [1m[31mE       AssertionError: assert 'Invalid line encountered in checksum manifest' in '[\x1b[91mERROR\x1b[0m] GPG signature verification failed.\n[\x1b[94mNOTE \x1b[0m] Re-run with the global --debug flag for more information.\n'[0m
[   22s] [1m[31mE        +  where '[\x1b[91mERROR\x1b[0m] GPG signature verification failed.\n[\x1b[94mNOTE \x1b[0m] Re-run with the global --debug flag for more information.\n' = CaptureResult(out='[\x1b[91mERROR\x1b[0m] GPG signature verification failed.\n[\x1b[94mNOTE \x1b[0m] Re-run with the global --debug flag for more information.\n', err='').out[0m
[   22s] 
[   22s] [1m[31mtests/test_cli.py[0m:163: AssertionError
[   22s] ------------------------------ Captured log call -------------------------------
[   22s] [33mWARNING [0m gnupg:gnupg.py:1087 gpg returned a non-zero error code: 2
[   22s] [31m[1m______ test_gpg_verify_manifest_scenario[checksum file with wrong hashes] ______[0m
[   22s] 
[   22s] capsys = <_pytest.capture.CaptureFixture object at 0x7f391b5ca310>
[   22s] request = <FixtureRequest for <Function test_gpg_verify_manifest_scenario[checksum file with wrong hashes]>>
[   22s] project_fixture = 'signed_project_modified_manifest'
[   22s] exp_stdout_substr = 'Checksum validation failed.', exp_stderr_substr = ''
[   22s] exp_rc = 2
[   22s] 
[   22s]     @pytest.mark.parametrize(
[   22s]         "project_fixture, exp_stdout_substr, exp_stderr_substr, exp_rc",
[   22s]         [
[   22s]             ("signed_project_and_gpg", "GPG signature verification succeeded", "", 0),
[   22s]             ("signed_project_broken_manifest", "Invalid line encountered in checksum manifest", "", 1),
[   22s]             ("signed_project_missing_manifest", "Checksum manifest file does not exist:", "", 1),
[   22s]             ("signed_project_modified_manifest", "Checksum validation failed.", "", 2),
[   22s]             ("signed_project_with_different_gpg_home", "Re-run with the global --debug flag", "", 3),
[   22s]             ("signed_project_broken_manifest_in", "An error was encountered while parsing MANIFEST.in: unknown action 'invalid-directive'", "", 1),
[   22s]         ],
[   22s]         ids=[
[   22s]             "valid checksum file and signature",
[   22s]             "valid signature but broken checksum file",
[   22s]             "missing checksum file entirely",
[   22s]             "checksum file with wrong hashes",
[   22s]             "matching pubkey does not exist in gpg home",
[   22s]             "broken MANIFEST.in after signing",
[   22s]         ],
[   22s]     )
[   22s]     def test_gpg_verify_manifest_scenario(capsys, request, project_fixture, exp_stdout_substr, exp_stderr_substr, exp_rc):
[   22s]         """
[   22s]         Test `ansible-sign project gpg-verify` given different project directory
[   22s]         scenarios (fixtures).
[   22s]         """
[   22s]         (project_root, gpg_home) = request.getfixturevalue(project_fixture)
[   22s]         keyring = os.path.join(gpg_home, "pubring.kbx")
[   22s]         args = [
[   22s]             "project",
[   22s]             "gpg-verify",
[   22s]             f"--keyring={keyring}",
[   22s]             str(project_root),
[   22s]         ]
[   22s]         rc = main(args)
[   22s]         captured = capsys.readouterr()
[   22s] >       assert exp_stdout_substr in captured.out
[   22s] [1m[31mE       AssertionError: assert 'Checksum validation failed.' in '[\x1b[91mERROR\x1b[0m] GPG signature verification failed.\n[\x1b[94mNOTE \x1b[0m] Re-run with the global --debug flag for more information.\n'[0m
[   22s] [1m[31mE        +  where '[\x1b[91mERROR\x1b[0m] GPG signature verification failed.\n[\x1b[94mNOTE \x1b[0m] Re-run with the global --debug flag for more information.\n' = CaptureResult(out='[\x1b[91mERROR\x1b[0m] GPG signature verification failed.\n[\x1b[94mNOTE \x1b[0m] Re-run with the global --debug flag for more information.\n', err='').out[0m
[   22s] 
[   22s] [1m[31mtests/test_cli.py[0m:163: AssertionError
[   22s] ------------------------------ Captured log call -------------------------------
[   22s] [33mWARNING [0m gnupg:gnupg.py:1087 gpg returned a non-zero error code: 2
[   22s] [31m[1m_____ test_gpg_verify_manifest_scenario[broken MANIFEST.in after signing] ______[0m
[   22s] 
[   22s] capsys = <_pytest.capture.CaptureFixture object at 0x7f391b5d4710>
[   22s] request = <FixtureRequest for <Function test_gpg_verify_manifest_scenario[broken MANIFEST.in after signing]>>
[   22s] project_fixture = 'signed_project_broken_manifest_in'
[   22s] exp_stdout_substr = "An error was encountered while parsing MANIFEST.in: unknown action 'invalid-directive'"
[   22s] exp_stderr_substr = '', exp_rc = 1
[   22s] 
[   22s]     @pytest.mark.parametrize(
[   22s]         "project_fixture, exp_stdout_substr, exp_stderr_substr, exp_rc",
[   22s]         [
[   22s]             ("signed_project_and_gpg", "GPG signature verification succeeded", "", 0),
[   22s]             ("signed_project_broken_manifest", "Invalid line encountered in checksum manifest", "", 1),
[   22s]             ("signed_project_missing_manifest", "Checksum manifest file does not exist:", "", 1),
[   22s]             ("signed_project_modified_manifest", "Checksum validation failed.", "", 2),
[   22s]             ("signed_project_with_different_gpg_home", "Re-run with the global --debug flag", "", 3),
[   22s]             ("signed_project_broken_manifest_in", "An error was encountered while parsing MANIFEST.in: unknown action 'invalid-directive'", "", 1),
[   22s]         ],
[   22s]         ids=[
[   22s]             "valid checksum file and signature",
[   22s]             "valid signature but broken checksum file",
[   22s]             "missing checksum file entirely",
[   22s]             "checksum file with wrong hashes",
[   22s]             "matching pubkey does not exist in gpg home",
[   22s]             "broken MANIFEST.in after signing",
[   22s]         ],
[   22s]     )
[   22s]     def test_gpg_verify_manifest_scenario(capsys, request, project_fixture, exp_stdout_substr, exp_stderr_substr, exp_rc):
[   22s]         """
[   22s]         Test `ansible-sign project gpg-verify` given different project directory
[   22s]         scenarios (fixtures).
[   22s]         """
[   22s]         (project_root, gpg_home) = request.getfixturevalue(project_fixture)
[   22s]         keyring = os.path.join(gpg_home, "pubring.kbx")
[   22s]         args = [
[   22s]             "project",
[   22s]             "gpg-verify",
[   22s]             f"--keyring={keyring}",
[   22s]             str(project_root),
[   22s]         ]
[   22s]         rc = main(args)
[   22s]         captured = capsys.readouterr()
[   22s] >       assert exp_stdout_substr in captured.out
[   22s] [1m[31mE       assert "An error was encountered while parsing MANIFEST.in: unknown action 'invalid-directive'" in '[\x1b[91mERROR\x1b[0m] GPG signature verification failed.\n[\x1b[94mNOTE \x1b[0m] Re-run with the global --debug flag for more information.\n'[0m
[   22s] [1m[31mE        +  where '[\x1b[91mERROR\x1b[0m] GPG signature verification failed.\n[\x1b[94mNOTE \x1b[0m] Re-run with the global --debug flag for more information.\n' = CaptureResult(out='[\x1b[91mERROR\x1b[0m] GPG signature verification failed.\n[\x1b[94mNOTE \x1b[0m] Re-run with the global --debug flag for more information.\n', err='').out[0m
[   22s] 
[   22s] [1m[31mtests/test_cli.py[0m:163: AssertionError

Tumbleweed uses:

[   10s] + pytest-3.11 --ignore=_build.python3 -v --basetemp /tmp/ansible-sign-pytest --color=yes -k 'not (test_pinentry_simple)'
[   10s] [1m============================= test session starts ==============================[0m
[   10s] platform linux -- Python 3.11.5, pytest-7.4.3, pluggy-1.3.0 -- /usr/bin/python3.11
[   10s] cachedir: .pytest_cache
[   10s] rootdir: /home/abuild/rpmbuild/BUILD/ansible-sign-0.1.1
[   10s] configfile: setup.cfg
[   10s] testpaths: tests
[   10s] plugins: cov-4.1.0, mock-3.12.0, libtmux-0.23.2
[   11s] [1mcollecting ... [0mcollected 40 items / 1 deselected / 39 selected

Leap 15.5 uses:

[   35s] + pytest-3.11 --ignore=_build.python311 -v --basetemp /tmp/ansible-sign-pytest --color=yes -k 'not (test_pinentry_simple)'
[   36s] [1m============================= test session starts ==============================[0m
[   36s] platform linux -- Python 3.11.2, pytest-7.3.1, pluggy-1.0.0 -- /usr/bin/python3.11
[   36s] cachedir: .pytest_cache
[   36s] rootdir: /home/abuild/rpmbuild/BUILD/ansible-sign-0.1.1
[   36s] configfile: setup.cfg
[   36s] testpaths: tests
[   36s] plugins: mock-3.10.0, cov-4.0.0, libtmux-0.21.0
[   36s] [1mcollecting ... [0mcollected 40 items / 1 deselected / 39 selected

[johanneskastl]

Any ideas on this? I do not want to ignore checks just to have a successful package build...

[relrod]

@johanneskastl The issue is that this is a legitimate failure and it needs someone to sit down and research it and I don't think anyone is currently working on that. The issue happens on newer Fedora, too.

This is due to newer gnupg using keyboxd by default and ansible-sign not currently supporting that. Specifically this breaks using --keyring to point to a specific keyring file (rather than a full gnupg home directory).

I'd gladly accept a PR if someone wanted to research how to make --keyring work properly given the gnupg changes, but I'm not sure when I'll have the time to properly research this out myself.

[johanneskastl]

OK, so at least it is a known issue and not only happening on openSUSE.

I am not experienced enough to dig into this, unfortunately. I am merely packaging it...

[relrod]

OK, so at least it is a known issue and not only happening on openSUSE.

Yeah, it's every distro with a newer gnupg 😢

For packaging, you could just skip the tests that use --keyring and perhaps document it as a known issue. Most of the functionality of ansible-sign should still work, it's just the --keyring flag that is broken.

Perhaps an easy fix for now is to just hide that option if we detect a too-new gnupg... But ideally, we'd figure out an actual solution and support keyboxd properly.