generate-ci'd emscripten pytest fails before reaching module code

[chrysn]

Bug Description

In cbor-diag-py, which is an almost trivial¹ project following the maturing scaffolding of maturin new and generate-ci, the emscripten tests fail, seemingly before they even touch code.

Excerpt of the latest CI run

[pytest]
Run set -e
xbuild environment already exists, skipping download                            
Installing xbuild environment                                                   
Creating Pyodide virtualenv at .venv                                            
... Configuring virtualenv                                                      
... Installing standard library                                                 
Successfully created Pyodide virtual environment!                               
Looking in indexes: https://pypi.org/simple, file:///home/runner/work/cbor-diag-py/cbor-diag-py/.pyodide-xbuildenv/xbuildenv/pyodide-root/pypa_index
Looking in links: dist
Collecting cbor-diag
  Downloading https://cdn.jsdelivr.net/pyodide/v0.25.1/full/cbor_diag-1.0.1-cp311-cp311-emscripten_3_1_46_wasm32.whl (362 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 362.9/362.9 kB 14.3 MB/s eta 0:00:00
Installing collected packages: cbor-diag
Successfully installed cbor-diag-1.0.1
WARNING: Location 'file:/home/runner/work/cbor-diag-py/cbor-diag-py/.pyodide-xbuildenv/xbuildenv/pyodide-root/pypa_index/pip/' is ignored: it is neither a file nor a directory.
Looking in indexes: https://pypi.org/simple, file:///home/runner/work/cbor-diag-py/cbor-diag-py/.pyodide-xbuildenv/xbuildenv/pyodide-root/pypa_index
Collecting pytest
  Downloading pytest-8.2.0-py3-none-any.whl.metadata (7.5 kB)
Collecting iniconfig (from pytest)
  Downloading https://cdn.jsdelivr.net/pyodide/v0.25.1/full/iniconfig-2.0.0-py3-none-any.whl (14 kB)
Requirement already satisfied: packaging in ./.venv/lib/python3.11/site-packages (from pytest) (23.1)
Collecting pluggy<2.0,>=1.5 (from pytest)
  Downloading pluggy-1.5.0-py3-none-any.whl.metadata (4.8 kB)
Downloading pytest-8.2.0-py3-none-any.whl (339 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 339.2/339.2 kB 8.8 MB/s eta 0:00:00
Downloading pluggy-1.5.0-py3-none-any.whl (20 kB)
Installing collected packages: pluggy, iniconfig, pytest
Successfully installed iniconfig-2.0.0 pluggy-1.5.0 pytest-8.2.0
============================= test session starts ==============================
platform emscripten -- Python 3.11.3, pytest-8.2.0, pluggy-1.5.0
rootdir: /home/runner/work/cbor-diag-py/cbor-diag-py
configfile: pytest.ini
collected 1 item / 10 errors

==================================== ERRORS ====================================
_ ERROR collecting emsdk-cache/emsdk-main/upstream/emscripten/test/test_benchmark.py _
ImportError while importing test module '/home/runner/work/cbor-diag-py/cbor-diag-py/emsdk-cache/emsdk-main/upstream/emscripten/test/test_benchmark.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/lib/python311.zip/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
emsdk-cache/emsdk-main/upstream/emscripten/test/test_benchmark.py:20: in <module>
    import clang_native
emsdk-cache/emsdk-main/upstream/emscripten/test/clang_native.py:10: in <module>
    from tools.shared import PIPE, run_process, CLANG_CC, CLANG_CXX
E   ModuleNotFoundError: No module named 'tools'
_ ERROR collecting emsdk-cache/emsdk-main/upstream/emscripten/test/test_browser.py _
ImportError while importing test module '/home/runner/work/cbor-diag-py/cbor-diag-py/emsdk-cache/emsdk-main/upstream/emscripten/test/test_browser.py'.
Hint: make sure your test modules/packages have valid Python names.

Given that nothing in the errors appear to be related to the cbor-diag module, it appears that this is an issue with maturin or the actions added in generate-ci. I'm a bit lost at why the emscripten tests are even called, do you have any pointers to track this down further?

Your maturin version (maturin --version)

1.5.1

Your Python version (python -V)

3.11.8

Your pip version (pip -V)

24.0

What bindings you're using

pyo3

Does cargo build work?

• Yes, it works

If on windows, have you checked that you aren't accidentally using unix path (those with the forward slash /)?

• Yes

Steps to Reproduce

The issue is with what happens on GitHub actions, I don't think this can be reproduced locally.

Footnotes

1. The package uses altered paths and a manual init to avoid the public modulename.modulename API guarantee tracked in Defining a module foo results in foo.foo being available #1399 ↩

[messense]

I don't know either, @hoodmane any idea?

[hoodmane]

cc @ryanking13.

I'm not sure why it's trying to run those tests but I think they were meant to run in the emscripten source tree and weren't meant to run on install. I wonder why they are running here.

[ryanking13]

It looks like emscripten is installed under emsdk-cache directory (the directory specified in actions-cache-folder option in the setup-emsdk action). And emscripten contains a bunch of Python test files in their repository, so pytest added them to the test suite.

So I guess you need to

1. Install emscripten somewhere out of current working directory, or
2. Specify the test file paths clearly when running pytest

[chrysn]

Just to not drop the ball, is that something I can act on in my repository (doesn't seem like it, it doesn't touch emsdk installation), something where we'd need to change the generate output, or something where we'd need to pull in maintainers of the action repositories supplying the CI steps?

[ryanking13]

is that something I can act on in my repository

Well, I think the easiest way, and what I would like to recommend, is specifying the files/directory to test by pytest clearly. By adding testpaths in pytest.ini, so that even if some files are unintentionally created in the local directory during CI, pytest will only put the specified directory in the test scope.