Installing packags through requirements.txt are not on the path

[mjkl-gh]

I've been trying out the new feature that enable installing python dependencies inside the container. However, something is failing in a weird way.

Some packages seem to be installed on the path others not, specifically httpx, which can be seen in the log:

Installing collected packages: proxmoxer, urllib3, sniffio, h11, certifi, requests, httpcore, anyio, httpx
  WARNING: The script httpx is installed in '/home/semaphore/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed anyio-4.3.0 certifi-2024.2.2 h11-0.14.0 httpcore-1.0.5 httpx-0.27.0 proxmoxer-2.0.1 requests-2.31.0 sniffio-1.3.1 urllib3-2.2.1

This makes any script that depends on this fail unless I add:

      - name: Verify httpx is installed
        delegate_to: localhost
        ansible.builtin.pip:
          name: httpx

Although it is a good practice to ensure installation, regardless where the script is running. This behavior is not what I expected. Especially since other packages do seem to work fine.

[tboerger]

My pending pr for the docker refactoring resolves that problem.

[tboerger]

After #1963 have been merged, could you give that develop container tag another try? I guess this issue should be solved.

[mjkl-gh]

I think watchtower automatically pulled it.

I'm now running into this:

No additional system dependencies to install
Installing additional python dependencies
error: externally-managed-environment
× This environment is externally managed
╰─> 
    The system-wide python installation should be maintained using the system
    package manager (apk) only.
    
    If the package in question is not packaged already (and hence installable via
    "apk add py3-somepackage"), please consider installing it inside a virtual
    environment, e.g.:
    
    python3 -m venv /path/to/venv
    . /path/to/venv/bin/activate
    pip install mypackage
    
    To exit the virtual environment, run:
    
    deactivate
    
    The virtual environment is not deleted, and can be re-entered by re-sourcing
    the activate file.
    
    To automatically manage virtual environments, consider using pipx (from the
    pipx package).
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

And semaphore no longer starts

[mjkl-gh]

I assume this happens because the pip install --user is called and that has been disabled by the maintainers of alpine, because system packages rely on certain python packages. And running pip upgrade on those might break them

[tboerger]

Looks like I got to add more flags similar to the dockerfile.

[mjkl-gh]

I assume you mean --break-system-packages?

I would advise against as it simply might break the container in unexpected ways, as the flag suggests

I'm currently busy with other things, but maybe I can find some time over the weekend to see if I can create a PR that follows a more distro approved way. Maybe just adding /home/semaphore/.local/bin to the PATH, which was the issue that started this, might be a better solution.

[tboerger]

That's just required because of the restrictions set by alpine devs. The container won't be managed by the alone package manager. Beside that the python deps are only installed in the user space.

[tboerger]

Even if I don't like it, the docker images are using a venev again.

[mjkl-gh]

Thx for fixing the issue and reconsidering. I did not manage to find the time to write a MR.

[ramiuslr]

Hi @tboerger, thanks for these informations, I have the same problem as described here even with the develop tag.
Here are the relevant logs:

semaphore-1  | Attempting to connect to database semaphore on semaphore-mariadb:3306 with user semaphore ...
semaphore-1  | No additional system dependencies to install
semaphore-1  | Installing additional python dependencies
semaphore-1  | ..Collecting git+https://github.com/vmware/vsphere-automation-sdk-python.git (from -r /etc/semaphore/requirements.txt (line 3))
semaphore-1  |   Cloning https://github.com/vmware/vsphere-automation-sdk-python.git to /tmp/pip-req-build-jip9xn58
semaphore-1  |   Running command git clone --filter=blob:none --quiet https://github.com/vmware/vsphere-automation-sdk-python.git /tmp/pip-req-build-jip9xn58
semaphore-1  |   Resolved https://github.com/vmware/vsphere-automation-sdk-python.git to commit 99ee630cb819495f572e9775b48ce0ac1b0263de
semaphore-1  |   Installing build dependencies: started
semaphore-1  |   Installing build dependencies: finished with status 'done'
semaphore-1  |   Getting requirements to build wheel: started
semaphore-1  |   Getting requirements to build wheel: finished with status 'done'
semaphore-1  |   Installing backend dependencies: started
semaphore-1  |   Installing backend dependencies: finished with status 'done'
semaphore-1  |   Preparing metadata (pyproject.toml): started
semaphore-1  |   Preparing metadata (pyproject.toml): finished with status 'done'
semaphore-1  | Collecting pyvmomi (from -r /etc/semaphore/requirements.txt (line 1))
semaphore-1  |   Downloading pyvmomi-8.0.2.0.1.tar.gz (852 kB)
semaphore-1  |      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 852.2/852.2 kB 6.6 MB/s eta 0:00:00
semaphore-1  |   Installing build dependencies: started
semaphore-1  |   Installing build dependencies: finished with status 'done'
semaphore-1  |   Getting requirements to build wheel: started
semaphore-1  |   Getting requirements to build wheel: finished with status 'done'
semaphore-1  |   Installing backend dependencies: started
semaphore-1  |   Installing backend dependencies: finished with status 'done'
semaphore-1  |   Preparing metadata (pyproject.toml): started
semaphore-1  |   Preparing metadata (pyproject.toml): finished with status 'done'
semaphore-1  | Collecting pywinrm[credssp] (from -r /etc/semaphore/requirements.txt (line 2))
semaphore-1  |   Downloading pywinrm-0.4.3-py2.py3-none-any.whl.metadata (11 kB)
semaphore-1  | Requirement already satisfied: six>=1.7.3 in /opt/semaphore/venv/lib/python3.11/site-packages (from pyvmomi->-r /etc/semaphore/requirements.txt (line 1)) (1.16.0)
semaphore-1  | Collecting xmltodict (from pywinrm[credssp]->-r /etc/semaphore/requirements.txt (line 2))
semaphore-1  |   Downloading xmltodict-0.13.0-py2.py3-none-any.whl.metadata (7.7 kB)
semaphore-1  | Requirement already satisfied: requests>=2.9.1 in /opt/semaphore/venv/lib/python3.11/site-packages (from pywinrm[credssp]->-r /etc/semaphore/requirements.txt (line 2)) (2.32.3)
semaphore-1  | Collecting requests-ntlm>=1.1.0 (from pywinrm[credssp]->-r /etc/semaphore/requirements.txt (line 2))
semaphore-1  |   Downloading requests_ntlm-1.2.0-py3-none-any.whl.metadata (2.4 kB)
semaphore-1  | Collecting requests-credssp>=1.0.0 (from pywinrm[credssp]->-r /etc/semaphore/requirements.txt (line 2))
semaphore-1  |   Downloading requests_credssp-2.0.0-py2.py3-none-any.whl.metadata (10 kB)
semaphore-1  | Processing //tmp/pip-req-build-jip9xn58/lib/vapi-runtime/vapi_runtime-2.44.0-py2.py3-none-any.whl (from vsphere-automation-sdk==1.86.0->-r /etc/semaphore/requirements.txt (line 3))
semaphore-1  | Processing //tmp/pip-req-build-jip9xn58/lib/vcenter-bindings/vcenter_bindings-4.2.0-py2.py3-none-any.whl (from vsphere-automation-sdk==1.86.0->-r /etc/semaphore/requirements.txt (line 3))
semaphore-1  | Processing //tmp/pip-req-build-jip9xn58/lib/vapi-common-client/vapi_common_client-2.44.0-py2.py3-none-any.whl (from vsphere-automation-sdk==1.86.0->-r /etc/semaphore/requirements.txt (line 3))
semaphore-1  | Processing //tmp/pip-req-build-jip9xn58/lib/vmwarecloud-aws/vmwarecloud_aws-1.64.0-py2.py3-none-any.whl (from vsphere-automation-sdk==1.86.0->-r /etc/semaphore/requirements.txt (line 3))
semaphore-1  | Processing //tmp/pip-req-build-jip9xn58/lib/nsx-python-sdk/nsx_python_sdk-4.1.2.0.0-py2.py3-none-any.whl (from vsphere-automation-sdk==1.86.0->-r /etc/semaphore/requirements.txt (line 3))
semaphore-1  | Processing //tmp/pip-req-build-jip9xn58/lib/nsx-policy-python-sdk/nsx_policy_python_sdk-4.1.2.0.0-py2.py3-none-any.whl (from vsphere-automation-sdk==1.86.0->-r /etc/semaphore/requirements.txt (line 3))
semaphore-1  | Processing //tmp/pip-req-build-jip9xn58/lib/nsx-vmc-policy-python-sdk/nsx_vmc_policy_python_sdk-4.1.2.0.0-py2.py3-none-any.whl (from vsphere-automation-sdk==1.86.0->-r /etc/semaphore/requirements.txt (line 3))
semaphore-1  | Processing //tmp/pip-req-build-jip9xn58/lib/nsx-vmc-aws-integration-python-sdk/nsx_vmc_aws_integration_python_sdk-4.1.2.0.0-py2.py3-none-any.whl (from vsphere-automation-sdk==1.86.0->-r /etc/semaphore/requirements.txt (line 3))
semaphore-1  | Processing //tmp/pip-req-build-jip9xn58/lib/vmwarecloud-draas/vmwarecloud_draas-1.23.0-py2.py3-none-any.whl (from vsphere-automation-sdk==1.86.0->-r /etc/semaphore/requirements.txt (line 3))
semaphore-1  | Collecting lxml>=4.3.0 (from vsphere-automation-sdk==1.86.0->-r /etc/semaphore/requirements.txt (line 3))
semaphore-1  |   Downloading lxml-5.2.2-cp311-cp311-musllinux_1_2_x86_64.whl.metadata (3.4 kB)
semaphore-1  | Requirement already satisfied: charset-normalizer<4,>=2 in /opt/semaphore/venv/lib/python3.11/site-packages (from requests>=2.9.1->pywinrm[credssp]->-r /etc/semaphore/requirements.txt (line 2)) (3.3.2)
semaphore-1  | Requirement already satisfied: idna<4,>=2.5 in /opt/semaphore/venv/lib/python3.11/site-packages (from requests>=2.9.1->pywinrm[credssp]->-r /etc/semaphore/requirements.txt (line 2)) (3.7)
semaphore-1  | Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/semaphore/venv/lib/python3.11/site-packages (from requests>=2.9.1->pywinrm[credssp]->-r /etc/semaphore/requirements.txt (line 2)) (2.2.1)
semaphore-1  | Requirement already satisfied: certifi>=2017.4.17 in /opt/semaphore/venv/lib/python3.11/site-packages (from requests>=2.9.1->pywinrm[credssp]->-r /etc/semaphore/requirements.txt (line 2)) (2024.2.2)
semaphore-1  | Requirement already satisfied: cryptography in /opt/semaphore/venv/lib/python3.11/site-packages (from requests-credssp>=1.0.0->pywinrm[credssp]->-r /etc/semaphore/requirements.txt (line 2)) (42.0.7)
semaphore-1  | Collecting pyspnego>=0.5.0 (from requests-credssp>=1.0.0->pywinrm[credssp]->-r /etc/semaphore/requirements.txt (line 2))
semaphore-1  |   Downloading pyspnego-0.10.2-py3-none-any.whl.metadata (5.4 kB)
semaphore-1  | Collecting pyOpenSSL (from vapi-runtime@ file://localhost//tmp/pip-req-build-jip9xn58/lib/vapi-runtime/vapi_runtime-2.44.0-py2.py3-none-any.whl->vsphere-automation-sdk==1.86.0->-r /etc/semaphore/requirements.txt (line 3))
semaphore-1  |   Downloading pyOpenSSL-24.1.0-py3-none-any.whl.metadata (12 kB)
semaphore-1  | Requirement already satisfied: setuptools in /opt/semaphore/venv/lib/python3.11/site-packages (from vcenter-bindings@ file://localhost//tmp/pip-req-build-jip9xn58/lib/vcenter-bindings/vcenter_bindings-4.2.0-py2.py3-none-any.whl->vsphere-automation-sdk==1.86.0->-r /etc/semaphore/requirements.txt (line 3)) (65.5.0)
semaphore-1  | Requirement already satisfied: cffi>=1.12 in /opt/semaphore/venv/lib/python3.11/site-packages (from cryptography->requests-credssp>=1.0.0->pywinrm[credssp]->-r /etc/semaphore/requirements.txt (line 2)) (1.16.0)
semaphore-1  | Requirement already satisfied: pycparser in /opt/semaphore/venv/lib/python3.11/site-packages (from cffi>=1.12->cryptography->requests-credssp>=1.0.0->pywinrm[credssp]->-r /etc/semaphore/requirements.txt (line 2)) (2.22)
semaphore-1  | Downloading lxml-5.2.2-cp311-cp311-musllinux_1_2_x86_64.whl (5.0 MB)
semaphore-1  |    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.0/5.0 MB 29.9 MB/s eta 0:00:00
semaphore-1  | Downloading requests_credssp-2.0.0-py2.py3-none-any.whl (10 kB)
semaphore-1  | Downloading requests_ntlm-1.2.0-py3-none-any.whl (6.0 kB)
semaphore-1  | Downloading pywinrm-0.4.3-py2.py3-none-any.whl (44 kB)
semaphore-1  |    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.1/44.1 kB 6.9 MB/s eta 0:00:00
semaphore-1  | Downloading xmltodict-0.13.0-py2.py3-none-any.whl (10.0 kB)
semaphore-1  | Downloading pyspnego-0.10.2-py3-none-any.whl (129 kB)
semaphore-1  |    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 129.9/129.9 kB 22.4 MB/s eta 0:00:00
semaphore-1  | Downloading pyOpenSSL-24.1.0-py3-none-any.whl (56 kB)
semaphore-1  |    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56.9/56.9 kB 9.9 MB/s eta 0:00:00
semaphore-1  | Building wheels for collected packages: pyvmomi, vsphere-automation-sdk
semaphore-1  |   Building wheel for pyvmomi (pyproject.toml): started
semaphore-1  |   Building wheel for pyvmomi (pyproject.toml): finished with status 'done'
semaphore-1  |   Created wheel for pyvmomi: filename=pyvmomi-8.0.2.0.1-py2.py3-none-any.whl size=527673 sha256=efc66b85ad5892d4cb6e2b1ba1839255c39d894b35b536d636c72fbe2674aa94
semaphore-1  |   Stored in directory: /home/semaphore/.cache/pip/wheels/58/00/35/035f172707b539c0edd6e086053a4f177a424c0ab807799d1f
semaphore-1  |   Building wheel for vsphere-automation-sdk (pyproject.toml): started
semaphore-1  |   Building wheel for vsphere-automation-sdk (pyproject.toml): finished with status 'done'
semaphore-1  |   Created wheel for vsphere-automation-sdk: filename=vsphere_automation_sdk-1.86.0-py3-none-any.whl size=2325 sha256=0e02684643bbf47532a5163dfbe1d33300bd4e9fadd32ba2c954de905d1bac85
semaphore-1  |   Stored in directory: /tmp/pip-ephem-wheel-cache-7kksthdp/wheels/32/3f/75/62d9c317cd99d4f8b7abfe4e2ac50658e57b7cc1984bb0bc28
semaphore-1  | Successfully built pyvmomi vsphere-automation-sdk
semaphore-1  | Installing collected packages: xmltodict, pyvmomi, lxml, pyspnego, pyOpenSSL, vapi-runtime, requests-ntlm, requests-credssp, vcenter-bindings, vapi-common-client, pywinrm, vmwarecloud-draas, vmwarecloud-aws, nsx-vmc-policy-python-sdk, nsx-vmc-aws-integration-python-sdk, nsx-python-sdk, nsx-policy-python-sdk, vsphere-automation-sdk
semaphore-1  |   WARNING: The script pyspnego-parse is installed in '/home/semaphore/.local/bin' which is not on PATH.
semaphore-1  |   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
semaphore-1  |   WARNING: The script vapi-server is installed in '/home/semaphore/.local/bin' which is not on PATH.
semaphore-1  |   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
semaphore-1  | Successfully installed lxml-5.2.2 nsx-policy-python-sdk-4.1.2.0.0 nsx-python-sdk-4.1.2.0.0 nsx-vmc-aws-integration-python-sdk-4.1.2.0.0 nsx-vmc-policy-python-sdk-4.1.2.0.0 pyOpenSSL-24.1.0 pyspnego-0.10.2 pyvmomi-8.0.2.0.1 pywinrm-0.4.3 requests-credssp-2.0.0 requests-ntlm-1.2.0 vapi-common-client-2.44.0 vapi-runtime-2.44.0 vcenter-bindings-4.2.0 vmwarecloud-aws-1.64.0 vmwarecloud-draas-1.23.0 vsphere-automation-sdk-1.86.0 xmltodict-0.13.0
semaphore-1  | Starting semaphore server
semaphore-1  | Loading config
semaphore-1  | Validating config
semaphore-1  | MySQL semaphore@semaphore-mariadb:3306 semaphore
semaphore-1  | Tmp Path (projects home) /tmp/semaphore
semaphore-1  | Semaphore develop-80e2c8e-1717270566
semaphore-1  | Interface
semaphore-1  | Port 3000
semaphore-1  | Server is running

The requirements.txt I'm using:

pyvmomi
pywinrm[credssp]
git+https://github.com/vmware/vsphere-automation-sdk-python.git

Here is the error I get running tasks (logs from web UI):

[WARNING]:  * Failed to parse
/tmp/semaphore/inventory_10/inventory/production/vcenter7.vmware.yml with auto
plugin: Please install "PyVmomi" Python module as this is required for VMware
Guest dynamic inventory plugin.

Just in case someone needs it, here is my compose.yml:

---
services:

  # Semaphore
  semaphore-mariadb:
    restart: unless-stopped
    image: mariadb:latest
    hostname: mariadb
    volumes:
      - /srv/semaphore/data:/var/lib/mysql
    expose:
      - 3306
    environment:
      MYSQL_RANDOM_ROOT_PASSWORD: 'yes'
      MYSQL_DATABASE: 'semaphore'
      MYSQL_USER: 'semaphore'
      MYSQL_PASSWORD: '<redacted>'

  semaphore:
    image: semaphoreui/semaphore:develop
    restart: unless-stopped
    environment:
      SEMAPHORE_DB_USER: 'semaphore'
      SEMAPHORE_DB_PASS: '<redacted>'
      SEMAPHORE_DB_HOST: 'semaphore-mariadb'
      SEMAPHORE_DB_PORT: 3306
      SEMAPHORE_DB_DIALECT: 'mysql'
      SEMAPHORE_DB: 'semaphore'
    depends_on:
      - semaphore-mariadb
    volumes:
      - ./config.json:/etc/semaphore/config.json:ro
      - ./requirements.txt:/etc/semaphore/requirements.txt:ro
    expose:
      - 3000

  # Caddy reverse proxy
  caddy:
    image: 'caddy:latest'
    restart: unless-stopped
    ports:
      - 0.0.0.0:80:80
      - 0.0.0.0:443:443
      - 0.0.0.0:443:443/udp

    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - /srv/caddy/data:/data
      - /srv/caddy/config:/config
      - ./certs:/certs:ro
...

[ramiuslr]

@fiftin @tboerger should I create a new issue for this case ? I'm not that familiar with GitHub issues process, so just let me know what is best for you !