Support developer virtualenv in Debian 12

[zenmonkeykstop]

Status

Ready for review

Description of Changes

Fixes #7153

make venv fails on Python 3.11 due to incompatibility between our pinned PyYAML version and cython 3.0. As a temporary measure, this PR (approach suggested by the original reporter on Gitter) constrains the cython version to allow the venv to be created.

(In future we should update PyYAML and development requirements in general.)

Testing

With Python 3.9 on debian 11:

• make venv and make dev complete successfully

With Python 3.11 on debian 12:

• make venv and make dev complete successfully

• CI passes

• Visual review

Deployment

n/a, development only.

[cfm]

make venv passes for me on Python 3.9 (Debian 11) and Python 3.11 (Debian 12). It fails for me on Python 3.12 (macOS, from MacPorts) with:

      _ruamel_yaml.c:28302:55: error: no member named 'ob_digit' in 'struct _longobject'
                  const digit* digits = ((PyLongObject*)x)->ob_digit;
                                        ~~~~~~~~~~~~~~~~~~  ^
      _ruamel_yaml.c:28357:55: error: no member named 'ob_digit' in 'struct _longobject'
                  const digit* digits = ((PyLongObject*)x)->ob_digit;
                                        ~~~~~~~~~~~~~~~~~~  ^
      _ruamel_yaml.c:28498:55: error: no member named 'ob_digit' in 'struct _longobject'
                  const digit* digits = ((PyLongObject*)x)->ob_digit;
                                        ~~~~~~~~~~~~~~~~~~  ^
      _ruamel_yaml.c:28553:55: error: no member named 'ob_digit' in 'struct _longobject'
                  const digit* digits = ((PyLongObject*)x)->ob_digit;
                                        ~~~~~~~~~~~~~~~~~~  ^
      _ruamel_yaml.c:28732:55: error: no member named 'ob_digit' in 'struct _longobject'
                  const digit* digits = ((PyLongObject*)x)->ob_digit;
                                        ~~~~~~~~~~~~~~~~~~  ^
      _ruamel_yaml.c:28787:55: error: no member named 'ob_digit' in 'struct _longobject'
                  const digit* digits = ((PyLongObject*)x)->ob_digit;
                                        ~~~~~~~~~~~~~~~~~~  ^
      _ruamel_yaml.c:29276:47: error: no member named 'ob_digit' in 'struct _longobject'
          const digit* digits = ((PyLongObject*)b)->ob_digit;
                                ~~~~~~~~~~~~~~~~~~  ^
      133 warnings and 12 errors generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for ruamel.yaml.clib
ERROR: Could not build wheels for ruamel.yaml.clib, which is required to install pyproject.toml-based projects
make: *** [venv] Error 1

Let me know if you'd like me to take this over and investigate further, @zenmonkeykstop.

[cfm]

I've tested make venv using this Dockerfile like so, with the same results as above:

user@sd:~/securedrop$ cat Dockerfile
ARG VERSION
FROM python:$VERSION

RUN apt-get update
RUN apt-get install -y git jq make
RUN apt-get install -y build-essential libssl-dev libffi-dev python3-dev dpkg-dev
RUN apt-get install -y python3-pip
RUN pip install virtualenv

RUN git clone -b 7153-cython-constraint https://github.com/freedomofpress/securedrop
WORKDIR securedrop
RUN make venv
user@sd:~/securedrop$ docker build -t 7153 . --build-arg VERSION=3.9-bullseye
[passes]
user@sd:~/securedrop$ docker build -t 7153 . --build-arg VERSION=3.11-bookworm
[passes]
user@sd:~/securedrop$ docker build -t 7153 . --build-arg VERSION=3.12-bookworm
[...]
91.69       _ruamel_yaml.c: In function ‘__Pyx_PyIndex_AsSsize_t’:
91.69       _ruamel_yaml.c:29276:45: error: ‘PyLongObject’ {aka ‘struct _longobject’} has no member named ‘ob_digit’
91.69       29276 |     const digit* digits = ((PyLongObject*)b)->ob_digit;
91.69             |                                             ^~
91.69       error: command '/usr/bin/gcc' failed with exit code 1
91.69       [end of output]
91.69   
91.69   note: This error originates from a subprocess, and is likely not a problem with pip.
91.69   ERROR: Failed building wheel for ruamel.yaml.clib
93.25 ERROR: Could not build wheels for ruamel.yaml.clib, which is required to install pyproject.toml-based projects
94.31 make: *** [Makefile:21: venv] Error 1
------
Dockerfile:12
--------------------
  10 |     RUN git clone -b 7153-cython-constraint https://github.com/freedomofpress/securedrop
  11 |     WORKDIR securedrop
  12 | >>> RUN make venv
  13 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c make venv" did not complete successfully: exit code: 2

Since there are newer versions of ruamel.yaml, I began upgrading development dependencies...

--- a/securedrop/requirements/python3/develop-requirements.in
+++ b/securedrop/requirements/python3/develop-requirements.in
@@ -34,10 +34,10 @@ pytest-xdist>=3.0.2
 python-vagrant
 pyyaml>=5.4.1
 requests>=2.31.0
-ruamel.yaml>=0.16.10
+ruamel.yaml>=0.18.6
 ruff
-safety>2.2.0
-semgrep>=1.57.0
+safety>2.3.1
+semgrep>=1.74.0
 setuptools>=56.0.0
 shellcheck-py
 six==1.15.0

...but have run into conflicting requirements:

There are incompatible versions in the resolved dependencies:
  ruamel.yaml>=0.18.6 (from -r requirements/python3/develop-requirements.in (line 37))
  ruamel.yaml<1,>=0.15.37 (from ansible-lint==4.2.0->-r requirements/python3/develop-requirements.in (line 2))
  ruamel.yaml<0.18,>=0.16.0 (from semgrep==1.74.0->-r requirements/python3/develop-requirements.in (line 40))
  ruamel.yaml>=0.17.21 (from safety==3.2.0->-r requirements/python3/develop-requirements.in (line 39))

@zenmonkeykstop, it seems these constraints will need to be accompanied by a careful set of upgrades. I hope this Dockerfile will make it easy to test those changes in a minimal, reproducible Python 3.12 environment.

[zenmonkeykstop]

Thanks for the effort - this is a release build blocker (for builds on modern systems), but I'll just work around it for now.

Reducing PR scope

[legoktm]

LGTM!