Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upgrade to Python 3.12 #2072

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
+6 −5
Open

Upgrade to Python 3.12 #2072

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
8f3dce1
Upgrade to 3.12
max-muoto Mar 10, 2024
e4b6dad
Upgrade Pandas
max-muoto Mar 10, 2024
51a4943
Merge branch 'main' into python-3.12
mathbunnyru Mar 27, 2024
4a9db95
Merge branch 'main' into python-3.12
mathbunnyru May 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,8 @@ If you want to use the older `Ubuntu` and/or `Python` version, you can use the f
| 2022-10-09 | 22.04 | 3.8 | `7285848c0a11` |
| 2022-10-09 | 22.04 | 3.9 | `ed2908bbb62e` |
| 2023-05-30 | 22.04 | 3.10 | `4d70cf8da953` |
| weekly build | 22.04 | 3.11 | `latest` |
| 2024-03-06 | 22.04 | 3.11 | `07dc2d08734b` |
| weekly build | 22.04 | 3.12 | `latest` |

## Contributing

Expand Down
2 changes: 1 addition & 1 deletion images/docker-stacks-foundation/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ RUN echo "auth requisite pam_deny.so" >> /etc/pam.d/su && \
USER ${NB_UID}

# Pin the Python version here, or set it to "default"
ARG PYTHON_VERSION=3.11
ARG PYTHON_VERSION=3.12

# Setup work directory for backward-compatibility
RUN mkdir "/home/${NB_USER}/work" && \
Expand Down
2 changes: 1 addition & 1 deletion images/pyspark-notebook/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ USER ${NB_UID}
RUN mamba install --yes \
'grpcio-status' \
'grpcio' \
'pandas=2.0.3' \
'pandas=2.2.1' \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you checked the comment above to make sure you’re using the proper version?

Copy link
Author

@max-muoto max-muoto Mar 10, 2024 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They have pandas<=2.2.1 so we should be good here.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/apache/spark/blob/master/dev/infra/Dockerfile

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to check the latest stable tag, not the current main branch.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah sorry there. Looks like we'll need for their next release, as this commit isn't included in the latest stable tag: ericm-db/spark@98ca3ea

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we'll need to wait on this as well, since we need at least Pandas 2.1.1 to ensure compatibility with 3.12.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bjornjorgensen could you please tell us when the Spark release will include this commit? (at least approximately)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm.. well we are waiting for hadoop 3.4.0 and a new hive release. We haven't started any RC release yet. I build and test my own jupyterlab https://github.com/bjornjorgensen/jupyter-spark-master-docker and I did try python 3.12 but it breake so match so I'm using python 3.11 as debian testing are using.
And Spark 3.5.1 don't support python 3.12 have a look at apache/spark#43922

Copy link

@shreve shreve Jun 11, 2024 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we just revert this particular change rather than waiting for the next release of Spark? This seems like an incredibly self-imposed blocker.

'pyarrow' && \
mamba clean --all -f -y && \
fix-permissions "${CONDA_DIR}" && \
Expand Down
2 changes: 1 addition & 1 deletion tests/docker-stacks-foundation/test_python_version.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
from tests.conftest import TrackedContainer

LOGGER = logging.getLogger(__name__)
EXPECTED_PYTHON_VERSION = "3.11"
EXPECTED_PYTHON_VERSION = "3.12"


def test_python_version(container: TrackedContainer) -> None:
Expand Down
2 changes: 1 addition & 1 deletion tests/pyspark-notebook/units/unit_pandas_version.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@
# Distributed under the terms of the Modified BSD License.
import pandas

assert pandas.__version__ == "2.0.3"
assert pandas.__version__ == "2.2.1"