[Task]: Support newer versions of Pyarrow in Beam

[tvalentyn]

What needs to happen?

We should upgrade beam to support newer versions of PyArrow. Currently Pyarrow 16 is the latest: https://pypi.org/project/pyarrow/#history.

When we do this upgrade, we should check whehter we can also unpin the upper bound for Windows tests:

beam/sdks/python/setup.py

Line 147 in 365c2d9

# https://github.com/apache/beam/issues/28410 - pyarrow>=13 seeing issues

If that is still the issue, we will have issues with Python 3.12 on windows that use arrow.

Issue Priority

Priority: 2 (default / most normal work should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[tvalentyn]

this is fixed.

[tvalentyn]

currently supporting up to: 'pyarrow>=3.0.0,<17.0.0',