Packages status summary

[paulvi]

This is of course external to this repository, but there should be a place for status (e.g. provided, works as is, fails) for many Python packages.

graalpython -m ginstall install --help currently lists

Known packages are pytest, pytest_parallel, py, attrs, pyparsing, packaging, more_itertools, atomicwrites, pluggy, zipp, wcwidth, PyYAML, six, Cython, setuptools, pkgconfig, wheel, protobuf,
Keras_preprocessing, gast, astor, absl_py, mock, Markdown, Werkzeug, h5py, sortedcontainers, hypothesis, setuptools_scm, numpy, dateutil, certifi, idna, chardet, urllib3, requests, lightfm, pytz, pandas, scipy,
cycler, cppy, cassowary, Pillow, matplotlib

For example, requests is within "Known packages", so it works when installing with ginstall.
But if it was not, it would be helpful to know, that the one installed with 'pip' does not work, and what exact errors are.
(Maybe it needs some fixes from package maintainers)

Summary example

Package	ginstall	pip
ast		#218
Cython	V	V
flask		#218
fcntl		#52
jax		? #208
matplotlib	V
pandas	V
pdb		#63
requests	V	init fails oracle/graal#3588, SO
scipy	V
sqlite3		#174

[msimacek]

Hi @paulvi. Internal modules (those that ship with the implementation, like fcntl, sqlite3...) are tracked in https://github.com/oracle/graalpython/blob/master/docs/contributor/MISSING.md.

For 3rd party packages, we're currently working on a testing infrastructure that would automatically and periodically run tests for 500 most popular packages on PyPI under our implementation and report the percentage of passing tests for each. It's a work in progress, there's no publicly available output yet. It will need quite some time, because there's no standard way to run tests for a package on PyPI and packages often don't pin their dependencies, so almost every package needs some amount of manual tinkering to get the tests work.

[paulvi]

Hi Michael @msimacek
So there are 3 categories:

• built-in packages, developed in this repo
• package "known" to ginstall, that are already fixed/work
• all other 3rd party package
  (Is this saying good?)

I think it is enormous undertaking to address all or top 500 packages, and it will be often on move as package and their dependency change. So not just current test state but also previous history would be needed, together with exact dependency version used. Is it really feasible?

Maybe other way (or at least for all other packages) it to suggest to package mantainer to test their package on several Python distributions, including graalpython. So specific result of this approach would be travis-ci or an other CI/CD configuration that does this, than some promotion/awareness wave.
It may take a lot of work to make all packages work (always) on graalpython,
while telling developers to suggest to packages maintainers to consider graalpython would be just matter of one documentation page.

And closer to issue title, consider enable wiki for this GitHub project,
so package status editable list could live at
https://github.com/oracle/graalpython/wiki/packages

[msimacek]

It's not really feasible to test all of the 500, some of them don't even make sense or don't have tests. It is meant to be a best effort where some percentage of packages would remain in an unknown state when they cannot be tested.

Having package maintainers run the tests themselves in their CI would of course be ideal :) But for that, all the tests would have to pass under graalpython. We cannot ask maintainers to add failing jobs. So we first need to do the testing ourselves and get the tests passing. Only then we can upstream the testing of packages that pass.

I'll ask about the wiki.

[letalvoj]

Any update on the testing infra? I would be interested in which packages and the produced errors.
Could it maybe even attract some package specific contributions from people wanting to use graalpython? 🤔

[msimacek]

Hi @letalvoj. The test infra is running, but as expected, most packages fail to test because they need some manual setup or environment preparation or the test dependencies fail to install. I'm working through the list but it takes time. The results for the latest release can be queried at https://www.graalvm.org/compatibility/ (if you want the whole data, the page is using just a static CSV), but unfortunately the logs are not yet publicly available anywhere.

[thedrow]

I'd be very interested to check what needs to be done for Celery to work with Graal. I hope you'll run the test suite for it soon.

[msimacek]

I tried running the celery unit test suite and unfortunately it failed to import the msgpack C extension module as there was some problem in handling datetime objects in the initialization code. @tstupka is currently reworking the C datetime API, so I'll try again when that work is done as it's likely to fix that particular problem.

@thedrow would you mind sharing your use case?

[thedrow]

I maintain Celery and I'd like to allow support for other interpreters as well.

[rainbowechoes]

Hi, I find that ginstall can't install mediapipe lib, either pip. Did I have some way to fix it ?