Provide binary python package for macOS M1

[memeplex]

Summary

pip install lightgm requires building from sources in my MBP (ARM Monterey 12.4), that wasn't the case in my previous MBP (Intel Big Sur). I assume the problem is with the ARM vs Intel part, not the Monterey vs Big Sur one.

Motivation

Building from sources requires previous installation of wheel, additional non-python dependencies (cmake) and is slower.

Description

pip install lightgbm should install binaries OOB.

[jameslamb]

If you use conda, you can get an arm64 precompiled binary from conda-forge. See #4369 (comment).

conda install -c conda-forge lightgbm

Otherwise, we already have some feature requests for providing binaries for the M1 Macs:

• Apple M1 chip compilation support #4369
• Mac M1 GPU support #4694
• Error when trying to use source-built python package for M1 mac #4843
• published maven jars don't support Apple Silicon (ARM) Macs (arm64e) because of missing dynamic library #5035

Unfortunately, we're blocked on this until we're able to build and test arm64 wheels on M1 Macs in a free CI service (#4843 (comment)). Without that ability to test, we can't take on this support.

[dandreevv]

@jameslamb Hello!
scikit-learn has already use MacOS ARM jobs to build wheels (https://github.com/scikit-learn/scikit-learn/blob/main/.github/workflows/wheels.yml#L110).
Isn't that what you're waiting for?

[jameslamb]

They are using cibuildwheel to cross-compile ARM wheels for Mac on Macs that have Intel chips.

https://github.com/scikit-learn/scikit-learn/blob/ec6efc57fac36820919184ffb5fae70a7e871b62/.github/workflows/wheels.yml#L111

https://github.com/scikit-learn/scikit-learn/blob/ec6efc57fac36820919184ffb5fae70a7e871b62/.github/workflows/wheels.yml#L141

Which we have been hesitant to rely on in this project because we don't have a way to actually test that wheels produced that way work on a real M1.

• published maven jars don't support Apple Silicon (ARM) Macs (arm64e) because of missing dynamic library #5035 (comment)
• Error when trying to use source-built python package for M1 mac #4843 (comment)

It's also possible that LightGBM might not be able to use scikit-build or cibuildwheel in its current configuration, because of the patterns that lightgbm's wheels use to build lib_lightgbm.so. See #5106 for significant discussion around this.

If you'd like to help with this issue, we're open to proposals and can answer questions as you try to develop. But providing M1 binaries in this project unfortunately won't be as easy as just copying that GitHub Actions config from scikit-learn, and will require a significant amount of time.

Otherwise, you can subscribe to this issue for updates.

[jameslamb]

@shiyu1994 Github Actions now supports a GitHub-hosted Apple Silicon runner: actions/runner-images#8439

It isn't available for free though. Could you look into getting Microsoft to sponsor such a runner for this project, so we can build Python wheels targeting the M1/M2 Macs?

[santiviquez]

Are there any updates for LightGBM to fully work on M1/M2 Macs?

From an outsider perspective it's a bit weird that now it looks that the blocking is github actions not been free but both GitHub and LightGBM are Microsoft companies/projects 🤔

[jameslamb]

I have tried several approaches over the last 2 years to get support from Microsoft or GitHub for native M1/M2 macOS builds in this project... so far none of those attenpts have resulted in any financial or development support towards resolving this.

I personally have an M2 laptop now (thanks to @guolinke !), so we may give up on native builds and attempt cross-compiling wheels in the near future...at least I'd be able to test them on that laptop.

Until then, you will have to continue building from source, and we're here to help with any issues you encounter doing that.

[santiviquez]

So sad there hasn't been any financial support from them yet :/

I really appreciate your work and all the time you have put into this!

[stephenpardy]

@jameslamb thank you for the effort here. Is the issue purely financial? And if you don't mind me asking do you know roughly how much it might cost to add the M1 runners?

I am interested to see if I can help find a solution to this.

[radeva]

@jameslamb You can try FlyCI's M1 and M2 runners. They are on average 2x faster and 2x cheaper than GitHub's AND we have a free tier for OSS projects (see below).

Install Instructions

1. Install FlyCI app and
2. Easily replace one line of code and start using FlyCI runners:

jobs:
 ci:
-    runs-on: macos-latest
+    runs-on: flyci-macos-large-latest-m1
   steps:
   - name: 👀 Checkout repo
     uses: actions/checkout@v4

500 mins/month Free for Public Repos

Since your repo is public, FlyCI offers 500 mins/month of free M1 runner usage with the flyci-macos-large-latest-m1 runner for public projects.

Don't hesitate to contact us in case the free tier doesn't suit your needs or you experience any issues with the runners. Our team is here to support you!

Best Regards,
Veselina Radeva
Product Manager at FlyCI

[jameslamb]

Is the issue purely financial?

No.

Even with budget and CI resources, this project is still facing a significant lack of people-hours doing work to move the project forward.

• this project is being maintained almost entirely by volunteers
  • no one from Microsoft is working on day-to-day maintenance tasks like testing, CI, packaging, backlog triage, etc.
• those limited volunteer hours have to be spread over a big surface area
  • for example, I personally have worked on 2 or 3 "CRAN is giving us 2 weeks to fix something in the R package" incidents over the last year)
• there are other macOS-compatibility issues that require development and testing effort, e.g.
  • LightGBM is incompatible with libomp 12 and 13 on macOS #4229
  • published maven jars don't support Apple Silicon (ARM) Macs (arm64e) because of missing dynamic library #5035
  • unable to install lightgbm with pip in macos 12.3.1 (chip M1) #5269
  • Provide binary python package for macOS M1 #5328
  • Some errors when installing python interface in mac with m2 #5700
  • [python-package] Can't import LightGBM into Jupyter notebook on M1 MacOS #6262

---

do you know roughly how much it might cost to add the M1 runners?

I don't know for sure. Here's what I've seen looking into this:

• GitHub Actions / Azure DevOps have GitHub-hosted M1 Mac runners but they're not free
  • https://github.com/orgs/community/discussions/69211
  • https://docs.github.com/en/actions/using-github-hosted-runners/about-larger-runners/about-larger-runners
• GitLab CI has macOS M1 runners in beta but only available in their paid SaaS product
  • https://gitlab.com/groups/gitlab-org/-/epics/8267
  • https://docs.gitlab.com/ee/ci/runners/saas/macos_saas_runner.html
  • they have a program for open source projects to apply to get free minutes, but I don't think LightGBM would qualify because Microsoft is a for-profit company and the terms say "your organization ... can't seek to make a profit by selling services" (docs).
  • I have been talking to @darbyfrey from GitLab about options there but for now that isn't available to use.
• TravisCI doesn't have them
  • https://travis-ci.community/t/m1-chip-support/12195
• CircleCI only supports them for paid customers
  • https://discuss.circleci.com/t/macos-intel-support-deprecation-in-january-2024/48718
  • https://circleci.com/docs/using-macos/
• CirrusCI supports them with usage-based pricing
  • based on https://cirrus-ci.org/pricing/ and the current activity level in LightGBM, I suspect we could provide high-quality macOS arm64 wheels with maybe $100/month of spending there

---

I am interested to see if I can help find a solution to this.

Thanks very much!

I think the best path available to us is to pay for CirrusCI and run a single job there that builds wheels + runs the tests natively on an arm64 macOS machine. I hope we could convince Microsoft to pay for that (@shiyu1994 could you comment on this?).

However... I don't want to wait around any longer not working on this, waiting for financial support. I'd like to focus on improved macOS support in the next release of LightGBM (after v4.3.0)

I think LightGBM should pursue cross-compilation. That's what conda-forge does (conda-forge docs) and at this point it's clear to me that providing experimental macOS arm64 wheels (cross-compiled but not tested) would be better for LightGBM's users than forcing them to build from source.

@stephenpardy do you or others from your organization have some capacity to help with adding x86_64 -> arm64 macOS cross-compilation support to LightGBM + the lightgbm Python package?

To start, I'd welcome some help generating a reproducible example for #5269 to help narrow down some of the build issues and to be able to test fixes (even if those fixes are only run interactively on our laptops).

[jameslamb]

You can try FlyCI's M1 and M2 runners

@radeva thanks very much for your post. I saw a similar on on XGBoost as well (dmlc/xgboost#9570 (comment)).

At this time, I don't support adding FlyCI to LightGBM's CI setup.

I'm nervous about the security implications of using an externally-provided hosted runner with GitHub Actions.

I looked at https://www.flyci.net/ and don't see the things I'd expect from a company that's offering CI services compliant with Apple's licensing terms for offering M1/M2/M3 services, like:

• contact information
• a license / terms-of-use
• documentation describing the architecture and security model

For example

• would processes on the FlyCI runner be able to access this repo's secrets?
• what software is installed on the runners? what will artifacts I build there be linking and testing against?
  • versions of macOS?
• how long are logs retained?

To be clear, I'm not looking for answers to these questions. I am just mentioning some examples of reasons that I am not open to investing this project's maintainers' limited time and attention into using FlyCI at this time. I'd be happy to explore it in the future if some of those things were addressed.

[radeva]

Thank you @jameslamb for the detailed feedback. We really appreciate it!
We're already working on providing the mentioned information and we'll get back to you once it is available.

[jmoralez]

We could look at using cibuildwheel, I've used it in a couple of projects and it's worked great for me. It also allows to customize the build process, for example scikit-learn packages conda's OpenMP for MacOs in their wheels.

[jameslamb]

Definitely could be an option! Although I don't think we should resort to bundling and redistributing a copy of OpenMP lightly, as that can cause other portability and legal issues.

I don't think we should pursue cibuildwheel until we can reliably build working arm64 wheels using just cmake commands and things like build and pip. I strongly suspect that some of the issues I've mentioned above have fixes that lie in the project's CMake setup.

[jameslamb]

It looks like GitHub is now providing free, GitHub-hosted macOS M1 runners!

https://github.blog/changelog/2024-01-30-github-actions-introducing-the-new-m1-macos-runner-available-to-open-source/

I'll test that in the next few days 😁

(thanks @leofang for telling me about this 😊 )

[jameslamb]

I'll test that in the next few days

Well that statement from January turned out to not be true, for most definitions of "next few days" 😬

I am working actively on this now. For anyone subscribed to this issue, please stay subscribed... when we have wheels that I feel are ready, I'll push them to test-PyPI and ask for volunteers to try installing them.

[jayqi]

Just to document:

GitHub recently on 2024-04-22 upgraded the GitHub Actions macos-latest image to macOS 14, and as part of that, changed the runners to be ARM64 instead of Intel/x86_64. This means that a wheel is no longer available if running pip install lightgbm on macos-latest, and pip will attempt to build the LightGBM Python package from source.

• Reference: macos 14 is only arm64 actions/runner-images#9741

I got the following failure during build time:

*** Building project with Ninja...
      2024-04-29 16:00:35,864 - scikit_build_core - INFO - RUN: /opt/homebrew/bin/cmake --build /var/folders/3m/p59k4qdj0f17st0gn2cmj3640000gn/T/tmp6n6643rr/build
      ninja: error: '/opt/homebrew/opt/libomp/lib/libomp.dylib', needed by '/private/var/folders/3m/p59k4qdj0f17st0gn2cmj3640000gn/T/pip-install-f6cmspx3/lightgbm_e756e84[80](https://github.com/jayqi/github-actions-sandbox/actions/runs/8881677094/job/24384593834#step:4:81)cc541db80a2e503a263c854/lib_lightgbm.so', missing and no known rule to make it
      
      *** CMake build failed

Example: https://github.com/jayqi/github-actions-sandbox/actions/runs/8881677094/job/24384594174

Installing OpenMP explicitly beforehand solves the problem, e.g.,

      - name: Install libomp with Homebrew
        if: ${{ runner.os == 'macOS' && runner.arch == 'ARM64' }}
        run: |
          brew install libomp

Example: https://github.com/jayqi/github-actions-sandbox/actions/runs/8881677094/job/24384593509

Full job source: https://github.com/jayqi/github-actions-sandbox/blob/bf292c1afd105ce77f4c855893f4d1d1b0e99da9/.github/workflows/lightgbm-macos.yml#L53-L79

Thanks to @jameslamb for helping me debug this offline.