(transferred) [python-package] how to install the Python package from source?

[jameslamb]

Description

migrated from #5053 (comment)

Hi!
I am trying to increase kMaxPosition = 10000 for my project, but I am not able to compile the source code.

I am stuck here.
Install the python package by cd into LightGBM/python-package and python setup.py install --precompile

Since, setup.py has been deprecated, I tried using flit to install the package. But I am getting this. Please help.
error: subprocess-exited-with-error

× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [24 lines of output]
Traceback (most recent call last):
File "C:\Users\snigam\AppData\AppData2\Lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 353, in
main()
File "C:\Users\snigam\AppData\AppData2\Lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\snigam\AppData\AppData2\Lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 149, in prepare_metadata_for_build_wheel
return hook(metadata_directory, config_settings)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\snigam\AppData\Local\Temp\pip-build-env-vyyms50h\overlay\Lib\site-packages\scikit_build_core\build_init_.py", line 72, in prepare_metadata_for_build_wheel
return _build_wheel_impl(
^^^^^^^^^^^^^^^^^^
File "C:\Users\snigam\AppData\Local\Temp\pip-build-env-vyyms50h\overlay\Lib\site-packages\scikit_build_core\build\wheel.py", line 142, in build_wheel_impl
metadata = get_standard_metadata(pyproject, settings)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\snigam\AppData\Local\Temp\pip-build-env-vyyms50h\overlay\Lib\site-packages\scikit_build_core\build\metadata.py", line 39, in get_standard_metadata
metadata = StandardMetadata.from_pyproject(new_pyproject_dict)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\snigam\AppData\Local\Temp\pip-build-env-vyyms50h\overlay\Lib\site-packages\scikit_build_core_vendor\pyproject_metadata_init.py", line 257, in from_pyproject
cls.get_license(fetcher, project_dir),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\snigam\AppData\Local\Temp\pip-build-env-vyyms50h\overlay\Lib\site-packages\scikit_build_core_vendor\pyproject_metadata_init.py", line 395, in _get_license
raise ConfigurationError(msg, key='project.license.file')
scikit_build_core._vendor.pyproject_metadata.ConfigurationError: License file not found ("LICENSE")
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
Traceback (most recent call last):
File "", line 198, in run_module_as_main
File "", line 88, in run_code
File "C:\Users\snigam\AppData\AppData2\Scripts\flit.exe_main.py", line 7, in
File "C:\Users\snigam\AppData\AppData2\Lib\site-packages\flit_init.py", line 219, in main
installer.install()
File "C:\Users\snigam\AppData\AppData2\Lib\site-packages\flit\install.py", line 432, in install
self.install_with_pip()
File "C:\Users\snigam\AppData\AppData2\Lib\site-packages\flit\install.py", line 370, in install_with_pip
check_call(cmd, shell=shell)
File "C:\Users\snigam\AppData\AppData2\Lib\subprocess.py", line 413, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['C:\Users\snigam\AppData\AppData2\python.exe', '-m', 'pip', 'install', '.[scikit-learn,pandas,arrow,dask]']' returned non-zero exit status 1.

I tried this as well python -m pip install --upgrade pip setuptools, but it didn't work.

If there's any other way to increase 'kMaxPosition', please share.

[jameslamb]

@NigamSomya To install the Python package from a local copy of LightGBM, run the following:

sh build-python.sh install

For details on how to customize that, follow the docs at https://github.com/microsoft/LightGBM/blob/master/python-package/README.rst. You can ask here if you have other questions or concerns.

[NigamSomya]

Hi! Thanks that worked successfully, but it installed the package in this directory C:\Users\snigam\AppData\AppData2\Lib\site-packages\lightgbm\ which is not picking up the changes that I made to the dcg_calculator.cpp

Plus, when I cloned the source code in a different directory, I expected the command that you shared would install the necessary files in that directory only. Please help.

[jameslamb]

which is not picking up the changes that I made to the dcg_calculator.cpp

What evidence do you see the the changes you've made to LightGBM's source code are not being picked up in the Python package?

Just to be sure you understand...the Python package will not contain any .h or .cpp files. Those files are compiled into a shared library, lib_lightgbm.dll, that the Python package code calls.

If you run the command I show you from the root of the same repo where you changed any .cpp files in LightGBM, the resulting Python package will reflect your changes. Be sure you're doing that.

I expected the command that you shared would install the necessary files in that directory only.

The command I shared will build the lightgbm Python package and install it to where pip install on your system places packages by default.

If you want to control that more tightly, omit the install.

sh build-python.sh bdist_wheel

That will create a file with a name like lightgbm-4.3.0.99-py3-none-win_amd64.whl in the dist/ directory at the root of the LightGBM repo. That file is a Python wheel and you can install it to wherever you want.

[NigamSomya]

Hi!
Because I am getting the same error (kMaxposition error), I also ran the command in the same directory where my source code is place but somehow it gets installed as if I am running the usual pip command to install the lightgbm (in the usual appdata directory).

Also, the same thing happened when I tried to use this command sh build-python.sh bdist_wheel. Couldn't find anything named lightgbm-4.3.0.99-py3-none-win_amd64.whl. Must have been installed in the default directory and it behaves like the usual lightgbm package.
I am not sure what I am doing wrong, can you please help me with this?

[jameslamb]

somehow it gets installed as if I am running the usual pip command to install the lightgbm (in the usual appdata directory).

LightGBM's Python package does not support editable installs.

This command builds a wheel and installs it with pip install:

sh build-python.sh bdist_wheel install

This command builds a wheel and places it in the directory ./dist, relative to the root of the repo:

sh build-python.sh bdist_wheel

And then you are free to install it however you'd install other Python wheels. For example, like this:

pip install --no-deps ./dist/lightgbm-4.3.0.99-py3-none-win_amd64.whl

Because I am getting the same error (kMaxposition error)

To help you, we need more information than what you've provided.

• what version of LightGBM? (output of git log -n 2)
• what version of Python?
• can you share all of the build and installation logs?

Some things to try:

• be sure you save the file dcg_calculator.cpp before compiling
• be sure the Python interpreter used wherever you're running your program is the same one that was active from wherever you installed LightGBM
  • for example, if you are running your training code in a separate environment managed by venv / conda / poetry, then you need to install lightgbm into that environment
  • for example, if you are using a Jupyter notebook or similar, you may need to restart the kernel