WIP: release v4.4.0

[jameslamb]

Release checklist:

Copied from #6277, with a few changes.

before merge

• Update VERSION.txt number.
• Update version in Appveyor config file.
• Update version in configure file of R-package: /gha run r-configure.
• Change development.mode from unreleased to release in pkgdown config file.
• Update version in python-package/pyproject.toml
• Add release branch to RTD versions, trigger a new build, check docs
• All docs for new behavior have Sphinx versionadded:: annotations (docs on those)
  • nothing in this release requires those
• All new parameters in config.h have *New in version {version}* comments added
  • no new parameters added in config.h in this release
• Run the valgrind checks with /gha run r-valgrind (docs link)
• test Python and R packages on M1/M2 Mac

Items to be addressed before merging:

• [R-package] always name the shared library 'lightgbm', not 'lib_lightgbm' #6432
• [c++] remove uses of '..' in headers #6409
• [python-package] clarify max_depth warning and limit when it is emitted #6402
• [python-package] Allow to pass early stopping min delta in params #6274
• deprecation warnings for these:
  • [RFC] remove 'categorical_feature' and 'feature_name' parameters in cv() and train() #6435
  • [RFC] remove HDFS support? #6436

I'd really like for this to be the first release with arm64 macOS wheels, if we can get there. But let's not delay the release waiting on that if everything else above is done.

• WIP: [ci] [python-package] add arm64 macOS wheels #6391

after merge

• Push v* tag to trigger GitHubRelease action at Azure Pipelines.
• Update stable tag at GitHub.

code to update tags (click me)

NEW_VERSION="4.4.0"
git fetch upstream --tags
git tag -d stable
git push upstream :refs/tags/stable
git tag stable
git tag "v${NEW_VERSION}"
git push upstream stable "v${NEW_VERSION}"

• Add release notes from release-drafter to the automatically-created GitHub release, convert it from Draft to published.
• Add new tag to RTD versions and trigger a new build.
• Upload release to CRAN
• Remove the release branch from RTD versions
• Upload release to test PyPI
• Upload release to PyPI.

code to publish to PyPI (summary)

mkdir /tmp/lgb-release
cd /tmp/lgb-release

NEW_VERSION="4.4.0"

# NOTE: requires a GitHub personal access token with "repo" scope
gh release download \
    --repo microsoft/LightGBM \
    --dir ./artifacts \
    --pattern 'lightgbm*-py3-*.whl' \
    --pattern "lightgbm-${NEW_VERSION}.tar.gz" \
    "v${NEW_VERSION}"

twine upload \
    -r testpypi \
    ./artifacts/*

(gh is the GitHub CLI, see https://cli.github.com/manual/gh_release_download)

Then confirmed that installing the latest wheel works.

pip install -i https://test.pypi.org/simple/ "lightgbm==${NEW_VERSION}"
python ./examples/python-guide/logistic_regression.py

Then pushed them to real PyPI.

twine upload \
    ./artifacts/*

• Update cran-comments.md when new release is accepted on CRAN.
• Upload release to NuGet.
• Update version and commit hash in Homebrew formula (docs)
• put up another PR incrementing development version here in GitHub (example: bump development version to 4.1.0.99 #6090)

code to open that dev version PR (click me)

RELEASE_PR_NUMBER="6439"
RELEASE_VERSION=$(cat ./VERSION.txt)
DEV_VERSION="${RELEASE_VERSION}.99"

git checkout -b ci/dev-version

echo "${DEV_VERSION}" > ./VERSION.txt
sed \
    -i .bak \
    "s|${RELEASE_VERSION}|${DEV_VERSION}|g" \
    .appveyor.yml

sed \
    -i .bak \
    "s|version = \"${RELEASE_VERSION}\"|version = \"${DEV_VERSION}\"|g" \
    ./python-package/pyproject.toml

sed \
    -i .bak \
    's|mode\: release|mode\: unreleased|g' \
    ./R-package/pkgdown/_pkgdown.yml

docker run \
    --rm \
    -v $(pwd):/opt/LightGBM \
    -w /opt/LightGBM \
    ubuntu:22.04 \
    ./R-package/recreate-configure.sh

git add \
    ./.appveyor.yml \
    ./R-package/configure \
    ./R-package/pkgdown/_pkgdown.yml \
    ./VERSION.txt \
    ./python-package/pyproject.toml

commit_msg="bump development version to ${DEV_VERSION}"
git commit -m "${commit_msg}"
git push upstream ci/dev-version

gh pr create \
    --repo microsoft/LightGBM \
    --base 'master' \
    --label 'maintenance' \
    --title "${commit_msg}" \
    --body "now that v${RELEASE_VERSION} has been released: #${RELEASE_PR_NUMBER}"

Notes for Reviewers

I believe this should be v4.4.0 instead of v4.3.1 because of some small breaking changes:

• [cmake] [c++] use .dylib extension unconditionally on macOS #6418
• ENH more stable gradient of CrossEntropy #6327
• [R-package] Rename slice() to lgb.slice.Dataset() #6293

See https://github.com/microsoft/LightGBM/releases.

[jameslamb]

@shiyu1994 @guolinke @jmoralez @borchero @btrotta to help with the release, could you please do the following?

• comment here if you want to change anything about the proposed list of issues/PRs to be addressed before merging
• comment on the open Request for Comment issues:
  • [RFC] remove 'categorical_feature' and 'feature_name' parameters in cv() and train() #6435
  • [RFC] remove HDFS support? #6436
• share any other thoughts here about this release (e.g. the version number, the timing, the testing plan)

Thank you!

[borchero]

Plan sounds great to me, thanks a lot for taking care of the release @jameslamb!

I think that v4.4.0 makes sense as the next version since we're introducing a few new features :)

[shiyu1994]

Thank you @jameslamb for the releasing. I don't think we have anything urgent to be included in v4.4.0 for now.

[celestinoxp]

Please add verbose fix in this new version...