[CICD] Improving CI/CD for Selective Dependency Installation

[oualib]

Description:

This epic focuses on enhancing the Continuous Integration/Continuous Deployment (CI/CD) pipeline for VerticaPy, allowing users to choose specific dependencies based on their requirements.

Tasks:

• Enable users to choose among multiple options when installing VerticaPy dependencies.
• Implement a mechanism to install only the needed modules based on user preferences.

Definition of Done:

• Users can install VerticaPy with a comprehensive set of dependencies using pip install verticapy.
• Users can selectively install only the necessary modules based on their preferences using pip install verticapy[option].

Contributions:

Contributions to this important enhancement are highly welcome! Whether it's implementing the mechanism for selecting dependencies or ensuring smooth functionality in both installation options, your participation is key. Let's collaborate to make VerticaPy installation more flexible! Thank you.

[oualib]

@afard I think we have to limit the dependencies and make the structure lighter with some useful options. Do you have any ideas so we can easily close this issue with a nice solution?

[afard]

Currently, we have four category for dependencies.

The first one is listed in requirements-testing.txt. This is the list of packages we need to run our tests. It is specifically used in tox.ini and .github/workflows/pylint.yml.

The second category is the packages we need for development. They are listed in requirements-dev.txt and used in .github/workflows/pylint.yml.

IIUC, this issue is more about the last two categories "install_requires" and "extras_require" listed in setup.py.

• Every package in the "install_requires" list needs to be installed in order to have VerticaPy successfully installed. Therefore, it should contain only the absolutely necessary packages. Nevertheless, we can condition the necessity of each package based on python_version or platform_system.
• Optional dependencies for installing VerticaPy are listed as "extras_require". Right now, we have defined only "all" group in this category. This group contains all optional packages. We can add several new groups with smaller subset of packages representing the needs of different group of users.

In addition to above notes, we can also drop usage of setup.py in favor of pyproject.toml. It is a new option (and the encouraged one) to configure Python builds, made available first in pip version 10.0.