Setup Python environment with pre-commit, pytest and github actions

This repository was created from some blogs and documents, which I will leave below.

Works in python and PySpark.

How to use:

You can clone the repo https://github.com/anthonypernia/pre-commit-python-action-example.git or just download the files that you need and use in your repo.

Install the necessary libraries:

• pre-commit
• mypy
• black
• pytest
• isort
• pylint
• coverage

Or just install the requirements.txt that is in this repo using: pip install -r requirements.txt

Pre-commit

Pre-commit is a tool that we use to identify issues in our code, such as type errors, syntax errors, functions without documentation, etc

To learn more about Pre-commit you can check the Documentation

To execute pre-commit use the following command:

pre-commit run --all-files

And then, the library starts checking the code with all the hooks that we are using in the pre-commit-config file. The first time it will take about 2 minutes

You could receive three messages in the pipeline:

• Passed - When everything is fine, and I pass the test
• Failed - When the test failed
• Skipped - When a test is skipped or does not apply to the file

The good news is that most hooks do the necessary modifications to improve the code automatically so you just had to add the file again with git add FILE_TO_ADD.py and run again pre-commit run --all-files

Here are two running examples

• The first example is when some test fails, in one case the Hook solves the problem and you only need to add the files that were modified, in the other case, you need to solve the code issue

• The second example is when the code passes all tests and is ready to be uploaded to the remote repository.

However, the main function of pre-commit is to be called automatically before the commit. You can do it with:

pre-commit install

You will receive a message like this:

pre-commit installed at .git/hooks/pre-commit

Then, every time you do a commit, the pre-commit will be executed

In case it passes all the tests, the commit will be executed, otherwise it will not

You can copy my pre-commit configuration:

Pre-commit file: .pre-commit-config.yaml Link

Github Action using Pre-Commit, PyTest and Coverage

Github actions is the way that we have to apply CICD or run customized workflows

You can take a look at the Documentation

The funniest thing is that pre-commit can be added to a workflow as an automated task, we can add unit testing, check the coverage, etc. All this task can be executed after push our code or create a pull request.

There are some examples with customized workflows:

• pre-commit-and-merge.yml, This workflow will merge all the push that pass the pre-commit tests, from the develop branch, into the master branch

• pre-commit-on-pr.yml, This workflow will execute pre-commit on all pull requests to main branch and will show the result in the PR

• pre-commit-on-push-and-pr.yml, Same as before , but in this case it will execute test on PR to main branch and all push to develop

• pre-commit-and-pytest.yml.yml, In this case, in addition to running the pre-commit tests, this workflow will run the pytest suite.

• pre-commit-pytest-coverage-codecov.yml, This workflow is the most complete. it will execute the following actions:

  • run pre-commit test
  • run pytest suite
  • check the coverage and upload to Codecov
  • it will fail if coverage is not ok
  You can find more info in Codecov

Best Option: pre-commit-pytest-coverage-codecov

The one that I consider the best is the last one because it checks the coverage, execute the pytest suite, and executes the pre-commit so that at the end you can see the coverage graphs in the PR

You need to create an account in Codecov, give it permission to scan repositories, get the codecov token , and store in secrets as CODECOV_TOKEN

If you want to configure coverage to approve, you need to create a file called codecov.yml

You can check the file that I use Here

Here is an example of a PR using that workflow, in that case, we have some errors in pre-commit and we need to solve it

When the problem is solved, when you push your code again, the tests will be updated, in that case, we need to update the test to complete the coverage

After solving all the errors, you can see all the tests that were approved

The documentation that I used to create this repository is:

https://pre-commit.com/#install

https://pre-commit.com/#usage

https://composed.blog/python/pre-commit

https://towardsdatascience.com/pre-commit-hooks-you-must-know-ff247f5feb7e

https://verdantfox.com/blog/view/how-to-use-git-pre-commit-hooks-the-hard-way-and-the-easy-way

Some hooks examples:

https://github.com/pre-commit/pre-commit-hooks

Information about Pytest:

https://docs.pytest.org/en/6.2.x/customize.html

Codecov Documentation:

https://docs.codecov.com/docs