Project Update - 3

This is continuation in the series about our side project. This post is a follow up to the pervious post on various tools used to build the process of building the application

The Environment

Core language used in our project is Python. Python is used to fetch details from API calls, cleaning the API return value, and also writing it into DB. Even though the language is very popular and user ranges from amateurs to big corporations, it undergoes development. Implying the new versions are created and released often. And so, setting up the environment is necessary.

During the start of the project, the Python 3.11 was newly released. So, its decided to use 3.9 as it is 2 versions earlier to then latest, as this will be more stable and less of breaking changes. However, the 3.11 seems to be the best version yet. Once the basic features are built and tested, we can move into 3.11.

Here is the yaml file for initializing the Conda environment.

Pre-commit

The Pre-commit gets driven by the precommit config file. This file holds the list of checks to be performed.

repos:
-   repo: https://github.com/psf/black
    rev: 22.12.0
    hooks:
    - id: black
      language_version: python3.9
-   repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.4.0
    hooks:
    - id: trailing-whitespace
    - id: end-of-file-fixer
    - id: check-toml
    - id: check-added-large-files
    - id: check-case-conflict
    # - id: detect-aws-credentials
    - id: detect-private-key
    - id: no-commit-to-branch
-   repo: https://github.com/asottile/reorder_python_imports
    rev: v3.9.0
    hooks:
    -   id: reorder-python-imports

Pytest

The project has almost 10k stars, forked 2.3k times. This is enough to understand the popularity of the module. Eventhough unittest comes with the Python installation, Pytest has several advantages over it.

For me to chose pytest over unittest has 2 main reasons - 1. plugins and 2. its simplicity (single assert to rule them all).

Plugins

It would be unfair if we are not using the plugins or not mentioning their usage.

1. pytest-clarity - as per the name, the error message carries better clarity
2. pytest-sugar - shows the progress bar against the tests

Code Coverage

Coverage.py is the Python code coverage tool that we use.

coverage is used along side pytest to generate the coverage report

coverage run -m pytest; coverage report; coverage html

coverage run -m pytest → This runs the pytest and fetches coverage

coverage report → Report coverage stats on modules.

coverage html → Create an HTML report

Once the coverage report is ready, we pass the report to codecov.io for badge creation.

Github Actions

The GH actions is our CI tool. With every push and merger of pull request, the below events gets triggred

1. Python 3.9 is created with modules declared in the requirements.txt.
2. Also, pytest and its dependencies are installed in the environment.
3. Execution of code coverage.

name: Workflow for Codecov example-python
on: [push, pull_request]
jobs:
  run:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Set up Python 3.9
        uses: actions/setup-python@v2
        with:
          python-version: '3.9'
      - name: Install dependencies
        run: |
          pip install -r requirements.txt
          pip install pytest
          pip install pytest-cov
          pip install coverage
      - name: Run tests and collect coverage
        run: |
          coverage run -m pytest
          coverage report
      - name: Upload coverage to Codecov
        uses: codecov/codecov-action@v3

Branching

During the development, the main is left undisturbed. Instead a new branch is created for new changes. Once the code residing in new branch is tested, code coverage verified, PR needs to be raised and merged to main branch.

• This ensures mistaken commits into main branch.
• Usage of non-main branch is enforced by Pre-commit.

The advantage of this post is that above mentioned code and process can be reused to other projects. This more or less acts as a template for future projects.