Can Aytöre

Industrial Engineer, M.Sc.
R & Python-ist 🐍

Guide to Building the ´Perfect´ Python Project

In this guide, I will try to walk you through some best practices for setting up an exemplary Python project to maintain the highest quality code. Happy reading!

Python is a versatile and powerful object-oriented programming language that has gained immense popularity in various domains, from web development and data science to automation and artificial intelligence. However, building a perfect Python project requires more than just writing code and make it work. Whether you’re an experienced programmer or just getting started with Python, spending a tiny amount of time to setup a project with the best tools will save immense time, and lead to a happier coding experience and long-term success.

It involves setting up a robust project structure, managing dependencies, adhering to coding standards, and ensuring efficient testing and code coverage, checking vulnerabilities or threats, and many more not covered here.

Step 1: Project Structure

A well-organized project structure is the foundation of any Python project. A common and recommended structure is as follows:

my_project/
|-- src/
|   |-- __init__.py
|   |-- module1.py
|   |-- module2.py
|-- tests/
|   |-- __init__.py
|   |-- test_module1.py
|   |-- test_module2.py
|-- changelog.md
|-- Dockerfile
|-- Pipfile
|-- Pipfile.lock
|-- README.md
|-- setup.py
|-- .gitignore

src: Houses your Python source code files reside.
tests: Stores your test files.
changelog: A log or record of all notable changes made to the project.
Dockerfile: A text document that allow us to build images for Docker.
README.md: Documentation of your project, including installation instructions and usage guidelines.
setup.py: Defines project metadata and dependencies.
Pipfile: Manages dependencies using Pipenv.
.gitignore: List of files and directories to be ignored by version control. A collection of useful templates can be found here.

Step 2: Dependency Management with Pipenv

Pipenv simplifies dependency management by creating a virtual environment and managing project-specific dependencies. Dependencies are external libraries or packages that your Python project relies on to function correctly. Pipenv combines the capabilities of both ´pip´ and ´virtualenv´ to provide a robust and user-friendly solution. Managing dependencies sometimes is a bit tricky, we could come across thousands of issues on StackOverFlow.

To install Pipenv, run:

pip install pipenv

To initialize your project with Pipenv, run:

pipenv --python 3.7   # Use your desired Python version

Then, (un)install your project dependencies:

pipenv install <package-name>
pipenv uninstall <package-name>

Pipenv will automatically update the Pipfile and Pipfile.lock files to reflect the new dependency.

To update all dependencies to their latest versions:

pipenv update

To activate the Virtual Environment:

pipenv shell

This puts you inside the project’s isolated environment.

To leave the virtual environment and return to your system’s Python, simply type:

exit

For best practice, I recommend you run the following to get the latest changes in the Pipfile (if any) before activating the virtualenv.

pipenv sync

Pipenv in general ensures a clean and isolated environment for your project, making it easy to replicate across different systems.

Step 3: Code Formatting with isort and black

Consistent code formatting enhances readability and maintainability. isort sorts and organizes your imports, while black formats your code according to PEP 8 standards.

Install isort and black using Pipenv:

pipenv install isort black --dev

To format your code, run:

pipenv run isort src/
pipenv run black src/

Note that Black and Isort have incompatible default settings, so we need to customize Isort to align with Black’s preferences. To do this, you should include the following in a setup.cfg file.

[
isort
]
multi_line_output=3
include_trailing_comma=True
force_grid_wrap=0
use_parentheses=True
line_length=88

Step 4: Style Enforcement with flake8

flake8 checks your code for style and potential issues.

Install it with Pipenv:

pipenv install flake8 --dev

Run flake8 to check your code:

pipenv run flake8 src/

Address any issues and ensure your code complies with PEP 8 guidelines.

Step 5: Testing with pytest

Testing is crucial for ensuring the correctness of your code. pytest is a popular testing framework that simplifies test writing and execution.

Install pytest using Pipenv:

pipenv install pytest --dev

Create test files in the tests/ directory and write your test cases using pytest conventions. Run tests with:

pipenv run pytest

Step 6: Test Coverage with pytest-cov

Code coverage helps identify areas of your codebase that lack test coverage. pytest-cov is an excellent tool for measuring code coverage.

Install pytest-cov using Pipenv:

pipenv install pytest-cov --dev

To generate a coverage report, run:

pipenv run pytest --cov=src --cov-report=html

Open the generated HTML report in your browser to view code coverage statistics.

Preferably create a new ´.coveragerc´ file to return only the coverage statistics of our application code assuming the application code lives in the ´src´ module.

[run]
source = best_practices

[report]
exclude_lines =
    # Have to re-enable the standard pragma
    pragma: no cover

    # Don't complain about missing debug-only code:
    def __repr__
    if self\.debug

    # Don't complain if tests don't hit defensive assertion code:
    raise AssertionError
    raise NotImplementedError

    # Don't complain if non-runnable code isn't run:
    if 0:
    if __name__ == .__main__.:

You may want the test coverage of application code to fail if the coverage is less than 80% by providing the ´–cov-fail-under=80´ flag.

Step 7: Check for vulnerabilities

It analyzes your project’s dependencies and display a report with any detected vulnerabilities. Follow the recommended actions to address the vulnerabilities.

To initialize the scanning, simply type:

pipenv check --output full-report

Bonus: Git hooks with pre-commit

Git hooks allow you to run scripts any time you want to commit or push. Git hook scripts are useful for identifying simple issues before submission to code review. We run our hooks on every commit to automatically point out issues in code.

Here we can configure all of tools above to run on any changed python files on committing. For that, we would need a new file called ´.pre-commit-config.yaml´.

repos:
  - repo: repo-name
    hooks:
      - id: isort
        name: isort
        stages: [commit]
        language: system
        entry: pipenv run isort
        types: [python]

      - id: black
        name: black
        stages: [commit]
        language: system
        entry: pipenv run black
        types: [python]

      - id: flake8
        name: flake8
        stages: [commit]
        language: system
        entry: pipenv run flake8
        types: [python]
        exclude: setup.py

      - id: pytest
        name: pytest
        stages: [commit]
        language: system
        entry: pipenv run pytest
        types: [python]

      - id: pytest-cov
        name: pytest
        stages: [push]
        language: system
        entry: pipenv run pytest --cov --cov-fail-under=80
        types: [python]
        pass_filenames: false

If you would like to skip these hooks you can run ´git commit –no-verify´ or ´git push –no-verify´

To finish setting up, follow these steps:

# Enter project directory
cd <repo_name>

# Initialize git repo
git init

# Install dependencies
pipenv install --dev

# Setup pre-commit and pre-push hooks
pipenv run pre-commit install -t pre-commit
pipenv run pre-commit install -t pre-push

Once you’re happy with the code you can then do your first ´git commit´ and all the hooks will be run.

Conclusion

Here are some additional best practices to consider:

Version Control: Use Git for version control and host your code on platforms like GitHub, GitLab, or Bitbucket.
Documentation: Write clear and concise docstrings for your functions and classes. Use tools like Sphinx for generating documentation. Also you may have a look Documentation As Code applications.
Continuous Integration (CI): Set up a CI/CD pipeline using services like TravisCI, CircleCI, Bamboo or GitHub Actions to automate testing and deployment.
Linter Integration: Integrate flake8 and pytest into your CI pipeline to enforce code style and run tests automatically on each commit.
Code Reviews: Collaborate with team members through code reviews/PRs to maintain code quality and share knowledge.
Security: Regularly update dependencies to patch security vulnerabilities. Using Blackduck could be an option.

Building the perfect Python project requires careful planning, adherence to best practices, and the use of modern tools. By following these steps, you can create a Python project that is clean, maintainable, well-tested, and ready for collaboration with other developers.

Also, I personally think it would be beneficial for you to take a look at 12factor. Happy coding!

2021 1

2021

Guide to Building the ´Perfect´ Python Project

Oct 2, 2021 10 min read

In this guide, I will try to walk you through some best practices for setting up an exemplary Python project to maintain the highest quality code. Happy read...