# Python Project Best Practices Use this guide when creating sibling repositories that should feel like this project: a Python-first package with clear package boundaries, typed public APIs, focused documentation, and repeatable validation commands. Replace `project_name` with the distribution name and `package_name` with the import package. They are often the same, but keep the distinction explicit. ## Repository Shape Prefer this top-level layout for new Python package repositories: ```text project-name/ README.md CONTRIBUTING.md LICENSE pyproject.toml setup.py # only when build-time customization is needed tox.ini docs/ conf.py index.md development.md architecture.md api.md examples/ 01-basics/ README.md 00-package-tour.py tests/ conftest.py core/ workflows/ package_name/ __init__.py version.py py.typed core/ workflows/ ``` This repository uses a flat package layout: the import package lives directly under the repository root rather than under `src/`. Keep that style for sibling projects when the goal is consistency with this project. Keep local-only or generated directories out of the source contract: - virtual environments and local package builds; - `__pycache__`, `.pytest_cache`, `.mypy_cache`, Ruff cache, and docs builds; - generated notebooks, checkpoints, prediction outputs, cache arrays, and large experiment artifacts; - scratch directories, backup directories, and one-off planning notes. ## Packaging Use `pyproject.toml` as the source of packaging metadata. Recommended baseline: ```toml [project] name = "project_name" dynamic = ["version"] requires-python = ">=3.10" authors = [ {name = "Project Developers"}, ] dependencies = [ ] readme = "README.md" license = {file = "LICENSE"} description = "Short project description" [project.optional-dependencies] docs = [ "myst-parser>=2", "sphinx>=7.2", ] dev = [ "mypy>=1.13", "pytest>=8", "ruff>=0.6", "tox>=4.39", ] [build-system] requires = [ "setuptools>=77", ] build-backend = "setuptools.build_meta" [tool.setuptools.packages.find] include = ["package_name*"] namespaces = false [tool.setuptools.package-data] package_name = ["py.typed"] [tool.setuptools.dynamic] version = {attr = "package_name.version.__version__"} ``` Keep required dependencies small. Put optional integrations behind extras such as `docs`, `dev`, backend names, and `all`. Base imports should work without optional engines, plotting stacks, native extensions, or development tools. Use `setup.py` only when setuptools needs executable build logic, such as optional C++ or CUDA extensions. Keep it thin and make native builds opt-in with environment variables. A pure-Python package usually does not need `setup.py`. Put the package version in `package_name/version.py`: ```python """Project version.""" __version__ = "0.1.0" ``` Add `package_name/py.typed` for PEP 561 typing support. ## Development Toolkit Use tox as the command surface for common checks: ```ini [tox] requires = tox >=4.39 envlist = lint type docs docs-tests tests [testenv] deps = pytest commands = python -m pytest {posargs} [testenv:lint] package = skip deps = ruff commands = ruff format --diff package_name tests examples setup.py ruff check package_name tests examples setup.py [testenv:format] package = skip deps = ruff commands = ruff format package_name tests examples setup.py ruff check --fix-only package_name tests examples setup.py [testenv:type] deps = mypy commands = mypy package_name {posargs} [testenv:docs-tests] deps = pytest commands = pytest --doctest-modules package_name {posargs} [testenv:docs] extras = docs commands = sphinx-build -W --keep-going -b html docs docs/_build/html {posargs} ``` Install locally from the repository root: ```sh python -m venv .venv source .venv/bin/activate python -m pip install --upgrade pip python -m pip install -e ".[dev,docs]" ``` Run these commands before review: ```sh tox -e lint tox -e type tox -e docs-tests tox -e docs tox -e tests ``` Use direct pytest commands for focused iteration: ```sh python -m pytest tests/core python -m pytest tests/workflows/test_example.py ``` ## Ruff And Mypy Use Ruff for formatting, linting, and import sorting: ```toml [tool.ruff.lint] select = ["E", "F", "B", "I"] [tool.ruff.lint.isort] lines-after-imports = 2 known-first-party = ["package_name"] [tool.ruff.format] docstring-code-format = true ``` Use mypy for the package, with a strict-but-gradual baseline: ```toml [tool.mypy] python_version = "3.10" files = ["package_name"] check_untyped_defs = true no_implicit_optional = true pretty = true show_error_codes = true strict_equality = true warn_redundant_casts = true warn_unused_configs = true warn_unused_ignores = true ``` Prefer precise types for public APIs, dataclasses, protocol-like contracts, and configuration objects. If a module must be baselined with `ignore_errors`, keep that override narrow and documented. Do not use broad type ignores to hide unclear ownership or unstable interfaces. ## Source Organization Organize by responsibility, not by implementation convenience. A small project does not need every directory below, but the dependency direction should remain clear. ```text package_name/ core/ # normalized inputs, outputs, validation, shared primitives terms/ # domain units or model components workflows/ # user-facing orchestration across lower layers adapters/ # optional external integrations analysis/ # diagnostics and plotting helpers benchmarks/ # performance smoke tests and expert entry points ``` Recommended dependency direction: ```text core -> terms or domain modules -> training, fitting, or workflows -> adapters, analysis, examples, benchmarks ``` Keep the lower layers free of optional engine objects and UI concerns. Adapters should translate external types into normalized internal types at the boundary. Workflows may compose lower layers for examples or common tasks, but they should not own kernels, file formats, cache layouts, or core validation rules. Use underscore-prefixed modules for implementation details: ```text package_name/terms/_shared.py package_name/terms/_parameters.py ``` Expose stable APIs through package `__init__.py` files and documentation. Keep `__init__.py` thin. If imports are expensive or require optional dependencies, use lazy exports rather than importing heavy modules at package import time. ## Public API Conventions Follow these naming and API rules: - classes use `PascalCase`; - functions, methods, modules, and files use `snake_case`; - constants use `UPPER_SNAKE_CASE`; - public modules avoid surprising import-time work; - public functions validate arguments early and raise clear `ValueError`, `TypeError`, or domain-specific exceptions; - deprecations use `warnings.warn(..., DeprecationWarning, stacklevel=2)`; - expensive optional integrations stay behind extras and adapter modules. Prefer small documented public surfaces. A helper is not public just because it is imported by another internal module. If users should rely on it, export it intentionally, add tests, and mention it in API docs. ## Documentation Use Markdown documentation with Sphinx and MyST. Recommended docs pages: - `docs/index.md`: navigation and project overview; - `docs/getting-started.md`: installation and first working example; - `docs/concepts.md`: user-facing mental model; - `docs/architecture.md`: package boundaries and dependency direction; - `docs/development.md`: setup, validation, and generated artifacts; - `docs/api.md`: documented public API. Recommended `docs/conf.py` choices: ```python extensions = [ "myst_parser", "sphinx.ext.autodoc", "sphinx.ext.napoleon", "sphinx.ext.viewcode", ] source_suffix = { ".md": "markdown", } napoleon_google_docstring = True napoleon_numpy_docstring = False autodoc_typehints = "description" ``` Use Google-style docstrings for Python APIs: ```python def load_record(path: str) -> Record: """Load a record from disk. Args: path: File path to read. Returns: Parsed record. Raises: ValueError: If the file contents are invalid. """ ``` Keep docs factual and close to implementation. When behavior depends on an optional backend, platform, or performance benchmark, state the scope and the fallback behavior. ## Examples Examples should be runnable and should double as lightweight integration coverage. Use numbered example folders for guided learning: ```text examples/ 01-basics/ 02-domain-case/ 03-advanced-workflow/ ``` Keep tracked `.py` files as the source of truth. Generated notebooks can be useful local presentation artifacts, but they should not become the canonical implementation unless the project is intentionally notebook-first. Track only small, stable input data that is required for examples or tests. Generated models, predictions, plots, caches, and checkpoints should normally be ignored. ## Testing Mirror package areas under `tests/`: ```text tests/ core/ terms/ workflows/ adapters/ analysis/ speed/ ``` Use pytest configuration in `pyproject.toml`: ```toml [tool.pytest.ini_options] testpaths = [ "tests/core", "tests/workflows", ] filterwarnings = [ "error", "ignore::DeprecationWarning", "error::DeprecationWarning:package_name.*", ] ``` Testing expectations: - core data structures need shape, dtype, validation, and error-path tests; - public APIs need behavior tests and import-surface tests; - workflows need small end-to-end tests with stable fixtures; - adapters need optional dependency skips and boundary-conversion tests; - performance-sensitive paths need targeted speed gates or benchmark smoke tests; - bug fixes should include a regression test unless the behavior is already covered by a higher-level test. Use warnings-as-errors for package deprecations. This keeps accidental warning regressions from becoming accepted behavior. ## Optional Native Code Keep native extensions optional. A sibling project should import and run in pure Python unless the user explicitly builds native code. Recommended pattern: - place sources under `package_name/csrc/`; - use environment variables such as `PROJECT_BUILD_NATIVE=1`; - fail clearly when a requested compiler, CUDA toolkit, or build dependency is unavailable; - choose automatic runtime fallback by default; - provide a strict runtime mode only when users need native code to be required. Do not let native extension availability change public API semantics. It should only affect performance or explicitly documented backend behavior. ## Review Checklist Use this checklist before opening a change: - package imports cleanly from a fresh editable install; - `README.md` explains the project, install command, and minimal example; - `pyproject.toml` has accurate dependencies, extras, package discovery, and tool configuration; - public APIs are typed, documented, exported intentionally, and tested; - optional dependencies are isolated behind extras and adapter modules; - generated files are ignored unless they are small, stable source artifacts; - `tox -e lint`, `tox -e type`, `tox -e docs-tests`, `tox -e docs`, and `tox -e tests` pass or have documented, scoped exceptions; - architecture docs match the actual dependency direction. ## Bootstrap Checklist For A Sibling Repository 1. Create the repository with `README.md`, `LICENSE`, `pyproject.toml`, `tox.ini`, `docs/`, `examples/`, `tests/`, and `package_name/`. 2. Add `package_name/version.py`, `package_name/py.typed`, and a thin `package_name/__init__.py`. 3. Configure setuptools package discovery and dynamic versioning. 4. Configure Ruff, mypy, pytest, and tox. 5. Add a minimal docs build with MyST Markdown and Google-style docstrings. 6. Add one import test, one behavior test, and one runnable example. 7. Install with `python -m pip install -e ".[dev,docs]"`. 8. Run the tox validation commands and fix failures before adding advanced features.