Contributing¶

Summary¶

PRs welcome!

Consider starting a discussion to see if there's interest in what you want to do.
Submit PRs from feature branches on forks to the develop branch.
Ensure PRs pass all CI checks.
Maintain test coverage at 100%.

Git¶

Why use Git? Git enables creation of multiple versions of a code repository called branches, with the ability to track and undo changes in detail.
Install Git by downloading from the website, or with a package manager like Homebrew.
Configure Git to connect to GitHub with SSH.
Fork this repo.
Create a branch in your fork.
Commit your changes with a properly-formatted Git commit message.
Create a pull request (PR) to incorporate your changes into the upstream project you forked.

Python¶

Hatch¶

This project uses Hatch for dependency management and packaging.

Highlights¶

Automatic virtual environment management: Hatch automatically manages the application environment.
Dependency resolution: Hatch will automatically resolve any dependency version conflicts using the pip dependency resolver.
Dependency separation: Hatch supports separate lists of optional dependencies in the pyproject.toml. Production installs can skip optional dependencies for speed.
Builds: Hatch has features for easily building the project into a Python package and publishing the package to PyPI.

Installation¶

Hatch can be installed with Homebrew or pipx.

Install project with all dependencies: hatch env create.

Key commands¶

# Basic usage: https://hatch.pypa.io/latest/cli/reference/
hatch env create  # create virtual environment and install dependencies
hatch env find  # show path to virtual environment
hatch env show  # show info about available virtual environments
hatch run COMMAND  # run a command within the virtual environment
hatch shell  # activate the virtual environment, like source venv/bin/activate
hatch version  # list or update version of this package
export HATCH_ENV_TYPE_VIRTUAL_PATH=.venv  # install virtualenvs into .venv

Testing with pytest¶

General info¶

Tests are in the tests/ directory.
Run tests by invoking pytest from the command-line within the virtual environment in the root directory of the repo.
pytest features used include:
pytest plugins include:
- pytest-mock
pytest configuration is in pyproject.toml.
Test coverage reports are generated by coverage.py. To generate test coverage reports, first run tests with coverage run, then generate a report with coverage report. To see interactive HTML coverage reports, run coverage html instead of coverage report.

Integration testing instructions¶

Integration tests will be skipped if cloud credentials are not present. Running integration tests locally will take some additional setup.

Make buckets on each supported cloud platform¶

Upload objects to each bucket¶

Upload an object to each bucket named .env.testing.

The file should have this content:

# .env
AWS_ACCESS_KEY_ID_EXAMPLE=AKIAIOSFODNN7EXAMPLE
AWS_SECRET_ACCESS_KEY_EXAMPLE=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLE
CSV_VARIABLE=comma,separated,value
EMPTY_VARIABLE=''
# comment
INLINE_COMMENT=no_comment  # inline comment
JSON_EXAMPLE='{"array": [1, 2, 3], "exponent": 2.99e8, "number": 123}'
PASSWORD='64w2Q$!&,,[EXAMPLE'
QUOTES_AND_WHITESPACE='text and spaces'
URI_TO_DIRECTORY='~/dev'
URI_TO_S3_BUCKET=s3://mybucket/.env
URI_TO_SQLITE_DB=sqlite:////path/to/db.sqlite
URL_EXAMPLE=https://start.duckduckgo.com/
OBJECT_STORAGE_VARIABLE='DUDE!!! This variable came from object storage!'

Generate credentials for each supported cloud platform¶

There are three sets of credentials needed:

AWS temporary credentials
AWS static credentials
Backblaze static credentials

The object storage docs have general info on generating the static credentials.

For AWS static credentials, create a non-admin user. The user will need an IAM policy like the following. This project doesn't do any listing or deleting at this time, so those parts can be omitted if you're going for least privilege.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": ["s3:ListBucket"],
            "Resource": ["arn:aws:s3:::<AWS_S3_BUCKET_NAME>"]
        },
        {
            "Effect": "Allow",
            "Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject"],
            "Resource": ["arn:aws:s3:::<AWS_S3_BUCKET_NAME>/*"]
        }
    ]
}

After attaching the IAM policy to the non-admin user, generate an access key for the non-admin user, set up an AWS CLI profile named fastenv, and configure it with the access key for the non-admin user. AWS static credentials are now ready.

AWS temporary credentials work a little differently. Create an IAM role, with a resource-based policy called a "role trust policy." The role trust policy would look like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<AWS_ACCOUNT_ID>:root"
            },
            "Action": ["sts:AssumeRole", "sts:TagSession"]
        }
    ]
}

Attach the identity-based policy you created for the IAM user to the role as well.

The end result is that the IAM user can assume the IAM role and obtain temporary credentials. The temporary credentials have the same IAM policy as the regular access key, so tests can be parametrized accordingly.

Run all the tests¶

Once you're finally done with all that, maybe go out for a walk or something.

Then come back, and run these magic words:

# set the required input variables
AWS_IAM_ROLE_NAME="paste-here"
AWS_S3_BUCKET_HOST="paste-here"
BACKBLAZE_B2_ACCESS_KEY_FASTENV="paste-here"
# leading space to avoid storing secret key in shell history
# set `HISTCONTROL=ignoreboth` for Bash or `setopt histignorespace` for Zsh
 BACKBLAZE_B2_SECRET_KEY_FASTENV="paste-here"
BACKBLAZE_B2_BUCKET_HOST="paste-here"
BACKBLAZE_B2_BUCKET_REGION="paste-here"
CLOUDFLARE_R2_ACCESS_KEY_FASTENV="paste-here"
 CLOUDFLARE_R2_SECRET_KEY_FASTENV="paste-here"
CLOUDFLARE_R2_BUCKET_HOST="paste-here"

# get AWS account ID from STS (replace jq with other JSON parser as needed)
AWS_ACCOUNT_ID=$(aws sts get-caller-identity | jq -r .Account)

# assume the IAM role to get temporary credentials
ASSUMED_ROLE=$(
  aws sts assume-role \
  --role-arn arn:aws:iam::$AWS_ACCOUNT_ID:role/$AWS_IAM_ROLE_NAME \
  --role-session-name fastenv-testing-local-aws-cli
)

# run all tests by providing the necessary input variables
AWS_IAM_ACCESS_KEY_FASTENV=$(aws configure get fastenv.aws_access_key_id) \
  AWS_IAM_SECRET_KEY_FASTENV=$(aws configure get fastenv.aws_secret_access_key) \
  AWS_IAM_ACCESS_KEY_SESSION=$(echo $ASSUMED_ROLE | jq -r .Credentials.AccessKeyId) \
  AWS_IAM_SECRET_KEY_SESSION=$(echo $ASSUMED_ROLE | jq -r .Credentials.SecretAccessKey) \
  AWS_IAM_SESSION_TOKEN=$(echo $ASSUMED_ROLE | jq -r .Credentials.SessionToken) \
  AWS_S3_BUCKET_HOST=$AWS_S3_BUCKET_HOST \
  BACKBLAZE_B2_ACCESS_KEY_FASTENV=$BACKBLAZE_B2_ACCESS_KEY_FASTENV \
  BACKBLAZE_B2_SECRET_KEY_FASTENV=$BACKBLAZE_B2_SECRET_KEY_FASTENV \
  BACKBLAZE_B2_BUCKET_HOST=$BACKBLAZE_B2_BUCKET_HOST \
  BACKBLAZE_B2_BUCKET_REGION=$BACKBLAZE_B2_BUCKET_REGION \
  CLOUDFLARE_R2_ACCESS_KEY_FASTENV=$CLOUDFLARE_R2_ACCESS_KEY_FASTENV \
  CLOUDFLARE_R2_SECRET_KEY_FASTENV=$CLOUDFLARE_R2_SECRET_KEY_FASTENV \
  CLOUDFLARE_R2_BUCKET_HOST=$CLOUDFLARE_R2_BUCKET_HOST \
  hatch run coverage run && coverage report

Code quality¶

Running code quality checks¶

Code quality checks can be run using the Hatch scripts in pyproject.toml.

Check: hatch run check
Format: hatch run format

Code style¶

Python code is formatted with Ruff. Ruff configuration is stored in pyproject.toml.
Other web code (JSON, Markdown, YAML) is formatted with Prettier.

Static type checking¶

To learn type annotation basics, see the Python typing module docs, Python type annotations how-to, the Real Python type checking tutorial, and this gist.
Type annotations are not used at runtime. The standard library typing module includes a TYPE_CHECKING constant that is False at runtime, but True when conducting static type checking prior to runtime. Type imports are included under if TYPE_CHECKING: conditions so that they are not imported at runtime. These conditions are ignored when calculating test coverage.
Type annotations can be provided inline or in separate stub files. Much of the Python standard library is annotated with stubs. For example, the Python standard library logging.config module uses type stubs. The typeshed types for the logging.config module are used solely for type-checking usage of the logging.config module itself. They cannot be imported and used to type annotate other modules.
The standard library typing module includes a NoReturn type. This would seem useful for unreachable code, including functions that do not return a value, such as test functions. Unfortunately mypy reports an error when using NoReturn, "Implicit return in function which does not return (misc)." To avoid headaches from the opaque "misc" category of mypy errors, these functions are annotated as returning None.
Mypy is used for type-checking. Mypy configuration is included in pyproject.toml.
Mypy strict mode is enabled. Strict includes --no-explicit-reexport (implicit_reexport = false), which means that objects imported into a module will not be re-exported for import into other modules. Imports can be made into explicit exports with the syntax from module import x as x (i.e., changing from import logging to import logging as logging), or by including imports in __all__. This explicit import syntax can be confusing. Another option is to apply mypy overrides to any modules that need to leverage implicit exports.

Spell check¶

Spell check is performed with CSpell. The CSpell command is included in the Hatch script for code quality checks (hatch run check).

GitHub Actions workflows¶

GitHub Actions is a continuous integration/continuous deployment (CI/CD) service that runs on GitHub repos. It replaces other services like Travis CI. Actions are grouped into workflows and stored in .github/workflows. See Getting the Gist of GitHub Actions for more info.

GitHub Actions and AWS¶

Static credentials¶

As explained in the section on generating credentials for local testing, a non-admin IAM user must be created in order to allow GitHub Actions to access AWS when using static credentials. The IAM user for this repo was created following IAM best practices. In AWS, there is a GitHubActions IAM group, with a fastenv IAM user (one user per repo). The fastenv user has an IAM policy attached specifying its permissions.

On GitHub, the fastenv user access key is stored in GitHub Secrets.

The bucket host is stored in GitHub Secrets in the "virtual-hosted-style" format (<bucketname>.s3.<region>.amazonaws.com).

Temporary credentials¶

In addition to the static access key, GitHub Actions also retrieves temporary security credentials from AWS using OpenID Connect (OIDC). See the GitHub docs for further info.

The OIDC infrastructure is provisioned with Terraform, using a similar approach to the example in br3ndonland/terraform-examples.

GitHub Actions and Backblaze B2¶

A B2 application key is stored in GitHub Secrets, along with the corresponding bucket host in "virtual-hosted-style" format (<bucket-name>.s3.<region-name>.backblazeb2.com).

See the Backblaze B2 S3-compatible API docs for further info.

GitHub Actions and Cloudflare R2¶

A Cloudflare S3 auth token (access key) is stored in GitHub Secrets, along with the corresponding bucket host in "virtual-hosted-style" format (https://<BUCKET>.<ACCOUNT_ID>.r2.cloudflarestorage.com).

See the Cloudflare R2 docs for further info.

Maintainers¶

Merges¶

The default branch is develop.
PRs should be merged into develop. Head branches are deleted automatically after PRs are merged.
The only merges to main should be fast-forward merges from develop.
Branch protection is enabled on develop and main.
- develop:
  - Require signed commits
  - Include administrators
  - Allow force pushes
- main:
  - Require signed commits
  - Include administrators
  - Do not allow force pushes
  - Require status checks to pass before merging (commits must have previously been pushed to develop and passed all checks)

Releases¶

To create a release:
- Bump the version number in fastenv.__version__ with hatch version and commit the changes to develop.
  - Follow SemVer guidelines when choosing a version number. Note that PEP 440 Python version specifiers and SemVer version specifiers differ, particularly with regard to specifying prereleases. Use syntax compatible with both.
  - The PEP 440 default (like 1.0.0a0) is different from SemVer. Hatch and PyPI will use this syntax by default.
  - An alternative form of the Python prerelease syntax permitted in PEP 440 (like 1.0.0-alpha.0) is compatible with SemVer, and this form should be used when tagging releases. As Hatch uses PEP 440 syntax by default, prerelease versions need to be written directly into fastenv.__version__.
  - Examples of acceptable tag names: 1.0.0, 1.0.0-alpha.0, 1.0.0-beta.1
- Push to develop and verify all CI checks pass.
- Fast-forward merge to main, push, and verify all CI checks pass.
- Create an annotated and signed Git tag.
  - List PRs and commits in the tag message:
```
git log --pretty=format:"- %s (%h)" \
  "$(git describe --abbrev=0 --tags)"..HEAD
```
  - Omit the leading v (use 1.0.0 instead of v1.0.0)
  - Example: git tag -a -s 1.0.0
- Push the tag. GitHub Actions will build and publish the Python package and update the changelog.
- Squash and merge the changelog PR, removing any Co-authored-by trailers before merging.

Deployments¶

Documentation is built with Material for MkDocs, deployed on Vercel, and available at fastenv.bws.bio and fastenv.vercel.app.

Vercel build configuration:

Build command: python3 -m pip install 'mkdocs-material>=9,<10' && mkdocs build --site-dir public. The version of mkdocs-material installed on Vercel is independent of the version listed in pyproject.toml. If the version of mkdocs-material is updated in pyproject.toml, it must also be updated in the Vercel build configuration.
Output directory: public (default)

Vercel site configuration is specified in vercel.json.