diff --git a/docs/release.qmd b/docs/release.qmd index 61f2347..a35da6a 100644 --- a/docs/release.qmd +++ b/docs/release.qmd @@ -1 +1,220 @@ # Release process {#sec-release} + +A release is a snapshot of the data package at a specific point in time. It +contains the data and metadata in their final state at that point in time and is +tagged with a specific version number. Releases are how we track changes over +time and make it easier to share or distribute data packages to researchers in a +structured and predictable way. + +The release process is dependent on the type of data contained in the data +package. For data packages that contain human (in particular health or +sensitive) data, the release process is a bit more complicated, as will be +explained later, than for data packages that contain data that doesn't fall +under legal restrictions (e.g. GDPR). This is because the data must remain on +secure servers and can't (usually) be uploaded to any public repository or +archive. That means we can't use services like GitHub workflows nor upload any +data for public access, so we can't use a continuous release process (or at +least it is more difficult to). + + + +For nonsensitive data, the release process is fairly simple. It can be done +through a GitHub workflow (e.g. `release.yml`) that runs on a schedule or is +triggered by specific events (e.g. a specific type of commit, described later in +this chapter). The data and metadata can be attached as release artifacts on +GitHub and/or be uploaded to public archives like [Zenodo](https://zenodo.org/). + +Regardless of the type of data, the general steps remain mostly the same with +some notable differences. The difference is in *where* the release is done (the +computing environment where the release process is executed, e.g. GitHub or on a +secure server) and *what triggers* a release. We'll start with the triggers. + +## Triggers + +There are a few ways to trigger the release process: manually, on a schedule, or +based on specific events whenever a change is merged into `main`. While we aim +to practice [continuous +delivery](https://en.wikipedia.org/wiki/Continuous_delivery) when we're able to, +this isn't always the case for more complicated situations. For data that falls +under legal regulations, you likely will need to use either a manual process or +a schedule-based one by running a [cron job](https://en.wikipedia.org/wiki/Cron) +on the server. For non-private data, you can use the merge/push-based process by +using a GitHub workflow. + +Regardless of the underlying trigger, the actual release process is the same: a +release is created based on specific text within the commit messages. Using +commit messages to determine a release is called [semantic +release](https://decisions.seedcase-project.org/why-semantic-release-with-cocogitto/), +which uses [Semantic +Versioning](https://decisions.seedcase-project.org/why-semver/index.html) and +[Conventional +Commits](https://decisions.seedcase-project.org/why-conventional-commits/) as +its foundation. + +### Commits + +In order to determine whether changes should result in a release, commits must +follow [Conventional +Commits](https://decisions.seedcase-project.org/why-conventional-commits/) to +structure the commit messages. The structure of a commit message looks like +this: + +```text +(optional scope): + +[optional body] + +[optional footer(s)] +``` + +The two main components of this structure for determining releases are the +"type" and the "footer". The "type" is the first part of the commit message and +is used to determine what type of change has been made. The "footer" is the last +part of the commit message and is used to include a `BREAKING CHANGE` note if +the change is a breaking change, i.e., a change that disrupt the compatibility +of existing code with newer versions. A breaking change can also be indicated by +appending a "!" to the commit "type". These two components determine which +version to set for the release. + +Semantic versions are made up of three numbers: `MAJOR.MINOR.PATCH`, e.g., +`0.1.2`. The `MAJOR` version is incremented when there are breaking changes, the +`MINOR` version is incremented when new features are added, and the `PATCH` +version is incremented when fixes are made. For semantic releases, the commit +"type" `feat` increases the `MINOR` version, while the commit "type" `fix`, +`refactor`, or `perf` increases the `PATCH`. If there is a breaking change, with +either `BREAKING CHANGE` in the footer or `!` in the commit message, then +the `MAJOR` version is increased. + +But how do you know which commit type to use? Unlike software development, +developing data packages is quite different and it can be a bit more difficult +to determine what a "feature", "fix", or "breaking change" is. To help determine +the commit "type", we use aspects of [Data Package's semantic +versioning](https://datapackage.org/recipes/data-package-version/) guide. + +Breaking changes with the `!` or `BREAKING CHANGE` in the footer format +*must only* happen after the first stable release of the data package. The first +stable release is defined as when the data package has all expected or planned +resources, metadata has been completed, and all "observational units" (e.g. +participant or animal) have been measured. Essentially, when the planned study +has been completed. Before that point, only `MINOR` and `PATCH` changes are +allowed. This means that the version will remain at `0.MINOR.PATCH` until the +stable release. Once a stable release has been made, a breaking change would +occur if you: + +- Change the data package, resource, or column name or identifier. +- Remove a resource or column from the data package. +- Move a column into another resource. +- Change a column type (e.g. from integer to string). +- Change a column's constraints to be more restrictive (e.g. reduce the distance + between the minimum and maximum values). +- Remove a participant's data (e.g. after they request their data be deleted). +- Substantially change the meaning of the text in the metadata (e.g. a column's + description or a resource's title). + +A good guideline to use for `MINOR` (`feat`) commits would be if something *new* +has been added or expanded on. Minor changes with the `feat` format would be if +you: + +- Add a new resource. +- Add data, either new rows or columns to an existing resource. +- Change a column's constraints to be less restrictive (e.g. increase the + distance between the minimum and maximum values). +- Add new text to the metadata, for example, when no metadata existed before, + but not correcting existing metadata (see `PATCH` below). + +A good guideline to use for `PATCH` (`fix`, `refactor`, or `perf`) commits would +be if something has been *corrected* or *refined*. Before the stable release, +many of the breaking change items above would be considered a patch change, as +they generally don't add any new content. Patch changes with the `fix`, +`refactor`, or `perf` commit type would be if you: + +- Correct errors in existing data, like a typo or data entry error. Depending on + the severity of the error, this could also be a breaking change. +- Change the text of the metadata without changing the meaning, for example + fixing typos, grammatical errors, or clarifying the text without changing its + meaning. +- Changes to how the data is processed so that it results in better compression + or other performance improvements. + +Any other commit types can be used, but they won't result in a releasable +change. For example, before any metadata or resources have been created as the +pipeline is being developed, you could use the `chore` commit type to indicate +that you are working on the pipeline, but no data or metadata has been created +yet. + +## Steps + +Now that we've covered the triggers, let's go over the actual steps involved in +the release process, whether it is manual, schedule-based, or merge/push-based. +[Cocogitto](https://decisions.seedcase-project.org/why-semantic-release-with-cocogitto/) +manages all these steps via the `cog.toml` file. + +The release process runs the following steps to check and potentially update the +release number: + +1. Check the commit history since the last release for any releasable changes. + If no releasable changes are found, then no release is created. Otherwise, + the process continues. + +2. Update the version based on the commit message and update the version in the + `pyproject.toml` file using + [`uv version`](https://docs.astral.sh/uv/reference/cli/#uv-version). If the + metadata format is `datapackage.json`, the version field uses the version in + `pyproject.toml` and will be updated automatically when the + `datapackage.json` file is (re)generated. + +3. Run the build process from start to end, excluding the step to pull from the + sources and saving to `raw/`. Raw data is only saved intentionally as a + separate commit/pull request. This is described in the build process in + @sec-build. The main artifacts of the build process are the + `_.tar` file for sensitive and nonsensitive data and + the `_.zip` for an artifact that can be publicly + uploaded when the data is sensitive. + +4. Generate the changelog based on the commit messages since the last release. + [git-cliff](https://decisions.seedcase-project.org/why-changelog-with-git-cliff/) + is used to generate the changelog. + +5. Commit the changes that were made in the `CHANGELOG.md` file and the metadata + file (e.g. `datapackage.json`), then create a tag for the new version on that + commit. No data in `raw/` is committed. Push the commit and the tag to + GitHub. + +6. Create a new GitHub release on GitHub from the new tag and changelog. Attach + the build artifacts to the release. For nonsensitive data and that is smaller + than 2 Gb (GitHub's limit), the `.tar` file is attached to the release. For + sensitive data or for data that is larger than 2 Gb, the `.zip` file is used + instead. Either way, the file is renamed to simply `.zip` (or + `.tar`), as the tag itself contains the version number. + +7. For nonsensitive data and that is smaller than 50 Gb (the limit for Zenodo), + upload the `.tar` file to Zenodo. For sensitive data or data + that is larger than 50 Gb, upload the `.zip` file instead. + +::: callout-important +For data that is larger than GitHub's 2 Gb limit or Zenodo's 50 Gb limit, the +best place to store the released `.tar` file is in your server in the +`releases/` folder. You'll likely need to delete older releases if space becomes +an issue on the server. Deleting older releases is fine, as the release process +is designed to be reproducible, so you can always recreate the release by +switching to the tagged commit and running the build process again. +::: + +## Practical considerations + +As you develop a data package, there are a few things to keep in mind in order +to make the release process easier. + +- We consider the first, non-stable (< 1.0.0) release to happen once there is + code that takes the first resource and its metadata from raw format into its + final resource state. The code must also be integrated into the `build.py` + file, so that the release can be reproducible. + +- Whenever you make a change, either directly to main or through a pull request, + you *always* need to make sure commits and pull requests are + [atomic](https://decisions.seedcase-project.org/why-atomic-commits-and-prs/). + This means that each commit or pull request contains only one *conceptual* + change. That's because the commit message (and consequently the changed files + in the commit) determine what type of release will be created. The commit + message will also be added to the changelog, so be aware of the message you + use.