diff --git a/docs/build.qmd b/docs/build.qmd index 44b1bee..1614783 100644 --- a/docs/build.qmd +++ b/docs/build.qmd @@ -1 +1,210 @@ -# Build process +# Build process {#sec-build} + +Like software packages (e.g. in Rust, Python, R), the Git repository contains +the source of the package rather than the package itself. It includes the raw +input data and the code needed to process it into the final format. But before +the data package can be distributed, it must first be "built" (just like code is +"compiled") by running the code to transform the raw data inputs to the final +packaged format. + +At a high level, the directories and files involved in the build process are +(non-relevant files and directories are removed): + +``` +/ +├── .github/ +│ └── workflows/ +│ └── release.yml # Optional, depends on legal requirements +├── raw/ +├── staging/ +├── resources/ +├── releases/ +├── src/ +│ └── / +│ ├── __init__.py +│ ├── build_raw.py +│ └── build_staging.py +├── . +├── CHANGELOG.md +├── README.md +├── LICENSE.md +└── +``` + +::: callout-note +Why do we "build" a data package? + +1. To keep a separation between the source code and raw input data contained + within the Git repository and the final data package that is released. +2. To ensure a reproducible build from input to output. +3. To keep a history of the changes made to the final data and to version the + data package for easier tracking of changes and updates. +4. To treat the data package as a formal "product" and apply rigorous and robust + software engineering and data engineering practices to its development and + release. +::: + +## Steps + +The initial steps, from the source of the data to staging, is managed by a +pipeline management tool, which should have its pipeline defined in +`src//build_*.py` (though some pipeline tools require a different +filename). We recommend using +[pytask](https://pytask-dev.readthedocs.io/en/stable/), which we've found works +well for developing data packages (for more details, see our [pytask for Python +workflow management](https://decisions.seedcase-project.org/why-pytask/) +decision post). All steps must be defined within this tool, from pulling the +source data to `raw/`, to processing the raw data into `staging/`, and to +processing or extracting any metadata obtained from the source or staged data. +Because data saved into `raw/` is committed in separate pull requests (described +more below and in @sec-release), there should be two build steps: downloading +data from source locations to `raw/` and taking the data in `raw/` and +processing it into `staging/`. We suggest calling these files `build_raw.py` and +`build_staging.py`, so that they can be run independently of each other. + +The steps after `staging/` are managed mainly by +[Sprout](https://sprout.seedcase-project.org/), which is our tool to handle the +processing from `staging/` to `resources/` and for (re-)generating the metadata +file. + +All the steps for building the data package into `.tar` and `.zip` files are +handled through the [`justfile`](https://just.systems/) file, which would be +(one of) the `` in the above directory structure. The `justfile`, +if you use our [template data +package](https://template-data-package.seedcase-project.org/), contains recipes +for each step of the build process. + +If the data package is based on open data or on data not covered by legal +restrictions (e.g. GDPR), the build and release process can be automated through +a GitHub Actions workflow defined in `.github/workflows/release.yml`. Otherwise, +the process needs to be done manually or through a scheduled workflow on a +secure server. We cover this more in @sec-release. + +```{mermaid} +%%| label: fig-build-process +%%| fig-cap: "The process for building a data package." +%%| fig-alt: A flowchart showing the process for building a data package. The +%%| process starts with the data from the source being pulled into the `raw/` +%%| directory. The data in `raw/` is then processed into `staging/`. The data +%%| in `staging/` is then joined into `resources/`. The metadata file is +%%| optionally regenerated from the code and checked against the data in +%%| `staging/`. Finally, the metadata and resources are bundled into a `.tar` +%%| and `.zip` file in `releases/`. + +flowchart TB + Source["source
(e.g., API
or database)"] -->|pull| Raw["raw/"] + Raw -->|process| Staging["staging/"] + Staging -->|join| Resources["resources/"] + Resources + Metadata["metadata
(Optionally
regenerated)"] + Metadata ---|check| Staging + Metadata & Resources -->|bundle| Build["releases/
.tar and .zip"] +``` + +::: callout-note +We plan on building a tool to automate and simplify the build steps, but for +now, you can use the structure and tools that we've set up in our [Template Data +Package](https://template-data-package.seedcase-project.org/). See @sec-setup +for details about that. +::: + +::: callout-important +It's important to note here that only the data in `raw/` and the metadata file +are saved into [Git LFS](https://git-lfs.github.com/) during the release +process. No other data artifacts or files are saved in the Git history. This is +described in more detail in @sec-release. +::: + +### Pull from source to raw + +To make sure we're getting the most up-to-date data, the first build step is to +pull from the various data sources into `raw/`. See @sec-raw and @sec-src for +details on how the raw data is organised and how the source code pulls the data +via the pipeline tool. The data is *not* processed in any way when it is pulled +from the source into `raw/`. + +### Raw to staging + +After pulling the source data into the `raw/` directory, the data (usually) +needs to be processed in some way. In particular, the data likely needs to be +reorganised so that it ends up in the correct resource location. The data in +`raw/` should be processed into `staging/` in a one-to-one mapping, so that +every `.` file in `raw/` is processed into a +`.parquet` file in `staging/`. This enables the potential for +parallel processing and to maintain a sequence between the two directories. Any +data in `staging/` must also match the contents of the metadata format. This is +all done to ensure that the data gets into the final resource correctly and +without issues. See @sec-staging, @sec-metadata, and @sec-src for more details +`staging/`. + +### Staging to resources + +At this stage, [Sprout](https://sprout.seedcase-project.org/) can take over and +process the data from `staging/` into `resources/`. All the timestamped files in +the individual `staging/` resource directories are processed and joined into a +single resource file in `resources/`. During processing, the data is checked +against the metadata. See @sec-resources, @sec-metadata, and [Sprout's +documentation](https://sprout.seedcase-project.org/) for more details on this +step. + +### Metadata (re-)generation (optional) + +For data packages using `datapackage.json` as the metadata format, Sprout will +also (re-)generate the file from the Python code. See @sec-metadata and +[Sprout's documentation](https://sprout.seedcase-project.org/) for a description +of this section. + +### Build artifacts + +At the end, the data package is built into one `_.tar` +file that contains the metadata file (e.g. `datapackage.json`), `LICENSE.md`, +`README.md`, `CHANGELOG.md`, and the resource files. If the data contains human +(especially health) data and if it is required to be on secure servers, an +additional `_.zip` file is also built with the same files +but **does not include** the data. This `.zip` file can be uploaded to public +archives to generate, for example, a [DOI](https://doi.org/) on +[Zenodo](https://zenodo.org), while the `.tar` file can remain on the server. +The `.tar` and `.zip` files are saved into a `releases/` directory ignored by +Git. + +## Development practices + +How does this build process impact developing the data package and the +development practices? Because of this formal build process, there are a few +things to be aware of. + +First, you should only save raw data in the Git history within the Git LFS +(after pulling it from the sources) during an intentional update. A pull request +(or commit) should only contain changes to `raw/` in that commit, meaning the +commit should be [atomic](https://en.wikipedia.org/wiki/Atomic_commit) as a data +update. You should do it this way because you need to know what has changed in +the raw data to be able to correctly use the right Conventional Commit type (see +@sec-release for more details). For example, if the raw data update has +completely new data, the type would be `feat`, but if it was correcting a data +entry error, it would be `fix`. + +Related to the above, outside of these intentional and atomic commits/pull +requests, you shouldn't commit the raw data to the Git history for any other +change, such as when changing the code. During these types of changes, treat any +data pulled from sources into `raw/` or processed into `staging/` or +`resources/` as temporary. + + + +Related to the above, if the metadata format is `datapackage.json`, pull +requests should *not* contain any changes to it. These files are generated +during the build process and should not be modified or added directly. For +`datapackage.json` based metadata, the metadata files within `src/` that contain +the metadata managed by [Sprout](https://sprout.seedcase-project.org/) should be +modified instead. For other metadata formats, the metadata file can be modified +directly and can be saved to the Git history. + +Commit messages should still be written in the Conventional Commits format, +though the specific commit types used are a bit different considering no data, +or metadata files if it is the `datapackage.json` format, are being modified or +saved directly. See @sec-release for more details on how Conventional Commits +are used in the release process and what commit messages to use. + +All files in `staging/`, `resources/`, and `releases/` should be ignored by Git +in the `.gitignore` file, aside from a `.gitkeep` or `README.md` file to keep +the directory structure.