Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions .all-contributorsrc
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
{
"projectName": "data-module-template",
"projectOwner": "modelblocks-org",
"repoType": "github",
"repoHost": "https://github.com",
"files": [
"README.md"
],
"imageSize": 100,
"commit": false,
"commitConvention": "none",
"contributors": [],
"contributorsPerLine": 7,
"linkToUsage": false,
"contributorsSortAlphabetically": true
}
56 changes: 34 additions & 22 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,48 +1,47 @@
# Modelblocks data module template

A template for modular data workflows using [`snakemake`](https://snakemake.readthedocs.io/en/stable/) workflows, part of the [Modelblocks](https://www.modelblocks.org/) toolset.
A template for modular data workflows built with [`Snakemake`](https://snakemake.readthedocs.io/en/stable/). This template is part of the [Modelblocks](https://www.modelblocks.org/) toolset.

> [!TIP]
> Looking for general information on Modelblocks?
> Check the [Modelblocks](https://www.modelblocks.org/) website and our [documentation and guidelines](https://modelblocks.readthedocs.io/en).

## Resources
## Learning resources

To familiarise yourself with Modelblocks data modules:
Looking for general information on Modelblocks or how data modules work?

- Visit the [Modelblocks](https://www.modelblocks.org/) website and read our [documentation and guidelines](https://modelblocks.readthedocs.io/en).
- Check the auto-generated minimal example. You can find it in `tests/integration/Snakefile`.
- Read about `snakemake` modularisation in [their documentation](https://snakemake.readthedocs.io/en/stable/snakefiles/modularization.html#modules).
- Read about `Snakemake` modularisation in the [`Snakemake` documentation](https://snakemake.readthedocs.io/en/stable/snakefiles/modularization.html#modules).

## Features

- Standardised layout compliant with the [snakemake workflow catalogue's](https://snakemake.github.io/snakemake-workflow-catalog/#) listing requirements, enabling them to be automatically included in their listings once published. Read more about those requirements [here](https://snakemake.github.io/snakemake-workflow-catalog/docs/catalog.html#standardized-usage-workflows).
- Standardised layout compliant with the [`Snakemake` workflow catalogue](https://snakemake.github.io/snakemake-workflow-catalog/#) listing requirements, so modules can be included automatically once published. Read more about those requirements [here](https://snakemake.github.io/snakemake-workflow-catalog/docs/catalog.html#standardized-usage-workflows).
- Standardised input-output structure across modules:
- `resources/`: files needed for the module's processes.
- `user/`: files that should be provided by users. Document them well!
- `automatic/`: files that the module downloads or prepares in intermediate steps.
- `results/`: files generated by the module's algorithms that are relevant to the user.
- Pre-made integration setup for your module.
- Preconfigured integration setup for your module.
- Continuous Integration (CI) settings, ready for [pre-commit.ci](https://pre-commit.ci/).
- GitHub actions to automate chores during pull requests and releases.
- Premade `pytest` setup.
- Contributor recognition via [All Contributors](https://allcontributors.org/en/).
- GitHub Actions to automate chores during pull requests and releases.
- Pre-made `pytest` setup.

> [!IMPORTANT]
>
> A few things to be aware of.
> Keep these points in mind.
>
> - **Modules do not work like regular snakemake workflows**
> - The primary way to test them should be external (calling `module:`, passing resources, and requesting results). Check the pre-made example in `tests/integration` for more info.
> - Internal access (e.g., calling the `all:` rule) may not work, as the module may not have the necessary `resources/` to execute properly.
> - **Modules do not work like regular `Snakemake` workflows**
> - They must be tested externally using the `module:` command in `Snakemake`, passing user resources, and requesting specific results.
Check the pre-made example in `tests/integration` for details.
> - Internal access (e.g., calling the `all:` rule) is discouraged, as the module may not have the necessary `resources/` to execute properly.
> - **Please be sure to maintain the following files to ensure Modelblocks compatibility**
> - `INTERFACE.yaml`: a simple description of the module's input/output structure.
> - `config/config.yaml`: a basic functioning example of how to configure this module.
> - `workflow/internal/config.schema.yaml`: the module's configuration schema, used by `snakemake` for [validation](https://snakemake.readthedocs.io/en/stable/snakefiles/configuration.html).
> - `workflow/internal/config.schema.yaml`: the module's configuration schema, used by `Snakemake` for [validation](https://snakemake.readthedocs.io/en/stable/snakefiles/configuration.html).
> - `AUTHORS` / `CITATION.cff` / `LICENSE`: licensing and attribution of this module's code and methods.


## How to use this template

We require [`pixi`](https://pixi.sh/) as a package manager. Once installed, do the following:
This template uses [`pixi`](https://pixi.sh/) as its package manager. Once installed, do the following:

1. Install the templater tool [`copier`](https://copier.readthedocs.io/en/stable/).

Expand All @@ -59,19 +58,32 @@ We require [`pixi`](https://pixi.sh/) as a package manager. Once installed, do t
```

> [!TIP]
> If your terminal does not have access to `copier` then you may need to update your `PATH` variable to include `~/.pixi/bin`.
> If `copier` is not available in your terminal, you may need to update your `PATH` variable to include `~/.pixi/bin`.

3. Answer some questions so can we pre-fill licensing, citation files, etc...
3. Answer a few questions so we can pre-fill licensing, citation files, etc.
4. Initialise the `pixi` project environment of your new module.

```shell
cd ./<module_name> # navigate to the new project
pixi install --all # install the project environment
```

5. Extra: run the auto-generated example module!
5. Register your project in [pre-commit.ci](https://pre-commit.ci/) and [allcontributors.org](https://allcontributors.org/en/) to benefit from CI and contributor task automation.
6. Extra: run the auto-generated example module!

```shell
cd tests/integration # go to the integration test...
pixi run snakemake --use-conda # run it!
```

## Contributors ✨

Thanks goes to these wonderful people, sorted alphabetically ([emoji key](https://allcontributors.org/en/reference/emoji-key/)):

<!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section -->
<!-- prettier-ignore-start -->
<!-- markdownlint-disable -->
<!-- markdownlint-restore -->
<!-- prettier-ignore-end -->
<!-- ALL-CONTRIBUTORS-LIST:END -->

This project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome!
16 changes: 16 additions & 0 deletions template/.all-contributorsrc.jinja
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
{
"projectName": "{{module_short_name}}",
"projectOwner": "{{github_org}}",
"repoType": "github",
"repoHost": "https://github.com",
"files": [
"README.md"
],
"imageSize": 100,
"commit": false,
"commitConvention": "none",
"contributors": [],
"contributorsPerLine": 7,
"linkToUsage": false,
"contributorsSortAlphabetically": true
}
13 changes: 13 additions & 0 deletions template/README.md.jinja
Original file line number Diff line number Diff line change
Expand Up @@ -65,3 +65,16 @@ This module is based on the following research and datasets:

*
*

## Contributors ✨

Thanks goes to these wonderful people, sorted alphabetically ([emoji key](https://allcontributors.org/en/reference/emoji-key/)):

<!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section -->
<!-- prettier-ignore-start -->
<!-- markdownlint-disable -->
<!-- markdownlint-restore -->
<!-- prettier-ignore-end -->
<!-- ALL-CONTRIBUTORS-LIST:END -->

This project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome!
Loading