So far in this tutorial, we've covered the basic commands for managing individual packages:
- spack install to install packages
- spack uninstall to remove them
- spack find to view and query installed packages
Now we'll explore Spack environments, a powerful feature that lets us manage collections of packages together in a documented and reproducible way. Spack environments are similar to virtual environments in other package managers (e.g., Python venv, Conda environments, or nix-env).
Managing a software stack with many packages and varying configuration can quickly become hard to track by hand. An environment lets you:
- Establish a standard set of requirements for a project,
- Install and use them together as a unit,
- Reproduce the same build on another machine, and
- Document your software stack for collaborators, CI/CD pipelines, and releases.
Each environment is captured in two shareable files: spack.yaml records the requested specs and configuration, while spack.lock captures the fully concrete build for reproducibility.
We'll look at both in detail later in this tutorial.
Let's look at the output of spack find at this point in the tutorial.
.. literalinclude:: outputs/environments/find-no-env-1.out :language: console
This is a complete, but cluttered list of all our installed packages and their dependencies.
It contains packages built with both openmpi and mpich, as well as multiple variants of other packages, like hdf5 and zlib-ng.
The query mechanism we learned about with spack find can help, but it would be nice if we could see only the software that is relevant to our current project instead of seeing everything on the machine.
Let's create a new environment called myproject using the spack env create command:
.. literalinclude:: outputs/environments/env-create-1.out :language: console
To see all of our environments we've created so far we can run spack env list:
.. literalinclude:: outputs/environments/env-list-1.out :language: console
Now let's activate our environment by running the spack env activate command or spacktivate alias.
The -p flag prepends the environment name to our shell prompt, which is a handy reminder of which environment we are in:
.. literalinclude:: outputs/environments/env-activate-1.out :language: console
Note
For quick experiments you don't intend to keep, spack env activate --temp creates and activates a fresh, unnamed environment in a temporary directory.
We'll use a regular named environment throughout this tutorial, but --temp is handy when you just want to try something out.
Once we activate an environment, spack find will only show what is in the current environment.
For example, because we just created this environment the output below doesn't show any installed packages.
.. literalinclude:: outputs/environments/find-env-1.out :language: console
Although Spack doesn't show all installed software when in an active environment, it will reuse packages across environments to save disk space and reduce build times.
Additionally the output now tells us that we're in the myproject environment, so there is no need to panic when we no longer see our previously installed packages.
It also states that there are no root specs.
We'll get back to what that means later.
While this detailed output is useful, if we only want to check what environment we are in, we can use spack env status:
.. literalinclude:: outputs/environments/env-status-1.out :language: console
To now exit out of this environment, we can use spack env deactivate or despacktivate if we're feeling fancy.
After deactivating, we can see everything installed in this Spack instance:
.. literalinclude:: outputs/environments/env-status-2.out :language: console
Now that we understand how creation and activation work, let's go back to our myproject environment and install a couple of packages, specifically tcl and trilinos.
Installing software in an environment follows three steps:
- add the specs we want, registering them as root specs of the environment;
- concretize the environment, resolving every root and its dependencies into a concrete set; and
- install the concretized specs.
We'll walk through them one at a time.
First, we add our two specs with spack add:
.. literalinclude:: outputs/environments/env-add-1.out :language: console
tcl and trilinos are now registered as root specs, i.e., the packages we've explicitly requested.
They're called "roots" because they sit at the top of the dependency graph, with their dependency packages sitting below them.
If we run spack find now, it lists them as roots but reports nothing concretized yet.
Adding a spec only records our intent:
.. literalinclude:: outputs/environments/env-add-find-1.out :language: console
Next, we concretize the environment, resolving those roots and all their dependencies into a concrete set:
.. literalinclude:: outputs/environments/env-concretize-1.out :language: console
The spack find above hinted at this with show with spack find -c.
Now that the environment is concretized, that command lists the concrete set, including the packages still to be installed:
.. literalinclude:: outputs/environments/env-find-c-1.out :language: console
Finally, we install:
.. literalinclude:: outputs/environments/env-install-1.out :language: console
We can see that Spack reused existing installations of tcl and the dependencies of trilinos that were already present on the system, rather than rebuilding them from scratch.
Additionally, the environment's view was automatically updated to include the installations.
This means all the software in this environment has been added to our PATH, making the installed packages readily accessible from the command line while we have the environment activated.
Let's confirm the contents of the environment using spack find:
.. literalinclude:: outputs/environments/find-env-2.out :language: console
We can see that the roots and their dependencies have been installed. The packages reported as concretized packages to be installed are build-only dependencies, which Spack skips when it installs from binaries.
Note
We walked through the three steps separately, but in practice spack install concretizes for you, so spack add followed by spack install is all you need.
You must still add a spec before installing it, which prevents you from accidentally changing a shared environment.
Adding all specs before installing also lets Spack concretize them together, so they share dependencies and can all build in parallel.
Activating an environment adds all of its installed software to our shell search paths through a feature called environment views.
A view is a directory tree of symlinks to all installed packages, structured like a standard Linux filesystem (bin/, lib/, include/, ...).
For example, tcl ships a shell called tclsh.
Activating myproject puts it on our PATH:
.. literalinclude:: outputs/environments/use-tcl-1.out :language: console
Notice the path includes the environment name and a view subdirectory.
With openmpi installed in myproject as a dependency of trilinos, mpicc is also on our PATH:
.. literalinclude:: outputs/environments/show-mpicc-1.out :language: console
This means we can compile code against the environment's packages without any manual -I or -L flags.
Let's build a small program that uses both MPI and zlib:
#include <stdio.h>
#include <mpi.h>
#include <zlib.h>
int main(int argc, char **argv) {
int rank;
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
printf("Hello world from rank %d\n", rank);
if (rank == 0) {
printf("zlib version: %s\n", ZLIB_VERSION);
printf("zlib-ng version: %s\n", ZLIBNG_VERSION);
}
MPI_Finalize();
}.. literalinclude:: outputs/environments/use-mpi-1.out :language: console
If we look at the full set of path variables set by activation:
.. literalinclude:: outputs/environments/show-paths-1.out :language: console
we find CMAKE_PREFIX_PATH and PKG_CONFIG_PATH too, so build tools find the environment's libraries and headers automatically.
Spack offers two ways to get rid of a package in an environment:
spack removetakes a spec out of the environmentspack uninstalldeletes the installed files from disk
In a setup with several environments these behave quite differently, so let's create a second environment to see how.
We'll create myproject2, activate it, and add both scr and trilinos:
.. literalinclude:: outputs/environments/env-create-2.out :language: console
then concretize and install as usual:
$ spack concretize
$ spack installspack find lets us confirm the environment now holds both roots and their dependencies:
.. literalinclude:: outputs/environments/env-create-2-find.out :language: console
So myproject contains tcl and trilinos, while myproject2 contains scr and trilinos.
Let's start with the simplest case.
scr is used only by myproject2, so we can remove it cleanly.
spack remove is the counterpart to spack add: it drops scr as a root spec.
.. literalinclude:: outputs/environments/env-remove-scr-1.out :language: console
That only updates the environment's configuration, though.
Right afterward, spack find still lists scr.
It is no longer a root, but it is still part of the environment:
.. literalinclude:: outputs/environments/env-remove-scr-2.out :language: console
To actually drop it, we reconcretize the environment:
.. literalinclude:: outputs/environments/env-remove-scr-3.out :language: console
Now scr, along with any dependencies nothing else needs, has been pruned, and spack find no longer lists it:
.. literalinclude:: outputs/environments/env-remove-scr-4.out :language: console
trilinos is a more interesting case, because myproject uses it too.
Environments only point to shared installations rather than owning the files, so Spack won't let us delete one out from under another environment.
If we try to uninstall trilinos from myproject2, Spack refuses, since myproject still references it:
.. literalinclude:: outputs/environments/env-uninstall-1.out :language: console
As the error suggests, spack remove is the right tool: it takes trilinos out of myproject2 without touching the shared installation.
.. literalinclude:: outputs/environments/env-remove-1.out :language: console
We know trilinos is still needed by myproject, so let's switch back and confirm it's still there:
.. literalinclude:: outputs/environments/env-swap-1.out :language: console
myproject still has trilinos as a root spec.
We have been treating an environment as an abstraction, but it is really just a directory on disk.
spack cd -e takes us to myproject's directory:
.. literalinclude:: outputs/environments/filenames-1.out :language: console
Because we created myproject with spack env create <name>, it lives under var/spack/environments inside the Spack installation.
This is what makes it a managed environment: we can refer to it by name.
Environments can also be independent, with their files placed in any directory of our choosing.
The directory holds the two files that define the environment:
spack.yaml: the manifest file containing the abstract specs we asked for plus configuration settings.spack.lock: the lockfile containing the concrete specs generated whenever the environment is concretized.
spack.yaml is the human-readable file we have been editing indirectly with spack add and spack remove:
.. literalinclude:: outputs/environments/cat-config-1.out :language: spec
The concretizer:unify: true setting ensures all specs in the environment share a single version of each dependency.
spack.lock records the fully concretized environment (every package, version, variant, and dependency) so the build can be reproduced exactly.
It is machine-readable JSON:
.. literalinclude:: outputs/environments/lockfile-1.out :language: console
The full file runs to thousands of lines.
A hidden .spack-env directory (not shown by ls) sits alongside these files and holds Spack's bookkeeping, including the view we used earlier.
Managed environments also appear by name in spack env list, with the active one highlighted:
.. literalinclude:: outputs/environments/env-list-2.out :language: console
We will come back to both files when we reproduce environments later.
We've seen spack.yaml contains the root specs of the environment.
It also controls configuration settings that shape how Spack behaves when the environment is used.
Let's edit spack.yaml to require mpich as our mpi provider using spack config edit (make sure EDITOR is set first).
You should now have spack.yaml open in your editor.
Change it to include the packages:mpi:require entry below:
# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
packages:
mpi:
require: [mpich]
# add package specs to the `specs` list
specs:
- tcl
- trilinosThe environment is now out of sync: myproject was installed with openmpi, but the configuration now requires mpich.
If we run spack concretize, Spack sees the existing concretization is still valid and does nothing:
.. literalinclude:: outputs/environments/concretize-1.out :language: console
To force Spack to re-concretize from scratch, use spack concretize --force:
.. literalinclude:: outputs/environments/concretize-f-1.out :language: console
All the specs are now concrete with mpich as the MPI implementation, ready to be installed.
Next you'd run spack install to rebuild the environment with the new MPI provider.
Note
In general, when building an environment it is also possible to add and install specs one at a time, but Spack will then concretize each new spec greedily against the versions already locked in. This can lead to a different (and sometimes less optimal) result than concretizing everything together.
The spack.yaml and spack.lock files we have been working with are not just a record of the environment: they are also the mechanism for sharing it.
You can commit them to version control, hand them to a colleague, or drop them into a CI/CD pipeline.
Anyone with a Spack installation can recreate the environment from them.
To demonstrate, we'll play the role of the recipient of a shared environment.
Let's build a fresh independent environment in a new code/ directory, which stands in for a project a colleague has shared with us:
.. literalinclude:: outputs/environments/independent-create-1.out :language: console
Since the environment is independent, it must be activated by path, not by name.
Edit the spack.yaml created in the directory to add trilinos and openmpi:
# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
# add package specs to the `specs` list
specs:
- trilinos
- openmpi
view: true
concretizer:
unify: true.. literalinclude:: outputs/environments/install-independent-1.out :language: console
Passing either of these files to spack env create recreates the environment.
The spack.yaml file preserves the abstract requirements but allows the concretizer to adapt to a new platform.
The spack.lock file fixes every concretization decision and reproduces the build exactly.
Recreating from spack.yaml preserves the root specs but leaves dependency resolution to the concretizer, which may produce different versions on a new platform or as packages are updated:
.. literalinclude:: outputs/environments/create-from-file-1.out :language: console
The new environment has no installed packages yet, only the root specs from the file are registered:
.. literalinclude:: outputs/environments/find-env-abstract-1.out :language: console
Recreating from spack.lock reproduces the exact build, and thus requires the same machine architecture.
All packages recorded in the lockfile are installed without a separate install step:
.. literalinclude:: outputs/environments/create-from-file-2.out :language: console
Since the environment was created from spack.lock, the root specs and all their dependencies are already installed:
.. literalinclude:: outputs/environments/find-env-concrete-1.out :language: console
Let's clean up the environments we created during the tutorial. First, deactivate the current environment:
$ spack env deactivateThen remove all the managed environments at once:
$ spack env rm myproject myproject2 abstract concreteThis tutorial only scratches the surface of environments and what they can do. For more information, take a look at the Spack resources below.
- Environments: reference docs
- Configuration tutorial: for customizing your environment
- Spack stacks tutorial: for configuring combinatorial environments
- Install-level parallel builds: for how to launch
spack installto build your environment in parallel
- Developer workflows: for developing code in an environment
- GitLab CI pipelines with Spack environments: for using environments to generate CI pipelines
- Container Images: for creating containers from environments
- Spack Stack Catalog: for discovering environments that you can explore on GitHub