ARD

ARD is a local development and test harness for VM-backed DevStack deployments. The primary local workflow uses Ansible provider playbooks to provision libvirt VMs, generate an ARD/Zuul-like inventory, and then run the existing DevStack deployment roles inside those VMs.

The primary local provider target is libvirt. KubeVirt and static are also provider paths; static targets pre-provisioned SSH hosts.

Docs navigation was reorganized; see:

docs/repo-navigation.md for repo layout and entry points
docs/README.md for the documentation index
docs/concepts/ard-render-model.md for user-facing render model concepts and examples
docs/providers/kubevirt-prerequisites.md for OpenShift/KubeVirt setup requirements
docs/architecture/render-contracts.md for contributor render/provider contracts
ansible/README.md for Ansible structure and canonical playbook paths

Quick start

Bootstrap the repository and host dependencies:

./bootstrap-repo.sh

Render a named deployment workspace:

make render ARD_DEPLOYMENT=devstack-a

Create the VMs and generated inventory:

make apply ARD_DEPLOYMENT=devstack-a
make ping ARD_DEPLOYMENT=devstack-a

SSH to a deployment node, or print the SSH command without running it:

make ssh ARD_DEPLOYMENT=devstack-a ARD_NODE=controller
make ssh-print ARD_DEPLOYMENT=devstack-a ARD_NODE=compute-1
# equivalent explicit dry-run/print mode
make ssh ARD_DEPLOYMENT=devstack-a ARD_NODE=compute-1 ARD_SSH_PRINT=1

For KubeVirt clusters where the Kubernetes API is reachable over VPN but VMs are not exposed with LoadBalancer services, OpenSSH can use virtctl port-forward as a proxy. Place this before any broad Host * block in ~/.ssh/config so the VM-specific settings take precedence:

Host vm/* vmi/*
  ProxyCommand virtctl port-forward --stdio=true %h %p
  StrictHostKeyChecking accept-new
  LogLevel QUIET
  ConnectTimeout 10
  User stack
  IdentityFile ~/.ssh/id_ed25519_stack
  ForwardX11 no
  ForwardAgent yes
  ControlMaster no
  ControlPath none
  ServerAliveInterval 30
  ServerAliveCountMax 3
  Compression yes

Deploy DevStack:

make deploy ARD_DEPLOYMENT=devstack-a
make verify ARD_DEPLOYMENT=devstack-a

Destroy provider resources when finished:

make destroy ARD_DEPLOYMENT=devstack-a

Remove the local deployment workspace only when you no longer need the rendered inputs or generated state:

make cleanup ARD_DEPLOYMENT=devstack-a

Local deployment workflow

The normal local workflow is:

render: create a concrete deployment workspace from presets and optional overlays.
apply: create libvirt network/domain/disk/seed resources, generate inventory, and wait for SSH/cloud-init readiness.
ping: verify SSH/Ansible connectivity again.
deploy: run the multinode DevStack playbook.
verify: run basic post-deploy checks.
destroy: remove libvirt resources but keep the workspace and generated artifacts for inspection.
clean-generated: remove generated inventory/state/rendered artifacts without touching provider resources.
cleanup: delete the workspace.

A deployment workspace lives under:

deployments/<deployment-name>/
  render.yaml             # optional persistent render intent
  local-vars.yaml         # optional site/workload vars for lifecycle playbooks
  deployment.yaml         # generated by render
  nodes.yaml              # generated by render
  devstack/
    common.yaml
    group_vars/
      controller.yaml
      compute.yaml
    host_vars/
  inventory.yaml          # generated by apply
  provider-state.yaml     # generated by apply
  rendered/               # generated provider artifacts
  logs/

render treats deployment.yaml, nodes.yaml, and devstack/*.yaml as generated output and may overwrite them. Keep render intent in render.yaml, an external render file, or a deployment-local overlay such as overrides/render.yaml. Keep site-local or workload deployment variables in local-vars.yaml; Make automatically passes that file to lifecycle playbooks when it exists.

After a workspace has been rendered, later lifecycle commands only need the logical deployment name. The workspace directory remains deployments/<deployment> even when provider resources, such as KubeVirt VMs in a shared namespace, use a user-prefixed resource name. If the workspace contains render.yaml, make render ARD_DEPLOYMENT=<name> also reloads it automatically.

make render ARD_DEPLOYMENT=oko-cyborg
make apply ARD_DEPLOYMENT=oko-cyborg
make ping ARD_DEPLOYMENT=oko-cyborg
make deploy ARD_DEPLOYMENT=oko-cyborg
make destroy ARD_DEPLOYMENT=oko-cyborg

Static provider

The static provider runs workloads on pre-provisioned SSH hosts. A static render intent supplies logical inventory nodes and their SSH transport details; make apply discovers the host addresses used by workload services.

A complete CentOS Stream 10 AIO example with Cyborg and the pci-sim kernel module is available under examples/devstack/static-cyborg-pci-sim/:

make render \
  ARD_DEPLOYMENT=static-cyborg-pci-sim \
  ARD_RENDER_FILE=examples/devstack/static-cyborg-pci-sim/render.yaml
make apply ARD_DEPLOYMENT=static-cyborg-pci-sim
make ping ARD_DEPLOYMENT=static-cyborg-pci-sim
make deploy ARD_DEPLOYMENT=static-cyborg-pci-sim

The host must already be reachable by SSH with passwordless sudo. See docs/providers/static.md for node declarations, address discovery, lifecycle behavior, and host reuse.

The equivalent managed-VM example uses the same Cyborg configuration with a CentOS Stream 10 guest on a remote libvirt execution host:

examples/devstack/remote-libvirt-cyborg-pci-sim/

make render \
  ARD_DEPLOYMENT=remote-libvirt-cyborg-pci-sim \
  ARD_RENDER_FILE=examples/devstack/remote-libvirt-cyborg-pci-sim/render.yaml
# Add the execution host to deployment-local local-vars.yaml, then:
make apply ARD_DEPLOYMENT=remote-libvirt-cyborg-pci-sim
make deploy ARD_DEPLOYMENT=remote-libvirt-cyborg-pci-sim

The remote-libvirt OKO example demonstrates that the same provider transport also composes with a MicroShift control plane and two pre-provisioned EDPM compute nodes. It reuses the existing OKO workload topology unchanged:

examples/oko/remote-libvirt-microshift-two-edpm-compute/

make render \
  ARD_DEPLOYMENT=remote-libvirt-oko \
  ARD_RENDER_FILE=examples/oko/remote-libvirt-microshift-two-edpm-compute/render.yaml
# Add the execution host to deployment-local local-vars.yaml, then:
make apply ARD_DEPLOYMENT=remote-libvirt-oko
make deploy ARD_DEPLOYMENT=remote-libvirt-oko
make verify ARD_DEPLOYMENT=remote-libvirt-oko

Make targets

The root Makefile wraps the provider playbooks. The default target is a full local rebuild workflow:

make
# equivalent to
make default

make default runs destroy-clean-generated, cleanup, render, apply, ping, deploy, and verify in order. The initial destroy and cleanup steps are best-effort so the target also works from a fresh checkout.

Individual workflow targets are also available:

make render
make apply
make ping
make ssh
make ssh-print
make deploy
make verify
make destroy
make destroy-clean-generated
make clean-generated
make cleanup
make site

Useful variables:

ARD_DEPLOYMENT       deployment name, default devstack-1
ARD_DEPLOYMENTS_DIR  deployment parent dir, default ./deployments
ARD_DEPLOYMENT_DIR   full workspace path; defaults to deployments/<deployment>
ARD_PROVIDER         provider, default libvirt; required for first render when selecting another provider
ARD_TOPOLOGY         topology preset
ARD_TARGET_BRANCH    DevStack target branch, default master
ARD_WORKLOAD         workload to deploy, default devstack; use microshift for MicroShift
ARD_SERVICES         comma-separated service profiles, default devstack,ovn,tempest; empty for MicroShift
ARD_PROVIDER_PROFILE provider profile, default local-libvirt
ARD_IMAGE            optional image key override; defaults to centos-stream-10 for MicroShift
ARD_NETWORK_CIDR     libvirt management CIDR, default 192.168.96.0/24
ARD_RENDER_FILE      optional render intent file; defaults to existing $(ARD_DEPLOYMENT_DIR)/render.yaml
ARD_NODE             inventory node for make ssh, default controller
ARD_SSH_PRINT        print SSH command without running it when set to 1
ARD_SSH_ARGS         extra arguments passed to ssh
ARD_EXTRA_VARS       extra Ansible vars appended to provider commands

When $(ARD_DEPLOYMENT_DIR)/local-vars.yaml exists, Make automatically passes it to apply, deploy, verify, destroy, destroy-clean-generated, and cleanup before ARD_EXTRA_VARS. Use it for local proxy settings, workload deploy-time settings, or development overrides that should persist with a rendered workspace but are not part of provider/topology render intent.

make apply is the provider-create phase: it creates provider resources, waits for SSH/cloud-init, and writes inventory.yaml plus provider-state.yaml. make deploy is the workload-converge phase: it uses reusable workload converge playbooks under ansible/playbooks/workloads/*/converge*.yaml. Top-level Molecule scenarios delegate to the same converge playbooks, so Make and Molecule exercise the same workload flow.

For KubeVirt OKO deployments, make deploy maps to the same high-level converge flow used by molecule/kubevirt-oko: deploy MicroShift when needed, configure the ARD multinode bridge overlay, then apply OKO network, control-plane, and dataplane resources. This keeps make apply equivalent to Molecule create and make deploy equivalent to Molecule converge.

Example:

make render \
  ARD_DEPLOYMENT=devstack-a \
  ARD_TARGET_BRANCH=master \
  ARD_TOPOLOGY=one-controller-two-compute \
  ARD_SERVICES=devstack,ovn,tempest \
  ARD_NETWORK_CIDR=192.168.99.0/24

Multiple local deployments

Use a unique deployment name and management CIDR for each local deployment:

make render ARD_DEPLOYMENT=devstack-a ARD_NETWORK_CIDR=192.168.99.0/24
make render ARD_DEPLOYMENT=devstack-b ARD_NETWORK_CIDR=192.168.100.0/24

Provider resources are named with the deployment name, for example:

ard-devstack-a-controller
ard-devstack-a-compute-1

Inventory hostnames remain logical names such as controller, compute-1, and compute-2.

Topology presets

Supported local render presets:

all-in-one
one-controller-one-compute
one-controller-two-compute

all-in-one renders:

controller

one-controller-one-compute renders:

controller
compute-1

one-controller-two-compute renders:

controller
compute-1
compute-2

Topology presets are built from generic node pools. Singleton pools can set an explicit name such as controller; counted pools default to readable hyphenated names such as {type}-{index}. Multinode topologies disable nova-compute on the controller through the rendered controller group vars when the topology says the controller does not run compute services.

Render intent, examples, and service profiles

The examples/ tree is the primary interface for reusable render intent. The same example can be used through Make, Molecule, or direct Ansible. Examples carry ard_render_schema_version: 1 and describe workload/topology intent; provider, identity, namespace, CIDR, and scenario-specific values can be overridden by the caller.

make render \
  ARD_DEPLOYMENT=stable-test \
  ARD_RENDER_FILE=examples/devstack/aio-plus-compute/render.yaml \
  ARD_TARGET_BRANCH=stable/2026.1 \
  ARD_NETWORK_CIDR=192.168.98.0/24

uv run ansible-playbook -i localhost, ansible/playbooks/provider/render.yaml \
  -e ard_deployment_name=stable-test \
  -e @examples/devstack/aio-plus-compute/render.yaml

Molecule scenarios set provisioner.ard_render_file to one of these examples and keep scenario-local overrides under provisioner.ard.

See docs/concepts/ard-render-model.md for the render model, provider identity, node classes, networks, and state artifacts. See docs/architecture/render-contracts.md for contributor-level schema and merge contracts.

Use ard_render_overrides for simple kustomize-like customizations. Overrides use ordinary recursive dictionary merge semantics: later dictionaries replace scalar and list values for the relevant section.

ard_management_network: ard-mgmt
ard_render_overrides:
  provider_defaults:
    image: ubuntu-24.04
  node_pools:
    compute:
      count: 2
      flavor: devstack-compute
      profiles:
        - ssh
        - nested_virt
        - performance
      networks:
        - name: ard-mgmt
          ip_start: 3
        - name: storage
          ip_start: 20
  networks:
    storage:
      cidr: 192.168.120.0/24
      provider_network: ard-storage
  devstack:
    common:
      enable_ceph: true
    controller:
      controller_localrc_extra:
        DEBUG_LIBVIRT_COREDUMPS: true

ard_render_node_overrides:
  compute-2:
    image: ubuntu-24.04
    flavor: devstack-compute
    profiles:
      - ssh
      - nested_virt
      - gpu
    networks:
      ard-mgmt:
        ip: 192.168.98.50

The libvirt provider supports multiple rendered networks. ard_management_network selects which attached network is used for SSH inventory and nodepool.private_ipv4; additional networks are rendered as extra libvirt networks and attached as additional VM interfaces.

The built-in tenant network preset is isolated and opt-in. It is not attached by default, but can be added to a node pool when guests need a bridge-only network for their own VLANs, DHCP, or overlay experiments:

ard_render_overrides:
  node_pools:
    compute:
      networks:
        - name: ard-mgmt
          ip_start: 3
        - name: tenant
          mac_start: 20

Isolated networks render as libvirt networks without host-side IP, NAT, or DHCP. Guest interfaces are still given deterministic MAC addresses and stable interface names.

Current service profiles are:

devstack
ovn
tempest
ceph

Images and flavors

Reusable provider defaults live in the ARD provider common role. Deployment workspaces normally only select the image/flavor by name instead of embedding the full registry.

Current image keys:

debian-13
ubuntu-24.04
centos-stream-10
fedora-eln
almalinux-10
rocky-linux-10

Current flavor keys:

devstack-control   8 vCPU, 16 GiB RAM, 80 GiB disk
devstack-compute   8 vCPU,  8 GiB RAM, 80 GiB disk
microshift-node    8 vCPU,  8 GiB RAM, 80 GiB disk

The default local image is Debian 13 genericcloud.

Cache and state paths

Base cloud images are cached under:

$XDG_CACHE_HOME/ard/images

or, if XDG_CACHE_HOME is unset:

~/.cache/ard/images

Per-deployment libvirt disks, seed ISOs, NVRAM, and console logs live under:

$XDG_STATE_HOME/ard/libvirt/images/<deployment-name>

or, if XDG_STATE_HOME is unset:

~/.local/state/ard/libvirt/images/<deployment-name>

make destroy removes per-deployment provider resources but keeps cached base images and generated workspace artifacts for inspection. Use make destroy-clean-generated to destroy provider resources and then remove inventory.yaml, provider-state.yaml, and rendered/. Use make clean-generated to remove those generated files without touching provider resources.

Molecule scenarios

Top-level Molecule scenarios are full ARD-backed validation flows. They call the same provider playbooks used by Make and do not use Vagrant. Migrated scenarios load reusable intent from provisioner.ard_render_file and keep only scenario-local overrides in molecule.yml under provisioner.ard; Molecule platforms are intentionally omitted so topology and node names are defined only once by ARD render presets.

Available scenarios:

default                    Debian 13, controller + compute-1, master
one-controller-two-compute Debian 13, controller + compute-1 + compute-2
stable-2026.1              Ubuntu 24.04, controller + compute-1, stable/2026.1

Run a full scenario test:

uv run molecule test -s default

For a cheaper loop:

uv run molecule create -s default
uv run ansible -i molecule/default/deployment/inventory.yaml all -m ping
uv run molecule converge -s default
uv run molecule verify -s default
uv run molecule destroy -s default

Role-level Molecule scenarios live under ansible/roles/*/molecule and use Podman where containers are sufficient:

make molecule-test
make molecule-role-ensure_kustomize

Remote libvirt execution host

The libvirt provider can execute through Ansible on a remote virtualization host while keeping deployment.yaml, generated inventory, rendered artifacts, and provider-state.yaml in the local deployment directory. Configure the host in deployment-local local-vars.yaml:

ard_libvirt_execution_host:
  name: virt-host
  ansible_host: hypervisor.example.com
  ansible_user: virt-admin
  ansible_port: 22
  ansible_private_key_file: ~/.ssh/id_ed25519
  ansible_ssh_common_args: ""
  ansible_become: false

The execution host defaults to localhost. In both modes ARD runs host-local virsh with qemu:///system; remote libvirt transport URIs are not used. Libvirt cloud-init only makes guests reachable by Ansible. The shared provider apply and workload roles perform subsequent configuration, preserving the same flow used by static and KubeVirt providers. Images, VM disks, seed media, and console logs live on the execution host. Their resolved paths and the non-secret SSH connection metadata are recorded in provider-state.yaml before resources are created, so later apply, destroy, deploy, verify, and make ssh commands use the same target. Private key contents and passwords are never copied into deployment state.

When VM management addresses are not routed to the controller, the generated workload inventory automatically proxies guest SSH through the execution host. The same inventory drives apply readiness, workload deployment, verification, and make ssh, including distinct hypervisor and guest keys or a nonstandard hypervisor SSH port.

Override execution-host storage locations when needed:

ard_libvirt_execution_image_dir: /var/lib/ard/libvirt/images
ard_libvirt_execution_image_cache_dir: /var/cache/ard/images

The remote account still needs the permissions required for qemu:///system, plus virsh, qemu-img, setfacl, compatible QEMU/KVM firmware, and access to the configured storage paths and either cloud-localds or genisoimage for NoCloud seed media. CentOS Stream 10 uses the EPEL genisoimage package. The controller needs ssh-keygen.

Host package installation is deliberately separate from normal provider apply. To opt in to bootstrapping a virtualization host, run the dedicated playbook against an explicit inventory target:

uv run ansible-playbook \
  -i virt-host, \
  ansible/playbooks/provider/bootstrap-libvirt-host.yaml \
  -e ard_libvirt_bootstrap_user=stack

The playbook contains Debian-family and Red Hat-family package mappings, installs the libvirt/QEMU, firmware, ACL, and NoCloud seed tooling, enables the appropriate libvirt service sockets, and verifies qemu:///system. The CentOS Stream 10 path has been exercised against a live execution host. Validate or override package choices for other distributions. On CentOS it enables EPEL to install genisoimage. Omit ard_libvirt_bootstrap_user if group membership should not be changed. Reconnect after adding a user to libvirt.

Troubleshooting

Libvirt access

The provider uses host-local qemu:///system on the selected execution host. That account normally needs libvirt/qemu group access. If bootstrap reports missing group membership, log out and back in or use newgrp libvirt before running provider commands.

On SELinux hosts, apply labels each generated console log for virtlogd. This per-file chcon label is not a persistent file-context policy; if an administrator runs restorecon over the image tree, rerun apply before restarting affected guests.

UEFI firmware

Libvirt firmware auto-selection is used for UEFI boot with secure boot disabled. Install the OVMF/edk2 firmware package for your distribution if libvirt cannot define or start the domain.

SSH not ready

apply creates VMs and inventory, then waits for SSH and cloud-init completion before returning. If apply times out, inspect the serial console logs under the libvirt deployment state directory or retry:

make apply ARD_DEPLOYMENT=devstack-a

Molecule create uses the same apply playbook, so it also waits for node readiness before converging.

CIDR conflicts

Use a management CIDR that does not conflict with existing host networks or other ARD deployments:

make render ARD_DEPLOYMENT=devstack-b ARD_NETWORK_CIDR=192.168.100.0/24

Cleanup after failure

Destroy provider resources while preserving generated artifacts for inspection:

make destroy ARD_DEPLOYMENT=devstack-a

Destroy provider resources and remove generated inventory/state/rendered files:

make destroy-clean-generated ARD_DEPLOYMENT=devstack-a

Remove generated files without touching provider resources:

make clean-generated ARD_DEPLOYMENT=devstack-a

If the workspace is no longer needed:

make cleanup ARD_DEPLOYMENT=devstack-a

Generated runtime files are ignored by git:

inventory.yaml
provider-state.yaml
rendered/
logs/

Name		Name	Last commit message	Last commit date
ansible		ansible
deployments		deployments
docs		docs
examples		examples
molecule		molecule
scripts		scripts
submodules		submodules
.gitignore		.gitignore
.gitmodules		.gitmodules
.yamllint		.yamllint
Makefile		Makefile
README.md		README.md
ansible.cfg		ansible.cfg
bindep.txt		bindep.txt
bootstrap-repo.sh		bootstrap-repo.sh
pyproject.toml		pyproject.toml
requirements.yml		requirements.yml
uv.lock		uv.lock

Navigation Menu

Repository files navigation

ARD

Quick start

Local deployment workflow

Static provider

Make targets

Multiple local deployments

Topology presets

Render intent, examples, and service profiles

Images and flavors

Cache and state paths

Molecule scenarios

Remote libvirt execution host

Troubleshooting

Libvirt access

UEFI firmware

SSH not ready

CIDR conflicts

Cleanup after failure

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

ARD

Quick start

Local deployment workflow

Static provider

Make targets

Multiple local deployments

Topology presets

Render intent, examples, and service profiles

Images and flavors

Cache and state paths

Molecule scenarios

Remote libvirt execution host

Troubleshooting

Libvirt access

UEFI firmware

SSH not ready

CIDR conflicts

Cleanup after failure

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages