Add authenticated blueapi/tiled scans and PV health checks

README.md

@@ -7,27 +7,158 @@
 
 Scheduled component tests and end-to-end experiment runs for ViSR
 
-This is where you should write a short paragraph that describes what your module does,
-how it does it, and why people should use it.
+`visr-tests` runs pytest checks against a ViSR beamline (b01-1) on a schedule,
+as CronJobs from the beamline's services repo, and writes a JUnit report per
+check. Component checks look at one service or device at a time; experiment
+checks run a plan through blueapi and check the data it wrote to tiled.
 
 What            | Where
 :---:           | :---:
 Source          | <https://github.com/DiamondLightSource/visr-tests>
 Docker          | `docker run ghcr.io/diamondlightsource/visr-tests:latest`
 Releases        | <https://github.com/DiamondLightSource/visr-tests/releases>
 
-This is where you should put some images or code snippets that illustrate
-some relevant examples. If it is a library then you might put some
-introductory code here:
+## Running checks
 
-```python
-from visr_tests import __version__
+```
+visr-tests run CHECK [ARGUMENT ...] [CHECK [ARGUMENT ...] ...]
+```
+
+Each check runs in its own pytest session, in the order given. Reports go to
+`$REPORT_DIR/<timestamp>/$JOB_NAME/<check>.xml`. The exit status is non-zero
+if any check fails.
+
+Check                      | What it checks
+:---                       | :---
+`components/blueapi`       | blueapi `/healthz` answers 200 through its public URL
+`components/tiled`         | tiled `/healthz`, an anonymous search, and that `/api/v1/binned/<unknown run>` answers 422 (the `dls_tiled.visr` router is mounted)
+`components/flipper`       | each flipper (PV prefixes as arguments, e.g. `BL01C-EA-FLIP-01`) answers `caget <prefix>:Position`, and its `Position_RBV` is 0 or 1 and not in INVALID alarm
+`components/align`         | each alignment stage (PV prefixes as arguments, e.g. `BL01C-EA-ALIGN-01`) answers `caget <prefix>:Pitch:Acceleration_RBV`
+`components/panda`         | each PandA (PV prefixes as arguments, e.g. `BL01C-EA-PANDA-01 BL01C-MO-PPANDA-01`) answers `pvget <prefix>:Pcap:Active`
+`components/iocs`          | each IOC in `IOC_NAMES` answers `caget <IOC NAME>:UPTIME` (devIocStats)
+`experiments/spectroscopy` | runs `demo_spectroscopy` as a step scan and as a fly scan through blueapi, then finds each run in tiled by `start.blueapi_task_id` and checks its exit status is `success`, it has 4 primary events, and `/binned?width=2&height=2` gives a 2x2 `RedTotal`
+`experiments/tomography`   | the tomography experiment
+
+The PV checks only read: they never write a PV or move anything. A PV must
+answer within `PV_TIMEOUT` seconds; one that does not is tried once more before
+the check fails. Moving each flipper between its two positions and back is a
+possible future check.
+
+The scans move the sample stage; the fly scan also needs the PandA and PMAC.
+The plan, its parameters and the expected results are in
+`src/visr_tests/experiments/spectroscopy.py`.
 
-print(f"Hello visr_tests {__version__}")
+### Environment
+
+Variable             | Default | Used by
+:---                 | :---    | :---
+`BLUEAPI_URL`        | `https://b01-1-blueapi.diamond.ac.uk` | blueapi checks. The public URL, so requests go through oauth2-proxy as users' do
+`TILED_URL`          | `http://b01-1-tiled:8000` | tiled checks. The in-cluster Service
+`TOKEN_URL`          | `https://identity.diamond.ac.uk/realms/dls/protocol/openid-connect/token` | experiments
+`CLIENT_ID`          | none | experiments: the Keycloak service account's client id
+`CLIENT_SECRET`      | none | experiments: its client secret
+`INSTRUMENT_SESSION` | none | experiments: the instrument session the plans run in, e.g. `cm44191-4`. The service account must be allowed to run plans in it
+`SCAN_TIMEOUT`       | `300` | experiments: seconds each scan may take, from submitting it to its run stopping in tiled
+`PV_TIMEOUT`         | `5` | PV checks: seconds to wait for each PV read, as `caget -w 5`
+`IOC_NAMES`          | none | `components/iocs`: the IOCs to check, separated by commas or spaces, e.g. `bl01c-di-dcam-01,bl01c-mo-ioc-01`. Each must load devIocStats (`devIocStats.iocAdminSoft` in its `ioc.yaml`)
+`EPICS_CA_NAME_SERVERS` | none | PV checks over CA: in the cluster the gateway, `b01-1-epics-gateways:9064`
+`EPICS_PVA_NAME_SERVERS` | none | PV checks over PVA: in the cluster the gateway, `b01-1-epics-gateways:9075`
+`EPICS_CA_AUTO_ADDR_LIST`, `EPICS_PVA_AUTO_ADDR_LIST` | `YES` | set both to `NO` in the cluster, so PVs are searched for only through the gateway
+
+Experiments authenticate to blueapi with an OAuth2 `client_credentials` token
+(`client_secret_basic`), fetched once per check. A check that needs a variable
+which is not set fails at setup and says which. The client secret and the token
+are never logged or written to reports.
+
+The PV checks use the standard EPICS client environment above (aioca for CA,
+p4p for PVA), as blueapi does in `b01-1-blueapi/values.yaml`. In the cluster
+the components job needs no host network: it reaches the IOCs through the
+gateway Service.
+
+For example, against the beamline from a workstation (tiled through its
+LoadBalancer address):
+
+```
+CLIENT_ID=... CLIENT_SECRET=... INSTRUMENT_SESSION=cm44191-4 \
+TILED_URL=http://<tiled LoadBalancer IP>:8000 \
+    uv run visr-tests run components/blueapi components/tiled experiments/spectroscopy
 ```
 
-Or if it is a commandline tool then you might put some example commands here:
+and the PV checks through the gateway's LoadBalancer address:
 
 ```
-python -m visr_tests --version
+EPICS_CA_NAME_SERVERS=<gateway IP>:9064 EPICS_CA_AUTO_ADDR_LIST=NO \
+EPICS_PVA_NAME_SERVERS=<gateway IP>:9075 EPICS_PVA_AUTO_ADDR_LIST=NO \
+IOC_NAMES=bl01c-di-dcam-01,bl01c-di-dcam-02,bl01c-di-dcam-04,bl01c-di-dcam-05,bl01c-di-locum-01,bl01c-mo-ioc-01 \
+    uv run visr-tests run \
+        components/flipper BL01C-EA-FLIP-01 BL01C-EA-FLIP-02 BL01C-EA-FLIP-03 \
+        components/align BL01C-EA-ALIGN-01 \
+        components/panda BL01C-EA-PANDA-01 BL01C-MO-PPANDA-01 \
+        components/iocs
 ```
+
+## Provisioning the client secret
+
+The experiment checks run as a Keycloak service account. Its client secret
+reaches the CronJob as a SealedSecret in the services repo.
+
+1.  Request the service account at the
+    [Keycloak service desk](https://jira.diamond.ac.uk/servicedesk/customer/portal/5)
+    with:
+
+    - audience: `account`
+    - instrument: `b01-1`
+    - client id: `b01-1-blueapi-test-runner`
+    - environment: Dev/Test and Production
+    - your GPG public key, so the secret can be sent to you encrypted (see
+      [request a registration with Keycloak](https://dev-guide.diamond.ac.uk/authn/how-tos/request-a-registration-with-keycloak/))
+
+    You may receive separate credentials for identity, identity-dev and
+    identity-test. Use the identity (production) ones: `TOKEN_URL` defaults
+    to it.
+
+2.  Decrypt the secret and seal it as the Secret `visr-tests-secret` in the
+    `b01-1-beamline` namespace, with keys `client-id` and `client-secret`.
+    From a clone of `b01-1-services`, with your GPG key available:
+
+    ```bash
+    module load k8s-b01-1   # kubectl and kubeseal for the b01-1 cluster
+
+    tmp=$(mktemp -d)
+    gpg --decrypt client-secret.pgp | tr -d '\r\n' > "$tmp/client-secret"
+    kubectl create secret generic visr-tests-secret \
+        --namespace b01-1-beamline \
+        --from-literal=client-id=b01-1-blueapi-test-runner \
+        --from-file=client-secret="$tmp/client-secret" \
+        --dry-run=client -o yaml |
+        kubeseal --format yaml \
+        > services/b01-1-nightly-experiments/templates/secret.yaml
+    rm -rf "$tmp"
+    ```
+
+    Replace `client-secret.pgp` with the file you received. kubeseal uses the
+    cluster's sealed-secrets controller with its default name and namespace,
+    as the other SealedSecrets in `b01-1-services` do. The sealed values only
+    decrypt as `visr-tests-secret` in `b01-1-beamline`: do not rename it. The
+    SealedSecret is safe to commit; the decrypted file is not, so make sure
+    it is gone.
+
+3.  Point the CronJob at it, in
+    `services/b01-1-nightly-experiments/values.yaml`:
+
+    ```yaml
+    scheduled-jobs:
+      env:
+        - name: CLIENT_ID
+          valueFrom:
+            secretKeyRef: {name: visr-tests-secret, key: client-id}
+        - name: CLIENT_SECRET
+          valueFrom:
+            secretKeyRef: {name: visr-tests-secret, key: client-secret}
+        - name: INSTRUMENT_SESSION
+          value: cm44191-4
+    ```
+
+    The components job needs no credentials. A Secret is per namespace, so
+    another service in `b01-1-beamline` can reference `visr-tests-secret` the
+    same way.

pyproject.toml

@@ -13,7 +13,13 @@ classifiers = [
     "Programming Language :: Python :: 3.14",
 ]
 description = "Scheduled component tests and end-to-end experiment runs for ViSR"
-dependencies = [] # Add project dependencies here, e.g. ["click", "numpy"]
+dependencies = [
+    "aioca>=2.1",
+    "httpx>=0.28.1",
+    "p4p>=4.3.0",
+    "pytest>=9.1.1",
+    "pytest-asyncio>=1.4.0",
+]
 dynamic = ["version"]
 license.file = "LICENSE"
 readme = "README.md"
@@ -26,6 +32,7 @@ dev = [
     "pytest",
     "pytest-cov",
     "ruff",
+    "softioc>=4.7.2",
     "tox-uv",
     "types-mock",
 ]

src/visr_tests/components/align.py

@@ -0,0 +1,11 @@
+"""Check the alignment stage's IOC answers over Channel Access.
+
+Arguments: stage PV prefixes, e.g. BL01C-EA-ALIGN-01. Reads the fastcs-ximc
+``Pitch:Acceleration_RBV``.
+"""
+
+from visr_tests.epics import ca_read
+
+
+async def test_align_answers(check_arg: str, pv_timeout: float):
+    await ca_read(f"{check_arg}:Pitch:Acceleration_RBV", pv_timeout)

src/visr_tests/components/blueapi.py

@@ -0,0 +1,8 @@
+"""Check blueapi is up, through its public URL (BLUEAPI_URL)."""
+
+from visr_tests.services import Blueapi
+
+
+def test_blueapi_healthz(blueapi: Blueapi):
+    status = blueapi.healthz()
+    assert status == 200, f"blueapi /healthz returned HTTP {status}"

src/visr_tests/components/flipper.py

@@ -1,9 +1,19 @@
-"""Check a flipper moves between its two positions."""
+"""Check a flipper's IOC serves its position. Read only: the flipper does not move.
 
-import logging
+Arguments: flipper PV prefixes, e.g. BL01C-EA-FLIP-01. fastcs-thorlabs-mff
+serves ``Position``, the setpoint (a boolOut), and ``Position_RBV``, the
+position read back from the device (a boolIn, ZNAM False, ONAM True).
+"""
 
-log = logging.getLogger(__name__)
+from visr_tests.epics import INVALID, ca_read
 
 
-def test_flipper_cycles(check_arg: str):
-    log.info("pass cycle of flipper %s", check_arg)
+async def test_flipper_position_connects(check_arg: str, pv_timeout: float):
+    await ca_read(f"{check_arg}:Position", pv_timeout)
+
+
+async def test_flipper_position_valid(check_arg: str, pv_timeout: float):
+    pv = f"{check_arg}:Position_RBV"
+    value = await ca_read(pv, pv_timeout)
+    assert value.severity < INVALID, f"{pv} is in INVALID alarm"
+    assert value in (0, 1), f"{pv} is {value!r}, expected 0 (False) or 1 (True)"

src/visr_tests/components/iocs.py

@@ -0,0 +1,11 @@
+"""Check each IOC in IOC_NAMES answers over Channel Access.
+
+Reads ``<IOC name in upper case>:UPTIME``, which every IOC that loads
+devIocStats (``devIocStats.iocAdminSoft`` in its ioc.yaml) serves.
+"""
+
+from visr_tests.epics import ca_read
+
+
+async def test_ioc_uptime(ioc: str, pv_timeout: float):
+    await ca_read(f"{ioc.upper()}:UPTIME", pv_timeout)

src/visr_tests/components/panda.py

@@ -0,0 +1,11 @@
+"""Check a PandA's IOC answers over PV Access.
+
+Arguments: PandA PV prefixes, e.g. BL01C-EA-PANDA-01 BL01C-MO-PPANDA-01.
+Reads the fastcs-pandablocks ``Pcap:Active``.
+"""
+
+from visr_tests.epics import pva_read
+
+
+async def test_panda_answers(check_arg: str, pv_timeout: float):
+    await pva_read(f"{check_arg}:Pcap:Active", pv_timeout)

src/visr_tests/components/tiled.py

@@ -0,0 +1,23 @@
+"""Check tiled is up, readable anonymously and serves the ViSR /binned route."""
+
+from visr_tests.services import Tiled
+
+# no run has this name, so /binned must reject it
+NO_SUCH_RUN = "visr-tests-no-such-run"
+
+
+def test_tiled_healthz(tiled: Tiled):
+    status = tiled.healthz()
+    assert status == 200, f"tiled /healthz returned HTTP {status}"
+
+
+def test_tiled_anonymous_read(tiled: Tiled):
+    assert tiled.count() >= 0
+
+
+def test_tiled_binned_route(tiled: Tiled):
+    """422 for an unknown run shows that the dls_tiled.visr router is mounted."""
+    status = tiled.binned_status(NO_SUCH_RUN)
+    assert status == 422, (
+        f"tiled /binned returned HTTP {status} for an unknown run, expected 422"
+    )