web-daemon.md

Web interface and automatic diagnostics

pg-diag-web provides an English web interface for local diagnostic jobs and downloads. pg-diag-daemon watches local host/PostgreSQL metrics and starts a report when a declarative rule fires. Both are foreground Linux processes in the pg_diag package. The existing pg-diag commands and defaults are unchanged.

The web UI serves the same render/styles/report.css that the HTML renderer embeds in standalone reports. Tables, controls, navigation, colors and light/dark themes share that stylesheet. Service-specific CSS supplies page layout.

Setup

Install the optional web dependencies in the same environment as pg-diag:

python -m pip install 'pg-diag[web]'
# From a source checkout:
python -m pip install -e '.[web]'

Copy examples/service/service.yaml, automation.yaml and users.yaml to a local writable configuration directory. Installed packages also ship them under share/doc/pg-diag/examples/service/ in the Python installation's data directory. Paths in service.yaml are resolved relative to that file. Use a common service.yaml and data_dir for both processes, under the same OS account.

Edit service.yaml for the local PostgreSQL port, database, role and passfile. Unix sockets and loopback addresses are accepted; remote targets and SSH collection are not offered by these modules. Existing CLI remote modes remain available separately. The CLI executable must be from the same package version. Point log_sources.postgres_csv.directory to the local PostgreSQL CSV log directory if the standalone logs profile is needed. Full database reports use the CLI's existing local log discovery; standalone logs uses the selected registered log source.

Generate each web user's password hash interactively, without putting a password in shell history:

python -c 'from getpass import getpass; from argon2 import PasswordHasher; print(PasswordHasher().hash(getpass("Password: ")))'

Paste the resulting $argon2id$... string as password_hash in users.yaml. There are no default passwords. Example placeholders intentionally prevent web startup until replaced. A writer can manage every job, control the daemon and edit automation; a reader can list/download completed archives and view the read-only host/instance inventory. Account changes take effect after restarting web, which invalidates browser sessions and stops its active collection. The daemon does not read users.yaml.

Use owner-only permissions for configuration and PostgreSQL credentials:

chmod 600 service.yaml automation.yaml users.yaml local.pgpass
pg-diag-web --config ./service.yaml

The supplied listener is 0.0.0.0:8080; open http://HOST:8080. The service uses plain HTTP as configured, without TLS or a reverse proxy requirement. An SSH tunnel can carry browser traffic when needed:

ssh -L 8080:127.0.0.1:8080 USER@HOST

Run the daemon separately, in another shell or with your usual process manager:

pg-diag-daemon --config ./service.yaml

Systemd, containers and cgroups are not required. Either process works alone. One instance of each is permitted per data_dir. The example automation rules start disabled; enable them after configuring the database and thresholds.

Reports and jobs

The example configuration supplies 34 editable presets for DBA, developer and security tasks. Shared workflows are labeled DBA / Dev or DBA / Dev / Security; dedicated developer profiles cover functions, bulk writes, schema/migration checks and recorded query plans. Other profiles cover general collection, SQL/I/O/memory, replication, maintenance, privileges, authentication, TLS, RLS, OS permissions and audit settings. See the preset catalog for their scope and collection windows. The writer's New report page reads presets from automation.yaml; the same profiles are available to automatic rules. Select all compatible items, specific item IDs, or any matching tags, optionally restricting item types. The full preset uses 300 seconds of snapshots, a 5-second interval, HTML + JSON and 10 minutes of logs. These are service profile settings, not new CLI defaults. The form shows how many selected items can run with the current mode, item types and log depth. For quick checks, use the current-locks preset or shorten a snapshot window to 30 seconds with a 5-second interval.

Jobs displays both execution slots and the shared history. Each job has a saved parameter snapshot, its initiator/trigger, phase, stdout/stderr, progress, cancel, retry and archive status. CLI progress describes collection; packaging has a separate phase. Missing progress stays indeterminate. Output is transported with server-sent events and resumes from a cursor after reconnection. An opened job has a #jobs/<id> URL, so refreshing the page or using browser Back/Forward restores its status and output. Its details appear above the history.

There is one manual slot and one automatic slot: at most two collections at once. An occupied slot rejects a new request; no queue exists. Finalization occupies the slot until packaging has been attempted. Retrying any job starts a new job in the web slot using its original report parameters. Repackaging an archive failure reuses the collected files and preserves their original completion time.

Cancellation and owner shutdown stop the CLI and its descendants, including children that create another session. A temporary per-job Linux child subreaper holds an inherited slot lock and watches a pipe owned only by web/daemon. EOF also detects an owner's SIGKILL. It escalates cancellation through SIGINT, SIGTERM and SIGKILL and then packages remaining files. The other owner's job continues. The supervisor exits with the job; it is not a third installed service.

Result and archive status are separate. CLI exit code 1 with valid requested reports is item_errors; cancellation/interruption may produce an incomplete archive containing only logs and a manifest. Packaging errors preserve source files and expose Retry archive. Archives contain available report.html, report.json, collector/output logs, a manifest with checksums, and automatic trigger evidence. They are downloaded as tar.gz; there is no server-side HTML preview. Open downloaded reports locally.

Automation YAML and forms

Writers can Start daemon, Stop daemon, and Apply rules on Automation. Starting launches the separate daemon with this installation's service.yaml; it continues when web exits. An already running standalone daemon sharing the same data_dir can also be controlled. Stop cancels its current automatic collection and waits for packaging; the UI shows stopping until it exits. Manual collections remain independent. Startup errors are shown in the UI; the daemon's startup log is data_dir/daemon.log.

Valid changes still load automatically on the next sampling cycle. Apply rules wakes the daemon immediately, including with a long sampling interval. Commands are recorded in SQLite, addressed to the process's PID/start-time/boot identity, and acknowledged by the daemon. The UI compares the saved file revision with the revision actually evaluated by the rule engine, rather than treating a save or signal as successful application. Invalid edits keep the last valid rules and show an error. Changes during measurement are reloaded before evaluating rules; existing collections retain their original profile and trigger revision.

Automation provides metric/rule tables and a graphical condition editor. Settings provides retention/default-profile controls and the same YAML as a text editor. Saving is atomic and checks the expected revision: a concurrent edit returns a conflict instead of overwriting another writer's changes. Duplicate YAML keys, aliases, unknown fields and unsupported profile combinations are rejected. Comments/order are not preserved by form edits.

External edits to automation.yaml are read automatically. A malformed file leaves the last valid configuration effective and displays an error. Without a valid current or previously accepted configuration, automatic collection and deletion stay disabled. The listener, executable and database credentials are local settings in service.yaml; restart the owners after changing that file.

For example, the supplied long-lock rule selects the waits preset:

id: long_lock_wait
enabled: true
condition:
  metric: pg.lock_wait_max_seconds
  aggregate: last
  operator: gte
  value: 30
for: 0s
recovery:
  condition:
    metric: pg.lock_waiting_sessions
    aggregate: last
    operator: eq
    value: 0
  for: 30s
cooldown: 10m
profile: waits_5m

Conditions can nest all and any. Leaf comparisons are gt, gte, lt, lte, eq, ne. Aggregation is last or mean/min/max with a window such as 3m. A mean is weighted by elapsed sample time. Incomplete windows, failed measurements and excessive gaps evaluate to unknown; they do not confirm a trigger. for requires continuous confirmation. Changing a condition resets its window/confirmation without discarding the already-handled episode latch.

After firing, the rule requires its recovery condition and cooldown before it can fire for another episode. Cooldown starts when collection ends, not when it starts. If the daemon slot is busy at the trigger, that episode is recorded as suppressed_busy and is not replayed later. Rule latches/cooldowns survive a daemon restart; in-memory windows warm up again. Disabling a rule does not cancel its running job.

Metric	Meaning
host.cpu_busy_pct	Whole-host CPU busy percent from /proc/stat; idle, iowait and steal are excluded.
host.memory_available_pct	MemAvailable / MemTotal, percent; lower values indicate pressure.
pg.active_sessions	Active client backends, excluding the observer.
pg.waiting_sessions	Active client backends with a wait event.
pg.lock_waiting_sessions	Client backends with wait type Lock.
pg.lock_wait_max_seconds	Greatest age of a currently ungranted lock request.

Only sources needed by enabled rules are polled. PostgreSQL observation uses a read-only connection, bounded statements and pg_read_all_stats (or superuser) visibility; insufficient visibility is unknown rather than a false zero. Report collection has its existing permissions requirements. Lock age comes from pg_locks.waitstart when available; otherwise it is a lower-bound observed age keyed by backend identity and lock identity. A lost/changed wait or sampling gap resets that observation. Query/transaction age is never substituted for wait age.

Host and PostgreSQL inventory

The authenticated header shows the service host's hostname and non-loopback IP addresses (loopback is used if no other address exists). Instances lists local PostgreSQL postmasters with their ports, server versions, primary/standby roles, PIDs and data directories, plus a separate replication-slot table. The connection from service.yaml is marked Diagnostic target (the service default). Writers can use Make Report under a database name to open New report with that instance and database selected. Direct navigation to New report starts with the service default, or retains the current form's explicit selection. The form shows address/socket, port, database user, version and log directory. Its target survives page reload via the URL and is not changed by a profile preset.

The server resolves the selection against fresh local discovery, checks the instance identity and database availability, and tests the selected connection before admitting a job. Arbitrary hosts, credentials or log paths are not accepted from the browser. Unavailable/stale selections fail explicitly, without falling back to the default. Each job and archive manifest stores its resolved target; history, archives and job details display it. Retry revalidates that target and rejects a changed instance or log directory. Older jobs without recorded target metadata are labelled accordingly.

Log directory is read from each instance's PostgreSQL settings; relative paths are resolved against its data directory. SQL modes let the existing collector discover logs through the selected connection. Logs mode passes the selected instance's directory and timezone to the existing CLI. Logs cover the entire instance, not only the selected database. Unreadable directories or unavailable CSV logging are reported and block logs-only submissions. No CLI options changed.

Saved profiles contain collection parameters, not the form's selected target. New logs profiles use log_source: instance; the standalone daemon resolves that against its configured default instance. Existing named log_sources remain supported for daemon/API jobs without an explicit target. Automation continues to monitor and collect from the service.yaml target.

Each instance has database subrows, including empty and template databases. Their Status cells show total client connections, active, idle, and idle in transaction (including aborted transactions). Other client states appear when present. Background/parallel workers and the inventory connection itself are excluded. Filtering by database name retains its parent instance; sorting and pagination keep each instance together with its databases.

If PostgreSQL hides session details from the configured database user, the UI marks counts as lower bounds (≥) and shows the number of hidden sessions; it does not mistake them for zero activity. Full visibility requires superuser or pg_read_all_stats privileges, as described in the PostgreSQL statistics documentation. Database-query failures are displayed separately from replication-slot errors.

Archives with failed, cancelled or interrupted results show No report and Download logs, rather than implying that a completed report exists. These archives retain stdout/stderr, manifest and any incomplete files for debugging. Successful reports and reports with item-level errors retain Download.

Discovery reads /proc and each accessible postmaster.pid, then makes bounded, read-only PostgreSQL connections. Other instances use the configured database user/passfile and their postgres database. Connection and slot-permission errors are displayed explicitly. Inventory is cached for 15 seconds. It is limited to the visible process/network namespace: inside a container it describes that container, not sibling containers or the underlying host. Processes hidden from the service OS user may not be discoverable.

Storage and lifecycle

data_dir contains SQLite state, interprocess locks, per-job work/output files, configuration history and archives/. Put it on a local filesystem; distributed execution and NFS locking are outside this service's scope. New state files are owner-only. Do not point independent installations at partially shared state.

Both processes rotate archives using one maintenance lock and one policy: archive.retention (default 30d) and the fixed max_total_bytes: 1000000000. The oldest completion times are evicted first. Downloads do not renew age and an already-open download continues if rotation unlinks its archive. Changing retention in the UI previews affected archives. Cleanup is also performed before publishing new archives, so concurrent finalization shares the same quota.

On Reports, writers can select individual archives or all rows on the current page and click Delete. The confirmation dialog lists the selected archives and requires the exact text DELETE, also checked by the API. Selection is kept across filtering, sorting and page changes. Deleted archive files are removed from disk; job history and collection output remain until normal retention expires. Readers cannot select or delete archives. Deletion uses the same maintenance lock as rotation and publication, and already-open downloads can finish. Files that could not be deleted remain listed, with an error shown.

The 1 GB cap covers published compressed archives, not all working files or the temporary space needed during packaging. A single archive that exceeds it fails packaging without evicting existing reports. Packaging/hash verification has a 300-second deadline. stdout and stderr are each capped at 16 MiB, with a bounded 36 MiB event journal; truncation does not stop collection. Expired job work, history and old configuration snapshots are also cleaned up.

After owner death, the supervisor finishes cancellation/packaging. A later owner can recover interrupted metadata/publication after confirming the old processes are gone. It does not resume collection. An uncertain live process tree retains its occupied slot. Direct terminal pg-diag invocations and different data_dir installations are not included in the two-slot limit.

Development checks

python -m pip install -e '.[dev,web,browser]'
pytest -q tests/unit/test_service.py tests/unit/test_service_runtime.py tests/unit/test_service_host.py
PG_DIAG_BROWSER_TESTS=1 pytest -q tests/unit/test_service_runtime.py -k browser
PG_DIAG_SERVICE_INTEGRATION=1 pytest -q tests/integration/test_service_postgres.py
# Optional short pgbench baseline / observer / two-collector comparison:
PG_DIAG_SERVICE_INTEGRATION=1 PG_DIAG_SERVICE_LOAD_TESTS=1 pytest -q -s tests/integration/test_service_postgres.py -k observer_cost

The integration tests create their own temporary local PostgreSQL cluster and stop it afterwards. They default to /usr/lib/postgresql/18/bin; set PG_DIAG_SERVICE_PG_BIN for another installation. They test actual lock waits, visibility, a triggered snapshots report, concurrent collectors, owner death and disappearance of the terminated collector's PostgreSQL connection. Inventory coverage starts a second cluster and verifies each server's version and slots.