[Architecture] Move BullMQ, Redis, and cron runtime implementations out of the application package

[mhbdev]

Summary

Queue consumers, Redis connection ownership, distributed locks, and scheduler implementations currently live inside @upstand/usecases:

• DeploymentWorker creates BullMQ workers and Redis connections and owns worker lifecycle.
• BackupRunWorker creates its own Redis connection, BullMQ worker, retry handling, lock-renewal timer, and scoped execution.
• NotificationDeliveryWorker creates and manages another BullMQ worker and Redis connection.
• BackupScheduler, GeneralScheduler, access-log cleanup, queue reconciliation, queue naming, and several lock helpers are also application-package runtime concerns.
• @upstand/usecases directly depends on @upstand/redis, bullmq, and croner.

The business capabilities—queue a deployment, trigger a backup, publish a notification, execute a scheduled action—belong in the application layer. BullMQ, Redis clients, cron timers, worker event listeners, process lifecycle, and reconnection behavior are runtime adapters.

Reviewed against master at 7f9e26400a474aa5a706d261f24fa2bf3d98ce8c.

Architectural impact

• Application tests and package consumers inherit Redis/BullMQ runtime dependencies.
• Queue technology is embedded in use-case modules, making replacement or alternate execution modes expensive.
• Worker lifecycle and graceful shutdown are split between the application package and the server host.
• Every worker independently configures connection and failure behavior, increasing operational drift.
• Business handlers cannot be executed synchronously or by another queue adapter without importing BullMQ-specific classes and job types.

Proposed direction

1. Define application ports for enqueueing deployment, backup, notification, and scheduled jobs.
2. Define typed job commands/events in a lightweight application or contracts module without BullMQ types.
3. Move BullMQ queue creation, Worker construction, Redis connections, worker event listeners, retry/backoff translation, and shutdown into @upstand/redis, @upstand/infrastructure, or a dedicated @upstand/runtime package.
4. Keep small application handlers such as ProcessDeploymentCommand, ExecuteBackupRunUseCase, and DeliverNotificationUseCase independent of queue transport.
5. Have the server host compose queue adapters with handlers and own start/readiness/drain/stop lifecycle.
6. Centralize Redis connection profiles, worker telemetry, stalled-job policy, and shutdown behavior.
7. Represent cron scheduling through a scheduler port; place Croner/timer implementations in runtime infrastructure.
8. Preserve idempotency, locking, retries, and reconciliation as explicit policies with adapter integration tests.

Incremental migration

1. Extract notification delivery first because its worker is a small transport wrapper.
2. Migrate backup processing and lock renewal.
3. Split deployment job processing from the BullMQ worker shell.
4. Move schedulers and reconciliation after queue command contracts stabilize.
5. Remove bullmq, croner, and @upstand/redis from @upstand/usecases dependencies.

Acceptance criteria

• No BullMQ Queue, Worker, Job, or DelayedError types appear in application-layer source.
• @upstand/usecases has no production dependency on BullMQ, Croner, or the concrete Redis package.
• Queue handlers can be invoked directly with typed commands in unit tests.
• One runtime module owns Redis worker connections, readiness, telemetry, and graceful shutdown.
• Queue names and payload schemas are versioned and validated at the adapter boundary.
• Existing retry, delayed-job, idempotency, lock-renewal, and reconciliation semantics are covered by integration tests.
• A non-BullMQ test adapter can enqueue and execute application commands.

Suggested tests

• Execute each command handler without Redis.
• Validate malformed and version-mismatched job payloads at the adapter boundary.
• Worker restart and graceful-drain integration tests.
• Redis interruption/reconnection behavior.
• Idempotent replay and stalled-job handling.
• Cron adapter invokes the same application command used by manual execution.