Skip to content

Decide whether the 8 macOS-skipped daemon-mode tests should be re-enabled (or the guard made explicit) #305

Description

@StefanSteiner

Decision needed

Eight daemon-mode integration tests in hyperdb-mcp/tests/daemon_tests.rs are skipped on macOS via:

#[cfg_attr(target_os = "macos", ignore = "flaky on macOS CI — daemon startup exceeds 150s timeout")]

They run on Linux and Windows CI — macOS is the only platform where they don't. This issue is to decide whether that guard is still warranted or should be lifted, rather than leaving it as a silent cfg_attr.

The eight:

  • daemon_mode_engine_connects_to_shared_hyperd
  • daemon_mode_two_engines_share_same_hyperd
  • daemon_mode_persistent_database_file_survives_engine_drop
  • daemon_mode_persistent_engine_data_is_queryable
  • hyperd_monitor_detects_killed_hyperd_and_restarts
  • client_report_triggers_restart_after_kill
  • engine_recovers_after_hyperd_killed
  • daemon_mode_ephemeral_database_cleaned_up_on_drop

Why it matters

These cover the daemon's core value: shared-hyperd reuse across engines, persistence across engine drops, crash detection and restart, and ephemeral cleanup. Until recently all eight were unconditionally #[ignore]d, so the daemon shipped resident-by-default for months with no automated crash-recovery coverage. Re-enabling them on Linux/Windows immediately turned CI red and exposed a real endpoint-publish-ordering bug (fixed in #286). So the coverage is load-bearing — and macOS is currently the one tier not getting it.

The evidence on both sides

For lifting the guard: on an Apple M3 Max these run in ~15–22s each, nowhere near the 150s budget. The reason string blames "daemon startup exceeds 150s timeout," which does not reproduce on a normal local macOS run.

For keeping it: the flakiness was observed specifically on shared/loaded CI runners. When macOS-ignored tests were forced to run under load (load average ~4.8, another build in progress), daemon_mode_engine_connects_to_shared_hyperd failed with "TestDaemon did not start within 150s". macOS GitHub-hosted runners are slower and more contended than the Linux ones, so a budget that is generous locally can still be tight there.

Options

  1. Lift the guard, raise the budget. Re-enable on macOS with a longer or adaptive startup timeout (scale the 150s wait, or key it off observed CI slowness) so all three platforms get crash-recovery coverage. Risk: reintroduces macOS CI flakiness if the real problem is variance rather than the absolute budget.
  2. Investigate the macOS startup cost first. Determine why daemon+hyperd startup can exceed 150s on macOS runners (cold binary, code-signing/quarantine checks on first exec, runner I/O) before changing the guard. Most likely to produce a durable fix.
  3. Keep the guard as a deliberate tradeoff, but say so explicitly in the reason string — "skipped on macOS CI runners due to startup variance; covered on Linux and Windows" — so it reads as a decision, not a suspected-stale workaround.

Recommendation: option 2 then 1 — measure the macOS startup path before widening the budget, since a 150s timeout that is already 7× the local runtime suggests the failure is a stall, not a slow-but-completing start.

Related recent daemon-test reliability items

Provenance

Raised while auditing the ignored-test surface. Verified against main at 8760df2: the eight cfg_attr(target_os = "macos", ignore) guards are at daemon_tests.rs:14691718.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions