Scout stale-cluster ARecord cleanup scoped by DNS zone

[prabhjotbawa]

Added a current_zone parameter to all four stale-selector builders (stale_arecord_label_selector, stale_httproute_arecord_label_selector, stale_tlsroute_arecord_label_selector,
stale_tcproute_arecord_label_selector) and their corresponding delete_stale_cluster_*_arecords functions, appending a zone=<current_zone> clause to the selector.

Zone is resolved per-call via resolve_zone() from the resource's annotations (falling back to ctx.default_zone), matching normal create-path zone resolution — including on the delete/opt-out cleanup paths, where
the zone is taken from the object being deleted.

Fixes: #474

[prabhjotbawa]

Code QL error fixed, Security Scan Passed as well

[ebourgeois]

Review: stale-cluster ARecord cleanup scoped by DNS zone

Thanks for this — the mechanical change is sound and consistently threaded. All four selector builders and all four delete_stale_cluster_*_arecords functions take current_zone, and all 12 call sites (4 create, 4 delete, 4 opt-out) resolve the zone from the right object's own annotations via resolve_zone. LABEL_ZONE is written by every build_*_arecord and predates the stale-cleanup feature, so there is no migration gap with unlabeled legacy records. CI is green.

Two things I'd like addressed before merge, plus one lower-severity item — all inline.

1. The fix doesn't cover the same-zone case, but the new rustdoc and user docs claim it does. The zone clause only disambiguates clusters publishing into different zones. Several clusters publishing into one shared zone — the common multi-cluster-DNS topology — still mutually clobber. The docs assert this limits cleanup "to true renames of the same physical Scout instance," which isn't true there.
2. Stale cleanup is now silently skipped on the delete and opt-out paths when no zone resolves, leaving permanently unreachable orphans. The None arm returns Ok(()), the finalizer is removed, and the object is gone — so no future reconcile can re-drive the cleanup. Pre-PR these paths deleted the records.
3. Low: a user-controlled annotation now feeds a delete-path label selector unvalidated, which can turn a previously-infallible cleanup into a 5-minute deletion stall.

Notes, not blocking

• delete_stale_cluster_arecords_deletes_only_returned_records mounts its GET mock without a labelSelector matcher, so it asserts nothing about zone scoping. That matches its stated purpose, but it isn't a second line of defense for the new clause.
• reconcile_service has no stale-cluster cleanup at all (only service_arecord_label_selector), so LoadBalancer Services are untouched by both the bug and the fix. Pre-existing and out of scope here — but relevant if #474 is to be closed as fully fixed.

[prabhjotbawa]

For completeness, review comments incorporated as below:

Fixed

• src/scout.rs, docs/src/guide/scout.md: the zone-scoping rustdoc and user docs claimed the fix "limits the stale-cleanup match to true renames of the some physical Scout instance," which is only true when the conflicting clusters publish into different zones. Several clusters publishing into one shared zone with matching namespace + resource name still mutually delete each other's live ARecords in a flapping loop. Added static_label_selector's doc and the "Changing the Cluster Name" guide section to state the actual quarantine and hint limitation explicitly, instead of overclaiming a fix this selector doesn't provide (needs an instance-level marker, which is out of scope here).

• src/scout.rs: on the delete and opt-out paths (Ingress/HTTPRoute/TLSRoute/TCPRoute), when no zone could be resolved for the object being removed, cleanup silently skipped stale-cluster deletion and the finalizer was released anyway — unlike the reconcile path, there is no future reconcile to retry, so the stale ARecord was orphaned permanently with no trace. Added log_unscoped_stale_cluster_arecord_candidates, called ARecords from other cluster names sharing the object's namespace + name, unscoped by zone (there being no zone to scope by), and logs their names so an operator can act on them. It never fails its caller — list errors are logged and swallowed, since it is a best-effort diagnostic, not part of the cleanup itself.

• src/scout.rs: the resolved bindy.firestoned.io/zone annotation reached the delete-path labelSelector unvalidated, unlike validate_record_name_override, which exists for the same class of annotation. A DNS-idiomatic but label-illegal value (trailing dot, over 63 chars) mode the API server reject the list call with a 400, stalling Ingress/Route deletion for full REMOTE_CLEANUP_GRACE_SECS (300s) grace period before the finalizer and filtered the zone through it before use in all 8 delete/opt-out call sites; an invalid zone now falls through to the same no-usable-zone path as a missing one (warn + log orphan candidates) instead of reaching the selector.

• src/scout.rs: 4 delete-path call sites (deleting_annotations for Ingress, HTTPRoute, TLSRoute, TCPRoute) cloned the whole annotations BTreeMap only to have resolve_zone read one key and drop it. Replaced with a borrowed EMPTY_ANNOTATIONS static fallback — static, not const, because a const re-materializes a fresh temporary at each use site that cannot outlive the borrowing statement.

• src/scout_tests.rs: added unit tests for is_valid_zone_label_value (accepts plain/hyphenated/63-char zones; rejects empty, leading/trailing dot, over-length, illegal characters), wiremock-backed tests for log_unscoped_stale_cluster_arecord_candidates (unscoped selector, never issues a DELETE, swallows list errors), and a test proving that stripping every bindy.firestoned.io/* annotation in one edit removes the opt-in at the zone simultaneously — the scenario that sends stale-cluster cleanup down the new no-usable-zone path instead of a normal zone-scoped delete.