Migrate the dashboard to spaday and serve it outside the cluster - #176
Merged
Conversation
The dashboard was a Ray Serve FastAPI ingress holding the perspective tables, paired with a hand-rolled esbuild/pnpm frontend. Because the browser had to reach that ingress, it required an inbound port on the cluster, which is not always possible. Replace the frontend with spaday, spaday-perspective and spaday-webawesome, and move the table host out of the cluster by default. The tracker actor now buffers table operations that the dashboard pulls over Ray's existing connection, so the browser only talks to localhost and the cluster accepts no new traffic. UI state syncs over transports; perspective keeps its own websocket for bulk data. Two dashboard modes are available: RayTaskTracker(dashboard="local") served from the caller's process RayTaskTracker(dashboard="cluster") served from Ray Serve, as before Pin spaday's asset layout to "installed". Its source/installed detection keys off a js/ directory beside the package, and unrelated wheels create one in site-packages, which 404s the runtime. Build the Ray Serve app in __serve_build_asgi_app__ rather than passing it to ingress. Ray pickles an ingress app to ship it to the replica, which fails on the perspective server and the transports store. Drop js/, the hatch-js build hook and the NodeJS toolchain; raydar is now pure Python. Requires Python 3.11 for spaday-perspective, and moves perspective-python from 3.4 to 4.5. Remove PerspectiveRayServer, PerspectiveProxyRayServer and setup_proxy_server. Table creation and updates go through RayTaskTracker.create_table and update_table as before. Signed-off-by: Tim Paine <3105306+timkpaine@users.noreply.github.com>
CI failed importing raydar: ray.serve's haproxy module does `from jinja2 import Environment` at import time, but ray[serve] does not declare jinja2. raydar imports ray.serve at module level, so it has to declare jinja2 itself. Dropping jinja2 along with the Jinja2 templating went unnoticed locally because the old dependency was still installed. pandas has the same shape and was already wrong before this branch: it is imported at the top of task_tracker.py but was listed only under the develop extra, so `pip install raydar; import raydar` failed. Verified by installing the wheel into a fresh environment with only the declared dependencies. Signed-off-by: Tim Paine <3105306+timkpaine@users.noreply.github.com>
Contributor
Test Results36 tests +34 36 ✅ +34 29s ⏱️ -19s Results for commit 2b06f9e. ± Comparison against base commit 24417eb. This pull request removes 1 and adds 35 tests. Note that renamed tests count towards both.♻️ This comment has been updated with latest results. |
A poll-loop failure was the worst of these: `apply` was called outside the try block, so one bad row killed the task and froze the dashboard with no error surfaced and no way to recover short of a restart. Move it inside the guard and cover it with a regression test. Stop churning the synced model. `drain` replays schemas on every call and returns a truthy dict even when nothing happened, so every poll rewrote the timestamp and broadcast a patch to every browser twice a second. Apply now tracks whether anything actually changed. Report rows from Perspective rather than a lifetime counter. The counter ignored both `limit` and `clear`, so a capped table showed 5 when it held 2, and a cleared one still showed 5 when it held 0. Scope teardown to raydar's own Serve application. `exit()` called the global `ray.serve.shutdown()` even for local and disabled dashboards, which would take unrelated deployments on the cluster down with it. Make the local dashboard's lifecycle safe: `stop` is idempotent and releases the socket, an unstarted dashboard no longer leaks its bound port, and a thread that outlives its join is no longer forgotten, which previously let a second server bind the same socket. Warn instead of failing silently in two places: a non-loopback bind now says it serves task metadata unauthenticated, and an existing actor whose dashboard mode differs from the requested one now says so rather than presenting a permanently empty dashboard. Also dedupe pending clears, which accumulated unboundedly between drains, and assert real table contents in the tests that previously only checked table names. Signed-off-by: Tim Paine <3105306+timkpaine@users.noreply.github.com>
Perspective 4 encodes rows as JSON, which has no datetime, so passing a `datetime` raised `TypeError: Object of type datetime is not JSON serializable`. Perspective 3 accepted them, so this broke the pattern the README documents and any user code carrying it over. Coerce `datetime` and `date` to epoch millis for columns declared `datetime` or `date`, keeping the documented API working. Epoch *seconds* were the more dangerous case: Perspective accepts a float and reads it as millis, so `time.time()` silently rendered every row as 1970. The demo and the metrics example in the wiki both did this; they now pass datetimes. Coercion deliberately does not guess a bare number's unit. The existing tests only used integer and string columns, which is why none of this showed up. Cover the datetime path. Separately, the chrome was tinted: #0f172a against Perspective's dark viewer, which is a neutral #242526, read as purple. Match Perspective's own greys in both themes so the header and the grid look like one application. Signed-off-by: Tim Paine <3105306+timkpaine@users.noreply.github.com>
test_construction_and_dataframe slept a flat 30 seconds and then asserted on row 0, so it failed on CI with an empty frame when the GCS metadata had not landed yet. The tracker code was untouched between the passing and failing runs, so this was the sleep, not a regression. Poll for the condition with a generous ceiling instead. The test now finishes in about 3 seconds rather than always taking 30, and tolerates a slow runner instead of failing on one. The suite drops from roughly 67 to 36 seconds. Signed-off-by: Tim Paine <3105306+timkpaine@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The dashboard was a Ray Serve FastAPI ingress holding the perspective tables, paired with a hand-rolled esbuild/pnpm frontend. Because the browser had to reach that ingress, it required an inbound port on the cluster, which is not always possible.
Replace the frontend with spaday, spaday-perspective and spaday-webawesome, and move the table host out of the cluster by default. The tracker actor now buffers table operations that the dashboard pulls over Ray's existing connection, so the browser only talks to localhost and the cluster accepts no new traffic. UI state syncs over transports; perspective keeps its own websocket for bulk data.
Two dashboard modes are available:
RayTaskTracker(dashboard="local") served from the caller's process
RayTaskTracker(dashboard="cluster") served from Ray Serve, as before
Pin spaday's asset layout to "installed". Its source/installed detection keys off a js/ directory beside the package, and unrelated wheels create one in site-packages, which 404s the runtime.
Build the Ray Serve app in serve_build_asgi_app rather than passing it to ingress. Ray pickles an ingress app to ship it to the replica, which fails on the perspective server and the transports store.
Drop js/, the hatch-js build hook and the NodeJS toolchain; raydar is now pure Python. Requires Python 3.11 for spaday-perspective, and moves perspective-python from 3.4 to 4.5.
Remove PerspectiveRayServer, PerspectiveProxyRayServer and setup_proxy_server. Table creation and updates go through RayTaskTracker.create_table and update_table as before.