Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ This file collects architecture and design notes for studio that aren't otherwis
- [`src/features/instance/config/roles/DESIGN.md`](src/features/instance/config/roles/DESIGN.md) — why the role editor's JSON text is the only source of truth, what forces the JSON view, and what Save changes on purpose.
- [`src/router/DESIGN.md`](src/router/DESIGN.md) — why a failed route load is reported and shown only by its route, and why a preload still toasts.

- [`docs/status-redesign.md`](docs/status-redesign.md) — port of analytics-viz visualizations + spec system into the instance Status tab. Covers: 7-tab IA, the data-path adapter (no SQL — `get_analytics` only), bucket-by-window clamps, chart-surface CSS tokens, feature-flag rollback, testing strategy. Read before touching `src/features/instance/status/**`.
- [`docs/status-redesign.md`](docs/status-redesign.md) — port of analytics-viz visualizations + spec system into the instance Status tab. Covers: 8-tab IA, the data-path adapter (no SQL — `get_analytics` only), the Jobs tab's own poll and permission gate, bucket-by-window clamps, chart-surface CSS tokens, feature-flag rollback, testing strategy. Read before touching `src/features/instance/status/**`.

## API explorer (the instance/cluster "APIs" tab)

Expand Down
16 changes: 14 additions & 2 deletions docs/status-redesign.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@ The instance Status tab is a spec-driven analytics dashboard built on top of Har

## Tabs

`Health → Traffic → Requests → Database → Replication → Storage → Overview` (Health is the default landing tab).
`Health → Traffic → Requests → Database → Replication → Storage → Overview → Jobs` (Health is the default landing tab).

Tab + time-range state is mirrored to `?tab=&range=&refresh=` so URLs are shareable. Below `md` the tab strip collapses to a `Select`. The time-range picker is a sticky sub-toolbar inside each tab (hidden on Overview).
Tab + time-range state is mirrored to `?tab=&range=&refresh=&jobsRefresh=` so URLs are shareable. Below `md` the tab strip collapses to a `Select`. The time-range picker is a sticky sub-toolbar inside each chart tab (hidden on Overview and Jobs, which has its own toolbar).

## Data Path

Expand All @@ -18,6 +18,18 @@ All analytics data flows through one query:
- **Capability probe:** `useAnalyticsCapability` runs one cheap probe per instance on mount. While loading, the Status tab shows an inline message; on error, it surfaces an "analytics unavailable" message (no legacy fallback).
- **Cap:** server responses larger than 50 000 rows are tail-truncated client-side with a console warning.

## Jobs tab

Jobs lists Harper's background jobs (CSV/S3 loads, exports, bulk deletes, backups) from `search_jobs_by_start_date` — `src/features/instance/status/jobs/` plus `src/integrations/api/instance/status/searchJobs.ts`. It reads no `get_analytics`, so like Overview it sits in `ANALYTICS_FREE_TABS` in `StatusTabs.tsx`: the capability probe neither gates its body nor redirects a `?tab=jobs` deep link.

- **Its own poll, not the window tick.** The analytics tick bakes `[start, end]` into every key; Jobs keys on the window _width_ and computes the bounds in the query function, polling with `pollUnlessForbidden(jobsRefresh)` so a 403 stops the timer. It shares the `range` search param and the option lists, but not `TimeRangePicker`, whose resolution sub-label and freshness stamp describe analytics queries only. The StatusTabs tick keeps running on Jobs on purpose: stopping it would leave the charts on a stale window when the user switches back.
- **Auto-refresh off by default, in its own param.** Each poll makes Harper run an unlimited `select * from system.hdb_job` over the window (harper `server/jobs/jobs.ts`), so Jobs fetches when it opens and on Refresh now, and polls only on an interval picked for it — not on refocus or reconnect either while off. That interval is `jobsRefresh`, not the charts' `refresh`: one value cannot default to 60s on the charts and to off on Jobs without a tab switch either starting Jobs polling or resetting the charts' choice. Every navigation carries both unchanged. Push updates are HarperFast/harper#3179.
- **Unbounded by the server.** The operation takes no limit or cursor, so every poll fetches the whole window. Jobs therefore offers windows up to 7 days (a `range=30d` link reads as 7 days there), and renders at most `MAX_RENDERED_JOBS` of the newest rows, saying how many it held back.
- **Per node.** `hdb_job` is a non-replicating system table (harper `resources/databases.ts`), so a cluster route lists the jobs of whichever node its request reached, and says so.
- **Super user only, by the role catalog's rule.** The operation is `requires_su`; `canListJobs` honours an allowlist grant only where the role editor's `gateVerdict` says a listing grants it for the instance's version band (5.3.0-beta.3 on), so the tab and the role editor never disagree. The version comes from the `registration_info` the instance route already loaded; if a refetch of it fails, a listed grant is tried rather than refused, and a 403 shows the same explanation.

The import modal's CSV progress toast uses the same job records: `waitForJob` reports each poll through `onUpdate`, and the toast is settled from `useImportDataMutation`'s own callbacks, which still run after the modal unmounts. Harper writes nothing to a job between starting and finishing it, so the toast shows state and elapsed time, never a percentage.

## Pipeline + Primitives

Ported from the analytics-viz project. Key directories under `src/features/instance/status/analytics/`:
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
import { afterEach, describe, expect, it, vi } from 'vitest';
import { createImportProgressToast, describeImportJob, IMPORT_PROGRESS_TITLE } from './importProgressToast';

const toast = vi.hoisted(() => ({
loading: vi.fn((_title: string, data?: { id?: string | number }) => data?.id ?? 'toast-1'),
success: vi.fn(),
dismiss: vi.fn(),
}));

vi.mock('sonner', () => ({ toast }));

afterEach(() => {
vi.clearAllMocks();
});

describe('describeImportJob', () => {
it('reads a job no worker has picked up yet as queued', () => {
expect(describeImportJob({ id: 'j1', status: 'CREATED' }, 2_000)).toBe('Queued · 2s · job j1');
});

it('reads any other unfinished job as running', () => {
expect(describeImportJob({ id: 'j1', status: 'IN_PROGRESS' }, 75_000)).toBe('Running · 1m 15s · job j1');
});
});

describe('createImportProgressToast', () => {
it('opens one loading toast on the first job update and keeps updating that same toast', () => {
let now = 1_000;
const progress = createImportProgressToast(() => now);

now = 1_500;
progress.onJobUpdate({ id: 'j1', status: 'CREATED' });
now = 4_000;
progress.onJobUpdate({ id: 'j1', status: 'IN_PROGRESS' });

expect(toast.loading).toHaveBeenNthCalledWith(1, IMPORT_PROGRESS_TITLE, {
id: undefined,
description: 'Queued · 500 ms · job j1',
});
expect(toast.loading).toHaveBeenNthCalledWith(2, IMPORT_PROGRESS_TITLE, {
id: 'toast-1',
description: 'Running · 3s · job j1',
});
});

it('turns the progress toast into the result on success', () => {
const progress = createImportProgressToast(() => 0);
progress.onJobUpdate({ id: 'j1', status: 'IN_PROGRESS' });
progress.onSuccess({ message: 'successfully loaded 10 of 10 records' });

expect(toast.success).toHaveBeenCalledWith('successfully loaded 10 of 10 records', {
id: 'toast-1',
description: undefined,
});
});

it('reports the result in a toast of its own when the import ran no job', () => {
const progress = createImportProgressToast(() => 0);
progress.onSuccess({ message: 'Inserted 3 records' });

expect(toast.loading).not.toHaveBeenCalled();
expect(toast.success).toHaveBeenCalledWith('Inserted 3 records', { id: undefined, description: undefined });
});

it('dismisses the progress toast on failure, leaving the error to the global handler', () => {
const progress = createImportProgressToast(() => 0);
progress.onJobUpdate({ id: 'j1', status: 'IN_PROGRESS' });
progress.onError();

expect(toast.dismiss).toHaveBeenCalledWith('toast-1');
expect(toast.success).not.toHaveBeenCalled();
});

it('dismisses nothing on a failure that came before any job', () => {
createImportProgressToast(() => 0).onError();
expect(toast.dismiss).not.toHaveBeenCalled();
});
});
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
import type { HarperJob } from '@/integrations/api/instance/database/getJob';
import type { ImportDataResult, ImportProgressListener } from '@/integrations/api/instance/database/importData';
import { formatDuration } from '@/lib/formatDuration';
import { toast } from 'sonner';

export const IMPORT_PROGRESS_TITLE = 'Importing data…';

/**
* The progress line for a polled load job. Harper writes nothing to a job between starting and
* finishing it, so its state and the time since submission are all there is to report.
*/
export function describeImportJob(job: HarperJob, elapsedMs: number): string {
const state = job.status?.toUpperCase() === 'CREATED' ? 'Queued' : 'Running';
return `${state} · ${formatDuration(elapsedMs)} · job ${job.id}`;
}

/** On failure the progress toast is only dismissed: the global mutation handler already toasts the error. */
export function createImportProgressToast(now: () => number = Date.now): ImportProgressListener {
const submittedAt = now();
let toastId: string | number | undefined;
return {
onJobUpdate(job: HarperJob) {
toastId = toast.loading(IMPORT_PROGRESS_TITLE, {
id: toastId,
description: describeImportJob(job, now() - submittedAt),
});
},
onSuccess({ message }: ImportDataResult) {
toast.success(message, { id: toastId, description: undefined });
},
onError() {
if (toastId !== undefined) {
toast.dismiss(toastId);
}
},
};
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
/**
* @vitest-environment jsdom
*/
import { ImportDataModal } from '@/features/instance/databases/modals/ImportDataModal';
import type { ImportDataParams } from '@/integrations/api/instance/database/importData';
import { cleanup, fireEvent, render, screen, waitFor } from '@testing-library/react';
import { afterEach, describe, expect, it, vi } from 'vitest';

const mutate = vi.hoisted(() => vi.fn());
const toast = vi.hoisted(() => ({
error: vi.fn(),
loading: vi.fn(() => 'toast-1'),
success: vi.fn(),
dismiss: vi.fn(),
}));

vi.mock('@tanstack/react-router', () => ({
useParams: () => ({ instanceId: 'instance-1' }),
}));

vi.mock('@/hooks/useAuth', () => ({
useInstanceAuth: () => ({ user: { role: { permission: { super_user: true } } } }),
isAdminMode: () => false,
useCloudAuth: () => ({ user: null }),
}));

vi.mock('@/config/useInstanceClient', () => ({
useInstanceClientIdParams: () => ({ instanceClient: {}, entityId: 'entity-1', entityType: 'instance' }),
}));

vi.mock('@/integrations/api/instance/database/importData', () => ({
useImportDataMutation: () => ({ mutate, isPending: false }),
}));

vi.mock('sonner', () => ({ toast }));

vi.mock('@/features/instance/config/roles/operations/useOperationsAllowlistSupported', () => ({
useOperationsAllowlistSupported: () => true,
}));

afterEach(() => {
cleanup();
vi.clearAllMocks();
});

const withTable = { data: { dog: { attributes: [{ attribute: 'id', is_primary_key: true }] } } };

describe('ImportDataModal progress', () => {
it('hands the import a progress toast and leaves the success toast to it', async () => {
const onImported = vi.fn();
render(
<ImportDataModal
isModalOpen
setIsModalOpen={() => undefined}
instanceDatabaseMap={withTable as never}
databaseName="data"
tableName="dog"
initialMethod="url"
onImported={onImported}
/>,
);

fireEvent.change(screen.getByPlaceholderText('https://example.com/data.csv'), {
target: { value: 'https://example.com/dogs.csv' },
});
fireEvent.click(screen.getByRole('button', { name: 'Import Data' }));

await waitFor(() => expect(mutate).toHaveBeenCalledTimes(1));
const [variables, callbacks] = mutate.mock.calls[0] as [
ImportDataParams,
{ onSuccess: (result: { message: string }) => void },
];
expect(variables.source).toEqual({ kind: 'csv-url', url: 'https://example.com/dogs.csv' });

variables.progress!.onJobUpdate({ id: 'job-1', status: 'IN_PROGRESS' });
expect(toast.loading).toHaveBeenCalledTimes(1);

callbacks.onSuccess({ message: 'loaded 5 of 5' });
expect(onImported).toHaveBeenCalledWith('data', 'dog');
expect(toast.success).not.toHaveBeenCalled();
});
});
5 changes: 3 additions & 2 deletions src/features/instance/databases/modals/ImportDataModal.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ import {
generateRandomRecords,
randomizableAttributes,
} from '@/features/instance/databases/functions/generateRandomRecords';
import { createImportProgressToast } from '@/features/instance/databases/functions/importProgressToast';
import { parseJsonRecords } from '@/features/instance/databases/functions/parseJsonRecords';
import { SchemaRelationshipMap } from '@/features/instance/databases/functions/schemaRelationships';
import { sampleDatasets } from '@/features/instance/databases/sampleDatasets';
Expand Down Expand Up @@ -312,11 +313,11 @@ export function ImportDataModal({
tableExists,
replicated: instanceParams.entityType === 'cluster',
source,
progress: createImportProgressToast(),
...instanceParams,
},
{
onSuccess: ({ message }) => {
toast.success(message);
onSuccess: () => {
closeModal();
onImported(database, table);
},
Expand Down
43 changes: 29 additions & 14 deletions src/features/instance/status/analytics/StatusTabs.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import { ANALYTICS_QUERY_KEY_PREFIX } from '@/integrations/api/instance/status/g
import { useQueryClient } from '@tanstack/react-query';
import { useNavigate, useSearch } from '@tanstack/react-router';
import { type ReactNode, useCallback, useEffect, useMemo, useRef, useState } from 'react';
import { JobsTab } from '../jobs/JobsTab';
import { type StatusTabId, validateStatusSearch } from '../statusSearch';
import { AnalyticsOnboardingHint } from './components/AnalyticsOnboardingHint';
import { TimeRangePicker } from './components/TimeRangePicker';
Expand Down Expand Up @@ -35,10 +36,14 @@ const TAB_DEFS = [
{ id: 'replication', label: 'Replication' },
{ id: 'storage', label: 'Storage' },
{ id: 'overview', label: 'Overview' },
{ id: 'jobs', label: 'Jobs' },
] as const satisfies readonly { id: StatusTabId; label: string }[];

type TabId = StatusTabId;

/** Tabs that read no `get_analytics` data, so the capability probe neither gates nor redirects them. */
const ANALYTICS_FREE_TABS: ReadonlySet<TabId> = new Set(['overview', 'jobs']);

export function StatusTabs({ instanceParams, isLocalStudio }: Props) {
// The capability probe only gates the analytics-backed chart tabs —
// Overview reads get_status / get_system_information and must stay
Expand Down Expand Up @@ -106,7 +111,10 @@ function StatusTabsInner({ instanceParams, isLocalStudio, capability }: InnerPro
// only narrows the `strict: false` typing to the same single source of
// truth (statusSearch.ts) instead of hand-rolled casts.
const raw = useSearch({ strict: false });
const { tab, range: presetId, refresh: refreshMs } = validateStatusSearch(raw as Record<string, unknown>);
// `jobsRefresh` is only Jobs' own interval; every navigation here passes it through unchanged.
const { tab, range: presetId, refresh: refreshMs, jobsRefresh } = validateStatusSearch(
raw as Record<string, unknown>,
);

// The analytics window is a fixed [start, end] snapshot baked into every
// panel's query key, so "refresh" means sliding the window forward to a
Expand Down Expand Up @@ -155,16 +163,16 @@ function StatusTabsInner({ instanceParams, isLocalStudio, capability }: InnerPro
}, [refreshMs, refresh, queryClient, tick, instanceParams.entityId]);

const updatePreset = useCallback((id: TimePresetId) => {
void navigate({ to: '.', search: { tab, range: id, refresh: refreshMs } });
}, [navigate, tab, refreshMs]);
void navigate({ to: '.', search: { tab, range: id, refresh: refreshMs, jobsRefresh } });
}, [navigate, tab, refreshMs, jobsRefresh]);

const updateTab = useCallback((id: TabId) => {
void navigate({ to: '.', search: { tab: id, range: presetId, refresh: refreshMs } });
}, [navigate, presetId, refreshMs]);
void navigate({ to: '.', search: { tab: id, range: presetId, refresh: refreshMs, jobsRefresh } });
}, [navigate, presetId, refreshMs, jobsRefresh]);

const updateRefreshMs = useCallback((ms: number) => {
void navigate({ to: '.', search: { tab, range: presetId, refresh: ms } });
}, [navigate, tab, presetId]);
void navigate({ to: '.', search: { tab, range: presetId, refresh: ms, jobsRefresh } });
}, [navigate, tab, presetId, jobsRefresh]);

// When the probe fails while a chart tab is selected, land on Overview so
// the page opens on live data instead of an error. One-shot per failure —
Expand All @@ -179,10 +187,14 @@ function StatusTabsInner({ instanceParams, isLocalStudio, capability }: InnerPro
}
if (landedOnOverviewRef.current) { return; }
landedOnOverviewRef.current = true;
if (tab !== 'overview') {
void navigate({ to: '.', search: { tab: 'overview', range: presetId, refresh: refreshMs }, replace: true });
if (!ANALYTICS_FREE_TABS.has(tab)) {
void navigate({
to: '.',
search: { tab: 'overview', range: presetId, refresh: refreshMs, jobsRefresh },
replace: true,
});
}
}, [capability.error, tab, navigate, presetId, refreshMs]);
}, [capability.error, tab, navigate, presetId, refreshMs, jobsRefresh]);

// The window snapshot lives in its own memo keyed ONLY on the preset and
// the refresh tick. Nothing else may mint a new window: it is baked into
Expand All @@ -209,7 +221,7 @@ function StatusTabsInner({ instanceParams, isLocalStudio, capability }: InnerPro
syncId: `${instanceParams.entityId}:${tab}`,
}), [timeRange, bucketMs, instanceParams, tab]);

const showTimePicker = tab !== 'overview';
const showTimePicker = !ANALYTICS_FREE_TABS.has(tab);
const picker = showTimePicker
? (
<TimeRangePicker
Expand Down Expand Up @@ -242,10 +254,10 @@ function StatusTabsInner({ instanceParams, isLocalStudio, capability }: InnerPro
<AnalyticsProvider value={ctxValue}>
<Tabs value={tab} onValueChange={(v) => updateTab(v as TabId)} className="px-4 py-2">
{
/* Hint applies to chart interactions; hide it on Overview which
has none of those affordances. */
/* Hint applies to chart interactions; hide it on the tabs that
have none of those affordances. */
}
{tab !== 'overview' && !capability.isLoading && !capability.error && <AnalyticsOnboardingHint />}
{!ANALYTICS_FREE_TABS.has(tab) && !capability.isLoading && !capability.error && <AnalyticsOnboardingHint />}
{
/*
* Tab strip layout:
Expand Down Expand Up @@ -293,6 +305,9 @@ function StatusTabsInner({ instanceParams, isLocalStudio, capability }: InnerPro
<TabsContent value="overview">
<OverviewTab instanceParams={instanceParams} isLocalStudio={isLocalStudio} />
</TabsContent>
<TabsContent value="jobs">
<JobsTab instanceParams={instanceParams} />
</TabsContent>
</Tabs>
</AnalyticsProvider>
);
Expand Down
Loading
Loading