Workflow manager - #70
Open
rvanderwerf wants to merge 46 commits into
Open
Conversation
…e workflow engine Portal: workflow builder (React Flow canvas, node palette, config panel with per-field help), workflow store/versioning APIs, validation/compilation via the shared workflow_core package, Greengrass packaging, Bedrock-based generation with temperature control, and a Fargate-sandboxed cloud test runner with live progress and simulated inference outcomes. Edge: additive workflow_engine (discovery, executor via GstPipelineManager, dio/MQTT/OPC UA output bindings incl. AWS IoT mutual TLS, conditional two-path routing, custom Python bridge) with alembic migration; existing gstreamer pipeline path untouched. Includes 15 property-based test suites (hypothesis/fast-check), security baseline re-pins for the intentional IAM changes, and CORS fixes for gateway error responses.
…orts, simulator, lifecycle states, quality-tier risk display)
… (Req 16), x86_64_nvidia target, unified quality classification
…, Plugin_Component packaging, x86_64_nvidia)
Node Designer feature area (all 78 spec tasks) and the iteration that followed the first live imports and builds: - Plugin records/importer/builds/components/simulator Lambdas, custom node type registration, node-designer API routes, DynamoDB tables, KMS artifact signing, per-arch CodeBuild projects (x86_64, x86_64_nvidia, arm64_jp4/jp5/jp6 on L4T/JetPack containers) - Async repository import with plugin-set selection, module listing with descriptions/doc links, per-architecture source revisions (arch_revisions fan-out with per-slug fetch settling), advisory platform compatibility (GStreamer requirement vs platform versions) - Auto-start of queued builds when an import settles; build image entrypoint fallback to full-tree meson configure for release branches without per-plugin options; failed-build log excerpts centered on the real error instead of post-failure upload boilerplate - Plugin record delete (with S3 source/artifact cleanup and promoted guard), lifecycle promote/demote controls, duplicate-registration guard (register switches to update mode when the plugin already backs a node type, GET /custom-node-types?plugin_id=...) - Workflow Builder palette merge of registered custom node types (test/prod gated, version pinning), scaffold generation and Bedrock node generator, plugin simulator, workflow engine edge-side plugin loading/verification, test-sandbox simulation harness - Frontend: Node Designer library/detail/import/create/generate/ register/review/simulator views, retry and delete actions, per-arch revision inputs and labels Suites: backend 883, workflow_core 355+1s, frontend 423 + build, infrastructure tsc + 30 jest, test-sandbox 175, workflow_engine 204+3s.
The palette catalog endpoint only merges the Use_Case's registered Custom_Node_Types when the request carries usecase_id; the builder called it bare, so promoted custom nodes (test/prod) never reached the palette. The builder now passes the selected Use_Case and reloads the catalog when it changes.
Build-time introspection capture (dda-gst-introspect + Dockerfile.x86_64 + dda-plugin-build), gstIntrospection stanza recording in plugin_builds.py, gst-properties serving route, frontend scan module with ParameterScanPanel wired into both wizards, plus hypothesis/fast-check property tests (Properties 1-12), unit tests, and the container integration test. Spec: .kiro/specs/gst-parameter-prepopulation
Register fast (25 examples, default) and ci (100 examples, HYPOTHESIS_PROFILE=ci) profiles in the test/backend-test conftest and lower per-test max_examples overrides to the fast budget, mirroring the existing workflow_engine engine-fast profile and the frontend's global fast-check numRuns cap.
Implements the camera-registry-sync spec (all 73 tasks):
Edge (LocalServer):
- camera_discovery: V4L2/CSI enumeration behind an injectable ioctl
layer, periodic re-enumeration with absence tracking
- camera_sync: build_inventory merge, versioned state store, and the
EdgeSyncAgent reporting the full inventory over the
dda-camera-registry shadow with debounce/backoff, plus the
portal-change apply path (on_delta) through existing accessors
- workflow_engine: resolve_bindings, CameraBindingStore, and watcher
integration resolving dda-camera-bindings against the local inventory
with re-resolution on discovery/bindings changes
- server_setup wiring behind failure isolation; no schema changes
Portal backend:
- camera_sync.py: reduce_report sync reducer (version guard, ack
matching, edge-wins conflicts) and the SQS shadow-report ingest
- camera_registry.py: seven /devices/{id}/cameras routes (read,
mutate, conflict re-apply, refresh) with RBAC and audit events
- deployments.py: validate_camera_bindings, binding context view, and
camera-bindings shadow delivery with prune/rollback semantics
- workflow_packaging.py: bindingPoints emission and version-item
discriminator; workflow_core serializer preserves advisory node data
- staleness threshold setting on the existing settings API
Infrastructure (CDK): dda-portal-camera-registry table, shadow-report
queue + DLQ, camera sync/registry Lambdas, CameraRegistryApiStack
routes, and the cross-account IoT topic rule in use-case onboarding.
Frontend: device cameras tab, workflow builder camera picker with
advisory binding hints, and the CreateDeployment binding matrix.
All 22 design properties covered by hypothesis/fast-check tests
(25 examples locally, 100 via HYPOTHESIS_PROFILE=ci). Suites: portal
backend 1067, frontend vitest 505 + build, infra jest 43 + tsc,
LocalServer camera/workflow suites green.
…s, design, tasks)
…apping writes The adjustment paths (fetch-success handler and ADJUST_REUSE) write arch_revisions/fetches then auto-start queued builds in the same invocation. start_queued_builds re-read the record eventually- consistently, could miss the just-written mapping, and submitted the build from the flat source_s3_prefix (the original revision's tree), failing the first post-adjustment build deterministically. get_version_item gains an opt-in consistent_read parameter (default False: byte-identical call for all other callers); start_queued_builds requests consistent_read=True. Spec: adjust-revision-stale-read-fix (20 tests incl. 3 Hypothesis properties; full backend suite 1159 passed).
The sibling-spec preservation guard recorded the pre-custom-node-designer sha256 of edge-cv-portal/backend/functions/components.py. Refresh it to the current committed hash so the build-time security audit gate passes from a clean checkout.
…stment-fix spec docs
…etectionApplication into workflow_manager
…plugin-revision-adjustment-fix)
…import opt-in) Three independent fixes under the workflow-designer-bugfixes spec: - Bedrock generation omits temperature/top_p when unset: defaults are now None (sent only when explicitly configured or overridden), the settings API accepts and round-trips null sampling values, and the admin form treats blank fields as unset. - Node-designer wizards seed default ports from the selected palette category (input nodes: no input port, one VideoFrames output); untouched defaults rewrite on category change, edited rows preserved; ports step states per-kind requirements. - Module import selection defaults to none with an explicit opt-in gate; wire serialization and fallbacks unchanged. Each fix carries exploration + preservation property tests (hypothesis / fast-check). Full suites green: backend 1182, frontend 636, tsc clean.
- New custom_python_preprocess node type (Custom Python (Frames)): fixed VideoFrames in/out, preprocessing palette section, same emlpython bridge mapping as custom_python; vendored workflow_core mirror synced with a byte-equality smoke test. - Frame-based handler contract process_frame(frame, metadata): NumPy uint8 arrays derived from stream caps, shape/dtype enforcement with row-padding preservation; legacy handle(frame_bytes, metadata) contract preserved; metadata["frame"] carries width/height/format. - dda_frames helper module injected by the runner: to_array/to_bytes, frame_info(), load_image() from local disk or s3:// URIs. - cv2/np/numpy pre-bound in handler namespaces; OpenBLAS/OpenCV thread pools capped in handler subprocesses so numpy imports reliably under the 512 MB RLIMIT_AS on many-core hosts. - Packaging gathers both Custom Python node types; catalog contract docs corrected (process(data, metadata) never existed). - 12 correctness properties covered by hypothesis/fast-check tests. Suites: portal backend 1184, workflow_core 364, frontend 641 + build, edge workflow_engine 267 — all green.
…aravis-camera-input)
Adds an aravis_camera_source node type threaded through the existing
camera-registry-sync seams end to end:
- Catalog: new descriptor (camera_id/gain/exposure, appsrc-headed device
mappings, dataset-fed sim stub) in both mirrored workflow_core copies;
compiler resolves {nodeId} tokens in element args.
- Edge discovery: injectable Aravis enumeration (camera_discovery/aravis.py)
with deterministic stable ids, wired into the CameraDiscovery loop;
build_inventory merges configured Camera-type Image_Sources with
discovered bus cameras by cameraId and reports AravisDiscovered entries.
- Designer: the camera picker offers Aravis-compatible registry entries
for the node's camera_id, records cameraBindingHint.
- Packaging/deployment: aravisBinding binding points, Aravis type
compatibility in validate_camera_bindings, matrix filtering.
- Edge execution: resolve_bindings produces aravis_assignments; new
aravis_feed planner; WorkflowExecutor gains a binding-resolution
provider and an Aravis frame feed via camera_manager + appsrc.
15 correctness properties covered by hypothesis/fast-check tests. Final
checkpoint: portal backend 1206, workflow_core 374, frontend 660 + build,
edge suites 402 — all green; vendored mirror byte-identical.
Workflow designer bugfixes, custom Python frames, Aravis camera input
…d stale version item
… cmake in base images
…resources re-sync
- Retarget the generated ECR recipe's component name (ComponentName, access-control keys, lifecycle script paths) to the component being published instead of inheriting whatever recipe.yaml was last written for; alternating JP5/JP6 publishes no longer register under the wrong name (CreateComponentVersion ConflictException). - Construct the component ARN directly for portal-discovery tagging; the paginated list-components filter query returned None whenever the component sat past the first page, silently skipping the tag.
…ackend images The JP5/JP6/base backend Dockerfiles never copied the three packages added by the camera-registry-sync and workflow-manager specs, so LocalServer crashed at startup (ModuleNotFoundError: camera_discovery; uncaught workflow_engine import in app.py). Adds the COPY lines and recaptures the docker preservation baselines accordingly.
Bedrock-backed prompt-to-code assistance for custom Python node modules on the Workflow_Builder and Node_Designer surfaces: - bedrock_common.py: shared Bedrock configuration/client/inference-config extracted from workflow_generator.py (identical semantics, one source of truth); workflow_generator refactored to import it - code_assist.py: POST /code-assist with the 400 validation matrix, per-surface RBAC (+unauthorized_access audit), forced provide_code tool Converse call, AST entry-point validation, and the full error envelope mapping; dispatched from the workflow generator handler - API Gateway: top-level /code-assist POST on the existing WorkflowGeneratorHandler integration (no new Lambda) - frontend: CodeAssistPanel (pure reducer state machine, error presenter, api client) integrated into NodeConfigPanel for the custom Python node types and into the Node_Designer .py scaffold tabs; debounced Import_Analyzer deriving pip requirements from imports with derived/verify badges and manual-entry preservation - tests: 13 design properties covered (hypothesis + fast-check), component/unit suites; all baselines green (backend 1243, frontend 753, infra build + jest)
…etectionApplication into workflow_manager
- publish region us-east-2 -> us-east-1: the components, artifact bucket (dda-component-us-east-1), and ECR repos live in us-east-1; the stale us-east-2 value made 'gdk component publish' resolve NEXT_PATCH against an empty region and start over at 1.0.0 - drop the stray third 'aarch64' argument: build-custom.sh takes exactly COMPONENT-NAME COMPONENT-VERSION and exits with a usage error otherwise
…ployments Portal-created deployments never configured aws.greengrass.ShadowManager, so it ran with default config (no synchronize section) and the camera-registry-sync named shadows (dda-camera-registry device->cloud reports, dda-camera-bindings cloud->device bindings) never mirrored to IoT Core - devices showed 'no camera registry shadow' / 'Never synced'. create_deployment now auto-includes ShadowManager (unpinned, implicit LocalServer dependency) with a synchronize merge for both named shadows whenever a LocalServer component is deployed, skipping it when the caller supplies their own ShadowManager entry. Takes effect on the next deployment revision per device.
…run-ux) Cloud-built workflows deployed to a device were registered and runnable via the edge HTTP API but invisible in the LocalServer UI. Adds the missing surface, additively: - api/WorkflowRegistrationAPI.ts over the existing registrations/ executions endpoints - Deployed workflows list page (identity, status, invalid reason, empty state) and details page (Run control only for registered registrations, invalid alert, execution history with failure details, 2s polling while runs are active, backend 409 surfaced via the flashbar) - one new route subtree in App.tsx and one SideNav link; legacy Pipeline_Configuration pages and backend untouched - first Jest test infrastructure for src/frontend (setupTests, fast-check): exploration test (failed pre-fix, passes post-fix), preservation suite, unit + property tests - 46 tests green, production build clean
…ments CreateDeployment requires componentVersion on every component entry. Resolve newest Nucleus-compatible public ShadowManager version with a 2.3.15 fallback; harden FakeGreengrass to reject versionless entries.
… exchange role ShadowManager's cloud sync loop calls the IoT data plane (Get/Update/ DeleteThingShadow, ListNamedShadowsForThing) with token-exchange-role credentials. Without these actions the sync fails with ForbiddenException (403) and the dda-camera-registry / dda-camera-bindings named shadows never mirror to IoT Core - the Portal shows devices as 'Never synced' and camera refresh fails. Scoped to things in the device's account. Matches the ShadowManagerSyncPolicy applied by hand to account 164152369890.
The golden pins the full file bytes; the ShadowManagerSyncPolicy addition (a8a46c9) legitimately changed it. Recaptured via the suite's capture-on-absence mechanism. Preservation suite: 147 passed locally (3 pre-existing IAM synth drifts unrelated to this change, skipped in the build container).
The golden records the pre-fix requests==2.32.3 pin line by design (the comparison normalizes the version token); the recapture in 7a0b8c4 wrongly picked up the fixed 2.32.4 pin from the working tree. Preservation suite green on the affected tests.
…n (portal-user-manager) - Portal: PortalAdmin-only User Manager page (list/filter accounts, password change, forgot-password via SES temp password, role change with last-PortalAdmin guard), strict two-phase audit helpers - Account sync: per-device staged sync over dda-user-accounts named shadow, ack ingest via IoT rule -> SQS, 5-min retry/timeout sweep Lambda - Edge: local_auth (PBKDF2 credential cache, HMAC session tokens, lockout, config poller), authorize_request decision matrix, LoginGate web UI - Infra: ComputeStack tables/queues/rules/Lambdas + UserAdminApiStack routes behind the JWT authorizer
…e (portal-user-manager) - Backend: POST /admin/users (Cognito-native invitation), disable/enable endpoints with read-state-first no-op, DELETE with D13 ordering and partial verifier-cleanup handling; last-PortalAdmin guard applied to disable and delete; new account_* audit action types in the shared layer - Frontend: Create User, disable/enable confirm, and delete confirm modals wired into the UserManager page with matching API client methods - Infra: pool-scoped Cognito IAM grants extended with AdminCreateUser, AdminEnableUser, AdminDisableUser, AdminDeleteUser - Tests: 87 new backend unit tests (273 total user_admin), frontend suite at 815 tests, all green
…ages The portal-user-manager feature (b04e5ab) added src/backend/local_auth and src/backend/user_accounts_sync but no Dockerfile COPY lines - the same missing-module startup crash pattern as camera_discovery/camera_sync (83f936d). server_setup.py imports user_accounts_sync at startup. Recaptured the affected security preservation goldens (masked Dockerfile bytes, out-of-scope hashes for src/backend/Dockerfile and app.py). All six pattern audits and the preservation suite pass locally.
…etectionApplication into workflow_manager
…points
POST /admin/users, POST /admin/users/{username}/disable|enable, and
DELETE /admin/users/{username} were implemented in user_admin.py but never
registered in the UserAdminApiStack route table, so API Gateway returned
bare 403s for the new User Manager actions.
finalize_audit_event queries dda-portal-audit-log to locate the pending entry before updating it; the shared role helper only grants write, so every mutating User Manager action failed with AccessDeniedException after the Cognito operation succeeded.
The pool has email as an alias attribute, so a new username with an email already used by another account raises UsernameExistsException; the fixed 'username already exists' string misled administrators about the real conflict. 409 now carries Cognito's own message.
The pool requires given_name and family_name; the NEW_PASSWORD_REQUIRED form submitted only given_name, so first sign-in for User Manager-created accounts failed with 'Invalid attributes given, family_name is missing'.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.