feat: data export utility, request validation middleware, dependency …

[whiteghost0001]

…vulnerability check, and CI test database

Resolves Issues #850, #851, #855, and #859 simultaneously:

1. Add Data Export Utility (closes Add Data Export Utility #850):

   • What was done: - Implemented an administrative data export service (DataExportService) in listener/src/services/data-export-service.ts for querying and exporting scheduled notifications (scheduled_notifications) and blockchain events (processed_events). - Added comprehensive filtering capabilities supporting status, channel/notification type, recipient substring, contract address, event type, date ranges (fromDate/toDate), limit, and offset. - Added support for documented export formats: standardized JSON structure with metadata envelope and RFC 4180 compliant CSV. - Built security-first redaction handling: automatically redacts secrets, tokens, credentials, API keys, webhook URLs with embedded tokens, and recipient emails via redactValue and sanitizeRecipient, with an explicit administrative --include-sensitive override. - Created CLI utility script at listener/src/scripts/export-data.ts and registered npm run export:data. - Exposed administrative endpoints GET /api/admin/export & POST /api/admin/export (and /api/export) on events-server.ts, secured with API key authentication. - Authored documentation in docs/DATA_EXPORT_UTILITY.md.
   • How it was done:
     • Leveraged parameterized SQLite queries against scheduled_notifications and processed_events.
     • Recursively walked payload objects using redactValue to mask sensitive keys (token, secret, apiKey, password, etc.) and regex sanitized webhook URLs. - Formatted tabular output into RFC 4180 CSV with escaped quotes and JSON objects.

2. Add Request Validation Middleware (closes Add Request Validation Middleware #851):

   • What was done: - Introduced centralized request validation middleware in listener/src/middleware/request-validator.ts to validate incoming requests before reaching business logic. - Created declarative schemas (Schemas.scheduleNotification, Schemas.createTemplate, Schemas.renderTemplate, Schemas.batchValidate, Schemas.dataExport, Schemas.updatePreferences). - Standardized error response format conforming to NotifyChain API response specifications (utils/response.ts), returning { success: false, error: { code, message, details } }. - Enhanced handleApiError in listener/src/api/error-handler.ts to catch ValidationError instances and format standard HTTP 400 responses with field-level issues. - Maintained full backward compatibility for existing valid requests and legacy client error expectations (MISSING_FIELDS, INVALID_DATE). - Authored documentation in docs/REQUEST_VALIDATION_MIDDLEWARE.md.
   • How it was done:
     • Built validatePayload and streaming body validator parseAndValidateBody with payload size inspection, JSON syntax error recovery, and field-type validation.
     • Integrated validation checks into POST /api/schedule, POST /api/templates, and administrative export endpoints on events-server.ts.

3. Add Dependency Vulnerability CI Check (closes Add Dependency Vulnerability CI Check #855):

   • What was done: - Added automated dependency vulnerability check workflow in .github/workflows/dependency-check.yml. - Configured automated triggers on pushes, pull requests to main and staging, scheduled weekly scans (Mondays 06:00 UTC), and manual workflow_dispatch. - Made vulnerability findings visible in CI through detailed console logs, structured markdown summary tables in $GITHUB_STEP_SUMMARY, and downloadable JSON report artifacts (reports/dependency-audit/). - Enforced strict credential security: restricted permissions to permissions: contents: read, eliminated all secret token injections, and executed scans in isolated read-only modes. - Created helper utility scripts/audit-dependencies.js to execute audits across workspaces (listener, dashboard, frontend, contract) and format step summaries. - Authored documentation in docs/DEPENDENCY_VULNERABILITY_CI_CHECK.md.
   • How it was done:
     • Orchestrated npm audit --json across package workspaces and checked Cargo lockfile dependencies.
     • Parsed audit metadata, calculated severity counts (Critical, High, Moderate, Low, Info), and appended GitHub step summary markdown tables.

4. Add CI Test Database Environment (closes Add CI Test Database Environment #859):

   • What was done: - Created reproducible test database provisioning and teardown utility in listener/src/test-utils/test-db-environment.ts. - Enabled automated database provisioning (provisionTestDatabase) that purges prior database files to guarantee a clean initial state. - Automated migration execution: runs SQLite baseline schema and executes all pending incremental migrations in order via MigrationRunner. - Verified clean state by asserting that tables exist and contain zero records prior to test runs. - Created standalone CLI scripts listener/src/scripts/setup-test-db.ts (npm run db:test:setup) and listener/src/scripts/clean-test-db.ts (npm run db:test:clean). - Integrated test database provisioning and integration testing in .github/workflows/ci.yml. - Authored documentation in docs/CI_TEST_DATABASE_ENVIRONMENT.md.
   • How it was done:
     • Implemented removeDatabaseFiles to delete database, -wal, -shm, and -journal files before and after runs.
     • Programmatically instantiated Database and MigrationRunner, applying migrations atomically within transactions and verifying schema state.

Closes #850
Closes #851
Closes #855
Closes #859

Overview

Related Issue

Closes #

Changes

Verification

# e.g.
cd dashboard && npm test
cd listener && npm run typecheck && npm test
cd contract/contracts/hello-world && cargo test

How to Test

Checklist

• Branch is up to date with main
• Tests added/updated and all pass locally
• cargo fmt --all run (if Rust changes)
• npm run lint passes (if TypeScript changes)
• Documentation updated if behavior changed

[drips-wave]

@whiteghost0001 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits