n8n failure triage: Event Router HMAC secret missing + Ops Health Check auth (10/14 errors)

[labgadget015-dotcom]

Summary

24h n8n failure sweep (2026-08-22) shows 14 error executions across 6 workflows. Two are CRITICAL and block the core GadgetLab pipeline; the other four are isolated side-project/transient issues. This issue tracks the triage + the credential/secret fixes required to clear 10 of 14 failures.

Root causes pulled directly from the n8n Cloud executions API (GET /api/v1/executions?status=error) + per-execution includeData=true node-level errors, cross-checked against repo webhook config via gh api.

CRITICAL — fix required (blocks DRC pipeline)

#1 GitHub Event Router rejecting ALL inbound webhooks (6/14)

• Workflow: GitHub Event Router — Agent Loop Intake (oijizIGJtRzBG94Z, active). Webhook: POST /webhook/github-events (repo hook id 639039220, events: issues, pull_request).
• Node: Validate HMAC Signature (code node, id 8b1b2330-58d0-4e1c-9b05-9019db5b0955)
• Error: [HMAC] GITHUB_WEBHOOK_SECRET not set in $env or $vars. Rejecting ALL webhook traffic.
• IMPORTANT CORRECTION (2026-08-22): the GitHub repo webhook (gh api repos/.../hooks) shows secret SET: False — the webhook was created WITHOUT a secret. So the fix is a two-end match, not just setting n8n:
  1. GitHub repo → Settings → Webhooks → the github-events hook → set a Secret (generate a strong random string).
  2. n8n → Settings → Env Vars (or the workflow's Variables) → add GITHUB_WEBHOOK_SECRET with the exact same string.
  • If only one end has the secret, the node still rejects (missing header, or signature mismatch). Both must match.
• Node already checks $env first then $vars — no code change needed. Publish the Event Router after step 2.
• Impact: every GitHub issue/push event is dropped → DRC Agent Loop never triggers.

#2 Ops Health Check auth failure (4/14)

• Workflow: Gadget Lab — Ops Health Check (4vFBQOwaB7enaopA)
• Node: Check Event Router (HTTP Request → n8n management API /api/v1/workflows/oijizIGJtRzBG94Z)
• Error: Authorization failed - please check your credentials
• Credential: Header Auth account (id FEYPjuMfn7JuXQ2C)
• INDEPENDENT of ⚠️ Test Coverage Below Threshold - 2025-11-27 #1 — pings the n8n mgmt API, not the GitHub webhook.
• Verified: the API key in ~/.config/n8n_watchdog/api_key DOES authenticate to this exact endpoint (GET returned 200). So the fix is to point Header Auth account at that same valid key.
• Fix (requires Gadget's hands — credential): edit Header Auth account credential → set Header Name X-N8N-API-KEY, Header Value = the key in ~/.config/n8n_watchdog/api_key. Save.

Fixing #1 + #2 clears 10 of 14 (71%) failures.

MODERATE — backlog (isolated, lower priority)

Workflow	Node	Error	Class
Nonprofit RevOps Pilot	Log Triage Result (Google Sheets append)	Column names were updated after the node's setup	Sheet "Triage Log" headers diverged from the node's hardcoded column list (name/email/category/urgencyScore/reasoning/suggestedAsk/triagedAt). Agent-fixable: re-map node columns to current sheet headers + rebuild schema. Needs one read of the sheet (OAuth token = Gadget's hands to grant, or reuse stored Google Sheets account cred id n7rMM1kTEBOug4jy).
Daily DRC Digest	Query Projects and Tasks	Connection timed out	Transient/upstream
Multi-LLM Compounding	Claude Draft	Authorization failed	Anthropic cred expired
Multi-LLM Collaboration	Gemini Critique	The resource you are requesting could not be found	Model/endpoint misconfig

Defer until core pipeline (Neon sink + DRC trigger) is green.

Related (separate workstream)

• DRC Agent Loop Store to TIM Postgres writes to frozen local Postgres (onError:continue hides loss). Fix = repoint to Neon; watchdog tim.agent_runs freshness (cron ae8fa69e33ea) detects it.

Action

• Gadget: set Secret on GitHub repo webhook github-events AND matching GITHUB_WEBHOOK_SECRET in n8n (⚠️ Test Coverage Below Threshold - 2025-11-27 #1)
• Gadget: refresh Header Auth account credential with valid X-N8N-API-KEY (⚠️ Test Coverage Below Threshold - 2025-11-27 #2)
• Verify via next n8n watchdog run (cron b58d908d6958) → ⚠️ Test Coverage Below Threshold - 2025-11-27 #1 + ⚠️ Test Coverage Below Threshold - 2025-11-27 #2 read 0 errors
• Triage Moderate backlog as cleanup

---

STAGED AGENT-FIX: Nonprofit RevOps #3 (Log Triage Result — column drift)

Node: Log Triage Result — n8n-nodes-base.googleSheets typeVersion 4.7, id 0eb35ca0-7092-433d-b848-73d081de102d, workflow 4gHFudC98pwVPmP6 (active).
Target: Google Sheet 1V49L8nrAG146efE9HsKqXbOmegEaxTLlG0K6POpVFlk, tab "Triage Log".
Error: Column names were updated after the node's setup → the sheet's headers diverged from the node's hardcoded 7-column schema.

Fix (agent-prepared, needs Gadget to Publish): switch the node from a fixed column map to auto-mapping so it tolerates header drift. In the node parameters:

• Set autoMapInputData: true
• Remove the brittle columns block: {"mappingMode":"defineBelow","value":{...},"schema":[...]}

The incoming item already carries the correct keys (name/email/category/urgencyScore/reasoning/suggestedAsk/triagedAt) from the AI Triage & Draft Agent → Triage Output Parser step, so auto-map will write them to whatever matching headers exist. This is the standard n8n remedy for this exact error and requires no read of the live sheet.

Apply via a strict PUT (strip server fields; keep only name/nodes/connections/settings with {"executionOrder":"v1"}), then Publish. Verification: re-run the workflow (submit the form once) → node appends without the column error.

NOTE: a read of the live sheet to confirm current headers was attempted but blocked — the only Google credential on this host (gcloud ADC) lacks Sheets scope for that doc. Not required for this fix (auto-map handles drift). If you'd prefer an exact column list instead of auto-map, grant Sheets scope to a credential and I'll read the headers and rebuild the schema precisely.

[labgadget015-dotcom]

@copilot Please analyze this bug report and suggest potential root causes and debugging steps.

[labgadget015-dotcom]

@copilot Please analyze this bug report and suggest potential root causes and debugging steps.

[labgadget015-dotcom]

@copilot Please analyze this CI failure and suggest fixes for the build or test issues.

[labgadget015-dotcom]

@copilot Please analyze this CI failure and suggest fixes for the build or test issues.