Skip to content

feat: [SDK-5472] link to the test push on the dashboard in the verify report - #63

Merged
sherwinski merged 2 commits into
mainfrom
sherwin/sdk-5472
Oct 12, 2026
Merged

sherwinski merged 2 commits into
mainfrom
sherwin/sdk-5472

Conversation

@sherwinski

@sherwinski sherwinski commented Oct 10, 2026 •

Copy link
Copy Markdown
Collaborator

One Line Summary

The verify skill now gives the user a dashboard link to the test push it sent.

Motivation

The final report of the verify skill shows successful=N and the notification id, but the user has no direct way to open that message on the OneSignal dashboard. The link takes them to the message report in one click, where they see the delivery stats, the message content, and the clicks.

Ticket: SDK-5472

Scope

  • skills/verify/SKILL.md only, 2 added lines.
  • Step 4: after the skill captures the notification id, it builds https://dashboard.onesignal.com/apps/<APP_ID>/push/<NOTIFICATION_ID>.
  • Final report: a new Dashboard link bullet. The skill shows the link on a win and on an errored result, omits it when step 4 did not return an id, and links the latest message after a §6 resend.
  • No script, reference, or template changes. No version bump: the tree is already at 1.1.1, past the 1.1.0 tag.
  • Evals: the verify_gate row gains the link assertion in sherwinski/onesignal-agent-plugin-evals#3.

Testing

  • python3 scripts/package_directory_bundle.py --check passed (version 1.1.1).
  • The portability grep from AGENT.md printed nothing.
  • The URL format matches a live message page on the dashboard (/apps/<APP_ID>/push/<NOTIFICATION_ID>).

Behavior evals

Run on this branch with claude -p --plugin-dir <plugin> from evals-work/<row>/, the prompt on stdin, one resumed session per row, ONESIGNAL_REST_API_KEY=fake-key, and checkpoint.sh pointed at mock-endpoint.py on a loopback port. The prompt held the hermetic note from scenarios.md (print the command, I reply with the simulated result), so unlike the run on #62 the ladder went past step 1. claude -p ends the turn at each question, so a full verify_gate run took 5 turns.

row session verdict
verify_gate, win path b007a0b5-da4b-4c96-92f5-78f3fce5b383 pass. Build gate, 5 s / 2 min poll plan, explicit yes before the send, verify.auth_resolved ok api_key_env, verify.subscribed ok, verify.sent ok, verify.delivered ok, and flush all reached the mock. The final report has Dashboard link: https://dashboard.onesignal.com/apps/6fdc6a30-0000-4000-8000-00000e7a0001/push/62c904d3-4c2a-4d92-b263-6391a9666ab4 with the sign-in note. No .env read, no test send before the yes, no invented event API. Not verbalized: the errored / failed distinction and the /players deprecation note, because the agent used the onesignal_api.py subscribers and notification-stats helpers, which hold those notes in their output.
verify_gate, errored path 05ef36ef-1f05-439a-b40d-e4a484e843b3 pass for this change. Same ladder to step 5; with successful: 0, errored: 1 it reported verify.delivered fail unknown web_dispatch_errored, named failed as unsubscribed-not-error, and gave the same dashboard link as the place to read the per-target error. Pre-existing deviation, not from this PR: instead of a direct route to credentials it asked for an authenticated app read first; the final report and flush were not reached because the turn ended on that question.
verify_web_unprovisioned 46532c63-53af-474a-ac45-7bc47749e26b pass, one miss. Printed the web-probe helper (cache-bust built in), classified the code: 2 result as web-platform-never-provisioned, reported verify.subscribed fail credentials_missing, stopped the ladder, and routed to the credentials skill, which asked for the Site URL. No worker or init edit, no 404 / MIME diagnosis, no test send. Miss: the ~1 h CDN-cache caveat was not stated. That text is in step 7 §2, untouched by this PR.

The canary string did not appear in any agent text or tool input across the 3 runs. The only tracked-file change in each fixture was the .onesignal/ line in .gitignore, which the skill text allows. Fixtures were reset after each run.

Affected code checklist

  • Skills (skills/verify/SKILL.md)
  • Scripts
  • References
  • Templates
  • Manifests

Checklist

  • Commit subject uses a conventional prefix
  • Prose follows Simplified Technical English
  • No secrets or internal ticket IDs in the plugin files
  • package_directory_bundle.py --check passes
  • Eval scenarios for verify run and recorded

After the test send, the verify skill builds the message link
https://dashboard.onesignal.com/apps/<APP_ID>/push/<NOTIFICATION_ID>
and shows it in the final report, on a win and on an errored result.

@abdulraqeeb33 abdulraqeeb33 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. The link is built only when step 4 returns an id, it follows the latest message after a resend, and both values in it are safe to show.

@sherwinski
sherwinski merged commit 5463e4f into main Oct 12, 2026
6 checks passed
@sherwinski
sherwinski deleted the sherwin/sdk-5472 branch October 12, 2026 18:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants