feat(prompts): add /m2_smoketest for app verification workflow - #24
Merged
Conversation
Discover or create smoketest_script/, extend it from new-feature diff, review and refine the script, execute it, then emit a quantified report plus an explicit bug list. Covers install, runtime, settings, UX and performance. Also route /m2_smoketest in copilot-instructions section 9, update README and bump CHANGELOG to v3.1.0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
新增
/m2_smoketest(.github/prompts/m2_smoketest.prompt.md)—— App 基本驗證的完整流程:探索既有 smoke test → 沒有就建立/有就依新功能補測試 → review 並精修腳本 → 實際執行 → 輸出量化報告與明確的 bug 清單。覆蓋安裝、執行、設定、使用者體驗、效能五個面向,約定目錄為smoketest_script/。同時把它接進
copilot-instructions.md§9 路由表與標準流程順序(/m2_review→ 修 →/m2_smoketest→/m2_pr),並更新 README 與 CHANGELOG(v3.1.0)。Why
既有六支指令裡沒有一支回答「這包東西還裝得起來、跑得起來嗎」。
/m2_review只看 diff,CI 只跑得動自動化測試,而桌面 App 最常壞的地方——安裝、升級覆蓋、設定遷移、關閉後殘留 process——全都在這兩者的射程之外。/m2_evo已經把 smoke test 當成每個迭代的回歸防線(§0.4「沒有 smoke test 就沒有這支 prompt」),但它只說「要有」,沒說「怎麼生出來、怎麼維護、怎麼判讀」。這支 prompt 補的就是那一段。第二個動機是誠實度。smoke test 最容易腐爛的方式不是沒人寫,而是寫了之後慢慢變成「永遠會綠」——空
catch、沒有斷言、把跑不動的案例標成 skip、效能量一次就下結論。所以這支 prompt 的重點不在「會跑測試」,而在讓放水無所遁形。How
流程分流(§1–§3)
git log -1 --format=%H -- smoketest_script找出腳本上次更新後的 diff,再依訊號對照該補哪些案例(新 CLI 參數、新設定項、新頁面、安裝變更、新相依、schema 變更、修過的 bug)。先輸出覆蓋缺口表、按鈕確認過才動手。五大面向與編號(§4)
SMK-<AREA>-<NNN>,AREA∈INS/RUN/CFG/UX/PERF;ID 發出去就不重用,否則歷史報告會對不上。manual-checklist.md標manual——沒驗證就不准寫 pass。輸出契約(§2.4)
result.json(summary/cases/metrics欄位固定),報告一律以它為資料來源,不靠 agent 讀 stdout 猜數字。0全過/1有 fail/2腳本或環境本身壞掉 —— 不分開的話,CI 紅了看不出是 App 壞還是測試壞。環境安全(§6.1、§6.4)
git clean -fd、git reset --hard,以及打到使用者目錄的Remove-Item -Recurse -Force。量化報告與 bug 清單(§7、§8)
pass + fail + flaky + skip,不含manual,但 manual 數量必須出現在報告上。檔案:行號+ 重現步驟 + 期望 vs 實際)與smoke test 自己的問題(每筆給可直接套用的修法)。範圍界線
smoketest_script/底下的東西且逐項確認;不修 App 的 bug、不改 CI workflow、不 commit/不 push/不開 PR —— 問題列清楚後交回使用者或/m2_review。results/的忽略設定寫在smoketest_script/.gitignore(巢狀),不動 repo 根目錄的.gitignore,以免在下游 repo 留下沒被要求的變更、也不牴觸上面那條範圍規則。auto:安裝與設定測試會實際動到使用者機器,確認節點不可免除;想少被打擾用/m2_smoketest run(只跑不改)。其他
copilot-instructions.md§9 的「強制確認節點」列舉由 four → five(納入m2_smoketest),並在 Auto Mode 一節寫明它為何被排除。scripts/validate.py會檢查路由表涵蓋所有 prompt file,所以 §9 的 routing 條目是必要而非選配。Impact
copilot-instructions.md§9 三處措辭;README 指令表/流程圖/auto說明/目錄樹。sync-manifest.txt已列prompts/(資料夾 overlay),新檔會自動同步,manifest 與下游 workflow 皆零改動。下次同步後 Copilot Chat 打/會多一支/m2_smoketest。auto」四道防線收斂,但建議先在自己的 App 專案實跑一次再推廣。Verification
python scripts/validate.py→ 6 個 prompt file 全數通過、0 錯誤 0 警告(含機密掃描與「路由表涵蓋所有 prompt」檢查)/m2_review→ 🔴 0 項、🟡 2 項,兩項 Should fix 皆已修(.gitignore自我矛盾、§9 確認節點列舉漏列)案例 = ✅+❌+⚠️+⏭️成立,總通過率 42/45 = 93.3%,效能 +2.9% / +25.2% 與離散度 3.7% 皆相符git diff --cached自檢無機密資料、無內部 IP、無殘留 debug 程式碼origin/main(git rev-list --left-right --count左側為 0)CHANGELOG.md(v3.1.0)/m2_smoketest尚未在真實 App 專案端到端實跑過(本 repo 沒有可安裝的 App,跑不到 INS/RUN/CFG/UX/PERF 任一面向)—— 屬規格,需在下游 App 專案首次使用時驗證-Depth、pwshfallback、報告欄名改計分案例)刻意保留,待實跑後再決定