During a Pathfinder test run, a test case failed and reported the following result state:
"result": { "shepherd-bte": { "status": "FAILED", "message": "Status code: 500" }, "shepherd-arax": { "status": "FAILED", "message": "Status code: 598" }, "shepherd-aragorn": { "status": "FAILED", "message": "Status code: 598" }, "ars": { "status": "NO_RESULTS", "message": "No results" } }
However, when pulling this PK from the ARS endpoint, it returns an empty object ({}) rather than a 404 or an error log.
Currently, there is a lack of observability in the Test Harness when tests fail due to transient infrastructure errors (e.g., Status code 598). When these failures occur, the Test Harness records a Primary Key (PK) for the failure, but querying that PK in the ARS returns a completely empty object.
Because the query itself is valid and runs successfully when executed manually, this lack of logging creates a blind spot that makes it impossible to debug transient errors during automated test runs. (Confirmed as a Test Harness bug via Slack discussion with @maximusunc).
During a Pathfinder test run, a test case failed and reported the following result state:
"result": { "shepherd-bte": { "status": "FAILED", "message": "Status code: 500" }, "shepherd-arax": { "status": "FAILED", "message": "Status code: 598" }, "shepherd-aragorn": { "status": "FAILED", "message": "Status code: 598" }, "ars": { "status": "NO_RESULTS", "message": "No results" } }However, when pulling this PK from the ARS endpoint, it returns an empty object ({}) rather than a 404 or an error log.
Currently, there is a lack of observability in the Test Harness when tests fail due to transient infrastructure errors (e.g., Status code 598). When these failures occur, the Test Harness records a Primary Key (PK) for the failure, but querying that PK in the ARS returns a completely empty object.
Because the query itself is valid and runs successfully when executed manually, this lack of logging creates a blind spot that makes it impossible to debug transient errors during automated test runs. (Confirmed as a Test Harness bug via Slack discussion with @maximusunc).