[BUG] 1.14.10 — Single-shot retry on 404 "Unknown API" during createAPIAccess does not trigger, causing intermittent rollback after API create

[Kekko5]

APIM-CLI version

1.4.10

API-Management version

7.7.20211130

Bug description

When importing a new Frontend API with one or more application grants, apim-cli intermittently fails with Can't grant access to API immediately after the API is created. API-Manager replies HTTP 404 — Unknown API on POST /applications/{id}/apis, referencing the API ID that was just created a few hundred milliseconds earlier. The retry logic present in APIManagerAPIAccessAdapter.createAPIAccess does not appear to trigger in this scenario, and the rollback runs immediately, deleting the freshly created API.

The behavior is intermittent (rate roughly 1–3% of deployments) and timing-dependent. Re-running the same import sometimes succeeds at the second attempt, sometimes requires several attempts, and occasionally needs the operator to remove existing application grants manually, rerun the import, and reassign the grants.

Environment

• apim-cli version: 1.14.10
• API-Manager version: 7.7.20211130 (SP2)
• JRE: 11
• Topology: clustered Admin Node, sticky session enabled on the load balancer in front of Admin Nodes
• Command: api import -retryDelay 3500 -c .json -s
• CLI log shows effective retry delay reported as: Retrying unexpected API-Manager REST-API responses with a delay of 7000 milliseconds

Expected behavior

The retry policy in APIManagerAPIAccessAdapter should kick in: wait retryDelay, repeat the POST /applications/{id}/apis. If the call succeeds at the second attempt, the import should complete without rollback.

Actual behavior

The retry does not appear to execute. Looking at timestamps in the log: Creating API-Access for application is logged at 18:27:36.182, and ERROR Error granting access to application id ... at 18:27:36.243 — a gap of 61 ms, while the configured retry delay is reported as 7000 ms. With retry actually running we would expect the ERROR to appear no earlier than ~18:27:43.182. There is no intermediate WARN Got unexpected error: 'Unknown API' while creating API-Access... line in the log, which is the message emitted by the retry path.

Code reference

Snippet from modules/apim-adapter/src/main/java/com/axway/apim/adapter/apis/APIManagerAPIAccessAdapter.java, tag 1.14.10, method createAPIAccess:

if ((statusCode == 403 || statusCode == 404) &&
    (response.contains("Unknown API") ||
     response.contains("The entity could not be found"))) {
  LOG.warn("Got unexpected error: 'Unknown API' while creating...");
  Thread.sleep(cmd.getRetryDelay());
  httpResponse = httpHelper.execute(request, true);

Two observations:

1. The retry is single-shot. In our scenario the API-Manager cache window seems to last longer than a single retryDelay, so even a fired retry might not be enough. A small bounded loop (e.g. 2–3 attempts with retryDelay) would be more resilient.
2. In our specific log the retry does not seem to fire at all (timing analysis above + absence of the WARN Got unexpected error line). We have not been able to determine from log alone why the condition does not match. A --debug run is on the roadmap to capture the raw response payload.

Related prior work

• [BUG] Error while creating/updating API Access #26 — original "Unknown API" 404 / 403 report
• Error when importing application using Jenkins #212 / PR Provide option to configure the Retry-Delay for some of the API-Manager REST-API calls #213 — first retry on 404 "Unknown API" (released in 1.3.6)
• Error upgrading access to newer API - The entity could not be found. #218 — variant message "The entity could not be found" (fixed in 1.3.13)
• Error 409 from API Gateway API when creating API-Access for application #538 — 409 on the same POST /applications/{id}/apis (fixed in 1.14.11)

This issue is in the same family but specifically about the 404 path on 1.14.10, where the retry exists in the source but does not appear to execute for our combination of API-Manager version + payload format.

Questions to the maintainers

1. Is there a known regression in 1.14.10 around the retry condition in APIManagerAPIAccessAdapter.createAPIAccess?
2. Would you accept a PR converting the single-shot retry into a bounded loop (configurable, e.g. -maxRetries) for the same (403|404) + (Unknown API|The entity could not be found) condition?
3. Has the 404 case in this flow been re-validated in 1.14.11 / 1.14.12 / 1.14.13? The changelogs of those releases do not explicitly mention it (only 409 in 1.14.11), and we would like to confirm before upgrading whether an upgrade alone is expected to mitigate the 404 occurrences as well.
4. Any recommended runtime workaround at JSON or command line level beyond -retryDelay?

Thank you.

Steps to reproduce

Not reliably reproducible on demand (timing-dependent). Conditions under which we observe it:

1. New API definition (does not yet exist in API-Manager).
2. JSON contains one or more applications entries with grants.
3. Import succeeds in creating the API; the immediately following POST /applications/{id}/apis to grant access returns 404 within ~60 ms of the create.

Relevant log output

2026-04-21 18:27:32,867 INFO   APIManagerCLI| API-Manager CLI: 1.14.10
2026-04-21 18:27:32,868 INFO   APIManagerCLI| Module: API - I M P O R T  (1.14.10)
2026-04-21 18:27:32,884 INFO   CoreParameters| Retrying unexpected API-Manager REST-API responses with a delay of 7000 milliseconds.
2026-04-21 18:27:33,495 INFO   APIManagerAdapter| Successfully connected to API-Manager (7.7.20211130) on: https://<host>:8575
2026-04-21 18:27:33,711 INFO   SpecificationFactory| Reading API-Definition (Swagger/WSDL) from file: <path>.json (relative path)
2026-04-21 18:27:33,736 INFO   SpecificationFactory| Detected: Swagger 2.0 specification.
2026-04-21 18:27:33,874 INFO   IImportConfigAdapter| Handling configured client-applications.
2026-04-21 18:27:34,380 INFO   IImportConfigAdapter| Application exists : <App1> (<orgKey>) (null)]
2026-04-21 18:27:34,811 INFO   IImportConfigAdapter| Application exists : <App2> (<orgKey>) (null)]
2026-04-21 18:27:34,838 INFO   APIImportApp| Lookup actual API based on Path: /<basepath>
2026-04-21 18:27:35,847 INFO   APIImportManager| No existing API found, creating new!
2026-04-21 18:27:35,941 INFO   CreateNewAPI| Create published API: <ApiName> 1.0  based on Swagger 2.0 specification.
2026-04-21 18:27:36,177 INFO   geClientOrganization| All desired organizations: [] have already access. Nothing to do.
2026-04-21 18:27:36,182 INFO   ManageClientApps| Creating API-Access for the following apps: [<App1>, <App2>]
2026-04-21 18:27:36,182 INFO   ManageClientApps| Creating API-Access for application <App1>
2026-04-21 18:27:36,243 ERROR  agerAPIAccessAdapter| Error granting access to application id : <appId> for  API-Proxy  : <apiId> using URI: https://<host>:8575/api/portal/v1.4/applications/<appId>/apis  Received Status-Code: 404 Response: {"errors":[{"code":404,"message":"Unknown API"}]}
2026-04-21 18:27:36,245 INFO   RollbackAPIProxy| Rollback FE-API: <ApiName> (ID: <apiId> / State: unpublished)
2026-04-21 18:27:36,689 INFO   APIManagerAPIAdapter| API: <ApiName> 1.0 ( <apiId> ) successfully deleted
2026-04-21 18:27:39,580 INFO   RollbackHandler| Rolled back: [Frontend-API: false, Backend-API: true]
2026-04-21 18:27:39,580 INFO   AppException| You may enable debug to get more details.
2026-04-21 18:27:39,585 ERROR  AppException| Can't grant access to API.

[rathnapandi]

Hi @Kekko5,

Thanks for detailed message.

Is there a known regression in 1.14.10 around the retry condition in APIManagerAPIAccessAdapter.createAPIAccess?

There are corner cases and I don't see integration test.
Would you accept a PR converting the single-shot retry into a bounded loop (configurable, e.g. -maxRetries) for the same (403|404) + (Unknown API|The entity could not be found) condition?
Yes, https://failsafe.dev/ libaray is part of the project. please use that library.
Has the 404 case in this flow been re-validated in 1.14.11 / 1.14.12 / 1.14.13? The changelogs of those releases do not explicitly mention it (only 409 in 1.14.11), and we would like to confirm before upgrading whether an upgrade alone is expected to mitigate the 404 occurrences as well.
Looks like no. It is a breaking change.
Any recommended runtime workaround at JSON or command line level beyond -retryDelay?
No.