Why
Found while trying to clear ~36,000 accumulated sources from a test project (#362). The cleanup could only be completed by writing an external paced driver with its own retry loop — the CLI could not do it.
Three causes compound.
1. No pagination on the commands that need it most
gateway source list, destination list and connection list have no --next/--prev flags, yet --output json returns pagination.next. event, request and transformation list all have them.
Maximum --limit is 255 (API-enforced), so page 2 of 36,000 sources is unreachable through the CLI. This is the clearest bug of the three and the cheapest fix: those three commands should have the cursor flags their siblings already have.
2. Deleting costs two requests
gateway <res> delete <id> --force issues a GET before the DELETE — confirmed at --log-level debug. Against a 240 req/min limit that halves throughput to a ceiling of roughly 110 deletes per minute.
The GET presumably exists to give a good "not found" message, which is worth something. But it doubles the cost of every delete, and for the --force path the API's own error would do.
3. No rate-limit handling at all
xargs -P 16 over 300 deletes: 193 of 300 failed with
failed to get source: error: unexpected http status code: 429, raw response body: Too Many Requests
No retry, no backoff, no use of the Retry-After header the API sends, and the message does not say "you are being rate limited, slow down" — it surfaces a raw status code and a raw body.
This one matters well beyond cleanup: any script doing bulk work hits it, and the error gives no clue what to do. SuppressRateLimitErrors already exists on the client for polling, so there is prior art for handling 429 deliberately.
What it costs today
Clearing one project (36,359 sources + 36,713 destinations) is about 11 hours of wall clock at the achievable rate.
Suggested order
- Add
--next/--prev to source, destination and connection list. Small, obvious, unblocks everything else.
- Honour
Retry-After on 429 with a bounded retry, and make the message say what happened. Benefits every command, not just bulk work.
- Consider dropping the pre-delete GET when
--force is set, or making it optional.
A --all or bulk-delete mode is deliberately not proposed here. It is a bigger design question and the three fixes above make the scripted approach viable, which is enough.
Related
Why
Found while trying to clear ~36,000 accumulated sources from a test project (#362). The cleanup could only be completed by writing an external paced driver with its own retry loop — the CLI could not do it.
Three causes compound.
1. No pagination on the commands that need it most
gateway source list,destination listandconnection listhave no--next/--prevflags, yet--output jsonreturnspagination.next.event,requestandtransformationlist all have them.Maximum
--limitis 255 (API-enforced), so page 2 of 36,000 sources is unreachable through the CLI. This is the clearest bug of the three and the cheapest fix: those three commands should have the cursor flags their siblings already have.2. Deleting costs two requests
gateway <res> delete <id> --forceissues a GET before the DELETE — confirmed at--log-level debug. Against a 240 req/min limit that halves throughput to a ceiling of roughly 110 deletes per minute.The GET presumably exists to give a good "not found" message, which is worth something. But it doubles the cost of every delete, and for the
--forcepath the API's own error would do.3. No rate-limit handling at all
xargs -P 16over 300 deletes: 193 of 300 failed withNo retry, no backoff, no use of the
Retry-Afterheader the API sends, and the message does not say "you are being rate limited, slow down" — it surfaces a raw status code and a raw body.This one matters well beyond cleanup: any script doing bulk work hits it, and the error gives no clue what to do.
SuppressRateLimitErrorsalready exists on the client for polling, so there is prior art for handling 429 deliberately.What it costs today
Clearing one project (36,359 sources + 36,713 destinations) is about 11 hours of wall clock at the achievable rate.
Suggested order
--next/--prevtosource,destinationandconnection list. Small, obvious, unblocks everything else.Retry-Afteron 429 with a bounded retry, and make the message say what happened. Benefits every command, not just bulk work.--forceis set, or making it optional.A
--allor bulk-delete mode is deliberately not proposed here. It is a bigger design question and the three fixes above make the scripted approach viable, which is enough.Related