feature: upload improvements

[madebydavid]

Background

• Game uploads are currently handled by the ship function in src/utils/ship/index.ts and uploadZip in src/utils/ship/upload.ts
• It is a straightforward zip and then upload to a given signed URL. This is going directly to a Digital Ocean space
• There is no retry handling, any connection issue would force the user to rerun the command - just a single fetch which can fail
• The zip is uploaded in a single PUT request, no multipart or resume
• If the user has not change the game code much, rerunning the shipthis game ship command re-uploads all files

Nice to haves

• A retry mechanism around the fetch (with an increasing backoff/delay)
• Compute the zip hash before upload and only upload if different from last upload?
• Allow the user to rerun a build on an old upload? (backend change)
• Implement multi-part upload for resilience
• Incremental upload - is there some way to determine what changed since last ship command run and only upload those changes?
• Estimate and show the upload time before starting?

[madebydavid]

Implement multi-part upload for resilience

This part requires a backend change - I am currently updating the API like this:

POST /upload/:projectId/multipart          {size?}         -> {id, partSize, maxParts}
POST /upload/multipart/:uploadId/parts     {partNumbers[]} -> {parts:[{partNumber,url}]}
POST /upload/multipart/:uploadId/complete  {parts[]}       -> {id, etag}
POST /upload/multipart/:uploadId/abort                     -> {id}

[madebydavid]

backend changes are done and deployed to develop

[madebydavid]

I think it might be good to clarify on what the goals of this issue are. As I see them:

• Make the upload more resilient to failures
• Improve the average speed of the uploads

The PR #246 which adds the multipart concurrency adds a MAX_PART_ATTEMPTS for each part with a small delay somewhat addresses this - but I think the retry could be improved and could be added into the legacy singleUpload method too (now used for smaller files)