Skip to content

Stale output.tar.zst from prior deploy not cleaned up; runtime extracts orphan over current output.tar.gz #263

@nxfwapd

Description

@nxfwapd

Summary

On Linux Python App Service, when consecutive deploys produce different compression formats (output.tar.zst from one, output.tar.gz from a later one), both files persist in /home/site/wwwroot/. The Oryx runtime startup script extracts the OLDER output.tar.zst instead of the newer output.tar.gz, causing production to serve stale code with no error indication.

Environment

  • App Service: wapd-image-finder (Linux, Python 3.11)
  • Image tag: appsvc/python:3.11_20260203.2.tuxprod
  • Oryx Version: 0.2.20251017.2
  • Date observed: 2026-05-10 UTC
  • Region: East US

Reproduction sequence

  1. Deploy A produces output.tar.zst (zstd-compressed) -- in our case a deploy attempt where COMPRESS_DESTINATION_DIR=false was set as an app setting but ignored by Oryx (separate issue, see Container panic 'Could not find file /home/site/wwwroot/output.tar.gz' on appsvc/python:3.11_20260203.2.tuxprod microsoft/Oryx#2921).
  2. Deploy B (later, via GitHub Actions OneDeploy) produces output.tar.gz (gzip).
  3. Both files now sit in /home/site/wwwroot/. Microsoft does not delete the older one.
  4. Container restarts. Runtime startup script extracts output.tar.zst (older) rather than output.tar.gz (current).
  5. Production serves the OLDER deploy's source/static-assets.

Evidence

$ ls -la /home/site/wwwroot/
-rwxrwxrwx  1 nobody nogroup        261 May 10 04:47 oryx-manifest.toml
-rwxrwxrwx  1 nobody nogroup  289966627 May 10 04:47 output.tar.gz   <- current deploy
-rwxrwxrwx  1 nobody nogroup  277256844 May 10 02:23 output.tar.zst  <- orphan from prior deploy

$ tar -tzf /home/site/wwwroot/output.tar.gz | grep assets/index
./frontend/dist/assets/index-w8ghkh8t.css
./frontend/dist/assets/index-DNf7ivpY.js   <- new bundle hash, 404 in production

$ zstd -d -c /home/site/wwwroot/output.tar.zst | tar -t | grep assets/index
./frontend/dist/assets/index-BRgwFiec.js   <- old bundle hash, 200 in production
./frontend/dist/assets/index-CjFrALqX.css

oryx-manifest.toml from the latest deploy says CompressDestinationDir="true". The startup script appears to scan for any output.tar.* file rather than reading the manifest's intended format, picking the older .zst over the newer .gz.

Workaround

Manually rm /home/site/wwwroot/output.tar.zst via Kudu shell, then az webapp restart. After the restart, runtime extracts only output.tar.gz and production serves the current deploy.

Asks

  1. Oryx runtime startup script should either (a) trust oryx-manifest.toml's declared format and only look for the matching tarball, or (b) prefer the newest output.tar.* file by mtime.
  2. Oryx build phase should clean up prior output.tar.* artifacts in wwwroot so only the newest exists after each deploy.
  3. Fix COMPRESS_DESTINATION_DIR=false honoring (currently ignored by Oryx, tracked in Container panic 'Could not find file /home/site/wwwroot/output.tar.gz' on appsvc/python:3.11_20260203.2.tuxprod microsoft/Oryx#2921). With that env var working, customers wouldn't end up with mixed-format tarballs in the first place.

Related Microsoft issues we've filed today

This is the third distinct App Service Linux regression we've hit in a 24h window. Would appreciate triage.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions