Detect and log stale package-manager lock holders during apt operations

[yashnap]

Issue:
Multiple VM's are failing patch assessments repeatedly with apt-get update returning exit code 100:

E: Could not get lock /var/lib/apt/lists/lock. It is held by process <pid> (apt-get)
E: Unable to lock directory /var/lib/apt/lists/

Root cause
A stale apt-get process spawned by an older LinuxPatchExtension version (e.g. v1.6.x) kept holding the apt lock for 70+ days and was never cleaned up after the extension upgraded (e.g. to v1.6.x1). Every subsequent assessment failed. Recovery required manual intervention (kill the process, remove the lock, re-run) on each VM.
Per maintainer guidance, we do not automatically clear package-manager locks (risky, can interfere with legitimate operations). The first step is detection + telemetry/logging only, so we can confirm the "orphaned, extension-owned, older-version" scenario in the field before considering any scoped, safe recovery later.

What the PR does
In AptitudePackageManager.invoke_package_manager_advanced, on the existing failure branch:

1. Detects the lock-contention case from apt's output (Could not get lock / Unable to lock).
2. Parses the holder PID directly from apt's message (held by process ).
3. Reads the holder's elapsed run time and command line via a single ps -p -o pid=,etime=,cmd=.
4. If the holder was launched by a LinuxPatchExtension- path, extracts that version and compares it to the current Constants.EXT_VERSION using the existing VersionComparator.
5. Emits a structured, log-only warning, e.g.:

   [APM] Detected package manager lock held by a LinuxPatchExtension process. No action taken (detection only).
   [HolderPid=14980][HolderVersion=1.6.64][CurrentVersion=1.6.72][IsOlderVersion=True][Holder=... LinuxPatchExtension-1.6.64..]

TESTING
Manual end-to-end on a VM (Ubuntu 24.04, current build 1.6.72):
ARM : /subscriptions/6acc8a91-e2b0-4041-a069-c2932ab42fd9/resourceGroups/apt_lock_issue_rg/providers/Microsoft.Compute/virtualMachines/apt-lock-vm-ubuntu

1. Created a fake stale holder that takes a POSIX (fcntl/lockf) lock on /var/lib/apt/lists/lock with an older LinuxPatchExtension-1.6.64 path in its argv using the below script(by copilot)
   `cat > /tmp/hold_apt_lock.py <<'EOF'
   import fcntl, time
   f = open('/var/lib/apt/lists/lock', 'w')
   fcntl.lockf(f, fcntl.LOCK_EX | fcntl.LOCK_NB) # POSIX lock apt's F_GETLK can see
   time.sleep(3600)
   EOF

sudo python3 /tmp/hold_apt_lock.py
-oDir::Etc::SourceParts=/var/lib/waagent/Microsoft.CPlat.Core.LinuxPatchExtension-1.6.64/tmp/azgps-src &`

2. Confirmed apt reports the holder PID
   sudo apt-get -q update # => E: Could not get lock ... It is held by process <pid> (python3) ps -p <pid> -o pid=,etime=,cmd= # cmdline shows the LinuxPatchExtension-1.6.64 path

3. Triggered an assessment while the lock was held.

4. Verified the detection line appeared on every retry, with HolderVersion=1.6.64, CurrentVersion=1.6.72, IsOlderVersion=True, and that no process was killed and no lock removed (assessment failed/retried normally).
   3.core.log

New log after recent change: ( September 23rd)

5.core.log

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.98%. Comparing base (119f110) to head (3e74a8d).

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #389      +/-   ##
==========================================
+ Coverage   94.95%   94.98%   +0.02%     
==========================================
  Files         113      113              
  Lines       21844    21936      +92     
==========================================
+ Hits        20743    20835      +92     
  Misses       1101     1101              

Flag	Coverage Δ
python27	94.98% <100.00%> (+0.02%)	⬆️
python312	94.98% <100.00%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Copilot review overview

🟡 Changes recommended

Address sensitive command-line logging, truncated process output, locale-dependent parsing, and incomplete safety-test assertions.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1

Open (1)

• Raw lock-holder command line may expose credentials in telemetry · New

What changed in this PR

Adds detection-only diagnostics for apt lock contention, including holder PID, runtime, command line, and LinuxPatchExtension version comparison.

Changes:

• Detects and parses apt lock-holder failures.
• Logs extension ownership and version comparisons without taking recovery action.
• Adds tests for stale extension and unrelated lock holders.

File	Summary
src/​core/​tests/​Test_AptitudePackageManager.py	Tests stale and unrelated lock-holder scenarios.
src/​core/​src/​package_managers/​AptitudePackageManager.py	Implements lock-holder inspection and structured warnings.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.