Fix extras pin un-hosted beside a pylock (#1794)

crates/socket-patch-cli/CLI_CONTRACT.md

@@ -1594,7 +1594,7 @@ command's integration tests check its output.
 | `redirect_unconfirmed`    | `skipped` (`details.mode: "hosted"`) | hosted `scan` / `get` (v5.0, additive): the patch was granted but no lockfile entry pinning it could be rewritten. The status and exit code are unchanged for now, pending the open hosted exit-policy decision (#704); `--silent` hides the human line. |
 | `lockfile_unreadable` / `lockfile_unparseable` / `patched_ref_invalid` / `patched_ref_unattributable` | run-level `warnings[]` | vex (every form): lockfile-discovery diagnostics — see "Manifest-less VEX (lockfile discovery)". Never flip the exit on their own. |
 | `vex_npm_shrinkwrap_only` | run warning and `failed[].reason` | vex (every form, #899): the patch is wired only in a root `npm-shrinkwrap.json` with no `package-lock.json` twin, which npm >= 12 never reads; no statement until the twin exists. `list` / `rollback` / `remove` still manage the wiring. |
-| `vendor_multiple_lockfiles` / `pypi_multiple_lockfiles` | `skipped` (warning) | vendor: a sibling lockfile of another package manager (for PyPI, also a root `requirements.txt` that pins the package beside the wired tool lock) will still install UNPATCHED bytes; names the wired winner + the ignored locks. Beside `Pipfile.lock`, `uv.lock` or a wired PEP 751 / script lock, an exact registry pin of the package in `requirements.txt` or an in-root `-r` include (`pipenv requirements` or `uv export` output, or the file `pip lock -r requirements.txt` read) is wired with the lock instead and never named here (#612, #1368, #1758): both files then refer to the committed wheel, the ledger entry records both, every revert restores both, a superseding patch re-wires both, and a re-run over a project whose lock is already wired (a vendor from before, or an export made since) wires the export too (`pypi_requirements_sibling_wired`). A pin the requirements wiring cannot rewrite (a range, extras, an include outside the root) stays a loser. Likewise, a `pylock.toml` / `pylock.<name>.toml` that `pipenv lock` writes beside `Pipfile.lock` under `[pipenv] use_pylock = true` is wired with it (#1368), with the same ledger, revert, supersede and re-run behavior (`pypi_pipenv_pylock_wired`); a pylock the wiring cannot rewrite in place (a symlink, a non-registry source) stays a loser. |
+| `vendor_multiple_lockfiles` / `pypi_multiple_lockfiles` | `skipped` (warning) | vendor: a sibling lockfile of another package manager (for PyPI, also a root `requirements.txt` that pins the package beside the wired tool lock) will still install UNPATCHED bytes; names the wired winner + the ignored locks. Beside `Pipfile.lock`, `uv.lock` or a wired PEP 751 / script lock, an exact registry pin of the package in `requirements.txt` or an in-root `-r` include (`pipenv requirements` or `uv export` output, or the file `pip lock -r requirements.txt` read) is wired with the lock instead and never named here (#612, #1368, #1758): both files then refer to the committed wheel, the ledger entry records both, every revert restores both, a superseding patch re-wires both, and a re-run over a project whose lock is already wired (a vendor from before, or an export made since) wires the export too (`pypi_requirements_sibling_wired`). An exact pin of the package with extras (`six[foo]==1.16.0`), or a file's only line for it when that line is an unpinned extras requirement (`six[foo]`, which hosted mode also rewrites), cannot be expressed as a wheel path line. It refuses a fresh wiring of the lock `pypi_extras_unsupported`, as the requirements-only wiring does, rather than leave `pip install -r` unpatched; a hosted → vendored switch then keeps both files' hosted pins (#1794). A lock already wired re-runs in sync with the extras pin still named here. Any other pin the requirements wiring cannot rewrite (a range, an include outside the root) stays a loser. Likewise, a `pylock.toml` / `pylock.<name>.toml` that `pipenv lock` writes beside `Pipfile.lock` under `[pipenv] use_pylock = true` is wired with it (#1368), with the same ledger, revert, supersede and re-run behavior (`pypi_pipenv_pylock_wired`); a pylock the wiring cannot rewrite in place (a symlink, a non-registry source) stays a loser. |
 | `vendor_npm_shrinkwrap_only` | `skipped` (warning) | vendor / scan / get `--mode vendored` (npm, #899): the package was wired into `npm-shrinkwrap.json`, the only npm lock (also on an in-sync re-run). npm >= 12 never reads the shrinkwrap and installs the unpatched registry bytes; rename the lock to `package-lock.json` (or commit a copy under that name) and re-run. |
 | `vendor_npm_deno_lock` | `skipped` (warning) | vendor / scan / get `--mode vendored` (npm, every lockfile flavor, #406): the package was vendored (also on an in-sync re-run), but the project root's `deno.lock` locks the same `name@version`. `deno install` installs a `package.json` project's npm dependencies from `deno.lock` and never reads the rewired lock, so installs Deno runs keep the registry copy and `vex` omits the patch (`vex_deno_lock_copy`); npm / pnpm / yarn / bun installs are patched. To patch what Deno installs, run `deno install` and then `socket-patch scan --mode agent`. |
 | `vendor_workspace_member_skipped` | `skipped` (warning) | vendor / scan / get `--mode vendored` (npm, #688): a lock entry with the package's `name@version` is the project's own source (a workspace member or a `file:` directory), so it is left alone while the lock's registry copies are vendored; patch that source directly if it needs the fix. When it is the only instance, vendoring refuses instead. |

crates/socket-patch-cli/tests/mode_migration_pypi.rs

@@ -3339,6 +3339,46 @@ async fn pip_lock_rerun_wires_a_requirements_pin_added_later() {
     );
 }
 
+/// #1794: the extras refusal guards a FRESH lock wiring only. A pylock
+/// already wired to the vendored wheel (by a run before the extras pin
+/// was added, or by a release before the refusal) re-runs in sync instead
+/// of failing every later `vendor`, and the extras pin stays a named
+/// loser as before.
+#[tokio::test]
+async fn pip_lock_rerun_keeps_a_wired_lock_beside_an_extras_pin() {
+    let (_tmp, root) = project();
+    stage_pip_lock(&root);
+    std::fs::write(root.join("requirements.txt"), "idna==3.7\n").unwrap();
+    stage_manifest(&root);
+    let (code, env) = run_cli(&root, &["vendor"], &[]);
+    assert_eq!(code, 0, "{env:#}");
+    let wired = std::fs::read_to_string(root.join("pylock.toml")).unwrap();
+    assert!(
+        wired.contains(&format!(".socket/vendor/pypi/{UUID}/")),
+        "{wired}"
+    );
+
+    std::fs::write(
+        root.join("requirements.txt"),
+        "idna==3.7\nsix[foo]==1.16.0\n",
+    )
+    .unwrap();
+    let (code, env) = run_cli(&root, &["vendor"], &[]);
+    assert_eq!(code, 0, "an in-sync lock is not refused: {env:#}");
+    assert!(
+        !env.to_string().contains("pypi_extras_unsupported"),
+        "{env:#}"
+    );
+    assert_eq!(
+        std::fs::read_to_string(root.join("pylock.toml")).unwrap(),
+        wired
+    );
+    assert_eq!(
+        std::fs::read_to_string(root.join("requirements.txt")).unwrap(),
+        "idna==3.7\nsix[foo]==1.16.0\n"
+    );
+}
+
 /// #1758: hosted mode pins six in both `requirements.txt` and the
 /// `pip lock` pylock.toml. The hosted → vendored switch (plain `vendor`,
 /// the eject, and `scan --mode vendored`) restores both files to their
@@ -3415,6 +3455,86 @@ async fn pip_lock_hosted_to_vendored_keeps_the_requirements_patched() {
     }
 }
 
+/// #1794: a `requirements.txt` pin with extras (`six[foo]==1.16.0`) can't
+/// be written as a vendored wheel path line, so the pylock beside it can't
+/// be vendored alone: the switch's upstream restore had just un-hosted the
+/// requirements pin, and `pip install -r requirements.txt` would then go
+/// from patched to unpatched. The switch refuses the purl with the
+/// requirements wiring's own code, and the restore is rolled back, so both
+/// files keep their hosted pin byte for byte. Before the fix both files
+/// were un-hosted, only pylock.toml was wired, and the run exited 0.
+#[tokio::test]
+async fn pip_lock_hosted_to_vendored_refuses_an_extras_pin() {
+    for takeover in [false, true] {
+        let server = MockServer::start().await;
+        let hosted_url = mount_hosted_api(&server, true).await;
+        mount_patch_view(&server).await;
+        let uri = server.uri();
+        let switch: Vec<&str> = if takeover {
+            vec![
+                "scan",
+                "--mode",
+                "vendored",
+                "--yes",
+                "--api-url",
+                &uri,
+                "--org",
+                ORG,
+                "--api-token",
+                "fake-token",
+                "--patch-server-url",
+                &uri,
+                "--vendor-url",
+                &uri,
+            ]
+        } else {
+            let mut args = vec!["vendor", "--patch-server-url", &uri, "--vendor-url", &uri];
+            args.extend(api_args(&uri));
+            args
+        };
+        let (_tmp, root) = project();
+        std::fs::write(
+            root.join("requirements.txt"),
+            "idna==3.7\nsix[foo]==1.16.0\n",
+        )
+        .unwrap();
+        std::fs::write(root.join("pylock.toml"), PIP_LOCK).unwrap();
+        let files = ["requirements.txt", "pylock.toml"];
+        let (code, env) = hosted_scan(&root, &server);
+        assert_eq!(code, 0, "hosted scan: {env:#}");
+        let hosted: Vec<String> = files
+            .iter()
+            .map(|file| std::fs::read_to_string(root.join(file)).unwrap())
+            .collect();
+        for (file, text) in files.iter().zip(&hosted) {
+            assert!(
+                text.contains(&hosted_url),
+                "hosted mode pins {file}:\n{text}\n{env:#}"
+            );
+        }
+
+        stage_manifest(&root);
+        prebuilt_common::mount_project(&server, &root).await;
+        let (code, env) = run_cli(&root, &switch, &[]);
+        assert_eq!(code, 1, "{switch:?}: the purl is refused: {env:#}");
+        assert!(
+            env.to_string().contains("pypi_extras_unsupported"),
+            "{switch:?}: refused with the requirements wiring's code: {env:#}"
+        );
+        for (file, before) in files.iter().zip(&hosted) {
+            let text = std::fs::read_to_string(root.join(file)).unwrap();
+            assert_eq!(
+                &text, before,
+                "{switch:?}: {file} keeps its hosted pin byte for byte\n{env:#}"
+            );
+        }
+        assert!(
+            !root.join(".socket/vendor/pypi").exists(),
+            "{switch:?}: nothing is vendored"
+        );
+    }
+}
+
 /// The `pipenv requirements > requirements.txt` export of [`stage_pipenv`]'s
 /// lock: the file Docker / plain-pip installs read.
 const PIPENV_EXPORT: &str = "-i https://pypi.org/simple\nsix==1.16.0 ; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2'\n";

crates/socket-patch-core/src/vendor/pypi.rs

@@ -1117,12 +1117,13 @@ async fn pypi_prelude<'p>(
                     warnings.extend(project.warnings.iter().cloned());
                     // #1368: a `uv export` requirements file is an install
                     // path of its own; wire its pin with the lock.
-                    let sibling = super::pypi_requirements::sibling_pin_wirable(
+                    let sibling = super::pypi_requirements::fresh_sibling_pin(
                         project_root,
                         &canon_name,
                         version,
                     )
-                    .await;
+                    .await
+                    .map_err(|(code, detail)| refused(code, detail))?;
                     if sibling && dry_run {
                         warnings.push(sibling_dry_run_notice(flavor, &canon_name, version));
                     }
@@ -1157,12 +1158,13 @@ async fn pypi_prelude<'p>(
                     // #1758: the requirements.txt the pylock was locked
                     // from is an install path of its own; wire its pin
                     // with the lock.
-                    let sibling = super::pypi_requirements::sibling_pin_wirable(
+                    let sibling = super::pypi_requirements::fresh_sibling_pin(
                         project_root,
                         &canon_name,
                         version,
                     )
-                    .await;
+                    .await
+                    .map_err(|(code, detail)| refused(code, detail))?;
                     if sibling && dry_run {
                         warnings.push(sibling_dry_run_notice(flavor, &canon_name, version));
                     }
@@ -1351,14 +1353,18 @@ async fn pypi_prelude<'p>(
             // requirements.txt`) is an install path of its own; wire its
             // pin with the lock, or re-wire the one the superseded entry
             // wired.
-            let sibling =
+            let sibling = if target == PipenvTarget::Fresh && superseded.is_none() {
+                super::pypi_requirements::fresh_sibling_pin(project_root, &canon_name, version)
+                    .await
+                    .map_err(|(code, detail)| refused(code, detail))?
+            } else {
                 super::pypi_requirements::sibling_pin_wirable(project_root, &canon_name, version)
                     .await
-                    || superseded.as_ref().is_some_and(|prev| {
-                        prev.wiring
-                            .iter()
-                            .any(|r| r.kind == super::pypi_requirements::RECORD_KIND)
-                    });
+            } || superseded.as_ref().is_some_and(|prev| {
+                prev.wiring
+                    .iter()
+                    .any(|r| r.kind == super::pypi_requirements::RECORD_KIND)
+            });
             // #1368: a pylock `pipenv lock` wrote beside Pipfile.lock
             // (`use_pylock = true`) still pinning the registry release.
             let pylock_states =