Grahamc/lockable inputs generally

doc/manual/rl-next/lockable-http-file-inputs.md

@@ -0,0 +1,21 @@
+---
+synopsis: "The Lockable HTTP Protocol now applies to file inputs"
+---
+
+The [Lockable HTTP Protocol](@docroot@/protocols/tarball-fetcher.md) only applied to tarball inputs.
+Nix ignored the `Link: <...>; rel="immutable"` header for an input with `flake = false` and a URL without a tarball extension, because such an input is a file input.
+
+Nix now applies the immutable URL to file inputs too:
+
+```nix
+inputs.determinate-pkg = {
+  url = "https://install.determinate.systems/determinate-pkg/stable/Universal";
+  flake = false;
+};
+```
+
+The immutable URL must have the same input type as the input that Nix fetched.
+For a file input whose immutable URL has a tarball extension, the server must put the prefix `file+` in front of the transport scheme.
+
+Nix also compares the contents that it downloads against the `narHash` in the immutable URL, and gives an error if the two values are different.
+Before, Nix ignored that value.

doc/manual/source/SUMMARY.md.in

@@ -128,7 +128,7 @@
     - [Build Trace Entry](protocols/json/build-trace-entry.md)
     - [Build Result](protocols/json/build-result.md)
     - [Store](protocols/json/store.md)
-  - [Serving Tarball Flakes](protocols/tarball-fetcher.md)
+  - [Lockable HTTP Protocol](protocols/tarball-fetcher.md)
   - [Store Path Specification](protocols/store-path.md)
   - [Nix Archive (NAR) Format](protocols/nix-archive/index.md)
   - [Binary Cache](protocols/binary-cache/index.md)

doc/manual/source/protocols/tarball-fetcher.md

@@ -1,33 +1,27 @@
-# Lockable HTTP Tarball Protocol
-
-Tarball flakes can be served as regular tarballs via HTTP or the file
-system (for `file://` URLs). Unless the server implements the Lockable
-HTTP Tarball protocol, it is the responsibility of the user to make sure that
-the URL always produces the same tarball contents.
-
-An HTTP server can return an "immutable" HTTP URL appropriate for lock
-files. This allows users to specify a tarball flake input in
-`flake.nix` that requests the latest version of a flake
-(e.g. `https://example.org/hello/latest.tar.gz`), while `flake.lock`
-will record a URL whose contents will not change
-(e.g. `https://example.org/hello/<revision>.tar.gz`). To do so, the
-server must return an [HTTP `Link` header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Link) with the `rel` attribute set to
-`immutable`, as follows:
+# Lockable HTTP Protocol
+
+Nix can fetch a flake input over HTTP, or from the file system for `file://` URLs.
+If the server does not support the Lockable HTTP Protocol, the user must make sure that the URL always gives the same contents.
+
+An HTTP server can return an "immutable" URL for the lock file.
+Then the user can put an input in `flake.nix` that asks for the most recent version of a resource (for example, `https://example.org/hello/latest.tar.gz`), while `flake.lock` records a URL whose contents do not change (for example, `https://example.org/hello/<revision>.tar.gz`).
+To do this, the server must send an [HTTP `Link` header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Link) with the `rel` attribute set to `immutable`, as follows:
 
 ```
 Link: <flakeref>; rel="immutable"
 ```
 
-(Note the required `<` and `>` characters around *flakeref*.)
+(The `<` and `>` characters around *flakeref* are necessary.)
+
+*flakeref* must have the same input type as the input that Nix fetched.
+Nix gives an error if the two types are different.
+For the two input types that this protocol applies to, refer to [Tarball inputs](#tarball-inputs) and [File inputs](#file-inputs).
 
-*flakeref* must be a tarball flakeref. It can contain the tarball flake attributes
-`narHash`, `rev`, `revCount` and `lastModified`. If `narHash` is included, its
-value must be the [NAR hash][Nix Archive] of the unpacked tarball (as computed via
-`nix hash path`). Nix checks the contents of the returned tarball
-against the `narHash` attribute. The `rev` and `revCount` attributes
-are useful when the tarball flake is a mirror of a fetcher type that
-has those attributes, such as Git or GitHub. They are not checked by
-Nix.
+*flakeref* can contain the flake attributes `narHash`, `rev`, `revCount` and `lastModified`.
+If *flakeref* contains `narHash`, its value must be the [NAR hash][Nix Archive] of the contents, as computed by `nix hash path`.
+Nix compares the contents that it downloaded against `narHash`, and gives an error if the two values are different.
+Nix does not check `rev` and `revCount`.
+These two attributes are useful when the input is a mirror of a fetcher type that has them, such as Git or GitHub.
 
 ```
 Link: <https://example.org/hello/442793d9ec0584f6a6e82fa253850c8085bb150a.tar.gz
@@ -36,10 +30,43 @@ Link: <https://example.org/hello/442793d9ec0584f6a6e82fa253850c8085bb150a.tar.gz
   &narHash=sha256-GUm8Uh/U74zFCwkvt9Mri4DSM%2BmHj3tYhXUkYpiv31M%3D>; rel="immutable"
 ```
 
-(The linebreaks in this example are for clarity and must not be included in the actual response.)
+(The line breaks in this example make it easy to read. Do not put them in the actual response.)
+
+## Tarball inputs
+
+Nix unpacks a tarball input into a tree.
+Nix uses this input type when the URL has a tarball extension, such as `.tar.gz` or `.zip`, or when the input is a flake.
+
+For a tarball input, *flakeref* must be a tarball flakeref.
+
+The value of the `lastModified` flake attribute is the timestamp of the most recent file in the tarball.
+
+## File inputs
+
+Nix puts a file input into the store as a single file.
+Nix uses this input type when the URL has no tarball extension and the input has `flake = false`.
+The URL does not have to look like a file, and it can contain query parameters.
+
+> **Example**
+>
+> ```nix
+> # flake.nix
+> {
+>   inputs.determinate-pkg = {
+>     url = "https://install.determinate.systems/determinate-pkg/stable/Universal";
+>     flake = false;
+>   };
+>   outputs = { foo }: { /* ... */ };
+> }
+> ```
+
+For a file input, *flakeref* must be a file flakeref.
+
+```
+Link: <https://install.determinate.systems/determinate-pkg/tag/v3.22.0/Universal>; rel="immutable"
+```
 
-For tarball flakes, the value of the `lastModified` flake attribute is
-defined as the timestamp of the newest file inside the tarball.
+The `lastModified` attribute does not apply to file inputs.
 
 ## Gitea and Forgejo support
 

src/libfetchers/tarball.cc

@@ -408,6 +408,48 @@ struct CurlInputScheme : InputScheme
     {
         return (bool) input.getNarHash();
     }
+
+    /* Apply the "immutable" URL that the server sent in a `Link:
+       <...>; rel="immutable"` header (see the Lockable HTTP Protocol)
+       to the input.
+
+       The immutable URL must denote an input of the same type, because
+       the server cannot change how Nix reads the resource. If the
+       immutable URL contains a `narHash`, it must agree with `narHash`,
+       the NAR hash of the data that Nix downloaded.
+
+       `requireTree` tells the parser whether the immutable URL denotes
+       a tree. */
+    Input applyImmutableUrl(
+        const Settings & settings,
+        const Input & input,
+        const std::string & immutableUrl,
+        const Hash & narHash,
+        bool requireTree) const
+    {
+        // FIXME: would be nice to support arbitrary flakerefs
+        // here, e.g. git flakes.
+        auto immutableInput = Input::fromURL(settings, immutableUrl, requireTree);
+
+        if (immutableInput.getType() != schemeName())
+            throw Error(
+                "the immutable URL '%s' of input '%s' is a '%s' input, but a '%s' input is necessary",
+                immutableUrl,
+                input.to_string(),
+                immutableInput.getType(),
+                schemeName());
+
+        if (auto expected = immutableInput.getNarHash(); expected && *expected != narHash)
+            throw Error(
+                (unsigned int) 102,
+                "NAR hash mismatch in the immutable URL '%s' of input '%s': expected '%s', but got '%s'",
+                immutableUrl,
+                input.to_string(),
+                expected->to_string(HashFormat::SRI, true),
+                narHash.to_string(HashFormat::SRI, true));
+
+        return immutableInput;
+    }
 };
 
 struct FileInputScheme : CurlInputScheme
@@ -445,6 +487,10 @@ struct FileInputScheme : CurlInputScheme
         auto file = downloadFile(store, settings, getStrAttr(input.attrs, "url"), input.getName());
 
         auto narHash = store.queryPathInfo(file.storePath)->narHash;
+
+        if (file.immutableUrl)
+            input = applyImmutableUrl(settings, input, *file.immutableUrl, narHash, false);
+
         input.attrs.insert_or_assign("narHash", narHash.to_string(HashFormat::SRI, true));
 
         auto accessor = ref{store.getFSAccessor(file.storePath)};
@@ -453,6 +499,14 @@ struct FileInputScheme : CurlInputScheme
 
         return {accessor, input};
     }
+
+    std::optional<std::string> getFingerprint(Store & store, const Input & input) const override
+    {
+        if (auto narHash = input.getNarHash())
+            return "file:" + narHash->to_string(HashFormat::SRI, true);
+        else
+            return std::nullopt;
+    }
 };
 
 struct TarballInputScheme : CurlInputScheme
@@ -510,21 +564,15 @@ struct TarballInputScheme : CurlInputScheme
             return std::nullopt;
         auto & result = *res;
 
-        if (result.immutableUrl) {
-            auto immutableInput = Input::fromURL(settings, *result.immutableUrl);
-            // FIXME: would be nice to support arbitrary flakerefs
-            // here, e.g. git flakes.
-            if (immutableInput.getType() != "tarball")
-                throw Error("tarball 'Link' headers that redirect to non-tarball URLs are not supported");
-            input = immutableInput;
-        }
+        auto narHash = settings.getTarballCache()->treeHashToNarHash(settings, result.treeHash);
+
+        if (result.immutableUrl)
+            input = applyImmutableUrl(settings, input, *result.immutableUrl, narHash, true);
 
         if (result.lastModified && !input.attrs.contains("lastModified"))
             input.attrs.insert_or_assign("lastModified", uint64_t(result.lastModified));
 
-        input.attrs.insert_or_assign(
-            "narHash",
-            settings.getTarballCache()->treeHashToNarHash(settings, result.treeHash).to_string(HashFormat::SRI, true));
+        input.attrs.insert_or_assign("narHash", narHash.to_string(HashFormat::SRI, true));
 
         return {{result.accessor, input}};
     }

tests/nixos/tarball-flakes.nix

@@ -8,7 +8,35 @@
 let
   pkgs = config.nodes.machine.nixpkgs.pkgs;
 
-  root = pkgs.runCommand "nixpkgs-flake" { } ''
+  # A resource that is not a tarball. An input that points at it is a
+  # `file` input, not a `tarball` input.
+  bom = pkgs.writeText "bom.json" ''
+    {"hello": "world"}
+  '';
+
+  # A flake with a `flake = false` input, to show that the Lockable HTTP
+  # Protocol applies to `file` inputs.
+  mkFileFlake =
+    name: url:
+    pkgs.writeTextFile {
+      inherit name;
+      destination = "/flake.nix";
+      text = ''
+        {
+          inputs.foo = {
+            url = "${url}";
+            flake = false;
+          };
+          outputs = { self, foo }: { };
+        }
+      '';
+    };
+
+  fileFlake = mkFileFlake "file-flake" "http://localhost/file/stable/aarch64-linux?a=1";
+  badHashFlake = mkFileFlake "bad-hash-flake" "http://localhost/file-bad-hash/stable/aarch64-linux";
+  badTypeFlake = mkFileFlake "bad-type-flake" "http://localhost/file-bad-type/stable/aarch64-linux";
+
+  root = pkgs.runCommand "nixpkgs-flake" { nativeBuildInputs = [ pkgs.nix ]; } ''
     mkdir -p $out/{stable,tags}
 
     set -x
@@ -27,6 +55,32 @@ let
     Redirect "/tags/latest.tar.gz" "/stable/${nixpkgs.rev}.tar.gz"
     Header always set Link "<http://localhost/stable/${nixpkgs.rev}.tar.gz?rev=${nixpkgs.rev}&revCount=1234>; rel=\"immutable\""
     EOF
+
+    # A non-tarball resource, served at a URL that has no tarball
+    # extension. Nix fetches it as a `file` input.
+    mkdir -p $out/file/{stable,v1} $out/file-bad-hash/stable $out/file-bad-type/stable
+    for d in file/stable file/v1 file-bad-hash/stable file-bad-type/stable; do
+      cp ${bom} $out/$d/aarch64-linux
+    done
+
+    # The NAR hash of the resource, percent-encoded for a URL query.
+    narHash=$(nix-hash --to-sri --type sha256 "$(nix-hash --type sha256 $out/file/v1/aarch64-linux)" \
+      | sed 's|+|%2B|g; s|/|%2F|g; s|=|%3D|g')
+
+    cat >$out/file/stable/.htaccess <<EOF
+    Header always set Link "<http://localhost/file/v1/aarch64-linux?narHash=$narHash>; rel=\"immutable\""
+    EOF
+
+    # An immutable URL with a NAR hash that does not agree with the
+    # contents.
+    cat >$out/file-bad-hash/stable/.htaccess <<EOF
+    Header always set Link "<http://localhost/file/v1/aarch64-linux?narHash=sha256-tbudgBSg%2BbHWHiHnlteNzN8TUvI80ygS9IULh4rklEw%3D>; rel=\"immutable\""
+    EOF
+
+    # An immutable URL that is a tarball flakeref, not a file flakeref.
+    cat >$out/file-bad-type/stable/.htaccess <<EOF
+    Header always set Link "<http://localhost/stable/${nixpkgs.rev}.tar.gz>; rel=\"immutable\""
+    EOF
   '';
 in
 
@@ -58,6 +112,9 @@ in
         virtualisation.additionalPaths = [
           pkgs.hello
           pkgs.fuse
+          fileFlake
+          badHashFlake
+          badTypeFlake
         ];
         virtualisation.memorySize = 4096;
         nix.settings.substituters = lib.mkForce [ ];
@@ -99,6 +156,32 @@ in
       # Check that fetching fails if we provide incorrect attributes.
       machine.fail("nix flake metadata --json http://localhost/tags/latest.tar.gz?rev=493300eb13ae6fb387fbd47bf54a85915acc31c0")
       machine.fail("nix flake metadata --json http://localhost/tags/latest.tar.gz?narHash=sha256-tbudgBSg+bHWHiHnlteNzN8TUvI80ygS9IULh4rklEw=")
+
+      # The protocol also applies to `file` inputs, that is, inputs with
+      # `flake = false` whose URL has no tarball extension.
+      machine.succeed("cp -rT ${fileFlake} /tmp/file-flake && chmod -R u+w /tmp/file-flake")
+      machine.succeed("nix flake lock /tmp/file-flake")
+      node = json.loads(machine.succeed("cat /tmp/file-flake/flake.lock"))["nodes"]["foo"]
+      print(node)
+
+      # Check that the lock file records the immutable URL, and that the
+      # input is still a file.
+      assert node["locked"]["url"] == "http://localhost/file/v1/aarch64-linux"
+      assert node["locked"]["type"] == "file"
+      assert node["original"]["url"] == "http://localhost/file/stable/aarch64-linux?a=1"
+
+      # Check that Nix compares the contents against the `narHash` in the
+      # immutable URL.
+      machine.succeed("cp -rT ${badHashFlake} /tmp/bad-hash-flake && chmod -R u+w /tmp/bad-hash-flake")
+      out = machine.fail("nix flake lock /tmp/bad-hash-flake 2>&1")
+      print(out)
+      assert "NAR hash mismatch in the immutable URL" in out
+
+      # Check that Nix rejects an immutable URL of a different input type.
+      machine.succeed("cp -rT ${badTypeFlake} /tmp/bad-type-flake && chmod -R u+w /tmp/bad-type-flake")
+      out = machine.fail("nix flake lock /tmp/bad-type-flake 2>&1")
+      print(out)
+      assert "is a 'tarball' input, but a 'file' input is necessary" in out
     '';
 
 }