Repository navigation
Node.js package, s3 backend, and the behavior of rclone - #32
Merged
Merged
Conversation
eme.js now follows the paper of Halevi and Rogaway and rfjakob/eme step by step, with the names of the paper (P, C, T, L, PPP, MP, MC, M, CCC), and a comment for each step. The doubling in GF(2^128) changes the block in place, with no branch on the data. The checks give a TypeError or a RangeError with a clear message, and the check of the sizes in xorBlocks, which used && in place of ||, goes away. The exports are encrypt and decrypt. The output is the same as before for 2000 random inputs, and the speed is the same. eme_test.js checks the two vectors of rfjakob/eme (16 and 2048 bytes) with the AES of aes-js and of node:crypto, the decrypt of each encrypt, the change of each block when one bit of the input or of the tweak changes, and the checks of the input.
decryptName gave 0 blocks to EME for an empty segment of a path, for example the end of a folder (x/), and EME threw. An empty segment now stays empty in the two directions, as in rclone. The test encode, with an empty name from fast-check, passes now.
aes.js gives AES-256-ECB of node:crypto with no padding as the block cipher of EME, and the dependency aes-js goes away. It works in Node.js and in workerd with nodejs_compat at the compatibility date of wrangler.toml (a check with wrangler dev). workerd refuses null as the IV, so the IV is empty. A block cipher of EME now takes one or more blocks (ECB), so each ECB layer of EME is one call: 4 calls for each name. With node:crypto, 2048 bytes take 58 us (176 us before), and a short name about 9 us. EME copies with new Uint8Array in place of slice: the slice of a Buffer is a view, and node:crypto gives Buffers. A new test examines a Buffer as input. The output is the same as the first code for 2000 random inputs.
deno.json, the Deno settings of VS Code and the engine deno go away. main.js runs with node, and the backends have no shebang of deno serve: a backend is a module, not a program. The two benchmarks run with node and lib/bench.js, a small helper with the timer of Deno.bench, in place of Deno.bench and Deno.Command (npm run bench). The README and CONTRIBUTING.md tell the npm commands and node:test. The tests give the same result as before: 68 of 93 pass on Node.js 26.
The key of obscure came from a top-level await. With the circular imports of mod.js and the backends, esbuild puts each module in a lazy function, and a top-level await cannot be in such a function: wrangler pages dev 4.142 could not build _worker.js, also on the first commit. The key is now a promise that getKey makes at its first use.
deps.js and dev_deps.js re-exported the packages, as deps.ts in Deno. Now each module imports its packages, and package.json lists them. The modules that use HTMLRewriter import it: crypt used the global that deps.js set. Node.js has the same functions for four packages, so they go away: @std/path is node:path, @std/encoding is Buffer, the MD5 of @std/crypto is createHash of node:crypto (with no WebAssembly, which Workers block), and nobody used @std/http. The tests import node:assert/strict and fast-check. The helpers of deps.js go to their users: toLocaleISOString to lib/time.js, config_dir to cmd/config. The tests give the same result as before (68 of 93 pass). In wrangler pages dev, PUT, GET and the list of memory and of crypt over memory work, and rclone 1.75.1 decodes the name that crypt wrote.
The date was 2024-06-28. 2026-09-01 is also the date of mkfifo.com, and workerd 1.20260926 has it. At this date nodejs_compat is on by default; the flag stays, so that the file tells it. In wrangler pages dev at this date, with no flag, PUT, GET and the list of memory and of crypt over memory work. workerd still refuses a null IV for AES-ECB, so aes.js keeps the empty IV.
These three packages came from JSR through npm:@jsr aliases and the .npmrc of this repository, so a different project could not install rfetch. Now small modules of rfetch do the work, with the APIs of the web and of Node.js, and .npmrc goes away: - lib/format.js: formatBytes and formatDuration give the text of @std/fmt for the options of rfetch (no difference in 160,103 cases), but with KiB in place of kiB, the symbol of IEC, as rclone writes it. - lib/ini.js: parse and stringify in the format of the configuration file of rclone: [name], key = value, and an empty line between sections. @std/ini 0.225 did not use the options section and whitespace that cmd/config gave it. Two sections with the same name become one, as in rclone, and a name such as __proto__ does not change the prototype. - lib/mime.js: mimeType as MimeTypeFromName of rclone: the table of Go, the types that rclone adds, and .txt. An unknown extension gives application/octet-stream, as in rclone, in place of an empty Content-Type. The new tests of lib pass.
With no configuration file, config file gave an empty path, and create wrote nowhere. Now the file is the default file of rclone, CONFIG_DIR/rclone/rclone.conf, as the test expects. With the new INI module, the 7 tests of cmd/config pass.
pages_build_output_dir makes this file the configuration of the deploys from Git and of wrangler pages dev, in place of the settings of the dashboard. The output is the repository root, with _worker.js. The name is rfetch: denoclr was the earlier name, rcloned in reverse, with Deno in it.
decodePath gives the path of a URL with no percent-encoding. It gives null for a path that can go out of the root, for example /..%2Fsecret.txt, which the URL parser keeps. escapeHTML puts a character reference in place of each character that has a meaning in HTML.
backendSuite(name, backend) runs the same tests on each backend, in a new folder that it deletes at the end: - 404 for a file or a folder that does not exist; - PUT and DELETE of files and folders; - the names of the test FsEncoding of rclone; - a file of 0 bytes, and the two files of fstests; - the list of each folder; - the bytes and the headers of GET and HEAD. remote() is a client of a backend in the same process. parseListing reads the HTML list of a folder.
The memory backend now: - removes the percent-encoding of the path, and gives 400 for a path that is not safe; - writes each name with percent-encoding in the link, and with escapes in the HTML; - gives the type of each file in the list, not the type of the folder; - gives Content-Type, Content-Length and Last-Modified for GET and HEAD of a file; - gives 404 for HEAD and DELETE of a path that does not exist, and 405 for other methods; - finds the files of a folder with no regular expression, which failed for a name with ( or [; - gives the size of a file of 0 bytes in the list. The tests of the memory backend are now the shared suite.
The local backend now: - removes the percent-encoding of the path, and gives 400 for a path that can go out of the current folder, for example /..%2Fsecret.txt; - gives 403 for DELETE /. Before, DELETE / deleted the current folder and all that is in it; - gives 404 for a path that does not exist, in place of an error; - creates the parent folders for PUT of a file, and accepts a PUT with no body; - writes each name with percent-encoding in the link, and with escapes in the HTML, and gives the type of each file. The tests use a new temporary folder as the current folder. They run the shared suite, and tests for the root folder.
The links of a list have percent-encoding. Thus lsjson gave the name a%20b.txt for the file a b.txt.
CONTRIBUTING.md tells the answers that a backend gives, how to use decodePath and escapeHTML, and how a backend runs the shared test suite.
…arser lib/ini.js now has the rules of goconfig v1.0.0, which rclone uses to read and write its configuration file: - the white space of Go, and the byte order mark; - ":" as the end of a key, as "="; - keys and values in quotes; - the keys before the first section in the section DEFAULT; - numbers for the keys "-". ParseStream is a TransformStream from the text, in chunks of any size, to an entry for each section and each key. parse(text) and read(stream) use the same reader of lines. stringify puts a key or a value in quotes when rclone does not read it the same with no quotes. It throws a TypeError for what rclone cannot read back. rclone 1.75.1 itself writes the value a`b as "a`b" and the value " a " with no quotes, and then it reads back different values. An error gives the number of the line, but not its text, which can have a password. The property tests: - parse reads what stringify writes, or stringify throws a TypeError; - stringify writes each configuration that rclone makes, and each configuration that parse reads; - read gives the result of parse for each split of the text; - parse reads each file and each line as rclone 1.75.1 reads it, and rclone reads what stringify writes. These tests run when rclone is installed, or when RCLONE is its path.
…he file cannot keep The old rule for names accepted a line break, and a space at the end. A line break in a name could break the file. Now the rule is fspath.CheckConfigName of rclone. create and update give 400, and do not change the file, for an option that the file cannot keep, for example a value with a line break. The error is in the body, because a status text cannot have all characters. show and update find only the own keys of the configuration: the name toString is not a remote.
The note on the tests against rclone had long lines.
ConfigFile changes only the lines of a change: update, set and delete. All other lines stay as they are: the comments, the empty lines and the format of each line. A comment directly above a section or a key, with no empty line between, is a comment of that line. It goes away only when that line goes away. All other comments always stay. rclone 1.75.1 does less: when it saves the file, it loses the comments at the end of the file, and a delete of a section also deletes the comments above the section that are for the whole file. Other changes: - The entries of ParseStream have a type (section, key or comment) and the number of the line. - A key that starts with "[" or U+FEFF needs no quotes. A line of a key that starts with "[" must not end with "]", thus its value goes in quotes then. The property tests found such keys, which parse reads but stringify could not write. - When a key "-" goes away, the keys "-" after it get their numbers as keys in quotes. Thus their numbers do not change. - The source has escapes in place of invisible characters. The new property tests: - ConfigFile keeps the text of each file that parse reads; - ConfigFile changes the configuration as a model of maps; - ConfigFile removes only the comments directly above a line that goes away; - rclone reads the file that ConfigFile changes.
create, update and delete change the file with ConfigFile. show with no name gives the whole file with its comments, as rclone config show does. The tests use a new folder as the current folder and as the home folder. Before, they wrote rclone.conf in the folder of the repository, as main_test.js does, and the two test files ran at the same time.
The data: - DecryptionStream reads the header and the blocks from chunks of any size. Before, the header had to be in the first chunk. - It checks the magic bytes, and gives the errors of rclone: file too short, truncated block header, bad magic, block that does not authenticate. - encryptedSize and decryptedSize are the functions of rclone, and decryptedSize throws for a size that no encrypted data has. The names: - PathCipher removes the padding with the checks of rclone, and refuses bad UTF-8, a size that is not a number of blocks, and a name of more than 2048 bytes. - padding.js goes away: crypt uses only PKCS#7. The tests use the vectors of backend/crypt/cipher_test.go of rclone: the names, the paths, the files of TestEncryptData, the sizes and the errors. Properties test each split of the chunks and each size of block.
The crypt backend now: - puts no password and no salt in the links of a list. Before, each link had the query of the request with the password, and a browser keeps the links in its history; - writes each name with percent-encoding in the link and with escapes in the HTML, and shows no link for a name that does not decrypt, as rclone skips such a name; - gives the error of the remote, for example 404, and does not decrypt it; - gives Content-Type, Content-Length and Last-Modified for GET and HEAD, and no other header of the encrypted data; - sends the encrypted size as Content-Length on PUT, and not the type of the file; - gives 400 for a path that is not safe, with no remote, with no password, or with a password that is not obscured; - makes the keys as rclone: an empty password gives keys of zero bytes. It keeps the keys of the last 16 passwords, as scrypt takes much time. The tests have the key vectors of rclone (TestKey) and run the shared suite over the memory backend. In a Worker, a response of a stream has no Content-Length. Thus crypt sends the data through a FixedLengthStream when the runtime has one, and the memory backend gives a Blob.
rclone copy encrypts files and rfetch reads them: the bytes, the sizes and the lists, as rclone lsjson -R. rfetch encrypts files and rclone reads them: rclone copy and rclone cryptcheck. A property sends random names and data in the two directions. The files have hard names and the sizes 0, 1, 65535, 65536, 65537 and 1 MiB + 1. The tests run when rclone is installed.
lib/listing.js reads and writes the HTML table of a list. The tests and the backends use it: - readListing gives the entries of a list, as rclone lsjson. It was lib/testing/listing.js. - renderListing writes a list from the entries. The memory backend uses it, and other backends can use it. - relink puts a query in place of the query of each link of a list, for a backend on top of another backend. - childOf gives the name of a file or a folder directly in a folder, from a link. A link of renderListing has a query only when the query is not empty.
The shared suite has two new options: - skipName: the tests skip the names that match it. Use it only for a limit that the backend has in rclone too. - writer: for a read only backend, the PUT and DELETE requests go to this backend, and the backend under test reads the files. When a test of a name fails, the test deletes its folder. Thus the next tests do not see its files.
The alias backend now: - gives 400 with no remote, and for a path that is not safe. Before, it threw an error; - sends the path with percent-encoding to the remote, with the method, the headers and the body of the request; - puts its own query in the links of a list, not the query of the remote; - gives the path of the alias in Content-Location. lib/path.js has encodePath, the reverse of decodePath.
parseSize reads a size as SizeSuffix of rclone: no suffix is KiB, b is bytes, and k, M, G, T, P and E are powers of 1024, with an optional i and B. off is -1. The test has the vectors of TestSizeSuffixSet of rclone.
The chunker backend now writes and reads the files of the chunker of rclone 1.75.1: - The names of the chunks are as name_format, with start_from, and with the suffix of a transaction for norename. A name with the form of a chunk, a control chunk or a temporary chunk is not a file of the list. - The metadata is simplejson, version 1 or 2, with the size, the number of chunks, md5 or sha1, and the transaction. As rclone, it reads at most 1023 bytes and writes at most 255 bytes, and it refuses metadata that is not correct. - meta_format none needs hash_type none, as in rclone. Then a file is always in chunks. - A small file goes to the remote as it is. When its content looks like metadata, it gets one chunk and metadata. - A file of unknown size goes in chunks with metadata when it is larger than 1 MiB or than the chunk size. - A list gives the sum of the sizes of the chunks, and it does not show a file with missing chunks. GET of such a file gives 500. - DELETE deletes the metadata first, then the chunks. - chunk_size is a size of rclone, 2 GiB by default. The backends of rfetch cannot rename. Thus with the transactions rename, a reader can see a file that is not complete. The transactions auto are norename. The tests run the shared suite in three modes, and check the files on the remote. With rclone, the tests copy files in the two directions, and check that rfetch and rclone make the same files. In the mode rename, the chunks and the metadata are the same bytes.
parseList reads an option of the type CommaSepList or SpaceSepList of rclone. As in rclone, it reads one record with the CSV rules of Go: a value in quotes can have the separator, a line break, and "" for one quote. A property test sends 5000 random texts to parseList and to encoding/csv of Go, and they must give the same values and the same errors. It runs when Go is installed.
toLocaleString and localeCompare make a new format or collator of the locale for each call. Now renderListing uses one Intl.Collator, and formatBytes uses one Intl.NumberFormat. The text and the order are the same. In Node.js, renderListing of 10000 entries: before 69 ms, after 58 ms.
A handler of an end tag is one more call from the HTML parser into JavaScript for each row, which costs much time in a Worker. Now the entry of a row goes in the list at its link. The last link of a row gives the entry, as before. The new tests in lib/listing_test.js pass with the old and the new code. In workerd, readListing of 10000 entries: before 397 ms, after 375 ms.
lsjson had its own HTMLRewriter for the rows, and a second TransformStream to remove the last comma. For --recursive, it parsed again the JSON lines of each folder below, and it thought that each chunk was one line. Now lsjson reads each list with readListing of lib, and one ReadableStream gives one chunk and line for each entry. The lists of the folders below start at the same time, as before. The output has a fixed order now: the entries of a folder, then the entries of each folder in it. A check of 12 cases against the old code gave the same output. A folder below that lsjson cannot list now stops the list with an error, as 'rclone lsjson -R' exits with an error. Before, the list ended with no entries of that folder, and with no error. Crypt lsjson of 1000 files, in workerd: before 188 ms, after 97 ms. lsjson of 1000 files of memory, in Node.js: before 56 ms, after 43 ms.
The data of rclone crypt is NaCl secretbox: XSalsa20 and Poly1305. Node.js, WebCrypto and Workers have no Salsa20, thus the blocks went through @noble/ciphers in JavaScript. Now backend/crypt/secretbox.wasm seals and opens the blocks, up to 16 blocks for each call. Its source is the Rust crate in backend/crypt/native, on crypto_secretbox of RustCrypto, as the crypt NIF of mkfifo. The module has no imports, and it uses the SIMD of WebAssembly. The tests of rclone (TestEncryptData) and the tests against real rclone pass. Node.js, Deno and Bun read the file. A Worker cannot compile a module at run time, thus the field imports of package.json gives wrangler a file that imports the module. The new job wasm of CI builds the module again, and compares its bytes with the file in the repository. The key of a password now comes from the scrypt of node:crypto, which is native code in Node.js and in workerd. @noble/ciphers and @noble/hashes are not dependencies now. The test of the imports of the workspaces accepts a name of the field imports of package.json, to a file in the folder. 8 MiB in Node.js: PUT before 78 ms, after 39 ms; GET before 67 ms, after 37 ms. 8 MiB over memory in workerd: PUT before 200 ms, after 138 ms; GET before 157 ms, after 109 ms. scrypt of a password: in workerd before 147 ms, after 34 ms; in Node.js before 92 ms, after 48 ms.
The memory backend kept each file as a File, and each folder as a key with no value. A list read all the keys of all the folders, and a GET gave the File as the body: a Worker reads a Blob body in chunks of 4 KiB. Now a file is its bytes, its media type and its time, and a GET gives a copy of the bytes as the body. Each folder has the names of its children, thus a list reads only its folder. The behavior is the same, and the shared suite passes. A list of 10 files, with 10000 other files in the store, in Node.js: before 0.39 ms, after 0.13 ms. 8 MiB in workerd, in one run with the two versions: PUT before 12 ms, after 8 ms; GET with a reader before 31 ms, after 24 ms. A delete of a folder of 10000 files: before 1.5 ms, after 3.7 ms.
The module had the path of the crates of cargo on the machine of the build, for example /root/.cargo/registry/..., thus the job wasm of CI built other bytes. Now build.sh changes the path of the crates to /cargo and the path of the crate to /crate, with --remap-path-prefix. Two builds in two folders, with two homes of cargo, gave the same bytes. The flags are in build.sh now, in place of .cargo/config.toml: the variable of the flags takes the place of the flags of the file.
rclone lsjson remote:tree lists the folder tree. rfetch lsjson read a path with no / at the end as a file: it listed the parent folder and gave only the entry of tree. Now, when no file has the name and a folder has it, lsjson lists that folder, as rclone. A file of the name is still a file. The comparison with rclone found the difference.
A GET read the file in parts of 64 KiB, with one call to the system for each part. Now a part is 1 MiB. The list of a folder made the stat of each file one after the other. Now the stats go at the same time, and a file that goes away before its stat is not in the list. rfetch cat of 64 MiB of a local file, the whole command: before 257 ms, after 217 ms; cat of a chunker file of 64 MiB in chunks of 8 MiB: before 324 ms, after 253 ms. In one process: the list of a folder of 100 files, before 11.5 ms, after 8.9 ms; lsjson --recursive of 1000 files, before 156 ms, after 102 ms.
bench/rclone.js runs the same commands with the two tools, with the same remotes in the variables RCLONE_CONFIG_*, on the same files: cat, rcat and lsjson --recursive on local files, crypt and chunker. rclone writes the files of crypt and chunker. First the script makes sure that the two tools give the same data and the same lists, and that rclone reads the files that rfetch writes. The time is the time of the whole process. The tools run in a temporary folder, with relative paths: the local backend of rfetch reads only the files in its current folder.
The command line loaded all the commands through mod.js, and with them HTMLRewriter. Now main.js loads only the module of its command, from the list cmd/folders.js. The fetch of rfetch is in fetch.js: mod.js, the commands and _worker.js import it. An unknown command gives an error and the help, with the exit code 1. The median time of 21 runs, in Node.js 22, before and after: - obscure: 135 ms, 109 ms. - help: 123 ms, 103 ms. - cat of a small file: 127 ms, 114 ms. The test cmd/folders_test.js makes sure that cmd/folders.js has each command of cmd/main.js.
The new module @rfetch/lib/rewriter gives htmlRewriter(). It loads HTMLRewriter at its first use, thus a command that reads no list does not load it. readListing, relink, the list of crypt and the index of http use it; relink and the list of crypt are now async. In Node.js, the HTMLRewriter of WebAssembly puts each chunk of its output in the queue of a stream. A list in one large chunk gave a queue of more than 16000 chunks, and a read from such a queue copies it: the time was quadratic. Now HTMLRewriter gets the HTML in chunks of at most 8 KiB, and the small chunks become one write. The HTMLRewriter of workerd and of Bun gets the chunks as they come: a stream of JavaScript between two of them costs more time. bench/node.js, before and after: - crypt: list 1000 files: 1022 ms, 85 ms. - crypt: lsjson of 1000 files: 627 ms, 150 ms. - lib: readListing of 10000 entries: 2990 ms, 349 ms. In workerd, the times do not change: crypt lsjson of 1000 files is 88 to 98 ms before, and 96 to 102 ms after.
node bench/rclone.js [RUNTIME...] runs rfetch on each runtime that the arguments name, for example node, deno and bun, and rclone. Each runtime must give the same data and the same lists as rclone. The output is a table in Markdown: one row for each case, and one column for each runtime and for rclone. The median of 5 runs, in ms, on this code (rclone 1.75.1): - start: node 22 164, node 24 140, node 26 124, deno 110, bun 43, rclone 47. - local lsjson --recursive, 1000 files: node 22 286, node 24 270, node 26 252, deno 281, bun 68, rclone 52. - crypt cat 64 MiB: node 22 449, node 24 474, node 26 488, deno 510, bun 448, rclone 216.
The new scanTags of @rfetch/lib/html reads the tags of an HTML document in JavaScript, as the tokenizer of the HTML standard: the attributes, the comments, and the text of script, style, textarea and title, which is not markup. readListing, and the index of the http backend, read the lists with it. The crypt list and relink of alias read the list of the other backend with readListing, change the entries, and write the list with renderListing. Thus rfetch does not use HTMLRewriter, and lib/rewriter.js goes away. @sntran/html-rewriter stays only as a devDependency of lib: a property test makes sure that readListing gives the entries of HTMLRewriter for each list of renderListing. An attribute value now gets the decode of the basic and the numeric character references, as the standard tells. HTMLRewriter gives the raw value. A list of rfetch has no reference in its links. bench/node.js, before and after: - lib: readListing of 10000 entries: 349 ms, 62 ms. - crypt: list 1000 files: 85 ms, 31 ms. - crypt: lsjson of 1000 files: 150 ms, 36 ms. workerd (wrangler dev, 3 runs), before and after: - lib: readListing of 10000 entries: 297 to 379 ms, 49 to 64 ms. - crypt: list 1000 files: 41 to 60 ms, 24 to 29 ms. - crypt: lsjson of 1000 files: 88 to 102 ms, 34 to 39 ms. The command line in Node.js 22, median of 21 runs, before and after: - lsjson of an empty folder: 140 ms, 116 ms. - lsjson --recursive of 1000 local files: 333 ms, 256 ms.
A local path of the command line or of the code is now relative to the current folder, as in rclone: file.txt, ../file.txt and /tmp/file.txt. Before, the local backend joined each path to the current folder, thus /tmp/x was the folder tmp/x in it, and ../x was x. fetch.js resolves the path as a URL on the file URL of the current folder, and gives the local backend the absolute path, with RFETCH_LOCAL_ROOT: / in env. RFETCH_LOCAL_ROOT is not an option: fetch.js removes it from the options, thus a config file, a variable, a header or the query cannot set it. :local:path, a named remote of type local, and a client of rfetch serve, which gives no env, stay in the current folder. The Worker gives no local path: a path with no remote is a file of its site. The local backend refuses to delete its root and the current folder. Tests: an absolute path and ../ read a file out of the current folder; a header, a variable and a remote cannot set the root; a delete of the current folder gives 403.
The command line gave the exit code 0 and no message for an error. Now an answer that is not ok writes the error to stderr, and gives the exit code 3 for 404, as "directory not found" of rclone, else 1. lsjson of a path that names no file and no folder gives 404, as rclone. Before, it gave an empty list. The output waits for drain when the pipe is full: before, a slow reader kept all the output in memory. A reader that closes the pipe, for example head, stops the command with no error. Before, Node.js wrote the stack of an EPIPE error. Node.js, Deno and Bun give the same data, and stop with no error.
rclone lsjson gives the time of a local file with 9 digits after the second, for example 2026-10-18T15:02:41.540476887Z: the format of Go for a remote with a precision of 1 ns. rfetch gave milliseconds. Now the local backend writes the time of each file and folder of a list with 9 digits, from the stat in nanoseconds. renderListing and toLocaleISOString keep the digits of a time, from 3 to 9, thus crypt and chunker on a local remote give them too. A remote with milliseconds, for example memory, keeps 3 digits. The size of a folder stays -1, as for the other backends. Node.js and Bun give the nanoseconds of a file. Deno gives milliseconds, thus on Deno the last 6 digits are 0. cmd/lsjson/rclone_test.js now compares the ModTime of each file and folder with rclone, digit for digit. bench/rclone.js compares the times to the millisecond, for Deno.
The crate salsa20 has no SIMD for WebAssembly, and Salsa20 was 55% of the time of the module. The new src/salsa.rs of the crate computes 4 blocks of Salsa20 at a time, one in each lane of the SIMD of WebAssembly, and HSalsa20 for the key of each box. Poly1305 is the crate poly1305, as before, and the compare of the tags takes the same time for each tag (subtle). crypto_secretbox is now only a dev-dependency. The boxes are the same bytes as before: - cargo test compares each box with crypto_secretbox for 16 sizes, with a counter of blocks near a carry, and a bad box and a bad tag do not open. The code with no SIMD runs on the machine of the build. - secretbox_test.js decrypts the box of the first test of NaCl, of 131 bytes, over 3 blocks of Salsa20, with the module. - The rclone tests of crypt compare the data with rclone. The job wasm of CI runs cargo test. The module only, 64 MiB in blocks of 64 KiB, in Node.js 22: seal 167 ms before, 110 ms after; open 183 ms before, 128 ms after. bench/node.js: crypt PUT 8 MiB 33.7 ms before, 20.8 ms after; crypt GET 8 MiB 35.7 ms before, 29.5 ms after. Bun gains less: its WebAssembly makes less of the SIMD.
EncryptionStream and DecryptionStream joined the bytes that were not a full block yet with each new chunk. A request body in a Worker comes in chunks of a few KiB, thus each chunk of 4 KiB copied up to 64 KiB. Now the chunks stay in a list, and join one time when a whole block, box or header is there. 8 MiB in chunks of 4 KiB, in Node.js, 2 runs: seal 73 and 90 ms before, 37 and 45 ms after; open 55 and 60 ms before, 33 and 37 ms after. Chunks of 64 KiB and of 1 MiB do not change. workerd (wrangler dev, 3 runs), with the module of the commit before: crypt PUT 8 MiB 107 to 145 ms before, 69 to 82 ms after; crypt GET 8 MiB 102 to 124 ms before, 72 to 78 ms after.
rfetch serve gave the request of a client to the backend with its query and its headers. A field of the query won over the option of the served remote, for example ?remote=/etc/ for an alias, and the headers RFETCH_* and RCLONE_* went on to the remotes below, where fetch.js reads them as options and remotes. Thus a client could read the other remotes of the config, and with the absolute local paths of the commit before, each file of the disk. Now serve removes the query and these headers of each request, and sets only the options of the served remote. The Worker has no local backend now: a client of the Worker tells all its options, thus an alias to / would read the files of the runtime. In workerd, node:fs reads no file today, but in Node.js it does. Tests: a client of serve with ?remote= or the headers of an option reads only the served folder; the Worker gives 404 for :local:, for an alias to /etc/, and for a named remote of type local. Both tests fail with the code before.
scanTags of d1133f0 was a small HTML parser of rfetch. A list can come from any server: the index page of the http backend, or the remote of a crypt at a URL. A small parser can read such a page wrongly. Now readListing and the index of http read HTML with readElements of the new @rfetch/lib/rewriter: HTMLRewriter, that is lol-html, the HTML parser of Cloudflare. - In workerd and in Bun, it is the HTMLRewriter of the runtime. - In Node.js and in Deno, it is @sntran/lol-html, lol-html 2.8 in WebAssembly, with no stream: each chunk of HTML goes to write(), and the output goes away. Thus no queue of output chunks grows, as it did with html-rewriter-wasm. It loads at its first use. HTMLRewriter gives the value of an attribute as it is in the document. attributeOf decodes the character references with decodeHTMLAttribute of the package entities, as the HTML standard tells. scanTags decoded only the basic and the numeric references. A new case in backend/http/rclone_test.js has a script, a comment and the reference é: rfetch and rclone list the same files. With scanTags, this case fails. The list of crypt and relink of alias stay as in d1133f0: they read the entries with readListing and write the list with renderListing. bench/node.js in Node.js 22, with scanTags and now (3 runs): - lib: readListing of 10000 entries: 62 ms, 140 to 154 ms. - crypt: list 1000 files: 31 ms, 38 to 43 ms. - crypt: lsjson of 1000 files: 36 ms, 49 to 53 ms. With html-rewriter-wasm, before d1133f0, these were 349, 85 and 150 ms. workerd (wrangler dev, 3 runs), with scanTags and now: - lib: readListing of 10000 entries: 49 to 64 ms, 256 to 297 ms. - crypt: list 1000 files: 24 to 29 ms, 42 to 58 ms. - crypt: lsjson of 1000 files: 34 to 39 ms, 73 to 102 ms. These are the times of the HTMLRewriter of workerd before d1133f0. The output goes to arrayBuffer(): a pipe to a WritableStream of JavaScript took 835 to 922 ms for 10000 entries. Bun, now: 49 to 53 ms, 16 to 23 ms and 21 to 32 ms.
rfetch serve http REMOTE:PATH served the root of the remote, or the current folder for a local path, and not PATH. The backend got only the options in the query of TRACE: the options of the config file and of the variables did not go, for example the password of a crypt remote in the config file. Now each request of a client goes to the fetch of rfetch, at REMOTE:PATH/ and the path of the request, with the flags of the backend in the query. Thus the remote gets all its options, as for the other commands. The server now has all the options of the remote, thus these rules protect it: - A client only reads, as with rclone serve http: each method but GET and HEAD gives 405. The fetch of rfetch answers TRACE with the options of the remote, with its secrets. - A name with a / after the decoding, for example ..%2F, gives 400. - The links of a list have no query: the server reads no query, and the options of the backend do not show. - An error gives 500 with no message, and the server continues. The server is now node:http, not @sntran/serve. In @sntran/serve 0.0.1 to 0.0.3, new Request() refuses the method TRACE with a TypeError, and no code catches it: one TRACE request of a client stopped rfetch serve with the exit code 1. node:http works in Node.js, Deno and Bun. The test of the global fetch in main_test.js uses listen of @rfetch/lib/testing/server. main.js knows the flag --addr, thus --addr 127.0.0.1:8080 works. Before, only --addr=127.0.0.1:8080 worked: parseArgs read the address as an argument of the command. Tests: cmd/serve/main_test.js serves a subfolder, a named remote of the variables, a crypt remote of a config file, and a crypt remote with the flag crypt_password. ..%2F gives 400. TRACE, PUT and DELETE give 405, and the file stays. The links of a list have no query. An error gives 500, and the next request works. main_test.js starts rfetch serve http . with --addr 127.0.0.1:0, sends TRACE, then reads a file.
The new section Runtimes tells how to run rfetch on Node.js, Deno and Bun, and gives one run of bench/rclone.js on Node.js 22, 24 and 26, Deno 2.9, Bun 1.4 and rclone 1.75.1. It recommends no runtime: the times change from one run to the next and from one machine to the next, thus a user runs the benchmark on the machine of the user to choose.
The property of f2e5f34 made a name of each text, and failed about one time in five runs. The link of the name . is the folder, the link of .. is its parent, and a name with / is two names: readListing cannot give them back, and decodePath refuses them. Now the property makes no such name. It passes in 150 runs.
The parseArgs of Node.js takes the next argument as the value only for a flag that it knows as a string. main.js told it few flags, with strict false: thus --crypt-password x or --max-depth 2 gave the flag true, and the value became an argument of the command. Now parseFlags of the new cmd/flags.js tells parseArgs the type of each flag, as rclone: - FLAGS has the flags of the commands. A boolean flag takes no value, or a value after =, for example --dir-slash=false. A string flag takes the next argument, or a value after =. - A flag of a backend, for example --crypt-password, has the type of the option in the options of the module of the backend. Only the backends of the flags load. alias, crypt and drive now export their options. - An unknown flag, a flag with no value, and a boolean value that Go does not read give the error of rclone, with the exit code 1. - A flag with the name no-... is not the negation of a flag. Before, allowNegative made --no-clobber of copyurl into clobber: false. - --help shows the help, and rfetch help flags lists the flags. - The short flags of rclone: -R, -F, -s, -d, -a, -p, -n. Two bugs of the boolean flags in the lists, fixed with isTrue of @rfetch/lib/options, which reads true and the text true: - lsd gave recursive the text false, which is true in JavaScript: thus lsd listed all the folders below. Now it lists one level, as rclone. - lsjson . and lsjson .. gave directory not found. A path with . or .. at the end is a folder now, as in rclone. Tests: cmd/flags_test.js reads flags after a space and after =, the flags of backends, the booleans of rclone, the arguments after --, the errors, and each option of each backend of rfetch as a flag. main_test.js runs lsf with flags after a space, an unknown flag, lsd and lsjson .
…, as rclone rclone gives a folder with no time the time of the flag --default-time, 2000-01-01T00:00:00Z. lsjson of rfetch gave "", for example for the folders of memory, http and s3. Now lsjson gives DEFAULT_TIME of @rfetch/lib/time. The precision of the http backend of rclone is 1 s: lsjson gives 2026-01-02T03:04:05Z. rfetch gave 2026-01-02T03:04:05.000Z. Now the http backend writes its times to the second, and toLocaleISOString keeps the digits of a time in a string from 0 to 9, not from 3 to 9: the digits tell the precision of the remote. A Date still gives 3 digits. Test: lsjson -R of rfetch and of rclone on the same http index, of python3 -m http.server, give the same items. lsjson of a folder of memory gives 2000-01-01T00:00:00Z.
The lists of text of rfetch had tabs and ISO times. Now each line is the line of rclone 1.75: - ls: the size on the right of 9 characters, then the path. - lsl: the size, the local time 2006-01-02 15:04:05.000000000, then the path. - lsd: the size on the right of 12 characters, the local time to the second, the count of items on the right of 9 characters, then the path with no / at the end. A folder has -1 for the size and the count. rclone gives the size of a local folder on the disk, for example 4096: rfetch gives -1, as the other backends. - --human-readable: the size with a suffix of 1024, for example 120.562Ki, and the count with a suffix of 1000, in fields of 9 and 8 characters. The rounding is the rounding of %.3f of Go: a half goes to the even digit. - lsf: the letter t gives the local time 2006-01-02 15:04:05. With --csv, the separator is , when the flag does not tell it, and a value with the separator, a quote, a line break or a space at the start is in quotes, as csv.Writer of Go. The new listLines of cmd/lsf/main.js reads the items of lsjson, and each command gives the line of an item. toListTime of @rfetch/lib/time writes the times, and sizeString, countString, sizeField and countField of @rfetch/lib/format write the sizes and the counts, as SizeString and SizeStringField of rclone. Tests: cmd/lsf/rclone_test.js compares the lines of ls, lsl, lsd and lsf with the lines of rclone on the same local files, with names that CSV puts in quotes. It passes with TZ UTC, America/New_York and Asia/Kathmandu. The tests of each command check the lines on memory, and lib/format_test.js checks the texts of SizeSuffix and CountSuffix of Go.
listLines read the body of lsjson with a TransformStream on chunks, not on lines. It worked only because lsjson gives each line in its own chunk. When the body comes in other chunks, for example from the Worker on the network, the list was empty and there was no error. The new LineStream of cmd/lsjson/main.js is a TransformStream from bytes to bytes. It keeps only the part of a line that has no "\n" yet, as the ParseStream of @rfetch/lib/ini, and gives the line of each item. A line that is not an item is an error of the stream. ls, lsl, lsd and lsf send the body of lsjson through it, thus any body of lsjson can go through it. Tests: LineStream reads a body in one chunk, in a chunk for each byte, and in chunks of 7 bytes, and gives an error for a broken line.
On Bun 1.3, rfetch cannot read the list of a local folder: the HTMLRewriter of Bun gives the error "Failed to pipe stream". An http remote works. On Bun 1.4.0 and 1.4.3, the lists of a local folder and of an http remote are correct. The README tells the minimum version and the error. The engines of package.json have bun >=1.4.0, next to node.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This branch makes rfetch a normal Node.js package and a stateless "rclone as a service" Worker.
Package and build
@rfetch/NAME. The shared code is the workspace@rfetch/lib.deno.json,deps.jsanddev_deps.jsare removed. Each module imports its own packages.@std/fmt,@std/iniand@std/media-types.node:cryptoreplacesaes-js.wrangler.tomlis the configuration of the Pages projectrfetch.Backends
s3backend: Signature Version 4, lists, ranges and multipart uploads. It is tested with R2.cryptgives the same results as rclone 1.75.1, in the two directions. XSalsa20-Poly1305 is in WebAssembly, and scrypt comes fromnode:crypto.Configuration
RFETCH_*is the primary prefix, andRCLONE_*also works.RFETCH_CONFIG_R2_TYPE: s3.Commands
lsjson,ls,lsl,lsdandlsfgive the same output as rclone, with--human-readable,--csvand the times of rclone.=, the same error messages and the same exit codes.../work, as in rclone.serve httpis read-only, as in rclone. It serves the path of the remote with all its options. A client cannot set an option of the remote, and TRACE is refused.Worker
/:s3:bucket/a.txtor/r2:bucket/a.txt.GETwithAccept: application/jsongives the list oflsjson. TheDepthheader sets how many levels.Breaking changes
node main.js,npm testandnpm start.Tests
deno fmt, and checks thatsecretbox.wasmbuilds to the same bytes.bench/measures the backends in Node.js and in workerd. The README has the times of the commands on each runtime and with rclone.