diff --git a/CHANGELOG.md b/CHANGELOG.md index e0caa48..1909a04 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,97 @@ [specification.md](specification.md) is the authority on behaviour. +## 0.1.7 (unreleased) + +The local API and the library for companion apps. No wire-format or CID changes: these nodes, 0.1.6, 0.1.5, 0.1.4 +and 0.1.3 nodes work together. The local API stays compatible: every route, field and status of 0.1.6 is +unchanged, and error bodies gain a field. The upgrade of an `` is automatic and one-way. + +Upgrade notes: +- **State:** the state database gains two tables (`pin_times`, `pin_labels`); an older rnfs can't open it + afterwards. Pins made before the upgrade have no `added` time. +- **`/daemon.json`:** the daemon writes it while it runs. Clients from 0.1.7 on read the API port from + it first, so one that can't read a newer daemon's `rnfs.toml` still finds it. +- **Settings:** `rnfs.toml` may now be rewritten by the daemon (`rnfs config set`, `POST /api/config/set`). Only the + assignments changed are replaced; comments and other lines are kept. Settings whose `api_port` equals a + non-zero `gateway_port` are now refused at start (such a daemon could never listen on both). + +Added, for apps: +- **Discovery:** `/daemon.json`, written atomically (0600) once the API listens and removed at stop: the + daemon's PID, API and gateway ports, version, start time, rnfs dir, Reticulum config directory, whether it + attached to a shared instance, and node ID. `rnfs.apiclient.read_daemon_info()` reads it, refusing a symlink, + a file others could have written, or one whose process has ended. +- **Stopping:** `POST /api/shutdown` stops the daemon as SIGTERM does, after answering; `rnfs stop [--wait S]` + uses it and waits until the API stops answering. Windows has no other clean way to stop a background daemon. +- **Error codes:** every API error is `{"error", "code"}`, the code machine-readable (`rnfs.errors.CODES`): the + node's errors by status (`bad_request`, `not_found`, `too_large`, `denied`, `invalid_dag`, `busy`, + `storage_full`, `internal`) and the API's own refusals (`no_such_endpoint`, `method_not_allowed`, + `unauthorized`, `bad_host`, `timeout`, `length_required`, `body_too_large`, `too_many_connections`, + `unsupported`). So a 404 for an endpoint a daemon lacks (`no_such_endpoint`) is told from one for missing + content (`not_found`). `ApiError.code`, and `NoSuchEndpoint` and `Unsupported` (both `NotFound`s) in the + library, which also recognises an older daemon's unknown-endpoint 404 by its message. +- **Capabilities:** `GET /api/capabilities` lists the daemon's routes, features, event kinds, error codes and + capabilities (`events`, `pin_details`, `cache`, `config`, `jobs_add`, `add_tar`, ...). +- **Events:** `GET /api/events?after=&kinds=&epoch=&wait=`, a long poll on the node's events (`rnfs/events.py`), + replacing an app's polling of each list: `job` (every state change, and a running job's progress at most once + a second), `pin` (added, removed, labelled), `cache` (added, removed), `inbox` (messages received, removed and + auto-fetched; outbox messages queued and their delivery state), `name` (a newer record of a name the node holds + or watches) and `peer` (a peer first heard, coming online, going offline). Events are numbered within a run; + the last 1024 are kept, and a client that falls behind, or asks of an earlier run (`epoch`), is told `lost` + and reloads. At most 8 polls wait at once. +- **Pins:** `pin ls --json` and `GET /api/pin/ls` give each pin's `label`, `origin` (where it was added from, + kept apart from the label), `size`, `entries` (a directory's entry count, as a number) and `added` (when first + pinned). `rnfs pin label [label]` / `POST /api/pin/label` give a pin a label, which becomes its listed + name. `name` is otherwise unchanged. +- **The cache:** `rnfs cache ls [--json]` / `GET /api/cache/ls` list the cached fetches (downloads not pinned), + most recently used first, with size, completeness and whether the same content is pinned or provided; + `rnfs cache rm …` / `POST /api/cache/rm` evict those leading to a CID now. Apps no longer need to look in + `blocks/`. +- **Settings:** `GET /api/config` gives every setting with its schema (type, choices, default, unit, range, + description, group, whether it applies without a restart), the running value, the value in `rnfs.toml`, and + whether the command line overrides it or a restart is pending; `POST /api/config/set` changes `rnfs.toml` + (validated as a start would, nothing written otherwise; `null` removes a setting), keeping its comments and + other lines, and applies at once what can be. `rnfs config [list|get|set|unset]`. `rnfs.config.settings_schema()` + needs no daemon. Settings that now apply without a restart: `max_storage` (enforced at once), + `max_download_size`, `max_output_entries`, `max_upload_rate`, `remote_pin_quota`, `follow_quota`, + `lxmf_auto_fetch_quota`, `max_jobs` (queued jobs start if the new limit allows), `max_finished_jobs`, + `serve_interfaces`, `no_serve_interfaces`, `quiet_hours`, `keep_max_names`, `keeper_query`, + `lxmf_accept_from`, `lxmf_auto_fetch_from`, `lxmf_send_from_hint` and `scan_interval`. +- **Add and import as jobs:** `POST /api/jobs` takes `kind` `add` (`path` and `POST /api/add`'s options) and + `import` (`path`), with progress (`total_bytes` to read, `done_bytes` read, `rate`, `eta`), cancellation at + the next read, and the result `POST /api/add` / `/api/import` would give; persisted, and run again from the + start if interrupted. `rnfs add --background`, `rnfs import --background`. +- **Streaming, so a daemon that can't see the app's files (Flatpak) works fully:** `POST /api/add/stream?format=tar` + (a tar stream becomes a directory; links, devices, absolute or `..` paths and invalid names are skipped with + warnings; bounded by `max_output_entries` and the DAG depth limit; extended headers over 64 KiB, global headers + over 4 KiB in all, data after the tar's end, and compressed or malformed tars refused), + `GET /api/get/stream?format=tar` (a target as a tar, fetched first, every limit checked before the response + starts, with an exact length; files as stored; entries whose names hold `\`, `＼` or `／`, or start as a Windows + drive does, `C:`, are skipped and counted), + `GET /api/export/stream` (an archive, byte-identical to `rnfs export`) and `POST /api/import/stream`. +- **Peers:** `GET /api/peers` and `rnfs peers` gain `first_heard`, `rtt` and `rtt_at` (the round trip RNS measured + on the last link to the peer, `ping` included) and `online`, worked out by the daemon: an rnfs link is open, or + within 15 minutes of its last sign of life — an answer, a request served, a link, or an announce emitted within + that time (a path answer, or an announce relayed late, doesn't count). Decided on the monotonic clock, so a + wall-clock step changes nothing. + `rnfs peers` gains RTT and ONLINE columns. +- **rnfs.Client and rnfs.Node:** typed methods for what was only reachable through `Client.api`: `peers()`, + `stats()`, `contacts.*`, `names.keep/unkeep/kept`, `names.follow/unfollow/following` and `lxmf.*` (address, + send, inbox, get, remove, outbox, sync), with objects (`Peer`, `Stats`, `Contact`, `Kept`, `Followed`, + `Message`, ...); `jobs.add` and `jobs.import_archive`; and the additions above (events, capabilities, pin + labels, the cache, settings, streams). A Node raises what a Client gets back. +- **Per-call timeouts:** `with client.timeout(seconds):` bounds the calling thread's requests only, long polls + fitting inside it (a Node accepts it and does nothing); `ApiConnection.request/call/get/post(timeout=)`. + +Changed: +- `rnfs.apiclient` (the CLI, `rnfs.Client`) refuses a daemon older than 0.1.5 before sending it the token, rather + than sending the request on a second connection as it did for them until now (the fallback planned to go in + 0.1.7): stop such a daemon and start the new one. Every daemon of 0.1.5 or later works as before. +- `POST /api/import` (and `rnfs import`) refuse anything but a regular file, without waiting on a FIFO. +- `Config.validate` refuses `api_port` equal to a non-zero `gateway_port`. +- A request that times out raises `ConnectionFailed` saying the daemon "didn't answer within its timeout". +- `Config.load` reports invalid TOML as " isn't valid TOML: …" (still a `ValueError`). + ## 0.1.6 Security fixes, maintainability work and performance work. No wire-format or CID changes: these nodes, 0.1.5, diff --git a/README.md b/README.md index 2d65a3e..be76141 100644 --- a/README.md +++ b/README.md @@ -440,6 +440,16 @@ lxmf_accept_from = [""] | `debug_log` | `false` | Write `~/.rnfs/debug.log` (readable by you only) for diagnosing problems (`daemon --debug-log`) | | `verbosity` | `0` | Extra logging: 1 per-query detail, 2 RNS debug output (`daemon -v`, `-vv`) | +**Changing settings while the daemon runs.** `rnfs config` lists every setting with the value the daemon runs +with (`*` set in `rnfs.toml`, `!` changed there but waiting for a restart, `^` set on the daemon's command line, +which wins); `rnfs config get ` describes one. `rnfs config set ` (a TOML value: `true`, +`10`, `"text"`, `["a", "b"]`; a bare word is text) and `rnfs config unset ` edit `rnfs.toml` in place, +keeping your comments, after checking the whole file as a start would. Some apply at once: `max_storage` (enforced +then), `max_download_size`, `max_jobs`, `quiet_hours`, the serve interfaces, the quotas, the LXMF sender lists +and a few more (`config get` says which); the rest at the next start. Apps do +the same through `GET /api/config` and `POST /api/config/set`, which carry each setting's type, range, unit, +default and description. + **Private mode.** `private = true` with a `members` list (identity hashes, from each node's `rnfs status`) makes the node link to, serve and keep records for members only. A name's announce still reaches everyone, so outsiders can see which CID it points at and when it changes (not fetch it), and @@ -458,7 +468,7 @@ Every command except `daemon` talks to the running daemon, so run them as the da | Command | Description | |---|---| | `rnfs daemon` | Run a node with its API and gateway (`--bootstrap`, `--rnsconfig`, `--private`, `--debug-log`, `-v`…) | -| `rnfs status` / `peers` / `stats` | Node ID and identity hash, storage, jobs, LXMF address / known nodes / traffic | +| `rnfs status` / `peers` / `stats` | Node ID and identity hash, storage, jobs, LXMF address / known nodes, their round-trip time and whether each is online / traffic | | `rnfs ping NODE_ID` / `providers CID` | Round-trip time to a node / nodes providing a CID | | `rnfs add PATH\|-` | Add a file, directory or stdin, pin it and print its CID (options below) | | `rnfs get TARGET [-o OUT] [--from NODE_ID] [--background]` | Fetch and write out a file or directory | @@ -525,6 +535,17 @@ with rnfs.Node("~/.myapp/rnfs") as node: # or: rnfs.Client(), for the d never stops your Reticulum. Never point it at the daemon's `~/.rnfs/`: a directory is used by one node at a time. - Callbacks run on one event thread; keep them short. +- Beyond content, pins, names and jobs: `peers()` and `stats()`, the address book (`contacts.add/get/list/remove`), + keeping and following names (`names.keep/unkeep/kept`, `names.follow/unfollow/following`), and LXMF + (`lxmf.address/send/inbox/get/remove/remove_all/outbox/sync`; off in an embedded node unless + `settings={"lxmf_enabled": True}`). +- To follow what changes instead of polling lists: `events.watch(callback, kinds=...)` (jobs, pins, the cache, the + inbox, names, peers; an event with `event.lost` means "read everything again"), or `events.since(...)`. Also + `capabilities()`, pin labels (`label_pin`), the cache (`cache()`, `uncache(cid)`), settings (`config()`, + `config_set({...})`), tars and archives as streams (`add_tar`, `get_tar`, `export_stream`, `import_stream`), and + for a `Client` `shutdown()` and `daemon_info()`. +- A `Client` waits as long as each request takes unless given `timeout=`; `with client.timeout(5): ...` bounds + just the calls your thread makes inside it (a `Node` accepts it too, and does nothing). - `examples/rnfs_share.py` is a runnable example using either class; design.md's "Public API" section is the reference. Other languages can use the local API's JSON endpoints (listed in `rnfs/api.py`). diff --git a/design.md b/design.md index 1a0b6a7..370fb65 100644 --- a/design.md +++ b/design.md @@ -114,6 +114,7 @@ rnfs/ blockstore.py one file per CID, atomic, verified (section 8) storage.py Storage: blocks + pins (sources) + cached fetches + protection + eviction/GC (section 8) archive.py export/import stream; archive_size for an exact Content-Length (section 8) + tarstream.py directories as tar streams: build_tar (an upload), plan/length/chunks (a download) shared_folder.py importing /files/ with pin source "folder" (section 8) dht/ no RNS dependency routing.py RoutingTable: buckets, seniority, failures, unreliable nodes, bans (section 6) @@ -382,13 +383,15 @@ check goes. `verify_daemon` checks `X-Rnfs-Proof` before anything else of the an `503` is `Busy` and every other status `AuthFailed`, both in the client's own words, so a program on the port chooses neither the error nor its text; it reads at most `PROOF_READ_LIMIT` (64 KiB), and connecting and proving take at most `PROOF_TIMEOUT` (15 s, or the client's timeout if shorter), after which the request gets the client's -own timeout. A daemon that closes the proved connection gets the request on a new one only if the version it proved -is below 0.1.5 (`KEEPS_PROOF_OPEN_SINCE`; such daemons close every connection), else `ConnectionFailed`: a new -connection could reach another listener. That fallback goes in 0.1.7. `request()` maps failures to `DaemonNotRunning`, `ConnectionFailed` (`_exchange`), `AuthFailed` (a +own timeout. A daemon that closes the proved connection gets nothing more (`ConnectionFailed`): a new connection could reach +another listener. A daemon proved older than 0.1.5 (`OLDEST_DAEMON`; such daemons close every connection) is +refused by `check_daemon` before the token is sent; until 0.1.7 the client sent it the request on a new connection. `request()` maps failures to `DaemonNotRunning`, `ConnectionFailed` (`_exchange`), `AuthFailed` (a failed proof, or 403) and the API's statuses (`_raise_for`: `STATUS_ERRORS`, `error_for`, the one mapping); `Stream` reads a body, raising `ConnectionFailed` if it ends short. Its methods (`status`, `add_path`, `add_stream`, `fetch`, `cat`, `info`, -`ls`, `stat`, `dir_set`, `dir_rm`, `export`, `import_archive`, `gc`, `verify`, `pin_*` (with `pin_remote`), +`ls`, `stat`, `dir_set`, `dir_rm`, `export`, `import_archive`, `add_tar`, `get_tar`, `export_stream`, +`import_stream` (uploads bytes or a binary file, chunked when their length is unknown; downloads a `Stream`), `gc`, +`verify`, `pin_*` (with `pin_remote`), `providers`, `ping`, `name_*` (with `name_watch`, `name_import`, `name_export`, `name_rm`), `submit_job`, `job`, `jobs`, `cancel_job`, `clear_jobs`, `deny*`) cover every core route (`ApiHandler.ROUTES`; checked by test_apiclient), so `remote.py` writes no path itself, and send only the parameters given; `ApiError` is an `RnfsError`, and each status's error a subclass of the @@ -402,13 +405,15 @@ subclasses it, turning each `ApiError` into `fail(str(e))`, so its messages are |---|---| | `add_path(path, wrap=False, only_hash=False, pin=True, to=None, to_self=False, seal_name=True, chunker=None)` | `{"cid", "size", "warnings": [str]}` (+ `"sealed"`); file or directory, pinned `user` (or cached with `pin=False`) | | `add_stream(stream, size=None, name=None, only_hash=False, …)` | as above; `name` wraps it as that entry (sealed: the sealed name) | +| `add_tar(stream, name=None, only_hash=False, pin=True, chunker=None)` | as above (+ `"warnings_total"` past 100): the directory a tar stream holds (`POST /api/add/stream?format=tar`), built by `tarstream.build_tar` as it is read; at most `max_output_entries` files and directories; recorded in pin_names as `name` or `(tar upload)` | | `get(target, output, from_node=None, raw=False)` | `{"cid", "warnings"}` (+ `"warnings_total"` past 100; + `"sealed", "name"`, the name only if `safe_default_name` accepts it); fetches and writes the file or directory under `.rnfs-get-.partial` beside `output`, then renames it to `output` (`rename_new`), decrypting a sealed file unless `raw`; a directory output is first checked to still be the one `write_out` made (`walker.check_unchanged`) and, on Windows, made to inherit its folder's access list (`winacl.inherit_from_parent`; if that fails it stays private, with a warning logged) | | `info(target, for_gateway=False, from_node=None, raw=False)` | `{"cid", "type": "file"\|"directory", "size", "executable"}` (+ `"sealed", "recipient", "stored_size", "name"`; the sealed name is None unless `safe_default_name` accepts it) | | `stored_head(target, length)` | a file's first `length` bytes if stored, else None; never fetches (the user's own gateway `HEAD` and ranges not at 0); `Denied` as a read: a denied CID met, or while there are file hashes a file that fails `_check_file` (one not wholly stored gives None) | | `read(target, offset=0, length=None, for_gateway=False, from_node=None, raw=False)` | iterator of bytes of a file (range fetched only; sealed files decrypted unless `raw` or `for_gateway`) | | `read_file(target, offset=0, length=None, from_node=None, raw=False)` | `(info, chunks)`: `info()` and `read()` in one fetch, resolving the target once; `chunks` reads `[offset, offset + length)` cut to the info's size only once iterated (a directory's raises `ValueError`), and closing it ends the fetch (`GET /api/cat`, which used to call `info` and `read`: two fetches) | | `block(cid, for_gateway=False)` | the CID's own block, verified (`cat --block`, `?format=raw`: the gateway calls it only for the user's own GET) | -| `export_stream(target, start, for_gateway=False)` | fetches the DAG, then writes its archive to `start(exact length)` (`?format=archive`; a `start` returning `None` writes nothing). The gateway calls it only for the user's own GET (never for a limited request or a HEAD, stored or not), inside `fetcher.observing(ClientWatch)`, which cancels the fetch once the client disconnects | +| `export_stream(target, start, for_gateway=False)` | fetches the DAG, then writes its archive to `start(exact length)` (`?format=archive`; a `start` returning `None` writes nothing). The gateway calls it only for the user's own GET (never for a limited request or a HEAD, stored or not), inside `fetcher.observing(ClientWatch)`, which cancels the fetch once the client disconnects. Built on `export_chunks` | +| `export_chunks(target, for_gateway=False)`, `get_tar(target)` | `(info, chunks)`, as `read_file`: the DAG fetched and checked first (`node_content._fetched_dag`: `max_download_size`, `max_output_entries`, the deny list's file hashes), then `info` (`{"roots", "blocks", "length"}` / `{"cid", "length", "skipped"}`) and an iterator of the archive's or tar's bytes, exactly `length` of them; closing it ends the fetch (`GET /api/export/stream`, `GET /api/get/stream?format=tar`, which run the fetch inside `observing(ClientWatch)` as the gateway does) | | `ls(target, for_gateway=False)` | `[{"name", "kind": "file"\|"executable"\|"directory", "size", "cid"}]` | | `stat(target)` | `{"cid", "size", "blocks", "local_blocks", "local_bytes"}` | | `is_local(target)` | `bool`: everything needed is stored, so no network fetch | @@ -425,7 +430,8 @@ subclasses it, turning each `ApiError` into `fail(str(e))`, so its messages are | `name_list()` | `[{"alias", "name", "cid" or None, "sequence"}]` | | `name_publish(alias, target)` | `{"name", "cid", "sequence"}`, plus `"warning"` in private mode (the name's announce reaches outsiders: `names.PRIVATE_NAME_WARNING`) | | `name_resolve(name, for_gateway=False)` | `{"name", "cid", "sequence", "age"}`; NotFound if unresolved | -| `export(target, output)` / `import_archive(path)` | `{"roots", "blocks"}` / `{"roots", "incomplete", "blocks"}` | +| `export(target, output)` / `import_archive(path, progress=None)` | `{"roots", "blocks"}` / `{"roots", "incomplete", "blocks"}`; `import_archive` opens `path` with `O_NONBLOCK` and refuses (ValueError) anything but a regular file, so a FIFO can't hold it (or an import job) | +| `import_stream(stream, label="(archive upload)", progress=None)` | as `import_archive`, of an archive read from `stream` to its end (`POST /api/import/stream`; `import_archive` opens the file and calls it). `progress` here and in `add_path` is told each read's length (`builder.BuildOptions.progress`, `builder.counted`) and may raise to stop: an add or import job's `Job.read` | | `status()` | (API: without the token, only `{"version"}`) `{"version", "node_id", "identity_hash", "dht_mode", "slow", "private", "known_nodes", "records", "pins", "provided", "blocks", "used_bytes", "max_storage", "names"}` plus every feature's `status()` fields (traffic, jobs, LXMF); `identity_hash` is the RNS identity hash (None on offline test nodes) | `RnfsNode.offline(config, node_id, client, names_network)` builds a node without Reticulum or background @@ -458,7 +464,9 @@ as a request body. `/api/name/watch` (a long poll for newer records: Public API, API additions); `POST /api/add`, `/api/add/stream`, `/api/get`, `/api/dir/set`, `/api/dir/rm`, `/api/pin/add`, `/api/pin/rm`, `/api/pin/remote`, `/api/gc`, `/api/verify`, `/api/name/create`, `/api/name/rm`, `/api/name/export`, -`/api/name/import`, `/api/name/publish`, `/api/export`, `/api/import`. `GET /api/info` and `POST /api/get` take +`/api/name/import`, `/api/name/publish`, `/api/export`, `/api/import`; for clients that share no files with the +daemon (a sandboxed app), `GET /api/get/stream?format=tar`, `GET /api/export/stream`, `POST /api/import/stream`, +and `POST /api/add/stream?format=tar`. `GET /api/info` and `POST /api/get` take an optional `from` (node ID), as does `GET /api/cat`; `GET /api/name/resolve?name=` takes a name or a name contact's nick. @@ -514,6 +522,56 @@ files as subresources but can't read any gateway response with `fetch()`; an ear would let any site's sandboxed frame read the gateway. A separate origin per root (a subdomain per CID) is the possible future way to let pages read their own DAG. +## Local API for apps (0.1.7) + +What a companion app needs from the daemon beyond the facade calls (specification section 11, "For apps"). + +- **Discovery.** `cli.run_daemon` writes `/daemon.json` (`apiclient.DAEMON_FILE`; `cli.write_daemon_info`, + `fsutil.write_atomic`, 0600, private) once its servers listen and its token is written, and removes it when it + stops if it still names its PID (`remove_daemon_info`). `apiclient.read_daemon_info` reads it without following a + symlink, refuses one other users could have written, one over 64 KiB, one without an integer `pid` and a valid + `api_port`, and (POSIX) one whose process has ended; `api_port()` and the CLI's `client()` use its port before + `rnfs.toml`'s, so an older client finds a daemon whose settings it can't parse. Only the port is used: the proof + check is unchanged, so a stale or planted file at worst points at a port where nothing proves itself. +- **Error codes.** `rnfs.errors.STATUS_CODES` (the node's errors, by status), `API_CODES` (the API's own + refusals: `bad_host`, `unauthorized`, `no_such_endpoint`, `method_not_allowed`, `timeout`, `length_required`, + `body_too_large`, `unsupported`, `too_many_connections`) and `error_code(error, status)`, which prefers an + exception's own `code` (`errors.Unsupported`: a 404 `unsupported`, for something this daemon can't do). + `ApiHandler.respond_error` sends `{"error", "code"}`; `ApiRequestError` carries its code. The client + (`apiclient.error_for`) keeps the code on every `ApiError` (`.code`) and raises `NoSuchEndpoint` for + `no_such_endpoint` (or, from a daemon before 0.1.7, a 404 whose message starts "no such API endpoint") and + `Unsupported` for `unsupported`, both `NotFound` subclasses, so existing `except NotFound` clauses still hold. + `http.server`'s own error pages (a malformed request line) have no code. +- **Capabilities.** `GET /api/capabilities`: `api.CAPABILITIES` (what a route list can't say: `events`, + `pin_details`, ...), every core and feature route, the loaded features, `events.EVENT_KINDS` and + `errors.CODES`. `ApiConnection.capabilities()` answers `{}` from an older daemon. +- **Events.** `rnfs/events.py`: `EventLog`, one per node (`RnfsNode.events`, made first in `_setup`), holding the + last `MAX_EVENTS` (1024) events, each `{"seq", "kind", "time", ...}` numbered from 1 per run, with a random + `epoch` per run. `publish()` appends under the log's condition (the innermost lock: callers hold the names lock + or storage's) and tells listeners (an embedded Node's event thread) without it; `events.publish(node, ...)` is + the never-raising form for code that may run on a stand-in node. `since(after, kinds, wait, epoch)` is + level-triggered: `after` absent answers `next` at once; events after `after`, or after `wait` none; `lost` (and + every event held) when `after` is no longer held, is beyond the log (a restart) or `epoch` differs. Kinds and + their fields: `pin` (`change`: added, removed or label; `source`; `cids`, at most `MAX_EVENT_CIDS` (100), and + `count`: storage's `pin_listener`, after the pins are saved; a label from `pin_label`), `cache` (`change`: added + or removed, `cids`, `count`: storage's `cache_listener`, from `_set_record` and `_drop_records`), `name` + (`name`, `cid`, `sequence`: `NameManager.held_listener`, for a newer record accepted for a name held — this + node's own, kept, followed or resolved — or watched), and `job`, `inbox` and `peer` from their features. + `GET /api/events` waits at most 30 s, `MAX_EVENT_WAITERS` (8) calls at once (`ApiHandler.event_waiters`, a + semaphore per server; 503 beyond). +- **Shutdown.** `ApiServer(shutdown=)`: the daemon passes its `stopping.set`, the event its signal handlers set, + so `POST /api/shutdown` answers `{"stopping": true}`, flushes, then stops the daemon exactly as SIGTERM does + (the servers stop, `daemon.json` and the token are removed, the node saves and stops). Without one (tests) it is + `unsupported`. `ApiConnection.shutdown(wait=)` polls the open status until the port stops answering; the CLI's + `rnfs stop` uses it. +- **Pins and the cache.** `pin_ls` adds `label`, `origin`, `size`, `entries` and `added` (`_PinDetails`, read + from stored blocks and remembered for complete pins as before; a large directory's entry count walks its stored + index and group nodes, None while one is missing); `name` is unchanged but for a label coming first. + `pin_label` (`POST /api/pin/label`) stores the label in `PinNames` (`pin_labels`). `cache_ls` lists + `storage.cached_fetches()` newest first; `cache_rm` is `Storage.uncache(cids)`, which drops the records leading + to them and collects as `gc` does (warming the DAGs that stay without the lock), then the publisher withdraws + what is no longer complete. + ## Public API (rnfs.Node, rnfs.Client) **Provisional until 0.2.** Names, signatures and objects may change after use in a real app. Said in the @@ -547,7 +605,8 @@ with rnfs.Node("~/.myapp/rnfs") as node: # or: rnfs.Client() for the u rnfs/__init__.py __version__, API_STATUS; every public name loaded on first use (PEP 562 __getattr__, __all__, __dir__), so `import rnfs` imports nothing more and the CLI's start-up is unchanged rnfs/objects.py the typed objects (Cid re-exported from rnfs.content.cid); no RNS -rnfs/interface.py Rnfs, Names, Name, Jobs, Job, Deny, Watch, File, the event thread: shared by both; no RNS +rnfs/interface.py Rnfs, Names, Name, Jobs, Job, Deny, Contacts, Lxmf, Fetched, Events, Watch, File, the event + thread: shared by both; no RNS rnfs/embedded.py Node: imports rnfs.node, so RNS, only when rnfs.Node is first used rnfs/remote.py Client, over rnfs.apiclient; no RNS rnfs/errors.py gains Cancelled (moved from fetcher, which imports it from there) and Closed @@ -556,10 +615,14 @@ rnfs/apiclient.py the API protocol layer (token, proof, typed errors, Stream); `Rnfs` implements every method once, `open` included, over a *backend* offering the facade's operations in the local API's argument and result forms (JSON-ready dicts; CIDs written out, node IDs and names as hex): for -`Node`, `embedded._Backend`, an adapter over the `RnfsNode`; for `Client`, `remote._Backend(api, long_wait)`, over +`Node`, `embedded._Backend`, an adapter over the `RnfsNode`; for `Client`, `remote._Backend(api)`, over an `ApiConnection` whose methods already mirror the facade (`add_path`, `ls`, `info`, `pin_add`, …). Both turn the same dicts into the same objects (`Entry.from_dict`), so they can't drift apart; only lifecycle, how name records -are heard, and the escape hatches differ per class. `rnfs.Client`, its objects +are heard, timeouts and the escape hatches differ per class. A feature's methods (peers, contacts, keep, follow, +LXMF) go, in a Node, through the feature as its API route calls it: the route's own lookup (each feature module's +`_feature(handler)`, or `_common.feature`, given the backend, which has the handler's `node` attribute), so a +feature that isn't loaded is the route's `NotFound`, with its message, then the route's checks of its arguments +(`check_kind`, `parse_name_arg`, lxmf's `_int`), in its order. `rnfs.Client`, its objects and errors import neither RNS nor anything that does (tested, as the CLI's start-up is). The CLI keeps using `apiclient` directly (its `ApiClient` subclasses `ApiConnection`), so its output can't change. @@ -597,6 +660,20 @@ and errors import neither RNS nor anything that does (tested, as the CLI's start Nothing is sent until the first call. Raises `SettingsError` if `rnfs.toml` can't be read or trusted; one made private on the way is reported as a `RuntimeWarning` (`warnings.warn`), without loading RNS. +**Timeouts.** A Client's `timeout` bounds each request (none by default: fetches can take hours). Rather than a +`timeout=` on every method of the shared interface, `with client.timeout(seconds):` (`Rnfs.timeout`, None: no +bound) replaces it for the requests the calling thread makes inside the block, nesting and restoring the outer one +on exit. It is thread-local (`ApiConnection.using_timeout`, a `threading.local` per connection), so other threads, +and the Client's own event, job-watch and name-watch threads, keep the Client's. A request that outlasts it raises +`ConnectionFailed`; connecting and proving take at most the smaller of it and `PROOF_TIMEOUT` (15 s). The long +polls on the calling thread fit it as they fit the Client's: `Job.wait` asks `GET /api/jobs/get` to wait at most +`remote.long_poll_wait(timeout)` (`LONG_POLL_MARGIN`, 2 s, less, whole seconds; with none left, it polls every +second), read from `ApiConnection.effective_timeout()` when each request is sent; the name poller, on its own +thread, keeps the Client's. A Node makes no requests: its `timeout()` checks `seconds` (None or a positive +number, else `ValueError`, as a Client's) and does nothing, so code using it runs on either class. Underneath, +`ApiConnection.request`, `call`, `get` and `post` take `timeout=` for one call (`apiclient.DEFAULT`: the thread's, +else the connection's; None: none, even over the connection's). + Both: `close()` (idempotent; both end their event and job-watch threads, then a Node stops its features and threads, saves, closes the state database, releases its rnfs.dht destination and the lock; a Client's name poller ends with the long poll in flight, within 30 s), `closed`, `with … as node:` (closes on @@ -621,14 +698,34 @@ Every object is a frozen dataclass with `to_dict()` (the API's JSON form, CIDs a | `Entry` | `name`, `kind` ("file", "executable", "directory"), `size`, `cid` | the name | | `Info` | `cid`, `type` ("file", "directory"), `size`, `executable`, `sealed`, `recipient`, `stored_size`, `name` | — | | `Stat` | `cid`, `size`, `blocks`, `local_blocks`, `local_bytes` | — | -| `Pin` | `cid`, `sources: tuple[str]`, `complete`, `type`, `sealed`, `name` | the CID | +| `Pin` | `cid`, `sources: tuple[str]`, `complete`, `type`, `sealed`, `name`, and (0.1.7; None from an older daemon) `label`, `origin`, `size`, `entries`, `added` | the CID | +| `Cached` / `Uncached` | `cid`, `root`, `path: tuple[str]`, `pinned`, `complete`, `type`, `size`, `provided` / `removed_records`, `removed_blocks`, `freed_bytes` | the CID / — | +| `Event` | `seq`, `kind` (`EVENT_KINDS`, or "lost": `lost` True), `time`, `fields: dict` (the kind's own; merged back by `to_dict`) | — | +| `EventBatch` | `events: tuple[Event]`, `next`, `lost`, `epoch` | — | +| `Capabilities` | `version`, `capabilities: frozenset[str]` (`has(name)`), `routes: tuple[(method, path)]`, `features`, `event_kinds`, `error_codes`; empty from a daemon before 0.1.7 | — | +| `Setting` | `rnfs.config.settings_schema()`'s `name`, `group`, `type`, `choices`, `nullable`, `default`, `unit`, `minimum`, `maximum`, `exclusive_minimum`, `description`, `live`, and `value`, `file`, `overridden`, `pending` (lists as tuples) | the name | +| `Settings` / `SettingsChange` | `path`, `settings: tuple[Setting]` (`settings["max_jobs"]`, iteration), `file_error`, `warnings` / `path`, `applied`, `restart_required`, `overridden`, `warnings` | — | | `Record` | `name: bytes`, `cid`, `sequence`, `age` (seconds since heard, or None) | — | | `Status` | `version`, `node_id`, `identity_hash`, `dht_mode`, `slow`, `private`, `known_nodes`, `records`, `pins`, `provided`, `blocks`, `used_bytes`, `max_storage`, `names`, `features: dict` (every other field) | — | | `Collected` / `Verified` | `removed_blocks`, `freed_bytes` / `checked`, `bad: tuple[Cid]` | — | | `Exported` / `Imported` | `roots: tuple[Cid]`, `blocks` / and `incomplete` | — | | `DenyState` | `entries: tuple[str]`, `lists: tuple[dict]` | — | - -`Name`, `Job`, `Watch` and `File` are bound to the `Node` or `Client` that made them (below). +| `Peer` | `node_id`, `mode`, `in_table`, `seniority`, `senior`, `hops`, `interface`, `bitrate`, `link`, `rate`, `last_heard`, `last_active`, `banned`, `sent`, `received`, `requests`, `answered`, `refused`, `failed`, `bad`, and (None from a daemon before 0.1.7) `first_heard`, `rtt`, `rtt_at`, `online` | node ID hex | +| `Stats` | `since`, `totals: dict`, `refused: dict`, `interfaces: dict`, `upload_budget`, `upload_budget_left` | — | +| `Contact` | `nick`, `kind` ("name", "node", "lxmf"), `id: bytes` | the nick | +| `Kept` | `name: bytes`, `reasons: tuple[str]` ("user", "follow", "own"), `sequence`, `cid`, `emitted` (the record held, or None) | name hex | +| `Followed` | `name: bytes`, `pinned`, `latest` (CIDs or None), `sequence`, `state` | name hex | +| `LxmfAddress` | `address: bytes`, `display_name`, `propagation_node`, `nearest_propagation_node` | address hex | +| `MessageLink` | `uri`, `cid`, `name`, `path: tuple[str]`, `size`, `from_node` (JSON `"from"`) | the URI | +| `Message` (inbox) | `id`, `hash`, `sender` (JSON `"from"`), `display_name`, `sent`, `received`, `title`, `text`, `method`, `links: tuple[MessageLink]`, `auto: tuple[dict] \| None`, `trusted`, `contact` | — | +| `Outgoing` (outbox) | `id`, `hash`, `to`, `time`, `links`, `text`, `method`, `state`, `progress`, `error`, `direct_hash`, `fallback` | — | +| `Sent` / `Synced` | `id`, `hash`, `to`, `method` / `propagation_node`, `started` | — | + +A JSON key that is a Python keyword is a field of another name (`_field(key="from")`); `from_dict` ignores keys +it doesn't know, so a newer daemon's additions don't break an older library. + +`Name`, `Job`, `Fetched` (`lxmf.get`'s per link: `target`, `output`, `job: Job | None`, `cid`, `warnings`, +`error`), `Watch` and `File` are bound to the `Node` or `Client` that made them (below). ### Methods @@ -656,6 +753,15 @@ Client the connection errors below). "Fetch errors" = `NotFound` (no such conten | `pin_remote(node_id, cid, pin=True) -> bool` | | `pin_remote` / `POST /api/pin/remote` | | `providers(cid) -> list[bytes]`, `ping(node_id) -> float \| None` | | `providers`, `ping` / `GET /api/providers`, `/api/ping` | | `export(target, output) -> Exported`, `import_archive(path) -> Imported` | fetch errors, `ValueError`, `OSError` | `export`, `import_archive` / `POST /api/export`, `/api/import` | +| `add_tar(source, *, name=None, pin=True, only_hash=False, chunker=None) -> Added` (bytes or a binary file, read up to the tar's end; `Added.warnings_total`) | `ValueError` (a malformed tar, a chunker), `Denied`, `TooLarge`, `StorageFull` | `add_tar` / `POST /api/add/stream?format=tar` | +| `get_tar(target) -> Iterator[bytes]`, `export_stream(target) -> Iterator[bytes]` (sent once iterated, as `cat`; the whole DAG fetched first; `close()` stops early) | fetch errors, `ValueError`, possibly once iterated | `get_tar`, `export_chunks` (the chunks, closed when done) / `GET /api/get/stream?format=tar`, `/api/export/stream` | +| `import_stream(source) -> Imported` (bytes or a binary file, read to its end) | `InvalidDag`, `Denied`, `StorageFull`, `ValueError` | `import_stream` / `POST /api/import/stream` | +| `label_pin(cid, label) -> Pin` (None or "" clears it) | `NotFound` (not pinned), `ValueError` | `pin_label` / `POST /api/pin/label` | +| `cache() -> list[Cached]`, `uncache(cid) -> Uncached` | `uncache`: `NotFound` (no cached fetch leads to it) | `cache_ls`, `cache_rm` / `GET /api/cache/ls`, `POST /api/cache/rm` | +| `capabilities() -> Capabilities` | | `api.capabilities(node, without=embedded.NODE_LACKS)` (a Node: not "shutdown" or "daemon_json") / `GET /api/capabilities` (`{}` from an older daemon) | +| `shutdown(wait=None) -> bool`, `daemon_info() -> dict \| None` | `shutdown`: `Unsupported` (a Node: `close()` stops it; a daemon without a way to stop it), `NoSuchEndpoint` (an older daemon) | a Node: refused / None; a Client: `ApiConnection.shutdown`, `daemon_info` (`daemon.json`, read locally) | +| `config() -> Settings`, `config_set(settings) -> SettingsChange` | `ValueError` (unknown, refused or malformed settings), `Busy` (rnfs.toml changed meanwhile), `NotFound` (no settings feature) | `SettingsFeature.describe`, `set` (a Node: through JSON, as the API carries them) / `GET /api/config`, `POST /api/config/set` | +| `events.watch(callback, kinds=None) -> Watch`, `events.since(after=None, kinds=None, wait=0, epoch=None) -> EventBatch` (epoch "" is None; a Client's wait fits its timeout, pausing up to a second on an empty answer when that leaves none) | `ValueError` (an unknown kind, after or wait not a whole number ≥ 0); a Client: `NoSuchEndpoint` (an older daemon), `since`: `Busy` (8 polls waiting) | `EventLog.add_listener`, `since` / `GET /api/events` (below) | | `gc() -> Collected`, `verify() -> Verified` | | `gc`, `verify` / `POST /api/gc`, `/api/verify` | | `names.create(alias) -> Name`, `names.import_key(alias, key) -> Name` | `ValueError` (alias invalid or taken, key malformed) | `name_create`, `name_import` / `POST /api/name/create`, `/api/name/import` | | `names.get(alias) -> Name`, `names.list() -> list[Name]` | `NotFound` | `name_list` / `GET /api/name/list` | @@ -663,10 +769,17 @@ Client the connection errors below). "Fetch errors" = `NotFound` (no such conten | `names.resolve(name) -> Record` (bytes, hex, `Name` or a contact's nick) | `NotFound` (unresolved), `ValueError` | `name_resolve` / `GET /api/name/resolve` | | `names.watch(name, callback) -> Watch` | `ValueError`; a Client: `NotFound` from a daemon without the endpoint | below / `GET /api/name/watch` | | `Name`: `alias`, `name: bytes`, `cid`, `sequence`, `str()` hex; `publish(target) -> Record`, `export_key() -> bytes`, `remove()`, `refresh() -> Name` | `publish`: `NotFound` (path not stored), `ValueError`, `RnfsError` (highest sequence) | `name_publish`, `name_export`, `name_rm` / `POST /api/name/publish`, `/api/name/export`, `/api/name/rm` | -| `jobs.get(target, output, *, from_node=None, sealed_name=False, fallback_dht=False, on_progress=None) -> Job`, `jobs.pin(target, *, on_progress=None) -> Job`, `jobs.export(target, output, *, on_progress=None) -> Job` | `ValueError` (output exists or in use), `Busy` (1000 active) | `JobsFeature.submit` / `POST /api/jobs` | +| `jobs.get(target, output, *, from_node=None, sealed_name=False, fallback_dht=False, on_progress=None) -> Job`, `jobs.pin(target, *, on_progress=None) -> Job`, `jobs.export(target, output, *, on_progress=None) -> Job`, `jobs.add(path, *, wrap=False, pin=True, only_hash=False, to=None, to_self=False, seal_name=True, chunker=None, on_progress=None) -> Job`, `jobs.import_archive(path, *, on_progress=None) -> Job` | `ValueError` (output exists or in use, an option), `NotFound` (add, import: no such path), `Busy` (1000 active) | `JobsFeature.submit` / `POST /api/jobs` | | `jobs.job(job_id) -> Job`, `jobs.list() -> list[Job]`, `jobs.clear(job_id=None) -> int` | `NotFound`, `ValueError` (not finished) | `JobsFeature.get`, `list`, `clear` / `GET /api/jobs/get`, `GET /api/jobs`, `POST /api/jobs/clear` | -| `Job`: `id`, `kind`, `target`, `output`, `from_node`, `sealed_name`, `fallback_dht`, `state`, `error`, `result: dict`, `total_bytes`, `done_bytes`, `downloaded_bytes`, `blocks`, `blocks_total`, `rate`, `eta`, `created`, `started`, `finished`, `runs`, `done` (finished); `refresh() -> Job`, `wait(timeout=None) -> Job`, `cancel() -> Job`, `watch(callback) -> Watch` | `wait`: `TimeoutError`; `NotFound` once cleared | `JobsFeature.get(wait=)`, `cancel` / `GET /api/jobs/get?wait=`, `POST /api/jobs/cancel` | +| `Job`: `id`, `kind`, `target` (None for add and import), `output`, `from_node`, `sealed_name`, `fallback_dht`, `path`, `options` (add and import), `state`, `error`, `result: dict`, `total_bytes`, `done_bytes`, `downloaded_bytes`, `blocks`, `blocks_total`, `rate`, `eta`, `created`, `started`, `finished`, `runs`, `done` (finished); `refresh() -> Job`, `wait(timeout=None) -> Job`, `cancel() -> Job`, `watch(callback) -> Watch` | `wait`: `TimeoutError`; `NotFound` once cleared | `JobsFeature.get(wait=)`, `cancel` / `GET /api/jobs/get?wait=`, `POST /api/jobs/cancel` | | `deny.list() -> DenyState`, `deny.add(entries) -> list[str]`, `deny.remove(entries) -> list[str]` | `ValueError` (malformed entry, too many), `NotFound` (remove: not on it) | `node.deny.entries`/`add`/`remove`, `DenyFeature.lists` / `GET /api/deny`, `POST /api/deny/add`, `/api/deny/rm` | +| `timeout(seconds) -> context manager` | `ValueError` (not None or a positive number) | a Client: `ApiConnection.using_timeout`; a Node: nothing (above) | +| `peers() -> list[Peer]`, `stats() -> Stats` | `NotFound` (no peers feature) | `PeersFeature.peers`, `stats` / `GET /api/peers`, `/api/stats` | +| `contacts.list() -> list[Contact]`, `contacts.get(nick) -> Contact`, `contacts.add(nick, id, kind=None) -> Contact` (id: bytes, hex or a `Name`; kind None: the node tells), `contacts.remove(nick)` | `ValueError` (nick, id or kind invalid, a kind it can't tell, full), `NotFound` (no such contact; no contacts feature) | `ContactsFeature.list`, `get`, `add`, `remove` / `GET /api/contacts`, `/api/contacts/get`, `POST /api/contacts/add`, `/api/contacts/rm` | +| `names.keep(name) -> Kept`, `names.unkeep(name)`, `names.kept() -> list[Kept]` (name: bytes, hex, `Name` or a name contact's nick) | `ValueError` (malformed, unknown nick, `keep_max_names`), `NotFound` (unkeep: not kept by the user; no keepers feature) | `KeepersFeature.keep`, `unkeep`, `kept` / `POST /api/name/keep`, `/api/name/unkeep`, `GET /api/name/kept` | +| `names.follow(name) -> Followed`, `names.unfollow(name)`, `names.following() -> list[Followed]` | as keep; `NotFound` (unfollow: not followed; no follow feature) | `FollowFeature.follow`, `unfollow`, `following` / `POST /api/name/follow`, `/api/name/unfollow`, `GET /api/name/following` | +| `lxmf.address() -> LxmfAddress`, `lxmf.send(to, target, message=None, *, propagate=False, nearest=False) -> Sent` (to: bytes, hex or an LXMF contact's nick; target a Cid, path or `rnfs:` URI, a name URI sent as a name link), `lxmf.sync(nearest=False) -> Synced` | `ValueError` (LXMF off, as by default in a Node; recipient or target invalid; no propagation node), `NotFound` (send: recipient never heard; no lxmf feature) | `LxmfFeature.address`, `send`, `sync` / `GET /api/lxmf/address`, `POST /api/lxmf/send`, `/api/lxmf/sync` | +| `lxmf.inbox() -> list[Message]`, `lxmf.outbox() -> list[Outgoing]`, `lxmf.get(id, *, link=None, output=None, directory=None, use_sent_cid=False) -> list[Fetched]`, `lxmf.remove(id) -> int`, `lxmf.remove_all() -> int` | get: `NotFound` (no such message; a name unresolved without `use_sent_cid`), `ValueError` (id or link not a positive integer, no or a clashing output); remove: `NotFound` | `LxmfFeature.inbox_list`, `outbox_list`, `inbox_get`, `inbox_rm` / `GET /api/lxmf/inbox`, `/api/lxmf/outbox`, `POST /api/lxmf/inbox/get`, `/api/lxmf/inbox/rm` | `File` (from `open`, an `io.BufferedIOBase`): `read`, `read1`, `readinto`, `readline`, iteration, `seek`, `tell`, `seekable()`/`readable()` True, `size`, `cid` (the file's own CID), `info`, `close()`, context manager; @@ -688,8 +801,7 @@ before submitting, as the CLI resolves a nick. Jobs are persisted, resumed at th `rnfs jobs` (a Client's) or only by the app (a Node's, in its own rnfs dir). **Escape hatches, unsupported:** `Node.internal` (the `RnfsNode`) and `Client.api` (the `ApiConnection`, whose -`get(path, **params)` and `post(path, **fields)` reach every endpoint: contacts, peers, stats, keep, follow, -LXMF). Not in the typed interface for 0.2: keep/follow, contacts, peers/stats, LXMF. +`get(path, **params)` and `post(path, **fields)` reach every endpoint, an app's own feature's included). **Errors.** One hierarchy in `rnfs.errors`, every class importable as `rnfs.`: `RnfsError` ⊃ `NotFound`, `TooLarge`, `InvalidDag`, `StorageFull`, `Busy`, `Denied`, `Cancelled`, `Closed`, `ApiError`, and @@ -707,12 +819,22 @@ ValueError)`. So `except rnfs.NotFound` and `except ValueError` catch both class |---|---|---|---| | `names.watch(name, callback)` | `Record`, each time a record of `name` newer than the last delivered (or than the one held when `watch` was called) is accepted under the sequence rules (section 9): announces, path responses, keepers' answers, this node's own publishes | `NameManager.watch(name)` (its announces count as asked in `_heard`, so a full name table never refuses them) and a record listener (`add_record_listener`), which `_offer` calls when it accepts a record and `publish` when it commits one, for watched names, with the names lock held; it only queues the record. Records heard before the watch knows its baseline wait and are compared with it (`_NameWatch.begin`). The keepers listener (`KeepersFeature.add_listener`) isn't used: its records reach `NameManager.offer` too | the `GET /api/name/watch` long poll, on the thread `rnfs-client-name-watch` | | `on_progress` / `Job.watch(callback)` | `Job`, at most twice a second (Node) or once a second (Client) while it changes, and once at its final state | the job-watch thread (`rnfs-job-watch`) polls `JobsFeature.get(id)` (a snapshot made under the jobs lock) every `fetcher.POLL` (0.5 s); `fetcher.observing` and the `Observer` (the `Job`, on the job's thread) stay the jobs feature's | `rnfs-client-job-watch` polls `GET /api/jobs/get?id=` once a second; `wait()` loops `GET /api/jobs/get?id=&wait=30` | -| LXMF links received | not offered in 0.2 (decision 5) | | | +| `events.watch(callback, kinds=None)` | `Event`, one call per event of `kinds` published after `watch()`, in order and never coalesced (an event says what changed, not a state); a "lost" `Event` (`event.lost`) where events were missed, after which the app reads again what it shows | the event log's listener (`EventLog.add_listener`, added at the first watch, removed at `close()`), called after `publish` releases the log's condition, perhaps under the names or storage lock: it only queues. The baseline is the log's `last` and `epoch`; a Node never misses events | `GET /api/events` long-polled for every kind on `rnfs-client-event-watch`, from the first watch's baseline (`after` absent, inside `watch()`, so an older daemon raises `NoSuchEndpoint` there), passing back `next` and `epoch`; on `lost` (fell over 1024 events behind, or the daemon restarted) each watch gets the marker and polling goes on from `next`. Idle, forgetting its position, while no watch is left; a failed poll is reported and retried after 5 s | +| LXMF links received | not offered in 0.2 (decision 5); `events.watch(kinds="inbox")` says when the inbox changes | | | Rules, the same for both: - Every callback runs on the instance's event thread (`rnfs-events` in a Node, `rnfs-client-events` in a Client), one at a time, in the order events arrived. Events are coalesced per name and per job, so a slow - callback gets the latest state, not a backlog. + callback gets the latest state, not a backlog; `events.watch`'s are not (each is posted under a key of its own), + so a callback slower than the events lets them queue. Each event watch (`interface._EventWatch`) keeps its + position, the run's epoch and the last sequence it passed: events heard before its baseline is known wait in it + and only those after it are delivered, and a "lost" marker moves it to the new run unless it began after it. + Its deliveries are queued under its lock, so they reach the event thread in the order its position passed them, + whichever threads heard them; `EventLog.publish` tells listeners in sequence order (its `_order` lock). + A Client takes one baseline at a time (`_baseline_lock`, the request and the position together, so two first + watches can't leave the poller past events the older one hasn't had) and sets the poller's position only when it + has none: a baseline from a restarted daemon leaves it on the old run, whose next poll answers "lost" and moves + every older watch on (the new watch began in the new run and skips that marker). - None of rnfs's locks is held: hooks inside rnfs only enqueue (`_Events.post`, whose condition is held just to queue, innermost), and the thread calls app code with nothing held. A callback may call any method of the Node or Client. - A callback must not block for long: it delays every other callback of that instance. Long work (a fetch, @@ -747,7 +869,8 @@ Rules, the same for both: | Settings, DHT mode, LXMF, lifecycle: a Node's own; the daemon's for a Client | Documented: a Client can't change them; `close()` only ends the Client | | No gateway or API in a Node | Not offered in 0.2 (decision 6) | | The API's `GET /api/info` has no `raw` | A Client's `info(raw=True)` (and `open(raw=True)`) takes a sealed file's `stored_size` as its size | -| Keep/follow, contacts, peers, stats, LXMF only through the escape hatches | Typed methods later, once an app needs them | +| A Client's requests can be bounded (`timeout`, `timeout()`); a Node's calls make none | A Node's `timeout()` checks its argument and does nothing; `Job.wait(timeout)` bounds a wait on both | +| LXMF is off in a Node unless `lxmf_enabled` | `lxmf.address`, `send` and `sync` raise `ValueError` saying so, as a daemon with it off answers (400) | | `OSError` in a Node is `ServerError` from a Client | Documented | | A Client's connection errors (`DaemonNotRunning`, …) | Client only, all `ApiError` | @@ -757,7 +880,10 @@ It stays the protocol layer: `Client` was renamed `ApiConnection` (unreleased; t `tests/test_apiclient.py` changed mechanically), keeping `request`, `call`, `get`, `post`, `Stream`, the token, proof and version check, and its dict-returning methods, which become `rnfs.Client`'s backend. Its errors are re-based as above. Its module docstring and README point apps to `rnfs.Client`. The CLI's output can't change: -it calls the same methods with the same parameters. It gained `name_watch` and `cat(from_node=)`. +it calls the same methods with the same parameters. It gained `name_watch` and `cat(from_node=)`, and in 0.1.7 +the feature methods `rnfs.Client` uses (`peers`, `stats`, `contacts`, `contact_get`/`_add`/`_rm`, +`name_keep`/`_unkeep`/`_kept`/`_follow`/`_unfollow`/`_following`, `lxmf_address`/`_send`/`_inbox`/`_inbox_get`/ +`_inbox_rm`/`_outbox`/`_sync`) and per-call timeouts (`timeout=`, `using_timeout`, `effective_timeout`). ### Example app @@ -789,13 +915,17 @@ in these, each tested (tests/test_embedded.py: the guards in both scopes, and a limits; LXMF can't shut its router down, so it runs until the process ends. - The lock, as above. Not done in a Node: the debug log, `threading.excepthook`, `faulthandler`, signal handlers, `RNS.logdest`, `RNS.exit()`. -- Threads added: `rnfs-events` and `rnfs-job-watch` (Node), `rnfs-client-events`, `rnfs-client-job-watch` and - `rnfs-client-name-watch` (Client), each started on first use. Locks added: `_Events`' and `_JobPoller`'s - conditions (innermost: held only to queue or take work); `Rnfs._watch_lock`, taken after the names lock (the - record listener runs under it), never before; `NameManager._watched`, under `names._lock`; - `api.NameWatches.lock`, taken before the names lock, never under it; and three leaves, never held while calling - out: `remote.Client._poll_cond` (the name poller's wake-up), `interface._NameWatch.lock` (a watch's baseline and - last sequence) and `interface.Watch._lock` (`cancel()`'s once-only check). Callbacks run with no rnfs lock held. +- Threads added: `rnfs-events` and `rnfs-job-watch` (Node), `rnfs-client-events`, `rnfs-client-job-watch`, + `rnfs-client-name-watch` and `rnfs-client-event-watch` (Client), each started on first use. Locks added: + `_Events`' and `_JobPoller`'s conditions (innermost: held only to queue or take work); `Rnfs._watch_lock`, taken + after the names lock and storage's (the record and event listeners run under them), never before; + `NameManager._watched`, under `names._lock`; `api.NameWatches.lock`, taken before the names lock, never under + it; `remote.Client._poll_cond` (the name and event pollers' wake-up and the event poller's position), taken + before `_watch_lock`, never under it, and never held while calling out; `remote.Client._baseline_lock`, taken + first, around one event baseline request and then `_poll_cond`; `interface._EventWatch.lock` (a watch's + baseline and position), held only while it queues deliveries (`_Events`' condition, under it); and two leaves: + `interface._NameWatch.lock` (a watch's baseline and last sequence) and `interface.Watch._lock` (`cancel()`'s + once-only check). Callbacks run with no rnfs lock held. ### Decisions taken at review @@ -858,6 +988,31 @@ the names of many nodes from one `get_path_table` RPC, for `rnfs peers`); the airtime policy (`TokenBucket`, `AirtimePolicy`, interface matching, quiet-hours parsing, all with injected clocks); and `Traffic`, thread-safe counters per peer (LRU, at most 1000), per interface (at most 64) and in total, never persisted. +- Per peer, `Traffic` also keeps `first_heard` (wall time, when first counted), `rtt` and `rtt_at` (`measured`: + `LinkPool` calls its `on_rtt`, which `PeerClient` sets to `traffic.measured`, with `Link.rtt` for every link + `_open` establishes, shared, one-shot (`ping`) or identified, only a finite non-negative number) and, on + `Traffic.monotonic` (so a wall-clock step decides nothing), `first_counted` and `last_seen`, the last sign the + peer was up: a request it answered (`ANSWERED` or `REFUSED`, not `FAILED`), a `/blocks` response sent to it + (`sent` with its node ID), a link established, or its own announce (`announced`). `DhtAnnounceHandler` takes + RNS's five-parameter call and runs `on_announced` (the node's `_announced`) only when `is_path_response` is + false, but that isn't enough: attached to a shared instance, RNS 1.5.6 passes path answers on to its clients as + plain announces, and an announce queue can deliver one hours late. So `_announced` reads the announce's emission + time (its signed random hash: the path table's newest, `RnsNetwork.emitted_time`, which is this announce's since + RNS calls handlers only for an announce that takes the path, after writing it; in memory, unlike the announce + cache) and `Traffic.announced` counts it only if it is later than the last one counted (`last_emitted`) and at + most `ONLINE_WINDOW` old, allowing `EMISSION_SKEW` (300 s) of clock difference, seeing the peer at its emission + time (never later than now). An unreadable emission doesn't count. Path responses and stale announces still + update `last_heard` and the mode (`heard`). `mesh.online(link, last_seen, now)` is the one online rule: an open + shared link, or `last_seen` within `ONLINE_WINDOW` (900 s: three `link_idle_timeout`s, far below + `announce_interval`) on the monotonic clock, a negative age being offline. +- The peers feature (`features/peers.py`) lists them (`first_heard` is the earliest of `Traffic`'s, the routing + table's entry (now − seniority) and RNS's `known_destinations` time, `_earliest`) and publishes `peer` events + (`{"node_id", "change": "new" | "online" | "offline", "online"}`) from its `rnfs-peers` thread: `check()` every + `CHECK_INTERVAL` (30 s) computes `mesh.online` for every peer `Traffic` counts or with an open link (both + bounded; not the listing, which a routing table beyond `MAX_PEERS` cuts, and whose routing-only peers are never + online) and diffs it with the last check's. The first check only records, and `new` needs a `Traffic` entry + first counted after it, so the start and the routing table seeded from RNS's cache publish nothing; a peer + online that is no longer counted nor linked gets `offline`. - `Fetcher.path_info` (default: nothing known) is set by the node to `RnsPaths.info`; `Fetch._rank` orders both providers and open-link probes, and each provider caches its `PathInfo`. - `ResponsePools` has `max_blocks_responses // 2` slow-link slots and `fast_blocks_responses` (or the rest, if @@ -938,7 +1093,7 @@ and in total, never persisted. ## Features `rnfs/features/__init__.py` defines `Feature` and `FEATURE_MODULES` (in start order: peers, jobs, keepers, follow, -contacts, lxmf, deny). The node builds every feature in its constructor (`node.feature(name)`), starts them after +contacts, lxmf, deny, settings). The node builds every feature in its constructor (`node.feature(name)`), starts them after its own threads and stops them, in reverse, before its own services. A feature may override: `start()`, `stop(deadline=None)` (asks its threads to end, then waits for them until `deadline`, a `time.monotonic()` value: the node passes its stop budget's, so `STOP_TIMEOUT`, 5 s, bounds the features' threads and its own together; @@ -958,7 +1113,8 @@ A feature declares the features it uses: `requires` (it can't work without them; which raises `features.FeatureError` before making any if a required feature is missing or later, or two features share a name; a feature reaches another only with `self.dependency(name)`, which raises `FeatureError` for a name it didn't declare and returns `None` for an optional one not loaded. Today follow requires keepers; jobs, keepers, -follow and lxmf use contacts, contacts uses keepers and lxmf uses jobs, all optionally. What the others call on +follow and lxmf use contacts, contacts uses keepers, lxmf uses jobs, and settings uses jobs, keepers and lxmf (to +hand them live settings), all optionally. What the others call on contacts is the `features.ContactsApi` Protocol (`lookup`, `has`, `nick_of`, `resolve_id`, `expand_target`), which types `node.feature("contacts")`, `self.dependency("contacts")` and `features.find(node, "contacts")`; `tests/test_contacts.py` checks `ContactsFeature` against it, so a renamed method fails there instead of silently @@ -975,13 +1131,46 @@ its `start()` raising is logged and the others still start (`RnfsNode._start_fea | Feature | Module | What it does | Hooks in shared code | |---|---|---|---| -| `peers` | `peers.py` | `rnfs peers`, `rnfs stats`; client-mode nodes' mode and last-heard time come from the node's `rnfs.dht` announce handler, which counts every announce in `Traffic` | reads `Traffic`, the routing table, links, paths | +| `peers` | `peers.py` | `rnfs peers`, `rnfs stats`; client-mode nodes' mode and last-heard time come from the node's `rnfs.dht` announce handler, which counts every announce in `Traffic`; `first_heard`, `rtt`, `online` and `peer` events (Mesh) | reads `Traffic`, the routing table, links, paths | | `jobs` | `jobs.py` | `JobsFeature` (the job manager), `Job`, CLI progress formatting | `fetcher.observing(observer)`: a thread-local observer every `Fetch` picks up (`total`, `block`, `cancelled` checked at each step, raising `fetcher.Cancelled`) | | `keepers` | `keepers.py` | signed records (`SignedRecord`, `parse_signed`, `sign_record`, `from_announce_data`), `/name`, `KeeperClient`, `KeeperStore` | `Feature.requests`; `publisher.DhtKey` and `Publisher.add_key_source` (non-CID keys published like CIDs); `NameManager.keepers` (`published`, `heard`, `query`, `always_ask`, `local_record`), `NameManager.offer()`/`record()`; the 5-argument announce handler (packet hash) | | `follow` | `follow.py` | followed names: each check and fetch one at a time, `fetcher.open(pinned=True)`, pin inside the fetch's protection, old pin released after | keepers' `FOLLOW` reason | | `contacts` | `contacts.py` | address book; `expand_target`, `lookup(nick, kind)`, `resolve_id(text, kind)`, `node_arg` | `RnfsNode._parse_target` (never for the gateway), `JobsFeature.submit` | -| `lxmf` | `lxmf.py` | pure parts (`Link`, `parse_uri`, `format_uri`, `find_links`, `encode_field`/`decode_field`); `MessageLog` (the inbox and outbox, in the state database); `LxmfFeature` (send with direct→propagated fallback, inbox, auto-fetch, inbox get, pin release); `LxmfNetwork`, made in `start()`, holds every RNS/LXMF object | `lxmf.validate_settings` from `Config.validate`; `LXMF_PREFIX` in `storage.check_source` | +| `lxmf` | `lxmf.py` | pure parts (`Link`, `parse_uri`, `format_uri`, `find_links`, `encode_field`/`decode_field`); `MessageLog` (the inbox and outbox, in the state database); `LxmfFeature` (send with direct→propagated fallback, inbox, auto-fetch, inbox get, pin release; `inbox` events with `box` "inbox" — `received` {id, from}, `removed` {ids}, `auto` {id, index, state} — or "outbox" — `queued` {id}, `state` {id, state}, `removed` {ids} — published under its lock by `_event`); `LxmfNetwork`, made in `start()`, holds every RNS/LXMF object | `lxmf.validate_settings` from `Config.validate`; `LXMF_PREFIX` in `storage.check_source` | | `deny` | `deny.py` | `deny_lists` sources (`parse_source`, `validate_settings` from `Config.validate`): `check(source)` resolves a name (persisted), fetches the target pinned, checks it is a file within `MAX_LIST_BYTES`, parses it, pins it `deny` and `accept`s it, then releases unused `deny` pins; `run_due` on `rnfs-deny`, after the join (`CHECK_INTERVAL` for names, `retry_delay` backoff); sources no longer configured are forgotten at start | `node.deny` (core); `DENY` in `storage.check_source` | +| `settings` | `settings.py` | `rnfs config`, `GET /api/config`, `POST /api/config/set`: rnfs.toml read and rewritten, `LIVE` settings applied to the running node | `config.settings_schema`, `Config.combine`, `config.edit_settings`; `Config.overrides`/`given_defaults` (set by `Config.load`); the live hooks below | + +**Settings.** The schema is `config.settings_schema()` (standard library only, so `rnfs.Client` apps and the CLI +read it without a daemon): one entry per `Config` field but `rnfs_dir`, its type and nullability from the field's +annotation, ranges from the tables `Config.validate` checks (`WHOLE_NUMBERS`, `OPTIONAL_WHOLE_NUMBERS`, `RATES`, +`DURATIONS`, `CHOICES`, and `FEATURE_LIMITS` for what features check), units, groups (`SECTIONS`, the markers in +`Config`), a one-line `DESCRIPTIONS` entry and `live` (`LIVE_SETTINGS`); `tests/test_settings.py` fails for a +field without one, and checks each entry against `validate`. `Config.load` records what it was given over +rnfs.toml (`overrides`: the daemon's command line, an embedded node's settings) and under it (`given_defaults`), +so `GET /api/config` can say for each setting the running value, rnfs.toml's (`file`, null where it isn't set: +TOML has no null), `overridden` and `pending` (a restart with the same command line would run another value), +plus `file_error` when a restart would refuse the file. `POST /api/config/set` runs under the feature's lock: it +reads rnfs.toml through `open_checked` (a symlink or another user's file is refused, 400), applies the changes to +its values (a name that isn't a setting, or `rnfs_dir`, may only be removed, and only if the file has it), and +checks them with `Config.combine` and `Config.check_reticulum_config` (`restart_config`) both alone (a start +without the command line, and clients reading the port, must accept the file) and with the overrides; then +`config.edit_settings` rewrites just the changed assignments, every statement of a changed name (a dotted key, an +inline table) included: lines end at `\n` only, as TOML's do, and each statement's extent comes from a small +lexer (`_statement_extent`: strings, multi-line ones too, and bracket depth), then one `tomllib` parse, so +editing is linear in the file's size; a comment ending it is kept; new settings are appended. Strings are written +with control characters, NEL, U+2028 and U+2029 escaped. The +result is checked to parse to exactly the intended values and to fit `MAX_SETTINGS_SIZE`, the file is re-read +(changed meanwhile: 503 busy, nothing written) and replaced with `fsutil.write_atomic` (0600, private, a symlink +at the path replaced, not followed). Last, each `LIVE` setting not overridden is assigned on `node.config`, the one +`Config` the node's parts share, and its hook, if any, hands it to what copied it at start: `Storage.max_storage` +(then `_enforce_storage` at once), `JobsFeature._schedule` (queued jobs a higher `max_jobs` allows), +`RequestServer.airtime`'s `serve`/`no_serve`/`quiet` (each replaced whole), `KeeperStore.max_names`, and +`LxmfFeature.accept_from`/`auto_fetch_from` (sets replaced whole). The others in `LIVE` are read from +`node.config` at each use. Not live, deliberately: identity, ports, Reticulum, `dht_mode`, private mode and +`members`, and `remote_pin_identities` (a removed identity's waiting remote pins are dropped only at a start), +`upload_budget` (its token bucket holds charges of responses in flight), `gateway_fetches` (the gateway isn't the +node's), `verbosity` (RNS clamps its loglevel, so a change can't be undone exactly) and `lxmf_sync_interval` +(read only after each download). `LxmfNetwork` makes an `LXMF.LXMRouter` with storage `/lxmf/` and `signal.signal` stubbed (LXMRouter installs SIGINT/SIGTERM handlers that exit at once), a 256 kB delivery limit, announce data without @@ -1037,11 +1226,19 @@ An output without `-o` is named with `fsutil.safe_default_name`. Sent links carr `GET /api/name/following`; `GET /api/contacts`, `GET /api/contacts/get?nick=`, `POST /api/contacts/add`, `/api/contacts/rm`; `GET /api/lxmf/address`, `POST /api/lxmf/send`, `GET /api/lxmf/inbox`, `POST /api/lxmf/inbox/get`, `/api/lxmf/inbox/rm`, `GET /api/lxmf/outbox`, `POST /api/lxmf/sync`; `GET /api/deny`, -`POST /api/deny/add`, `/api/deny/rm` (`{"entries": [...]}`). `POST /api/jobs` +`POST /api/deny/add`, `/api/deny/rm` (`{"entries": [...]}`); `GET /api/config`, `POST /api/config/set` +(`{"settings": {name: value or null}}`). `POST /api/jobs` takes, for a get, the JSON booleans `sealed_name` (write a sealed file this node opens under its safe sealed name in the output's directory; the CLI sets it for `rnfs get ` without `-o`, so it no longer calls `GET /api/info` first) and `fallback_dht` (a get with `from` that fails, other than by cancellation, -`TooLarge` or `StorageFull`, is retried once through the DHT). `POST /api/lxmf/send` and `/api/lxmf/sync` take +`TooLarge` or `StorageFull`, is retried once through the DHT); for `kind` `add`, `path` and `POST /api/add`'s +options (flat, as `/api/add` takes them; kept in the job as `options`, only those set), for `import`, `path`. +Add and import jobs run `add_path`/`import_archive` with `progress=Job.read`, which counts `done_bytes` (and the +rate, of bytes read) and raises `Cancelled` once cancelled or stopping; their `total_bytes` is `builder.tree_size` +(measured first, cancellable between entries) or the archive's size. They hold no protection session (they fetch +nothing) and an interrupted one runs again from the start. Every job state change, removal (`removed`: true) and, +at most every `EVENT_INTERVAL` (1 s) on the job's thread, its progress is a `job` event (`Job.event`: `id`, +`job_kind`, `state`, `error` and the progress fields; never the result). `POST /api/lxmf/send` and `/api/lxmf/sync` take `nearest`; `POST /api/lxmf/inbox/get` takes `use_sent_cid` (without it, an unresolvable name link is 404); `GET /api/lxmf/address` returns `propagation_node` (the configured one only) and `nearest_propagation_node`. `GET /api/lxmf/inbox` adds to each record, computed when listed, `trusted` (`LxmfFeature._trusted_hex`) and @@ -1063,9 +1260,10 @@ Core routes gained options: `POST /api/add` `pin`, `to`, `to_self`, `seal_name`, | `rnfs-job-` | jobs | one per running job, at most `max_jobs`; `stop()` waits for them until the node's stop deadline | | `rnfs-keepers`, `rnfs-keeper-query` | keepers | pending refreshes (≤ 64 queued), own-name sync, saves, re-asking every `announce_interval`; parallel `/name` queries | | `rnfs-follow` | follow | every check and fetch, one at a time | +| `rnfs-peers` | peers | every 30 s, the peers' online states diffed into `peer` events (no network I/O) | | `rnfs-deny` | deny | every deny-list source's check and fetch, one at a time, once joined | | `rnfs-events`, `rnfs-job-watch` | rnfs.Node | callbacks into the program, one at a time; polling watched jobs (both started on first use) | -| `rnfs-client-events`, `rnfs-client-job-watch`, `rnfs-client-name-watch` | rnfs.Client (in the program) | callbacks; polling watched jobs; the `GET /api/name/watch` long poll | +| `rnfs-client-events`, `rnfs-client-job-watch`, `rnfs-client-name-watch`, `rnfs-client-event-watch` | rnfs.Client (in the program) | callbacks; polling watched jobs; the `GET /api/name/watch` and `GET /api/events` long polls | | `rnfs-lxmf`, `rnfs-lxmf-fetch` | lxmf | announces, propagation-node downloads, polling outbound states each second while a message is followed or held or a download runs (otherwise it sleeps until the next announce or download; `LxmfNetwork.send`, `_hold` and `sync` wake it); auto-fetch, one at a time, senders in turn (LXMF runs its own job thread) | The API and gateway run one thread per connection, capped (32 and 64). The daemon's main thread waits for @@ -1110,11 +1308,15 @@ runs while a lock is held: `/state/rnfs.db` (rnfs/statedb.py), one table set per store; "FULL" commits are on disk when the change returns, "NORMAL" ones are batched saves a power loss may lose the last of: -- storage: `pins` (one row per source; each pin and release FULL, after the blocks are synced) and `cache` - (cached fetches, LRU order in `used`, at most 10 000; NORMAL every 30 s and at stop, changed records only); +- storage: `pins` (one row per source; each pin and release FULL, after the blocks are synced), `pin_times` + (0.1.7: when each pinned CID was first pinned by any source, written in the same transaction as the pin that + made it pinned and the release that ended it; pins of earlier versions have none, and a row whose pin is gone + is deleted at start; a table of its own so the released `pins` stays as 0.1.5 made it) and `cache` (cached + fetches, LRU order in `used`, at most 10 000; NORMAL every 30 s and at stop, changed records only); - `pin_names` (rnfs/pin_names.py: where each pin was added from, local only, pruned to the pin list; NORMAL, each change written at once, after the pin it names was committed FULL: a power loss can lose a name, never the - pin, and an add costs one durable commit, not two); + pin, and an add costs one durable commit, not two) and `pin_labels` (0.1.7: the label its user gave a pin, + apart from where it came from; FULL, each change written at once; pruned with the names); - node: `fetched` (targets fetched for the user, for `roots`; with the cache records) and `remote_pins` (remote pins accepted and not yet fetched, in order; about once a second and at stop, rewritten whole); fetcher: `invalid` (invalid roots, oldest first; with the cache records); diff --git a/rnfs/__init__.py b/rnfs/__init__.py index bbdf541..3ad8fe1 100644 --- a/rnfs/__init__.py +++ b/rnfs/__init__.py @@ -28,13 +28,19 @@ "Node": "rnfs.embedded", "Client": "rnfs.remote", **{name: "rnfs.objects" for name in ("Cid", "Added", "Written", "Entry", "Info", "Stat", "Pin", "Record", - "Status", "Collected", "Verified", "Exported", "Imported", "DenyState")}, - **{name: "rnfs.interface" for name in ("Rnfs", "Name", "Names", "Job", "Jobs", "Deny", "Watch", "File")}, + "Status", "Collected", "Verified", "Exported", "Imported", "DenyState", + "Peer", "Stats", "Contact", "Kept", "Followed", "LxmfAddress", + "MessageLink", "Message", "Outgoing", "Sent", "Synced", "Cached", + "Uncached", "Event", "EventBatch", "Capabilities", "Setting", "Settings", + "SettingsChange")}, + **{name: "rnfs.interface" for name in ("Rnfs", "Name", "Names", "Job", "Jobs", "Deny", "Watch", "File", + "Contacts", "Lxmf", "Fetched", "Events")}, **{name: "rnfs.errors" for name in ("RnfsError", "NotFound", "TooLarge", "InvalidDag", "StorageFull", "Busy", "Denied", "Cancelled", "Closed", "BlockMismatch", "SealError", - "NotRecipient")}, + "NotRecipient", "Unsupported")}, **{name: "rnfs.apiclient" for name in ("ApiError", "DaemonNotRunning", "ConnectionFailed", "TokenError", - "AuthFailed", "SettingsError", "BadRequest", "ServerError")}, + "AuthFailed", "SettingsError", "BadRequest", "ServerError", + "NoSuchEndpoint")}, } __all__ = ["API_STATUS", "__version__", *_EXPORTS] diff --git a/rnfs/api.py b/rnfs/api.py index c51753e..8d0c9c4 100644 --- a/rnfs/api.py +++ b/rnfs/api.py @@ -2,7 +2,7 @@ Every command of the CLI is one request. Targets are `[/name…]` strings; node IDs and names are 32 hex characters; paths are absolute paths on the daemon's machine. Successful responses are JSON -(except cat and name export, which send bytes); errors are `{"error": message}`. +(except cat and name export, which send bytes); errors are `{"error": message, "code": code}` (below). | Method | Path | Parameters | Response / facade call | |---|---|---|---| @@ -11,20 +11,26 @@ | GET | /api/cat | `target`, `offset`, `length`, `raw`, `block`, `from` | the bytes of a file (`read`; sealed files decrypted unless `raw`; fetched from node `from` if given), Content-Length set; `block=1`: the CID's own block | | GET | /api/ls | `target` | `{"entries": ls(target)}` | | GET | /api/stat | `target` | `stat(target)` | -| GET | /api/pin/ls | `cid` (optional) | `{"pins": pin_ls(cid)}` | +| GET | /api/pin/ls | `cid` (optional) | `{"pins": pin_ls(cid)}`: `cid`, `sources`, `complete`, `type`, `sealed`, `name`, and since 0.1.7 `label`, `origin`, `size`, `entries`, `added` | | GET | /api/providers | `cid` | `{"providers": [node ID]}` | | GET | /api/ping | `node` | `{"rtt": seconds or null}` | | GET | /api/name/list | | `{"names": name_list()}` | | GET | /api/name/resolve | `name` | `name_resolve(name)` | | GET | /api/name/watch | `names` (comma-separated), `after` (a sequence or `-` per name), `wait` (≤ 30 s) | `{"records": [{"name", "cid", "sequence", "age"}]}`: the records held whose sequence is above `after`, as soon as there is one or after `wait` (below) | | POST | /api/add | JSON `path`, `wrap`, `only_hash`, `pin`, `to` (list), `to_self`, `seal_name` (bool), `chunker` | `add_path(...)` | -| POST | /api/add/stream | query `name`, `only_hash`, `pin`, `to` (comma-separated), `to_self`, `seal_name` (the name), `chunker`; body is the data | `add_stream(...)` | +| POST | /api/add/stream | query `name`, `only_hash`, `pin`, `to` (comma-separated), `to_self`, `seal_name` (the name), `chunker`, `format`; body is the data | `add_stream(...)`; `format=tar`: the body is a tar stream, `add_tar(stream, name, ...)` builds the directory it holds (no `to`, `to_self` or `seal_name`) | +| GET | /api/get/stream | `target`, `format` (`tar`) | the target as a tar stream (`get_tar`), fetched first, Content-Length set | +| GET | /api/export/stream | `target` | the target's archive (`export_chunks`), byte-identical to `export`, Content-Length set | +| POST | /api/import/stream | body is an archive | `import_stream(body)`, as `/api/import` answers | | POST | /api/get | JSON `target`, `output`, `from`, `raw` | `get(...)` | | POST | /api/dir/set | JSON `target`, `name`, `cid`, `executable` | `dir_set(...)` | | POST | /api/dir/rm | JSON `target` | `dir_rm(target)` | | POST | /api/pin/add | JSON `target` | `pin_add(target)` | | POST | /api/pin/rm | JSON `cid` | `{}` after `pin_rm(cid)` | | POST | /api/pin/remote | JSON `node`, `cid`, `pin` | `{"accepted": bool}` | +| POST | /api/pin/label | JSON `cid`, `label` (null or "" clears it) | `pin_label(cid, label)`: the pin's `pin_ls` entry | +| GET | /api/cache/ls | | `{"cache": cache_ls()}`: the cached fetches, most recently used first | +| POST | /api/cache/rm | JSON `cid` | `cache_rm(cid)`: `{"removed_records", "removed_blocks", "freed_bytes"}` | | POST | /api/gc, /api/verify | | `gc()`, `verify()` | | POST | /api/name/create | JSON `alias` | `name_create(alias)` | | POST | /api/name/rm | JSON `alias` | `{}` after `name_rm(alias)` | @@ -33,6 +39,9 @@ | POST | /api/name/publish | JSON `alias`, `target` | `name_publish(alias, target)` | | POST | /api/export | JSON `target`, `output` | `export(target, output)` | | POST | /api/import | JSON `path` | `import_archive(path)` | +| GET | /api/capabilities | | `{"version", "capabilities": [...], "routes": [[method, path]...], "features": [...], "event_kinds": [...], "error_codes": [...]}` (below) | +| GET | /api/events | `after` (a sequence; absent: none, only `next`), `kinds` (comma-separated), `epoch`, `wait` (≤ 30 s) | `{"events": [event], "next", "lost", "epoch"}` (rnfs.events): the events after `after`, as soon as there is one or after `wait` | +| POST | /api/shutdown | | `{"stopping": true}`, then the daemon stops as on SIGTERM (`unsupported` 404 where nothing can stop it) | The features (rnfs.features, each's `api_routes()`) add these; `name` is a name or a name contact's nick: @@ -65,9 +74,19 @@ | GET | /api/deny | | `{"entries": [local entry], "lists": [{"source", "cid", "entries", "state"}]}` | | POST | /api/deny/add | JSON `entries` | `{"added": [entries new to the local list]}` | | POST | /api/deny/rm | JSON `entries` | `{"removed": [...]}` (404 if one isn't on the local list) | +| GET | /api/config | | `{"path", "settings": [schema entry + "value", "file", "overridden", "pending"], "file_error", "warnings"}` (rnfs.features.settings) | +| POST | /api/config/set | JSON `settings` ({name: value, or null to remove it from rnfs.toml}) | `{"path", "applied", "restart_required", "overridden", "warnings"}`; nothing written unless the whole result is valid (400) | Both tables are checked against ApiHandler.ROUTES and the features' routes (tests/test_api.py). +Errors are `{"error": message, "code": code}`: code is machine-readable (rnfs.errors.CODES, design.md "Error +codes"), so a client tells "no such endpoint" (`no_such_endpoint`, an older daemon) from "no such content" +(`not_found`), both 404, without reading the message. `GET /api/capabilities` lists what this daemon offers +(CAPABILITIES, every route and feature), so a client checks once instead of trying each endpoint. + +Events (rnfs.events, `GET /api/events`): a long poll like the name watch below, at most MAX_EVENT_WAITERS waiting +at once (503 beyond); `after` absent answers at once with the `next` to start from. + GET parameters are in the query string. Request bodies may use Content-Length or chunked encoding. Watching names (rnfs.Client's names.watch): `GET /api/name/watch` is a long poll, level-triggered so a client loses @@ -116,12 +135,12 @@ from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer from urllib.parse import parse_qs, urlsplit -from rnfs import __version__, fsutil, ids +from rnfs import __version__, events, fsutil, ids from rnfs import node_content # --- content: add options (pin, to, chunker) --- from rnfs.apiclient import API_HEADER, CHALLENGE_HEADER, PROOF_HEADER, api_proof from rnfs.content.cid import Cid -from rnfs.errors import STATUSES, BlockMismatch, Busy -from rnfs.log import LOG_ERROR, LOG_WARNING, log +from rnfs.errors import CODES, STATUSES, BlockMismatch, Busy, Unsupported, error_code +from rnfs.log import LOG_ERROR, LOG_NOTICE, LOG_WARNING, log MAX_JSON_BODY = 1024 * 1024 # JSON parameters are tiny; this only stops a runaway body filling memory MAX_KEY_BODY = 64 * 1024 # an RNS identity file is 64 bytes @@ -133,6 +152,11 @@ MAX_WATCH_WAIT = 30 # seconds one may wait, as GET /api/jobs/get's wait= WATCH_GRACE = 60.0 # seconds a name stays watched after the last call naming it ends MAX_WATCHED = 1024 # names watched for API clients at most; the soonest to end go first +MAX_EVENT_WAITERS = 8 # GET /api/events calls waiting at once +# What this daemon offers beyond the routes themselves (GET /api/capabilities), each a promise kept once released +CAPABILITIES = ("error_codes", "capabilities", "events", "shutdown", "daemon_json", "pin_details", "pin_label", + "cache", "peer_details", "config", "jobs_add", "jobs_import", "add_tar", "get_tar", "export_stream", + "import_stream") # Exception class → HTTP status, checked in order (design.md); anything else is 500. The one table, rnfs.errors' ERROR_STATUSES = STATUSES @@ -150,6 +174,17 @@ def feature_routes() -> dict: return _feature_routes +def capabilities(node, without=()) -> dict: + """GET /api/capabilities' answer for node: {"version", "capabilities" (CAPABILITIES but those in without), + "routes" (every core and feature route, by path), "features" (those node loaded), "event_kinds", + "error_codes"}. rnfs.Node's capabilities() too, without what an embedded node can't do.""" + loaded = getattr(node, "features", None) or {} + routes = sorted({*ApiHandler.ROUTES, *feature_routes()}, key=lambda route: (route[1], route[0])) + return {"version": __version__, "capabilities": sorted(set(CAPABILITIES) - set(without)), + "routes": [list(route) for route in routes], "features": sorted(loaded), + "event_kinds": list(events.EVENT_KINDS), "error_codes": list(CODES)} + + def error_status(error: Exception) -> int: """The HTTP status for an exception raised by the node.""" for error_class, status in ERROR_STATUSES: @@ -613,21 +648,26 @@ class ApiServer(LocalServer): max_connections = 32 # the CLI makes one connection per command; this is plenty for scripts too - def __init__(self, node, port: int, token: str | None = None): - """Bind to 127.0.0.1:port, requiring token (a new random one if None); serves once start() is called.""" + def __init__(self, node, port: int, token: str | None = None, shutdown=None): + """Bind to 127.0.0.1:port, requiring token (a new random one if None); serves once start() is called. + shutdown, if given, is called (with no arguments) once POST /api/shutdown has been answered: the daemon + passes what its signal handlers do. Without it that route answers 404 unsupported.""" self.token = token if token is not None else secrets.token_urlsafe(32) if not self.token: raise ValueError("the API token must not be empty") - super().__init__(ApiHandler, port, node=node, token=self.token, watches=NameWatches(node)) + super().__init__(ApiHandler, port, node=node, token=self.token, watches=NameWatches(node), + event_waiters=threading.BoundedSemaphore(MAX_EVENT_WAITERS), + shutdown_callback=None if shutdown is None else staticmethod(shutdown)) class ApiRequestError(Exception): - """A request the API refuses before reaching the node: status and message.""" + """A request the API refuses before reaching the node: status, message and code (rnfs.errors.CODES).""" - def __init__(self, status: int, message: str): - """A refusal with status and message.""" + def __init__(self, status: int, message: str, code: str): + """A refusal with status, message and code.""" super().__init__(message) self.status = status + self.code = code class ApiHandler(TimedHandler): @@ -636,6 +676,8 @@ class ApiHandler(TimedHandler): node = None token: str | None = None watches: NameWatches | None = None + event_waiters: threading.BoundedSemaphore | None = None + shutdown_callback = None # what POST /api/shutdown calls once answered (ApiServer's shutdown) server_version = "rnfs" sys_version = "" @@ -655,6 +697,9 @@ class ApiHandler(TimedHandler): ("GET", "/api/name/watch"): "api_name_watch", ("POST", "/api/add"): "api_add", ("POST", "/api/add/stream"): "api_add_stream", + ("GET", "/api/get/stream"): "api_get_stream", + ("GET", "/api/export/stream"): "api_export_stream", + ("POST", "/api/import/stream"): "api_import_stream", ("POST", "/api/get"): "api_get", ("POST", "/api/dir/set"): "api_dir_set", ("POST", "/api/dir/rm"): "api_dir_rm", @@ -670,6 +715,12 @@ class ApiHandler(TimedHandler): ("POST", "/api/name/publish"): "api_name_publish", ("POST", "/api/export"): "api_export", ("POST", "/api/import"): "api_import", + ("POST", "/api/pin/label"): "api_pin_label", + ("GET", "/api/cache/ls"): "api_cache_ls", + ("POST", "/api/cache/rm"): "api_cache_rm", + ("GET", "/api/capabilities"): "api_capabilities", + ("GET", "/api/events"): "api_events", + ("POST", "/api/shutdown"): "api_shutdown", } OPEN_ROUTES = {("GET", "/api/status")} # allowed without the token PROOF_ROUTE = ("GET", "/api/status") @@ -682,7 +733,8 @@ def log_message(self, format, *args): @classmethod def busy_response(cls) -> bytes: """503 for a connection over max_connections.""" - body = json.dumps({"error": "too many API connections; try again shortly"}).encode() + body = json.dumps({"error": "too many API connections; try again shortly", + "code": "too_many_connections"}).encode() return raw_response("503 Service Unavailable", {"Content-Type": "application/json", "Retry-After": "10", "Cache-Control": "no-store", "X-Content-Type-Options": "nosniff"}, body) @@ -715,9 +767,9 @@ def send_json(self, status: int, body): self.end_headers() self.wfile.write(data) - def send_error_json(self, status: int, message: str, headers: dict | None = None): - """Send {"error": message} with the given status.""" - data = json.dumps({"error": message}).encode() + def send_error_json(self, status: int, message: str, code: str, headers: dict | None = None): + """Send {"error": message, "code": code} with the given status.""" + data = json.dumps({"error": message, "code": code}).encode() self.send_response(status) self.send_header("Content-Type", "application/json") self.send_header("Content-Length", str(len(data))) @@ -769,18 +821,19 @@ def dispatch(self): self.headers_sent = False try: if not host_allowed(self.headers.get("Host"), self.server.server_address[1]): - raise ApiRequestError(403, "Host header must be 127.0.0.1: or localhost:") + raise ApiRequestError(403, "Host header must be 127.0.0.1: or localhost:", "bad_host") route = (self.command, url.path) name = self.ROUTES.get(route) feature_route = None if name is not None else feature_routes().get(route) if name is None and feature_route is None: allowed = sorted(method for method, path in (*self.ROUTES, *feature_routes()) if path == url.path) if allowed: - return self.send_error_json(405, f"use {' or '.join(allowed)}", {"Allow": ", ".join(allowed)}) - raise ApiRequestError(404, f"no such API endpoint: {url.path}") + return self.send_error_json(405, f"use {' or '.join(allowed)}", "method_not_allowed", + {"Allow": ", ".join(allowed)}) + raise ApiRequestError(404, f"no such API endpoint: {url.path}", "no_such_endpoint") if route not in self.OPEN_ROUTES and not self.token_valid(): raise ApiRequestError(403, f"API requests must carry the header {API_HEADER}: , with the " - f"token the daemon wrote to /api_token") + f"token the daemon wrote to /api_token", "unauthorized") if self.command != "POST": self.request_read() # no body; POST endpoints call it once they have read theirs self.keep_open = (route == self.PROOF_ROUTE and not self.proved @@ -798,9 +851,9 @@ def dispatch(self): self.deadline = time.monotonic() + self.request_timeout self.phase = "request" except ApiRequestError as e: - self.respond_error(e.status, str(e)) + self.respond_error(e.status, str(e), e.code) except RequestTimeout as e: - self.respond_error(408, str(e)) + self.respond_error(408, str(e), "timeout") except Exception as e: # noqa: BLE001 - every error becomes a response, never a dropped connection status = error_status(e) if status == 500: @@ -809,14 +862,15 @@ def dispatch(self): # Answered 400 as a ValueError, as before, but the node's own fault: logged, with its traceback log(f"API {self.command} {loggable(url.path)} failed: {e!r}", LOG_WARNING) message = str(e) or type(e).__name__ - self.respond_error(status, message, {"Retry-After": "10"} if status == 503 else None) + self.respond_error(status, message, error_code(e, status), + {"Retry-After": "10"} if status == 503 else None) - def respond_error(self, status: int, message: str, headers: dict | None = None): + def respond_error(self, status: int, message: str, code: str, headers: dict | None = None): """Send an error response, unless a response is already under way (then just drop the connection).""" if self.headers_sent: return try: - self.send_error_json(status, message, headers) + self.send_error_json(status, message, code, headers) except OSError: pass # the client has gone away @@ -846,13 +900,13 @@ def body_reader(self) -> BodyReader: return BodyReader(self.rfile, 0, chunked=True) length = self.headers.get("Content-Length") if length is None: - raise ApiRequestError(411, "Content-Length or chunked encoding required") + raise ApiRequestError(411, "Content-Length or chunked encoding required", "length_required") try: size = int(length) except ValueError: - raise ApiRequestError(411, "invalid Content-Length") from None + raise ApiRequestError(411, "invalid Content-Length", "length_required") from None if size < 0: - raise ApiRequestError(400, "invalid Content-Length") + raise ApiRequestError(400, "invalid Content-Length", "bad_request") return BodyReader(self.rfile, size, chunked=False) def stream_reader(self) -> BodyReader: @@ -871,7 +925,7 @@ def read_body(self, limit: int) -> bytes: return b"" data = self.body_reader().read(limit + 1) if len(data) > limit: - raise ApiRequestError(413, "request body too large") + raise ApiRequestError(413, "request body too large", "body_too_large") self.request_read() return data @@ -944,6 +998,52 @@ def api_cat(self): if close: close() # stop a generator's fetch if the client went away + def api_get_stream(self): + """GET /api/get/stream?target=&format=tar: the target as a tar (RnfsNode.get_tar), its whole DAG fetched + first so the response has an exact Content-Length.""" + target = parse_target(self.param("target")) + if self.param("format") != "tar": + raise ValueError("format must be tar") + self.send_chunks("get/stream", target, "application/x-tar", lambda: self.node.get_tar(target)) + + def api_export_stream(self): + """GET /api/export/stream?target=: the target's archive (RnfsNode.export_chunks), as `export` writes it.""" + target = parse_target(self.param("target")) + self.send_chunks("export/stream", target, "application/vnd.rnfs.archive", + lambda: self.node.export_chunks(target)) + + def send_chunks(self, what: str, target: str, content_type: str, open_chunks): + """Answer with what open_chunks() gives, (info with "length", chunks), sent with that exact Content-Length. + + open_chunks fetches the whole DAG first, while a client that disconnects cancels it (as the gateway's + ?format=archive); an error before the first byte gets its status, one after it cuts the body short, which + the Content-Length tells the client.""" + from rnfs.errors import Cancelled + from rnfs.fetcher import observing + from rnfs.gateway_formats import ClientWatch + try: + with observing(ClientWatch(self.connection)): + info, chunks = open_chunks() + except Cancelled: + self.close_connection = True # nobody to answer + return + try: + self.send_response(200) + self.send_header("Content-Type", content_type) + self.send_header("Content-Length", str(info["length"])) + self.send_common_headers() + self.end_headers() + for chunk in chunks: + self.wfile.write(chunk) + except Exception as e: + if not self.headers_sent: + raise + if not isinstance(e, OSError): # an OSError is the client gone + log(f"API {what} {loggable(target)} failed mid-stream: {e!r}", LOG_ERROR) + self.close_connection = True + finally: + chunks.close() # ends the fetch + def api_ls(self): """GET /api/ls?target=: directory entries.""" self.send_json(200, {"entries": list(self.node.ls(parse_target(self.param("target"))))}) @@ -1012,7 +1112,16 @@ def api_add_stream(self): name = self.param("name", False) or None only_hash = parse_bool(self.param("only_hash", False) or "") options = node_content.add_options(lambda key: self.param(key, False), stream=True) # pin, to, seal_name… - self.send_json(200, self.node.add_stream(reader, size=size, name=name, only_hash=only_hash, **options)) + stream_format = self.param("format", False) + if stream_format is None: + self.send_json(200, self.node.add_stream(reader, size=size, name=name, only_hash=only_hash, **options)) + return + if stream_format != "tar": + raise ValueError("format must be tar") + if {"to", "to_self", "seal_name"} & options.keys(): + raise ValueError("a tar stream can't be sealed: sealing is for single files") + # add_tar reads the body to its end, checking that only zeros follow the tar's (its padding) + self.send_json(200, self.node.add_tar(reader, name=name, only_hash=only_hash, **options)) def api_get(self): """POST /api/get: fetch and write out to a path on this machine (parameters: the module docstring's table).""" @@ -1044,6 +1153,19 @@ def api_pin_rm(self): self.node.pin_rm(parse_cid(self.json_body().get("cid"))) self.send_json(200, {}) + def api_pin_label(self): + """POST /api/pin/label {"cid", "label"}: label a pin (null or "" clears it); its pin ls entry.""" + body = self.json_body() + self.send_json(200, self.node.pin_label(parse_cid(body.get("cid")), body.get("label"))) + + def api_cache_ls(self): + """GET /api/cache/ls: the cached fetches (downloads not pinned), most recently used first.""" + self.send_json(200, {"cache": self.node.cache_ls()}) + + def api_cache_rm(self): + """POST /api/cache/rm {"cid"}: evict the cached fetches leading to cid now.""" + self.send_json(200, self.node.cache_rm(parse_cid(self.json_body().get("cid")))) + def api_pin_remote(self): """POST /api/pin/remote {"node", "cid", "pin"}: ask a node to pin (or release) a CID.""" body = self.json_body() @@ -1096,3 +1218,49 @@ def api_export(self): def api_import(self): """POST /api/import {"path"}: import an archive from a path on this machine.""" self.send_json(200, self.node.import_archive(parse_path(self.json_body().get("path")))) + + def api_import_stream(self): + """POST /api/import/stream: import the archive the request body holds, read to its end.""" + self.send_json(200, self.node.import_stream(self.stream_reader())) + + # Capabilities, events, shutdown + + def api_capabilities(self): + """GET /api/capabilities: what this daemon offers (the module docstring).""" + self.send_json(200, capabilities(self.node)) + + def api_events(self): + """GET /api/events?after=&kinds=&epoch=&wait=: the node's events after a sequence (rnfs.events).""" + event_log = getattr(self.node, "events", None) + if event_log is None: + raise Unsupported("this node keeps no event log") + after = parse_int(self.param("after", False), "after", None) + kinds_text = self.param("kinds", False) + kinds = [kind for kind in kinds_text.split(",") if kind] if kinds_text else None + if kinds_text and not kinds: + raise ValueError("kinds lists no event kind") + for kind in kinds or (): + if kind not in events.EVENT_KINDS: + raise ValueError(f"unknown event kind {kind!r}; known: {', '.join(events.EVENT_KINDS)}") + epoch = self.param("epoch", False) or None + wait = min(parse_int(self.param("wait", False), "wait", 0), MAX_WATCH_WAIT) if after is not None else 0 + if wait > 0: + if not self.event_waiters.acquire(blocking=False): + raise Busy(f"{MAX_EVENT_WAITERS} event polls are waiting already") + try: + result = event_log.since(after, kinds, wait, epoch) + finally: + self.event_waiters.release() + else: + result = event_log.since(after, kinds, 0, epoch) + self.send_json(200, result) + + def api_shutdown(self): + """POST /api/shutdown: answer, then stop the daemon as SIGTERM does (ApiServer's shutdown).""" + self.json_body() + if self.shutdown_callback is None: + raise Unsupported("this daemon can't be stopped through its API") + log("stopping: asked through the API", LOG_NOTICE) + self.send_json(200, {"stopping": True}) + self.wfile.flush() + self.shutdown_callback() diff --git a/rnfs/apiclient.py b/rnfs/apiclient.py index 6fdbd77..ce695e1 100644 --- a/rnfs/apiclient.py +++ b/rnfs/apiclient.py @@ -14,29 +14,44 @@ Connecting and proving take at most PROOF_TIMEOUT seconds, reading at most PROOF_READ_LIMIT bytes. The proof is made on the connection that then carries the request (the daemon keeps it open for that one request), for every request, so a client used for a long time sends nothing to a program listening on the port after the - daemon stopped. Only a daemon proved older than 0.1.5, which closes every connection, gets the request on a - new one (KEEPS_PROOF_OPEN_SINCE, until 0.1.7). + daemon stopped. A daemon that closes that connection gets nothing more (ConnectionFailed). +- The port: a running daemon (0.1.7 on) writes `/daemon.json` (read_daemon_info: its PID, ports, + version, start time and Reticulum config), which is read first, so a client finds the daemon whatever its + rnfs.toml holds (an older client can't read a newer one's settings); without it (or with a stale one), the + api_port in rnfs.toml. Nothing is trusted from it but the port, and the proof below still comes first. - That proof request is also where the daemon's version is read (Client.daemon_version), and check_daemon is the - one place to refuse a daemon by it; every version is accepted for now. + one place to refuse a daemon by it: one before 0.1.5 (OLDEST_DAEMON), which closes the connection it proved + itself on so the request could only go on a new one, is refused before the token is sent (until 0.1.7 the + client did send it on a new connection). - Every failure raises an ApiError subclass whose message says what went wrong, as the CLI prints it: no daemon (DaemonNotRunning), the connection failing (ConnectionFailed), the token (TokenError), the proof or the token refused (AuthFailed), the settings (SettingsError), and the API's own error statuses (STATUS_ERRORS, from rnfs.errors.STATUSES: 400 BadRequest, 404 NotFound, 413 TooLarge, 422 InvalidDag, 451 Denied, 503 Busy, 507 StorageFull, other 5xx ServerError). ApiError is an rnfs.errors.RnfsError, and each status's error also a subclass of the node's own (NotFound of rnfs.errors.NotFound, BadRequest of ValueError, ...), so one except - clause catches what rnfs.Node and rnfs.Client raise alike. -- Each request opens its own connection (its proof, then the request), with no timeout for the request unless - one is given (fetches can take hours), so an ApiConnection may be used from several threads. Paths are on the + clause catches what rnfs.Node and rnfs.Client raise alike. Every ApiError has the daemon's machine-readable + code (rnfs.errors.CODES; None from a daemon before 0.1.7): a 404 for an endpoint the daemon doesn't have is + NoSuchEndpoint (also told from an older daemon's message), one for something it can't do Unsupported, both + NotFound subclasses. +- Each request opens its own connection (its proof, then the request), so an ApiConnection may be used from + several threads. A request has no timeout unless one is given (fetches can take hours): the connection's + (timeout=), replaced for every request one thread makes inside using_timeout(seconds), or for one call by + request/call/get/post's own timeout= (None: none, even if the connection has one). Connecting and proving take + at most the smaller of it and PROOF_TIMEOUT. A request outlasting it raises ConnectionFailed. Paths are on the daemon's machine, which is this one: they are made absolute here. """ +import contextlib import hashlib import hmac import http.client import json +import math import os import re import secrets import stat +import threading +import time import warnings from urllib.parse import urlencode @@ -45,6 +60,8 @@ DEFAULT_DIR = Config.rnfs_dir TOKEN_FILE = "api_token" # in the rnfs directory; written by the daemon, read by every client +DAEMON_FILE = "daemon.json" # in the rnfs directory: the running daemon's PID, ports, version (read_daemon_info) +MAX_DAEMON_FILE = 64 * 1024 # bytes of daemon.json read at most (the daemon's is a few hundred) API_HEADER = "X-Rnfs-Api" # Before sending the token, a client makes whatever listens on the API port prove it knows it (api_proof) CHALLENGE_HEADER = "X-Rnfs-Challenge" @@ -52,18 +69,20 @@ PROOF_TIMEOUT = 15 # seconds to connect and prove (at most the client's timeout): a listener mustn't hang a client PROOF_READ_LIMIT = 64 * 1024 # the most read of a proof's response (the daemon's is a few dozen bytes) COPY_SIZE = 64 * 1024 # what a Stream reads at a time when iterated -# A daemon before 0.1.5 closes the connection it proved itself on; only one proved that old gets the request on a -# new connection (the fallback is to be dropped in 0.1.7) -KEEPS_PROOF_OPEN_SINCE = (0, 1, 5) +# The oldest daemon a client talks to: one before 0.1.5 closes the connection it proved itself on, so the request +# could only go on a new connection, which another listener could take (clients before 0.1.7 sent it there) +OLDEST_DAEMON = (0, 1, 5) class ApiError(errors.RnfsError): - """An API call failed; str() says why. status is the HTTP status, or None if the daemon never answered.""" + """An API call failed; str() says why. status is the HTTP status, or None if the daemon never answered; code + the daemon's machine-readable code (rnfs.errors.CODES), or None (no response, or a daemon before 0.1.7).""" - def __init__(self, message: str, status: int | None = None): - """An error with message, from a response with status (None: no response).""" + def __init__(self, message: str, status: int | None = None, code: str | None = None): + """An error with message, from a response with status (None: no response) and code.""" super().__init__(message) self.status = status + self.code = code class DaemonNotRunning(ApiError): @@ -95,6 +114,14 @@ class NotFound(ApiError, errors.NotFound): """404: no such content, name, pin, job or endpoint.""" +class NoSuchEndpoint(NotFound): + """404 no_such_endpoint: the daemon has no such endpoint (it predates it), not that some content is missing.""" + + +class Unsupported(NotFound, errors.Unsupported): + """404 unsupported: the daemon can't do this at all (a daemon that can't be stopped through its API, say).""" + + class TooLarge(ApiError, errors.TooLarge): """413: over a size limit (max_download_size, a quota, the request body).""" @@ -137,10 +164,18 @@ def _status_errors() -> dict[int, type[ApiError]]: STATUS_ERRORS = _status_errors() -def error_for(status: int, message: str) -> ApiError: - """The ApiError subclass for an error response's status, with its message.""" +# A daemon before 0.1.7 sends no code; its 404 for an unknown endpoint says this +_NO_ENDPOINT_MESSAGE = "no such API endpoint" + + +def error_for(status: int, message: str, code: str | None = None) -> ApiError: + """The ApiError subclass for an error response's status (and code), with its message.""" + if status == 404 and (code == "no_such_endpoint" or (code is None and message.startswith(_NO_ENDPOINT_MESSAGE))): + return NoSuchEndpoint(message, status, code or "no_such_endpoint") + if status == 404 and code == "unsupported": + return Unsupported(message, status, code) error_class = STATUS_ERRORS.get(status) or (ServerError if status >= 500 else ApiError) - return error_class(message, status) + return error_class(message, status, code) def version_numbers(version) -> tuple[int, ...] | None: @@ -163,10 +198,89 @@ def load_settings(rnfs_dir: str = DEFAULT_DIR) -> Config: raise SettingsError(f"can't read the settings in {rnfs_dir}: {e}") from None +def daemon_info_path(rnfs_dir: str = DEFAULT_DIR) -> str: + """Where the daemon using rnfs_dir describes itself (DAEMON_FILE).""" + return os.path.join(os.path.expanduser(rnfs_dir), DAEMON_FILE) + + +def _process_alive(pid: int) -> bool: + """False only if no process pid runs; True when it does, or when that can't be told.""" + if fsutil.IS_WINDOWS: + return _windows_process_alive(pid) + try: + os.kill(pid, 0) + except ProcessLookupError: + return False + except OSError: # PermissionError: it runs, as another user + return True + return True + + +def _windows_process_alive(pid: int) -> bool: + """Windows: False only if OpenProcess says no process pid exists, or it has exited (GetExitCodeProcess); + True otherwise, also when that can't be told (another user's process can't be opened: it runs).""" + try: + import ctypes + from ctypes import wintypes + kernel32 = ctypes.WinDLL("kernel32", use_last_error=True) + kernel32.OpenProcess.restype = wintypes.HANDLE + kernel32.OpenProcess.argtypes = (wintypes.DWORD, wintypes.BOOL, wintypes.DWORD) + kernel32.GetExitCodeProcess.argtypes = (wintypes.HANDLE, ctypes.POINTER(wintypes.DWORD)) + kernel32.CloseHandle.argtypes = (wintypes.HANDLE,) + handle = kernel32.OpenProcess(0x1000, False, pid) # PROCESS_QUERY_LIMITED_INFORMATION + if not handle: + return ctypes.get_last_error() != 87 # ERROR_INVALID_PARAMETER: no such process + try: + code = wintypes.DWORD() + if not kernel32.GetExitCodeProcess(handle, ctypes.byref(code)): + return True + return code.value == 259 # STILL_ACTIVE + finally: + kernel32.CloseHandle(handle) + except (OSError, AttributeError, ValueError): + return True + + +def read_daemon_info(rnfs_dir: str = DEFAULT_DIR) -> dict | None: + """What the daemon using rnfs_dir wrote about itself at its start (DAEMON_FILE): {"pid", "api_port", + "gateway_port", "version", "started", "rnfs_dir", "reticulum_config", "shared_instance", "node_id"}, or None + if there is none, it can't be trusted (a symlink, not a regular file, writable by other users), it doesn't + parse, or its process no longer runs (a daemon that crashed). Never raises for a bad file: a client then uses + rnfs.toml, as before daemons wrote one.""" + path = daemon_info_path(rnfs_dir) + try: + fd = os.open(path, os.O_RDONLY | fsutil.O_NOFOLLOW | fsutil.O_BINARY | getattr(os, "O_NONBLOCK", 0)) + except OSError: + return None + try: + with os.fdopen(fd, "rb") as f: + info = os.fstat(f.fileno()) + if not stat.S_ISREG(info.st_mode): + return None + if not fsutil.IS_WINDOWS and (info.st_mode & 0o022 or info.st_uid != os.getuid()): + return None + raw = f.read(MAX_DAEMON_FILE + 1) + if len(raw) > MAX_DAEMON_FILE: + return None + data = json.loads(raw) + except (OSError, ValueError): + return None + if not isinstance(data, dict): + return None + pid, port = data.get("pid"), data.get("api_port") + if type(pid) is not int or type(port) is not int or not 1 <= port <= 65535 or not _process_alive(pid): + return None + return data + + def api_port(rnfs_dir: str = DEFAULT_DIR) -> int: - """The port the daemon using rnfs_dir listens on: api_port in its rnfs.toml, else DEFAULT_API_PORT. A - settings file made private on the way (others could access it) is reported as a RuntimeWarning. Raises - SettingsError if rnfs.toml can't be read.""" + """The port the daemon using rnfs_dir listens on: the one in its daemon.json (read_daemon_info), else api_port + in its rnfs.toml, else DEFAULT_API_PORT. A settings file made private on the way (others could access it) is + reported as a RuntimeWarning. Raises SettingsError if there is no usable daemon.json and rnfs.toml can't be + read.""" + running = read_daemon_info(rnfs_dir) + if running is not None: + return running["api_port"] config = load_settings(rnfs_dir) for message in config.warnings: warnings.warn(message, RuntimeWarning, stacklevel=2) @@ -217,6 +331,10 @@ def read_all(response: http.client.HTTPResponse) -> bytes: return response.read() except http.client.IncompleteRead: raise ConnectionFailed("the response was cut short; see the daemon's log") from None + except TimeoutError: + raise ConnectionFailed("the daemon didn't finish its answer within the timeout") from None + except OSError as e: # a reset + raise ConnectionFailed(f"the response failed ({e!r}); see the daemon's log") from None class Stream: @@ -241,6 +359,12 @@ def read(self, size: int = -1) -> bytes: chunk = self.response.read(size) except http.client.IncompleteRead: chunk = b"" + except TimeoutError: + raise ConnectionFailed(f"the daemon stopped sending within the timeout, after {self.received} bytes") \ + from None + except OSError as e: # a reset + raise ConnectionFailed(f"the transfer failed after {self.received} bytes ({e!r}); see the daemon's " + f"log") from None self.received += len(chunk) if not chunk and size and self.length is not None and self.received != self.length: raise ConnectionFailed(f"the transfer stopped after {self.received} of {self.length} bytes; " @@ -269,13 +393,38 @@ def _flag(value: bool) -> int | None: return 1 if value else None +class _Default: + """The type of DEFAULT.""" + + def __repr__(self) -> str: + """Its name, for signatures in help().""" + return "DEFAULT" + + +# A request's timeout when none is given for it: this thread's (using_timeout), else the connection's. Not None, +# which is a timeout of its own: none at all +DEFAULT = _Default() + + +def check_timeout(timeout) -> float | None: + """timeout if it is None or a positive number of seconds; ValueError otherwise (0 would make the socket + non-blocking, and a bool is a mistake).""" + if timeout is None: + return None + if isinstance(timeout, bool) or not isinstance(timeout, (int, float)) or not 0 < timeout < math.inf: + raise ValueError(f"a timeout is None or a positive number of seconds, not {timeout!r}") + return timeout + + class ApiConnection: """The local API of the daemon using rnfs_dir (default ~/.rnfs/), on 127.0.0.1:port (default: the api_port in its rnfs.toml). token_file defaults to /api_token and is read on the first request needing it. request() and call() are the whole protocol; get() and post() call any endpoint (features' too); the other - methods are every core route (api.ApiHandler.ROUTES), jobs and the deny list, returning what the API answers - (design.md, "Node facade"). Every failure raises an ApiError subclass. + methods are every core route (api.ApiHandler.ROUTES), jobs, the deny list and the features' routes rnfs.Client + uses (peers, contacts, keep, follow, LXMF), returning what the API answers (design.md, "Node facade"). Every + failure raises an ApiError subclass. timeout bounds each request (None: no bound; see the module docstring for + using_timeout and a call's own). """ def __init__(self, rnfs_dir: str | None = None, port: int | None = None, token_file: str | None = None, @@ -283,23 +432,51 @@ def __init__(self, rnfs_dir: str | None = None, port: int | None = None, token_f """A client; nothing is sent until the first request. Raises SettingsError if port isn't given and rnfs.toml can't be read.""" rnfs_dir = DEFAULT_DIR if rnfs_dir is None else rnfs_dir + self.rnfs_dir = rnfs_dir self.port = api_port(rnfs_dir) if port is None else port self.token_file = token_path(rnfs_dir) if token_file is None else token_file - self.timeout = timeout + self.timeout = timeout # each request's, unless the thread or the call gives another + self._local = threading.local() # .timeout: this thread's, inside using_timeout() self.token: str | None = None # the token read last, once a daemon has proved it knows it self.daemon_version: str | None = None # what the daemon said in the last proof request + # Timeouts + + def effective_timeout(self, timeout=DEFAULT) -> float | None: + """The timeout a request made now by this thread gets: timeout unless DEFAULT, else the one this thread's + using_timeout() set, else the connection's. ValueError for a timeout given that isn't one.""" + if timeout is not DEFAULT: + return check_timeout(timeout) + timeout = getattr(self._local, "timeout", DEFAULT) + return self.timeout if timeout is DEFAULT else timeout + + @contextlib.contextmanager + def using_timeout(self, timeout: float | None): + """A context manager: requests this thread makes inside it get timeout (None: none) instead of the + connection's, unless a call gives its own. Thread-local, so other threads (and their requests) keep + theirs; nests, restoring the outer one on exit. ValueError unless None or a positive number.""" + check_timeout(timeout) + previous = getattr(self._local, "timeout", DEFAULT) + self._local.timeout = timeout + try: + yield self + finally: + self._local.timeout = previous + # The protocol def request(self, method: str, path: str, params: dict | None = None, json_body: dict | None = None, body=None, length: int | None = None, extra_headers: dict | None = None, - authenticated: bool = True) -> http.client.HTTPResponse: + authenticated: bool = True, timeout=DEFAULT) -> http.client.HTTPResponse: """Send a request and return the successful response, whose body the caller reads (read_all, Stream). params go in the query string (None values left out); json_body is sent as JSON; body may be bytes or a binary file, sent chunked without length. Unless authenticated is False (version(), and the proof request itself), the token is sent, on a connection whose other end has just proved it knows it. + timeout bounds this request (None: no bound; DEFAULT: effective_timeout()), connecting and proving within + at most PROOF_TIMEOUT of it; it stays the response's, for reading its body. """ + timeout = self.effective_timeout(timeout) url = path + ("?" + urlencode({k: v for k, v in params.items() if v is not None}) if params else "") # Host is set explicitly: the API only accepts 127.0.0.1:, even for port 80 headers = {"Host": f"127.0.0.1:{self.port}", **(extra_headers or {})} @@ -311,13 +488,13 @@ def request(self, method: str, path: str, params: dict | None = None, json_body: if length is not None: headers["Content-Length"] = str(length) if authenticated: - connection, headers[API_HEADER] = self._prove(self._connect(self._proof_timeout())) - # Proved: the request itself may take as long as the client allows - connection.timeout = self.timeout + connection, headers[API_HEADER] = self._prove(self._connect(self._proof_timeout(timeout)), timeout) + # Proved: the request itself may take as long as its timeout allows + connection.timeout = timeout if connection.sock is not None: - connection.sock.settimeout(self.timeout) + connection.sock.settimeout(timeout) else: - connection = self._connect() + connection = self._connect(timeout) response = self._exchange(connection, method, url, body, headers) try: return self._raise_for(response) @@ -325,18 +502,20 @@ def request(self, method: str, path: str, params: dict | None = None, json_body: connection.close() raise - def _connect(self, timeout: float | None = None) -> http.client.HTTPConnection: - """A new connection to the API port (opened by its first request), with timeout (default: the client's).""" - return http.client.HTTPConnection("127.0.0.1", self.port, timeout=self.timeout if timeout is None else timeout) + def _connect(self, timeout: float | None) -> http.client.HTTPConnection: + """A new connection to the API port (opened by its first request), with timeout (None: none).""" + return http.client.HTTPConnection("127.0.0.1", self.port, timeout=timeout) - def _proof_timeout(self) -> float: - """The timeout for connecting and proving: PROOF_TIMEOUT, or the client's own if shorter.""" - return PROOF_TIMEOUT if self.timeout is None else min(self.timeout, PROOF_TIMEOUT) + @staticmethod + def _proof_timeout(timeout: float | None) -> float: + """The timeout for connecting and proving before a request with timeout: PROOF_TIMEOUT, or timeout if + shorter (a listener mustn't hold a request longer than the caller allows it in all).""" + return PROOF_TIMEOUT if timeout is None else min(timeout, PROOF_TIMEOUT) def _exchange(self, connection: http.client.HTTPConnection, method: str, url: str, body, headers: dict) -> http.client.HTTPResponse: """Send one request on connection and return its response, whatever its status; a connection that fails - raises DaemonNotRunning or ConnectionFailed.""" + raises DaemonNotRunning or ConnectionFailed (also when the answer doesn't come within its timeout).""" try: connection.request(method, url, body=body, headers=headers) return connection.getresponse() @@ -345,6 +524,9 @@ def _exchange(self, connection: http.client.HTTPConnection, method: str, url: st if isinstance(e, ConnectionRefusedError): raise DaemonNotRunning(f"no daemon is running on API port {self.port}. Start one with: rnfs daemon") \ from None + if isinstance(e, TimeoutError) and connection.timeout is not None: # else the system's (ETIMEDOUT) + raise ConnectionFailed(f"the daemon on API port {self.port} didn't answer within its timeout " + f"({connection.timeout:g} seconds)") from None if isinstance(e, (http.client.HTTPException, ConnectionError)): raise ConnectionFailed(f"the daemon dropped the connection ({e!r}); see its log") from None raise ConnectionFailed(f"can't reach the daemon on API port {self.port}: {e}") from None @@ -357,41 +539,39 @@ def _raise_for(self, response: http.client.HTTPResponse, data: bytes | None = No if data is None: data = read_all(response) try: - message = json.loads(data)["error"] + body = json.loads(data) + message = body["error"] except (ValueError, KeyError, TypeError): - message = f"HTTP {response.status} {response.reason}" - raise error_for(response.status, str(message)) + body, message = None, f"HTTP {response.status} {response.reason}" + code = body.get("code") if isinstance(body, dict) else None + raise error_for(response.status, str(message), code if isinstance(code, str) else None) - def _prove(self, connection: http.client.HTTPConnection) -> tuple[http.client.HTTPConnection, str]: + def _prove(self, connection: http.client.HTTPConnection, + timeout: float | None) -> tuple[http.client.HTTPConnection, str]: """Make what listens on the API port prove, on connection, that it knows the token, and return the - connection to send the request on and the token. The daemon keeps a connection it proved itself on - open for one more request, so the token and the request's body can only reach it: a proof made once, + connection to send a request with timeout on, and the token. The daemon keeps a connection it proved itself + on open for one more request, so the token and the request's body can only reach it: a proof made once, on another connection, would hold no longer than the daemon, and any local user can listen on its port once it stops. The token read is kept, and read again if the proof fails (a daemon restarted since - wrote a new one). A daemon that closes the connection anyway gets the request on a new one only if it - proved a version before 0.1.5 (whose daemons close every connection); for any other, ConnectionFailed.""" + wrote a new one). A daemon that closes the connection anyway gets nothing more: ConnectionFailed (one + before 0.1.5, which always does, was refused by check_daemon already).""" token = self.token or read_token(self.token_file) try: - version = self.verify_daemon(token, connection) + self.verify_daemon(token, connection) except AuthFailed: fresh = read_token(self.token_file) if fresh == token: raise connection.close() - connection = self._connect(self._proof_timeout()) - version = self.verify_daemon(fresh, connection) + connection = self._connect(self._proof_timeout(timeout)) + self.verify_daemon(fresh, connection) token = fresh self.token = token if connection.sock is None: - # A new connection could reach another listener (the daemon may stop in between), so only a daemon too - # old to keep it open gets the request on one, as its clients sent it - numbers = version_numbers(version) - if numbers is None or numbers >= KEEPS_PROOF_OPEN_SINCE: - raise ConnectionFailed(f"the daemon on API port {self.port} closed the connection it proved itself " - f"on, so the request wasn't sent; see its log") - connection = self._connect() - else: - connection.auto_open = 0 # never reconnect: a new connection could reach another listener + # A new connection could reach another listener (the daemon may stop in between): nothing more is sent + raise ConnectionFailed(f"the daemon on API port {self.port} closed the connection it proved itself on, " + f"so the request wasn't sent; see its log") + connection.auto_open = 0 # never reconnect: a new connection could reach another listener return connection, token def verify_daemon(self, token: str, connection: http.client.HTTPConnection | None = None) -> str | None: @@ -404,7 +584,7 @@ def verify_daemon(self, token: str, connection: http.client.HTTPConnection | Non request; every other status AuthFailed) nor its text. A proved daemon's error status raises its ApiError.""" challenge = secrets.token_hex(16) headers = {"Host": f"127.0.0.1:{self.port}", CHALLENGE_HEADER: challenge, "Connection": "keep-alive"} - connection = connection or self._connect(self._proof_timeout()) + connection = connection or self._connect(self._proof_timeout(self.effective_timeout())) response = self._exchange(connection, "GET", "/api/status", None, headers) proof = (response.getheader(PROOF_HEADER) or "").strip().encode("ascii", "replace") if not hmac.compare_digest(proof, api_proof(token, self.port, challenge).encode()): @@ -446,24 +626,34 @@ def verify_daemon(self, token: str, connection: http.client.HTTPConnection | Non def check_daemon(self, version: str | None): """The one place to refuse a daemon by its version (None if it didn't say), before the token is sent; - raise an ApiError to refuse. Every version is accepted for now.""" + raise an ApiError to refuse. A daemon proved older than OLDEST_DAEMON is refused (one that says no version, + or one that doesn't parse, is a newer one's business: a daemon that closes the connection is refused after + anyway).""" + numbers = version_numbers(version) + if numbers is not None and numbers < OLDEST_DAEMON: + raise ApiError(f"the daemon on API port {self.port} is rnfs {version}, too old for this client (it needs " + f"{'.'.join(map(str, OLDEST_DAEMON))} or later); upgrade it, or use rnfs {version}'s own " + f"command line") def call(self, method: str, path: str, params: dict | None = None, json_body: dict | None = None, - body=None, length: int | None = None, authenticated: bool = True): - """Send a request and return its decoded JSON response.""" - response = self.request(method, path, params, json_body, body, length, authenticated=authenticated) + body=None, length: int | None = None, authenticated: bool = True, timeout=DEFAULT): + """Send a request and return its decoded JSON response (timeout as request's).""" + response = self.request(method, path, params, json_body, body, length, authenticated=authenticated, + timeout=timeout) try: return json.loads(read_all(response)) except ValueError: raise ConnectionFailed("the daemon sent an invalid response") from None - def get(self, path: str, /, **params): - """GET path with query parameters; JSON response.""" - return self.call("GET", path, params=params) + def get(self, path: str, /, *, timeout=DEFAULT, **params): + """GET path with query parameters; JSON response (timeout as request's: no endpoint takes a parameter + called timeout).""" + return self.call("GET", path, params=params, timeout=timeout) - def post(self, path: str, /, **fields): - """POST path with a JSON body; JSON response.""" - return self.call("POST", path, json_body=fields) + def post(self, path: str, /, *, timeout=DEFAULT, **fields): + """POST path with a JSON body; JSON response (timeout as request's: no endpoint takes a field called + timeout).""" + return self.call("POST", path, json_body=fields, timeout=timeout) def _post(self, path: str, /, **fields): """POST path with the fields that aren't None.""" @@ -479,6 +669,61 @@ def status(self) -> dict: """The node's status: node ID, DHT mode, counts, storage and every feature's fields.""" return self.get("/api/status") + def capabilities(self) -> dict: + """What the daemon offers: {"version", "capabilities", "routes", "features", "event_kinds", "error_codes"}; + from a daemon before 0.1.7, which has no such endpoint, {} (so every capability is absent).""" + try: + return self.get("/api/capabilities") + except NoSuchEndpoint: + return {} + + def events(self, after: int | None = None, kinds: list[str] | None = None, wait: int = 0, + epoch: str | None = None) -> dict: + """The daemon's events after sequence after (of kinds, if given), as soon as there is one or after wait + seconds (the API caps it): {"events", "next", "lost", "epoch"}. after None answers at once with the + "next" to start from; pass the epoch given, so a restarted daemon answers "lost". NoSuchEndpoint from a + daemon before 0.1.7.""" + return self.get("/api/events", after=after, kinds=",".join(kinds) if kinds else None, wait=wait or None, + epoch=epoch) + + def shutdown(self, wait: float | None = None) -> bool: + """Ask the daemon to stop (as SIGTERM does). With wait, poll up to that many seconds (at least once) until + it has stopped: its API no longer answers and, when its daemon.json in this client's rnfs dir names this + port, its process has ended (it saves its state and releases the rnfs dir's lock after its API stops, so + a daemon started next can take it); True once it has, False if not within wait. Without wait, True once it + agreed. Unsupported from a daemon that can't be stopped this way, NoSuchEndpoint from one before 0.1.7.""" + running = self.daemon_info() + pid = running["pid"] if running is not None and running["api_port"] == self.port else None + self.post("/api/shutdown") + if wait is None: + return True + deadline = time.monotonic() + wait + while True: + remaining = deadline - time.monotonic() + if self._stopped(pid, max(0.1, remaining)): + return True + if remaining <= 0: + return False + time.sleep(min(0.2, remaining)) + + def _stopped(self, pid: int | None, timeout: float) -> bool: + """True if the daemon's API no longer answers (within timeout: whatever holds the port then mustn't hold + the caller) and its process pid, if known, has ended.""" + try: + self.call("GET", "/api/status", authenticated=False, timeout=timeout) + return False + except DaemonNotRunning: + pass + except ConnectionFailed: + pass # dropped, or a listener that doesn't answer: not the daemon's API either way + except ApiError: + return False # something answered + return pid is None or not _process_alive(pid) + + def daemon_info(self) -> dict | None: + """What the daemon wrote about itself (read_daemon_info) in this client's rnfs dir, or None.""" + return read_daemon_info(self.rnfs_dir) + # Content def add_path(self, path: str, wrap: bool = False, only_hash: bool = False, pin: bool = True, @@ -502,6 +747,33 @@ def add_stream(self, data, name: str | None = None, length: int | None = None, o "chunker": chunker} return self.call("POST", "/api/add/stream", params=params, body=data, length=length) + def add_tar(self, data, name: str | None = None, length: int | None = None, only_hash: bool = False, + pin: bool = True, chunker: str | None = None) -> dict: + """Add the directory a tar holds (bytes, or a binary file read to its end, sent chunked unless length is + given): regular files and directories, the rest skipped with a warning; name wraps it in a directory as that + entry. {"cid", "size", "warnings"} (+ "warnings_total"). For a program whose files the daemon can't open.""" + if isinstance(data, (bytes, bytearray, memoryview)) and length is None: + length = len(data) + params = {"format": "tar", "name": name, "only_hash": _flag(only_hash), "pin": None if pin else 0, + "chunker": chunker} + return self.call("POST", "/api/add/stream", params=params, body=data, length=length) + + def get_tar(self, target: str) -> Stream: + """A target (file or directory) as a tar, read as it arrives: the daemon fetches all of it first and sends its + exact length, so a transfer cut short raises ConnectionFailed. Files are as stored (a sealed file sealed).""" + return Stream(self.request("GET", "/api/get/stream", params={"target": target, "format": "tar"})) + + def export_stream(self, target: str) -> Stream: + """A target's archive, as export writes it, read as it arrives (fetched first, with its exact length).""" + return Stream(self.request("GET", "/api/export/stream", params={"target": target})) + + def import_stream(self, data, length: int | None = None) -> dict: + """Import an archive sent as bytes or a binary file read to its end (chunked unless length is given): as + import_archive, {"roots", "incomplete", ...}.""" + if isinstance(data, (bytes, bytearray, memoryview)) and length is None: + length = len(data) + return self.call("POST", "/api/import/stream", body=data, length=length) + def fetch(self, target: str, output: str, from_node: str | None = None, raw: bool = False) -> dict: """Fetch a file or directory and have the daemon write it to output (a path on this machine, which must not exist), in one request: {"cid", "warnings"} (+ "sealed", "name"). For progress, use submit_job.""" @@ -565,9 +837,26 @@ def pin_rm(self, cid: str): self.post("/api/pin/rm", cid=cid) def pin_ls(self, cid: str | None = None) -> list[dict]: - """Every pin (or cid's): [{"cid", "sources", "complete", "type", "sealed", "name"}].""" + """Every pin (or cid's): [{"cid", "sources", "complete", "type", "sealed", "name", and from a daemon since + 0.1.7 "label", "origin", "size", "entries", "added"}].""" return self.get("/api/pin/ls", cid=cid)["pins"] + def pin_label(self, cid: str, label: str | None) -> dict: + """Label a pin (None or "" clears its label): its pin_ls entry. NotFound if cid isn't pinned.""" + return self.post("/api/pin/label", cid=cid, label=label) + + # The cache + + def cache_ls(self) -> list[dict]: + """The cached fetches (downloads not pinned), most recently used first: [{"cid", "root", "path", "pinned", + "complete", "type", "size", "provided"}].""" + return self.get("/api/cache/ls")["cache"] + + def cache_rm(self, cid: str) -> dict: + """Evict the cached fetches leading to cid now: {"removed_records", "removed_blocks", "freed_bytes"}. + NotFound if none does.""" + return self.post("/api/cache/rm", cid=cid) + def pin_remote(self, node: str, cid: str, pin: bool = True) -> bool: """Ask node (hex) to pin cid for this node, or (pin=False) to release that pin; True if it accepted.""" return self.post("/api/pin/remote", node=node, cid=cid, pin=pin)["accepted"] @@ -637,14 +926,34 @@ def deny_rm(self, entries: list[str]) -> list[str]: """Remove entries from the local deny list (NotFound, removing none, if one isn't on it).""" return self.post("/api/deny/rm", entries=list(entries))["removed"] + # Settings (rnfs.toml; the schema alone is rnfs.config.settings_schema(), without a daemon) + + def config(self) -> dict: + """The daemon's settings: {"path" (its rnfs.toml), "settings": [each setting's schema entry with "value" + (running), "file" (rnfs.toml's, None if it doesn't set it), "overridden" (by the daemon's command line, + which wins), "pending" (a restart would change it)], "file_error" (why a restart would refuse rnfs.toml, + or None), "warnings"}. NoSuchEndpoint from a daemon before 0.1.7.""" + return self.get("/api/config") + + def config_set(self, settings: dict) -> dict: + """Set settings ({name: value, or None to remove it from rnfs.toml}) in the daemon's rnfs.toml, applying at + once those that can be: {"path", "applied", "restart_required", "overridden", "warnings"}. BadRequest, + writing nothing, unless the whole result is valid.""" + return self.post("/api/config/set", settings=dict(settings)) + # Jobs (background transfers, with progress) - def submit_job(self, kind: str, target: str, output: str | None = None, from_node: str | None = None, - **options) -> dict: - """Start a job: kind "get" (output needed), "pin" or "export" (output needed). Its record, with its - "id". options: get's sealed_name and fallback_dht.""" + def submit_job(self, kind: str, target: str | None = None, output: str | None = None, + from_node: str | None = None, path: str | None = None, options: dict | None = None, + **flags) -> dict: + """Start a job: kind "get" (output needed), "pin" or "export" (output needed) of a target, or "add" or + "import" of a path on this machine. Its record, with its "id". flags: get's sealed_name and fallback_dht; + options: an add's, as add_path takes them in the API's form (wrap, only_hash, pin, to, to_self, seal_name, + chunker; only those set).""" output = os.path.abspath(os.path.expanduser(output)) if output is not None else None - return self._post("/api/jobs", kind=kind, target=target, output=output, **{"from": from_node}, **options) + path = os.path.abspath(os.path.expanduser(path)) if path is not None else None + return self._post("/api/jobs", kind=kind, target=target, output=output, path=path, **{"from": from_node}, + **(options or {}), **flags) def job(self, job_id: str, wait: int = 0) -> dict: """A job's record; waits up to wait seconds (the API caps it) for it to finish.""" @@ -661,3 +970,96 @@ def cancel_job(self, job_id: str) -> dict: def clear_jobs(self, job_id: str | None = None) -> int: """Forget finished jobs (or that one); how many.""" return self._post("/api/jobs/clear", id=job_id)["removed"] + + # The features' routes rnfs.Client uses (rnfs/features/: NotFound from a daemon without the feature). A name + # is 32 hex characters or a name contact's nick; an LXMF recipient an address (hex) or an LXMF contact's nick + + def peers(self) -> list[dict]: + """Every node the daemon knows, nearest first: [{"node_id", "mode", "in_table", "seniority", "senior", + "hops", "interface", "bitrate", "link", "rate", "last_heard", "last_active", "banned", "sent", + "received", "requests", "answered", "refused", "failed", "bad", ...}] (peers feature).""" + return self.get("/api/peers")["peers"] + + def stats(self) -> dict: + """Traffic since the daemon started: {"since", "totals", "refused", "interfaces", "upload_budget", + "upload_budget_left"} (peers feature).""" + return self.get("/api/stats") + + def contacts(self) -> list[dict]: + """Every contact, by nick: [{"nick", "kind", "id"}].""" + return self.get("/api/contacts")["contacts"] + + def contact_get(self, nick: str) -> dict: + """One contact: {"nick", "kind", "id"}; NotFound if there is none.""" + return self.get("/api/contacts/get", nick=nick) + + def contact_add(self, nick: str, ident: str, kind: str | None = None) -> dict: + """Add or replace a contact for ident (32 hex characters): kind "name", "node" or "lxmf", or None to have + the daemon tell (ValueError if it can't). The contact.""" + return self._post("/api/contacts/add", nick=nick, id=ident, kind=kind) + + def contact_rm(self, nick: str): + """Delete a contact (NotFound if there is none).""" + self.post("/api/contacts/rm", nick=nick) + + def name_keep(self, name: str) -> dict: + """Keep a name's signed record: {"name", "reasons", "sequence", "cid", "emitted"} (keepers feature).""" + return self.post("/api/name/keep", name=name) + + def name_unkeep(self, name: str): + """Stop keeping a name (NotFound if it isn't kept by the user).""" + self.post("/api/name/unkeep", name=name) + + def name_kept(self) -> list[dict]: + """Every kept name: [{"name", "reasons", "sequence", "cid", "emitted"}].""" + return self.get("/api/name/kept")["kept"] + + def name_follow(self, name: str) -> dict: + """Follow a name, pinning its latest target: {"name", "pinned", "latest", "sequence", "state"} (follow + feature).""" + return self.post("/api/name/follow", name=name) + + def name_unfollow(self, name: str): + """Stop following a name, releasing its pin (NotFound if it isn't followed).""" + self.post("/api/name/unfollow", name=name) + + def name_following(self) -> list[dict]: + """Every followed name: [{"name", "pinned", "latest", "sequence", "state"}].""" + return self.get("/api/name/following")["following"] + + def lxmf_address(self) -> dict: + """{"address", "display_name", "propagation_node", "nearest_propagation_node"}; BadRequest if LXMF is off.""" + return self.get("/api/lxmf/address") + + def lxmf_send(self, to: str, target: str, message: str | None = None, propagate: bool = False, + nearest: bool = False) -> dict: + """Send a link to target (and message) over LXMF, queued: {"id", "hash", "to", "method"}.""" + return self._post("/api/lxmf/send", to=to, target=target, message=message, propagate=propagate or None, + nearest=nearest or None) + + def lxmf_inbox(self) -> list[dict]: + """The inbox, oldest first: [{"id", "hash", "from", "display_name", "sent", "received", "title", "text", + "method", "links", "auto", "trusted", "contact"}].""" + return self.get("/api/lxmf/inbox")["messages"] + + def lxmf_inbox_get(self, entry_id: int, link: int | None = None, output: str | None = None, + directory: str | None = None, use_sent_cid: bool = False) -> list[dict]: + """Fetch inbox message entry_id's links (or link, 1-based) to output or into directory (paths on this + machine): [{"target", "output", "job" | "cid", "warnings"?, "error"?}].""" + output = os.path.abspath(os.path.expanduser(output)) if output is not None else None + directory = os.path.abspath(os.path.expanduser(directory)) if directory is not None else None + return self._post("/api/lxmf/inbox/get", id=entry_id, link=link, output=output, directory=directory, + use_sent_cid=use_sent_cid or None)["results"] + + def lxmf_inbox_rm(self, entry_id: int | None = None, every: bool = False) -> int: + """Remove inbox message entry_id, or every one (every); how many (NotFound for an id there isn't).""" + return self.post("/api/lxmf/inbox/rm", **({"all": True} if every else {"id": entry_id}))["removed"] + + def lxmf_outbox(self) -> list[dict]: + """The messages sent, oldest first: [{"id", "hash", "to", "time", "links", "text", "method", "state", + "progress", ...}].""" + return self.get("/api/lxmf/outbox")["messages"] + + def lxmf_sync(self, nearest: bool = False) -> dict: + """Start downloading waiting messages from the propagation node: {"propagation_node", "started"}.""" + return self._post("/api/lxmf/sync", nearest=nearest or None) diff --git a/rnfs/cli.py b/rnfs/cli.py index ff66e05..cda1c2a 100644 --- a/rnfs/cli.py +++ b/rnfs/cli.py @@ -1,8 +1,9 @@ """rnfs command line (specification section 11). `daemon` runs a node with its API and gateway, holding a lock on /daemon.lock from before it touches -anything there, so a second daemon on the same rnfs dir stops at once; every other command is one request to a -running daemon's local API (see rnfs/api.py). Client commands never import RNS, so they start quickly; paths given to them +anything there, so a second daemon on the same rnfs dir stops at once, and describes itself in +/daemon.json (its PID, ports, version, start time and Reticulum config) while it runs, so clients find +it whatever rnfs.toml holds; every other command is one request to a running daemon's local API (see rnfs/api.py). Client commands never import RNS, so they start quickly; paths given to them are resolved to absolute paths here, since the daemon reads and writes them itself. Client commands talk to the daemon through rnfs.apiclient, the library companion apps use too: it reads the token @@ -22,6 +23,7 @@ from rnfs import __version__, apiclient, fsutil from rnfs.apiclient import CHALLENGE_HEADER, PROOF_HEADER, TOKEN_FILE, api_proof, token_path # noqa: F401 +from rnfs.apiclient import daemon_info_path from rnfs.config import DEFAULT_API_PORT, LOCK_FILE, Config DEFAULT_DIR = apiclient.DEFAULT_DIR @@ -62,6 +64,24 @@ def remove_token(path: str, token: str): pass # already gone, or unreadable: nothing of ours to clean up +def write_daemon_info(path: str, info: dict): + """Write the running daemon's description (apiclient.DAEMON_FILE) to path atomically, owner-only like the + token beside it.""" + fsutil.write_atomic(path, (json.dumps(info, indent=2) + "\n").encode("utf-8"), mode=0o600, private=True) + + +def remove_daemon_info(path: str, pid: int): + """Delete the daemon description at path if it is still this process's (pid): another daemon may have + replaced it.""" + try: + with open(path, encoding="utf-8") as f: + if json.load(f).get("pid") != pid: + return + fsutil.remove(path) + except (OSError, ValueError, AttributeError): + pass # already gone, or not ours to clean up + + def read_token(path: str) -> str: """The daemon's API token from path (apiclient.read_token); exits with an explanation if it can't be used.""" try: @@ -131,9 +151,13 @@ def copy_response(stream: apiclient.Stream, out, progress=None): def client(args) -> ApiClient: - """An ApiClient for --api-port, or the api_port in /rnfs.toml, or the default port; token from .""" + """An ApiClient for --api-port, or the port in the running daemon's /daemon.json, or the api_port in + /rnfs.toml, or the default port; token from .""" port = args.api_port if port is None: + running = apiclient.read_daemon_info(args.dir) + if running is not None: + return ApiClient(running["api_port"], token_path(args.dir)) try: config = apiclient.load_settings(args.dir) except apiclient.ApiError as e: @@ -155,6 +179,17 @@ def port_number(text: str, minimum: int = 1) -> int: return port +def positive_seconds(text: str) -> float: + """A number of seconds above 0 from the command line (argparse type).""" + try: + value = float(text) + except ValueError: + raise argparse.ArgumentTypeError(f"invalid number of seconds {text!r}") from None + if not 0 < value < float("inf"): + raise argparse.ArgumentTypeError(f"invalid number of seconds {text!r}: must be above 0") + return value + + def absolute(path: str) -> str: """path made absolute here, since the daemon may run in another directory.""" return os.path.abspath(os.path.expanduser(path)) @@ -255,6 +290,8 @@ def run_daemon(args): print(f"rnfs: {printable(str(e))}", file=sys.stderr, flush=True) RNS.exit(1) # Reticulum has started: ends with os._exit, so its threads don't keep the process servers = [] + # Set by a signal (below) or POST /api/shutdown (Windows has no clean way to stop a background daemon otherwise) + stopping = threading.Event() def abort(message: str): """Shut down what has started and exit with status 1 and message.""" @@ -267,7 +304,7 @@ def abort(message: str): RNS.exit(1) # ends with os._exit, so the message is printed first try: - servers.append(ApiServer(node, config.api_port)) + servers.append(ApiServer(node, config.api_port, shutdown=stopping.set)) servers.append(GatewayServer(node, config, config.gateway_port)) except (OSError, OverflowError) as e: # OverflowError: a port over 65535, which validate() should have refused abort(f"can't listen on 127.0.0.1: {e}. Is another daemon running? (--api-port, --gateway-port)") @@ -280,6 +317,15 @@ def abort(message: str): abort(f"can't write the API token {token_file}: {e}") for server in servers: server.start() + # After the token and the servers: a client finding it finds a daemon to talk to + info_file = daemon_info_path(config.root) + try: + write_daemon_info(info_file, { + "pid": os.getpid(), "api_port": api.port, "gateway_port": gateway.port, "version": __version__, + "started": time.time(), "rnfs_dir": config.root, "reticulum_config": RNS.Reticulum.configdir, + "shared_instance": bool(node.reticulum.is_connected_to_shared_instance), "node_id": node.node_id.hex()}) + except OSError as e: + abort(f"can't write {info_file}: {e}") print(f"rnfs node {node.node_id.hex()}, API on http://127.0.0.1:{api.port}, " f"gateway on http://127.0.0.1:{gateway.port}", flush=True) debug = debuglog.current() @@ -293,7 +339,6 @@ def abort(message: str): # Replace RNS's handlers, which exit immediately, so the node shuts down first. On Windows, Ctrl+C is # SIGINT and Ctrl+Break (or CTRL_BREAK_EVENT sent to the daemon's process group) is SIGBREAK, whose # default would end the process at once; SIGTERM never comes from another process there - stopping = threading.Event() for name in ("SIGINT", "SIGTERM", "SIGBREAK"): if hasattr(signal, name): signal.signal(getattr(signal, name), lambda *_: stopping.set()) @@ -309,6 +354,7 @@ def status(): for server in servers: server.stop() + remove_daemon_info(info_file, os.getpid()) remove_token(token_file, api.token) node.stop() if debug is not None: @@ -363,8 +409,38 @@ def sealed_output(args, output: str) -> str: return os.path.join(os.path.dirname(output), fsutil.safe_default_name(name, os.path.basename(output))) +def print_added(result: dict): + """Print an add's warnings (on stderr) and CID, as `rnfs add` does (an add job's too).""" + print_warnings(result) + print(printable(result["cid"])) + + +def print_imported(result: dict): + """Print an import's roots, and warn of those it didn't pin, as `rnfs import` does (an import job's too).""" + for root in result["roots"]: + print(printable(root)) + for root in result.get("incomplete") or []: + warn(f"{root} is incomplete in the archive, so it wasn't pinned") + + def run_add(args): - """Add a file, directory or stdin and print its CID.""" + """Add a file, directory or stdin and print its CID; with --background, start an add job and print its ID. + + In the foreground an add stays one request, not a job: a job would wait behind max_jobs transfers (one on slow + nodes), and a local add mustn't queue behind an hour-long LoRa download.""" + if getattr(args, "background", False): + if args.path == "-": + fail("--background adds a path; stdin is added in the foreground") + if args.name: + fail("--name is only for stdin (add -)") + path = absolute(args.path) + if not os.path.lexists(path): + fail(f"no such file or directory: {args.path}") + from rnfs.features import jobs + fields = {"kind": "add", "path": path, **{name: True for name in ("wrap", "only_hash") if getattr(args, name)}, + **add_options(args)} + jobs.run_transfer(args, fields, None) # prints the job's ID (and fails on a daemon without jobs) + return api = client(args) if args.path == "-": if args.wrap and not args.name: @@ -387,8 +463,7 @@ def run_add(args): if not os.path.lexists(path): fail(f"no such file or directory: {args.path}") result = api.post("/api/add", path=path, wrap=args.wrap, only_hash=args.only_hash, **add_options(args)) - print_warnings(result) - print(printable(result["cid"])) + print_added(result) def default_output(target: str) -> str: @@ -516,6 +591,39 @@ def run_pin_ls(args): print(f"{printable(pin['cid'])} {','.join(printable(s) for s in pin['sources'])}{state}{name}") +def run_pin_label(args): + """Label a pin, or clear its label (no label given).""" + pin = client(args).pin_label(args.cid, args.label) + if pin.get("label"): + print(printable(f"{pin['cid']} {pin['label']}")) + else: + print(printable(f"{pin['cid']}: label cleared")) + + +def run_cache_ls(args): + """List the cached fetches: downloads kept until evicted, not pinned.""" + from rnfs.features.peers import human_size # no RNS there + cache = client(args).cache_ls() + if args.json: + print(json.dumps(cache, indent=2)) + return + for entry in cache: + target = "/".join([entry["root"], *entry["path"]]) + state = ("" if entry["complete"] else " (incomplete)") + (" (pinned)" if entry["pinned"] else "") + line = f"{printable(entry['cid'])} {human_size(entry.get('size'))}{state}" + if target != entry["cid"]: + line += f" {printable(target)}" # names from other people's DAGs: escaped + print(line) + + +def run_cache_rm(args): + """Evict cached fetches now, by CID.""" + api = client(args) + for cid in args.cids: + result = api.cache_rm(cid) + print(printable(f"{cid}: removed {result['removed_blocks']} blocks, freed {result['freed_bytes']} bytes")) + + def run_pin_remote(args): """Ask a node to pin or release a CID.""" from rnfs.features.contacts import node_arg # a node contact's nick works as a node ID @@ -644,15 +752,33 @@ def run_export(args): def run_import(args): - """Import an archive, pinning its complete roots.""" + """Import an archive, pinning its complete roots; with --background, as an import job (one request otherwise, + as run_add explains).""" path = absolute(args.file) if not os.path.isfile(path): fail(f"no such file: {args.file}") - result = client(args).post("/api/import", path=path) - for root in result["roots"]: - print(printable(root)) - for root in result.get("incomplete") or []: - warn(f"{root} is incomplete in the archive, so it wasn't pinned") + if getattr(args, "background", False): + from rnfs.features import jobs + jobs.run_transfer(args, {"kind": "import", "path": path}, None) # prints the job's ID + return + print_imported(client(args).post("/api/import", path=path)) + + +def run_stop(args): + """Ask the daemon to stop, and wait (up to --wait seconds) until it has.""" + found = client(args) + # The library's own connection: the CLI's would exit on the error that means the daemon has stopped + api = apiclient.ApiConnection(args.dir, port=found.port, token_file=found.token_file) + try: + stopped = api.shutdown(wait=args.wait) + except apiclient.NoSuchEndpoint: + fail(f"the daemon (rnfs {api.daemon_version or '?'}) can't be stopped through its API; stop its process " + f"instead") + except apiclient.ApiError as e: + fail(str(e)) + if not stopped: + fail(f"the daemon hasn't stopped after {args.wait:g} s; it may still be saving its state") + print("rnfs: the daemon stopped") def run_status(args): @@ -711,6 +837,8 @@ def command(subparsers, name: str, run, help: str) -> argparse.ArgumentParser: add.add_argument("--wrap", action="store_true", help="put it in a directory") add.add_argument("--only-hash", action="store_true", help="just print the CID; store nothing") add.add_argument("--name", help="with stdin: wrap it in a directory as this entry") + add.add_argument("--background", action="store_true", + help="start an add job (a path only), print its ID and return (rnfs jobs)") # --- content: unpinned adds, sealed files, chunkers --- add.add_argument("--no-pin", action="store_true", help="keep it as evictable cache instead of pinning it") add.add_argument("--to", nargs="+", action="extend", metavar="RECIPIENT", @@ -761,6 +889,9 @@ def command(subparsers, name: str, run, help: str) -> argparse.ArgumentParser: pin_ls = command(pin, "ls", run_pin_ls, "list pins, their sources and where each came from") pin_ls.add_argument("cid", nargs="?") pin_ls.add_argument("--json", action="store_true", help="print the full listing as JSON") + pin_label = command(pin, "label", run_pin_label, "give a pin a label (none given: clear it)") + pin_label.add_argument("cid") + pin_label.add_argument("label", nargs="?", default=None) remote = pin.add_parser("remote", help="remote pinning").add_subparsers(dest="action", required=True, metavar="command") for action in ("add", "rm"): @@ -769,6 +900,12 @@ def command(subparsers, name: str, run, help: str) -> argparse.ArgumentParser: sub.add_argument("cid") command(commands, "gc", run_gc, "evict all cached fetches") + cache = commands.add_parser("cache", help="cached fetches (downloads not pinned)").add_subparsers( + dest="cache_command", required=True, metavar="command") + cache_ls = command(cache, "ls", run_cache_ls, "list cached fetches, most recently used first") + cache_ls.add_argument("--json", action="store_true", help="print the full listing as JSON") + cache_rm = command(cache, "rm", run_cache_rm, "evict cached fetches now") + cache_rm.add_argument("cids", nargs="+", metavar="CID") command(commands, "verify", run_verify, "re-check every stored block against its CID") providers = command(commands, "providers", run_providers, "find nodes providing a CID") @@ -800,8 +937,13 @@ def command(subparsers, name: str, run, help: str) -> argparse.ArgumentParser: export.add_argument("--background", action="store_true", help="start a job, print its ID and return (rnfs jobs)") import_ = command(commands, "import", run_import, "import an archive, pinning its roots") import_.add_argument("file") + import_.add_argument("--background", action="store_true", + help="start an import job, print its ID and return (rnfs jobs)") command(commands, "status", run_status, "node ID, known nodes, records held, storage use") + stop = command(commands, "stop", run_stop, "stop the daemon (as Ctrl+C or SIGTERM would) and wait until it has") + stop.add_argument("--wait", type=positive_seconds, default=30.0, metavar="SECONDS", + help="how long to wait for it to stop (default 30)") from rnfs import features for feature in features.feature_classes(): diff --git a/rnfs/config.py b/rnfs/config.py index 01788ef..bddae9b 100644 --- a/rnfs/config.py +++ b/rnfs/config.py @@ -9,7 +9,8 @@ import os import stat import tomllib -from dataclasses import dataclass, field, fields +from dataclasses import MISSING, dataclass, field, fields +from types import MappingProxyType from rnfs import fsutil, ids from rnfs.constants import DAY, GIB, HOUR, KIB, MIB, PROVIDER_TTL @@ -42,8 +43,102 @@ MAX_DURATION = 30 * DAY FLAGS = ("private", "gateway_scripts", "default_bootstrap", "debug_log") IDENTITY_LISTS = ("members", "remote_pin_identities") # RNS identity hashes: 32 hex digits, optionally +# Settings that take one of a few words (Config.validate), core and feature ones alike +CHOICES = {"slow": ("yes", "no", "auto"), "dht_mode": ("server", "client"), "provide_strategy": ("pinned", "roots"), + "keeper_query": ("fallback", "always")} _HEX_ID = ids.HEX16 # the name other modules used before rnfs.ids +# The settings schema (settings_schema): what an app needs to show and edit the settings without a copy of their +# metadata. Types and ranges come from the tables above and the fields' annotations and defaults; what those +# can't say is below. tests/test_settings.py fails for a field none of this describes + +# Groups, as the markers in Config mark them: (group, its first field), in field order; a group runs to the next +SECTIONS = (("core", "reticulum_dir"), ("transfers", "max_jobs"), ("mesh", "upload_budget"), + ("names", "keep_max_names"), ("lxmf", "lxmf_enabled"), ("deny lists", "deny_lists")) +# Feature settings' limits (their features check them): name -> (minimum, maximum or None) +FEATURE_LIMITS = {"max_jobs": (1, None), "max_finished_jobs": (1, None), "upload_budget": (1, None), + "keep_max_names": (0, None), "follow_quota": (0, None), "lxmf_auto_fetch_quota": (0, None), + "lxmf_sync_interval": (0, None)} +# Units of what the tables don't give one (DURATIONS are seconds, RATES bits/s) +UNITS = {"min_rate_sample": "bytes", "max_storage": "bytes", "max_download_size": "bytes", + "remote_pin_quota": "bytes", "follow_quota": "bytes", "lxmf_auto_fetch_quota": "bytes", + "upload_budget": "bytes/hour", "lxmf_sync_interval": "seconds"} +# Settings a running node applies at once when changed through the API (rnfs/features/settings.py, whose LIVE table +# must name exactly these); every other one applies at the next start +LIVE_SETTINGS = frozenset({ + "max_storage", "max_download_size", "max_output_entries", "max_upload_rate", "remote_pin_quota", + "scan_interval", "max_jobs", "max_finished_jobs", "serve_interfaces", "no_serve_interfaces", "quiet_hours", + "keep_max_names", "keeper_query", "follow_quota", "lxmf_accept_from", "lxmf_auto_fetch_from", + "lxmf_auto_fetch_quota", "lxmf_send_from_hint"}) +# One line per setting (specification section 12 has the full notes) +DESCRIPTIONS = { + "reticulum_dir": "Reticulum config directory; unset: RNS's own, the one rnsd uses", + "api_port": "Local API port, on 127.0.0.1", + "gateway_port": "Gateway port, on 127.0.0.1; 0: any free port", + "bootstrap_nodes": "Node IDs (hex) contacted first when joining the DHT", + "default_bootstrap": "Also contact the default bootstrap nodes when joining (never in private mode)", + "verbosity": "Extra logging: 1 per-query detail, 2 RNS debug output", + "debug_log": "Also write every message to /debug.log, readable by its owner only", + "slow": "Whether this node is slow: yes, no, or auto (from its interfaces)", + "slow_bitrate": "Below this, links and nodes are slow", + "min_rate_sample": "Smallest response used for a rate estimate", + "unmeasured_bitrate": "Rate assumed for deadlines before a link has a rate estimate", + "dht_mode": "server (store records and answer queries) or client (only ask)", + "alpha": "Parallel lookup queries; unset: 3, or 1 on slow nodes", + "beta": "Closest nodes that must answer before a lookup is complete", + "republish_interval": "How often provider records are republished; unset: 24 h, or 3 days on slow nodes", + "announce_interval": "How often the node announces itself", + "startup_announce_delay": "Delay of the second announce after startup", + "max_parallel_providers": "Providers fetched from at once; unset: 3, or 1 on slow nodes", + "provider_seniority": "Providers in the routing table this long are tried first", + "response_target": "Transfer time a /blocks response aims for", + "deadline_slack": "Added to response deadlines; minimum age before a response can be displaced", + "stall_timeout": "A response with no progress for this long is cancelled", + "blocks_cooldown": "A link whose response made no progress is refused /blocks this long", + "lookup_timeout": "Longest a DHT lookup runs", + "path_timeout": "Longest to wait for an RNS path", + "max_failures": "Unanswered queries in a row before a node is dropped", + "link_idle_timeout": "Idle links are closed after this long", + "max_incoming_links": "Links other nodes may have open to this one", + "max_blocks_responses": "/blocks responses in progress at once (half of them for slow links); at least 2", + "fast_blocks_responses": "/blocks responses in progress on links that aren't slow, if more than the rest", + "max_upload_rate": "Upload cap of every /blocks response; unset: half the rate estimate on slow links", + "max_records": "Provider records stored for others", + "ban_time": "How long a misbehaving node is ignored", + "max_storage": "Storage used before cached fetches are evicted (pins are never evicted)", + "max_download_size": "Largest target fetched; unset: no limit", + "max_output_entries": "Most entries a fetch, listing or write-out may meet", + "name_table_size": "Records kept for names this node hasn't resolved itself", + "remote_pin_quota": "Storage the remote pins of one identity may use", + "remote_pin_identities": "Identity hashes (hex) allowed to ask this node to pin (/pin)", + "members": "Private mode's member identity hashes (hex)", + "private": "Private mode: talk to members only", + "scan_interval": "How often the shared folder is scanned", + "provide_strategy": "roots (pins and complete cached downloads) or pinned (pins only) are provided", + "gateway_scripts": "Let gateway pages run scripts", + "gateway_fetches": "Gateway network fetches at a time for requests that aren't the user's own navigation", + "max_jobs": "Background jobs running at once; unset: 2, or 1 on slow nodes", + "max_finished_jobs": "Finished jobs kept for rnfs jobs (each at least 60 s)", + "upload_budget": "Bytes per hour of /blocks responses, node-wide; unset: unlimited", + "serve_interfaces": "Serve /blocks only on these RNS interfaces (names or prefixes); empty: all", + "no_serve_interfaces": "Never serve /blocks on these RNS interfaces (names or prefixes)", + "quiet_hours": "Local times when /blocks is refused, e.g. 22:00-06:00; empty: none", + "keep_max_names": "Names kept (keep and follow) for others; this node's own names don't count", + "keeper_query": "fallback: ask keepers when a path request heard nothing; always: on every network resolve", + "follow_quota": "Bytes a followed name's new target may hold; 0: followed names aren't fetched", + "lxmf_enabled": "LXMF on or off; unset: on if the LXMF package is installed", + "lxmf_display_name": "Name announced with the LXMF address; unset: rnfs node", + "lxmf_propagation_node": "LXMF propagation node (hex) for propagated messages and downloads; unset: none", + "lxmf_send_from_hint": "Sent links carry this node's ID as a provider hint (linking the LXMF address to it)", + "lxmf_sync_interval": "How often messages are downloaded from the propagation node; 0: never", + "lxmf_accept_from": "LXMF addresses (hex) whose messages are kept; empty: everyone", + "lxmf_auto_fetch_from": "LXMF addresses (hex) whose links are fetched and pinned", + "lxmf_auto_fetch_quota": "Storage the pins of one auto-fetch sender may use", + "deny_lists": "Deny-list sources: [/path] or name/[/path]", +} +_TYPE_NAMES = {int: "int", float: "float", bool: "bool", str: "string", list[str]: "list[string]"} +MAX_TOML_INT = 2 ** 63 - 1 # TOML integers are 64-bit signed + def _is_number(value) -> bool: """True if value is a finite int or float; not a bool, which Python counts as an int.""" @@ -55,10 +150,15 @@ class Config: """Settings for an RnfsNode. warnings: what Config.load changed to trust the settings, for the caller to show (a plain attribute, not a - field, so rnfs.toml can't set it). + field, so rnfs.toml can't set it). overrides and given_defaults: what Config.load was given over and under + rnfs.toml (the daemon's command line, an embedded node's settings and defaults), so the settings feature can + tell what a restart would run with. """ - warnings = () # unannotated, so not a dataclass field; Config.load sets a list + # Unannotated, so not dataclass fields; Config.load sets them (never changed in place: a copy is shared) + warnings = () + overrides = MappingProxyType({}) + given_defaults = MappingProxyType({}) rnfs_dir: str = "~/.rnfs/" # identity, blocks, pins, names and shared folder live here # Reticulum config directory. None uses RNS's own default (/etc/reticulum, ~/.config/reticulum or @@ -219,12 +319,9 @@ def effective_max_upload_rate(self, slow_link: bool, rate_estimate: float | None def validate(self): """Raise ValueError for settings the spec forbids, or of the wrong type or out of range.""" self._validate_core_types() - if self.slow not in ("yes", "no", "auto"): - raise ValueError("slow must be yes, no or auto") - if self.dht_mode not in ("server", "client"): - raise ValueError("dht_mode must be server or client") - if self.provide_strategy not in ("pinned", "roots"): - raise ValueError("provide_strategy must be pinned or roots") + for name, choices in CHOICES.items(): + if getattr(self, name) not in choices: + raise ValueError(f"{name} must be {', '.join(choices[:-1])} or {choices[-1]}") # transfers # type() rather than isinstance(): true is an int to Python, but not a count anyone means if self.max_jobs is not None and (type(self.max_jobs) is not int or self.max_jobs < 1): @@ -238,8 +335,6 @@ def validate(self): # type(): bool is an int subclass, and keep_max_names = true must not mean 1 if type(self.keep_max_names) is not int or self.keep_max_names < 0: raise ValueError("keep_max_names must be a whole number, not negative") - if self.keeper_query not in ("fallback", "always"): - raise ValueError("keeper_query must be fallback or always") if type(self.follow_quota) is not int or self.follow_quota < 0: raise ValueError("follow_quota must be a whole number of bytes, not negative") # lxmf @@ -251,6 +346,9 @@ def validate(self): for slow_node in (False, True): if self.effective_republish_interval(slow_node) >= PROVIDER_TTL / 2: raise ValueError("republish_interval must be less than PROVIDER_TTL / 2") + # Both are bound on 127.0.0.1 at start, so one port for both stops the daemon then + if self.gateway_port != 0 and self.api_port == self.gateway_port: + raise ValueError("api_port and gateway_port must differ") def _validate_core_types(self): """Raise ValueError for a core setting of the wrong type or out of range. @@ -303,22 +401,38 @@ def load(cls, rnfs_dir: str = "~/.rnfs/", defaults: dict | None = None, **overri load. Raises PermissionError if either file can't be trusted, ValueError for settings that aren't valid (also a settings file over MAX_SETTINGS_SIZE), OSError if the file can't be read. """ + path = settings_path(rnfs_dir) + data, warnings = read_settings_file(path) + config = cls.combine(rnfs_dir, parse_settings(data, path), defaults, overrides, path) + warnings.extend(config.check_reticulum_config()) + config.warnings = warnings + return config + + def check_reticulum_config(self) -> list[str]: + """Check RNS's `config` when reticulum_dir is the rnfs dir (earlier versions' layout, where RNS reads its + interfaces, and the commands they run, from there) as open_checked does: the warnings of making it + private. Raises PermissionError if it can't be trusted, OSError if it can't be opened; nothing to check + for another reticulum_dir.""" + root = self.root + reticulum_root = self.reticulum_root + if reticulum_root is None or os.path.realpath(reticulum_root) != os.path.realpath(root): + return [] + opened = open_checked(os.path.join(root, "config"), "the Reticulum config") + if opened is None: + return [] + fd, warning = opened + os.close(fd) + return [f"{warning}. Check that its interfaces are yours"] if warning is not None else [] + + @classmethod + def combine(cls, rnfs_dir: str, file_values: dict, defaults: dict | None = None, overrides: dict | None = None, + path: str = SETTINGS_FILE) -> "Config": + """The settings of rnfs_dir from rnfs.toml's values (file_values, as parse_settings gives them, read from + path), over defaults and under overrides (None values ignored), validated as Config.load does, with + overrides and given_defaults recorded. Raises ValueError (TypeError for an override that isn't a field). + """ values: dict = dict(defaults or {}) - warnings: list[str] = [] - root = os.path.expanduser(rnfs_dir) - path = os.path.join(root, SETTINGS_FILE) - # Its settings choose the Reticulum config, and with it programs rnfs starts, and the API port the CLI - # sends its token to: whoever could change it could choose them - opened = open_checked(path, "the settings file") - if opened is not None: - fd, warning = opened - if warning is not None: - warnings.append(f"{warning}. Check that its settings are yours") - with os.fdopen(fd, "rb") as f: - data = f.read(MAX_SETTINGS_SIZE + 1) - if len(data) > MAX_SETTINGS_SIZE: - raise ValueError(f"{path} is larger than {MAX_SETTINGS_SIZE // MIB} MiB") - values.update(tomllib.loads(data.decode("utf-8"))) + values.update(file_values) known = {f.name for f in fields(cls)} unknown = values.keys() - known if unknown: @@ -326,23 +440,277 @@ def load(cls, rnfs_dir: str = "~/.rnfs/", defaults: dict | None = None, **overri if "rnfs_dir" in values: # The file lives in the rnfs dir, so it can't move it; --dir does raise ValueError(f"rnfs_dir can't be set in {path}; use --dir") - values.update({k: v for k, v in overrides.items() if v is not None}) + given = {k: v for k, v in (overrides or {}).items() if v is not None} + values.update(given) values["rnfs_dir"] = rnfs_dir config = cls(**values) config.validate() - reticulum_root = config.reticulum_root - if reticulum_root is not None and os.path.realpath(reticulum_root) == os.path.realpath(root): - # Earlier versions' layout: RNS reads its config (interfaces, and their commands) from the rnfs dir - opened = open_checked(os.path.join(root, "config"), "the Reticulum config") - if opened is not None: - fd, warning = opened - os.close(fd) - if warning is not None: - warnings.append(f"{warning}. Check that its interfaces are yours") - config.warnings = warnings + config.overrides = MappingProxyType(given) + config.given_defaults = MappingProxyType(dict(defaults or {})) return config +def settings_path(rnfs_dir: str) -> str: + """Where the settings of rnfs_dir are: /rnfs.toml.""" + return os.path.join(os.path.expanduser(rnfs_dir), SETTINGS_FILE) + + +def read_settings_file(path: str) -> tuple[bytes | None, list[str]]: + """rnfs.toml's bytes at path (None if there is none) and the warnings of making it private, once checked as + open_checked checks it. Raises PermissionError if it can't be trusted, ValueError if it is larger than + MAX_SETTINGS_SIZE, OSError if it can't be read.""" + # Its settings choose the Reticulum config, and with it programs rnfs starts, and the API port the CLI + # sends its token to: whoever could change it could choose them + opened = open_checked(path, "the settings file") + if opened is None: + return None, [] + fd, warning = opened + warnings = [f"{warning}. Check that its settings are yours"] if warning is not None else [] + with os.fdopen(fd, "rb") as f: + data = f.read(MAX_SETTINGS_SIZE + 1) + if len(data) > MAX_SETTINGS_SIZE: + raise ValueError(f"{path} is larger than {MAX_SETTINGS_SIZE // MIB} MiB") + return data, warnings + + +def parse_settings(data: bytes | None, path: str = SETTINGS_FILE) -> dict: + """The values rnfs.toml's bytes data set ({} for None), not yet checked. Raises ValueError (naming path) if + they aren't UTF-8 TOML.""" + if data is None: + return {} + try: + return tomllib.loads(data.decode("utf-8")) + except ValueError as e: # UnicodeDecodeError and tomllib.TOMLDecodeError are ValueErrors + raise ValueError(f"{path} isn't valid TOML: {e}") from None + + +# The schema + + +def settings_schema() -> list[dict]: + """Every setting rnfs.toml may hold (rnfs_dir, which it can't, is left out), in Config's order, each a new + JSON-ready dict: {"name", "group" (SECTIONS), "type" ("int", "float", "bool", "string", "list[string]" or + "choice"), "choices" (a list for "choice", else None), "nullable" (whether it may be unset, taking the default + its description says), "default", "unit" ("seconds", "bytes", "bits/s", "bytes/hour" or None), "minimum", + "maximum" (None for no bound), "exclusive_minimum" (true: the value must be above minimum), "description", + "live" (a change through the API applies without a restart)}. Standard library only, so apps and the CLI can + use it without a daemon.""" + starts = dict((first, group) for group, first in SECTIONS) + group = "core" + schema = [] + for f in fields(Config): + group = starts.get(f.name, group) + if f.name == "rnfs_dir": + continue + schema.append(_schema_entry(f, group)) + return schema + + +def _schema_entry(f, group: str) -> dict: + """The schema entry of dataclass field f of Config, in group (see settings_schema).""" + name = f.name + nullable = type(None) in getattr(f.type, "__args__", ()) + base = next((t for t in getattr(f.type, "__args__", ()) if t is not type(None)), f.type) if nullable else f.type + kind = "choice" if name in CHOICES else _TYPE_NAMES[base] + minimum = maximum = None + exclusive = False + if name in WHOLE_NUMBERS: + minimum, maximum = WHOLE_NUMBERS[name] + elif name in OPTIONAL_WHOLE_NUMBERS: + minimum = OPTIONAL_WHOLE_NUMBERS[name] + elif name in RATES: + minimum, exclusive = 0, True + elif name in DURATIONS: + minimum, maximum, exclusive = 0, MAX_DURATION, DURATIONS[name] + elif name in FEATURE_LIMITS: + minimum, maximum = FEATURE_LIMITS[name] + unit = "seconds" if name in DURATIONS else "bits/s" if name in RATES else UNITS.get(name) + default = f.default_factory() if f.default is MISSING else f.default + return {"name": name, "group": group, "type": kind, "choices": list(CHOICES[name]) if name in CHOICES else None, + "nullable": nullable, "default": default, "unit": unit, "minimum": minimum, "maximum": maximum, + "exclusive_minimum": exclusive, "description": DESCRIPTIONS[name], "live": name in LIVE_SETTINGS} + + +# Editing rnfs.toml + + +def toml_value(value) -> str: + """value written as a TOML value: a bool, a whole number (64-bit), a finite float, a string, or a list of + these. Raises ValueError for anything else, which rnfs.toml can't hold.""" + if type(value) is bool: + return "true" if value else "false" + if type(value) is int: + if not -MAX_TOML_INT - 1 <= value <= MAX_TOML_INT: + raise ValueError(f"{value} is too large for rnfs.toml (64-bit whole numbers)") + return str(value) + if type(value) is float: + if not math.isfinite(value): + raise ValueError("rnfs.toml can't hold inf or nan") + return repr(value) # "0.5", "1e+16": both TOML floats + if type(value) is str: + return _toml_string(value) + if type(value) is list: + return "[" + ", ".join(toml_value(item) for item in value) + "]" + raise ValueError(f"rnfs.toml can't hold a {type(value).__name__} value") + + +def _toml_string(text: str) -> str: + """text as a TOML basic string: quote, backslash and control characters escaped, and the characters other + tools take for line breaks (NEL, U+2028, U+2029: TOML allows them raw, but an editor may split the line + there). ValueError for text that isn't Unicode text (a lone surrogate, which JSON can carry).""" + try: + text.encode("utf-8") + except UnicodeEncodeError: + raise ValueError("a setting's text must be valid Unicode") from None + escapes = {'"': '\\"', "\\": "\\\\", "\b": "\\b", "\t": "\\t", "\n": "\\n", "\f": "\\f", "\r": "\\r"} + out = [] + for c in text: + if c in escapes: + out.append(escapes[c]) + elif ord(c) < 0x20 or ord(c) == 0x7F or c in "\x85

": + out.append(f"\\u{ord(c):04x}") + else: + out.append(c) + return '"' + "".join(out) + '"' + + +def edit_settings(text: str, changes: dict) -> str: + """rnfs.toml's text with each setting in changes (name -> value, or None to remove it) changed, and every + other line kept: comments, blank lines and the other settings as they were written. + + text must be flat TOML (settings only, no tables), as Config.load accepts. A setting already there has its + assignment replaced where it is, all the lines its value spans (a multi-line array), keeping a comment that + ends its last line; one that isn't is added at the end; None removes the assignment. Every statement setting + a name is replaced (a dotted key, `name.x = 1`, or an inline table too): the first by the new assignment, the + others removed. Comments inside a replaced multi-line value go with it. Lines end at "\\n" only, as TOML's + do (NEL, U+2028 and U+2029 may be in its strings and comments). Linear in text's size: each statement's + extent is found by _statement_extent and parsed once. Raises ValueError if text isn't flat TOML or a value + can't be written (toml_value). + """ + tomllib.loads(text) # TOMLDecodeError (a ValueError) here: every statement below is complete + first_line = text.split("\n", 1)[0] + newline = "\r\n" if first_line.endswith("\r") and len(first_line) < len(text) else "\n" + out: list[str] = [] + seen: set[str] = set() + position = 0 + while position < len(text): + line_break = text.find("\n", position) + line_end = len(text) if line_break < 0 else line_break + 1 + line = text[position:line_end] + stripped = line.strip(" \t\r\n") + if not stripped or stripped.startswith("#"): + out.append(line) + position = line_end + continue + where = f"line {text.count(chr(10), 0, position) + 1} of rnfs.toml" + if stripped.startswith("["): + raise ValueError(f"{where} starts a table, which rnfs settings never use; edit it by hand") + value_end, end = _statement_extent(text, position) + try: + parsed = tomllib.loads(text[position:end]) + except tomllib.TOMLDecodeError: + raise ValueError(f"{where} isn't one setting; edit it by hand") from None + if len(parsed) != 1: + raise ValueError(f"{where} isn't one setting; edit it by hand") + name = next(iter(parsed)) + if name in changes: + if changes[name] is not None and name not in seen: + indent = line[:len(line) - len(line.lstrip(" \t"))] + comment = _trailing_comment(text, position, value_end, end, parsed) + out.append(f"{indent}{name} = {toml_value(changes[name])}{comment}{newline}") + seen.add(name) + else: + out.append(text[position:end]) + position = end + if out and not out[-1].endswith("\n"): + out[-1] += newline + for name, value in changes.items(): + if name not in seen and value is not None: + out.append(f"{name} = {toml_value(value)}{newline}") + return "".join(out) + + +def _statement_extent(text: str, start: int) -> tuple[int, int]: + """The statement (key = value) starting at text[start]: (the index after the last character of its value, + before any spaces and comment that end its line; the index after the line break ending it, or len(text)). + + A small lexer, linear in the statement's length: it skips TOML's strings (basic with escapes, literal, and + their multi-line forms, whose closing delimiter may take up to two more quotes) and counts brackets and + braces, so a line break or # ends the statement only outside them. Only an extent: the caller parses it.""" + depth = 0 + last = start # after the last character that isn't a space, a line break or a comment + i, n = start, len(text) + while i < n: + c = text[i] + if c in "\"'": + i = _string_end(text, i) + last = i + continue + if c == "#": # a comment, to the end of its line + line_break = text.find("\n", i) + if line_break < 0: + return last, n + if depth == 0: + return last, line_break + 1 + i = line_break + continue + if c == "\n": + if depth == 0: + return last, i + 1 + elif c in "[{": + depth += 1 + elif c in "]}": + depth -= 1 + if c not in " \t\r\n": + last = i + 1 + i += 1 + return last, n + + +def _string_end(text: str, start: int) -> int: + """The index after the TOML string starting at text[start] (a quote), or where a single-line one meets a line + break or text ends (not TOML: the caller's parse refuses it).""" + quote = text[start] + escapes = quote == '"' # literal strings have none + if text.startswith(quote * 3, start): + i = start + 3 + while i < len(text): + if escapes and text[i] == "\\": + i += 2 + elif text.startswith(quote * 3, i): + i += 3 + for _ in range(2): # """a"""" is a" and a closing """ + if i < len(text) and text[i] == quote: + i += 1 + return i + else: + i += 1 + return len(text) + i = start + 1 + while i < len(text) and text[i] != "\n": + if escapes and text[i] == "\\": + i += 2 + elif text[i] == quote: + return i + 1 + else: + i += 1 + return min(i, len(text)) + + +def _trailing_comment(text: str, start: int, value_end: int, end: int, parsed: dict) -> str: + """The comment ending the statement text[start:end] (which parses as parsed, its value ending at value_end), + with the spaces before it, or "". Checked: the statement without it parses the same.""" + tail = text[value_end:end].rstrip("\r\n") + if not tail.lstrip(" \t").startswith("#"): + return "" + try: + if tomllib.loads(text[start:value_end]) != parsed: + return "" + except tomllib.TOMLDecodeError: + return "" + return tail + + def open_checked(path: str, what: str) -> tuple[int, str | None] | None: """Open the file at path for reading, once checked through its descriptor: (the descriptor, a warning if it was made private) or None if nothing is at path. what (e.g. "the settings file") names it in messages. diff --git a/rnfs/content/archive.py b/rnfs/content/archive.py index 710bc05..458d744 100644 --- a/rnfs/content/archive.py +++ b/rnfs/content/archive.py @@ -5,7 +5,7 @@ every root's DAG, depth-first from each root, each block once. CIDs are binary. """ import struct -from typing import BinaryIO, Callable +from typing import BinaryIO, Callable, Iterator from RNS.vendor import umsgpack @@ -32,14 +32,21 @@ def export_archive(roots: list, get: walker.Get, out: BinaryIO, max_entries: int if max_entries is not None: for expect in expects: walker.count_entries(expect, get, max_entries) - out.write(umsgpack.packb([ARCHIVE_MAGIC, ARCHIVE_VERSION, [expect.cid.binary for expect in expects]])) - written = 0 - for block_expect, block in walker.iter_dags(expects, get): # each block once over every root - out.write(umsgpack.packb([block_expect.cid.binary, block])) + written = -1 # the header isn't a block + for data in archive_chunks(expects, get): + out.write(data) written += 1 return {"roots": [expect.cid for expect in expects], "blocks": written} +def archive_chunks(expects: list[walker.Expect], get: walker.Get) -> Iterator[bytes]: + """The bytes of the archive of expects (at most MAX_ROOTS, checked by the caller) as export_archive writes them: + the header, then one record per block, each block once over every root (GET /api/export/stream).""" + yield umsgpack.packb([ARCHIVE_MAGIC, ARCHIVE_VERSION, [expect.cid.binary for expect in expects]]) + for block_expect, block in walker.iter_dags(expects, get): # each block once over every root + yield umsgpack.packb([block_expect.cid.binary, block]) + + def archive_size(roots: list[Cid], blocks) -> int: """The exact bytes export_archive writes for roots whose DAGs hold blocks, (CID, block length) pairs each listed once (the gateway's ?format=archive sends it as Content-Length before writing).""" diff --git a/rnfs/content/builder.py b/rnfs/content/builder.py index 0895c11..9ef067c 100644 --- a/rnfs/content/builder.py +++ b/rnfs/content/builder.py @@ -128,11 +128,15 @@ class BuildOptions: chunker: the name of the chunker cutting files (chunker.FIXED, the default, or FASTCDC); it decides their CIDs. check: called with each file's content SHA-256 once it is read, before the file's root node is stored; it may raise to stop the build (the deny list's file hashes). None checks nothing, and hashes nothing for it. + progress: called with the length of each read from a file of the tree being built (build_open: a path's files, + not a stream's), and with 0 for each directory entry met, so a walk over empty files reports too; it may raise + to stop the build (an add job's cancel). None reports nothing. Raises ValueError for an unknown chunker. """ chunker: str = chunker.FIXED check: Callable[[bytes], None] | None = None + progress: Callable[[int], None] | None = None def __post_init__(self): """Check the chunker's name (None is the default).""" @@ -532,7 +536,27 @@ def build_open(handle: Handle, st: os.stat_result, path: str, put: Put, warnings return _build_cached(handle, put, kind, options, cache) # Reading a regular file never blocks, so O_NONBLOCK can stay; the caller keeps the descriptor with os.fdopen(handle, "rb", closefd=False) as f: - return build_file(f, put, kind=kind, options=options) + return build_file(counted(f, options.progress), put, kind=kind, options=options) + + +class _Counted: + """A binary reader passing the length of each read on to progress (BuildOptions.progress).""" + + def __init__(self, stream: BinaryIO, progress: Callable[[int], None]): + """Read from stream, reporting to progress.""" + self.stream = stream + self.progress = progress + + def read(self, size: int = -1) -> bytes: + """Up to size bytes of the stream, reported once read (progress may raise to stop the build).""" + data = self.stream.read(size) + self.progress(len(data)) + return data + + +def counted(stream: BinaryIO, progress: Callable[[int], None] | None) -> BinaryIO: + """stream, reporting the length of each read to progress if it is given (BuildOptions.progress).""" + return stream if progress is None else _Counted(stream, progress) def _build_cached(handle: Handle, put: Put, kind: int, options: BuildOptions, cache: FileCache) -> Built: @@ -565,6 +589,8 @@ def _build_tree(directory: Handle, path: str, put: Put, warnings: list[str], dep child_height = 0 children = list_directory(directory) for item in children: + if options.progress is not None: + options.progress(0) # a cancel check even where no file is read (a tree of empty directories) child_path = os.path.join(path, item.name) shown = display_path(child_path) if not dag.is_valid_name(item.name): @@ -598,6 +624,46 @@ def _build_tree(directory: Handle, path: str, put: Put, warnings: list[str], dep return build_directory(entries, put, child_height) +def tree_size(path: str, check: Callable[[], None] | None = None) -> int: + """The bytes add_path would read at path: a regular file's size (path followed if a symlink, as the add follows + it), or the sizes of the regular files below a directory that _build_tree reads, which skips symlinks, other + types and invalid names. An add job's total_bytes, measured before it reads anything: files changing meanwhile + make it an estimate. Unreadable directories count nothing (the add reports them); check(), if given, is called + for each entry and may raise (a job's cancel).""" + st = os.stat(path) + if stat.S_ISREG(st.st_mode): + return st.st_size + if not stat.S_ISDIR(st.st_mode): + return 0 + total = 0 + stack = [(path, 1)] + while stack: + directory, depth = stack.pop() + if depth > constants.MAX_DEPTH: + continue # the add refuses it (TooLarge) when it gets there + try: + listing = os.scandir(directory) + except OSError: + continue + with listing: + for item in listing: + if check is not None: + check() + if not dag.is_valid_name(item.name): + continue + try: + st = item.stat(follow_symlinks=False) + except OSError: + continue + if is_link(st): + continue + if stat.S_ISREG(st.st_mode): + total += st.st_size + elif stat.S_ISDIR(st.st_mode): + stack.append((item.path, depth + 1)) + return total + + def hard_link_reason(links: int) -> str: """Why a file with links other hard links is skipped (shared folder).""" return (f"it has {links} other hard link{'s' if links > 1 else ''}, so it is also a file elsewhere that may " diff --git a/rnfs/content/storage.py b/rnfs/content/storage.py index 791a682..a133051 100644 --- a/rnfs/content/storage.py +++ b/rnfs/content/storage.py @@ -23,9 +23,11 @@ (or counting) under the lock, where a DAG memoised as complete is taken whole rather than walked again. """ import itertools +import math import os import shutil import threading +import time from collections import OrderedDict from concurrent.futures import ThreadPoolExecutor from contextlib import contextmanager @@ -277,7 +279,16 @@ def __init__(self, blocks_dir: str, state_dir: str, max_storage: int): self._sizes: dict[Cid, int] = dict(self.blocks.scan()) self._used = sum(self._sizes.values()) self._pins: dict[Cid, set[str]] = {} + # When each pinned CID was first pinned by any source (Unix time; pin_times); one pinned before rnfs 0.1.7 + # has none + self._added: dict[Cid, float] = {} self._pinned: frozenset[Cid] | None = None # pinned_cids(), built when asked; None again once pins change + # Called as pin_listener(change, cids, source) once pins of source were added ("added") or released + # ("removed") and saved, and as cache_listener(change, cids) with the targets of cache records added + # ("added"), dropped ("removed") or retargeted (both), each with the lock held: the node's "pin" and "cache" + # events. Must only queue (rnfs.events) + self.pin_listener = None + self.cache_listener = None self._cache: OrderedDict[CacheKey, Cid] = OrderedDict() # oldest first self._sessions: list[Session] = [] self._links: dict[Cid, tuple[Cid, ...]] = {} # children of stored DAG nodes; CIDs never change content @@ -320,6 +331,18 @@ def _load(self): bad_pins.append((binary, source)) continue self._pins.setdefault(cid, set()).add(source) + stale_times = [] + for binary, added in self._db.query("SELECT cid, added FROM pin_times"): + try: + cid = Cid.from_bytes(binary) + except (ValueError, TypeError): + cid = None + # A time without its pin is deleted: they are written in one transaction, so only a pin row refused + # above (or a database edited by hand) leaves one + if cid in self._pins and isinstance(added, (int, float)) and math.isfinite(added): + self._added[cid] = float(added) + else: + stale_times.append((binary,)) bad_cache = [] for root, path, target, used in self._db.query("SELECT root, path, target, used FROM cache ORDER BY used"): try: @@ -333,17 +356,22 @@ def _load(self): capped = [_cache_row(key) for key in list(self._cache)[:max(0, len(self._cache) - MAX_CACHE_RECORDS)]] self._cap_cache() self._cache_dirty.clear() - if bad_pins or bad_cache or capped: + if bad_pins or bad_cache or capped or stale_times: with self._db.transaction() as tx: tx.executemany("DELETE FROM pins WHERE cid = ? AND source = ?", bad_pins) tx.executemany("DELETE FROM cache WHERE root = ? AND path = ?", bad_cache + capped) + tx.executemany("DELETE FROM pin_times WHERE cid = ?", stale_times) - def _save_pins(self, source: str, added=(), removed=()): - """Persist source's new pins of added and releases of removed, durably (called with the lock held).""" + def _save_pins(self, source: str, added=(), removed=(), first=(), at: float = 0.0, gone=()): + """Persist source's new pins of added and releases of removed, durably, with the CIDs first pinned at + all now (first, at time at) and no longer pinned at all (gone) (called with the lock held).""" self.blocks.sync() # so pins saved never outlast the blocks they keep with self._db.transaction(durable=True) as tx: tx.executemany("INSERT OR IGNORE INTO pins (cid, source) VALUES (?, ?)", [(cid.binary, source) for cid in added]) + tx.executemany("INSERT OR IGNORE INTO pin_times (cid, added) VALUES (?, ?)", + [(cid.binary, at) for cid in first]) + tx.executemany("DELETE FROM pin_times WHERE cid = ?", [(cid.binary,) for cid in gone]) tx.executemany("DELETE FROM pins WHERE cid = ? AND source = ?", [(cid.binary, source) for cid in removed]) def save(self): @@ -473,8 +501,10 @@ def pin_many(self, cids, source: str): return self._changes += 1 self._pinned = None + now = time.time() + first = [cid for cid in added if self._pins[cid] == {source}] # pinned by nothing else before try: - self._save_pins(source, added=added) + self._save_pins(source, added=added, first=first, at=now) except BaseException: # Not saved (a full or failing disk): not pinned, so memory says what the database does for cid in added: @@ -484,6 +514,9 @@ def pin_many(self, cids, source: str): del self._pins[cid] self._pinned = None raise + for cid in first: # a pin another source held already keeps its first time (or its lack of one) + self._added[cid] = now + self._tell(self.pin_listener, "added", added, source) def unpin(self, cid: Cid, source: str) -> bool: """Release source's hold on cid; True if it held it. Blocks go at the next evict() or gc().""" @@ -497,14 +530,33 @@ def unpin(self, cid: Cid, source: str) -> bool: del self._pins[cid] self._maybe_garbage = True self._pinned = None + gone = [cid] if cid not in self._pins else [] try: - self._save_pins(source, removed=[cid]) + self._save_pins(source, removed=[cid], gone=gone) except BaseException: self._pins.setdefault(cid, set()).add(source) # still pinned, as the database says self._pinned = None raise + if gone: + self._added.pop(cid, None) + self._tell(self.pin_listener, "removed", [cid], source) return True + def _tell(self, listener, *args): + """Call a pin or cache listener, if set (lock held); one that fails is logged, never raised: the change + it reports is made and saved already.""" + if listener is None: + return + try: + listener(*args) + except Exception as e: # noqa: BLE001 - see above + log(f"storage: a change listener failed: {e!r}", LOG_ERROR) + + def pin_added(self, cid: Cid) -> float | None: + """When cid was first pinned (Unix time; None if not pinned, or pinned before rnfs 0.1.7 recorded it).""" + with self._lock: + return self._added.get(cid) if cid in self._pins else None + def pins(self) -> dict[Cid, set[str]]: """Every pin and its sources (a copy).""" with self._lock: @@ -563,6 +615,10 @@ def _set_record(self, key: CacheKey, target: Cid): self._cache[key] = target self._cache.move_to_end(key) self._cache_dirty.add(key) + if previous != target: + if previous is not None: # a retargeted record (rare: a path whose name now leads elsewhere) + self._tell(self.cache_listener, "removed", [previous]) + self._tell(self.cache_listener, "added", [target]) def _drop_records(self, keys) -> list[Cid]: """Remove the records of keys, with the bookkeeping every removal needs: the save deletes their rows @@ -575,6 +631,9 @@ def _drop_records(self, keys) -> list[Cid]: if targets: self._maybe_garbage = True self._cache_generation += 1 + # One per target of the records dropped, as cache_ls lists records: another record may still lead to + # the same target, which a client reloading the list then sees + self._tell(self.cache_listener, "removed", list(dict.fromkeys(targets))) return targets def _cap_cache(self) -> set[Cid]: @@ -1002,6 +1061,41 @@ def act(before: dict[Cid, frozenset[Cid]]) -> set[Cid]: stats["incomplete"] = self._newly_incomplete(targets, act) return stats + def uncache(self, cids: set[Cid]) -> dict: + """Drop every cached fetch leading to one of cids (cache rm), then delete every block nothing reaches, as + gc() does for every fetch. A pin of the same content keeps its blocks. + + Returns {"uncached": set[Cid] targets no longer cached, "removed_records", "removed_blocks", + "freed_bytes", "incomplete": set[Cid] no longer complete}. + """ + with self._lock: + records = {key: target for key, target in self._cache.items() if target in cids} + targets = self._targets() + if not records: + return {"uncached": set(), "removed_records": 0, "removed_blocks": 0, "freed_bytes": 0, + "incomplete": set()} + stats = {} + + def act(before: dict[Cid, frozenset[Cid]]) -> set[Cid]: + """Drop the records still leading to one of cids and collect; the blocks deleted.""" + with self._lock: + rest = dict(self._cache) + roots = self._roots() + # What stays is walked without the lock first, as in gc(), so the collection under it reads little + self._warm(roots | {target for target in rest.values() if target not in cids}, + [key for key, target in rest.items() if key[1] and target not in cids]) + with self._lock: + self._add_memoised(before) + dropped = [key for key, target in self._cache.items() if target in cids] + uncached = set(self._drop_records(dropped)) - set(self._cache.values()) + deleted, freed = self._collect() if dropped else (set(), 0) + stats.update(uncached=uncached, removed_records=len(dropped), removed_blocks=len(deleted), + freed_bytes=freed) + return deleted + + stats["incomplete"] = self._newly_incomplete(targets, act) + return stats + def is_complete(self, cid: Cid) -> bool: """True if every block of cid's DAG is stored and the DAG is valid.""" try: diff --git a/rnfs/content/tarstream.py b/rnfs/content/tarstream.py new file mode 100644 index 0000000..5a22fa6 --- /dev/null +++ b/rnfs/content/tarstream.py @@ -0,0 +1,436 @@ +"""Directories as tar streams, for a client whose files the daemon can't open (a sandboxed app, such as a Flatpak's): +POST /api/add/stream?format=tar builds a directory from an uploaded tar (build_tar), GET /api/get/stream?format=tar +sends a target back as one (plan, length, chunks). No RNS beyond rnfs.log's, through walker. + +Reading treats the stream as untrusted input, read once, in order, never held whole (tarfile's stream mode): +- Regular files and directories become the tree, under the name rules of `add` (builder._build_tree): a member + whose name is absolute, has a `..` component, or has a part that isn't a valid entry name (not UTF-8, too + long) is skipped with a warning, as are symlinks, hard links, devices and FIFOs. Directories a path implies + are made without a member of their own; a later file replaces an earlier one of the same path (tar's own + rule, which `tar -r` relies on), and a member whose path a file or directory already holds as the other type + is skipped with a warning. A file is executable if its mode has the owner's execute bit, on every platform: a + tar's mode is its sender's, not a guess of Windows'. +- Bounds: at most max_entries files and directories (TooLarge), nesting at most MAX_DEPTH (TooLarge, as `add`), + and warnings kept up to a limit and counted beyond it (Warnings), each naming at most MAX_SHOWN characters of + the member's name, so the memory a stream can take is bounded by those and not by its length. tarfile reads a + pax or GNU long-name header whole, keeps global pax headers' keys for the rest of the stream (copying and + applying them for every member), and reads a sparse file's map as it goes, so a header over MAX_EXTENSION + bytes, more than MAX_EXTENSIONS of them in a row, global headers of more than MAX_GLOBAL_BYTES in all, and + sparse files are refused (_header_class). +- A damaged upload is never taken for a shorter one: a header that isn't valid anywhere in the stream is an error + (not tarfile's quiet end of the archive), and so is anything but zeros after the tar's end (a zeroed header + block in the middle would otherwise end it there). +- Plain tar only (no compression): the stream comes from a program on this machine, where compressing gains + nothing. + +Writing (for a target from anyone's DAG): PAX format, written by hand so the exact length is known before the first +byte (an exact Content-Length, so a client can tell a download that was cut short). Entry names are DAG names, +which hold no `/` or NUL and are never `.` or `..`, so a member's path is relative and stays inside the directory +it is extracted to; a name holding a character some extractors take for a separator (UNSAFE_CHARACTERS: a +backslash, and the fullwidth reverse solidus and solidus that "best fit" code pages map to them) is skipped, with everything below it, and +counted in the skipped Warnings. Every member is planned (paths kept as parent links, joined only when written) and +the small chunks its files would meet counted before the length is given, so writing can't fail on a limit after +a response has started. Files are their stored bytes: a sealed file stays sealed, as in a directory's `get` (GET +/api/cat decrypts one). Directories are mode 0755, files 0644 (0755 if executable), every member dated mtime (the +request's time), owned by uid and gid 0 with no names. +""" +import tarfile +from typing import BinaryIO, Callable, Iterator, NamedTuple + +from rnfs import constants, log +from rnfs.content import builder, dag, walker +from rnfs.content.builder import BuildOptions, Built, Put +from rnfs.content.dag import Entry +from rnfs.errors import InvalidDag, TooLarge + +BLOCK = tarfile.BLOCKSIZE # 512: headers, and file data padded to a multiple of it +RECORD = tarfile.RECORDSIZE # 10240: a tar ends with two zero blocks, padded to a multiple of it, as tarfile pads +# Bytes of one pax or GNU long-name header read at most: a valid path is at most MAX_DEPTH names of 255 bytes +MAX_EXTENSION = 64 * 1024 +MAX_EXTENSIONS = 8 # extension headers in a row (each read whole, tarfile following them recursively) +# Bytes of global pax headers in a stream, all told: tarfile applies their keys to every later member (git archive +# writes one of 52 bytes) +MAX_GLOBAL_BYTES = 4096 +MAX_SHOWN = 300 # characters of a member's name a warning shows (one may be 64 KiB, escaped six times longer) +# Characters some extractors read as a path separator or a drive (Windows'), never written in a tar's member names +# Not ":": ordinary POSIX names hold it (12:30.log), and to an extractor it is no separator, only a character a +# Windows file name can't have (which that extractor refuses or replaces); a name starting as a drive does is +# refused apart (safe_name) +UNSAFE_CHARACTERS = frozenset("\\\uff3c\uff0f") +COPY_SIZE = 64 * 1024 # what is read at a time of what follows a tar's end +_EXTENSION_TYPES = frozenset({tarfile.XHDTYPE, tarfile.XGLTYPE, tarfile.SOLARIS_XHDTYPE, tarfile.GNUTYPE_LONGNAME, + tarfile.GNUTYPE_LONGLINK}) +# tarfile's readers of sparse maps, which the header class refuses +SPARSE_READERS = ("_proc_sparse", "_proc_gnusparse_00", "_proc_gnusparse_01", "_proc_gnusparse_10") +# The TarInfo internals the header class overrides (_proc_member: where every header, however parsed, is +# processed), the same in Python 3.11 to 3.14: the tests check each exists, so a Python that renames one is noticed +# rather than read unbounded +TARFILE_INTERNALS = ("_proc_member", *SPARSE_READERS) + + +class Warnings: + """What a build skipped: the first limit warnings kept, the rest counted (a tar of a million symlinks mustn't + fill memory). Called with each warning.""" + + def __init__(self, limit: int = log.MAX_WARNINGS): + """No warnings yet; keep at most limit.""" + self.limit = limit + self.kept: list[str] = [] + self.total = 0 + + def __call__(self, text: str): + """Count a warning, keeping it while fewer than limit are kept.""" + self.note(lambda: text) + + def note(self, make: Callable[[], str]): + """Count a warning, making its text (make()) only if it is kept: past the limit it costs nothing.""" + self.total += 1 + if len(self.kept) < self.limit: + self.kept.append(make()) + + def result(self) -> dict: + """{"warnings": the kept ones} and, past the limit, a last one counting the rest and "warnings_total", as + log.cap_warnings gives them.""" + if self.total <= len(self.kept): + return {"warnings": list(self.kept)} + rest = self.total - len(self.kept) + return {"warnings": [*self.kept, f"... and {log.plural(rest, 'more warning')}"], "warnings_total": self.total} + + +def _refused(reason: str) -> tarfile.SubsequentHeaderError: + """The error a header check raises: one TarFile.next() turns into ReadError, wherever in the stream it is.""" + return tarfile.SubsequentHeaderError(reason) + + +def _header_class() -> type: + """A TarInfo subclass of its own for one stream, holding that stream's header counts, refusing what tarfile would + read without bound (the module docstring) and every malformed header.""" + + class Header(tarfile.TarInfo): + """A header of the stream being read.""" + + extensions = 0 # extension headers since the last member's + global_bytes = 0 # bytes of global pax headers so far + + @classmethod + def fromtarfile(cls, tarfile_): + """The next member; a malformed header anywhere is an error (tarfile ends the archive quietly at one + past the first member, which would take a damaged upload for a shorter one).""" + try: + return super().fromtarfile(tarfile_) + except (tarfile.InvalidHeaderError, tarfile.TruncatedHeaderError) as e: + raise _refused(f"malformed tar header ({e})") from None + + def _proc_member(self, tarfile_): + """Check a header just parsed against the limits, before tarfile reads what it announces (tarfile calls + this for every header block, extension headers' too, however it parsed it).""" + cls = type(self) + if self.type == tarfile.GNUTYPE_SPARSE: + raise _refused("sparse files aren't supported") + if self.type in _EXTENSION_TYPES: + cls.extensions += 1 + if self.size > MAX_EXTENSION: + raise _refused(f"a tar extended header of {self.size} bytes (at most {MAX_EXTENSION})") + if cls.extensions > MAX_EXTENSIONS: + raise _refused(f"more than {MAX_EXTENSIONS} tar extended headers in a row") + if self.type == tarfile.XGLTYPE: + cls.global_bytes += self.size + if cls.global_bytes > MAX_GLOBAL_BYTES: + raise _refused(f"global pax headers of more than {MAX_GLOBAL_BYTES} bytes") + else: + cls.extensions = 0 + return super()._proc_member(tarfile_) + + def sparse(self, *args): + """Refuse a sparse file: its map would be read without bound.""" + raise _refused("sparse files aren't supported") + + for name in SPARSE_READERS: + setattr(Header, name, sparse) + return Header + + +def _shown(text: str) -> str: + """text for a warning or log line, cut to MAX_SHOWN characters (a member's name may be 64 KiB).""" + return text if len(text) <= MAX_SHOWN else text[:MAX_SHOWN] + "..." + + +def _parts(name: str) -> list[str] | str: + """A member name's path parts ("." and empty ones dropped: `./a//b` is a/b), or why it is skipped.""" + if name.startswith("/"): + return "an absolute path" + parts = [part for part in name.split("/") if part not in ("", ".")] + if ".." in parts: + return "a path with a .. component" + if not all(dag.is_valid_name(part) for part in parts): + return "name is not valid UTF-8 or breaks the name rules" + return parts + + +class _Tree: + """The directories a tar builds, as nested dicts of name to a dict (a directory) or a Built (a file).""" + + def __init__(self, max_entries: int): + """An empty root; at most max_entries files and directories below it.""" + self.root: dict = {} + self.entries = 0 + self.max_entries = max_entries + + def _count(self): + """One more entry. Raises TooLarge past max_entries.""" + self.entries += 1 + if self.entries > self.max_entries: + raise TooLarge(f"the tar has more than max_output_entries ({self.max_entries}) files and directories") + + def directory(self, parts: list[str]) -> dict | None: + """The directory at parts, made with those above it as needed; None if a file holds one of the names.""" + current = self.root + for part in parts: + child = current.get(part) + if child is None: + self._count() + child = current[part] = {} + elif not isinstance(child, dict): + return None + current = child + return current + + def file_parent(self, parts: list[str]) -> dict | str: + """The directory a file at parts goes in (made with those above it as needed, and the file counted if + new: a file already there is replaced), or why it can't go there.""" + parent = self.directory(parts[:-1]) + if parent is None: + return "a file holds the name of one of its directories" + existing = parent.get(parts[-1]) + if isinstance(existing, dict): + return "a directory of that name came first" + if existing is None: + self._count() + return parent + + def build(self, put: Put) -> Built: + """Build every directory, deepest first (recursion bounded by MAX_DEPTH, checked as members came).""" + + def build(directory: dict) -> Built: + """One directory, after those below it.""" + entries, child_height = [], 0 + for name, child in directory.items(): + built = build(child) if isinstance(child, dict) else child + entries.append(Entry(name, built.cid, built.size, built.kind)) + child_height = max(child_height, built.height) + return builder.build_directory(entries, put, child_height) + + return build(self.root) + + +def build_tar(stream: BinaryIO, put: Put, warn: Callable[[str], None], max_entries: int, + options: BuildOptions | None = None, wrap: str | None = None) -> Built: + """Build the directory a tar stream holds (the module docstring's rules), wrapped as the entry wrap if given; + warn(text) is told what was skipped. options as for builder.build_file (the chunker, the deny list's check). + + Reads stream to its end, which after the tar's own may hold only zeros (its padding). Raises ValueError for a + stream that isn't a valid plain tar, ends early or has data after its end, TooLarge past max_entries or + MAX_DEPTH, and what options.check raises. + """ + if wrap is not None: + dag.check_name(wrap) # before reading a possibly large stream + tree = _Tree(max_entries) + try: + archive = tarfile.open(fileobj=stream, mode="r|", tarinfo=_header_class(), encoding="utf-8", + errors="surrogateescape") + with archive: + while (member := archive.next()) is not None: + # Stream mode keeps every member read, for a later getmembers(); nothing here needs them + archive.members.clear() + _add_member(archive, member, tree, put, warn, options) + # tarfile stops at the first zero block, which damage in the middle of a tar can make: through the + # stream it read ahead in, the rest must be the end blocks and padding + while data := archive.fileobj.read(COPY_SIZE): + if data.count(0) != len(data): + raise ValueError("not a valid tar stream: data after its end (a damaged header?)") + except tarfile.TarError as e: + raise ValueError(f"not a valid tar stream: {e}") from None + built = tree.build(put) + return builder.wrap(wrap, built, put) if wrap is not None else built + + +def _add_member(archive: tarfile.TarFile, member: tarfile.TarInfo, tree: _Tree, put: Put, + warn: Callable[[str], None], options: BuildOptions | None): + """Put one member into tree (or warn why it is skipped); a file's data is built as it is read.""" + shown = _shown(builder.display_path(member.name)) + parts = _parts(member.name) + if isinstance(parts, str): + warn(f"skipped {shown}: {parts}") + return + if not parts: # "." or "./": the root itself + if not member.isdir(): + warn(f"skipped {shown}: not a valid entry name") + return + if len(parts) > constants.MAX_DEPTH: + raise TooLarge(f"{shown} is nested deeper than MAX_DEPTH ({constants.MAX_DEPTH})") + if member.issym(): + warn(f"skipped symlink {shown}") + elif member.islnk(): + warn(f"skipped hard link {shown}") + elif member.isdir(): + if tree.directory(parts) is None: + warn(f"skipped {shown}: a file holds its name or the name of one of its directories") + elif not member.isreg(): + warn(f"skipped {shown}: not a regular file or directory") + else: + parent = tree.file_parent(parts) # checked before its data is read: a skipped file's is passed over + if isinstance(parent, str): + warn(f"skipped {shown}: {parent}") + return + kind = dag.KIND_EXECUTABLE if member.mode & 0o100 else dag.KIND_FILE + parent[parts[-1]] = builder.build_file(archive.extractfile(member), put, member.size, kind, options) + + +# Writing + + +class Member(NamedTuple): + """One member of a tar being written: the directory member it is in (None at the top), its name, kind + (dag.KIND_*), size (0 for a directory) and expectation. Its path is joined only when its header is written, so a + plan of many deep members holds each name once (a DAG's own, shared by its repeated subtrees).""" + + parent: "Member | None" + name: str + kind: int + size: int + expect: walker.Expect + + @property + def path(self) -> str: + """The member's path in the tar (its parents' names, then its own, joined by /).""" + names = [] + member: Member | None = self + while member is not None: + names.append(member.name) + member = member.parent + return "/".join(reversed(names)) + + +def safe_name(name: str) -> bool: + """True if name is safe as a part of a member's path: none of UNSAFE_CHARACTERS, and not starting as a Windows + drive does (`C:`, which as a member's first part would make a drive-relative path there; the rest is the DAG's + name rules).""" + drive = len(name) >= 2 and name[1] == ":" and name[0].isascii() and name[0].isalpha() + return not drive and UNSAFE_CHARACTERS.isdisjoint(name) + + +_SMALL = constants.CHUNK_SIZE // 16 # a piece shorter than this counts against max_entries (walker.SmallLeaves) + + +def _small_chunks(expect: walker.Expect, size: int, get: walker.Get, memo: dict, limit: int) -> int: + """How many small pieces reading the whole file expect names (of size bytes) meets, as walker.SmallLeaves counts + them, every occurrence counted; file nodes are read (each distinct one once: memo), chunks never. Stops counting + past limit.""" + if expect.cid.is_raw: + return 1 if 0 < size < _SMALL else 0 + if expect in memo: + return memo[expect] + walker.check_cid(expect) + node = walker.decode_checked(expect, get(expect)) + total = 0 + for child in walker.children(expect, node): + total += _small_chunks(child, child.size, get, memo, limit) + if total > limit: + break + memo[expect] = total + return total + + +def plan(expect: walker.Expect, get: walker.Get, name: str, max_entries: int, skipped: Warnings) -> list[Member]: + """The members of the tar of the file or directory expect names (its blocks all stored): a file is one member + called name; a directory's entries are at the top (its contents, as `tar -C dir .` makes them), each directory + before what it holds. Names unsafe in a tar are skipped with what is below them, counted in skipped (their + paths kept up to its limit). Raises TooLarge past max_entries entries, counting every occurrence, or past + max_entries small chunks in all (walker.SmallLeaves, which chunks() applies): before anything is sent.""" + small_left = max_entries + 2 # as SmallLeaves allows + memo_small: dict = {} + + def file_member(parent: Member | None, member_name: str, kind: int, child: walker.Expect) -> Member: + """A file's member, its small chunks counted.""" + nonlocal small_left + size = walker.size_of(child, get(child)) + small_left -= _small_chunks(child, size, get, memo_small, small_left) + if small_left < 0: + raise TooLarge("a file has more small chunks than max_output_entries allows") + return Member(parent, member_name, kind, size, child) + + block = get(expect) + if walker.kind_of(expect, block) == walker.FILE_TYPE: + kind = dag.KIND_EXECUTABLE if expect.kind == dag.KIND_EXECUTABLE else dag.KIND_FILE + return [file_member(None, name, kind, expect)] + members: list[Member] = [] + memo: dict = {} + stack: list[tuple[walker.Expect, Member | None]] = [(expect, None)] + while stack: + directory, parent = stack.pop() + below = [] + for entry, child in walker.directory_entries(directory, get, memo): + if not safe_name(entry.name): + skipped.note(lambda entry=entry, parent=parent: _shown(Member(parent, entry.name, 0, 0, child).path)) + continue + if len(members) >= max_entries: + raise TooLarge(f"more than max_output_entries ({max_entries}) entries") + if entry.kind == dag.KIND_DIRECTORY: + member = Member(parent, entry.name, entry.kind, 0, child) + below.append((child, member)) + else: + member = file_member(parent, entry.name, entry.kind, child) + members.append(member) + stack.extend(reversed(below)) + return members + + +def _header(member: Member, mtime: int) -> bytes: + """A member's header blocks (a pax header first where its path or size needs one).""" + info = tarfile.TarInfo(member.path) + info.mtime = mtime + if member.kind == dag.KIND_DIRECTORY: + info.type = tarfile.DIRTYPE + info.mode = 0o755 + else: + info.size = member.size + info.mode = 0o755 if member.kind == dag.KIND_EXECUTABLE else 0o644 + return info.tobuf(tarfile.PAX_FORMAT, "utf-8", "strict") + + +def _padding(size: int, unit: int) -> int: + """The zero bytes after size bytes up to a multiple of unit.""" + return -size % unit + + +def length(members: list[Member], mtime: int) -> int: + """The exact bytes chunks() writes for members.""" + total = sum(len(_header(member, mtime)) + member.size + _padding(member.size, BLOCK) for member in members) + total += 2 * BLOCK + return total + _padding(total, RECORD) + + +def chunks(members: list[Member], get: walker.Get, mtime: int, max_entries: int) -> Iterator[bytes]: + """The tar of members, as length() counts it. Raises InvalidDag if a file's bytes aren't its size (its + blocks were checked, so only a bug could), TooLarge past max_entries small chunks (walker.SmallLeaves, as a + write-out; plan has counted them, so only a bug could either).""" + small = walker.SmallLeaves(max_entries) # shared by every file, as write_out's + written = 0 + for member in members: + header = _header(member, mtime) + yield header + written += len(header) + if member.kind == dag.KIND_DIRECTORY: + continue + sent = 0 + for data in walker.read_range(member.expect, get, small=small): + sent += len(data) + yield data + if sent != member.size: + raise InvalidDag(f"{member.expect.cid} held {sent} bytes, not its size {member.size}") + written += sent + if _padding(sent, BLOCK): + yield bytes(_padding(sent, BLOCK)) + written += _padding(sent, BLOCK) + end = 2 * BLOCK + _padding(written + 2 * BLOCK, RECORD) + yield bytes(end) diff --git a/rnfs/content/walker.py b/rnfs/content/walker.py index 6374ee8..5572e9e 100644 --- a/rnfs/content/walker.py +++ b/rnfs/content/walker.py @@ -404,6 +404,12 @@ def _directory_entries(expect: Expect, get: Get, memo: dict | None = None) -> It stack.extend(reversed(children(current, node))) +def directory_entries(expect: Expect, get: Get, memo: dict | None = None) -> Iterator[tuple[Entry, Expect]]: + """Every entry of a directory in name order with the entry's expectation (memo: directory and index nodes + already decoded, kept for repeated subtrees). Raises ValueError for a file. Unbounded: the caller counts.""" + return _directory_entries(expect, get, memo) + + def list_directory(expect: Expect, get: Get, max_entries: int | None = None) -> Iterator[Entry]: """The entries of a directory in name order. Raises ValueError for a file, TooLarge past max_entries.""" for count, (entry, _) in enumerate(_directory_entries(expect, get), 1): diff --git a/rnfs/embedded.py b/rnfs/embedded.py index ba6efe0..243879b 100644 --- a/rnfs/embedded.py +++ b/rnfs/embedded.py @@ -18,7 +18,9 @@ Name records are heard through NameManager's record listener, which only queues them for the event thread; jobs are the jobs feature's, polled for progress. """ +import contextlib import io +import json import os import threading import time @@ -27,8 +29,9 @@ import RNS from rnfs import fetcher, fsutil, interface +from rnfs.apiclient import check_timeout from rnfs.config import Config -from rnfs.errors import RnfsError +from rnfs.errors import RnfsError, Unsupported from rnfs.log import LOG_ERROR, LOG_WARNING, log from rnfs.node import RnfsNode, lock_rnfs_dir, make_private_root from rnfs.objects import Record @@ -36,6 +39,8 @@ # An embedded node's settings where neither rnfs.toml nor the program's settings give them: an app's node is # often short-lived, and short-lived nodes should be DHT clients (section 6); the program has its own LXMF EMBEDDED_DEFAULTS = {"dht_mode": "client", "lxmf_enabled": False} +# The capabilities (api.CAPABILITIES) a Node lacks: it serves no API to be stopped through, and writes no daemon.json +NODE_LACKS = ("shutdown", "daemon_json") _process_lock = threading.Lock() _running: "Node | None" = None # the Node running in this process: one at a time, as RNS.Reticulum is one @@ -103,6 +108,7 @@ def _close(self): global _running node = self._node node.names.remove_record_listener(self._heard) + node.events.remove_listener(self._published) # if events.watch added it try: node.stop() finally: @@ -134,14 +140,42 @@ def _baseline(self, name: bytes) -> int | None: record = self._node.names.record(name) return record.sequence if record is not None else None + # Event watches + + def _published(self, event: dict): + """The event log's listener (any thread, perhaps under the names or storage lock): queue the event for the + event watches. A Node hears every event of its own run, so it never reports any lost.""" + self._events_heard(self._node.events.epoch, [event]) + + def _event_baseline(self) -> tuple[int, str]: + """Listen to the node's event log (from the first watch on), then its last sequence and epoch: an event + published in between reaches the watch, which skips it as at or before its baseline.""" + event_log = self._node.events + with self._watch_lock: + if not getattr(self, "_listening", False): # (getattr: tests build a Node without __init__) + event_log.add_listener(self._published) + self._listening = True + return event_log.last, event_log.epoch + class _Backend: - """rnfs.interface's backend for an RnfsNode: its facade, taking and returning the API's forms.""" + """rnfs.interface's backend for an RnfsNode: its facade, taking and returning the API's forms. + + A feature's methods are called as its API route calls them, so a Node gives a Client's results and errors: the + feature looked up through the route's own lookup (each feature module's _feature, or _common.feature), which + takes anything with the handler's node attribute, as this has, and raises the route's NotFound when the + feature isn't loaded; then the route's checks of its arguments, in its order.""" def __init__(self, node: RnfsNode): """The backend of node.""" self.node = node + def timeout(self, seconds: float | None): + """Nothing to bound (no request is made): a context manager doing nothing, once seconds is checked as a + Client checks it.""" + check_timeout(seconds) + return contextlib.nullcontext() + def status(self) -> dict: """The node's status.""" return self.node.status() @@ -264,8 +298,8 @@ def name_resolve(self, name: str) -> dict: from rnfs.features.keepers import parse_name_arg return self.node.name_resolve(parse_name_arg(self.node, name).hex()) - def submit_job(self, kind: str, target: str, output: str | None, from_node: str | None, **flags) -> dict: - """JobsFeature.submit.""" + def submit_job(self, kind: str, target: str | None, output: str | None, from_node: str | None, **flags) -> dict: + """JobsFeature.submit (flags: get's sealed_name and fallback_dht, or an add's or import's path and options).""" return self.node.feature("jobs").submit(kind, target, output, from_node, **flags) def job(self, job_id: str, wait: float) -> dict: @@ -298,3 +332,190 @@ def deny_add(self, entries: list[str]) -> list[str]: def deny_rm(self, entries: list[str]) -> list[str]: """DenyList.remove.""" return self.node.deny.remove(entries) + + # Features (see the class) + + def peers(self) -> list[dict]: + """PeersFeature.peers, as GET /api/peers.""" + from rnfs.features import _common, peers + return _common.feature(self, peers.NAME).peers() + + def stats(self) -> dict: + """PeersFeature.stats, as GET /api/stats.""" + from rnfs.features import _common, peers + return _common.feature(self, peers.NAME).stats() + + def contacts(self) -> list[dict]: + """ContactsFeature.list, as GET /api/contacts.""" + from rnfs.features import contacts + return contacts._feature(self).list() + + def contact_get(self, nick: str) -> dict: + """ContactsFeature.get, as GET /api/contacts/get.""" + from rnfs.features import contacts + return contacts._feature(self).get(nick) + + def contact_add(self, nick: str, ident: str, kind: str | None) -> dict: + """ContactsFeature.add, as POST /api/contacts/add (kind checked after the feature is found).""" + from rnfs.features import contacts + return contacts._feature(self).add(nick, ident, None if kind is None else contacts.check_kind(kind)) + + def contact_rm(self, nick: str): + """ContactsFeature.remove, as POST /api/contacts/rm.""" + from rnfs.features import contacts + contacts._feature(self).remove(nick) + + def name_keep(self, name: str) -> dict: + """KeepersFeature.keep of a name or a name contact's nick, as POST /api/name/keep.""" + from rnfs.features import keepers + feature = keepers._feature(self) + return feature.keep(keepers.parse_name_arg(self.node, name)) + + def name_unkeep(self, name: str): + """KeepersFeature.unkeep, as POST /api/name/unkeep.""" + from rnfs.features import keepers + feature = keepers._feature(self) + feature.unkeep(keepers.parse_name_arg(self.node, name)) + + def name_kept(self) -> list[dict]: + """KeepersFeature.kept, as GET /api/name/kept.""" + from rnfs.features import keepers + return keepers._feature(self).kept() + + def name_follow(self, name: str) -> dict: + """FollowFeature.follow of a name or a name contact's nick, as POST /api/name/follow.""" + from rnfs.features import follow, keepers + feature = follow._feature(self) + return feature.follow(keepers.parse_name_arg(self.node, name)) + + def name_unfollow(self, name: str): + """FollowFeature.unfollow, as POST /api/name/unfollow.""" + from rnfs.features import follow, keepers + feature = follow._feature(self) + feature.unfollow(keepers.parse_name_arg(self.node, name)) + + def name_following(self) -> list[dict]: + """FollowFeature.following, as GET /api/name/following.""" + from rnfs.features import follow + return follow._feature(self).following() + + def lxmf_address(self) -> dict: + """LxmfFeature.address, as GET /api/lxmf/address.""" + from rnfs.features import lxmf + return lxmf._feature(self).address() + + def lxmf_send(self, to: str, target: str, message: str | None, propagate: bool, nearest: bool) -> dict: + """LxmfFeature.send, as POST /api/lxmf/send.""" + from rnfs.features import lxmf + return lxmf._feature(self).send(to, target, message, propagate, nearest) + + def lxmf_inbox(self) -> list[dict]: + """LxmfFeature.inbox_list, as GET /api/lxmf/inbox.""" + from rnfs.features import lxmf + return lxmf._feature(self).inbox_list() + + def lxmf_inbox_get(self, entry_id, link, output: str | None, directory: str | None, + use_sent_cid: bool) -> list[dict]: + """LxmfFeature.inbox_get, as POST /api/lxmf/inbox/get: id and link checked as it checks them.""" + from rnfs.features import lxmf + return lxmf._feature(self).inbox_get(lxmf._int(entry_id, "id"), lxmf._int(link, "link", False), output, + directory, use_sent_cid) + + def lxmf_inbox_rm(self, entry_id, every: bool) -> int: + """LxmfFeature.inbox_rm of entry_id (checked as POST /api/lxmf/inbox/rm checks it), or every message.""" + from rnfs.features import lxmf + if every: + return lxmf._feature(self).inbox_rm(None) + return lxmf._feature(self).inbox_rm(lxmf._int(entry_id, "id")) + + def lxmf_outbox(self) -> list[dict]: + """LxmfFeature.outbox_list, as GET /api/lxmf/outbox.""" + from rnfs.features import lxmf + return lxmf._feature(self).outbox_list() + + def lxmf_sync(self, nearest: bool) -> dict: + """LxmfFeature.sync, as POST /api/lxmf/sync.""" + from rnfs.features import lxmf + return lxmf._feature(self).sync(nearest) + + # Streams (rnfs 0.1.7) + + def add_tar(self, data, name: str | None, pin: bool, only_hash: bool, chunker: str | None) -> dict: + """RnfsNode.add_tar of bytes or a binary file.""" + stream = io.BytesIO(data) if isinstance(data, bytes) else data + return self.node.add_tar(stream, name=name, only_hash=only_hash, pin=pin, chunker=chunker) + + def get_tar(self, target: str): + """RnfsNode.get_tar's chunks: a generator, fetching (and raising) once iterated, as a Client's.""" + def chunks(): + """The tar's bytes, the fetch ended when done or closed.""" + _, inner = self.node.get_tar(target) + try: + yield from inner + finally: + inner.close() + return chunks() + + def export_stream(self, target: str): + """RnfsNode.export_chunks' chunks: a generator, fetching (and raising) once iterated, as a Client's.""" + def chunks(): + """The archive's bytes, the fetch ended when done or closed.""" + _, inner = self.node.export_chunks(target) + try: + yield from inner + finally: + inner.close() + return chunks() + + def import_stream(self, data) -> dict: + """RnfsNode.import_stream of bytes or a binary file.""" + return self.node.import_stream(io.BytesIO(data) if isinstance(data, bytes) else data) + + # Pins, the cache, the node (rnfs 0.1.7) + + def pin_label(self, cid: str, label: str | None) -> dict: + """RnfsNode.pin_label.""" + return self.node.pin_label(cid, label) + + def cache_ls(self) -> list[dict]: + """RnfsNode.cache_ls.""" + return self.node.cache_ls() + + def cache_rm(self, cid: str) -> dict: + """RnfsNode.cache_rm.""" + return self.node.cache_rm(cid) + + def capabilities(self) -> dict: + """GET /api/capabilities' answer for this node (api.capabilities), but what an embedded node can't do: be + stopped through an API, or describe itself in daemon.json.""" + from rnfs import api + return api.capabilities(self.node, without=NODE_LACKS) + + def shutdown(self, wait: float | None) -> bool: + """Refused: a Node is the program's; close() stops it.""" + raise Unsupported("an rnfs.Node runs inside this program: close() it to stop it") + + def daemon_info(self) -> dict | None: + """None: a Node is no daemon, and its rnfs dir has no daemon.json.""" + return None + + def config(self) -> dict: + """SettingsFeature.describe, as GET /api/config answers it (through JSON, so the values have the forms a + Client's have).""" + from rnfs.features import _common, settings + return json.loads(json.dumps(_common.feature(self, settings.NAME).describe())) + + def config_set(self, changes: dict) -> dict: + """SettingsFeature.set of changes as POST /api/config/set's JSON body carries them (tuples as lists: the + feature checks rnfs.toml reads back as exactly what it was given).""" + from rnfs.features import _common, settings + feature = _common.feature(self, settings.NAME) + return feature.set(json.loads(json.dumps(changes))) + + def events_since(self, after: int | None, kinds: list[str] | None, wait: int, epoch: str | None) -> dict: + """EventLog.since, as GET /api/events calls it: wait capped as it caps it, and none without after.""" + from rnfs import api + event_log = getattr(self.node, "events", None) + if event_log is None: + raise Unsupported("this node keeps no event log") + return event_log.since(after, kinds, min(wait, api.MAX_WATCH_WAIT) if after is not None else 0, epoch) diff --git a/rnfs/errors.py b/rnfs/errors.py index db9373a..9ef8e02 100644 --- a/rnfs/errors.py +++ b/rnfs/errors.py @@ -21,6 +21,13 @@ class NotFound(RnfsError): """Content, a path inside a DAG, a name or an alias doesn't exist, or no provider could supply it.""" +class Unsupported(NotFound): + """This node can't do what was asked at all (a daemon started without a way to stop it from the API, say), as + opposed to there being no such content. A NotFound (404) whose code is "unsupported".""" + + code = "unsupported" + + class TooLarge(RnfsError): """A fetch or read would exceed a bound: max_download_size, max_output_entries, the DAG byte or byte-range bound (section 7), a remote pin's or auto-fetch's quota, or the free space an output needs.""" @@ -69,3 +76,21 @@ class NotRecipient(ValueError): # error, and rnfs.gateway has a fixed message for each STATUSES = ((ValueError, 400), (NotFound, 404), (TooLarge, 413), (Denied, 451), (InvalidDag, 422), (Busy, 503), (StorageFull, 507)) + +# The machine-readable code an API error carries ({"error", "code"}; design.md, "Error codes"), so a client needn't +# read the message: the node's errors by STATUSES' classes, then the API's own refusals. An error with a code +# attribute of its own (one of CODES) gives that instead. A code is never renamed once released +STATUS_CODES = {400: "bad_request", 404: "not_found", 413: "too_large", 451: "denied", 422: "invalid_dag", + 503: "busy", 507: "storage_full", 500: "internal"} +API_CODES = ("bad_host", "unauthorized", "no_such_endpoint", "method_not_allowed", "timeout", "length_required", + "body_too_large", "unsupported", "too_many_connections") +CODES = (*STATUS_CODES.values(), *API_CODES) + + +def error_code(error: Exception, status: int) -> str: + """The code for error, answered with status: its own code attribute if it has one of CODES, else the + status's ("internal" for any status without one).""" + own = getattr(error, "code", None) + if isinstance(own, str) and own in CODES: + return own + return STATUS_CODES.get(status, "internal") diff --git a/rnfs/events.py b/rnfs/events.py new file mode 100644 index 0000000..3e25502 --- /dev/null +++ b/rnfs/events.py @@ -0,0 +1,126 @@ +"""The node's event log for local clients: what changed (jobs, pins, the cache, the LXMF inbox, names, peers), so +an app learns of it through one long poll (GET /api/events, rnfs/api.py) instead of polling every list. No RNS. + +Every event is a dict: "seq" (1, 2, ... for the node's run), "kind" (EVENT_KINDS), "time" (Unix time) and the +kind's own fields (design.md, "Events"). The log keeps the last MAX_EVENTS; a client that asks for events after a +sequence the log no longer holds, of a kind it asks for (it fell behind), or of another run (the daemon restarted: +"epoch" changed) gets "lost": true and every event held, and reloads its lists. Level-triggered like GET /api/name/watch: a client that asks again with +the "next" it was given loses nothing between calls. + +publish() is called from anywhere, some callers holding their own locks (the names lock, storage's): it only +appends under the log's condition, the innermost lock, and never calls out. Listeners (an embedded rnfs.Node's +events thread) are told with that condition released, in sequence order (publishers take turns, _order), and must +only queue. +""" +import secrets +import threading +import time +from collections import deque + +MAX_EVENTS = 1024 # events kept; a client further behind gets "lost" +MAX_EVENT_CIDS = 100 # CIDs one event lists; "count" says how many changed (a gc may uncache thousands) +# Every kind published, for GET /api/events?kinds= and the capability list +EVENT_KINDS = ("job", "pin", "cache", "inbox", "name", "peer") +RESERVED_FIELDS = frozenset({"seq", "kind", "time"}) # every event's own + + +class EventLog: + """The events of one node run, numbered from 1 (see the module docstring). Thread-safe.""" + + def __init__(self, clock=time.time, max_events: int = MAX_EVENTS): + """An empty log; clock gives each event's time.""" + self.clock = clock + self.epoch = secrets.token_hex(8) # this run's: a client seeing another knows the daemon restarted + self._cond = threading.Condition() + # Held by a publisher from numbering its event until its listeners are told, so listeners hear events in + # sequence order (two publishers at once would otherwise race to them, and a listener tracking the last + # sequence it heard would skip the lower one). Taken before _cond, never under it; reentrant, for a + # listener that publishes + self._order = threading.RLock() + self._events: deque[dict] = deque(maxlen=max_events) + self._last = 0 # the last sequence given + self._dropped: dict[str, int] = {} # kind -> the last sequence of that kind the log no longer holds + self._listeners: tuple = () + + def publish(self, kind: str, /, **fields) -> dict: + """Append an event of kind with fields, wake the waiting calls and tell the listeners; the event. Raises + ValueError for an unknown kind, or a field named as one of the event's own (seq, kind, time).""" + if kind not in EVENT_KINDS: + raise ValueError(f"unknown event kind {kind!r}") + if fields.keys() & RESERVED_FIELDS: + raise ValueError(f"an event's fields can't be named {', '.join(sorted(fields.keys() & RESERVED_FIELDS))}") + with self._order: + with self._cond: + self._last += 1 + event = {"seq": self._last, "kind": kind, "time": self.clock(), **fields} + if len(self._events) == self._events.maxlen: + oldest = self._events[0] + self._dropped[oldest["kind"]] = oldest["seq"] + self._events.append(event) + self._cond.notify_all() + listeners = self._listeners + for listener in listeners: + try: + listener(event) + except Exception: # noqa: BLE001 - a listener only queues; one failing mustn't stop the publisher + pass + return event + + def add_listener(self, listener): + """Call listener(event) for every event published from now on (it must only queue).""" + with self._cond: + self._listeners = (*self._listeners, listener) + + def remove_listener(self, listener): + """Stop calling listener.""" + with self._cond: + self._listeners = tuple(item for item in self._listeners if item != listener) + + @property + def last(self) -> int: + """The sequence of the last event published (0 before any).""" + with self._cond: + return self._last + + def since(self, after: int | None, kinds=None, wait: float = 0.0, epoch: str | None = None) -> dict: + """{"events", "next", "lost", "epoch"}: the events held after sequence after (of kinds, if given), once + there is one or after wait seconds. after None: none, only "next" (where to start from now). "lost" is + True when events after after of kinds are no longer held, or epoch (the one a client was given) isn't this run's: + the events held are all returned, and the client should reload what it shows.""" + wanted = frozenset(kinds) if kinds else None + deadline = time.monotonic() + wait + with self._cond: + if after is None: + return {"events": [], "next": self._last, "lost": False, "epoch": self.epoch} + lost = (epoch is not None and epoch != self.epoch) or after > self._last + if lost: + after = 0 + while True: + # Lost: an event of a kind asked for, after after, that the log dropped (others don't matter here) + lost = lost or any(seq > after for kind, seq in self._dropped.items() + if wanted is None or kind in wanted) + found = [event for event in self._events + if event["seq"] > after and (wanted is None or event["kind"] in wanted)] + remaining = deadline - time.monotonic() + if found or lost or remaining <= 0: + return {"events": found, "next": self._last, "lost": lost, "epoch": self.epoch} + self._cond.wait(remaining) + + +def cid_fields(cids) -> dict: + """{"cids": the first MAX_EVENT_CIDS of cids written out, "count": how many}: an event about many pins or + cached fetches stays small, and a client seeing count above len(cids) reloads the list.""" + cids = list(cids) + return {"cids": [str(cid) for cid in cids[:MAX_EVENT_CIDS]], "count": len(cids)} + + +def publish(node, kind: str, /, **fields): + """Publish an event on node's log, if it has one (a stand-in node in tests may not). Never raises: an event + lost is a client reloading later, never a failed operation.""" + log = getattr(node, "events", None) + if log is None: + return + try: + log.publish(kind, **fields) + except Exception: # noqa: BLE001 - see above + pass diff --git a/rnfs/features/__init__.py b/rnfs/features/__init__.py index 811c82f..7c0c929 100644 --- a/rnfs/features/__init__.py +++ b/rnfs/features/__init__.py @@ -34,7 +34,7 @@ # Module paths, in start order. A feature may use the features before it FEATURE_MODULES: list[str] = ["rnfs.features.peers", "rnfs.features.jobs", "rnfs.features.keepers", "rnfs.features.follow", "rnfs.features.contacts", "rnfs.features.lxmf", - "rnfs.features.deny"] + "rnfs.features.deny", "rnfs.features.settings"] ApiRoute = Callable[[object], None] # handler(api_handler) API_METHODS = ("GET", "POST") # what a feature's API route may answer diff --git a/rnfs/features/jobs.py b/rnfs/features/jobs.py index 6bd376e..b171b3d 100644 --- a/rnfs/features/jobs.py +++ b/rnfs/features/jobs.py @@ -1,13 +1,27 @@ -"""Background jobs for long transfers (feature "jobs"): get, pin and export run by the daemon, with progress. +"""Background jobs for long transfers (feature "jobs"): get, pin, export, add and import run by the daemon, with +progress. On LoRa a 1 MB file can take an hour, so a transfer runs as a job the daemon owns instead of inside one API request. `rnfs get`, `pin add` and `export` start a job and follow it with a progress line on stderr (or -return at once with --background); `rnfs jobs` lists, shows, waits for, cancels and clears jobs. +return at once with --background); `rnfs add --background` and `rnfs import --background` start an add +or import job; `rnfs jobs` lists, shows, waits for, cancels and clears jobs. -- A job runs one node facade call (get, pin_add, export) on a thread of its own, inside +- A job runs one node facade call (get, pin_add, export, add_path, import_archive) on a thread of its own, inside fetcher.observing(): the fetch reports the verified size of what it fetches and each block it meets, and raises Cancelled at its next step (within fetcher.POLL) once the job is cancelled or the node stops. At most max_jobs run at once (2; 1 on slow nodes); queued jobs start oldest first. +- Add and import jobs read a file, directory or archive on this machine (path) instead of fetching a target: + their total_bytes is measured before reading (builder.tree_size, or the archive's size), each read is counted + as done (Job.read, the facade's progress callback), which raises Cancelled at once when the job is cancelled + or the node stops, and their rate is of bytes read. They hold no protection session: an interrupted one runs + again from the start, which is safe because adding is idempotent (the same content gives the same CIDs, and + blocks stored already are kept), and what a cancelled one stored is unreferenced until the next collection. + An add job's options are POST /api/add's (ADD_OPTIONS); a sealed add run again seals afresh, so only the run + that finishes gives the job its CID. +- Every change of a job's state (queued, running, done, failed, cancelled, and forgotten: "removed") and, at + most once a second (EVENT_INTERVAL), a running job's progress is published as a "job" event (rnfs.events) with + the job's state and progress fields (Job.event; the job's kind as "job_kind", an event's "kind" being "job"), + not its result: a client fetches the job for that. - Jobs are saved in the state database's jobs table (rnfs.statedb), durably, at every state change and with the node's periodic saves (keeping a running job's progress recent); each save writes only the jobs changed since the last. At startup, jobs that were queued or running are queued again and resume where they @@ -41,6 +55,7 @@ import time from collections import deque from contextlib import ExitStack +from typing import Callable from rnfs import ids from rnfs.constants import PARTIAL_SUFFIX @@ -49,7 +64,12 @@ from rnfs.fsutil import printable NAME = "jobs" -KINDS = ("get", "pin", "export") +KINDS = ("get", "pin", "export", "add", "import") +LOCAL_KINDS = frozenset({"add", "import"}) # read a path on this machine rather than fetch a target +# An add job's options: POST /api/add's, each with its JSON type (absent: add_path's default) +ADD_OPTIONS = {"wrap": bool, "only_hash": bool, "pin": bool, "to": list, "to_self": bool, "seal_name": bool, + "chunker": str} +MAX_RECIPIENT_TEXT = 256 # characters of one of an add job's to= recipients (a key is 128 hex digits) QUEUED, RUNNING, DONE, FAILED, CANCELLED = "queued", "running", "done", "failed", "cancelled" STATES = (QUEUED, RUNNING, DONE, FAILED, CANCELLED) ACTIVE = frozenset({QUEUED, RUNNING}) @@ -61,6 +81,11 @@ POLL_TTY = 1 # seconds between progress updates the CLI shows POLL_QUIET = 10 # seconds a CLI without a progress line waits per request (it returns early when the job ends) FINISHED_GRACE = 60.0 # seconds a finished job is kept whatever max_finished_jobs says +EVENT_INTERVAL = 1.0 # seconds between a running job's progress events, at least +# A "job" event's fields (rnfs.events): what changes, not the result (clients fetch the job for that); "kind" is +# sent as "job_kind", since an event's own "kind" is "job" +EVENT_FIELDS = ("id", "kind", "state", "error", "total_bytes", "done_bytes", "blocks", "blocks_total", + "downloaded_bytes", "rate", "eta") _JOB_ID = re.compile(r"[0-9a-f]{8}") _NODE_ID = ids.HEX16_LOWER # a saved job's from_node, as bytes.hex() writes it (a row that fails is deleted) @@ -73,10 +98,11 @@ def _log(message: str, level: int = LOG_INFO): class Job: """One transfer: what it is, its state and its progress. Also the fetch Observer of its runs.""" - def __init__(self, job_id: str, kind: str, target: str, output: str | None = None, + def __init__(self, job_id: str, kind: str, target: str | None, output: str | None = None, from_node: str | None = None, created: float | None = None, sealed_name: bool = False, - fallback_dht: bool = False): - """A queued job; from_node is a node ID in hex. sealed_name and fallback_dht are for get jobs.""" + fallback_dht: bool = False, path: str | None = None, options: dict | None = None): + """A queued job; from_node is a node ID in hex. sealed_name and fallback_dht are for get jobs; path (and + for add, options: ADD_OPTIONS) for add and import jobs, which have no target.""" self.id = job_id self.kind = kind self.target = target @@ -84,6 +110,8 @@ def __init__(self, job_id: str, kind: str, target: str, output: str | None = Non self.from_node = from_node self.sealed_name = sealed_name # write a sealed file under its sealed name, beside output self.fallback_dht = fallback_dht # if from_node fails, try again with ordinary DHT discovery + self.path = path # what an add or import job reads, on this machine + self.options = options # an add job's add_path options (only those set) self.state = QUEUED self.created = time.time() if created is None else created self.started: float | None = None # when the latest run started @@ -101,14 +129,23 @@ def __init__(self, job_id: str, kind: str, target: str, output: str | None = Non self.stop_event: threading.Event | None = None # the feature's, set when the node stops self.protection: ExitStack | None = None # the session keeping its blocks while queued or running self.position: int | None = None # its row's place in the jobs table (creation order), once queued or loaded - self._lock = threading.Lock() # guards _samples and _seen - self._samples: deque[tuple[float, int]] = deque() # (monotonic time, downloaded_bytes) + # Called with the job at most every EVENT_INTERVAL while it progresses (the feature's progress event) + self.on_progress: Callable[["Job"], None] | None = None + self._lock = threading.Lock() # guards _samples, _moved and _seen + self._moved = 0 # what the rate counts: bytes downloaded (transfers) or read (add, import) this run + self._samples: deque[tuple[float, int]] = deque() # (monotonic time, _moved) self._seen: set = set() # CIDs met this run, so each counts once + self._last_event = float("-inf") # monotonic time of the last progress event + + @property + def subject(self) -> str: + """What the job works on, for messages: its target, or an add's or import's path.""" + return self.target if self.target is not None else str(self.path) # Observer (rnfs.fetcher.Observer), called on the job's thread def total(self, size: int): - """The fetch knows the verified size of what it fetches.""" + """The fetch knows the verified size of what it fetches (or an add or import measured what it reads).""" self.total_bytes = size def block(self, cid, length: int, downloaded: bool): @@ -117,17 +154,46 @@ def block(self, cid, length: int, downloaded: bool): with self._lock: if downloaded: self.downloaded_bytes += length - if not self._samples or now - self._samples[-1][0] >= SAMPLE_INTERVAL: - self._samples.append((now, self.downloaded_bytes)) - # Prune here too, not only in rate(): a job nobody polls would otherwise keep a sample a - # second for as long as it runs (about 10 MB a day) - self._prune(now) - if cid in self._seen: - return - self._seen.add(cid) - self.blocks += 1 - if cid.is_raw: - self.done_bytes += length + self._sample(now, length) + if cid not in self._seen: + self._seen.add(cid) + self.blocks += 1 + if cid.is_raw: + self.done_bytes += length + self._progressed(now) + + def read(self, length: int): + """An add or import read length bytes of its path (the facade's progress callback, on the job's thread): + counted as done. Raises Cancelled once the job is cancelled or the node stops, which stops the read at + once.""" + from rnfs.errors import Cancelled + if self.cancelled(): + raise Cancelled(f"job {self.id} was cancelled") + now = time.monotonic() + with self._lock: + self.done_bytes += length + self._sample(now, length) + self._progressed(now) + + def check_cancelled(self): + """Raise Cancelled once the job is cancelled or the node stops (an add's measuring, between entries).""" + self.read(0) + + def _sample(self, now: float, length: int): + """Count length bytes moved for the rate, sampling at most every SAMPLE_INTERVAL. Called with _lock held.""" + self._moved += length + if not self._samples or now - self._samples[-1][0] >= SAMPLE_INTERVAL: + self._samples.append((now, self._moved)) + # Prune here too, not only in rate(): a job nobody polls would otherwise keep a sample a second for + # as long as it runs (about 10 MB a day) + self._prune(now) + + def _progressed(self, now: float): + """Tell on_progress, if EVENT_INTERVAL has passed since it was last told (on the job's thread, no lock).""" + if self.on_progress is None or now - self._last_event < EVENT_INTERVAL: + return + self._last_event = now + self.on_progress(self) def cancelled(self) -> bool: """True once the job is cancelled or the node stops.""" @@ -141,9 +207,10 @@ def begin_run(self): self.state = RUNNING self.started = time.time() self.runs += 1 - self.done_bytes = self.blocks = self.downloaded_bytes = 0 + self.done_bytes = self.blocks = self.downloaded_bytes = self._moved = 0 self._seen = set() self._samples = deque([(time.monotonic(), 0)]) + self._last_event = time.monotonic() # the "running" event has just been published def _prune(self, now: float): """Drop samples older than RATE_WINDOW, keeping one at or before the window's start as the base to @@ -152,18 +219,19 @@ def _prune(self, now: float): self._samples.popleft() def rate(self, now: float | None = None) -> float | None: - """Bytes/s downloaded over the last RATE_WINDOW of a running job (it decays while nothing arrives).""" + """Bytes/s downloaded (read, for add and import) over the last RATE_WINDOW of a running job (it decays + while nothing arrives).""" now = time.monotonic() if now is None else now with self._lock: if self.state != RUNNING or not self._samples: return None self._prune(now) since, base = self._samples[0] - downloaded = self.downloaded_bytes + moved = self._moved elapsed = now - since if elapsed < SAMPLE_INTERVAL: return None - return max(0.0, (downloaded - base) / elapsed) + return max(0.0, (moved - base) / elapsed) def to_dict(self) -> dict: """The job as the API shows it, with its current rate and ETA.""" @@ -173,14 +241,24 @@ def to_dict(self) -> dict: eta = max(0.0, (self.total_bytes - self.done_bytes) / rate) return {**self.to_record(), "state": self.state, "rate": rate, "eta": eta} + def event(self) -> dict: + """The job's "job" event fields (EVENT_FIELDS): its state and progress, not its result. Its kind is + "job_kind": an event's "kind" is the event's own ("job").""" + job = self.to_dict() + return {("job_kind" if name == "kind" else name): job[name] for name in EVENT_FIELDS} + def to_record(self) -> dict: - """The job as saved (a running job is saved as such, and queued again when loaded).""" - return {"id": self.id, "kind": self.kind, "target": self.target, "output": self.output, - "from": self.from_node, "sealed_name": self.sealed_name, "fallback_dht": self.fallback_dht, - "state": self.state, "created": self.created, "started": self.started, - "finished": self.finished, "error": self.error, "result": self.result, "runs": self.runs, - "total_bytes": self.total_bytes, "done_bytes": self.done_bytes, "blocks": self.blocks, - "blocks_total": self.blocks_total, "downloaded_bytes": self.downloaded_bytes} + """The job as saved (a running job is saved as such, and queued again when loaded). path and options + only for the kinds that have them (add, import), so the record of every other job is as 0.1.6 saved it.""" + record = {"id": self.id, "kind": self.kind, "target": self.target, "output": self.output, + "from": self.from_node, "sealed_name": self.sealed_name, "fallback_dht": self.fallback_dht} + if self.path is not None or self.options is not None: + record.update(path=self.path, options=self.options) + record.update({"state": self.state, "created": self.created, "started": self.started, + "finished": self.finished, "error": self.error, "result": self.result, "runs": self.runs, + "total_bytes": self.total_bytes, "done_bytes": self.done_bytes, "blocks": self.blocks, + "blocks_total": self.blocks_total, "downloaded_bytes": self.downloaded_bytes}) + return record @classmethod def from_record(cls, record) -> "Job": @@ -194,9 +272,10 @@ def from_record(cls, record) -> "Job": if kind not in KINDS or state not in STATES: raise ValueError("bad kind or state") sealed_name, fallback_dht = record.get("sealed_name", False), record.get("fallback_dht", False) - check_job_params(kind, target, output, from_node, sealed_name, fallback_dht) + path, options = record.get("path"), record.get("options") + check_job_params(kind, target, output, from_node, sealed_name, fallback_dht, path, options) job = cls(job_id, kind, target, output, from_node, _number(record.get("created"), time.time()), - sealed_name, fallback_dht) + sealed_name, fallback_dht, path, options) job.state = state job.started = _number(record.get("started"), None) job.finished = _number(record.get("finished"), None) @@ -237,17 +316,31 @@ def _capped(result: dict) -> dict: return result -def check_job_params(kind: str, target, output, from_node, sealed_name=False, fallback_dht=False): - """Raise ValueError unless these fit a job of kind: a well-formed target, an absolute output path for - get and export (none for pin), and from (a node ID in hex), sealed_name and fallback_dht (booleans) - only for get.""" +def check_job_params(kind: str, target, output, from_node, sealed_name=False, fallback_dht=False, path=None, + options=None): + """Raise ValueError unless these fit a job of kind: for get, pin and export a well-formed target, an absolute + output path for get and export (none for pin), and from (a node ID in hex), sealed_name and fallback_dht + (booleans) only for get; for add and import an absolute path and neither target nor output, and options + (check_add_options) only for add.""" from rnfs.content import walker # validates the root CID and every path name if kind not in KINDS: raise ValueError(f"kind must be one of {', '.join(KINDS)}") - walker.parse_target(target) - if kind == "pin": + if kind in LOCAL_KINDS: + if target is not None or output is not None: + raise ValueError(f"an {kind} job reads a path: it takes no target or output") + if not isinstance(path, str) or not os.path.isabs(path) or "\x00" in path: + raise ValueError(f"path must be an absolute path: {path!r}") + if options is not None: + if kind != "add": + raise ValueError("only add jobs take options") + check_add_options(options) + elif path is not None or options is not None: + raise ValueError(f"a {kind} job takes a target, not a path") + else: + walker.parse_target(target) + if kind == "pin" or kind in LOCAL_KINDS: if output is not None: - raise ValueError("a pin job has no output") + raise ValueError(f"a {kind} job has no output") elif not isinstance(output, str) or not os.path.isabs(output) or "\x00" in output: raise ValueError(f"output must be an absolute path: {output!r}") if from_node is not None: @@ -262,6 +355,37 @@ def check_job_params(kind: str, target, output, from_node, sealed_name=False, fa raise ValueError(f"only get jobs take {name}") +def check_add_options(options): + """Raise ValueError unless options are an add job's: a JSON object of ADD_OPTIONS, each of its type, to a list of + at most MAX_RECIPIENTS recipient strings, chunker a known chunker's name. What the node checks only when the + add runs (recipients it knows, only_hash with to) fails the job then, with the add's own message.""" + from rnfs.content import chunker, seal + if not isinstance(options, dict): + raise ValueError("an add job's options must be an object") + for name, value in options.items(): + expected = ADD_OPTIONS.get(name) + if expected is None: + raise ValueError(f"unknown add option {name!r}") + if type(value) is not expected: + raise ValueError(f"add option {name} must be a {expected.__name__}") + recipients = options.get("to", []) + if len(recipients) > seal.MAX_RECIPIENTS or not all( + isinstance(text, str) and 0 < len(text) <= MAX_RECIPIENT_TEXT for text in recipients): + raise ValueError(f"to must list at most {seal.MAX_RECIPIENTS} recipients") + if "chunker" in options: + chunker.check(options["chunker"]) + + +def add_job_options(value) -> dict | None: + """An add job's options from a JSON body's value(name) (None when absent), as POST /api/add takes them: wrap + and only_hash (flags), and node_content.add_options' (pin, to, to_self, seal_name, chunker); only those set, or + None for none. Raises ValueError as POST /api/add does.""" + from rnfs import api, node_content + options = {name: True for name in ("wrap", "only_hash") if api.parse_bool(value(name))} + options.update(node_content.add_options(value)) + return options or None + + def import_jobs(value: dict, tx) -> int: """Import jobs.json (statedb.migrate): its jobs, in order, replace the jobs table, each checked as rnfs 0.1.4 loaded it (a malformed one is logged and skipped; of a repeated id, the first is kept) and saved as @@ -337,6 +461,7 @@ def _load(self, db): bad.append((job_id,)) continue job.position = position + job.on_progress = self._publish if job.state in ACTIVE: if job.state == RUNNING: self._store.changed(job.id) # saved as queued by the next save @@ -392,12 +517,29 @@ def _cap_finished(self): if (job.finished or job.created) < recent: del self._jobs[job.id] self._store.changed(job.id) + self._removed(job) + + # Events + + def _publish(self, job: Job): + """Publish a "job" event of job's state and progress (Job.event; rnfs.events never raises). Also each job's + on_progress, called on its thread with no lock held.""" + from rnfs import events + events.publish(self.node, "job", **job.event()) + + def _removed(self, job: Job): + """Publish that job was forgotten (cleared, or past max_finished_jobs): its last state, and "removed".""" + from rnfs import events + events.publish(self.node, "job", id=job.id, job_kind=job.kind, state=job.state, removed=True) # Protection def _protect(self, job: Job): - """Keep everything stored below the job's root while it is queued or running.""" + """Keep everything stored below the job's root while it is queued or running; an add or import job has none + (it runs again from the start: the module docstring).""" from rnfs.content import walker + if job.target is None: + return root, _ = walker.parse_target(job.target) stack = ExitStack() session = stack.enter_context(self.node.storage.protect()) @@ -454,9 +596,11 @@ def _schedule(self) -> bool: self._store.changed(job.id) thread = threading.Thread(target=self._run, args=(job,), name=f"rnfs-job-{job.id}", daemon=True) self._threads[job.id] = thread - starting.append(thread) + starting.append((job, thread)) running += 1 - for thread in starting: + for job, _ in starting: + self._publish(job) # before its thread runs, so its first progress event comes after + for _, thread in starting: thread.start() if starting: self._save_quietly() @@ -478,7 +622,7 @@ def _run(self, job: Job): error = str(e) or type(e).__name__ except Exception as e: # noqa: BLE001 - a job's failure mustn't take its thread's bookkeeping with it error = str(e) or type(e).__name__ - _log(f"job {job.id} ({job.kind} {job.target}) failed: {e!r}", LOG_ERROR) + _log(f"job {job.id} ({job.kind} {job.subject}) failed: {e!r}", LOG_ERROR) self._finish(job, state, error, result) def _execute(self, job: Job) -> dict: @@ -488,11 +632,32 @@ def _execute(self, job: Job) -> dict: result = self._get(job) elif job.kind == "pin": result = node.pin_add(job.target) - else: + elif job.kind == "export": result = node.export(job.target, job.output) + elif job.kind == "add": + result = self._add(job) + else: + result = self._import(job) job.blocks_total = self._dag_block_count(result) return result + def _add(self, job: Job) -> dict: + """An add job: measure what it will read, then node.add_path with its options, as POST /api/add calls it.""" + from rnfs.content import builder + try: + job.total(builder.tree_size(job.path, job.check_cancelled)) + except OSError: + pass # add_path reports it, in its own words + return self.node.add_path(job.path, progress=job.read, **(job.options or {})) + + def _import(self, job: Job) -> dict: + """An import job: node.import_archive, as POST /api/import calls it; its size is what it will read.""" + try: + job.total(os.path.getsize(job.path)) + except OSError: + pass # import_archive reports it, in its own words + return self.node.import_archive(job.path, progress=job.read) + def _get(self, job: Job) -> dict: """A get job: write to its partial path, then rename that to the output (or the sealed name).""" from rnfs import fsutil @@ -561,29 +726,38 @@ def _finish(self, job: Job, state: str, error: str | None, result: dict | None): self._threads.pop(job.id, None) self._store.changed(job.id) self._cap_finished() + self._publish(job) if state in FINISHED: job.ended.set() - _log(f"job {job.id} ({job.kind} {job.target}) {state}" + (f": {error}" if error else "")) + _log(f"job {job.id} ({job.kind} {job.subject}) {state}" + (f": {error}" if error else "")) if not self._schedule(): # starting the next saves this outcome with it self._save_quietly() # Operations (API) - def submit(self, kind: str, target: str, output: str | None = None, from_node: str | None = None, - sealed_name: bool = False, fallback_dht: bool = False) -> dict: - """Queue a job; its dict. Raises ValueError for bad parameters or an output in use, Busy when full. + def submit(self, kind: str, target: str | None = None, output: str | None = None, from_node: str | None = None, + sealed_name: bool = False, fallback_dht: bool = False, path: str | None = None, + options: dict | None = None) -> dict: + """Queue a job; its dict. Raises ValueError for bad parameters or an output in use, NotFound for an add's + or import's path that doesn't exist, Busy when full. For get: sealed_name writes a sealed file under its sealed name beside output; fallback_dht retries - by DHT discovery if from_node fails. + by DHT discovery if from_node fails. For add and import: path, and for add options (ADD_OPTIONS). """ - from rnfs.errors import Busy + from rnfs.errors import Busy, NotFound contacts = self.dependency("contacts") - if contacts is not None: + if contacts is not None and target is not None: # `alice/docs`: a name contact's nick becomes the CID its name points at now (rnfs.features.contacts) target = contacts.expand_target(target) - check_job_params(kind, target, output, from_node, sealed_name, fallback_dht) + check_job_params(kind, target, output, from_node, sealed_name, fallback_dht, path, options) if output is not None and os.path.lexists(output): raise ValueError(f"{output} already exists") + if path is not None and not os.path.lexists(path): + # As add_path and import_archive say it, but before queueing: the CLI and apps learn it at once + raise NotFound(f"no such file or directory: {path}" if kind == "add" else f"no such file: {path}") + if kind == "import" and not os.path.isfile(path): + # A FIFO or device would hold a job slot until something writes to it (import_archive refuses it too) + raise ValueError(f"{path} is not a regular file") with self._lock: active = [job for job in self._jobs.values() if job.state in ACTIVE] if len(active) >= MAX_ACTIVE: @@ -594,14 +768,17 @@ def submit(self, kind: str, target: str, output: str | None = None, from_node: s job_id = secrets.token_hex(4) while job_id in self._jobs: job_id = secrets.token_hex(4) - job = Job(job_id, kind, target, output, from_node, sealed_name=sealed_name, fallback_dht=fallback_dht) + job = Job(job_id, kind, target, output, from_node, sealed_name=sealed_name, fallback_dht=fallback_dht, + path=path, options=options) job.stop_event = self._stop_event + job.on_progress = self._publish job.position = self._position self._position += 1 self._protect(job) self._jobs[job_id] = job self._store.changed(job_id) - _log(f"job {job_id} queued: {kind} {target}" + (f" to {output}" if output else "")) + _log(f"job {job_id} queued: {kind} {job.subject}" + (f" to {output}" if output else "")) + self._publish(job) # queued, before _schedule may start it if not self._schedule(): # else saved as it started self._save_quietly() return job.to_dict() @@ -629,7 +806,8 @@ def list(self) -> list[dict]: return [job.to_dict() for job in jobs] def cancel(self, job_id: str) -> dict: - """Cancel a job: a queued one at once, a running one at its fetch's next step. Finished ones stay.""" + """Cancel a job: a queued one at once, a running one at its fetch's next step (an add's or import's next + read). Finished ones stay.""" job = self._job(job_id) ended = False with self._lock: @@ -643,6 +821,7 @@ def cancel(self, job_id: str) -> dict: if ended: self._release(job) job.ended.set() + self._publish(job) with self._lock: self._cap_finished() self._save_quietly() @@ -656,7 +835,7 @@ def clear(self, job_id: str | None = None) -> int: removed = [jid for jid, job in self._jobs.items() if job.state in FINISHED and (job_id is None or jid == job_id)] for jid in removed: - del self._jobs[jid] + self._removed(self._jobs.pop(jid)) self._store.changed(*removed) if removed: self._save_quietly() @@ -713,12 +892,20 @@ def _feature(handler) -> JobsFeature: def api_start(handler): - """POST /api/jobs {"kind", "target", "output", "from", "sealed_name", "fallback_dht"}: queue a job; the job.""" + """POST /api/jobs {"kind", "target", "output", "from", "sealed_name", "fallback_dht"} for get, pin and export; + {"kind": "add", "path", and POST /api/add's options} or {"kind": "import", "path"}: queue a job; the job.""" from rnfs import api body = handler.json_body() kind = body.get("kind") if kind not in KINDS: raise ValueError(f"kind must be one of {', '.join(KINDS)}") + if kind in LOCAL_KINDS: + path = api.parse_path(body.get("path")) + options = add_job_options(body.get) if kind == "add" else None + # A target or output given is refused by submit's check, as for the other kinds + handler.send_json(200, _feature(handler).submit(kind, body.get("target"), body.get("output"), path=path, + options=options)) + return target = api.parse_target(body.get("target")) output = api.parse_path(body.get("output")) if kind != "pin" else body.get("output") from_node = api.parse_hex_id(body["from"]).hex() if body.get("from") else None @@ -799,6 +986,8 @@ def format_progress(job: dict, width: int = 20) -> str: if state == RUNNING: rate = job.get("rate") speed = f"{human_size(rate)}/s" if rate is not None else "-/s" + if total is None and job.get("kind") in LOCAL_KINDS: + return f"reading: {human_size(done)}, {speed}" # its size is being measured if total is None: blocks = job.get("blocks") or 0 return f"finding the target: {blocks} block{'' if blocks == 1 else 's'}, {speed}" @@ -813,17 +1002,20 @@ def format_progress(job: dict, width: int = 20) -> str: def format_job(job: dict) -> str: - """One `rnfs jobs` line: ID, state, kind, progress, target (and output).""" + """One `rnfs jobs` line: ID, state, kind, progress, target (and output), or an add's or import's path.""" where = f" -> {job['output']}" if job.get("output") else "" - return printable(f"{job['id']} {job['state']:<9} {job['kind']:<6} {format_progress(job)} {job['target']}{where}") + subject = job.get("target") or job.get("path") + return printable(f"{job['id']} {job['state']:<9} {job['kind']:<6} {format_progress(job)} {subject}{where}") def job_fields(job: dict) -> dict: """A job's fields for `rnfs jobs show`, readable.""" - fields = {"id": job["id"], "kind": job["kind"], "state": job["state"], "target": job["target"]} - for name in ("output", "from", "sealed_name", "fallback_dht"): + fields = {"id": job["id"], "kind": job["kind"], "state": job["state"]} + for name in ("target", "path", "output", "from", "sealed_name", "fallback_dht"): if job.get(name): fields[name] = job[name] + if job.get("options"): + fields["options"] = json.dumps(job["options"]) fields["progress"] = format_progress(job) total = job.get("total_bytes") fields["bytes"] = f"{job.get('done_bytes') or 0} of {total if total is not None else '?'}" @@ -942,7 +1134,9 @@ def has_jobs(api) -> bool: def run_transfer(args, fields: dict, fallback) -> dict | None: """Run a transfer as a job and follow it; its result, or None with --background (its ID is printed). - fallback(api) does the transfer in one request instead, for a daemon without the job manager. + fallback(api) does the transfer in one request instead, for a daemon without the job manager (None: only with + --background, which fails then). A daemon whose job manager predates the kind refuses it (400, kind must be one + of ...). """ from rnfs import cli api = cli.client(args) @@ -974,6 +1168,10 @@ def print_result(job: dict): print(cli.printable(result.get("output") or job["output"])) elif job["kind"] == "pin": print(cli.printable(result.get("cid"))) + elif job["kind"] == "add": + cli.print_added(result) + elif job["kind"] == "import": + cli.print_imported(result) else: print(cli.printable(f"exported {result.get('blocks')} blocks of {', '.join(result.get('roots') or [])} " f"to {job['output']}")) diff --git a/rnfs/features/lxmf.py b/rnfs/features/lxmf.py index 65b5207..48cecc6 100644 --- a/rnfs/features/lxmf.py +++ b/rnfs/features/lxmf.py @@ -21,6 +21,12 @@ batches, each save writing the records changed since the last; outbox messages still unfinished at a start are marked failed (LXMF doesn't persist its outbound queue). - Messages waiting at lxmf_propagation_node are downloaded every lxmf_sync_interval. +- Changes are published as "inbox" events (rnfs.events), each with "box" ("inbox" or "outbox") and "change": + inbox "received" ({"id", "from"}), "removed" ({"ids"}: by inbox rm, one event for --all, or dropped to make room) + and "auto" ({"id", "index", "state"}: an auto-fetch result recorded); outbox "queued" ({"id"}, by send), "state" + ({"id", "state"}: its delivery state changed, or it fell back to propagation) and "removed" ({"ids"}: dropped to + make room). Progress alone publishes nothing. The inbox records a start restores (_record_kept) publish nothing: + a client reloads at a start anyway (the events' epoch changes). Everything received is untrusted: fields are parsed strictly and capped, text is capped and escaped when printed. A message is only accepted once LXMF has validated its signature against the sender's identity; one @@ -44,7 +50,7 @@ from dataclasses import dataclass from urllib.parse import quote, unquote_to_bytes -from rnfs import features, fsutil, ids +from rnfs import events, features, fsutil, ids from rnfs.features import _common from rnfs.features._common import LOG_ERROR, LOG_INFO, LOG_NOTICE, LOG_VERBOSE, LOG_WARNING from rnfs.constants import ID_LENGTH, MAX_DEPTH @@ -976,6 +982,16 @@ def _propagated_sent(entry: dict) -> bool: """A propagated message is done once the propagation node has it.""" return entry.get("method") == "propagated" and entry.get("state") == "sent" + def _event(self, box: str, change: str, **fields): + """Publish an "inbox" event about box (INBOX or OUTBOX); see the module docstring. Safe with the lock held: + publishing only appends to the node's event log.""" + events.publish(self.node, "inbox", box=box, change=change, **fields) + + def _removed_event(self, box: str, removed: list[dict]): + """Publish one "removed" event for the records of box removed or dropped together, if any.""" + if removed: + self._event(box, "removed", ids=[entry["id"] for entry in removed]) + # Lifecycle def start(self): @@ -1178,7 +1194,10 @@ def send(self, to: str, target: str, text: str | None = None, propagate: bool = method = "propagated" if propagate else "direct" entry = {"hash": message_hash.hex(), "to": address.hex(), "time": time.time(), "links": [link.to_json()], "text": text or "", "method": method, "state": "outbound", "progress": 0.0} - entry_id, _ = self.outbox.add(entry) + entry_id, dropped = self.outbox.add(entry) + self._removed_event(OUTBOX, dropped) + if entry_id is not None: + self._event(OUTBOX, "queued", id=entry_id) if not propagate: with self._lock: self._resend[message_hash.hex()] = (address, content, fields, nearest) @@ -1202,17 +1221,19 @@ def outbound_changed(self, message_hash: bytes, state: str, progress: float | No elif network is not None: try: new_hash = network.send(resend[0], resend[1], resend[2], True, nearest) - if entry is not None: - self.outbox.update(entry["id"], hash=new_hash.hex(), method="propagated", - state="outbound", progress=0.0, direct_hash=message_hash.hex(), - fallback=True) + if entry is not None and self.outbox.update(entry["id"], hash=new_hash.hex(), + method="propagated", state="outbound", + progress=0.0, direct_hash=message_hash.hex(), + fallback=True): + self._event(OUTBOX, "state", id=entry["id"], state="outbound") _log(f"direct delivery of {message_hash.hex()} failed; sending it through the propagation " f"node as {new_hash.hex()}", LOG_NOTICE) return except Exception as e: # noqa: BLE001 values["error"] = f"propagation fallback failed: {fsutil.printable(e)}" - if entry is not None: - self.outbox.update(entry["id"], **values) + # Progress alone (polled each second while sending) publishes nothing + if entry is not None and self.outbox.update(entry["id"], **values) and entry.get("state") != state: + self._event(OUTBOX, "state", id=entry["id"], state=state) # Receiving @@ -1269,6 +1290,10 @@ def received(self, message: Incoming) -> int | None: reserved=lambda record: self._trusted_hex(record.get("from")), share=self._auto_share() if auto else None) self._release(dropped) + self._removed_event(INBOX, dropped) + if entry_id is not None: + # Under the lock, so an inbox rm of it can't publish "removed" first + self._event(INBOX, "received", id=entry_id, **{"from": sender}) if entry_id is None: _log(f"message {message_hash} from {sender} not kept: the inbox is full of messages from trusted " f"senders or holding auto-fetched pins (rnfs inbox rm)", LOG_NOTICE) @@ -1359,6 +1384,7 @@ def inbox_rm(self, entry_id: int | None) -> int: with self._lock: removed = self.inbox.remove(entry_id) self._release(removed) + self._removed_event(INBOX, removed) # one event, whatever their number if entry_id is not None and not removed: from rnfs.errors import NotFound raise NotFound(f"no inbox message #{entry_id}") @@ -1567,7 +1593,8 @@ def _set_auto(self, entry_id: int, index: int, result: dict): auto = list(entry.get("auto") or []) if 0 <= index < len(auto): auto[index] = result - self.inbox.update(entry_id, auto=auto) + if self.inbox.update(entry_id, auto=auto): + self._event(INBOX, "auto", id=entry_id, index=index, state=result.get("state")) def auto_fetch(self, entry_id: int, index: int): """Fetch one link of an inbox message and pin it with source lxmf:, within the sender's quota. diff --git a/rnfs/features/peers.py b/rnfs/features/peers.py index d07775c..4009039 100644 --- a/rnfs/features/peers.py +++ b/rnfs/features/peers.py @@ -5,17 +5,44 @@ this feature reads them. The node's rnfs.dht announce handler counts every announce in them too (Traffic.heard), so the DHT mode and the last-heard time of client-mode nodes (which never enter the routing table) are known. +Per peer the listing also gives: +- first_heard: the earliest of when this run first counted the peer, when it entered the routing table (kept + across a short absence) and when RNS's announce cache (Identity.known_destinations, which RNS persists) last + heard it; null if none knows. +- rtt, rtt_at: the round trip RNS measured on the last link established to it (a shared link, ping's, or an + identified one for /provide or /pin; LinkPool.on_rtt), in seconds, and when; null if none was. +- online: mesh.online(), the one rule: an rnfs link to it is open, or it showed it was up within + mesh.ONLINE_WINDOW (15 minutes, on the monotonic clock): an answer to a request (a refusal too), a request of its + own served, a link established, or an announce of its own emitted within the window (its signed emission time, + allowing mesh.EMISSION_SKEW of clock difference, and later than the last counted: Traffic.announced). A path + response, or an announce held in a queue or passed on late by a shared instance, says nothing of now. No network + I/O. + +"peer" events (rnfs.events): {"node_id", "change": "new" | "online" | "offline", "online"}. A thread checks every +CHECK_INTERVAL (offline comes with time passing, so it can't be a hook) and diffs the online states of the peers +Traffic counts and those with an open link, both bounded (not the listing, which the routing table may fill: a peer +only in the routing table is never online). Its first check, at start, only records them, and "new" is for a peer +first counted in Traffic (first heard or contacted this run) after that, so neither the start nor the routing table +seeded from RNS's announce cache floods the log: a client lists the peers once, then follows the events, taking +"new" as add-or-update (a peer listed from the routing table gets "new" once it is first heard). A new peer's event +says whether it is online and no "online" follows for it; one online that is no longer counted nor linked gets +"offline"; a peer forgotten beyond mesh.MAX_PEERS and heard again is "new" again. + Nothing is persisted: counts are since the daemon started. Memory is bounded by mesh.MAX_PEERS. """ import json import math +import threading import time -from rnfs import mesh +from rnfs import events, mesh from rnfs.features import Feature, _common from rnfs.fsutil import printable NAME = "peers" +# Seconds between checks of the peers' online states for "peer" events: an offline event is at most this late after +# mesh.ONLINE_WINDOW has passed, and a check reads only counters and the pool's open links +CHECK_INTERVAL = 30.0 class PeersFeature(Feature): @@ -31,6 +58,69 @@ def __init__(self, node): from rnfs.log import ThrottledLog # here, not at module level: it loads RNS, and the CLI imports features # A listing that can't read something is a bug or an RNS change, met per peer: a line a minute self._failure_log = ThrottledLog() + self._stop = threading.Event() + # The online state of each peer at the last check (None before the first), and that check's time + # (Traffic.monotonic): only a peer counted after the first is "new". Only the check thread (or a test) uses them + self._states: dict[bytes, bool] | None = None + self._since: float | None = None + + # Lifecycle + + def start(self): + """Start the thread publishing "peer" events (an offline test node calls check() itself).""" + self._spawn(self._run, "rnfs-peers") + + def stop(self, deadline: float | None = None): + """Stop the events thread, waiting for it until deadline (as Feature.stop).""" + self._stop.set() + self._join(deadline) + + def _run(self): + """check() now (recording the states without events), then every CHECK_INTERVAL until stop.""" + while True: + try: + self.check() + except Exception as e: # noqa: BLE001 - the thread keeps running; logged + self._failed(f"couldn't check the peers' online states: {e!r}") + if self._stop.wait(CHECK_INTERVAL): + return + + def check(self): + """Publish a "peer" event for each peer new since the last check (first counted after the first check) or + whose online state changed; the first check only records the states. See the module docstring. + + Over the peers Traffic counts and those with an open link (both bounded), not the listing, which the + routing table may fill to mesh.MAX_PEERS: a peer cut from it isn't offline. A peer only in the routing table + is never online (nothing of it was seen), so leaving it out loses no event. Decided by Traffic.monotonic.""" + now = self.traffic.monotonic() + counters = self.traffic.peers() + linked = self._linked() + watched = [node_id for node_id in dict.fromkeys([*linked, *counters]) if node_id != self.node.node_id] + states = {node_id: mesh.online(node_id in linked, (counters.get(node_id) or {}).get("last_seen"), now) + for node_id in watched} + previous, self._states = self._states, states + if previous is None: + self._since = now + return + for node_id, online in states.items(): + before = previous.get(node_id) + if before is None: + first = (counters.get(node_id) or {}).get("first_counted") + if first is not None and first >= self._since: + change = "new" + elif online: # counted before the first check, without a link then (forgotten, counted again) + change = "online" + else: + continue + elif before != online: + change = "online" if online else "offline" + else: + continue + events.publish(self.node, "peer", node_id=node_id.hex(), change=change, online=online) + # One no longer watched (its link closed with nothing counted, or forgotten beyond mesh.MAX_PEERS) is offline + for node_id, before in previous.items(): + if before and node_id not in states: + events.publish(self.node, "peer", node_id=node_id.hex(), change="offline", online=False) def status(self) -> dict: """Traffic totals and airtime refusals for `rnfs status`.""" @@ -49,19 +139,31 @@ def _budget_remaining(self) -> int | None: # Queries + def _linked(self) -> set[bytes]: + """The nodes with an open shared link (empty, logged, if they can't be listed).""" + try: + return set(self.node.client.open_links()) + except Exception as e: # noqa: BLE001 - a listing shows what it can; logged + self._failed(f"couldn't list the open links: {e!r}") + return set() + + def _known(self, counters: dict, linked: set[bytes]) -> list[bytes]: + """The peers listed: in the routing table, with an open link, or in counters (most recent first); this node + left out, at most mesh.MAX_PEERS.""" + node = self.node + ids = [n for n in dict.fromkeys([*node.routing.nodes(), *linked, *reversed(list(counters))]) + if n != node.node_id] + return ids[:mesh.MAX_PEERS] + def peers(self) -> list[dict]: """Every known node: in the routing table, with an open link, or in the traffic counters; nearest first.""" node = self.node routing = node.routing paths = getattr(node, "paths", None) counters = self.traffic.peers() - try: - linked = set(node.client.open_links()) - except Exception as e: # noqa: BLE001 - a listing shows what it can; logged - self._failed(f"couldn't list the open links: {e!r}") - linked = set() - ids = [n for n in dict.fromkeys([*routing.nodes(), *linked, *reversed(list(counters))]) if n != node.node_id] - ids = ids[:mesh.MAX_PEERS] + linked = self._linked() + ids = self._known(counters, linked) + now, monotonic = self.traffic.clock(), self.traffic.monotonic() # wall time to show, monotonic to decide # All at once: attached to a shared instance, RnsPaths reads the interfaces from one RPC, not one each known = paths.infos(ids) if paths is not None else {} peers = [] @@ -69,6 +171,7 @@ def peers(self) -> list[dict]: counts = counters.get(node_id) or {} seniority = routing.seniority(node_id) path = known.get(node_id, mesh.NO_PATH) + recalled = self._recalled_heard(node_id) # RNS's announce cache: one lookup for both times rate = None if node_id in linked: try: @@ -84,11 +187,15 @@ def peers(self) -> list[dict]: "senior": seniority is not None and seniority >= node.config.provider_seniority, "hops": path.hops, "interface": path.interface, "bitrate": path.bitrate, "link": node_id in linked, "rate": rate, - "last_heard": counts.get("last_heard") or self._recalled_heard(node_id), + "last_heard": counts.get("last_heard") or recalled, "last_active": counts.get("last_active"), "banned": routing.is_banned(node_id), **{key: counts.get(key, 0) for key in ("sent", "received", "requests", "answered", "refused", "failed", "bad")}, + "first_heard": _earliest(counts.get("first_heard"), + None if seniority is None else now - seniority, recalled), + "rtt": counts.get("rtt"), "rtt_at": counts.get("rtt_at"), + "online": mesh.online(node_id in linked, counts.get("last_seen"), monotonic), }) peers.sort(key=lambda p: (p["hops"] if p["hops"] is not None else math.inf, p["node_id"])) return peers @@ -146,12 +253,20 @@ def api_routes(cls) -> dict: def cli_commands(cls, commands, command): """rnfs peers [--json] and rnfs stats [--json].""" peers = command(commands, "peers", run_peers, - "known nodes: mode, seniority, hops, interface, link rate, last heard, traffic") + "known nodes: mode, seniority, hops, interface, link rate, last heard, traffic, " + "round trip, online") peers.add_argument("--json", action="store_true", help="print JSON") stats = command(commands, "stats", run_stats, "traffic since start: totals, refusals, per interface") stats.add_argument("--json", action="store_true", help="print JSON") +def _earliest(*times: float | None) -> float | None: + """The earliest of times that aren't None, or None. first_heard: each source (this run's counters, the routing + table, RNS's announce cache) knows of a time the peer had been heard by, so the earliest is the best answer.""" + known = [value for value in times if value is not None] + return min(known) if known else None + + def _api_peers(handler): """GET /api/peers: {"peers": [...]}.""" handler.send_json(200, {"peers": _common.feature(handler, NAME).peers()}) @@ -197,22 +312,31 @@ def human_age(seconds: float | None) -> str: return f"{seconds // 86400:.0f}d" +def human_rtt(seconds) -> str: + """A round-trip time like "85ms" or "2.4s", or "-" for None (or anything but a number).""" + if not isinstance(seconds, (int, float)) or isinstance(seconds, bool) or not 0 <= seconds < math.inf: + return "-" + return f"{seconds * 1000:.0f}ms" if seconds < 1 else f"{seconds:.1f}s" + + def format_peers(peers: list[dict], now: float) -> list[str]: - """The `rnfs peers` table, one line per node, escaped (the daemon's fields mustn't drive the terminal).""" + """The `rnfs peers` table, one line per node, escaped (the daemon's fields mustn't drive the terminal). RTT and + ONLINE come last, so the older columns keep their places; a daemon before 0.1.7 sends neither ("-").""" lines = [f"{'NODE':<32} {'MODE':<7} {'TABLE':>6} {'HOPS':>4} {'INTERFACE':<20} {'RATE':>10} {'HEARD':>6} " - f"{'SENT':>9} {'RECV':>9} {'FAIL':>5}"] + f"{'SENT':>9} {'RECV':>9} {'FAIL':>5} {'RTT':>7} ONLINE"] for peer in peers: table = human_age(peer["seniority"]) if peer["in_table"] else "-" heard = human_age(now - peer["last_heard"]) if peer.get("last_heard") else "-" rate = human_rate(peer["rate"]) if peer["link"] else "-" interface = (peer.get("interface") or "-")[:20] + online = {True: "yes", False: "no"}.get(peer.get("online"), "-") flags = (" banned" if peer["banned"] else "") + (f" bad:{peer['bad']}" if peer.get("bad") else "") # Each text field escaped before it is padded, so the columns line up; the whole line again for the rest lines.append(printable( f"{printable(peer['node_id']):<32} {printable(peer['mode'] or '?'):<7} {table:>6} " f"{peer['hops'] if peer['hops'] is not None else '?':>4} {printable(interface):<20} " f"{rate:>10} {heard:>6} {human_size(peer['sent']):>9} {human_size(peer['received']):>9} " - f"{peer['failed']:>5}{flags}")) + f"{peer['failed']:>5} {human_rtt(peer.get('rtt')):>7} {online}{flags}")) return lines diff --git a/rnfs/features/settings.py b/rnfs/features/settings.py new file mode 100644 index 0000000..d3b43b4 --- /dev/null +++ b/rnfs/features/settings.py @@ -0,0 +1,449 @@ +"""The settings feature: `rnfs config` and `GET /api/config`, `POST /api/config/set` — the settings with their +schema, read and changed through the daemon, some applied without a restart. + +- The schema (types, defaults, units, ranges, descriptions, groups, which apply live) is rnfs.config's + settings_schema(), standard library only, so an app or the CLI needs no copy of it and no daemon to read it. +- `GET /api/config` lists every setting with the running node's value, its value in rnfs.toml and whether a + restart would change it (pending). The daemon's command line (`--api-port`, `--private`, ...; an embedded node's + settings) wins over rnfs.toml, as at start: such a setting is "overridden", and a restart with the same command + line keeps the command line's value. +- `POST /api/config/set` checks the whole resulting configuration as Config.load would (unknown names, types, + ranges, the rules between settings) before writing anything, then rewrites rnfs.toml atomically (owner-only; + read through config.open_checked, so a symlink, a file of another user's or one over the size limit is refused, + and a symlink at its path is replaced, never written through), keeping comments and other lines + (config.edit_settings). Then it applies the settings in LIVE to the running node. rnfs_dir is refused: rnfs.toml + lives in it. +- What applies live is only what was checked to: most of LIVE is read from node.config at each use (the node and + its components share that one Config object), so assigning the attribute is enough; the rest copy the value at + startup and get it here too (each hook below says what). Settings that choose identity, ports, Reticulum, the + DHT mode or private mode and its members are never live, nor are those whose old value is persisted with what + it made. + +Whoever holds the API token can already read and write any file as the daemon's user, so editing rnfs.toml through +the API gives them nothing new; every check Config.load makes still applies. +""" +import copy +import datetime +import json +import math +import threading + +from rnfs import config as settings +from rnfs.errors import Busy +from rnfs.features import Feature, _common +from rnfs.fsutil import printable + +NAME = "settings" + + +class SettingsFeature(Feature): + """Reads and changes rnfs.toml for the API, applying LIVE settings to the running node. Thread-safe.""" + + name = NAME + # What some live settings are copied into: the job manager, the keeper store, LXMF's sender lists + optional = ("jobs", "keepers", "lxmf") + + def __init__(self, node): + """Nothing to load: rnfs.toml is read at each request.""" + super().__init__(node) + self._lock = threading.Lock() # one change at a time: read, check, write and apply + + # Reading + + def describe(self) -> dict: + """GET /api/config's answer: {"path", "settings": [schema entry + "value", "file", "overridden", + "pending"], "file_error", "warnings"} (see the module docstring and settings_entry).""" + config = self.node.config + path = settings.settings_path(config.rnfs_dir) + warnings: list[str] = [] + file_values: dict = {} + file_error = None + try: + data, warnings = settings.read_settings_file(path) + file_values = settings.parse_settings(data, path) + # What a restart would make of it, so a file it would refuse says so before the restart does + restart_config(config, file_values, path) + except (ValueError, TypeError, AttributeError, OSError) as e: # OSError: PermissionError too (untrusted) + file_error = str(e) + entries = [settings_entry(entry, config, file_values) for entry in settings.settings_schema()] + return {"path": path, "settings": entries, "file_error": file_error, "warnings": warnings} + + # Changing + + def set(self, changes) -> dict: + """Set the settings in changes (name -> value, None: removed from rnfs.toml, so back to the default) in + rnfs.toml, then apply those in LIVE that the command line doesn't override. Nothing is written unless the + whole result is valid. Returns {"path", "applied": [applied now], "restart_required": [whose new value + applies at the next start], "overridden": [set on the daemon's command line, which keeps winning], + "warnings"}, names in the order given; a setting set to the value running is in none of the first two. + A name that isn't a setting (or rnfs_dir) may only be removed, and only if rnfs.toml has it: so a file a + start refuses for it can be mended. Raises ValueError (refused), Busy (rnfs.toml changed meanwhile), + OSError (it can't be written).""" + if not isinstance(changes, dict) or not changes: + raise ValueError("settings must be a JSON object of setting names and values (null: back to the " + "default)") + known = {entry["name"] for entry in settings.settings_schema()} + with self._lock: + running = self.node.config + path = settings.settings_path(running.rnfs_dir) + try: + data, warnings = settings.read_settings_file(path) + except PermissionError as e: # not a file rnfs trusts: a symlink, another user's, not a regular file + raise ValueError(f"can't edit the settings file: {e}") from None + values = settings.parse_settings(data, path) + for name, value in changes.items(): + if name not in known and not (value is None and name in values): + if name == "rnfs_dir": + raise ValueError("rnfs_dir can't be set in rnfs.toml; use --dir") + raise ValueError(f"unknown setting {name!r}") + if value is None: + values.pop(name, None) + else: + values[name] = value + try: + candidate = restart_config(running, values, path) + except PermissionError as e: # the Reticulum config it would use can't be trusted + raise ValueError(str(e)) from None + except (TypeError, AttributeError) as e: # a value of a type a check didn't expect + raise ValueError(f"invalid settings: {e}") from None + text = settings.edit_settings(data.decode("utf-8") if data is not None else "", changes) + new_data = text.encode("utf-8") + if settings.parse_settings(new_data, path) != values: # not expected: edit_settings checks each step + raise ValueError("rnfs.toml can't be rewritten without changing other settings; edit it by hand") + if len(new_data) > settings.MAX_SETTINGS_SIZE: + raise ValueError(f"the settings would be larger than {settings.MAX_SETTINGS_SIZE} bytes") + changes = {name: value for name, value in changes.items() if name in known} # what's left: settings + live = [name for name in changes if name in LIVE and name not in running.overrides] + # The running settings with the live ones changed must hold together too (they do whenever the file's + # do, as no rule spans a live and another setting; checked so a later rule can't break that unseen) + mixed = copy.copy(running) + for name in live: + setattr(mixed, name, getattr(candidate, name)) + mixed.validate() + # A hand edit since the read would be lost: refused, for the caller to try again on the new file + if settings.read_settings_file(path)[0] != data: + raise Busy("rnfs.toml changed while it was being edited; try again") + from rnfs import fsutil + fsutil.write_atomic(path, new_data, mode=0o600, private=True) + restart = [name for name in changes if name not in live and name not in running.overrides + and not same_value(getattr(candidate, name), getattr(running, name))] + for name in live: + self._apply(name, getattr(candidate, name)) + overridden = [name for name in changes if name in running.overrides] + _common.log(f"settings: rnfs.toml changed ({', '.join(changes)}); applied now: {', '.join(live) or 'none'}" + f"; at the next start: {', '.join(restart) or 'none'}", _common.LOG_NOTICE) + return {"path": path, "applied": live, "restart_required": restart, "overridden": overridden, + "warnings": warnings} + + def _apply(self, name: str, value): + """Run the node with setting name at value (already checked): node.config changed, then the hook that + hands it to what copied it at start, if any.""" + setattr(self.node.config, name, value) + hook = LIVE[name] + if hook is not None: + hook(self, value) + + # API + + @classmethod + def api_routes(cls) -> dict: + """GET /api/config and POST /api/config/set.""" + return {("GET", "/api/config"): _api_config, ("POST", "/api/config/set"): _api_config_set} + + # CLI + + @classmethod + def cli_commands(cls, commands, command): + """rnfs config [list|get|set|unset].""" + import argparse + config = command(commands, "config", run_list, + "list the daemon's settings (* set in rnfs.toml, ! restart needed, ^ command line)") + config.add_argument("--json", action="store_true", help="print JSON, with each setting's schema") + sub = config.add_subparsers(dest="config_command", metavar="command") + # SUPPRESS: unset after the subcommand, --json keeps what it was given before it (`config --json get x`) + listing = command(sub, "list", run_list, "list the settings") + listing.add_argument("--json", action="store_true", default=argparse.SUPPRESS, help="print JSON") + get = command(sub, "get", run_get, "show one setting: value, default, type, whether it applies at once") + get.add_argument("name") + get.add_argument("--json", action="store_true", default=argparse.SUPPRESS, help="print JSON") + set_ = command(sub, "set", run_set, "set a setting in rnfs.toml and apply it if it can be applied at once") + set_.add_argument("name") + set_.add_argument("value", help='a TOML value: true, 10, 0.5, "text", ["a", "b"]; a bare word is text') + unset = command(sub, "unset", run_unset, "remove a setting from rnfs.toml (back to its default)") + unset.add_argument("name") + + +# Live hooks: what copied the setting at start gets the new value. node.config already holds it + + +def _max_storage(feature: SettingsFeature, value: int): + """The store's own copy (Storage.max_storage, which eviction uses), then max_storage enforced now rather than + at the next maintenance pass.""" + node = feature.node + node.storage.max_storage = value + try: + node._enforce_storage() + except Exception as e: # noqa: BLE001 - the setting is applied; the maintenance pass evicts again; logged + _common.log(f"settings: enforcing max_storage failed: {e!r}", _common.LOG_ERROR) + + +def _max_jobs(feature: SettingsFeature, value): + """Start queued jobs a higher max_jobs allows now (fewer running ones finish first, none is stopped).""" + jobs = feature.dependency("jobs") + if jobs is not None: + jobs._schedule() + + +def _airtime(feature: SettingsFeature, value): + """The request server's airtime policy copies serve_interfaces, no_serve_interfaces and quiet_hours (as a + parsed list): each replaced whole, so a request sees the old value or the new one. None without a server + (an offline node).""" + from rnfs import mesh + airtime = getattr(getattr(feature.node, "server", None), "airtime", None) + if airtime is None: + return + config = feature.node.config + airtime.serve = list(config.serve_interfaces) + airtime.no_serve = list(config.no_serve_interfaces) + airtime.quiet = mesh.parse_quiet_hours(config.quiet_hours) + + +def _keep_max_names(feature: SettingsFeature, value: int): + """The keeper store's cap. Names kept beyond a lower one stay kept, as at a start; only new ones are refused.""" + keepers = feature.dependency("keepers") + if keepers is not None: + keepers.store.max_names = value + + +def _lxmf_senders(attribute: str): + """A hook replacing LXMF's set of normalised addresses attribute (accept_from or auto_fetch_from) whole. A + sender taken off lxmf_auto_fetch_from has its queued links refused when their turn comes, as at a start.""" + + def hook(feature: SettingsFeature, value: list[str]): + """Replace the set.""" + lxmf = feature.dependency("lxmf") + if lxmf is not None: + from rnfs.features.lxmf import parse_address + setattr(lxmf, attribute, {parse_address(address).hex() for address in value}) + + hook.__name__ = f"_lxmf_{attribute}" + return hook + + +# Settings applied at once: name -> hook (None: read from node.config at each use, which is enough). Each was +# checked against the code using it; config.LIVE_SETTINGS names exactly these (tests/test_settings.py checks) +LIVE = { + "max_storage": _max_storage, # eviction: Storage's copy, and enforced now + "max_download_size": None, # Fetcher, at each size check + "max_output_entries": None, # the fetcher, walker calls and exports, at each use + "max_upload_rate": None, # RequestServer, at each /blocks response (effective_max_upload_rate) + "remote_pin_quota": None, # RnfsNode, at each remote pin + "scan_interval": None, # the maintenance pass + "max_jobs": _max_jobs, # JobsFeature.max_jobs, at each scheduling + "max_finished_jobs": None, # JobsFeature._cap_finished, when a job finishes + "serve_interfaces": _airtime, + "no_serve_interfaces": _airtime, + "quiet_hours": _airtime, + "keep_max_names": _keep_max_names, + "keeper_query": None, # KeepersFeature.always_ask, at each resolve + "follow_quota": None, # FollowFeature, at each fetch of a new target + "lxmf_accept_from": _lxmf_senders("accept_from"), + "lxmf_auto_fetch_from": _lxmf_senders("auto_fetch_from"), + "lxmf_auto_fetch_quota": None, # LxmfFeature._fetch_and_pin + "lxmf_send_from_hint": None, # LxmfFeature, at each link sent +} + + +# Shared by the API and the CLI (no RNS here: the CLI must not load it) + + +def restart_config(running: settings.Config, values: dict, path: str) -> settings.Config: + """The Config a restart of the node running with running would load from rnfs.toml's values (read from + path): with the command line's options (overrides) and the embedded node's defaults as at start, checked as + Config.load checks it, the Reticulum config included. rnfs.toml on its own is checked too: a start without the + command line's options, and the CLI and rnfs.Client (which read it for the API port), must accept it whatever + the overrides hide. Raises ValueError, PermissionError (the Reticulum config), OSError; TypeError or + AttributeError for a value of a type a check didn't expect.""" + alone = settings.Config.combine(running.rnfs_dir, values, dict(running.given_defaults), None, path) + alone.check_reticulum_config() + config = settings.Config.combine(running.rnfs_dir, values, dict(running.given_defaults), dict(running.overrides), + path) + config.check_reticulum_config() + return config + + +def same_value(a, b) -> bool: + """Whether two setting values are the same: equal, and not a flag against a number (True == 1 to Python).""" + return a == b and isinstance(a, bool) == isinstance(b, bool) + + +def jsonable(value): + """value with what JSON can't carry made text: TOML dates and times in a hand-edited rnfs.toml (ISO 8601), and + its inf and nan ("inf", "nan": json.dumps would write Infinity and NaN, which aren't JSON).""" + if isinstance(value, (datetime.date, datetime.time)): + return value.isoformat() + if isinstance(value, float) and not math.isfinite(value): + return str(value) + if isinstance(value, list): + return [jsonable(item) for item in value] + if isinstance(value, dict): + return {str(key): jsonable(item) for key, item in value.items()} + return value + + +def settings_entry(entry: dict, config: settings.Config, file_values: dict) -> dict: + """A setting's schema entry with: "value" (the running node's), "file" (rnfs.toml's, or None where it doesn't + set it: TOML has no null), "overridden" (the daemon's command line, or an embedded node's settings, set it, + and wins over rnfs.toml) and "pending" (a restart with the same command line would run another value). An + embedded node's own defaults (given_defaults, e.g. dht_mode "client") replace the schema's "default".""" + name = entry["name"] + value = getattr(config, name) + default = config.given_defaults.get(name, entry["default"]) + overridden = name in config.overrides + if overridden: + restart = config.overrides[name] + elif name in file_values: + restart = file_values[name] + else: + restart = default + return {**entry, "default": jsonable(default), "value": jsonable(value), "file": jsonable(file_values.get(name)), + "overridden": overridden, "pending": not same_value(restart, value)} + + +def _api_config(handler): + """GET /api/config: every setting with its schema, running value and rnfs.toml's (SettingsFeature.describe).""" + handler.send_json(200, _common.feature(handler, NAME).describe()) + + +def _api_config_set(handler): + """POST /api/config/set {"settings": {name: value or null}}: rnfs.toml changed, live ones applied.""" + body = handler.json_body() + handler.send_json(200, _common.feature(handler, NAME).set(body.get("settings"))) + + +# CLI + + +def parse_value(text: str): + """A setting's value from the command line: TOML (true, 10, 0.5, "text", ["a", "b"]), or the text itself if + it isn't one TOML value (a bare word such as auto or 22:00-06:00, or a TOML date, which no setting takes).""" + import tomllib + try: + parsed = tomllib.loads(f"value = {text}") + except tomllib.TOMLDecodeError: + return text + if parsed.keys() != {"value"} or isinstance(parsed["value"], (datetime.date, datetime.time, dict)): + return text + return parsed["value"] + + +def show_value(value) -> str: + """A value as rnfs.toml writes it, or "(unset)" for None; escaped (the daemon's text mustn't drive the + terminal).""" + if value is None: + return "(unset)" + try: + return printable(settings.toml_value(value)) + except ValueError: + return printable(json.dumps(value)) + + +def format_settings(result: dict) -> list[str]: + """The `rnfs config` listing: a line per setting, " name = value", the marker * for one set in + rnfs.toml, ! for one a restart would change (with rnfs.toml's value), ^ for one the command line sets; then + a legend for the markers used. Escaped.""" + lines, used = [], set() + for entry in result["settings"]: + marker = "^" if entry["overridden"] else "!" if entry["pending"] else "*" if entry["file"] is not None else " " + used.add(marker) + line = f"{marker} {printable(entry['name'])} = {show_value(entry['value'])}" + if marker == "!": + line += f" (rnfs.toml: {show_value(entry['file'])}; applies at the next start)" + lines.append(line) + legend = {"*": "* set in rnfs.toml", "!": "! changed in rnfs.toml, applies at the next start", + "^": "^ set on the daemon's command line, which wins over rnfs.toml"} + if used - {" "}: + lines.append("") + lines.extend(text for marker, text in legend.items() if marker in used) + return lines + + +def setting_fields(entry: dict) -> dict: + """One setting for `rnfs config get`, as "name: value" fields.""" + limits = [] + if entry["minimum"] is not None: + limits.append(f"{'above' if entry['exclusive_minimum'] else 'at least'} {entry['minimum']}") + if entry["maximum"] is not None: + limits.append(f"at most {entry['maximum']}") + kind = entry["type"] + (f" ({', '.join(entry['choices'])})" if entry["choices"] else "") + fields = {"name": entry["name"], "value": show_value(entry["value"]), + "rnfs.toml": show_value(entry["file"]) if entry["file"] is not None else "(not set)", + "default": show_value(entry["default"]), + "type": kind + (", or unset" if entry["nullable"] else "")} + if entry["unit"]: + fields["unit"] = entry["unit"] + if limits: + fields["range"] = " and ".join(limits) + fields["applies"] = "at once" if entry["live"] else "at the next start" + if entry["overridden"]: + fields["overridden"] = "by the daemon's command line" + if entry["pending"]: + fields["pending"] = "rnfs.toml's value applies at the next start" + fields["description"] = entry["description"] + return fields + + +def print_change(result: dict, name: str): + """What `config set` or `config unset` did to name.""" + from rnfs import cli + cli.print_warnings(result) + if name in result["applied"]: + print(f"{printable(name)}: saved in rnfs.toml and applied") + elif name in result["restart_required"]: + print(f"{printable(name)}: saved in rnfs.toml; restart the daemon to apply it") + elif name in result["overridden"]: + print(f"{printable(name)}: saved in rnfs.toml, but the daemon's command line sets it, which wins") + else: + print(f"{printable(name)}: saved in rnfs.toml (the daemon already runs with that value)") + + +def run_list(args): + """Print the settings.""" + from rnfs import cli + result = cli.client(args).config() + if args.json: + print(json.dumps(result, indent=2)) + return + for line in format_settings(result): + print(line) + cli.print_warnings(result) + if result.get("file_error"): + cli.warn(f"rnfs.toml: {result['file_error']}") + + +def run_get(args): + """Print one setting.""" + from rnfs import cli + result = cli.client(args).config() + entry = next((entry for entry in result["settings"] if entry["name"] == args.name), None) + if entry is None: + cli.fail(f"unknown setting {args.name!r}") + if args.json: + print(json.dumps(entry, indent=2)) + return + cli.print_fields(setting_fields(entry)) + + +def run_set(args): + """Set one setting.""" + from rnfs import cli + print_change(cli.client(args).config_set({args.name: parse_value(args.value)}), args.name) + + +def run_unset(args): + """Remove one setting from rnfs.toml.""" + from rnfs import cli + print_change(cli.client(args).config_set({args.name: None}), args.name) + + +FEATURE = SettingsFeature diff --git a/rnfs/interface.py b/rnfs/interface.py index 66e9fcf..c3a026d 100644 --- a/rnfs/interface.py +++ b/rnfs/interface.py @@ -7,7 +7,13 @@ A target is a Cid, `[/name…]` (whose first part may be a name contact's nick), or an rnfs: URI (section 13): a name URI's name is resolved first (persisted, as names.resolve does), and its from= is the from_node of the -methods taking one. Node IDs are 16 bytes or 32 hex characters; names 16 bytes, 32 hex characters or a Name. +methods taking one. Node IDs are 16 bytes or 32 hex characters; names 16 bytes, 32 hex characters or a Name, and a +name contact's nick where the API takes one (resolve, keep, follow); an LXMF recipient 16 bytes, 32 hex characters +or an LXMF contact's nick. + +The features' methods (peers, stats, contacts, names.keep and follow, lxmf) call the feature as its API route does, +so a Node raises what a Client gets back: NotFound when the feature isn't loaded, ValueError for what it refuses. +timeout() bounds a Client's requests per thread; a Node, which makes none, only checks its argument. Callbacks (names.watch, a job's on_progress or watch) run on the instance's event thread, one at a time, in the order their events arrived, with no rnfs lock held: whatever hears an event only queues it, coalesced per watch @@ -15,22 +21,28 @@ delays every other callback of the instance; close() from one doesn't wait for the thread. An exception in a callback is logged and dropped. Watched jobs are polled by a thread of their own, which queues a call when a job changes (at most every job_poll seconds) and once more when it finishes. Both threads start on first use. +events.watch's callbacks are the exception to coalescing: each event is one call, in order (an event says what +changed, not a state), so a callback slower than the events lets them queue up. """ import io +import itertools import os import threading import time import warnings from collections import OrderedDict +from collections.abc import Mapping from dataclasses import dataclass, field from rnfs import ids from rnfs.content.cid import Cid from rnfs.errors import Closed, NotFound, RnfsError +from rnfs.events import EVENT_KINDS # A job's final states, and the longest wait one request for a job asks for (the API's cap) from rnfs.features.jobs import FINISHED, MAX_WAIT as MAX_JOB_WAIT -from rnfs.objects import (Added, Collected, DenyState, Entry, Exported, Imported, Info, Pin, Record, Stat, Status, - Verified, Written) +from rnfs.objects import (Added, Cached, Capabilities, Collected, Contact, DenyState, Entry, Event, EventBatch, + Exported, Followed, Imported, Info, Kept, LxmfAddress, Message, Outgoing, Peer, Pin, Record, + Sent, Settings, SettingsChange, Stat, Stats, Status, Synced, Uncached, Verified, Written) FILE_BUFFER = 64 * 1024 # what a File reads at least per refill: one read() of the node, one GET /api/cat # The most one refill of a File read sequentially fetches: each sequential refill doubles from FILE_BUFFER up @@ -57,6 +69,46 @@ def name_bytes(name) -> bytes: return bytes.fromhex(node_hex(name)) +def name_arg(name) -> str: + """A name as the API takes it: a string that isn't 32 hex characters as it is (a name contact's nick, which + the node checks), anything else (16 bytes, hex, a Name) as 32 hex characters. ValueError for anything else.""" + if isinstance(name, str) and not ids.HEX16.fullmatch(name): + return name + return node_hex(name_bytes(name)) + + +def hash_arg(value) -> str: + """A hash (a node ID, name or LXMF address) as the API takes it: 16 bytes or a Name as hex, a string as it is + (32 hex characters, or a contact's nick where one is taken), for the node to check. ValueError otherwise.""" + if isinstance(value, Name): + return value.hex + if isinstance(value, (bytes, bytearray)): + return node_hex(value) + if isinstance(value, str): + return value + raise ValueError(f"a hash is 16 bytes or 32 hex characters, not {type(value).__name__}") + + +def recipients(to) -> list[str] | None: + """Sealing recipients as a list of strings (node IDs, identity hashes or contact nicks, for the node to + check), None for none. ValueError for anything but a list or tuple of strings, as the API refuses it.""" + if not to: + return None + if isinstance(to, str) or not isinstance(to, (list, tuple)) or not all(isinstance(item, str) for item in to): + raise ValueError("to must be a list of recipients") + return list(to) + + +def _body(source, method: str): + """What a stream method sends: bytes-like data as bytes (a memoryview's len() counts its items, not its + bytes), a binary file as it is. ValueError for anything else.""" + if isinstance(source, (bytes, bytearray, memoryview)): + return bytes(source) + if hasattr(source, "read"): + return source + raise ValueError(f"{method}() takes bytes or a binary file, not {type(source).__name__}") + + def _abspath(path) -> str: """path made absolute, as the daemon and the facade take paths.""" return os.path.abspath(os.path.expanduser(os.fspath(path))) @@ -177,6 +229,73 @@ def deliver(self, record: Record): self.callback(record) +class _EventWatch: + """One events.watch: its kinds (None: all), its callback, and its position: the node's run (epoch) and the + last sequence delivered or skipped. Events heard before the baseline is known wait in pending, then only those + after it are delivered; a "lost" marker (kind "lost", seq the sequence the source resumes after) moves the + position there and is delivered unless the watch began after it.""" + + def __init__(self, kinds: frozenset | None, callback): + """A watch calling callback, not ready until begin().""" + self.kinds = kinds + self.callback = callback + # Held while a delivery is queued (deliver: _Events.post, whose condition is innermost), so deliveries are + # queued in the order the position passes them, whichever thread hears them; never held calling out else + self.lock = threading.Lock() + self.ready = False + self.epoch: str | None = None + self.after = 0 + self.pending: list[tuple[str, dict]] = [] + self.handle: Watch | None = None + + def begin(self, after: int, epoch: str, deliver) -> list[dict]: + """Set the position the watch begins at and deliver(event) the events heard meanwhile that are after it, + in order, before any heard later; those delivered.""" + with self.lock: + self.ready, self.after, self.epoch = True, after, epoch + pending, self.pending = self.pending, [] + delivered = [event for run, event in pending if self._take(run, event)] + for event in delivered: + deliver(event) + return delivered + + def offer(self, epoch: str, event: dict, deliver) -> bool: + """deliver(event) of run epoch if the watch wants it (kept for begin() while the baseline isn't known); + whether it did.""" + with self.lock: + if not self.ready: + self.pending.append((epoch, event)) + return False + if not self._take(epoch, event): + return False + deliver(event) + return True + + def _take(self, epoch: str, event: dict) -> bool: + """Whether to deliver event, moving the position past it (lock held, ready).""" + if event["kind"] == "lost": + if epoch == self.epoch and event["seq"] <= self.after: + return False # the watch began after what was missed + self.epoch, self.after = epoch, event["seq"] + return True + if epoch != self.epoch or event["seq"] <= self.after: + return False + self.after = event["seq"] + return self.kinds is None or event["kind"] in self.kinds + + +def event_kinds(kinds) -> list[str] | None: + """kinds (a kind, an iterable of them, or None) as a list, None for every kind (also for none given). + ValueError for a kind the node doesn't publish (rnfs.events.EVENT_KINDS), as GET /api/events words it.""" + if kinds is None: + return None + kinds = [kinds] if isinstance(kinds, str) else list(kinds) + for kind in kinds: + if kind not in EVENT_KINDS: + raise ValueError(f"unknown event kind {kind!r}; known: {', '.join(EVENT_KINDS)}") + return kinds or None + + class _JobPoller: """The thread polling watched jobs, queueing each watch's callback when its job changes and when it ends.""" @@ -290,10 +409,15 @@ def __init__(self, backend): self._jobs = _JobPoller(self, self.jobs_thread, self.job_poll) self._watch_lock = threading.Lock() # taken after the names lock (records are heard under it), never before self._name_watches: dict[bytes, list[_NameWatch]] = {} + self._event_watches: dict[int, _EventWatch] = {} # under _watch_lock + self._event_keys = itertools.count() # each event watch's key, and each delivery's (never coalesced) self._node_id: bytes | None = None self.names = Names(self) self.jobs = Jobs(self) self.deny = Deny(self) + self.contacts = Contacts(self) + self.lxmf = Lxmf(self) + self.events = Events(self) # Lifecycle @@ -313,9 +437,13 @@ def close(self): with self._watch_lock: watches = [watch for each in self._name_watches.values() for watch in each] self._name_watches.clear() + event_watches = list(self._event_watches.values()) + self._event_watches.clear() for watch in watches: watch.handle.active = False self._unwatch_name(watch.name) + for watch in event_watches: + watch.handle.active = False self._close() def _close(self): @@ -342,6 +470,60 @@ def wait_joined(self, timeout: float | None = None) -> bool: self._check() return self._backend.wait_joined(timeout) + def timeout(self, seconds: float | None): + """A context manager bounding each request a Client makes for this thread inside it to seconds (None: no + bound), instead of the timeout the Client was made with: `with client.timeout(5): client.peers()`. + + Thread-local: other threads, and the event, job-watch and name-watch threads, keep the Client's. A long + poll in it (Job.wait) waits within it, as within the Client's; a request outlasting it raises + ConnectionFailed. Nests, the outer one back on exit. A Node makes no requests, so it only checks seconds: + code using it runs unchanged on either class. ValueError unless None or a positive number.""" + return self._backend.timeout(seconds) + + # The node: what it offers, the daemon, settings + + def capabilities(self) -> Capabilities: + """What the node offers: its version, capabilities (has("events"), ...), API routes, features loaded, + event kinds and error codes. A daemon before 0.1.7 can't say: an empty Capabilities, every has() False. A + Node's are a daemon's but "shutdown" and "daemon_json", which it can't do; its routes are the API's, whose + operations it offers through its methods.""" + self._check() + return Capabilities.from_dict(self._backend.capabilities()) + + def shutdown(self, wait: float | None = None) -> bool: + """Stop the daemon (a Client), as SIGTERM does; this Client stays usable once a daemon runs again. With + wait, True once its API stops answering within wait seconds (False: still answering); without, True once + it agreed. Raises Unsupported: from a Node (close() stops it), or a daemon that can't be stopped this way; + NoSuchEndpoint (a NotFound) from a daemon before 0.1.7.""" + self._check() + return self._backend.shutdown(wait) + + def daemon_info(self) -> dict | None: + """What the running daemon wrote about itself in its rnfs dir (daemon.json: "pid", "api_port", + "gateway_port", "version", "started", "rnfs_dir", "reticulum_config", "shared_instance", "node_id"), or None: + none there (a daemon before 0.1.7), it can't be trusted or its process has ended; always None for a Node, + which is no daemon. Read from the file, not asked of the daemon.""" + self._check() + return self._backend.daemon_info() + + def config(self) -> Settings: + """The node's settings: each with its schema, running value, rnfs.toml's, and whether the command line (a + Node's settings) overrides it or a restart would change it; rnfs.toml's path and why a restart would refuse + it, if it would. Raises NotFound without the settings feature (a daemon before 0.1.7: NoSuchEndpoint).""" + self._check() + return Settings.from_dict(self._backend.config()) + + def config_set(self, settings: Mapping) -> SettingsChange: + """Set settings ({name: value, None: removed from rnfs.toml, so back to the default}) in the node's + rnfs.toml, applying at once those that apply live and that the command line doesn't override. Nothing is + written unless the whole result is valid. Raises ValueError (unknown, refused or malformed settings), Busy + (rnfs.toml changed meanwhile: try again), NotFound without the settings feature; a Node OSError, a Client + ServerError, if rnfs.toml can't be written.""" + self._check() + if not isinstance(settings, Mapping): + raise ValueError("settings must be a dict of setting names and values (None: back to the default)") + return SettingsChange.from_dict(self._backend.config_set(dict(settings))) + # Targets def _target(self, target) -> tuple[str, str | None]: @@ -393,9 +575,10 @@ def add(self, source, *, name: str | None = None, wrap: bool = False, pin: bool ValueError, NotFound (no such path), Denied (a file's hash is denied), StorageFull, OSError. """ self._check() + to, wrap, pin, only_hash, to_self, seal_name = (recipients(to), bool(wrap), bool(pin), bool(only_hash), + bool(to_self), bool(seal_name)) sealing = bool(to or to_self) - options = {"only_hash": only_hash, "pin": pin, "to": list(to) if to else None, "to_self": to_self, - "chunker": chunker} + options = {"only_hash": only_hash, "pin": pin, "to": to, "to_self": to_self, "chunker": chunker} if isinstance(source, (str, os.PathLike)): if name is not None: raise ValueError("name is for adding data; a path is added under its own name") @@ -496,6 +679,12 @@ def pins(self, cid=None) -> list[Pin]: self._check() return [Pin.from_dict(pin) for pin in self._backend.pin_ls(str(cid) if cid is not None else None)] + def label_pin(self, cid, label: str | None) -> Pin: + """Give cid's pin a label (None or "" clears it), kept apart from where it was added from and lasting while + it stays pinned; the pin. Raises NotFound (not pinned), ValueError (a label refused).""" + self._check() + return Pin.from_dict(self._backend.pin_label(str(cid), label)) + def pin_remote(self, node_id, cid, pin: bool = True) -> bool: """Ask node_id to pin (or release this node's pin of) cid; True if it accepted.""" self._check() @@ -511,6 +700,18 @@ def verify(self) -> Verified: self._check() return Verified.from_dict(self._backend.verify()) + def cache(self) -> list[Cached]: + """The cached fetches (downloads not pinned, kept until evicted), most recently used first. Never + fetches.""" + self._check() + return [Cached.from_dict(entry) for entry in self._backend.cache_ls()] + + def uncache(self, cid) -> Uncached: + """Evict the cached fetches leading to cid now, deleting the blocks nothing else needs (a pin of the same + content keeps its own). Raises NotFound (no cached fetch leads to cid), ValueError.""" + self._check() + return Uncached.from_dict(self._backend.cache_rm(str(cid))) + # Network def providers(self, cid) -> list[bytes]: @@ -523,6 +724,19 @@ def ping(self, node_id) -> float | None: self._check() return self._backend.ping(node_hex(node_id)) + def peers(self) -> list[Peer]: + """Every node this one knows (in its routing table, with a link open, or counted in its traffic), nearest + first, with its path, link and traffic since the node started. Raises NotFound without the peers + feature.""" + self._check() + return [Peer.from_dict(peer) for peer in self._backend.peers()] + + def stats(self) -> Stats: + """The node's traffic since it started: totals, refusals, per interface, the upload budget left. Raises + NotFound without the peers feature.""" + self._check() + return Stats.from_dict(self._backend.stats()) + # Archives def export(self, target, output) -> Exported: @@ -537,6 +751,38 @@ def import_archive(self, path) -> Imported: self._check() return Imported.from_dict(self._backend.import_archive(_abspath(path))) + # Streams: tars and archives through the program, for files the daemon can't open (or a Node's app has + # in memory). A stream out is sent once iterated, as cat's, its target's whole DAG fetched first + + def add_tar(self, source, *, name: str | None = None, pin: bool = True, only_hash: bool = False, + chunker: str | None = None) -> Added: + """Add the directory a tar holds (bytes, or a binary file read up to the tar's end): its regular files + and directories, the rest skipped with a warning; wrapped as the entry name if given, pinned "user" unless + pin is False (evictable cache) or only_hash (nothing stored); chunker as add's. Raises ValueError (a + malformed tar, an unknown chunker), Denied (a file's hash), TooLarge (max_output_entries), StorageFull.""" + self._check() + return Added.from_dict(self._backend.add_tar(_body(source, "add_tar"), name or None, pin, only_hash, chunker)) + + def get_tar(self, target): + """An iterator of target (a file or directory) as a tar, files as stored (a sealed file sealed); entries + whose names are unsafe in a tar are left out. The whole DAG is fetched first (max_download_size, + max_output_entries), once iterated; close() it to stop early. Raises the fetch errors, ValueError, + possibly only once iterated.""" + self._check() + return self._backend.get_tar(self._target(target)[0]) + + def export_stream(self, target): + """An iterator of target's archive, byte for byte what export() writes, its whole DAG fetched first once + iterated; close() it to stop early. Raises the fetch errors, ValueError, possibly only once iterated.""" + self._check() + return self._backend.export_stream(self._target(target)[0]) + + def import_stream(self, source) -> Imported: + """Import an archive (bytes, or a binary file read to its end), as import_archive: its complete roots + pinned "user". Raises InvalidDag, Denied, StorageFull, ValueError.""" + self._check() + return Imported.from_dict(self._backend.import_stream(_body(source, "import_stream"))) + # Name watches (Names.watch) def _watch_name(self, name: bytes, callback) -> Watch: @@ -603,6 +849,65 @@ def _baseline(self, name: bytes) -> int | None: """The sequence of name's record held now, or None (subclasses).""" return None + # Event watches (Events.watch) + + def _watch_events(self, kinds: frozenset | None, callback) -> Watch: + """Call callback(event) for each event of kinds (None: every kind) published from now on. The watch is + registered before its baseline is asked for, so whatever is heard meanwhile waits in it, not lost.""" + self._check() + key = next(self._event_keys) + watch = _EventWatch(kinds, callback) + watch.handle = Watch(lambda: self._end_event_watch(key)) + self._events.start() + with self._watch_lock: + self._event_watches[key] = watch + try: + after, epoch = self._event_baseline() + except BaseException: + watch.handle.cancel() + raise + watch.begin(after, epoch, lambda event: self._deliver_event(watch, event)) + return watch.handle + + def _end_event_watch(self, key: int): + """Forget a cancelled event watch.""" + with self._watch_lock: + self._event_watches.pop(key, None) + + def _event_watch_count(self) -> int: + """How many event watches there are (begun or not).""" + with self._watch_lock: + return len(self._event_watches) + + def _events_heard(self, epoch: str, events: list[dict]): + """Events of the node's run epoch were heard, in order: queue each for every watch wanting it (any + thread; never blocks).""" + with self._watch_lock: + watches = list(self._event_watches.values()) + for watch in watches: + for event in events: + watch.offer(epoch, event, lambda event, watch=watch: self._deliver_event(watch, event)) + + def _events_lost(self, next_seq: int, epoch: str): + """Events up to next_seq were missed (the watcher fell behind, or the node's run is now epoch): queue the + "lost" marker for every watch whose position is before it.""" + self._events_heard(epoch, [{"seq": next_seq, "kind": "lost", "time": None, "epoch": epoch}]) + + def _deliver_event(self, watch: "_EventWatch", event: dict): + """Queue watch's callback with event, under a key of its own: events are never coalesced.""" + value = Event.from_dict(event) + + def call(): + """The app's callback, unless cancelled since.""" + if watch.handle.active: + watch.callback(value) + self._events.post(("event", next(self._event_keys)), call) + + def _event_baseline(self) -> tuple[int, str]: + """(the sequence of the last event published, the run's epoch) now, the event source started if needed + (subclasses).""" + raise NotImplementedError + class _FileReader(io.RawIOBase): """File's raw stream: each refill is one read of the target opened from the position, of what readinto asks @@ -829,8 +1134,7 @@ def resolve(self, name) -> Record: """The newest record of name (16 bytes, 32 hex characters, a Name or a name contact's nick), after asking the network; persisted, so it never goes back. Raises NotFound (unresolved), ValueError.""" self._owner._check() - text = name if isinstance(name, str) and not ids.HEX16.fullmatch(name) else node_hex(name_bytes(name)) - return Record.from_dict(self._owner._backend.name_resolve(text)) + return Record.from_dict(self._owner._backend.name_resolve(name_arg(name))) def watch(self, name, callback) -> Watch: """Call callback(record) on the event thread for each record of name (16 bytes, 32 hex characters or a @@ -839,20 +1143,62 @@ def watch(self, name, callback) -> Watch: daemon that can't watch names.""" return self._owner._watch_name(name_bytes(name), callback) + # Keepers and follow (rnfs/features/keepers.py, follow.py). After list(), so their annotations name the + # builtin through strings + + def keep(self, name) -> Kept: + """Keep name (16 bytes, 32 hex characters, a Name or a name contact's nick): hold and serve its signed + record so it stays resolvable while its owner is offline (section 9), asking keepers for it in the + background if none is held. Raises ValueError (a malformed name or unknown nick, keep_max_names reached), + NotFound without the keepers feature.""" + self._owner._check() + return Kept.from_dict(self._owner._backend.name_keep(name_arg(name))) + + def unkeep(self, name): + """Stop keeping name for the user (a name followed stays kept for that). Raises NotFound (not kept by the + user, or no keepers feature), ValueError.""" + self._owner._check() + self._owner._backend.name_unkeep(name_arg(name)) + + def kept(self) -> "list[Kept]": + """Every name kept, whatever for, with the record held.""" + self._owner._check() + return [Kept.from_dict(entry) for entry in self._owner._backend.name_kept()] + + def follow(self, name) -> Followed: + """Follow name (as keep takes it): keep it and keep its latest target pinned, fetched in the background as + the name moves (within follow_quota). Raises ValueError (as keep), NotFound without the follow feature.""" + self._owner._check() + return Followed.from_dict(self._owner._backend.name_follow(name_arg(name))) + + def unfollow(self, name): + """Stop following name, releasing its pin. Raises NotFound (not followed, or no follow feature), + ValueError.""" + self._owner._check() + self._owner._backend.name_unfollow(name_arg(name)) + + def following(self) -> "list[Followed]": + """Every name followed, with the target pinned for it and what its last check did.""" + self._owner._check() + return [Followed.from_dict(entry) for entry in self._owner._backend.name_following()] + @dataclass(frozen=True) class Job: - """A background transfer (get, pin or export) the node runs, persists and resumes after a restart: its - record as of the last refresh(), wait() or callback. done is True once it has finished (state "done", - "failed" or "cancelled"); result is what the matching call returns, error why it failed.""" + """A background transfer (get, pin, export, add or import) the node runs, persists and resumes after a restart: + its record as of the last refresh(), wait() or callback. done is True once it has finished (state "done", + "failed" or "cancelled"); result is what the matching call returns, error why it failed. An add or import job + has a path (and an add its options) instead of a target.""" id: str kind: str - target: str + target: str | None output: str | None = None from_node: bytes | None = None sealed_name: bool = False fallback_dht: bool = False + path: str | None = None + options: dict | None = None state: str = "queued" created: float | None = None started: float | None = None @@ -869,7 +1215,8 @@ class Job: eta: float | None = None _owner: Rnfs | None = field(default=None, repr=False, compare=False) - FIELDS = ("id", "kind", "target", "output", "sealed_name", "fallback_dht", "state", "created", "started", + FIELDS = ("id", "kind", "target", "output", "sealed_name", "fallback_dht", "path", "options", "state", "created", + "started", "finished", "error", "result", "runs", "total_bytes", "done_bytes", "blocks", "blocks_total", "downloaded_bytes", "rate", "eta") @@ -941,10 +1288,31 @@ def export(self, target, output, *, on_progress=None) -> Job: """Write an archive of target to output (which must not exist), as a job.""" return self._submit("export", target, _abspath(output), None, on_progress) + def add(self, path, *, wrap: bool = False, pin: bool = True, only_hash: bool = False, + to: list[str] | None = None, to_self: bool = False, seal_name: bool = True, chunker: str | None = None, + on_progress=None) -> Job: + """Add a file or directory on the node's machine as a job: add()'s options for a path, its result add()'s + Added as a dict; progress counts the bytes read of the size measured first. An interrupted add runs again + from the start (adding is idempotent). Raises NotFound (no such path), ValueError (an option), Busy.""" + # Flags as bools and recipients checked here, so both classes refuse alike (a Client's JSON would carry + # wrap=1 as 1, which the daemon takes, while a Node's job refuses it) + options = {"wrap": True if wrap else None, "pin": None if pin else False, + "only_hash": True if only_hash else None, "to": recipients(to), + "to_self": True if to_self else None, "seal_name": None if seal_name else False, + "chunker": chunker} + return self._submit("add", None, None, None, on_progress, path=_abspath(path), + options={name: value for name, value in options.items() if value is not None} or None) + + def import_archive(self, path, *, on_progress=None) -> Job: + """Import an archive on the node's machine as a job (import_archive()'s result as a dict); progress counts + the archive's bytes read. Raises NotFound (no such file), Busy.""" + return self._submit("import", None, None, None, on_progress, path=_abspath(path)) + def _submit(self, kind: str, target, output: str | None, from_node, on_progress, **flags) -> Job: - """Queue a job; with on_progress, watched from the start.""" + """Queue a job (target None for add and import, which take path= instead); with on_progress, watched from + the start.""" self._owner._check() - text, hint = self._owner._target(target) + text, hint = self._owner._target(target) if target is not None else (None, None) from_node = self._owner._from(from_node, hint) if kind == "get" else None job = Job.from_dict(self._owner._backend.submit_job(kind, text, output, from_node, **flags), self._owner) if on_progress is not None: @@ -995,3 +1363,182 @@ def list(self) -> DenyState: """The local entries, and each deny_lists source with its state.""" self._owner._check() return DenyState.from_dict(self._owner._backend.deny()) + + +class Contacts: + """A node's address book (Rnfs.contacts; rnfs/features/contacts.py): nicks for names, nodes and LXMF + addresses, taken wherever the node takes one of those (a name contact's nick also starts a target). A nick is + 1-32 lowercase letters, digits, '-', '_' or '.', starting with a letter or digit. Every method raises NotFound + without the contacts feature.""" + + def __init__(self, owner: Rnfs): + """The address book of owner.""" + self._owner = owner + + def get(self, nick: str) -> Contact: + """The contact called nick. Raises NotFound (none), ValueError (not a nick).""" + self._owner._check() + return Contact.from_dict(self._owner._backend.contact_get(nick)) + + def add(self, nick: str, id, kind: str | None = None) -> Contact: + """Add or replace the contact nick for id (16 bytes, 32 hex characters or a Name) of kind "name", "node" + or "lxmf"; None has the node tell what id is (one of its names or a name it knows, a node it knows, an + RNS destination it has heard). Raises ValueError (nick, id or kind invalid, a kind it can't tell, the + address book full).""" + self._owner._check() + return Contact.from_dict(self._owner._backend.contact_add(nick, hash_arg(id), kind)) + + def remove(self, nick: str): + """Delete the contact nick. Raises NotFound (none), ValueError (not a nick).""" + self._owner._check() + self._owner._backend.contact_rm(nick) + + # After the methods above, whose annotations name the builtin list + def list(self) -> list[Contact]: + """Every contact, by nick.""" + self._owner._check() + return [Contact.from_dict(contact) for contact in self._owner._backend.contacts()] + + +@dataclass(frozen=True) +class Fetched: + """One link fetched by Lxmf.get: its target (`[/path]`, a name link's name resolved) and output path; + the Job fetching it (a Node's and a daemon's jobs feature runs the fetch as a job) or, fetched in the call, its + CID; warnings (a name link fetched by the CID it was sent with), and error if it failed after an earlier link + had started (the first link's failure is raised instead).""" + + target: str + output: str + job: Job | None = None + cid: Cid | None = None + warnings: tuple[str, ...] = () + error: str | None = None + + @classmethod + def from_dict(cls, data: dict, owner: Rnfs | None = None) -> "Fetched": + """From the API's form: {"target", "output", "job" | "cid", "warnings"?, "error"?}.""" + job, cid = data.get("job"), data.get("cid") + return cls(data["target"], data["output"], Job.from_dict(job, owner) if job else None, + Cid.parse(cid) if cid else None, tuple(data.get("warnings") or ()), data.get("error")) + + def to_dict(self) -> dict: + """The API's form, every key present (None where the node leaves one out).""" + return {"target": self.target, "output": self.output, "job": self.job.to_dict() if self.job else None, + "cid": str(self.cid) if self.cid else None, "warnings": list(self.warnings), "error": self.error} + + +class Lxmf: + """A node's LXMF messaging (Rnfs.lxmf; rnfs/features/lxmf.py): rnfs links sent to and received from LXMF + addresses (Sideband, NomadNet, MeshChat, other rnfs nodes). A Node runs it only with settings + {"lxmf_enabled": True}; while it is off, address, send and sync raise ValueError saying why (inbox and outbox + list what is stored). Every method raises NotFound without the lxmf feature.""" + + def __init__(self, owner: Rnfs): + """The LXMF messaging of owner.""" + self._owner = owner + + def address(self) -> LxmfAddress: + """This node's LXMF address and display name, the propagation node set and the nearest one heard.""" + self._owner._check() + return LxmfAddress.from_dict(self._owner._backend.lxmf_address()) + + def send(self, to, target, message: str | None = None, *, propagate: bool = False, + nearest: bool = False) -> Sent: + """Send a link to target (a Cid, `[/path]`, a name contact's nick path or an rnfs: URI, a name URI + sent as a name link that follows the name), with message as its text, to to (an LXMF address: 16 bytes + or 32 hex characters; or an LXMF contact's nick). Direct unless propagate (through the propagation node; + nearest: without lxmf_propagation_node set, the nearest heard); a direct message that fails is sent again + through the propagation node, if one is set. Returns once queued: outbox() says how it went. Raises + ValueError (LXMF off, recipient or target invalid, no propagation node), NotFound (no announce heard + from the recipient yet: a path request was sent, try again shortly).""" + self._owner._check() + if isinstance(target, Cid): + target = str(target) + elif not isinstance(target, str): + raise ValueError(f"a target is a Cid, [/path] or an rnfs: URI, not {type(target).__name__}") + return Sent.from_dict(self._owner._backend.lxmf_send(hash_arg(to), target, message, bool(propagate), + bool(nearest))) + + def inbox(self) -> list[Message]: + """The messages received with rnfs links, oldest first.""" + self._owner._check() + return [Message.from_dict(entry) for entry in self._owner._backend.lxmf_inbox()] + + def get(self, id: int, *, link: int | None = None, output=None, directory=None, + use_sent_cid: bool = False) -> list[Fetched]: + """Fetch the links of inbox message id (only link, 1-based, if given) and write each to output (one + link; must not exist) or into directory under its last path name (the root CID if that name is unsafe: + the sender chose it), asking the sender's node first, then the DHT. Each fetch is a job (Fetched.job). + A name link whose name can't be resolved raises NotFound before anything starts, unless use_sent_cid: + then the CID it was sent with is fetched (nothing ties it to the name), with a warning. Raises NotFound (no + such message), ValueError (id or link invalid, neither or a clashing output).""" + self._owner._check() + results = self._owner._backend.lxmf_inbox_get( + id, link, _abspath(output) if output is not None else None, + _abspath(directory) if directory is not None else None, bool(use_sent_cid)) + return [Fetched.from_dict(result, self._owner) for result in results] + + def remove(self, id: int) -> int: + """Remove inbox message id, releasing the pins its links were auto-fetched under; 1. Raises NotFound (no + such message), ValueError.""" + self._owner._check() + return self._owner._backend.lxmf_inbox_rm(id, False) + + def remove_all(self) -> int: + """Remove every inbox message, releasing their auto-fetched pins; how many.""" + self._owner._check() + return self._owner._backend.lxmf_inbox_rm(None, True) + + def outbox(self) -> list[Outgoing]: + """The messages sent, oldest first, with their delivery state.""" + self._owner._check() + return [Outgoing.from_dict(entry) for entry in self._owner._backend.lxmf_outbox()] + + def sync(self, nearest: bool = False) -> Synced: + """Start downloading the messages waiting at the propagation node (nearest: without lxmf_propagation_node + set, the nearest heard). Raises ValueError (LXMF off).""" + self._owner._check() + return Synced.from_dict(self._owner._backend.lxmf_sync(bool(nearest))) + + +def _whole(value, name: str, none: bool = False) -> int | None: + """value if it is a whole number of at least 0 (or None, if none); ValueError otherwise, as the API's + parse_int refuses it.""" + if value is None and none: + return None + if type(value) is not int or value < 0: + raise ValueError(f"{name} must be a non-negative integer") + return value + + +class Events: + """What changes on a node (Rnfs.events; rnfs/events.py; design.md, "Events"): jobs, pins, the cache, the LXMF + boxes, names and peers, each an Event, so an app follows them instead of polling every list. A Node hears its + own as they are published; a Client long-polls GET /api/events. A daemon before 0.1.7 has no events: + NoSuchEndpoint (a NotFound).""" + + def __init__(self, owner: Rnfs): + """The events of owner.""" + self._owner = owner + + def watch(self, callback, kinds=None) -> Watch: + """Call callback(event) on the event thread for each event of kinds (a kind or several; None: all) + published from now on, in order, one call each. If events were missed (a Client fell more than the node's + 1024 kept events behind, or the daemon restarted), callback gets an Event of kind "lost" (event.lost) in + their place: read again what the app shows. A Client polls on a thread of its own + (rnfs-client-event-watch), within its timeout as the name poller does. Raises ValueError (an unknown + kind); a Client NoSuchEndpoint from a daemon without events.""" + return self._owner._watch_events(frozenset(kinds) if (kinds := event_kinds(kinds)) else None, callback) + + def since(self, after: int | None = None, kinds=None, wait: int = 0, epoch: str | None = None) -> EventBatch: + """The events after sequence after (of kinds, if given) as soon as there is one, or none after wait whole + seconds (at most 30; a Client's within its timeout, as Job.wait's). after None answers at once with only + the next to ask after. Pass back the batch's next and epoch: lost then says events were missed (the + node's run changed, or they are no longer held), and events holds every one held. Raises ValueError; a + Client NoSuchEndpoint from a daemon without events, Busy while 8 other polls wait.""" + self._owner._check() + kinds = event_kinds(kinds) + # epoch "" is no epoch, as the API reads an empty parameter (a Node's log would call it another run) + data = self._owner._backend.events_since(_whole(after, "after", none=True), kinds, _whole(wait, "wait"), + epoch or None) + return EventBatch.from_dict(data) diff --git a/rnfs/mesh.py b/rnfs/mesh.py index 66e0106..b3c95ef 100644 --- a/rnfs/mesh.py +++ b/rnfs/mesh.py @@ -7,7 +7,8 @@ - Airtime controls for /blocks (AirtimePolicy): a node-wide upload budget (a token bucket), interface rules and quiet hours. /find, /provide and /pin are tiny and never refused by them. - Traffic: bounded, thread-safe counters of bytes and requests per peer and per interface, for `rnfs peers`, - `rnfs stats` and `rnfs status`. + `rnfs stats` and `rnfs status`, with each peer's last link round-trip time and the last sign it was up, from + which online() tells whether it is online. Pure policy with injected clocks; nothing here imports RNS (or rnfs.log, which does) at module level, since the CLI imports the peers feature, which uses this module. @@ -30,6 +31,14 @@ PATH_CACHE = 15.0 MAX_CACHED_PATHS = 1024 PATHFINDER_M = 128 # RNS.Transport.PATHFINDER_M: what hops_to returns for an unknown path +# Seconds a peer stays online after the last sign it was up (online()). Three link_idle_timeouts (300 s by +# default): a peer just talked to doesn't turn offline the moment its idle link closes, nor between a fetch's +# requests or a republish's. Far below announce_interval (6 h by default), so one announce doesn't keep a node +# shown online for hours after it went away; a node that only announces is shown online for this long after each +ONLINE_WINDOW = 900.0 +# Seconds an announce's emission time (its sender's clock) may be off from ours and still count as now +# (Traffic.announced): nodes on a mesh without NTP drift by minutes +EMISSION_SKEW = 300.0 # Why a /blocks request was refused by an airtime control (Traffic.refused keys) REFUSED_BUDGET = "budget" @@ -418,19 +427,37 @@ def shared_instance_warning(config, reticulum) -> str | None: # Traffic counters +def online(link: bool, last_seen: float | None, now: float) -> bool: + """Whether a peer counts as online (`rnfs peers`, "peer" events): an rnfs link to it is open, or the last sign + it was up (Traffic: last_seen, monotonic like now) is at most ONLINE_WINDOW old; a negative age (a clock that + went back) counts as offline. A guess from what the node already knows, without network I/O: a node up but + silent shows offline (ping tells for sure), and one that went away shows online until the window passes or + its link closes.""" + return link or (last_seen is not None and 0 <= now - last_seen <= ONLINE_WINDOW) + + class _Peer: - """Counters for one node.""" + """Counters for one node. Times shown to users are wall time; those decisions are made from (last_seen, + first_counted) are Traffic.monotonic's, so a step of the wall clock can't turn peers online, offline or new.""" __slots__ = ("sent", "received", "requests", "answered", "refused", "failed", "bad", "mode", "last_heard", - "last_active") + "last_active", "first_heard", "first_counted", "last_seen", "last_emitted", "rtt", "rtt_at") - def __init__(self): - """Nothing counted yet.""" + def __init__(self, now: float, monotonic: float): + """Nothing counted yet; first counted at now (wall time), monotonic (Traffic.monotonic).""" self.sent = self.received = 0 self.requests = self.answered = self.refused = self.failed = self.bad = 0 self.mode: str | None = None # "server" or "client", from its announces - self.last_heard: float | None = None # wall time of its last announce + self.last_heard: float | None = None # wall time of its last announce (or path response) self.last_active: float | None = None # wall time of the last traffic with it + self.first_heard = now # wall time it was first counted (this run, or since it was last forgotten) + self.first_counted = monotonic # the same, monotonic: what "new" peer events are decided from + # Monotonic time of the last sign it was up: an announce emitted recently (Traffic.announced), an answer to a + # request (a refusal too), a request of its own served, or a link established + self.last_seen: float | None = None + self.last_emitted: float | None = None # emission time (its clock) of the newest announce counted + self.rtt: float | None = None # seconds: the round trip RNS measured on the last link established to it + self.rtt_at: float | None = None # wall time it was measured def as_dict(self) -> dict: """JSON-ready counters.""" @@ -441,15 +468,17 @@ class Traffic: """Bytes and requests per peer, per interface and in total since start. Thread-safe, bounded. Peers and interfaces are kept in LRU order and capped (MAX_PEERS, MAX_INTERFACES), so a flood of - identities or spawned interfaces costs a fixed amount of memory. clock is wall time (for display). + identities or spawned interfaces costs a fixed amount of memory. clock is wall time (for display, and for + announces' emission times); monotonic is what online() and "new" peers are decided from. """ def __init__(self, max_peers: int = MAX_PEERS, max_interfaces: int = MAX_INTERFACES, - clock: Callable[[], float] = time.time): + clock: Callable[[], float] = time.time, monotonic: Callable[[], float] = time.monotonic): """Empty counters.""" self.max_peers = max_peers self.max_interfaces = max_interfaces self.clock = clock + self.monotonic = monotonic self.started = clock() self._peers: OrderedDict[bytes, _Peer] = OrderedDict() self._interfaces: OrderedDict[str, dict[str, int]] = OrderedDict() @@ -462,7 +491,7 @@ def _peer(self, node_id: bytes) -> _Peer: """node_id's counters, made most recent (created, evicting the oldest, if new). Caller holds the lock.""" peer = self._peers.get(node_id) if peer is None: - peer = self._peers[node_id] = _Peer() + peer = self._peers[node_id] = _Peer(self.clock(), self.monotonic()) while len(self._peers) > self.max_peers: self._peers.popitem(last=False) else: @@ -492,6 +521,7 @@ def sent(self, node_id: bytes | None, interface: str | None, nbytes: int): peer = self._peer(node_id) peer.sent += nbytes peer.last_active = self.clock() + peer.last_seen = self.monotonic() # it just asked def request(self, node_id: bytes, outcome: str, received: int = 0, interface: str | None = None, bad: int = 0): @@ -508,14 +538,45 @@ def request(self, node_id: bytes, outcome: str, received: int = 0, interface: st peer.received += received peer.bad += bad peer.last_active = self.clock() + if outcome != FAILED: # it answered, if only to refuse + peer.last_seen = self.monotonic() def heard(self, node_id: bytes, mode: str): - """node_id announced itself in DHT mode ("server" or "client").""" + """node_id announced itself in DHT mode ("server" or "client"), or a path response for it arrived.""" with self._lock: peer = self._peer(node_id) peer.mode = mode peer.last_heard = self.clock() + def announced(self, node_id: bytes, emitted: float | None) -> bool: + """An announce of node_id's own arrived, emitted at emitted (Unix time on its clock, signed; None if + unknown). True if it counts as a sign node_id was up: emitted later than the newest counted before (a replay + doesn't count twice) and at most ONLINE_WINDOW ago, allowing EMISSION_SKEW of clock difference. An announce + delivered late (held in an announce queue, or a path answer a shared instance passes on as an announce) says + nothing of now. It is seen when it was emitted (less the skew, never later than now), not when it arrived.""" + if not isinstance(emitted, (int, float)) or isinstance(emitted, bool) or not math.isfinite(emitted): + return False + with self._lock: + peer = self._peer(node_id) + if peer.last_emitted is not None and emitted <= peer.last_emitted: + return False + peer.last_emitted = float(emitted) + age = max(0.0, self.clock() - emitted - EMISSION_SKEW) + if age > ONLINE_WINDOW: + return False + seen = self.monotonic() - age + if peer.last_seen is None or seen > peer.last_seen: + peer.last_seen = seen + return True + + def measured(self, node_id: bytes, rtt: float): + """A link to node_id was established, RNS measuring a round trip of rtt seconds.""" + with self._lock: + peer = self._peer(node_id) + peer.rtt = float(rtt) + peer.rtt_at = self.clock() + peer.last_seen = self.monotonic() + def refused(self, reason: str): """A /blocks request was refused by an airtime control (one of REFUSALS).""" with self._lock: diff --git a/rnfs/names.py b/rnfs/names.py index f33cea3..997be11 100644 --- a/rnfs/names.py +++ b/rnfs/names.py @@ -715,6 +715,9 @@ def __init__(self, config: Config, pins, network=None, clock=time.time, monotoni # each newer record accepted for a watched name; swapped whole, so they are read without the lock self._watched: dict[bytes, int] = {} self._record_listeners: tuple = () + # Called as held_listener(name, record), with the lock held, for each newer record accepted for a name held + # (persisted: this node's own, kept, followed, resolved) or watched: the node's "name" events. Must only queue + self.held_listener = None # Pins of names whose key file was deleted by hand, released by release_gone() once the node is set up self._gone: list[tuple[str, Cid]] = [] # Aliases whose key file is there but couldn't be read (permissions, damage): not names this node can @@ -1051,7 +1054,13 @@ def remove_record_listener(self, listener): self._record_listeners = tuple(item for item in self._record_listeners if item != listener) def _accepted(self, name: bytes, record: Record): - """A newer record was accepted for name: tell the record listeners if it is watched. Lock held.""" + """A newer record was accepted for name: tell the record listeners if it is watched, and held_listener if it + is held or watched. Lock held.""" + if self.held_listener is not None and (name in self._held or name in self._watched): + try: + self.held_listener(name, record) + except Exception as e: # noqa: BLE001 - as below + log(f"record listener failed: {e!r}", LOG_ERROR) if name not in self._watched: return for listener in self._record_listeners: diff --git a/rnfs/network/announces.py b/rnfs/network/announces.py index b3bd33c..b5dcbd5 100644 --- a/rnfs/network/announces.py +++ b/rnfs/network/announces.py @@ -81,8 +81,13 @@ class DhtAnnounceHandler: """RNS announce handler for rnfs.dht, including path responses, reporting each node's DHT mode.""" def __init__(self, on_server: Callable[[bytes], None], on_client: Callable[[bytes], None], - is_banned: Callable[[bytes], bool] = lambda node_id: False, own_id: bytes | None = None): - """Handler calling on_server(node_id) or on_client(node_id) for each announce not from a banned node.""" + is_banned: Callable[[bytes], bool] = lambda node_id: False, own_id: bytes | None = None, + on_announced: Callable[[bytes], None] | None = None): + """Handler calling on_server(node_id) or on_client(node_id) for each announce or path response not from a + banned node, then on_announced(node_id), if given, for an announce only: a transport node may answer a + path request from its path table long after the node went away. Not enough on its own: a shared instance + passes path answers on as announces, and announces can arrive late, so on_announced (RnfsNode._announced) + goes by the announce's emission time.""" # RNS only calls us for announces whose destination hash matches this full name self.aspect_filter = f"{APP_NAME}.{DHT_ASPECT}" self.receive_path_responses = True @@ -90,9 +95,12 @@ def __init__(self, on_server: Callable[[bytes], None], on_client: Callable[[byte self.on_client = on_client self.is_banned = is_banned self.own_id = own_id + self.on_announced = on_announced - def received_announce(self, destination_hash, announced_identity, app_data): - """RNS calls this (on a thread of its own) for each rnfs.dht announce or path response.""" + def received_announce(self, destination_hash, announced_identity, app_data, announce_packet_hash=None, + is_path_response=False): + """RNS calls this (on a thread of its own) for each rnfs.dht announce or path response; with five + parameters, it says which (is_path_response).""" node_id = bytes(destination_hash) if node_id == self.own_id or self.is_banned(node_id): return @@ -101,6 +109,8 @@ def received_announce(self, destination_hash, announced_identity, app_data): self.on_server(node_id) else: self.on_client(node_id) + if not is_path_response and self.on_announced is not None: + self.on_announced(node_id) except Exception as e: # noqa: BLE001 - runs on RNS's announce thread; logged log(f"error handling the announce of {node_name(node_id)}: {e}", LOG_WARNING) diff --git a/rnfs/network/client.py b/rnfs/network/client.py index 1480edc..6e6df60 100644 --- a/rnfs/network/client.py +++ b/rnfs/network/client.py @@ -44,6 +44,8 @@ def __init__(self, links: LinkPool, config: Config): self.links = links self.config = config self.traffic = mesh.Traffic() # requests and bytes per peer (mesh); the node shares it with the server + # Every link opened (shared, ping's, identified) records its round trip, and that the node is up, there + links.on_rtt = self.traffic.measured # Links @@ -160,7 +162,8 @@ def pin(self, node_id: bytes, cid: Cid, pin: bool) -> bool: # ping def ping(self, node_id: bytes) -> float | None: - """Open a new link to node_id and return its round-trip time in seconds, or None if unreachable.""" + """Open a new link to node_id and return its round-trip time in seconds, or None if unreachable. The pool + records it in traffic as it does for every link (LinkPool.on_rtt).""" peer = self.links.open_oneshot(node_id, identify=False) if peer is None: return None diff --git a/rnfs/network/links.py b/rnfs/network/links.py index 57ab6f5..dcdce4f 100644 --- a/rnfs/network/links.py +++ b/rnfs/network/links.py @@ -240,6 +240,9 @@ def __init__(self, identity: RNS.Identity, config: Config, clock: Callable[[], f self.lock = threading.Lock() self.stop_event = threading.Event() self.sweeper: threading.Thread | None = None + # Called with (node ID, seconds) for the round trip RNS measured on each link established, shared, one-shot + # (ping's) or identified, on the thread that opened it; PeerClient sets its traffic counters' (`rnfs peers`) + self.on_rtt: Callable[[bytes, float], None] | None = None # Reject response advertisements whose transfer outgrows their size (in the scope the node set) guards.install() @@ -358,8 +361,18 @@ def on_closed(closed_link): link.identify(self.identity) rtt = getattr(link, "rtt", None) log(f"connected to {name}" + (f" (round trip {rtt * 1000:.0f} ms)" if rtt else ""), LOG_VERBOSE) + self._report_rtt(node_id, rtt) return True + def _report_rtt(self, node_id: bytes, rtt): + """Hand on_rtt the round trip of a link just established to node_id, if it is a finite, non-negative number + (RNS measures it on a link this node opens: the time from its request to the other node's proof).""" + listener = self.on_rtt + if listener is None or not isinstance(rtt, (int, float)) or isinstance(rtt, bool): + return + if 0 <= rtt < float("inf"): + listener(node_id, float(rtt)) + def _recall(self, node_id: bytes): """The node's identity from RNS, asking for a path again (whose response carries it) if needed. diff --git a/rnfs/node.py b/rnfs/node.py index daa9e06..3faddf9 100644 --- a/rnfs/node.py +++ b/rnfs/node.py @@ -33,11 +33,11 @@ from collections import OrderedDict from contextlib import contextmanager from functools import partial -from typing import BinaryIO, Callable, Iterator, Literal, Protocol, TypeVar, overload +from typing import BinaryIO, Callable, Iterator, Literal, NamedTuple, Protocol, TypeVar, overload import RNS -from rnfs import __version__, denylist, features, fsutil, statedb, winacl +from rnfs import __version__, denylist, events, features, fsutil, statedb, winacl from rnfs.config import LOCK_FILE, Config from rnfs.constants import (APP_NAME, DEFAULT_BOOTSTRAP_NODES, DHT_ASPECT, K, MISSED_ANNOUNCES, MODE_SERVER, PARTIAL_SUFFIX) @@ -53,7 +53,7 @@ from rnfs.fetcher import Fetch, Fetcher, Observer, describe, observing from rnfs.mesh import RnsPaths from rnfs.log import LOG_DEBUG, LOG_ERROR, LOG_NOTICE, LOG_VERBOSE, LOG_WARNING, cap_warnings, log, node_name, plural, size -from rnfs.names import NameManager, parse_name +from rnfs.names import NameManager, RnsNetwork, parse_name from rnfs.pin_names import PinNames from rnfs.network import guards, rates from rnfs.network.announces import DhtAnnounceHandler, DhtAnnouncer, parse_mode @@ -375,6 +375,16 @@ def __init__(self, save: Callable[[], None] = lambda: None, close: Callable[[], self.close = close +class _PinDetails(NamedTuple): + """What pin_ls reads from a pin's stored blocks (RnfsNode._pin_details): None for what they can't tell.""" + + kind: str | None # "file" or "directory" + inside: str | None # a directory's one entry's name, or "(directory: N entries)" + sealed: bool | None # a sealed file + entries: int | None # a directory's entry count + size: int | None # content bytes, as the root block states them + + class _NamePins: """The pins NameManager uses: storage pins that also update what the node provides.""" @@ -509,7 +519,7 @@ def _start_network(self, config: Config, slow: bool): self.server = RequestServer(self.destination, config, self.backend, traffic=self.client.traffic) self.announcer = DhtAnnouncer(self.destination, config) self.announce_handler = DhtAnnounceHandler(self._heard_server, self._heard_client, self.routing.is_banned, - self.node_id) + self.node_id, on_announced=self._announced) self.announce_handler.register() self.names.start() log(f"node {node_name(self.node_id)} running as a DHT {config.dht_mode}" @@ -546,6 +556,8 @@ def _setup(self, config: Config, node_id: bytes, client, slow: bool, names_netwo self.config = config self.node_id = node_id self.client = client + # What changed, for local clients (GET /api/events, rnfs.events); first, so every part may publish + self.events = events.EventLog() self.slow = slow self._is_known = is_known self.members = config.member_set @@ -578,6 +590,8 @@ def _setup(self, config: Config, node_id: bytes, client, slow: bool, names_netwo if removed: log(f"private mode: dropped {plural(removed, 'provider record')} of non-members", LOG_NOTICE) self.storage = self._add_store(Storage(config.blocks_dir, config.state_dir, config.max_storage)) + self.storage.pin_listener = self._pin_event + self.storage.cache_listener = self._cache_event # Before anything is fetched or provided: the fetcher and the publisher check it through storage.deny self.deny = self._add_store(denylist.DenyList(config.state_dir)) self.storage.deny = self.deny @@ -588,8 +602,8 @@ def _setup(self, config: Config, node_id: bytes, client, slow: bool, names_netwo self._not_sealed: OrderedDict[walker.Expect, None] = OrderedDict() self._not_sealed_deny: frozenset[bytes] | None = None self._not_sealed_lock = threading.Lock() - # pin_ls's memo: pinned CID -> (type, name from inside, sealed), for complete pins only (_pin_details) - self._pin_memo: dict[Cid, tuple[str | None, str | None, bool | None]] = {} + # pin_ls's memo: pinned CID -> its _PinDetails, for complete pins only (_pin_details) + self._pin_memo: dict[Cid, _PinDetails] = {} self._pin_memo_lock = threading.Lock() # The shared folder is written through: nothing to save. Its per-file cache (what the files inside its entries # were built as) is saved with the batched stores; the shared folder's close releases it too @@ -599,6 +613,7 @@ def _setup(self, config: Config, node_id: bytes, client, slow: bool, names_netwo # Where pins came from, for pin ls (local only). Written through, not durably: a power loss can lose a # name, never the pin it names, which is committed durably first; its save writes a name whose write failed self.pin_names = self._add_store(PinNames(config.state_dir), batched=False) + self.pin_names.prune_labels(self.storage.pinned_cids()) self.fetcher = self._add_store(Fetcher(self.storage, client, config, node_id, self.routing, self._lookup, slow, config.state_dir, after_store=self._enforce_storage, make_room=self._make_room)) @@ -608,6 +623,7 @@ def _setup(self, config: Config, node_id: bytes, client, slow: bool, names_netwo generation=self._provide_generation), batched=False) # Saves its state itself, and at stop (names.stop) self.names = NameManager(config, _NamePins(self), network=names_network) + self.names.held_listener = self._name_event self._add_store(_StoreParts(close=self.names.close), batched=False) self.backend = NodeBackend(self) @@ -815,6 +831,25 @@ def _answered(self, node_id: bytes): if self.routing.answered(node_id): self.publisher.wake() # a CID that found no nodes may now be publishable + def _pin_event(self, change: str, cids: list[Cid], source: str): + """Storage's pin_listener (storage lock held): pins of source added or removed, as a "pin" event; a CID + no source pins any more loses its label (pin_names' lock is taken under storage's, never the other way).""" + pin_names = getattr(self, "pin_names", None) # made after storage; prune_labels covers the gap + if change == "removed" and pin_names is not None: + for cid in cids: + if not self.storage.is_pinned(cid): + pin_names.forget_label(cid) + self.events.publish("pin", change=change, source=source, **events.cid_fields(cids)) + + def _cache_event(self, change: str, cids: list[Cid]): + """Storage's cache_listener (storage lock held): cached fetches added or removed, as a "cache" event.""" + self.events.publish("cache", change=change, **events.cid_fields(cids)) + + def _name_event(self, name: bytes, record): + """NameManager's held_listener (names lock held): a newer record of a name this node holds (its own, kept, + followed or resolved) or watches, as a "name" event.""" + self.events.publish("name", name=name.hex(), cid=str(record.cid), sequence=record.sequence) + def _heard_server(self, node_id: bytes): """A server-mode announce (or path response) from node_id.""" self._count_heard(node_id, "server") @@ -835,6 +870,23 @@ def _count_heard(self, node_id: bytes, mode: str): if traffic is not None: traffic.heard(node_id, mode) + def _announced(self, node_id: bytes): + """node_id's own announce (not one RNS marks as a path response) arrived: Traffic.announced counts it as a + sign it is up, which `rnfs peers` reads its online state from (mesh.online), if it was emitted recently. + Attached to a shared instance, path answers come as announces too, and an announce queue can hold one for + hours, so only its emission time tells. + + That is the newest emission in node_id's path (RnsNetwork.emitted_time: RNS internal, None if it can't be + read, and then the announce doesn't count), which is this announce's: RNS calls announce handlers only for + an announce that takes the path, which needs one emitted later than the path's newest (or an expired path: + then the newest may be an earlier announce's, which counts only if no later one was). An in-memory read: no + disk, unlike the + announce cache, which a shared instance's client doesn't write and whose packet hash a path answer passed on + as an announce doesn't match.""" + traffic = getattr(self.client, "traffic", None) + if traffic is not None: + traffic.announced(node_id, RnsNetwork().emitted_time(node_id)) + def _is_member_node(self, node_id: bytes) -> bool: """Private mode's routing table admission: node_id's identity is a member.""" # Runs on every admission: a check, not a contact, so it doesn't mark the destination used @@ -1232,20 +1284,22 @@ def _record_fetch(self, fetch: Fetch, for_gateway: bool): def add_path(self, path: str, wrap: bool = False, only_hash: bool = False, pin: bool = True, to: list[str] | None = None, to_self: bool = False, seal_name: bool = True, - chunker: str | None = None) -> dict: + chunker: str | None = None, progress: Callable[[int], None] | None = None) -> dict: """Import a file or directory on this machine, pinned "user" (cache if not pin) unless only_hash. With to/to_self the file is sealed for those recipients first (seal.py; directories are refused), - its name sealed inside unless not seal_name; chunker is "fixed" (default) or "fastcdc". + its name sealed inside unless not seal_name; chunker is "fixed" (default) or "fastcdc". progress, if + given, is told the bytes read as they are (builder.BuildOptions.progress: an add job's), and may raise to + stop the add; blocks stored by then are unreferenced, for the next collection. """ if not os.path.lexists(path): raise NotFound(f"no such file or directory: {path}") recipients = node_content.seal_recipients(self, to, to_self, only_hash) - options = builder.BuildOptions(chunker) # ValueError for an unknown chunker, before anything is read + options = builder.BuildOptions(chunker, progress=progress) # ValueError for an unknown chunker, at once if only_hash: built, warnings = builder.add_path(path, builder.NO_PUT, wrap, options) else: - options = builder.BuildOptions(options.chunker, self._hash_check()) + options = builder.BuildOptions(options.chunker, self._hash_check(), progress) with self._refusing(), self.storage.protect() as session: with session.writer() as put: if recipients: @@ -1280,6 +1334,31 @@ def add_stream(self, stream: BinaryIO, size: int | None = None, name: str | None self._added(built) return {"cid": str(built.cid), "size": built.size, "warnings": [], **({"sealed": True} if recipients else {})} + def add_tar(self, stream: BinaryIO, name: str | None = None, only_hash: bool = False, pin: bool = True, + chunker: str | None = None) -> dict: + """Import the directory a tar stream holds (POST /api/add/stream?format=tar; rnfs/content/tarstream.py's + rules and limits, max_output_entries files and directories at most), wrapped as the entry name if given, + pinned "user" (cache if not pin) unless only_hash; chunker as for add_path. {"cid", "size", "warnings"} + (+ "warnings_total" past MAX_WARNINGS, as get caps them). Reads the stream to its end, where only zeros + may follow the tar's own.""" + from rnfs.content import tarstream + + options = builder.BuildOptions(chunker) # ValueError for an unknown chunker, before anything is read + warnings = tarstream.Warnings() + max_entries = self.config.max_output_entries + if only_hash: + built = tarstream.build_tar(stream, builder.NO_PUT, warnings, max_entries, options, name) + else: + options = builder.BuildOptions(options.chunker, self._hash_check()) + with self._refusing(), self.storage.protect() as session: + with session.writer() as put: + built = tarstream.build_tar(stream, put, warnings, max_entries, options, name) + node_content.keep_added(self, built.cid, pin) + if pin: + self.pin_names.record(built.cid, name or "(tar upload)") + self._added(built) + return {"cid": str(built.cid), "size": built.size, **warnings.result()} + def block(self, cid: str, for_gateway: bool = False) -> bytes: """A CID's own block, fetched if needed and re-verified (the gateway's ?format=raw).""" return node_content.block(self, cid, for_gateway) @@ -1289,6 +1368,20 @@ def export_stream(self, target: str, start, for_gateway: bool = False) -> dict: writes nothing: HEAD). The gateway's ?format=archive (rnfs/node_content.py).""" return node_content.export_stream(self, target, start, for_gateway) + def export_chunks(self, target: str, for_gateway: bool = False) -> tuple[dict, Iterator[bytes]]: + """(info, chunks): the target's DAG fetched and checked, its archive's {"roots", "blocks", "length"}, and an + iterator of the archive's bytes, byte-identical to `rnfs export` (GET /api/export/stream). Close chunks when + done, read or not: that ends the fetch.""" + chunks = node_content.export_chunks(self, target, for_gateway) + return next(chunks), chunks + + def get_tar(self, target: str) -> tuple[dict, Iterator[bytes]]: + """(info, chunks): the target's DAG fetched and checked, its tar's {"cid", "length", "skipped"}, and an + iterator of the tar's bytes (GET /api/get/stream?format=tar; node_content.tar_chunks). Close chunks when + done, read or not: that ends the fetch.""" + chunks = node_content.tar_chunks(self, target) + return next(chunks), chunks + @contextmanager def _refusing(self) -> Iterator[None]: """Around an add or import, which the deny list may stop (a denied block, or a file whose content hash is @@ -1549,14 +1642,18 @@ def pin_rm(self, cid: str): self.publisher.refresh() def pin_ls(self, cid: str | None = None) -> list[dict]: - """Pins with their sources, whether they're complete, their type, whether they're sealed files and a - local name when one is known. - - The name is where the pin was added from on this machine (pin_names), the shared-folder entry it - came from, or for a stored directory its single entry's name (add --wrap) or its entry count. For a - sealed file it is still the local path or the name given to add (this node's own record, for its user): - `sealed` tells callers that show pins to others not to show the name or the content. `sealed` is None - when it can't be told without fetching (the root or the file's first chunk isn't stored). + """Pins with their sources, whether they're complete, their type, whether they're sealed files, a local + name when one is known, and (rnfs 0.1.7) their label, origin, size, entry count and when added. + + origin is where the pin was added from on this machine (pin_names) or the shared-folder entry it came + from; label the name its user gave it (pin_label), kept apart from origin. name, as before 0.1.7, is the + label, else the origin, else for a stored directory its single entry's name (add --wrap) or its entry + count as text. For a sealed file origin is still the local path or the name given to add (this node's own + record, for its user): `sealed` tells callers that show pins to others not to show the name or the content. + `sealed` is None when it can't be told without fetching (the root or the file's first chunk isn't stored). + size is the content size the root block states (None until it is stored), entries a directory's entry + count (None for a file, or while a large directory's index isn't all stored), added when it was first + pinned (None for a pin made before rnfs 0.1.7). """ pins = self.storage.pins() self.pin_names.prune(set(pins)) @@ -1570,14 +1667,32 @@ def pin_ls(self, cid: str | None = None) -> list[dict]: result = [] for c, sources in sorted(pins.items(), key=lambda item: str(item[0])): complete = self.storage.is_complete(c) - kind, inside, sealed = self._pin_details(c, complete) - name = self.pin_names.get(c) or (folder_names.get(c) if FOLDER in sources else None) or inside + details = self._pin_details(c, complete) + origin = self.pin_names.get(c) or (folder_names.get(c) if FOLDER in sources else None) + label = self.pin_names.label(c) result.append({"cid": str(c), "sources": sorted(sources), "complete": complete, - "type": kind, "sealed": sealed, "name": name}) + "type": details.kind, "sealed": details.sealed, + "name": label or origin or details.inside, "label": label, "origin": origin, + "size": details.size, "entries": details.entries, "added": self.storage.pin_added(c)}) return result - def _pin_details(self, cid: Cid, complete: bool) -> tuple[str | None, str | None, bool | None]: - """(type, name from inside, sealed) of a pin (_pin_shape, _pin_sealed), remembered while it is complete. + def pin_label(self, cid: str, label: str | None) -> dict: + """Give a pin a label (None or "" clears it), kept apart from where it was added from: its pin_ls entry. + NotFound if cid isn't pinned; ValueError for a label pin_names.check_label refuses. A label lasts while + the CID stays pinned (by any source).""" + parsed = Cid.parse(cid) + if not self.storage.is_pinned(parsed): + raise NotFound(f"{cid} is not pinned") + if self.pin_names.set_label(parsed, label): + events.publish(self, "pin", change="label", source=None, label=self.pin_names.label(parsed), + **events.cid_fields([parsed])) + listed = self.pin_ls(str(parsed)) + if not listed: # released meanwhile + raise NotFound(f"{cid} is not pinned") + return listed[0] + + def _pin_details(self, cid: Cid, complete: bool) -> "_PinDetails": + """A pin's _PinDetails (_pin_shape, _pin_sealed), remembered while it is complete. They are read from stored blocks, which a complete pin has all of and whose content never changes, so they hold for as long as it stays complete; an incomplete pin's are read each time (a block may arrive, @@ -1589,10 +1704,11 @@ def _pin_details(self, cid: Cid, complete: bool) -> tuple[str | None, str | None known = self._pin_memo.get(cid) if known is not None: return known - kind, inside = self._pin_shape(cid) - details = (kind, inside, self._pin_sealed(cid, kind)) - # A complete DAG's blocks always tell both; None means a block went while they were read - if complete and kind is not None and details[2] is not None: + shape = self._pin_shape(cid) + details = shape._replace(sealed=self._pin_sealed(cid, shape.kind)) + # A complete DAG's blocks always tell them all; None means a block went while they were read + if complete and details.kind is not None and details.sealed is not None \ + and (details.kind == "file" or details.entries is not None): with self._pin_memo_lock: if len(self._pin_memo) < PIN_MEMO: self._pin_memo[cid] = details @@ -1609,25 +1725,59 @@ def _pin_sealed(self, cid: Cid, kind: str | None) -> bool | None: head = node_content.stored_head(self, str(cid), len(seal.MAGIC), deny=False) return None if head is None else seal.is_sealed(head) - def _pin_shape(self, cid: Cid) -> tuple[str | None, str | None]: - """(type, name from inside) of a pin whose root block is stored: "file" or "directory", and for a - directory its one entry's name or "(directory: N entries)". (None, None) if the root isn't stored.""" + def _pin_shape(self, cid: Cid, count: bool = True) -> "_PinDetails": + """The _PinDetails of a pin whose root block is stored, but sealed (None): its type ("file" or + "directory"), for a directory its one entry's name or "(directory: N entries)" (inside; a large + directory's index root, "(directory)" as before 0.1.7) and, if count, its entry count, and its content + size. All None if the root isn't stored (or isn't a valid DAG node).""" + unknown = _PinDetails(None, None, None, None, None) + if cid.is_raw: # a raw block is its own content: its size is known without reading it + size = self.storage.block_size(cid) + return unknown if size is None else _PinDetails("file", None, None, None, size) block = self.storage.get(cid) if block is None: - return None, None - if cid.is_raw: - return "file", None + return unknown try: node = dag.decode(block) except InvalidDag: - return None, None + return unknown if isinstance(node, dag.FileNode): - return "file", None + return _PinDetails("file", None, None, None, node.size) if isinstance(node, dag.DirectoryNode): - if len(node.entries) == 1: - return "directory", node.entries[0].name - return "directory", f"(directory: {len(node.entries)} entries)" - return "directory", "(directory)" + inside = node.entries[0].name if len(node.entries) == 1 else f"(directory: {len(node.entries)} entries)" + return _PinDetails("directory", inside, None, len(node.entries), node.size) + return _PinDetails("directory", "(directory)", None, self._index_entries(node) if count else None, node.size) + + def _index_entries(self, index: dag.IndexNode) -> int | None: + """The entries below a large directory's index node, counted from its stored groups; None if one of its + index or group nodes isn't stored (or isn't valid), or if the count would pass max_output_entries. Never + fetches. A node the index lists twice is invalid (a valid index never repeats a child), which also stops + a few crafted blocks from making the walk grow exponentially; at most max_output_entries nodes are read.""" + count = 0 + seen: set[Cid] = set() + stack = list(index.child_cids) + limit = self.config.max_output_entries + while stack: + cid = stack.pop() + if cid in seen or len(seen) >= limit: + return None + seen.add(cid) + block = self.storage.get(cid) + if block is None: + return None + try: + node = dag.decode(block) + except InvalidDag: + return None + if isinstance(node, dag.DirectoryNode): + count += len(node.entries) + if count > limit: + return None + elif isinstance(node, dag.IndexNode): + stack.extend(node.child_cids) + else: + return None + return count def pin_remote(self, node_id: bytes, cid: str, pin: bool) -> bool: """Ask node_id to pin (or release our pin of) cid over an identified link; True if accepted.""" @@ -1851,6 +2001,38 @@ def gc(self) -> dict: log(f"gc removed {plural(stats['removed_blocks'], 'block')} ({size(stats['freed_bytes'])})", LOG_NOTICE) return {"removed_blocks": stats["removed_blocks"], "freed_bytes": stats["freed_bytes"]} + def cache_ls(self) -> list[dict]: + """The cached fetches (downloads not pinned, kept until evicted; section 8), most recently used first: + {"cid" (the target), "root", "path" (the names from root to it, [] for a bare CID), "pinned" (the same + content is also pinned), "complete", "size" (as its root block states it, None until stored), "type", + "provided" (offered to the network: the roots strategy, a complete download for the user)}. One target + reached through several paths is listed once per path. Never fetches.""" + provided = self.publisher.provided() + pinned = self.storage.pinned_cids() + result = [] + for root, names, target in reversed(self.storage.cached_fetches()): + shape = self._pin_shape(target, count=False) # the type and size only: no walk of a large directory + result.append({"cid": str(target), "root": str(root), "path": list(names), "pinned": target in pinned, + "complete": self.storage.is_complete(target), "type": shape.kind, "size": shape.size, + "provided": target in provided}) + return result + + def cache_rm(self, cid: str) -> dict: + """Evict the cached fetches leading to cid now (as gc does for all of them), deleting the blocks nothing + else needs: {"removed_records", "removed_blocks", "freed_bytes"}. A pin of the same content keeps its + blocks. NotFound if no cached fetch leads to cid.""" + parsed = Cid.parse(cid) + stats = self.storage.uncache({parsed}) + if not stats["removed_records"]: + raise NotFound(f"{cid} is not in the cache") + self.publisher.refresh() + self._prune_fetched() + self._save_state() + log(f"cache rm {parsed} removed {plural(stats['removed_blocks'], 'block')} ({size(stats['freed_bytes'])})", + LOG_NOTICE) + return {"removed_records": stats["removed_records"], "removed_blocks": stats["removed_blocks"], + "freed_bytes": stats["freed_bytes"]} + def verify(self) -> dict: """Re-check every stored block against its CID, deleting bad ones.""" stats = self.storage.verify() @@ -1942,12 +2124,34 @@ def export(self, target: str, output: str) -> dict: raise return {"roots": [str(cid) for cid in stats["roots"]], "blocks": stats["blocks"]} - def import_archive(self, path: str) -> dict: - """Import an archive, pinning its complete roots with source "user".""" + def import_archive(self, path: str, progress: Callable[[int], None] | None = None) -> dict: + """Import an archive on this machine, pinning its complete roots with source "user" (import_stream; progress + as there: an import job's). Raises ValueError for anything but a regular file: opened without blocking, so a + FIFO can't hold the call (or an import job's slot) waiting for a writer.""" try: - stream = open(path, "rb") + fd = os.open(path, os.O_RDONLY | fsutil.O_BINARY | getattr(os, "O_NONBLOCK", 0)) except FileNotFoundError: raise NotFound(f"no such file: {path}") from None + except IsADirectoryError: + raise ValueError(f"{path} is not a regular file") from None + try: + if not stat.S_ISREG(os.fstat(fd).st_mode): + raise ValueError(f"{path} is not a regular file") + stream = os.fdopen(fd, "rb") + except BaseException: + os.close(fd) + raise + with stream: + return self.import_stream(stream, f"archive {os.path.abspath(path)}", progress) + + def import_stream(self, stream: BinaryIO, label: str = "(archive upload)", + progress: Callable[[int], None] | None = None) -> dict: + """Import an archive read from stream to its end (POST /api/import/stream), pinning its complete roots with + source "user", each recorded in pin_names as label unless it has a name there. progress, if given, is told + the bytes read as they are (an import job's) and may raise to stop the import: blocks stored by then stay as + unpinned blocks, for the next collection. {"roots", "incomplete", "blocks"}.""" + stream = builder.counted(stream, progress) + def check(roots: list[Cid]): """Refuse the import if a file the roots hold has a denied content hash.""" if self.deny.hashes: @@ -1956,10 +2160,10 @@ def check(roots: list[Cid]): for file in walker.iter_files(walker.root_expect(root), get): self._check_file(file, get) # a complete root: every block is stored and checked - with stream, self._refusing(): + with self._refusing(): stats = archive.import_archive(stream, self.storage, check=check) for root in stats.get("complete", []): - self.pin_names.record_if_unknown(root, f"archive {os.path.abspath(path)}") + self.pin_names.record_if_unknown(root, label) self.publisher.refresh() self._enforce_storage() return {"roots": [str(cid) for cid in stats["roots"]], "incomplete": [str(cid) for cid in stats["incomplete"]], diff --git a/rnfs/node_content.py b/rnfs/node_content.py index 56ccb0b..d806f98 100644 --- a/rnfs/node_content.py +++ b/rnfs/node_content.py @@ -4,13 +4,14 @@ - Sealed files (rnfs/content/seal.py): resolving recipients to public keys, sealing a file being added, and opening a sealed target with the node's identity for get, cat and info. - Gateway formats: a CID's own block (?format=raw) and a target's archive streamed with its exact length - (?format=archive). + (?format=archive, and GET /api/export/stream); a target as a tar streamed the same way (GET /api/get/stream). Functions take the RnfsNode first. RNS is imported inside functions: the daemon has loaded it already. """ import os import stat -from collections.abc import Callable +import time +from collections.abc import Callable, Iterator from typing import BinaryIO from rnfs.content import archive, builder, dag, seal, walker @@ -84,7 +85,7 @@ def sealing_stream(stream: BinaryIO, size: int | None, recipients: list[bytes], def add_sealed_path(path: str, put: builder.Put, wrap: bool, recipients: list[bytes], seal_name: bool, options: builder.BuildOptions | None = None) -> builder.Built: """Seal the regular file at path and build it (wrapped under its visible name if wrap), with options (the - chunker, and the deny check of the sealed bytes; builder.build_file).""" + chunker, the deny check of the sealed bytes, builder.build_file; and progress, given the plaintext read).""" handle, st = builder.open_path(path) try: if not stat.S_ISREG(st.st_mode): @@ -92,7 +93,8 @@ def add_sealed_path(path: str, put: builder.Put, wrap: bool, recipients: list[by name = os.path.basename(os.path.normpath(os.path.abspath(path))) sealed_name = name if seal_name and dag.is_valid_name(name) else None with os.fdopen(handle, "rb", closefd=False) as f: - return builder.add_stream(seal.SealingReader(f, recipients, sealed_name), put, None, + plain = builder.counted(f, options.progress if options is not None else None) + return builder.add_stream(seal.SealingReader(plain, recipients, sealed_name), put, None, name if wrap else None, options=options) finally: builder.close(handle) @@ -234,24 +236,73 @@ def block(node, cid: str, for_gateway: bool = False) -> bytes: def export_stream(node, target: str, start: Callable[[int], BinaryIO | None], for_gateway: bool = False) -> dict: """Fetch the target's DAG, then write its archive to start(exact archive length): the stream to write to, - or None to write nothing. Limits as for export: max_download_size, max_output_entries, and the DAG - byte bound of fetch_dag. Every file is checked against the deny list's file hashes before anything is sent, - as the gateway checks what it sends.""" - root, names = walker.parse_target(target) + or None to write nothing. {"roots", "blocks", "length"}; limits and checks as export_chunks.""" + chunks = export_chunks(node, target, for_gateway) + try: + info = next(chunks) + out = start(info["length"]) + if out is not None: + for data in chunks: + out.write(data) + return info + finally: + chunks.close() # ends the fetch, as leaving its block would + + +def _fetched_dag(node, fetch, expect: walker.Expect): + """Fetch the whole DAG of a resolved target to send it (an archive or a tar): limits as for export + (max_download_size, max_output_entries counting every occurrence, the DAG byte bound of fetch_dag), and every + file checked against the deny list's file hashes before anything is sent, as the gateway checks what it + sends.""" + fetch.fetch_dag(expect, node.config.max_output_entries) + walker.count_entries(expect, fetch.get, node.config.max_output_entries) + if node.deny.hashes: + for file in walker.iter_files(expect, fetch.get): + node._check_file(file, fetch.get) + + +def export_chunks(node, target: str, for_gateway: bool = False) -> Iterator: + """The target's archive as a generator: first {"roots", "blocks", "length"} once its DAG is fetched and + checked (_fetched_dag), then its bytes, byte-identical to `rnfs export` (?format=archive, GET + /api/export/stream). The fetch stays open until the generator ends or is closed. A leading contact nick is + expanded unless for_gateway.""" + root, names = node._parse_target(target, for_gateway) with node._fetching(root, names, None, for_gateway) as fetch: expect = fetch.resolve() - fetch.fetch_dag(expect, node.config.max_output_entries) - walker.count_entries(expect, fetch.get, node.config.max_output_entries) - if node.deny.hashes: - for file in walker.iter_files(expect, fetch.get): - node._check_file(file, fetch.get) + _fetched_dag(node, fetch, expect) blocks = node.storage.dag_blocks(expect.cid) length = archive.archive_size([expect.cid], ((cid, node.storage.block_size(cid) or 0) for cid in blocks)) - out = start(length) - if out is None: - return {"roots": [str(expect.cid)], "blocks": len(blocks), "length": length} - stats = archive.export_archive([expect], fetch.get, out) - return {"roots": [str(cid) for cid in stats["roots"]], "blocks": stats["blocks"], "length": length} + yield {"roots": [str(expect.cid)], "blocks": len(blocks), "length": length} + yield from archive.archive_chunks([expect], fetch.get) + + +SKIPS_LOGGED = 10 # entries a tar of a target skips that are logged by name (the rest only counted) + + +def tar_chunks(node, target: str) -> Iterator: + """The target as a tar (rnfs/content/tarstream.py), as a generator: first {"cid", "length", "skipped"} once its + DAG is fetched and checked (_fetched_dag), then the tar's bytes (GET /api/get/stream?format=tar). A file is + one member named by the target's last path name (or its CID for a bare CID, or one unsafe in a tar); entries + whose names are unsafe in a tar are skipped (counted in "skipped", the first SKIPS_LOGGED logged). The fetch + stays open until the generator ends or is closed.""" + from rnfs.content import tarstream + from rnfs.log import LOG_WARNING, log + + root, names = node._parse_target(target) + max_entries = node.config.max_output_entries + with node._fetching(root, names) as fetch: + expect = fetch.resolve() + _fetched_dag(node, fetch, expect) + name = names[-1] if names and tarstream.safe_name(names[-1]) else str(expect.cid) + skipped = tarstream.Warnings(SKIPS_LOGGED) + members = tarstream.plan(expect, fetch.get, name, max_entries, skipped) + for path in skipped.kept: + log(f"tar of {target}: skipped {path!r}: a character in its name could be read as a path", LOG_WARNING) + if skipped.total > len(skipped.kept): + log(f"tar of {target}: skipped {skipped.total - len(skipped.kept)} more such entries", LOG_WARNING) + mtime = int(time.time()) + yield {"cid": str(expect.cid), "length": tarstream.length(members, mtime), "skipped": skipped.total} + yield from tarstream.chunks(members, fetch.get, mtime, max_entries) # API parameters diff --git a/rnfs/objects.py b/rnfs/objects.py index 87794d9..5909306 100644 --- a/rnfs/objects.py +++ b/rnfs/objects.py @@ -1,15 +1,18 @@ """The objects rnfs.Node and rnfs.Client return (design.md, "Public API"; provisional until 0.2). No RNS. Each is a frozen dataclass made from the local API's JSON form (from_dict, which the facade's dicts share) and -turned back into it (to_dict: CIDs as their written form, node IDs and names as hex). str() gives the CID, name -or path where an object has one obvious text form. Cid is rnfs.content.cid's. +turned back into it (to_dict: CIDs as their written form, node IDs, names and hashes as hex, nested objects as +their own form). str() gives the CID, name or path where an object has one obvious text form. A field whose JSON +key is a Python keyword ("from") is named otherwise (sender, from_node). Cid is rnfs.content.cid's. """ from dataclasses import MISSING, dataclass, field, fields from rnfs.content.cid import Cid __all__ = ["Cid", "Added", "Written", "Entry", "Info", "Stat", "Pin", "Record", "Status", "Collected", "Verified", - "Exported", "Imported", "DenyState"] + "Exported", "Imported", "DenyState", "Peer", "Stats", "Contact", "Kept", "Followed", "LxmfAddress", + "MessageLink", "Message", "Outgoing", "Sent", "Synced", "Cached", "Uncached", "Event", "EventBatch", + "Capabilities", "Setting", "Settings", "SettingsChange"] def _cid(text) -> Cid: @@ -23,24 +26,36 @@ def _hex(value) -> bytes | None: def _dump(value): - """A field's JSON form: CIDs written out, bytes as hex, tuples as lists.""" + """A field's JSON form: CIDs written out, bytes as hex, tuples as lists, objects as their to_dict().""" if isinstance(value, Cid): return str(value) if isinstance(value, (bytes, bytearray)): return bytes(value).hex() if isinstance(value, tuple): return [_dump(item) for item in value] + if isinstance(value, frozenset): + return sorted(_dump(item) for item in value) + if isinstance(value, _Object): + return value.to_dict() return value -def _field(load=None, default=MISSING): - """A dataclass field whose JSON value from_dict turns into its type with load.""" +def _field(load=None, default=MISSING, key: str | None = None): + """A dataclass field whose JSON value from_dict turns into its type with load; key is its JSON key, if not + the field's name.""" metadata = {"load": load} if load is not None else {} + if key is not None: + metadata["key"] = key if default is MISSING: return field(metadata=metadata) return field(default=default, metadata=metadata) +def _key(f) -> str: + """A field's JSON key.""" + return f.metadata.get("key", f.name) + + def _json_fields(cls) -> list: """cls's fields that are JSON fields of their own (not those marked metadata={"json": False}).""" return [f for f in fields(cls) if f.metadata.get("json", True)] @@ -56,28 +71,31 @@ class _Object: def to_dict(self) -> dict: """The local API's JSON form.""" - return {f.name: _dump(getattr(self, f.name)) for f in _json_fields(self)} + return {_key(f): _dump(getattr(self, f.name)) for f in _json_fields(self)} @classmethod def from_dict(cls, data: dict): - """The object for a JSON form (the API's, or the facade's dict); fields missing take their defaults.""" + """The object for a JSON form (the API's, or the facade's dict); fields missing take their defaults, and + keys it doesn't know (a newer daemon's) are left out.""" values = {} for f in _json_fields(cls): - if f.name in data: + if _key(f) in data: load = f.metadata.get("load") - value = data[f.name] + value = data[_key(f)] values[f.name] = load(value) if load is not None and value is not None else value return cls(**values) @dataclass(frozen=True) class Added(_Object): - """What add() stored: the root's CID and size, warnings (entries a directory add skipped), whether sealed.""" + """What add() (or add_tar()) stored: the root's CID and size, warnings (entries a directory add or a tar + skipped; warnings_total counts them past the first 100, where an add_tar caps them), whether sealed.""" cid: Cid = _field(_cid) size: int = 0 warnings: tuple[str, ...] = _field(tuple, ()) sealed: bool = False + warnings_total: int | None = None def __str__(self) -> str: """The CID's written form.""" @@ -149,7 +167,10 @@ class Stat(_Object): @dataclass(frozen=True) class Pin(_Object): """A pin: its CID, sources ("user", "folder", "name:", ...), whether complete, its type and whether a - sealed file (None when its stored blocks can't tell), and a local name for it if one is known.""" + sealed file (None when its stored blocks can't tell), and a local name for it if one is known. From 0.1.7 (None + from an older daemon, or where not known): the label its user gave it (label_pin), where it was added from on + this machine (origin), its content size (once its root block is stored), a directory's entry count, and when + it was first pinned (Unix time; None for a pin made before 0.1.7).""" cid: Cid = _field(_cid) sources: tuple[str, ...] = _field(tuple, ()) @@ -157,6 +178,11 @@ class Pin(_Object): type: str | None = None sealed: bool | None = None name: str | None = None + label: str | None = None + origin: str | None = None + size: int | None = None + entries: int | None = None + added: float | None = None def __str__(self) -> str: """The CID's written form.""" @@ -246,3 +272,384 @@ class DenyState(_Object): entries: tuple[str, ...] = _field(tuple, ()) lists: tuple[dict, ...] = _field(tuple, ()) + + +@dataclass(frozen=True) +class Peer(_Object): + """A node this one knows (peers()): its node ID, DHT mode ("server", "client", None unknown), whether in the + routing table and for how many seconds (seniority; senior once past provider_seniority), the path's hops, + interface and bitrate, whether a link is open and its rate estimate (bits/s), when it was last heard and last + active (Unix times), whether banned, the traffic counted since the node started (bytes sent and received, + requests made, answered, refused, failed, and bad blocks), and from 0.1.7 when it was first heard, the + round-trip seconds of the last link to it (rtt, measured at rtt_at) and whether it is online now (None from a + daemon that predates them).""" + + node_id: bytes = _field(_hex) + mode: str | None = None + in_table: bool = False + seniority: float | None = None + senior: bool = False + hops: int | None = None + interface: str | None = None + bitrate: float | None = None + link: bool = False + rate: float | None = None + last_heard: float | None = None + last_active: float | None = None + banned: bool = False + sent: int = 0 + received: int = 0 + requests: int = 0 + answered: int = 0 + refused: int = 0 + failed: int = 0 + bad: int = 0 + first_heard: float | None = None + rtt: float | None = None + rtt_at: float | None = None + online: bool | None = None + + def __str__(self) -> str: + """The node ID in hex.""" + return self.node_id.hex() + + +@dataclass(frozen=True) +class Stats(_Object): + """The node's traffic since it started (stats()): since (Unix time), totals ({"sent", "received", + "responses", "requests", "answered", "refused", "failed"}), the /blocks refusals by reason, the bytes sent and + received per interface ({name: {"sent", "received"}}), and the upload budget per hour and what is left of it + (None without one).""" + + since: float | None = None + totals: dict = field(default_factory=dict) + refused: dict = field(default_factory=dict) + interfaces: dict = field(default_factory=dict) + upload_budget: int | None = None + upload_budget_left: int | None = None + + +@dataclass(frozen=True) +class Contact(_Object): + """An address book entry (contacts): its nick, its kind ("name", "node" or "lxmf") and the hash it names (16 + bytes).""" + + nick: str + kind: str + id: bytes = _field(_hex) + + def __str__(self) -> str: + """The nick.""" + return self.nick + + +@dataclass(frozen=True) +class Kept(_Object): + """A name this node keeps (names.keep): the name (16 bytes), why it is kept ("user", "follow", "own"), and the + signed record held, if any: its sequence, CID and emission time (Unix).""" + + name: bytes = _field(_hex) + reasons: tuple[str, ...] = _field(tuple, ()) + sequence: int | None = None + cid: Cid | None = _field(_cid, None) + emitted: float | None = None + + def __str__(self) -> str: + """The name in hex.""" + return self.name.hex() + + +@dataclass(frozen=True) +class Followed(_Object): + """A name this node follows (names.follow): the name (16 bytes), the target pinned for it (None until the + first is complete), the latest target and sequence known, and what the last check did (state: "waiting", + "current", "unresolved", "fetching ", or why it failed).""" + + name: bytes = _field(_hex) + pinned: Cid | None = _field(_cid, None) + latest: Cid | None = _field(_cid, None) + sequence: int | None = None + state: str = "waiting" + + def __str__(self) -> str: + """The name in hex.""" + return self.name.hex() + + +@dataclass(frozen=True) +class LxmfAddress(_Object): + """This node's LXMF identity (lxmf.address()): its address (16 bytes), display name, the propagation node set + (lxmf_propagation_node) and the nearest one heard (None: none).""" + + address: bytes = _field(_hex) + display_name: str | None = None + propagation_node: bytes | None = _field(_hex, None) + nearest_propagation_node: bytes | None = _field(_hex, None) + + def __str__(self) -> str: + """The address in hex.""" + return self.address.hex() + + +@dataclass(frozen=True) +class MessageLink(_Object): + """An rnfs link in an LXMF message (section 13): its rnfs: URI, the CID (a name link's: what it pointed at when + sent) or the name (16 bytes) and the path below it, the size and provider hint (from_node) as the sender gave + them (untrusted: fetches verify).""" + + uri: str + cid: Cid | None = _field(_cid, None) + name: bytes | None = _field(_hex, None) + path: tuple[str, ...] = _field(tuple, ()) + size: int | None = None + from_node: bytes | None = _field(_hex, None, key="from") + + def __str__(self) -> str: + """The rnfs: URI.""" + return self.uri + + +def _links(values) -> tuple[MessageLink, ...]: + """A message's links from their JSON form.""" + return tuple(MessageLink.from_dict(value) for value in values) + + +@dataclass(frozen=True) +class Message(_Object): + """A message in the LXMF inbox (lxmf.inbox()): its inbox id, hash, sender's LXMF address (sender, the API's + "from") and display name (the sender's own word: trusted and contact say what this node knows of it), when + sent (the sender's clock; None if implausible) and received, title and text, delivery method, its rnfs links, + each link's auto-fetch result ({"state", "cid" or "error"}; None unless the sender is in lxmf_auto_fetch_from), + whether the sender is trusted now, and the nick of the LXMF contact with its address.""" + + id: int + hash: bytes = _field(_hex) + sender: bytes = _field(_hex, key="from") + display_name: str | None = None + sent: float | None = None + received: float | None = None + title: str = "" + text: str = "" + method: str = "unknown" + links: tuple[MessageLink, ...] = _field(_links, ()) + auto: tuple[dict, ...] | None = _field(tuple, None) + trusted: bool = False + contact: str | None = None + + +@dataclass(frozen=True) +class Outgoing(_Object): + """A message in the LXMF outbox (lxmf.outbox()): its outbox id, hash, recipient, when queued, its links and + text, method ("direct" or "propagated"), state ("outbound", "sending", "sent", "delivered", "failed", ...) and + progress, the error of a failed propagation fallback, and for a direct message sent again through the + propagation node, the direct attempt's hash (direct_hash) and fallback True.""" + + id: int + hash: bytes = _field(_hex) + to: bytes = _field(_hex) + time: float | None = None + links: tuple[MessageLink, ...] = _field(_links, ()) + text: str = "" + method: str = "direct" + state: str = "outbound" + progress: float | None = None + error: str | None = None + direct_hash: bytes | None = _field(_hex, None) + fallback: bool = False + + +@dataclass(frozen=True) +class Sent(_Object): + """A message lxmf.send() queued: its outbox id, hash, recipient's address and method; the outbox says how it + went.""" + + id: int + hash: bytes = _field(_hex) + to: bytes = _field(_hex) + method: str = "direct" + + +@dataclass(frozen=True) +class Synced(_Object): + """What lxmf.sync() did: the propagation node asked (None: none set or heard) and whether a download + started.""" + + propagation_node: bytes | None = _field(_hex, None) + started: bool = False + + +@dataclass(frozen=True) +class Cached(_Object): + """A cached fetch (cache()): a download kept until evicted, not pinned. cid is the target, reached from root + through path (() for a bare CID); whether the same content is pinned too, complete, its type and size (None + until its root block is stored), and whether it is provided to the network.""" + + cid: Cid = _field(_cid) + root: Cid | None = _field(_cid, None) + path: tuple[str, ...] = _field(tuple, ()) + pinned: bool = False + complete: bool = False + type: str | None = None + size: int | None = None + provided: bool = False + + def __str__(self) -> str: + """The CID's written form.""" + return str(self.cid) + + +@dataclass(frozen=True) +class Uncached(_Object): + """What uncache() evicted: the cache records, and the blocks (and bytes) nothing else needed.""" + + removed_records: int = 0 + removed_blocks: int = 0 + freed_bytes: int = 0 + + +@dataclass(frozen=True) +class Event(_Object): + """Something that changed on the node (events): seq (1, 2, ... in the node's run), kind ("job", "pin", + "cache", "inbox", "name", "peer"; design.md, "Events", lists each kind's fields), time (Unix), and the kind's + own fields in fields. kind "lost" (lost is True) is no node's: events.watch delivers it when events were missed + (the watch fell behind, or the daemon restarted), so what the app shows should be read again; its fields hold + the run's "epoch".""" + + seq: int + kind: str + time: float | None = None + fields: dict = field(default_factory=dict, metadata={"json": False}) + + @property + def lost(self) -> bool: + """True for the "lost" marker.""" + return self.kind == "lost" + + def to_dict(self) -> dict: + """The API's form: seq, kind and time with the kind's fields merged in.""" + return {**self.fields, **super().to_dict()} + + @classmethod + def from_dict(cls, data: dict) -> "Event": + """From the API's form, every key but seq, kind and time going into fields.""" + known = {"seq", "kind", "time"} + return cls(data["seq"], data["kind"], data.get("time"), + {key: value for key, value in data.items() if key not in known}) + + +def _events(values) -> tuple[Event, ...]: + """Events from their JSON form.""" + return tuple(Event.from_dict(value) for value in values) + + +@dataclass(frozen=True) +class EventBatch(_Object): + """What events.since() answered: the events, the sequence to ask after next, whether events were missed + (lost: the node's run changed, or they are no longer held; events then holds every one held, and the app should + read again what it shows), and the run's epoch to pass back.""" + + events: tuple[Event, ...] = _field(_events, ()) + next: int = 0 + lost: bool = False + epoch: str | None = None + + +def _routes(values) -> tuple[tuple[str, str], ...]: + """Routes ([method, path] pairs) as tuples.""" + return tuple(tuple(route) for route in values) + + +@dataclass(frozen=True) +class Capabilities(_Object): + """What the node offers (capabilities()): its version, its capabilities (api.CAPABILITIES: "events", + "pin_label", "config", ...), the API routes ((method, path)), the features loaded, the event kinds and the + error codes. Empty from a daemon before 0.1.7, which can't say: every has() is False.""" + + version: str | None = None + capabilities: frozenset[str] = _field(frozenset, frozenset()) + routes: tuple[tuple[str, str], ...] = _field(_routes, ()) + features: tuple[str, ...] = _field(tuple, ()) + event_kinds: tuple[str, ...] = _field(tuple, ()) + error_codes: tuple[str, ...] = _field(tuple, ()) + + def has(self, capability: str) -> bool: + """Whether the node offers capability.""" + return capability in self.capabilities + + +def _frozen(value): + """A setting's JSON value with lists made tuples (so a frozen Setting compares and reads alike from either + class: the API's JSON has lists where a Node may have either).""" + if isinstance(value, (list, tuple)): + return tuple(_frozen(item) for item in value) + return value + + +@dataclass(frozen=True) +class Setting(_Object): + """One setting (config()): its schema (rnfs.config.settings_schema: name, group, type, choices, nullable, + default, unit, minimum, maximum, exclusive_minimum, description, live: a change applies without a restart), + the running value, rnfs.toml's (file; None where it doesn't set it), whether the command line (a Node's + settings) overrides rnfs.toml, and whether a restart would run another value (pending). Lists are tuples.""" + + name: str + group: str | None = None + type: str | None = None + choices: tuple | None = _field(tuple, None) + nullable: bool = False + default: object = _field(_frozen, None) + unit: str | None = None + minimum: float | None = None + maximum: float | None = None + exclusive_minimum: bool = False + description: str | None = None + live: bool = False + value: object = _field(_frozen, None) + file: object = _field(_frozen, None) + overridden: bool = False + pending: bool = False + + def __str__(self) -> str: + """The setting's name.""" + return self.name + + +def _settings(values) -> tuple[Setting, ...]: + """Settings from their JSON form.""" + return tuple(Setting.from_dict(value) for value in values) + + +@dataclass(frozen=True) +class Settings(_Object): + """The node's settings (config()): its rnfs.toml's path, every setting in the schema's order (settings[name] + gives one, iterating gives each), why a restart would refuse rnfs.toml (file_error, None if it wouldn't), and + the warnings reading it gave.""" + + path: str | None = None + settings: tuple[Setting, ...] = _field(_settings, ()) + file_error: str | None = None + warnings: tuple[str, ...] = _field(tuple, ()) + + def __getitem__(self, name: str) -> Setting: + """The setting called name. KeyError if there is none.""" + for setting in self.settings: + if setting.name == name: + return setting + raise KeyError(name) + + def __iter__(self): + """Each setting.""" + return iter(self.settings) + + +@dataclass(frozen=True) +class SettingsChange(_Object): + """What config_set() did to rnfs.toml (path): the settings applied to the running node now, those whose new + value applies at the next start (restart_required), those the command line overrides (overridden, which keeps + winning), and the warnings reading the file gave.""" + + path: str | None = None + applied: tuple[str, ...] = _field(tuple, ()) + restart_required: tuple[str, ...] = _field(tuple, ()) + overridden: tuple[str, ...] = _field(tuple, ()) + warnings: tuple[str, ...] = _field(tuple, ()) diff --git a/rnfs/pin_names.py b/rnfs/pin_names.py index 8d94008..6af41a1 100644 --- a/rnfs/pin_names.py +++ b/rnfs/pin_names.py @@ -1,4 +1,5 @@ -"""Local names for pins: where each pin came from on this machine, for `rnfs pin ls`. No RNS dependency. +"""Local names for pins: where each pin came from on this machine, and the label its user gave it, for `rnfs pin +ls`. No RNS dependency. A CID names only content, so `add photo.jpg` keeps no file name. This records, for this node's own user, what each pin was added from (an absolute path, "(stdin)", the target given to `pin add`, an archive) so @@ -6,6 +7,11 @@ database's pin_names table (rnfs.statedb): each change is one row written at once, but not durably (synchronous=NORMAL): the pin it names was committed durably before it, so a power loss can lose at most the latest names, never a pin, and an add costs one durable commit rather than two. + +A label (`rnfs pin label`, POST /api/pin/label; rnfs 0.1.7) is the name the user chose for a pin, kept apart from +where it was added from, which stays as recorded: the pin_labels table, each change written at once and durably +(the user asked for it). Names are pruned with the pins they name; a label goes as its CID stops being pinned +(forget_label, from the node's pin listener), and at start for one a stop left behind (prune_labels). """ import threading @@ -14,6 +20,7 @@ from rnfs.log import LOG_WARNING, log MAX_NAME_LENGTH = 4096 # an origin longer than this (a very deep path) is cut, not refused +MAX_LABEL_LENGTH = 1024 # characters a label may have; a longer one is refused def import_names(value: dict, tx) -> int: @@ -35,13 +42,27 @@ def import_names(value: dict, tx) -> int: return len(imported) +def check_label(label) -> str | None: + """label as stored: None (or "") clears it, else a string of 1 to MAX_LABEL_LENGTH characters, surrounding + whitespace removed. Raises ValueError for anything else.""" + if label is None: + return None + if not isinstance(label, str): + raise ValueError("a label must be a string (or null to clear it)") + label = label.strip() + if len(label) > MAX_LABEL_LENGTH: + raise ValueError(f"a label may have at most {MAX_LABEL_LENGTH} characters") + return label or None + + class PinNames: - """Pin → where it was added from. Thread-safe; entries for CIDs no longer pinned are pruned.""" + """Pin → where it was added from, and its label. Thread-safe; entries for CIDs no longer pinned are pruned.""" def __init__(self, state_dir: str): - """Load the names saved in state_dir; a row that doesn't parse is logged and deleted.""" + """Load the names and labels saved in state_dir; a row that doesn't parse is logged and deleted.""" self._lock = threading.Lock() self._names: dict[Cid, str] = {} + self._labels: dict[Cid, str] = {} self._db = statedb.open_store(state_dir, self._load) def _load(self, db: statedb.StateDb): @@ -61,12 +82,75 @@ def _load(self, db: statedb.StateDb): log(f"pin names: ignoring invalid record {binary!r} {text!r}", LOG_WARNING) bad.append((binary,)) statedb.delete_rows(db, "pin_names", ("cid",), bad) + self._label_store = statedb.RowStore(db, "pin_labels", ("cid",), durable=True, lock=self._lock, + row=self._label_row, key_values=lambda cid: (cid.binary,)) + bad = [] + for binary, text in db.query("SELECT cid, label FROM pin_labels"): + try: + label = check_label(statedb.load(text)) + if label is None: + raise ValueError("empty") + self._labels[Cid.from_bytes(binary)] = label + except (ValueError, TypeError): + log(f"pin names: ignoring invalid label {binary!r} {text!r}", LOG_WARNING) + bad.append((binary,)) + statedb.delete_rows(db, "pin_labels", ("cid",), bad) def _row(self, cid: Cid) -> tuple | None: """cid's pin_names row, or None if it has no name (lock held).""" name = self._names.get(cid) return None if name is None else (cid.binary, statedb.dump(name)) + def _label_row(self, cid: Cid) -> tuple | None: + """cid's pin_labels row, or None if it has no label (lock held).""" + label = self._labels.get(cid) + return None if label is None else (cid.binary, statedb.dump(label)) + + def set_label(self, cid: Cid, label: str | None) -> bool: + """Label cid (None clears its label), saved durably; True if that changed it. Raises ValueError for a + label check_label refuses, OSError if it can't be saved (kept in memory, saved with the next change).""" + label = check_label(label) + with self._lock: + previous = self._labels.get(cid) + if previous == label: + return False + self._put_label(cid, label) + try: + self._label_store.changed(cid) + except BaseException: + self._put_label(cid, previous) # not saved: memory says what the database does + raise + return True + + def _put_label(self, cid: Cid, label: str | None): + """Set or (None) remove cid's label in memory (lock held).""" + if label is None: + self._labels.pop(cid, None) + else: + self._labels[cid] = label + + def forget_label(self, cid: Cid): + """Drop cid's label, once it is no longer pinned by anything (the node's pin listener). Raises OSError if + that can't be saved (dropped in memory all the same, and saved with the next change).""" + with self._lock: + if self._labels.pop(cid, None) is not None: + self._label_store.changed(cid) + + def prune_labels(self, pinned): + """Drop the labels of CIDs not in pinned, at start: a label is dropped as its pin goes (forget_label), so + only a stop between the two leaves one.""" + with self._lock: + gone = [cid for cid in self._labels if cid not in pinned] + for cid in gone: + del self._labels[cid] + if gone: + self._label_store.changed(*gone) + + def label(self, cid: Cid) -> str | None: + """cid's label, or None.""" + with self._lock: + return self._labels.get(cid) + def record(self, cid: Cid, name: str): """Record that cid was added from name; a later add of the same content replaces it. Raises OSError if it can't be saved (kept in memory, and saved with the next change).""" @@ -91,7 +175,9 @@ def get(self, cid: Cid) -> str | None: return self._names.get(cid) def prune(self, pinned: set[Cid]): - """Forget the names of CIDs that are no longer pinned, so the table stays as small as the pin list.""" + """Forget the names of CIDs that are no longer pinned, so the table stays as small as the pin list. Not the + labels: a label is the user's own choice, dropped exactly when its pin goes (forget_label), never by a + listing working from a snapshot that a new pin and label may have overtaken.""" with self._lock: gone = [cid for cid in self._names if cid not in pinned] if not gone: @@ -101,8 +187,9 @@ def prune(self, pinned: set[Cid]): self._store.changed(*gone) def save(self): - """Write the names whose write failed, if any (each change is written as it is made).""" + """Write the names and labels whose write failed, if any (each change is written as it is made).""" self._store.save() + self._label_store.save() def close(self): """Release the state database (the node calls this once it has saved everything at shutdown).""" diff --git a/rnfs/remote.py b/rnfs/remote.py index 5abbd9b..1996947 100644 --- a/rnfs/remote.py +++ b/rnfs/remote.py @@ -13,6 +13,17 @@ within MAX_WATCH_WAIT seconds, and nothing heard meanwhile is lost (the daemon answers from the records it holds). With a timeout, a long poll (names, Job.wait) waits at most LONG_POLL_MARGIN seconds less than it, so the request never outlasts its own socket; one that leaves no wait at all polls every second instead. + +Watched events come from GET /api/events, long-polled the same way on a thread of its own (rnfs-client-event-watch) +for every kind, each watch keeping those it wants: it starts where the first watch's baseline (a call without +after, made inside watch()) is, passes back the "next" and "epoch" each answer gives, and on "lost" (it fell more +than the daemon's kept events behind, or the daemon restarted) tells each watch so and goes on from the answer's +"next". It stops polling once no watch is left. A failed poll is reported and retried after RETRY_DELAY. + +Timeouts: the client's (timeout=) bounds every request; timeout(seconds) replaces it for the requests one thread +makes inside it (ApiConnection.using_timeout, thread-local), Job.wait's and events.since's long polls included, +each fitted to the timeout in force when it is sent (long_poll_wait). The name and event pollers, on their own +threads, keep the client's. """ import logging import math @@ -25,18 +36,24 @@ from rnfs.objects import Record MAX_WATCH_WAIT = 30 # seconds one GET /api/name/watch or /api/jobs/get waits (the API's cap) -LONG_POLL_MARGIN = 2 # seconds a long poll leaves of the client's timeout +LONG_POLL_MARGIN = 2 # seconds a long poll leaves of the request's timeout RETRY_DELAY = 5.0 # seconds before polling names again after a failure SHORT_POLL = 1.0 # seconds between polls when the timeout leaves no wait _logger = logging.getLogger("rnfs") +def long_poll_wait(timeout: float | None) -> int: + """The longest (whole) seconds a long poll may ask the daemon to wait within a request timeout: MAX_WATCH_WAIT + without one, else LONG_POLL_MARGIN less than it, rounded down (0: it leaves none).""" + return MAX_WATCH_WAIT if timeout is None else max(0, min(MAX_WATCH_WAIT, math.floor(timeout) - LONG_POLL_MARGIN)) + + class Client(interface.Rnfs): """The daemon using rnfs_dir (default ~/.rnfs/), on 127.0.0.1:port (default: the api_port in its rnfs.toml), with the token in token_file (default /api_token). timeout bounds each request (default: none, as - fetches can take hours). Nothing is sent until the first call. Raises SettingsError if rnfs.toml can't be read - for the port.""" + fetches can take hours; timeout() sets another for one thread's calls). Nothing is sent until the first call. + Raises SettingsError if rnfs.toml can't be read for the port.""" events_thread = "rnfs-client-events" jobs_thread = "rnfs-client-job-watch" @@ -46,13 +63,16 @@ def __init__(self, rnfs_dir: str | None = None, *, port: int | None = None, toke timeout: float | None = None): """A client of the daemon (see the class).""" self._api = ApiConnection(rnfs_dir, port=port, token_file=token_file, timeout=timeout) - # The longest a long poll may wait within the timeout - self._long_wait = MAX_WATCH_WAIT if timeout is None else \ - max(0, min(MAX_WATCH_WAIT, math.floor(timeout) - LONG_POLL_MARGIN)) - super().__init__(_Backend(self._api, self._long_wait)) - self._poll_cond = threading.Condition() + # The longest the name poller's long poll may wait within the client's timeout (its thread has no other) + self._long_wait = long_poll_wait(timeout) + super().__init__(_Backend(self._api)) + self._poll_cond = threading.Condition() # taken before _watch_lock, never under it self._poller: threading.Thread | None = None self._names_changed = False + self._event_poller: threading.Thread | None = None + self._baseline_lock = threading.Lock() # one event baseline at a time (_event_baseline); taken first + # (the sequence the event poller asks after, the daemon run's epoch); None while no event watch needs it + self._event_position: tuple[int, str] | None = None # The highest sequence each name's poll has answered: the next asks above it, so a delivery still waiting # on the event thread isn't fetched again and again (only the poller uses it) self._polled: dict[bytes, int] = {} @@ -69,7 +89,7 @@ def _report(self, message: str): _logger.error("rnfs: %s", printable(message)) def _close(self): - """Wake the name poller so it ends.""" + """Wake the name and event pollers so they end.""" with self._poll_cond: self._poll_cond.notify_all() @@ -121,14 +141,75 @@ def _poll_names(self): with self._poll_cond: self._poll_cond.wait(SHORT_POLL) + # Event watches + + def _event_baseline(self) -> tuple[int, str]: + """GET /api/events without after: the daemon's last sequence and run now (NoSuchEndpoint from a daemon + without events). The event poller starts there unless it has a position already, which is at or before + it: the watch skips what it began after. Even one of another run (the daemon restarted) is kept: the next + poll then answers "lost", which moves every older watch on to the new run (the new one began in it, and + skips the marker unless it missed events too). + + One baseline at a time, the request and the position set together (_baseline_lock, not _poll_cond, so + close() never waits for a request): two first watches can't leave the position past events the one with + the older baseline hasn't had.""" + with self._baseline_lock: + batch = self._api.events() + with self._poll_cond: + if self._event_position is None: + self._event_position = (batch["next"], batch["epoch"]) + if self._event_poller is None: + self._event_poller = threading.Thread(target=self._poll_events, name="rnfs-client-event-watch", + daemon=True) + self._event_poller.start() + self._poll_cond.notify_all() + return batch["next"], batch["epoch"] + + def _poll_events(self): + """Long-poll the daemon's events (every kind) for the event watches until closed; idle, and forgetting + where it was, while there are none.""" + while True: + with self._poll_cond: + while not self.closed and (self._event_position is None or not self._event_watch_count()): + if not self._event_watch_count(): + self._event_position = None # the next watch's baseline sets it + self._poll_cond.wait() + if self.closed: + return + after, epoch = self._event_position + started = time.monotonic() + try: + batch = self._api.events(after, None, self._long_wait, epoch) + except Exception as e: # noqa: BLE001 - a daemon restarting answers again later (then "lost") + if self.closed: + return + self._report(f"watching events failed: {e}") + with self._poll_cond: + self._poll_cond.wait(RETRY_DELAY) + continue + with self._poll_cond: + self._event_position = (batch["next"], batch["epoch"]) + if batch["lost"]: + # The events held are older than what was missed, or of another run: the watches read again + self._events_lost(batch["next"], batch["epoch"]) + else: + self._events_heard(batch["epoch"], batch["events"]) + if not batch["events"] and not batch["lost"] and self._long_wait == 0 \ + and time.monotonic() - started < SHORT_POLL: + with self._poll_cond: + self._poll_cond.wait(SHORT_POLL) + class _Backend: """rnfs.interface's backend for the daemon's API (ApiConnection's methods, in the forms the API takes).""" - def __init__(self, api: ApiConnection, long_wait: int): - """The backend of api, whose long polls wait at most long_wait seconds.""" + def __init__(self, api: ApiConnection): + """The backend of api.""" self.api = api - self.long_wait = long_wait + + def timeout(self, seconds: float | None): + """ApiConnection.using_timeout: this thread's requests bounded by seconds inside it.""" + return self.api.using_timeout(seconds) def status(self) -> dict: """GET /api/status.""" @@ -262,21 +343,23 @@ def name_resolve(self, name: str) -> dict: """GET /api/name/resolve.""" return self.api.name_resolve(name) - def submit_job(self, kind: str, target: str, output: str | None, from_node: str | None, **flags) -> dict: - """POST /api/jobs (only the flags set, as the CLI sends them).""" + def submit_job(self, kind: str, target: str | None, output: str | None, from_node: str | None, **flags) -> dict: + """POST /api/jobs (only the flags set, as the CLI sends them; an add's or import's path and options too).""" return self.api.submit_job(kind, target, output, from_node, **{k: v for k, v in flags.items() if v}) def job(self, job_id: str, wait: float) -> dict: - """GET /api/jobs/get, waiting up to wait seconds (whole ones, within the timeout) for the job to finish; - when the timeout leaves no wait, a second's pause after an unfinished answer instead.""" + """GET /api/jobs/get, waiting up to wait seconds (whole ones, within the timeout this thread's request has) + for the job to finish; when that timeout leaves no wait, a second's pause after an unfinished answer + instead.""" if wait <= 0: return self.api.job(job_id, 0) - if self.long_wait == 0: + long_wait = long_poll_wait(self.api.effective_timeout()) + if long_wait == 0: job = self.api.job(job_id, 0) if job.get("state") not in interface.FINISHED: time.sleep(min(wait, SHORT_POLL)) return job - return self.api.job(job_id, min(self.long_wait, math.ceil(wait))) + return self.api.job(job_id, min(long_wait, math.ceil(wait))) def jobs(self) -> list[dict]: """GET /api/jobs.""" @@ -301,3 +384,146 @@ def deny_add(self, entries: list[str]) -> list[str]: def deny_rm(self, entries: list[str]) -> list[str]: """POST /api/deny/rm.""" return self.api.deny_rm(entries) + + def peers(self) -> list[dict]: + """GET /api/peers.""" + return self.api.peers() + + def stats(self) -> dict: + """GET /api/stats.""" + return self.api.stats() + + def contacts(self) -> list[dict]: + """GET /api/contacts.""" + return self.api.contacts() + + def contact_get(self, nick: str) -> dict: + """GET /api/contacts/get.""" + return self.api.contact_get(nick) + + def contact_add(self, nick: str, ident: str, kind: str | None) -> dict: + """POST /api/contacts/add.""" + return self.api.contact_add(nick, ident, kind) + + def contact_rm(self, nick: str): + """POST /api/contacts/rm.""" + self.api.contact_rm(nick) + + def name_keep(self, name: str) -> dict: + """POST /api/name/keep.""" + return self.api.name_keep(name) + + def name_unkeep(self, name: str): + """POST /api/name/unkeep.""" + self.api.name_unkeep(name) + + def name_kept(self) -> list[dict]: + """GET /api/name/kept.""" + return self.api.name_kept() + + def name_follow(self, name: str) -> dict: + """POST /api/name/follow.""" + return self.api.name_follow(name) + + def name_unfollow(self, name: str): + """POST /api/name/unfollow.""" + self.api.name_unfollow(name) + + def name_following(self) -> list[dict]: + """GET /api/name/following.""" + return self.api.name_following() + + def lxmf_address(self) -> dict: + """GET /api/lxmf/address.""" + return self.api.lxmf_address() + + def lxmf_send(self, to: str, target: str, message: str | None, propagate: bool, nearest: bool) -> dict: + """POST /api/lxmf/send.""" + return self.api.lxmf_send(to, target, message, propagate, nearest) + + def lxmf_inbox(self) -> list[dict]: + """GET /api/lxmf/inbox.""" + return self.api.lxmf_inbox() + + def lxmf_inbox_get(self, entry_id, link, output: str | None, directory: str | None, + use_sent_cid: bool) -> list[dict]: + """POST /api/lxmf/inbox/get.""" + return self.api.lxmf_inbox_get(entry_id, link, output, directory, use_sent_cid) + + def lxmf_inbox_rm(self, entry_id, every: bool) -> int: + """POST /api/lxmf/inbox/rm.""" + return self.api.lxmf_inbox_rm(entry_id, every) + + def lxmf_outbox(self) -> list[dict]: + """GET /api/lxmf/outbox.""" + return self.api.lxmf_outbox() + + def lxmf_sync(self, nearest: bool) -> dict: + """POST /api/lxmf/sync.""" + return self.api.lxmf_sync(nearest) + + def pin_label(self, cid: str, label: str | None) -> dict: + """POST /api/pin/label.""" + return self.api.pin_label(cid, label) + + def cache_ls(self) -> list[dict]: + """GET /api/cache/ls.""" + return self.api.cache_ls() + + def cache_rm(self, cid: str) -> dict: + """POST /api/cache/rm.""" + return self.api.cache_rm(cid) + + def capabilities(self) -> dict: + """GET /api/capabilities ({} from a daemon before 0.1.7).""" + return self.api.capabilities() + + def shutdown(self, wait: float | None) -> bool: + """POST /api/shutdown, then (wait) polling the open status until it stops answering.""" + return self.api.shutdown(wait) + + def daemon_info(self) -> dict | None: + """The daemon's daemon.json (read_daemon_info).""" + return self.api.daemon_info() + + def config(self) -> dict: + """GET /api/config.""" + return self.api.config() + + def config_set(self, settings: dict) -> dict: + """POST /api/config/set.""" + return self.api.config_set(settings) + + def add_tar(self, data, name: str | None, pin: bool, only_hash: bool, chunker: str | None) -> dict: + """POST /api/add/stream?format=tar.""" + return self.api.add_tar(data, name=name, only_hash=only_hash, pin=pin, chunker=chunker) + + def get_tar(self, target: str): + """GET /api/get/stream?format=tar: the body, read as it arrives; sent once iterated, as read()'s.""" + def chunks(): + """The body's chunks.""" + with self.api.get_tar(target) as stream: + yield from stream + return chunks() + + def export_stream(self, target: str): + """GET /api/export/stream: the body, read as it arrives; sent once iterated, as read()'s.""" + def chunks(): + """The body's chunks.""" + with self.api.export_stream(target) as stream: + yield from stream + return chunks() + + def import_stream(self, data) -> dict: + """POST /api/import/stream.""" + return self.api.import_stream(data) + + def events_since(self, after: int | None, kinds: list[str] | None, wait: int, epoch: str | None) -> dict: + """GET /api/events, waiting at most what this thread's timeout leaves (long_poll_wait); when it leaves no + wait, a pause of up to a second after an empty answer instead, as job()'s, so a caller looping on it + doesn't spin.""" + long_wait = long_poll_wait(self.api.effective_timeout()) + batch = self.api.events(after, kinds, min(wait, long_wait), epoch) + if after is not None and wait > 0 and long_wait == 0 and not batch["events"] and not batch["lost"]: + time.sleep(min(wait, SHORT_POLL)) + return batch diff --git a/rnfs/statedb.py b/rnfs/statedb.py index e2a6952..2dc20f6 100644 --- a/rnfs/statedb.py +++ b/rnfs/statedb.py @@ -143,6 +143,13 @@ "CREATE TABLE shared_folder_files (name TEXT NOT NULL, state TEXT NOT NULL, cid BLOB NOT NULL," " height INTEGER NOT NULL, digest BLOB, PRIMARY KEY (name, state)) WITHOUT ROWID", )), + ("pin_details", ( + # When each pinned CID was first pinned (Unix time; rnfs 0.1.7), by any source; a pin made by an earlier + # version has no row. A table of its own: the released tables stay exactly as 0.1.5 made them + "CREATE TABLE pin_times (cid BLOB PRIMARY KEY, added REAL NOT NULL) WITHOUT ROWID", + # A label the user gave a pin (pin label), kept apart from where it was added from (pin_names); dump()ed + "CREATE TABLE pin_labels (cid BLOB PRIMARY KEY, label TEXT NOT NULL) WITHOUT ROWID", + )), ) diff --git a/roadmap.md b/roadmap.md index b88b7e7..d0ae095 100644 --- a/roadmap.md +++ b/roadmap.md @@ -31,6 +31,31 @@ CHANGELOG.md). # 0.1.7 +## Local API and library for companion apps (done) + +What companion apps need from the daemon, most impactful first. + +* Discovery file: the daemon writes `/daemon.json` at start with its PID, API ports, version, start time + and the Reticulum config it uses, so a client finds the daemon without parsing `rnfs.toml` (an older client + can't find the port today if `rnfs.toml` has a newer setting) +* `POST /api/shutdown`: a background daemon can't be stopped cleanly on Windows without it (the app ships a second + console program to send Ctrl+Break through a hidden console) +* Capability list and machine-readable error codes: apps string-match error messages today, and a 404 can mean + either "no such endpoint" or "content not found" +* Push endpoint (`GET /api/events`) for jobs, pins, the LXMF inbox, names and peers, replacing the apps' polling +* Richer `pin ls`: size, time added, folder entry counts as fields (not "(directory: 28 entries)" strings), and a + renamable label kept separate from the added-from path (nothing can set a file's local name today) +* `cache ls` and `cache rm`: nothing lists downloads that aren't pinned, short of reading `blocks/` on disk +* Config API that carries its own schema (so apps don't keep a copy of the setting metadata), with some settings + applied without a restart (every setting needs one today) +* Add and import as jobs with progress, and directory streaming for upload and export, so a sandboxed (Flatpak) + daemon works fully +* Peers: `first_heard`, the last link RTT the node already measured, and an online state worked out by the daemon +* `rnfs.Client`: typed methods for peers, stats, contacts, follow, keep and LXMF, and timeouts per call instead + of per connection + +# 0.1.8 + ## Faster block writes on SD cards (plan item B, then A2) Sync blocks to disk at the durability point (before a pin or cache save) instead of once per block, with one `syncfs()` on Linux; keep the per-block fsync on other platforms and on stacked filesystems (ecryptfs, overlayfs), diff --git a/specification.md b/specification.md index cb891f7..da16c62 100644 --- a/specification.md +++ b/specification.md @@ -4,7 +4,7 @@ rnfs is content-addressed file sharing over [Reticulum](https://reticulum.networ features of IPFS: content IDs, chunked files, directories, a provider DHT, verified multi-source transfer, pinning, mutable names and a gateway — built on what RNS already provides. -**Version.** This document describes rnfs 0.1.6. The wire format (request paths and messages, CIDs and DAG +**Version.** This document describes rnfs 0.1.7. The wire format (request paths and messages, CIDs and DAG bytes, chunk boundaries, provider and name records, announce app_data and the LXMF field) is byte-identical since rnfs 0.1.3, so nodes of 0.1.3 and later work together. `tests/test_wire.py` checks the current code against vectors that 0.1.3's and 0.1.4's own code produced (`tests/fixtures/wire`). @@ -787,7 +787,7 @@ and repeat. - Each cached fetch is recorded as the `/` it fetched, which keeps the path's nodes as well as the target's DAG. - When storage exceeds `max_storage`, cached fetches are evicted least recently used first; `gc` evicts them - all. + all, and `cache rm ` those leading to one CID (a pin of the same content keeps its blocks). - `add --no-pin` stores what it adds as a cached fetch of its CID instead of pinning it: evicted least recently used like any other, removed by `gc`, never provided under `pinned`, and under `roots` provided while it is cached (it counts as a target the user fetched). @@ -1221,13 +1221,14 @@ are local conveniences, not part of the protocol. | Command | Does | |---|---| -| `add \|- [--wrap] [--only-hash] [--name N] [--no-pin] [--to …] [--to-self] [--no-sealed-name] [--chunker fixed\|fastcdc]` | Import a file, a directory or stdin, pin it with source `user`, print its CID. `--wrap` puts it in a directory; `--only-hash` just prints the CID; `--no-pin` keeps it as cache (section 8); `--to` seals it for recipients (section 5, Sealed files); `--chunker fastcdc` uses content-defined chunks (section 5) | +| `add \|- [--wrap] [--only-hash] [--name N] [--no-pin] [--to …] [--to-self] [--no-sealed-name] [--chunker fixed\|fastcdc] [--background]` | Import a file, a directory or stdin, pin it with source `user`, print its CID. `--wrap` puts it in a directory; `--only-hash` just prints the CID; `--no-pin` keeps it as cache (section 8); `--to` seals it for recipients (section 5, Sealed files); `--chunker fastcdc` uses content-defined chunks (section 5); `--background` adds a path as an `add` job (below) and prints its ID | | `get [/path] [-o out] [--from ] [--background] [--raw]` | Fetch and write out a file or directory. Without `-o` the output is, in the current directory, the target's last path name if it is printable, not hidden (no leading `.`), doesn't start with `-` (which a later command would read as an option) and is a valid file name here, else its root CID. A sealed file this node is a recipient of is decrypted, and without `-o` (for a bare CID) written under its sealed name by the same rule; `--raw` writes the stored bytes | | `cat [/path] [--offset N] [--length N] [--raw] [--block]` | Fetch and write a file, or part of one, to stdout; sealed files are decrypted as by `get` (only the segments covering the range); `--raw` writes the stored (encrypted) bytes, which is what scripts that show content to others must use; `--block` writes the CID's own block | | `ls [/path]` | List a directory: name, kind, size, CID | | `stat [/path]` | Size, block count, and how much is held locally | | `dir set [/path] [--exec]`, `dir rm /` | Make a new directory with an entry added, replaced or removed, and pin it with source `user` (fetching the complete directory, file contents included) | -| `pin add [/path] [--background]`, `pin rm `, `pin ls [] [--json]` | Manage pins; `ls` shows their sources, whether each is a sealed file (`sealed`: true, false, or null when its stored blocks can't tell), and a local name: where each was added from on this machine (recorded locally, never sent; for a sealed file too), the shared-folder entry, or what a stored directory holds | +| `pin add [/path] [--background]`, `pin rm `, `pin ls [] [--json]`, `pin label [