diff --git a/.agents/skills/bump-version/SKILL.md b/.agents/skills/bump-version/SKILL.md index 4473605..063ecf0 100644 --- a/.agents/skills/bump-version/SKILL.md +++ b/.agents/skills/bump-version/SKILL.md @@ -14,6 +14,12 @@ part of it -- see the end of this file. uv run scripts/bump_version.py ``` +There is exactly one place a version lives: `[workspace.package]` in +`Cargo.toml`. `frontend/package.json` deliberately has no `version` field -- the +dashboard reports the binary's, read from the configuration the listener injects -- +so nothing there needs bumping, and adding a version to it would create a second +thing to forget. + It sets the version under `[workspace.package]` in `Cargo.toml`, renames `## Development` in CHANGES.md to `## -- `, and opens a fresh empty Development section above it. It prints what it changed and what to do diff --git a/.agents/skills/check-docs/SKILL.md b/.agents/skills/check-docs/SKILL.md index fbdc767..203b61e 100644 --- a/.agents/skills/check-docs/SKILL.md +++ b/.agents/skills/check-docs/SKILL.md @@ -88,6 +88,43 @@ grep -rn 'env::var\|env = "DOPPEL' crates/ statuses, must match the table in the docs. A code added without a doc row is invisible to whoever has to handle it. +**The parameter reference.** `docs/usage/parameters.md` is generated from +`doppel-config.schema.json` by `scripts/parameters_doc.py`. Never edit it by hand +-- regenerate it and read the diff: + +```bash +uv run scripts/config_schema.py && uv run scripts/parameters_doc.py +``` + +What the generator cannot judge is whether a field's *description* still says +something true, or whether its example is still the example worth showing. Those +come from doc comments and `examples(...)` in `crates/doppel-core/src/config/`, so +read them there when the behaviour changed. + +**The links the dashboard builds.** The (i) beside a field links into that page, +and the link is assembled at runtime from the field's path and the running version +-- so mkdocs cannot see it and `--strict` cannot check it. A wrong fragment is +silent: the browser opens the page at the top and nobody notices. + +```bash +uv run scripts/check_docs_links.py +``` + +It holds three things, and each has a failure it exists for: + +- every `info="..."` path in `frontend/src` has a section under the anchor + `frontend/src/services/docs.ts` derives from it -- a renamed field is the way + this breaks; +- every documentation URL in the frontend is built through `docsUrl()`, so it + carries the version. A hand-written link to the site root shows whoever follows + it the rules of whatever has been released since; +- every relative link and fragment inside the generated page resolves. + +Both checks are in `make gate` and in CI, so what is left for a human is the +half a script cannot see: follow two or three of the (i) links in a running +dashboard and read the sections they land on. An anchor that exists but explains +a different field passes every check here. + **`main.example.yaml`.** It is the schema made concrete and is asserted against by tests, so it cannot be stale -- but it can be *misleading*, which tests do not catch. Read it as a newcomer copying it: does the ordering of its mocks @@ -120,3 +157,34 @@ Say what you compared, not just that you checked. "Every config field has a reference entry; the rule-map tests pass; two `usage/` pages named crates and were reworded; `mkdocs build --strict` is clean" is a useful report. "Docs look fine" is not. + +## The dashboard and the frontend + +`docs/usage/dashboard.md` is the operator-facing page: the two configuration +fields, who may use it, what each screen does, the theme, and what a 503 at the +root means. `docs/development/architecture.md` carries the developer-facing half. + +Check these together, because they go stale as a set: + +- `admin.dashboard` and `admin.title` in `docs/usage/configuration.md` and in + `main.example.yaml`. +- `GET /api/v1/access` and the routes in `docs/usage/admin-api.md`, plus + `DASHBOARD_NOT_BUILT` in its error table. The endpoint table is the one thing + there that is checked by a test -- `openapi.rs` refuses a document describing a + route that is not served -- so a path that changed shows up in `cargo test` + before it shows up here. What does not is prose quoting a path, and every + healthcheck: the compose file, the Docker Hub overview, `observability.md`. +- The build order in `README.md`: assets first, then the binary. A reader who + follows a stale version of that gets a 503 and no idea why. +- `make help`. Every target carries its own description, so the list cannot go + stale on its own -- but a documented command that no longer matches a target is + worth catching here. +- The four pages a reader reaches for before the dashboard's own: `docs/index.md` + (the feature list and the reading order), `usage/getting-started.md` (a first run + ends by opening it), `usage/troubleshooting.md` (the 503, disabled controls, the + refresh timer) and `DOCKERHUB.md` (the image serves it too). A feature documented + only on its own page is a feature nobody finds. + +The icons live in `assets/` at the repository root and are staged into +`docs/assets/` by `scripts/mkdocs_assets.py` on every build. `docs/assets/` is +git-ignored: a link that points there is fine, a file committed there is not. diff --git a/.agents/skills/check-licenses/SKILL.md b/.agents/skills/check-licenses/SKILL.md index f03b6f5..bf2755e 100644 --- a/.agents/skills/check-licenses/SKILL.md +++ b/.agents/skills/check-licenses/SKILL.md @@ -1,6 +1,6 @@ --- name: check-licenses -description: Use after editing any Cargo.toml, adding or removing a dependency, or before cutting a release. Verifies every direct dependency's licence against the project policy and updates LICENSE. +description: Use after editing any Cargo.toml or frontend/package.json, adding or removing a dependency, or before cutting a release. Verifies every direct dependency's licence against the project policy and updates LICENSE and THIRD-PARTY.md. --- # Check licences @@ -21,6 +21,20 @@ so the dynamic-linking exception does not apply. ## How to check +Both halves are generated, so start by running the generator and reading what it +says: + +```bash +uv run scripts/third_party.py # rewrites THIRD-PARTY.md +uv run scripts/third_party.py --check # what CI runs +``` + +It exits non-zero and prints `SOME LICENCES ARE NOT COVERED BY THE POLICY` with +the offending packages named. That is the check; the rest of this skill is for +deciding what to do about a new dependency. + +### The cargo half + Read the licence from the registry rather than from memory: ```bash @@ -33,6 +47,30 @@ Transitive crates come in under their parents' justification; if one of them carries a forbidden licence that is a real problem, but it is found by auditing the tree, not by this skill. +### The npm half + +The direct set is `dependencies` and `devDependencies` in +`frontend/package.json`. The two are not equivalent, and the difference decides +whether a licence matters at all: + +- `dependencies` is compiled into `frontend/dist` and embedded in the binary, so + it is **redistributed**. Its licences bind, and it belongs in `LICENSE` and + `THIRD-PARTY.md`. +- `devDependencies` -- vite, jest, eslint, tailwind's compiler, the tooling + underneath them -- runs on a developer's machine and ships nowhere. It is + deliberately absent from both files. + +A build tool in `dependencies` is therefore a real mistake and not a matter of +taste: it drags the policy over licences the project never distributes. +Tailwind sat there once, and the tooling closure it pulled in carried Blue Oak +and CC-BY -- which would have meant widening the permitted set for code nobody +ships. + +```bash +npm view license # one package +npm ls --omit=dev --all --prefix frontend # exactly what ships +``` + ## Ask the question the policy is actually for For each dependency that is new since the last check, answer in one line: what @@ -51,8 +89,17 @@ sweep late in phase 1, each declared and referenced by no source file. grep -rn "" crates --include='*.rs' | head ``` -## Update LICENSE +## Update LICENSE and THIRD-PARTY.md + +`LICENSE` carries the Apache 2.0 text plus a table of the *direct* set, from both +ecosystems. `THIRD-PARTY.md` carries the whole closure and is generated -- never +edit it by hand. + +```bash +uv run scripts/third_party.py +``` -`LICENSE` carries the Apache 2.0 text plus the third-party notices. When the -direct set changes, update the notices to match. Report what you changed and +Both modes need `frontend/node_modules` present and refuse to run without it, +rather than writing a cargo-only file that looks complete or reporting a stale one +that is not. Report what you changed and what you found -- including "nothing changed", which is a useful result. diff --git a/.agents/skills/pre-release-check/SKILL.md b/.agents/skills/pre-release-check/SKILL.md index dd3a004..3fe5282 100644 --- a/.agents/skills/pre-release-check/SKILL.md +++ b/.agents/skills/pre-release-check/SKILL.md @@ -27,6 +27,28 @@ changes things. `regenerate-config-schema` is the one exception, and only because the thing it writes is generated: if it produces a diff, that diff is a commit of its own before the release, not part of the release commit. +## Then check the dashboard + +`run-tests-and-linters` covers the frontend gate, but a release has two questions +of its own, because the dashboard is embedded at compile time and a binary without +it still builds: + +```bash +make e2e # the browser suite, which the frontend gate leaves out +``` + +Then confirm the release will not ship a page-less binary. The workflow greps each +archive's binary for the configuration element before publishing -- read that step +and check it is still there, since it is the only thing standing between a missing +artifact and a release whose root answers 503: + +```bash +grep -n "carries the dashboard" .github/workflows/release.yml +``` + +The dashboard's own version is not bumped and must not be: `frontend/package.json` +has no `version` field, and the page reports the binary's. + ## Then check the version ```bash diff --git a/.agents/skills/run-tests-and-linters/SKILL.md b/.agents/skills/run-tests-and-linters/SKILL.md index 00db585..8545edc 100644 --- a/.agents/skills/run-tests-and-linters/SKILL.md +++ b/.agents/skills/run-tests-and-linters/SKILL.md @@ -5,7 +5,18 @@ description: Use before marking any task done, before opening a pull request, or # Run tests and linters -The gate is three commands, in this order, all of which must pass: +The whole gate, in one command: + +```bash +make gate +``` + +That is `fmt-check`, `lint`, `test`, `test-frontend`, `size`, `docs`, `schema` +and `licences`. It deliberately leaves out the browser suite -- `make e2e` +downloads Chromium on first run -- so run that too whenever the change touches +`frontend/` or the dashboard routes. + +The Rust half by hand, in this order, all of which must pass: ```bash cargo fmt --check @@ -13,6 +24,35 @@ cargo clippy --all-targets -- -D warnings cargo test ``` +## The frontend half + +The dashboard has its own gate, and one ordering that matters: the assets are +embedded into the binary at compile time, so they are built before anything that +reads them. + +```bash +npm --prefix frontend ci +npm --prefix frontend run lint # eslint +npm --prefix frontend run typecheck # tsc, the application +(cd frontend && npx tsc --noEmit -p tsconfig.e2e.json) # tsc, the specs +npm --prefix frontend run build # writes frontend/dist +npm --prefix frontend test # jest, including the bundle-size budgets +cargo build -p doppel-cli # re-embeds what the build just wrote +npm --prefix frontend run e2e # Playwright, against that binary +``` + +Two traps this order exists to avoid: + +- `npm run build` alone changes nothing about a running binary. Rebuilding the + frontend without rebuilding the binary is how half an hour goes into chasing a + fixed bug that is still in the embedded bundle. +- The bundle-size suite measures `frontend/dist`. Running jest before the build + measures the previous one, or fails because there is nothing to measure. + +`DOPPEL_REQUIRE_DASHBOARD_ASSETS=1` on `cargo test` turns a skipped +asset-delivery suite into a failure, exactly as `DOPPEL_REQUIRE_DATABASE` does +for the store. `make test` sets it. + ## Capture the output, do not recall it Run the gate so its output lands in a file, and quote the file: diff --git a/.dockerignore b/.dockerignore index 394d0ee..4b04f04 100644 --- a/.dockerignore +++ b/.dockerignore @@ -1,9 +1,45 @@ -# The image copies one already-built binary in; nothing else in the repository -# belongs in the build context. Keeping it small matters more than usual here: -# `target/` alone is tens of gigabytes on a working checkout, and every byte of -# the context is sent to the daemon before the first instruction runs. +# The image prefers an already-built binary and only compiles one when none was +# staged, so the context is what those two paths need and nothing else. Keeping it +# small matters more than usual here: `target/` alone is tens of gigabytes on a +# working checkout, and every byte of the context is sent to the daemon before the +# first instruction runs. * -# The staged binary, which the `docker` release job puts here and the local -# build instructions in the Dockerfile copy here too. +# This file itself, because the Dockerfile copies it. That is not decoration: +# `COPY .dockerignore dist//doppel* /staged/` is how "copy the binary if +# it is there" is expressed, and it needs one source that always matches or the +# COPY fails when no binary was staged. +!.dockerignore + +# The staged binary, which the `docker` release job puts here and the local build +# instructions in the Dockerfile copy here too. When it is present nothing below +# is compiled. !dist/ + +# What the fallback build needs: the workspace, and the dashboard it embeds. +!Cargo.toml +!Cargo.lock +!rust-toolchain.toml +!crates/ + +# The dashboard, either already built or buildable. `frontend/dist` is generated, so +# it is listed explicitly; the sources beside it are what the image builds from when +# it is not there, which is why this list is exactly what `vite build` reads and not +# the whole directory -- `node_modules` is bigger than everything else here together. +# +# `assets/favicon.ico` is in the list because the build's `prebuild` step copies it +# into `public/`: one icon, one source of truth, and the copy is git-ignored. +!frontend/dist/ +!frontend/package.json +!frontend/package-lock.json +!frontend/index.html +!frontend/vite.config.ts +!frontend/tsconfig.json +!frontend/src/ +!frontend/scripts/ +!assets/favicon.ico + +# Nothing inside the workspace that the build does not read. `target/` is the one +# that matters; the rest keeps the context honest. +crates/**/target/ +**/*.rs.bk diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..7279f89 --- /dev/null +++ b/.gitattributes @@ -0,0 +1,34 @@ +# Line endings are normalised on commit, so a checkout on Windows does not turn +# every file into a diff, and the ASCII-only rule in AGENTS.md is not quietly +# broken by a stray CR. +* text=auto eol=lf + +# The icons are binary. git-lfs keeps them out of the packfile, and the `-text` +# marker stops git from ever trying to reconcile two versions line by line. +# +# Any workflow that reads these files must check out with `lfs: true`. Without +# it the working tree holds a pointer file, and the documentation build then +# ships a few lines of text where an image was expected -- which fails as a +# broken image rather than as a missing file, so it is worth stating here. +*.png filter=lfs diff=lfs merge=lfs -text +*.ico filter=lfs diff=lfs merge=lfs -text + +# One exemption, and it is not a matter of taste. DOCKERHUB.md points Docker +# Hub's overview image at +# raw.githubusercontent.com/lorem-dev/doppel/main/assets/icon-128.png, and that +# endpoint serves the LFS *pointer* rather than the bytes -- verified on +# 2026-08-11, 130 bytes of ASCII beginning "version +# https://git-lfs.github.com/spec/v1". Tracked through LFS, the overview would +# render a broken image, and nothing in this repository would fail to say so. +# This file is small enough that a plain blob costs nothing. +assets/icon-128.png -filter -diff -merge text=auto + +# The documentation's screenshots, for the same reason as the icons: a new +# screenshot is a new whole file rather than a diff, and every UI change brings +# one. `docs/screenshots/` is read only by the mkdocs build, whose workflow +# checks out with `lfs: true` -- without that it would ship the pointer file as +# a broken image. +*.webp filter=lfs diff=lfs merge=lfs -text + +# An SVG is XML, and a readable diff is worth more than treating it as opaque. +*.svg text diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index df46591..e29c4d5 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -64,10 +64,33 @@ jobs: # supposed to exercise the store and silently did not is worse than one # that failed. DOPPEL_REQUIRE_DATABASE: "1" + # Same idea as DOPPEL_REQUIRE_DATABASE, for the dashboard: the suites that + # exercise real asset delivery skip themselves when the binary was built + # without `frontend/dist`, which is right on a laptop and wrong here. + DOPPEL_REQUIRE_DASHBOARD_ASSETS: "1" steps: + # `lfs: true` because the icons are tracked through git-lfs and the frontend + # build copies the favicon out of `assets/`. Without it the checkout holds a + # pointer file and the copy lands a few lines of text where an image goes. - name: Checkout repository uses: actions/checkout@v7 + with: + lfs: true + + - name: Install Node + uses: actions/setup-node@v7 + with: + node-version: 24 + cache: npm + cache-dependency-path: frontend/package-lock.json + + # Built here rather than pulled from the frontend job as an artifact: this + # keeps the two jobs independent and parallel, and twenty seconds of npm is + # cheaper than serialising the graph. The admin crate's build script embeds + # whatever is in `frontend/dist`, so this has to happen before `cargo test`. + - name: Build the dashboard for the admin crate to embed + run: npm --prefix frontend ci && npm --prefix frontend run build # No toolchain is named here. `rust-toolchain.toml` pins the channel and # the components, and rustup honours it; naming a version in the workflow @@ -110,14 +133,22 @@ jobs: - name: Configuration schema is up to date run: uv run scripts/config_schema.py --check + # The parameter reference is generated from that schema, so it goes stale the + # same way and for the same reason: nobody regenerates a file by hand twice. + - name: Parameter reference is up to date + run: uv run scripts/parameters_doc.py --check + docs: name: Documentation builds runs-on: ubuntu-latest timeout-minutes: 10 steps: + # `lfs: true` for the icons the documentation shows; see the rust job. - name: Checkout repository uses: actions/checkout@v7 + with: + lfs: true # Pinned to an exact release, unlike every other action here. astral-sh # publishes `v9.0.0` but stopped publishing the sliding major tag after @@ -131,3 +162,111 @@ jobs: # moved. - name: Build the site (strict) run: uv run --with-requirements docs/requirements.txt mkdocs build --strict + + # `--strict` catches a broken link between pages; this catches the links the + # dashboard builds at runtime, which mkdocs cannot see. A wrong fragment is + # silent in a browser -- the page opens at the top and nobody notices. + - name: Dashboard documentation links resolve + run: uv run scripts/check_docs_links.py + + frontend: + name: Frontend lint, types, tests, size + runs-on: ubuntu-latest + timeout-minutes: 15 + + steps: + - name: Checkout repository + uses: actions/checkout@v7 + with: + lfs: true + + - name: Install Node + uses: actions/setup-node@v7 + with: + node-version: 24 + cache: npm + cache-dependency-path: frontend/package-lock.json + + - name: Install dependencies + run: npm ci + working-directory: frontend + + - name: Lint + run: npm run lint + working-directory: frontend + + # Two projects: the application and the Playwright suite. They have + # different globals -- jest's and Playwright's `expect` collide -- so a + # single project would have to pick one and leave the other unchecked. + - name: Types + run: npm run typecheck && npx tsc --noEmit -p tsconfig.e2e.json + working-directory: frontend + + # Before the tests, because the bundle-size suite measures what this + # produces. A test that silently skipped when `dist` was missing would make + # the budget a preference again. + - name: Build + run: npm run build + working-directory: frontend + + - name: Tests, including the bundle-size budgets + run: npm test + working-directory: frontend + + e2e: + name: Dashboard end to end + runs-on: ubuntu-latest + timeout-minutes: 20 + + steps: + - name: Checkout repository + uses: actions/checkout@v7 + with: + lfs: true + + - name: Install the toolchain pinned in rust-toolchain.toml + run: rustup show active-toolchain + + - name: Cache cargo registry and target + uses: Swatinem/rust-cache@v2 + with: + cache-on-failure: true + + - name: Install Node + uses: actions/setup-node@v7 + with: + node-version: 24 + cache: npm + cache-dependency-path: frontend/package-lock.json + + - name: Install dependencies + run: npm ci + working-directory: frontend + + # The dashboard is embedded at compile time, so the order matters: build the + # assets, then the binary. Building the binary first embeds nothing and every + # scenario then fails against a 503. + - name: Build the dashboard + run: npm run build + working-directory: frontend + + - name: Build the binary the specs drive + run: cargo build -p doppel-cli + + - name: Install Chromium + run: npx playwright install --with-deps chromium + working-directory: frontend + + - name: Run the scenarios + run: npm run e2e + working-directory: frontend + + # Only on a failure, and only the report: a trace for a green run is an + # upload nobody opens. + - name: Upload the report + if: failure() + uses: actions/upload-artifact@v7 + with: + name: playwright-report + path: frontend/playwright-report + retention-days: 7 diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index d4a0028..7b4c983 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -63,6 +63,11 @@ jobs: # mike reads and rewrites the gh-pages branch, so it needs history # rather than the default shallow clone of one ref. fetch-depth: 0 + # The icons are tracked through git-lfs, and the build hook copies them + # into the docs tree. Without this the published site would carry a + # pointer file where the logo goes -- which renders as a broken image + # rather than failing the build. + lfs: true # Pinned to an exact release, unlike every other action here. astral-sh # publishes `v9.0.0` but stopped publishing the sliding major tag after diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index afe9811..1c0d2c3 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -30,10 +30,47 @@ concurrency: cancel-in-progress: false jobs: + frontend: + name: Build the dashboard + runs-on: ubuntu-latest + timeout-minutes: 15 + + steps: + # `lfs: true` for the favicon the build copies out of `assets/`. + - name: Checkout repository + uses: actions/checkout@v7 + with: + ref: ${{ github.event.inputs.tag || github.ref }} + lfs: true + + - name: Install Node + uses: actions/setup-node@v7 + with: + node-version: 24 + cache: npm + cache-dependency-path: frontend/package-lock.json + + - name: Build + run: npm ci && npm run build + working-directory: frontend + + # Built once and handed to every build job. The opposite choice from CI, + # and for a reason: there are five of them here, so five npm builds would + # cost more than the artifact round trip -- and every binary published + # under one tag has to carry the same bytes. + - name: Upload the built dashboard + uses: actions/upload-artifact@v7 + with: + name: dashboard + path: frontend/dist + retention-days: 1 + if-no-files-found: error + build: name: Build ${{ matrix.target }} runs-on: ${{ matrix.runner }} timeout-minutes: 45 + needs: frontend strategy: # One target failing should not throw away the others: they take @@ -62,6 +99,16 @@ jobs: # `rust-toolchain.toml` pins the channel; only the target has to be added # on top of it, and `rustup target add` uses that same pinned toolchain. + # The admin crate's build script embeds whatever is in `frontend/dist`, so + # this has to land before `cargo build`. Without it the binary compiles and + # answers 503 at `/`, which is a release nobody would notice was broken + # until they opened it. + - name: Restore the built dashboard + uses: actions/download-artifact@v8 + with: + name: dashboard + path: frontend/dist + - name: Install the toolchain pinned in rust-toolchain.toml run: | rustup show active-toolchain @@ -98,6 +145,7 @@ jobs: musl: name: Build ${{ matrix.target }} (for the image) + needs: frontend runs-on: ${{ matrix.runner }} timeout-minutes: 45 @@ -135,6 +183,16 @@ jobs: sudo apt-get install -y --no-install-recommends \ -o DPkg::Lock::Timeout=300 musl-tools + # The admin crate's build script embeds whatever is in `frontend/dist`, so + # this has to land before `cargo build`. Without it the binary compiles and + # answers 503 at `/`, which is a release nobody would notice was broken + # until they opened it. + - name: Restore the built dashboard + uses: actions/download-artifact@v8 + with: + name: dashboard + path: frontend/dist + - name: Install the toolchain pinned in rust-toolchain.toml run: | rustup show active-toolchain @@ -237,6 +295,12 @@ jobs: with: context: . platforms: linux/amd64,linux/arm64 + # The Dockerfile compiles a binary when none is staged, and this job + # always stages one -- so it points the builder stage at a base with no + # toolchain in it. Without this the build pulls a Rust image, for both + # platforms, to run a `cp`. + build-args: | + BUILDER=alpine:3.24 push: true tags: ${{ steps.meta.outputs.tags }} labels: ${{ steps.meta.outputs.labels }} @@ -326,6 +390,37 @@ jobs: # flatten them into one directory. merge-multiple: true + # The one thing that cannot be checked before the archives exist: that the + # binaries about to be published actually carry the dashboard. The build + # script embeds `frontend/dist` silently -- a missing artifact yields a + # binary that compiles, passes nothing that would notice, and answers 503 at + # `/` for whoever downloads it. Grepping for the configuration element is + # enough: it comes from `index.html`, so its presence means the page was + # embedded. + - name: Every published binary carries the dashboard + run: | + set -euo pipefail + found=0 + for archive in dist/doppel-*.tar.gz; do + tmp=$(mktemp -d) + tar -xzf "$archive" -C "$tmp" + binary=$(find "$tmp" -type f -name doppel -perm -u+x | head -1) + if [ -z "$binary" ]; then + echo "no doppel binary inside $archive" >&2 + exit 1 + fi + if ! grep -qa 'id="doppel-config"' "$binary"; then + echo "$archive was built without the dashboard" >&2 + exit 1 + fi + echo "$archive carries the dashboard" + found=$((found + 1)) + done + if [ "$found" -eq 0 ]; then + echo "no archives were checked" >&2 + exit 1 + fi + # Attached so a configuration can be validated against the exact release # it will run under, rather than against whatever `main` says today. The # checked-in file is taken as-is: CI has already refused a stale one diff --git a/.gitignore b/.gitignore index 97987e3..56b142a 100644 --- a/.gitignore +++ b/.gitignore @@ -3,15 +3,35 @@ **/*.rs.bk *.pdb -# Runtime artifacts +# Runtime artifacts. Anchored with a leading slash, all of them: `config/` +# without one matches a directory of that name at any depth, which quietly +# covered `crates/doppel-core/src/config/` -- git keeps tracking files it +# already knows, so nothing failed, but every editor drew the crate's own +# configuration module as excluded. /templates *.sock -main.yaml +/main.yaml +# The configuration the compose file mounts, and anything else kept beside it. +/config/ .env # mkdocs build output /site +# Staged into the docs tree from assets/ by scripts/mkdocs_assets.py on every +# build. Committing the copy would be two files claiming to be the same icon. +/docs/assets + +# Frontend build and tooling output. `dist` is what the admin crate's build +# script embeds, so it is generated on the way to a binary and never committed; +# `public/favicon.ico` is copied in from assets/ by the prebuild script. +/frontend/node_modules +/frontend/dist +/frontend/coverage +/frontend/playwright-report +/frontend/test-results +/frontend/public/favicon.ico + # Python tooling for mkdocs .venv __pycache__/ diff --git a/AGENTS.md b/AGENTS.md index 7e9571e..d9c8326 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -23,12 +23,23 @@ cannot drift. `usage/` for how to use it, `development/` for how to work on it. `.superpowers/specs/` (git-ignored, not shipped) holds the full design. +`frontend/` is the browser dashboard the admin listener serves; see +[the frontend](#the-frontend) below before touching it. + --- ## Running the Gates Prerequisites: a stable Rust toolchain via rustup (pinned in -`rust-toolchain.toml`, with `rustfmt` and `clippy`) +`rust-toolchain.toml`, with `rustfmt` and `clippy`), Node for the dashboard, and +`uv` for the Python tooling. + +```bash +make gate # everything below, in the right order +make help # every target, with a description +``` + +The Rust half by hand: ```bash cargo fmt --check # formatting @@ -36,16 +47,33 @@ cargo clippy --all-targets -- -D warnings # lints, warnings are errors cargo test # tests ``` -All of these must pass before a pull request is ready. +All of these must pass before a pull request is ready. `make gate` leaves out +`make e2e`, which downloads a browser on first run -- run it too when the change +touches `frontend/` or the dashboard routes. ### Verification workflow -Run the full gate (`cargo fmt --check`, `cargo clippy`, `cargo test`) as a single -dedicated step at the END of a change -- not after every task. While iterating, -write only the minimal tests needed to guarantee the code works, and run just the -focused test for what you changed (at most a quick `cargo test -p `). -The comprehensive gate above is the single final check before a change is -considered done or a pull request opened. +Run `make gate` as a single dedicated step at the END of a change -- not after +every task, and not while iterating. It builds the dashboard, runs both Rust +suites, jest, the bundle budgets, mkdocs and three generators; on a change to one +file, nearly all of that re-checks code the change did not touch. + +While iterating, run only what addresses the part being changed: + +| Changed | Run | +| --- | --- | +| a crate | `cargo test -p `, `cargo clippy -p `, `cargo fmt` | +| `frontend/src` | `npm --prefix frontend test -- `, `npx eslint `, `npx tsc --noEmit` | +| the dashboard's behaviour in a browser | `npx playwright test ` | +| `docs/`, a doc comment, a docs link | `make docs`, `make parameters`, `make links`, separately | +| a config type or its schema | `make schema` | +| a dependency | `make licences` | + +Write only the minimal tests needed to guarantee the code works, and check a new +test by mutation with the same focused run -- not with the gate. + +`make gate` is the single final check before a change is considered done or a +pull request opened. Read its exit code as its own step, then commit. Capture the gate's output when you report it. Over this project's history, several reports stated test counts or quoted compiler diagnostics that did not @@ -127,9 +155,64 @@ for it: | `check-changes` | After a batch of commits -- verify CHANGES.md (Development section) reflects every change. | | `check-docs` | Before a release or after updating commands/options -- verify docs/ and README.md are current. | | `run-tests-and-linters` | Before marking any task done -- run `cargo fmt --check`, `cargo clippy --all-targets -- -D warnings` and `cargo test`, and capture the output. | -| `check-licenses` | After editing any `Cargo.toml` -- verify every direct dependency is licence-compliant and update LICENSE. | +| `check-licenses` | After editing any `Cargo.toml` or `frontend/package.json` -- verify every direct dependency is licence-compliant and regenerate LICENSE and THIRD-PARTY.md. | | `pre-release-check` | Before cutting a release -- runs the five `check-*` and `run-*` skills above (not `bump-version`) plus version-bump and commit-format checks. | +## The frontend + +`frontend/` holds the dashboard: TypeScript, React, react-router, zustand, +Tailwind, built by vite. `docs/development/architecture.md` has the reasoning; +what follows is what you need before editing it. + +``` +frontend/src/ + components/ presentational; may not import a service (eslint enforces it) + pages/ one screen each: fetches, holds its state, renders components + services/ the only code that talks to the API + store/ state that outlives one screen (zustand) + types/ the API's shapes, and the one error type +frontend/e2e/ + src/ the harness the specs drive: the server, its configurations + tests/ the Playwright specs +``` + +Four things that will otherwise cost you an hour each: + +- **The assets are embedded at compile time.** `npm run build` alone changes + nothing about a running binary; `cargo build` after it is what re-embeds them. + A binary built with no `frontend/dist` compiles and answers 503 at `/`. +- **jest cannot render anything that routes.** react-router ships ESM only and + uses `import.meta`, which does not survive the transform to CommonJS. Anything + that navigates belongs in the Playwright suite, against a real browser and a + real binary. +- **Rights are read through `useMay()`, and that is not decoration.** It + subscribes to the report so the component re-renders when rights arrive. A hook + that returned the same callback without subscribing would leave every gated + control disabled for the life of the page -- which is what happened, and what + the Playwright suite caught. `permits()` holds the precedence rules; there is + deliberately no second way to ask. +- **TypeScript is pinned to 5.9 on purpose.** typescript-eslint declares + `<6.1.0` and ts-jest declares `<7`, so the latest compiler is one no linter + accepts yet. Raising it breaks the lint, not the build. + +Commands: `make frontend`, `make test-frontend`, `make size`, `make e2e`. + +`npm ci` prints deprecation warnings for `inflight`, `glob@7`, `glob@10` and +`whatwg-encoding`. All four are inside jest's own tree -- `@jest/transform` and +`@jest/reporters` pull the globs, jsdom pulls the encoder -- so nothing there +ships: the redistributed set is the ten packages in `THIRD-PARTY.md`, none of them +these. They are deliberately not silenced with `overrides`, which would pin +versions jest was not tested against to quieten a message about a package the +project does not distribute. + +`frontend/package.json` carries no `version`: the page shows the *binary's* +version, taken from the configuration the listener injects, and a second version +in a second file is a second thing to forget at release time. + +The favicon is generated from `assets/favicon.svg`; the command that rebuilds +`assets/favicon.ico` is in a comment at the top of that file. `assets/` is staged +into `docs/assets/` by the mkdocs build hook, and `docs/assets/` is git-ignored. + ## CodeGraph diff --git a/CHANGES.md b/CHANGES.md index 40b0878..9dc8f56 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -8,6 +8,47 @@ release promotes it to a version heading; the `bump-version` skill does that. ## Development +### Added + +- `rewrite_urls` (default on): the upstream's own address is replaced by Doppel's + inside text bodies it relays, so a page or a JSON document cannot send a client + past every fault and mock. Exact host only. +- Metrics that exist before anything happens: `doppel_build_info`, + `doppel_dashboard_info`, `doppel_proxy_last_error_timestamp_seconds` and + `doppel_proxy_mocks`, the last following the configuration in force. +- `doppel_admin_request_duration_seconds`, the admin API's own latency, labelled + by route template so path and query parameters cannot multiply series. +- `server.external_url` and `DOPPEL_EXTERNAL_URL`: the address clients reach + Doppel at, used when rewriting an upstream's redirects. +- A browser dashboard on the admin listener's root, compiled into the binary: + proxy CRUD, mock templates, status and reload. `admin.dashboard`, `admin.title`. +- `GET /api/v1/access` reports the calling token's own rights, so a client can + disable an action instead of offering it and being refused. +- A Makefile: `make help` lists every target, `make gate` runs the whole check + suite. + +### Changed + +- The Prometheus exposition is `GET /metrics` again, the path every scraper + looks for; under `/api/v1/` the dashboard answered scrapes with HTML. +- `doppel_proxy_request_duration_seconds` carries `replace`, `loss` and + `upstream_error`, and its buckets reach 60s for injected latency. +- The admin listener accepts a trailing slash on every path. The proxy listener + still relays one verbatim. +- The admin listener compresses its responses, `br` or `gzip`, so the dashboard + is 74 KB on the wire instead of 140 KB, and `index.html` is minified. +- The Swagger UI is `/swagger-ui/` and the OpenAPI document `/openapi.json`, + both outside `/api/`: a page and a description, not resources of the API. +- A write through the admin API is in force when it answers: it promotes the + stored configuration before replying, so no reload is needed. +- A rewritten `Location` names Doppel's host instead of being relative, and now + covers a redirect to the upstream's own host outside the proxied path. + +- The PostgreSQL store keeps the configuration as JSON, so a new field no longer + needs a migration. Migration 0005 backfills it. +- **Breaking:** every admin endpoint now lives under `/api/`. `/status` and + `/metrics` moved to `/api/v1/`, the OpenAPI document and Swagger UI to `/api/`. + ## 0.4.1 -- 2026-08-11 ### Added diff --git a/Cargo.lock b/Cargo.lock index 1f7e1f7..1e4f439 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -26,6 +26,21 @@ dependencies = [ "memchr", ] +[[package]] +name = "alloc-no-stdlib" +version = "2.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cc7bb162ec39d46ab1ca8c77bf72e890535becd1751bb45f64c597edb4c8c6b3" + +[[package]] +name = "alloc-stdlib" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0e76a019e91224d279006ff972f1e984179a6e9feb050adba6ce8274aef23195" +dependencies = [ + "alloc-no-stdlib", +] + [[package]] name = "allocator-api2" version = "0.2.21" @@ -115,6 +130,18 @@ dependencies = [ "rustversion", ] +[[package]] +name = "async-compression" +version = "0.4.43" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3976abdc8fe7d1133d43d304afd42abdf5bc3e1319d263d223bde07b5efc4be8" +dependencies = [ + "compression-codecs", + "compression-core", + "pin-project-lite", + "tokio", +] + [[package]] name = "async-trait" version = "0.1.91" @@ -279,6 +306,27 @@ dependencies = [ "objc2", ] +[[package]] +name = "brotli" +version = "8.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5cc91aac060a7a1e25823bdccbfb6af1875b88f17c6daac97894eed8207166b3" +dependencies = [ + "alloc-no-stdlib", + "alloc-stdlib", + "brotli-decompressor", +] + +[[package]] +name = "brotli-decompressor" +version = "5.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3a32acac15fe1967bc3986b2a6347dffc965602354ea6f450ad07e8bfd253583" +dependencies = [ + "alloc-no-stdlib", + "alloc-stdlib", +] + [[package]] name = "bumpalo" version = "3.20.3" @@ -403,6 +451,24 @@ dependencies = [ "memchr", ] +[[package]] +name = "compression-codecs" +version = "0.4.38" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ce2548391e9c1929c21bf6aa2680af86fe4c1b33e6cea9ac1cfeec0bd11218cf" +dependencies = [ + "brotli", + "compression-core", + "flate2", + "memchr", +] + +[[package]] +name = "compression-core" +version = "0.4.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cc14f565cf027a105f7a44ccf9e5b424348421a1d8952a8fc9d499d313107789" + [[package]] name = "const-oid" version = "0.10.2" @@ -596,6 +662,7 @@ dependencies = [ "async-trait", "axum", "doppel-core", + "httpdate", "metrics", "metrics-exporter-prometheus", "serde", @@ -603,6 +670,7 @@ dependencies = [ "tempfile", "tokio", "tower", + "tower-http", "tracing", "utoipa", "utoipa-swagger-ui", @@ -657,6 +725,7 @@ name = "doppel-proxy" version = "0.4.1" dependencies = [ "axum", + "bytes", "doppel-core", "doppel-render", "futures-util", @@ -792,6 +861,7 @@ version = "1.1.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "843fba2746e448b37e26a819579957415c8cef339bf08564fe8b7ddbd959573c" dependencies = [ + "crc32fast", "miniz_oxide", "zlib-rs", ] @@ -3143,12 +3213,16 @@ version = "0.6.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4cfcf7e2740e6fc6d4d688b4ef00650406bb94adf4731e43c096c3a19fe40840" dependencies = [ + "async-compression", "bitflags", "bytes", + "futures-core", "futures-util", "http", "http-body", "pin-project-lite", + "tokio", + "tokio-util", "tower", "tower-layer", "tower-service", diff --git a/Cargo.toml b/Cargo.toml index d06a5d5..a2d2ba9 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -15,7 +15,12 @@ arc-swap = "1.9" async-trait = "0.1" axum = "0.8" clap = { version = "4", features = ["derive", "env"] } +bytes = "1" futures-util = "0.3" +# Already in the tree under hyper, and the only calendar in it. The dashboard's +# footer needs the year the binary was built, and turning a unix timestamp into +# one is leap years and century rules -- not arithmetic worth writing here. +httpdate = "1" metrics = "0.24" metrics-exporter-prometheus = { version = "0.18", default-features = false } minijinja = "2" @@ -63,6 +68,13 @@ tokio = { version = "1", features = [ "time", ] } tower = "0.5" +# Already in the graph through utoipa-swagger-ui. Two features are taken from it: +# response compression for the admin listener, which is the reason a dashboard +# does not have to ship 140 KB uncompressed. +tower-http = { version = "0.6", default-features = false, features = [ + "compression-gzip", + "compression-br", +] } tracing = "0.1" tracing-subscriber = { version = "0.3", features = ["env-filter", "json"] } diff --git a/DOCKERHUB.md b/DOCKERHUB.md index 45127aa..7814717 100644 --- a/DOCKERHUB.md +++ b/DOCKERHUB.md @@ -14,7 +14,7 @@ a version that exists. The version checked in is the last one released; the tags table stays illustrative, like the documentation's. --> -![Doppel](https://raw.githubusercontent.com/lorem-dev/doppel/main/docs/assets/icon-128.png) +![Doppel](https://raw.githubusercontent.com/lorem-dev/doppel/main/assets/icon-128.png) # Doppel @@ -60,14 +60,14 @@ start. ```bash docker run --rm \ -p 8080:8080 -p 8081:8081 \ - -v "$PWD/main.yaml:/etc/doppel/main.yaml:ro" \ + -v "$PWD/config:/etc/doppel" \ -v doppel-templates:/var/lib/doppel/templates \ loremdev/doppel:0.1.0-alpine ``` `8080` carries proxied traffic, `8081` the admin API. -A minimal `main.yaml`: +A minimal `config/main.yaml`: ```yaml server: @@ -120,12 +120,12 @@ services: - "8080:8080" - "8081:8081" volumes: - - ./main.yaml:/etc/doppel/main.yaml:ro + - ./config:/etc/doppel - doppel-templates:/var/lib/doppel/templates environment: DOPPEL_ADMIN_TOKENS: '{"ci":{"token":"...","group":"admin"}}' healthcheck: - test: ["CMD", "curl", "-fsS", "-o", "/dev/null", "http://127.0.0.1:8081/status"] + test: ["CMD", "curl", "-fsS", "-o", "/dev/null", "http://127.0.0.1:8081/api/v1/status"] interval: 5s timeout: 3s start_period: 5s @@ -135,11 +135,22 @@ volumes: doppel-templates: ``` -`/status` needs no token by default and answers only once the runtime is +`/api/v1/status` needs no token by default and answers only once the runtime is compiled and both listeners are bound. `-f` matters: without it `curl` exits 0 on a 500 and the check passes for answering at all rather than for answering correctly. +## The dashboard + +The admin port serves a browser dashboard from its root -- the proxy set, a form +over every field of a proxy, mock templates, status and reload -- compiled into +the image. Publish `8081` as the quick start above does and open +`http://127.0.0.1:8081/`. + +It is a client of the admin API and bound by the same token rules, so it offers +only what the caller's token may actually do. `admin.dashboard: false` turns it +off; the JSON API is unaffected either way. + ## Tokens `DOPPEL_ADMIN_TOKENS` keeps secrets out of the configuration file that gets @@ -165,7 +176,11 @@ curl -X POST -H "X-Proxy-Authorization: Bearer $TOKEN" \ docker exec doppel config reload --socket /tmp/doppel.sock ``` -Editing a bind-mounted `main.yaml` on the host changes the file the container +The directory is mounted rather than the file on purpose: a save renames a new +file over the old one, and a file that is itself a mount point cannot be replaced -- +the admin API would refuse every write with "Resource busy". + +Editing `config/main.yaml` on the host changes the file the container reads, so a reload picks it up without restarting anything. ## What the image contains diff --git a/Dockerfile b/Dockerfile index a49f7d0..82bf341 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,33 +1,120 @@ # Doppel, on Alpine. # -# The binary is built outside this file and copied in -- see the `docker` job -# in .github/workflows/release.yml. Building Rust inside a multi-architecture -# `docker buildx` means QEMU emulation for the non-native architecture, which -# turns a two-minute compile into most of an hour. Copying an already-built -# binary makes the image build the same speed on both. +# Prefers an already-built binary and compiles one only when none was staged. # -# Build locally against a binary you already have: +# The preference is not a style choice. Building Rust inside a multi-architecture +# `docker buildx` means QEMU emulation for the non-native architecture, which turns +# a two-minute compile into most of an hour -- so the release workflow builds each +# binary on a native runner and stages it, and this file copies it in. # -# cargo build --release --target x86_64-unknown-linux-musl -p doppel-cli -# mkdir -p dist && cp target/x86_64-unknown-linux-musl/release/doppel dist/ -# docker build --build-arg BIN=dist/doppel -t doppel:dev . +# The fallback exists because a developer should not have to know that. `docker +# build -t doppel:dev .` works with an empty `dist/`: the binary is compiled in the +# builder stage below, for the platform being built, with no cross toolchain on the +# host. `make image` stages one first and this file then skips the compile. # -# The binary must be musl-linked. A glibc build does not run here, and the -# failure is a bare "not found" from the shell rather than anything that says -# why. +# docker build -t doppel:dev . # compiles inside, or uses dist/ if it is there +# make image # builds the dashboard first, then this +# +# The dashboard is built inside too, when `frontend/dist` is not in the context. It +# is embedded at compile time, so a binary built without it answers 503 at its own +# root -- and a fresh checkout should not have to know that before it can build an +# image. `make image` still builds it on the host first, which is faster and is what +# a working checkout already has. +# +# A binary you built yourself is used by putting it at `dist/doppel`, or at +# `dist//doppel` for a multi-platform build. There is no build argument +# to point elsewhere: one place to look is one thing to get wrong. +# +# A staged binary must be musl-linked. A glibc build does not run here, and the +# failure is a bare "not found" from the shell rather than anything that says why. -FROM alpine:3.24 +# The toolchain from rust-toolchain.toml, so a compile here matches a compile on a +# host. Override with `--build-arg BUILDER=alpine:3.24` when the binary is +# certainly staged: the copy branch needs no toolchain, and the release workflow +# passes exactly that to avoid pulling a Rust image it will not use. +ARG BUILDER=rust:1.94.0-alpine -# `TARGETPLATFORM` is set by buildx per platform (`linux/amd64`, -# `linux/arm64`), so one Dockerfile picks the right binary out of a directory -# laid out by platform without emulating anything. -# -# `BIN` defaults from it and can be overridden for a build by hand, where -# there is one binary and no buildx: +FROM ${BUILDER} AS builder + +# Set by buildx per platform, and the layout the release workflow stages into: +# `dist/linux/amd64/doppel`, `dist/linux/arm64/doppel`. This is what lets one +# Dockerfile pick the right binary out of a multi-platform build without emulating +# anything. # -# docker build --build-arg BIN=dist/doppel -t doppel:dev . +# `TARGETOS`/`TARGETARCH` are looked at as well as `TARGETPLATFORM`, because they +# do not always agree: the classic builder reports `linux/arm64/v8` where buildx +# reports `linux/arm64`, and a directory named after the first would be a binary +# nobody finds. ARG TARGETPLATFORM -ARG BIN=dist/${TARGETPLATFORM}/doppel +ARG TARGETOS +ARG TARGETARCH + +WORKDIR /src + +# Whatever is in `dist/`, if anything. +# +# `.dockerignore` is always in the context, so this always has one source that +# matches and cannot fail when nothing was staged -- which is the whole point of +# the fallback. Docker has no conditional `COPY`, and this is how "copy it if it is +# there" is expressed. `dist*` rather than a path into it: a glob tolerates a +# missing file but not a missing parent directory, and `dist/linux/arm64/` is +# exactly the parent that does not exist on a laptop. +COPY .dockerignore dist* /staged/ + +# The workspace, for the branch that compiles. Bounded by `.dockerignore`, which +# admits the crates, the manifests and the built dashboard and nothing else. +COPY . . + +# One of two things, decided by what was staged. +# +# The compile branch needs `frontend/dist`, because the admin crate embeds whatever is +# there and an empty directory yields a binary that starts, serves the API, and +# answers 503 at its own root. So it builds the dashboard when nobody has: the +# alternative was refusing, which meant the image could not be built from a fresh +# checkout without knowing to run `make frontend` first. +RUN set -eu; \ + staged=""; \ + for candidate in \ + "/staged/${TARGETPLATFORM}/doppel" \ + "/staged/${TARGETOS}/${TARGETARCH}/doppel" \ + "/staged/doppel"; \ + do \ + if [ -f "$candidate" ]; then staged="$candidate"; break; fi; \ + done; \ + if [ -n "$staged" ]; then \ + echo "doppel: using the binary staged at $staged"; \ + install -m 0755 "$staged" /doppel; \ + exit 0; \ + fi; \ + echo "doppel: nothing staged for ${TARGETPLATFORM}, compiling"; \ + if ! command -v cargo >/dev/null; then \ + echo "doppel: this builder image has no Rust toolchain, and there is no" >&2; \ + echo " binary in dist/ to copy. Either stage one or drop the" >&2; \ + echo " BUILDER build argument, whose default carries a toolchain." >&2; \ + exit 1; \ + fi; \ + if [ ! -f frontend/dist/index.html ]; then \ + if [ ! -f frontend/package.json ]; then \ + echo "doppel: there is no built dashboard in frontend/dist and no frontend" >&2; \ + echo " sources to build one from. Check .dockerignore, or stage a" >&2; \ + echo " binary in dist/." >&2; \ + exit 1; \ + fi; \ + echo "doppel: no dashboard in frontend/dist, building it"; \ + apk add --no-cache nodejs npm >/dev/null; \ + npm --prefix frontend ci; \ + npm --prefix frontend run build; \ + fi; \ + if [ ! -f frontend/dist/index.html ]; then \ + echo "doppel: the dashboard build produced no frontend/dist/index.html, and a" >&2; \ + echo " binary embedding nothing answers 503 at its own root." >&2; \ + exit 1; \ + fi; \ + apk add --no-cache musl-dev >/dev/null; \ + cargo build --release --locked -p doppel-cli; \ + install -m 0755 target/release/doppel /doppel + +FROM alpine:3.24 # `ca-certificates` is not optional: a proxy whose upstream is https cannot # verify it without a trust store, and the failure surfaces as UPSTREAM_ERROR @@ -58,7 +145,8 @@ RUN apk add --no-cache ca-certificates curl tini # which is the right answer in a container. RUN addgroup -S doppel && adduser -S -G doppel -h /var/lib/doppel doppel -COPY --chmod=0755 ${BIN} /usr/local/bin/doppel +# From the builder stage, which either copied the staged binary or compiled one. +COPY --from=builder /doppel /usr/local/bin/doppel # Two mount points, both owned by the runtime user. # diff --git a/LICENSE b/LICENSE index 0620daa..bd0273e 100644 --- a/LICENSE +++ b/LICENSE @@ -210,11 +210,18 @@ This project uses the following third-party libraries. Each library retains its original copyright and is distributed under its respective license. -The 26 crates below are the ones this workspace names directly. A -released binary links its dependencies statically, so it also contains their -dependencies: the complete list, generated from the resolve graph, is in +The packages below are the ones this project names directly: the crates in +`Cargo.toml`, and the npm packages in `frontend/package.json` whose code is +compiled into the dashboard. A released binary links its crates statically and +embeds that dashboard, so it also contains everything those depend on: the +complete list, generated from both resolve graphs, is in [THIRD-PARTY.md](THIRD-PARTY.md). +Build-time npm packages -- vite, jest, eslint, tailwind's compiler and the +tooling underneath them -- are deliberately absent. They run on a developer's +machine and ship nowhere, so listing them would claim a redistribution that does +not happen. + | Package | Ecosystem | Version | License | |---|---|---|---| | anyhow | crates.io | 1.0.104 | MIT OR Apache-2.0 | @@ -243,4 +250,9 @@ dependencies: the complete list, generated from the resolve graph, is in | tracing-subscriber | crates.io | 0.3.23 | MIT | | utoipa | crates.io | 5.5.0 | MIT OR Apache-2.0 | | utoipa-swagger-ui | crates.io | 9.0.2 | MIT OR Apache-2.0 | - +| prismjs | npm | 1.30.0 | MIT | +| react | npm | 19.2.8 | MIT | +| react-dom | npm | 19.2.8 | MIT | +| react-router | npm | 8.3.0 | MIT | +| react-simple-code-editor | npm | 0.14.1 | MIT | +| zustand | npm | 5.0.14 | MIT | diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..d004d03 --- /dev/null +++ b/Makefile @@ -0,0 +1,203 @@ +# Doppel, in one command at a time. +# +# Nothing here is required to work on the project -- every target is a short +# sequence of `cargo`, `npm` and `uv` that could be typed out instead. What it +# buys is the ordering that is easy to get wrong: the dashboard is embedded into +# the binary at compile time, so the assets have to be built *before* cargo runs, +# and the image needs a musl binary staged where the Dockerfile expects it. +# +# `make` on its own prints the list. + +CARGO ?= cargo +NPM ?= npm +UV ?= uv +FRONTEND ?= frontend + +# The image's tag. Override for a real publish: `make image IMAGE=loremdev/doppel:1.2.3-alpine`. +IMAGE ?= doppel:dev + +# Nothing here sets a target for an ordinary build: `cargo build` produces a binary +# for the machine it runs on, which is what `build` and `release` mean on any +# platform. +# +# The image needs a Linux one, and that is the Dockerfile's problem rather than +# this file's: it uses a binary staged in `dist/` when there is one and compiles +# inside its own builder stage when there is not. Which is what makes `make image` +# work on a Mac, where `ring` cannot be linked against musl without a cross +# toolchain nobody has installed. + +# A database for the store suites. The port is deliberately not 5432; see +# docker-compose.yml. +TEST_DATABASE_URL ?= postgres://doppel:doppel@127.0.0.1:55432/doppel + +.DEFAULT_GOAL := help + +# Every target here is a name, not a file, so make must not go looking for one +# and must not skip a target because a directory of that name happens to exist -- +# `frontend`, `docs` and `dist` all do. +.PHONY: build release frontend run gate fmt fmt-check lint test test-db \ + test-frontend size e2e schema schema-write licences licences-write \ + docs docs-serve image image-rebuild image-run db-up db-down migrate \ + clean clean-frontend help + +# --------------------------------------------------------------------------- +# Building +# --------------------------------------------------------------------------- + +build: frontend ## Build the debug binary with the dashboard embedded + $(CARGO) build + +release: frontend ## Build the release binary with the dashboard embedded + $(CARGO) build --release + +frontend: ## Build the dashboard into frontend/dist + $(NPM) --prefix $(FRONTEND) ci + $(NPM) --prefix $(FRONTEND) run build + +run: build ## Run the debug binary against ./config/main.yaml + ./target/debug/doppel serve --config config/main.yaml + +# --------------------------------------------------------------------------- +# Checks +# --------------------------------------------------------------------------- + +gate: fmt-check lint test test-frontend size docs schema parameters links licences ## Everything CI checks, except the browser suite + @echo "gate: clean" + +fmt: ## Format the Rust sources + $(CARGO) fmt --all + +fmt-check: ## Fail if anything is unformatted + $(CARGO) fmt --all --check + +lint: ## Clippy with warnings as errors, eslint, and both TypeScript projects + $(CARGO) clippy --workspace --all-targets -- -D warnings + $(NPM) --prefix $(FRONTEND) run lint + $(NPM) --prefix $(FRONTEND) run typecheck + cd $(FRONTEND) && npx tsc --noEmit -p tsconfig.e2e.json + +# The dashboard is built first and required, so the asset-delivery suites cannot +# skip themselves the way they may on a machine with no Node. Same reasoning as +# DOPPEL_REQUIRE_DATABASE, which is set only when a database is actually up. +test: frontend ## Run the Rust suites, requiring the embedded dashboard + DOPPEL_REQUIRE_DASHBOARD_ASSETS=1 $(CARGO) test --workspace + +test-db: frontend ## Run the Rust suites with the PostgreSQL store required + DOPPEL_REQUIRE_DASHBOARD_ASSETS=1 \ + DOPPEL_REQUIRE_DATABASE=1 \ + DOPPEL_TEST_DATABASE_URL=$(TEST_DATABASE_URL) \ + $(CARGO) test --workspace + +test-frontend: frontend ## Run the jest suites and the bundle-size budgets + $(NPM) --prefix $(FRONTEND) test + +size: frontend ## Report what the dashboard weighs, gzipped + @cd $(FRONTEND) && node -e "\ + const { gzipSync } = require('node:zlib'); \ + const { readdirSync, readFileSync } = require('node:fs'); \ + const dir = 'dist/assets'; let total = 0; \ + for (const file of readdirSync(dir).sort()) { \ + const kb = gzipSync(readFileSync(dir + '/' + file)).length / 1024; \ + total += kb; \ + console.log(kb.toFixed(1).padStart(7) + ' KB ' + file); \ + } \ + console.log(total.toFixed(1).padStart(7) + ' KB total'); \ + " + +# Not part of `gate`: it downloads a browser on first run and takes half a minute, +# which is the wrong trade for a check meant to be run constantly. +e2e: build ## Drive the built binary with Playwright + cd $(FRONTEND) && npx playwright install chromium + $(NPM) --prefix $(FRONTEND) run e2e + +schema: ## Fail if doppel-config.schema.json is stale + $(UV) run scripts/config_schema.py --check + +schema-write: ## Regenerate doppel-config.schema.json + $(UV) run scripts/config_schema.py + +licences: ## Fail if THIRD-PARTY.md is stale, or a licence is outside the policy + $(UV) run scripts/third_party.py --check + +licences-write: frontend ## Regenerate THIRD-PARTY.md from both dependency graphs + $(UV) run scripts/third_party.py + +# --------------------------------------------------------------------------- +# Documentation +# --------------------------------------------------------------------------- + +parameters: ## Fail if the generated parameter reference is stale + $(UV) run scripts/parameters_doc.py --check + +links: ## Fail if the dashboard links to documentation that is not there + $(UV) run scripts/check_docs_links.py + +docs: ## Build the documentation site, strictly + $(UV) run --with-requirements docs/requirements.txt mkdocs build --strict + +docs-serve: ## Serve the documentation with live reload + $(UV) run --with-requirements docs/requirements.txt mkdocs serve + +# --------------------------------------------------------------------------- +# The image +# --------------------------------------------------------------------------- + +# The Dockerfile takes an already-built binary rather than compiling inside a +# multi-architecture build, where the non-native half runs under emulation and +# turns a two-minute compile into most of an hour. So the binary is built first, +# for this machine's architecture, and staged where the default BIN points. +image: frontend ## Build the container image for this architecture + docker build -t $(IMAGE) . + +# `rm -rf dist` as well as `--no-cache`: a binary left there from an earlier build +# is what the image would use, and "from scratch" has to mean the compile happens. +image-rebuild: clean-frontend ## Build the image from scratch: no npm or docker cache + $(NPM) --prefix $(FRONTEND) ci + $(NPM) --prefix $(FRONTEND) run build + rm -rf dist + docker build --no-cache -t $(IMAGE) . + +image-run: ## Run the built image against ./config/main.yaml + docker run --rm -p 8080:8080 -p 8081:8081 \ + -v "$(PWD)/config:/etc/doppel:ro" \ + $(IMAGE) + +# --------------------------------------------------------------------------- +# The development database +# --------------------------------------------------------------------------- + +db-up: ## Start PostgreSQL and wait for it + docker compose up -d --wait postgres + +db-down: ## Stop PostgreSQL and remove its volume + docker compose down -v + +migrate: build ## Apply the migrations to the development database + DOPPEL_DATABASE_URL=$(TEST_DATABASE_URL) ./target/debug/doppel config migrate + +# --------------------------------------------------------------------------- +# Cleaning +# --------------------------------------------------------------------------- + +clean: clean-frontend ## Remove every build artifact + $(CARGO) clean + rm -rf dist site + +clean-frontend: ## Remove the dashboard's build output + rm -rf $(FRONTEND)/dist $(FRONTEND)/playwright-report $(FRONTEND)/test-results + +# --------------------------------------------------------------------------- +# Help +# --------------------------------------------------------------------------- + +# Every target documented with `## ...` on its own line, in file order, so the +# list cannot drift from the targets: a target without a comment simply does not +# appear, and one that is renamed takes its description with it. +help: ## Print this list + @printf 'Doppel. Targets:\n\n' + @grep -hE '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) \ + | awk 'BEGIN { FS = ":.*?## " } { printf " \033[1m%-16s\033[0m %s\n", $$1, $$2 }' + @printf '\nThis machine: %s. An ordinary build targets it. The image needs a\n' \ + '$(shell uname -s -m)' + @printf 'Linux binary, which the Dockerfile compiles inside unless dist/ holds one.\n' + @printf '\nVariables: IMAGE=%s\n' '$(IMAGE)' diff --git a/README.md b/README.md index f79549b..de638da 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,5 @@

- Doppel Logo + Doppel Logo

$$Doppel{\color{lightblue}ganger}$$

@@ -37,7 +37,7 @@ Or run the image: ```bash docker run --rm -p 8080:8080 -p 8081:8081 \ - -v "$PWD/main.yaml:/etc/doppel/main.yaml:ro" \ + -v "$PWD/config:/etc/doppel" \ loremdev/doppel:1.2.3-alpine ``` @@ -61,6 +61,12 @@ version. See concurrency, template upload, reload, a status endpoint, Prometheus metrics and a generated Swagger UI, behind token access control. See [the documentation](https://lorem-dev.github.io/doppel/usage/admin-api/). +- Serves a browser dashboard from that same port's root, compiled into the + binary: the proxy list, a form over the whole proxy document, mock templates, + status and reload. It is a client of the admin API and bound by the same token + rules, so it offers only what the caller may actually do -- and works without a + token wherever the API does. `admin.dashboard: false` turns it off. See + [the documentation](https://lorem-dev.github.io/doppel/usage/dashboard/). - Reports errors to Sentry, optionally, behind the `sentry` cargo feature. - Stores its configuration in a YAML file or in PostgreSQL, selected by `--store`, with `config push` and `config pull` to move a configuration @@ -86,6 +92,16 @@ never `pip`: uv run --with-requirements docs/requirements.txt mkdocs serve ``` +The dashboard's source is `frontend/`; building it needs Node, and the binary +embeds whatever `frontend/dist` holds at compile time: + +```bash +npm --prefix frontend ci && npm --prefix frontend run build +cargo build --release -p doppel-cli +``` + +Skipping that leaves a working binary whose root answers 503 and says so. + The workspace layout -- the crates, what each owns, and the dependency direction -- is in [the architecture page](https://lorem-dev.github.io/doppel/development/architecture/). @@ -99,11 +115,16 @@ Prerequisites: a stable Rust toolchain via rustup, edition 2024 (see `Cargo.toml` for the pinned `rust-version`). ```bash -cargo build --release -cp main.example.yaml main.yaml # edit to taste; main.yaml is git-ignored -./target/release/doppel serve --config main.yaml +make release # the dashboard, then the binary +mkdir -p config && cp main.example.yaml config/main.yaml # config/ is git-ignored +./target/release/doppel serve --config config/main.yaml ``` +`make help` lists every target: `make gate` runs the whole check suite, +`make image-rebuild` builds the container image from scratch, `make docs-serve` +serves the documentation. `cargo build --release` works too, and skips the +dashboard -- which then answers 503 at the admin root and says so. + `cargo run -- ` works the same way during development. ## Commands @@ -123,6 +144,10 @@ Configuration path, store selection and similar are set with flags or their `DOPPEL_*` environment equivalents; run any command with `--help` for the full list. +`DOPPEL_EXTERNAL_URL` names the address clients reach Doppel at, for rewriting an +upstream's redirects. Needed behind a port mapping or an ingress; otherwise +`server.host:server.port` is used. + ## A minimal configuration ```yaml diff --git a/THIRD-PARTY.md b/THIRD-PARTY.md index 94bbb47..777f79d 100644 --- a/THIRD-PARTY.md +++ b/THIRD-PARTY.md @@ -1,386 +1,406 @@ # Third-party notices -Doppel links its dependencies statically, so a released binary contains the -work below. Each retains its own copyright and is distributed under its own +Doppel links its dependencies statically and embeds the dashboard's compiled +JavaScript, so a released binary contains the work below -- crates and npm +packages alike. Each retains its own copyright and is distributed under its own licence. Generated from the resolved dependency graph by `scripts/third_party.py`. Do not edit by hand -- run the script. -372 packages. +391 packages: 379 from crates.io, 12 from npm. -| Package | Version | Licence | -|---|---|---| -| [addr2line](https://github.com/gimli-rs/addr2line) | 0.25.1 | Apache-2.0 OR MIT | -| [adler2](https://github.com/oyvindln/adler2) | 2.0.1 | 0BSD OR MIT OR Apache-2.0 | -| [aho-corasick](https://github.com/BurntSushi/aho-corasick) | 1.1.4 | Unlicense OR MIT | -| [allocator-api2](https://github.com/zakarumych/allocator-api2) | 0.2.21 | MIT OR Apache-2.0 | -| [android_system_properties](https://github.com/nical/android_system_properties) | 0.1.5 | MIT/Apache-2.0 | -| [anstream](https://github.com/rust-cli/anstyle.git) | 1.0.0 | MIT OR Apache-2.0 | -| [anstyle](https://github.com/rust-cli/anstyle.git) | 1.0.14 | MIT OR Apache-2.0 | -| [anstyle-parse](https://github.com/rust-cli/anstyle.git) | 1.0.0 | MIT OR Apache-2.0 | -| [anstyle-query](https://github.com/rust-cli/anstyle.git) | 1.1.5 | MIT OR Apache-2.0 | -| [anstyle-wincon](https://github.com/rust-cli/anstyle.git) | 3.0.11 | MIT OR Apache-2.0 | -| [anyhow](https://github.com/dtolnay/anyhow) | 1.0.104 | MIT OR Apache-2.0 | -| [arbitrary](https://github.com/rust-fuzz/arbitrary/) | 1.4.2 | MIT OR Apache-2.0 | -| [arc-swap](https://github.com/vorner/arc-swap) | 1.9.2 | MIT OR Apache-2.0 | -| [async-trait](https://github.com/dtolnay/async-trait) | 0.1.91 | MIT OR Apache-2.0 | -| [atoi](https://github.com/pacman82/atoi-rs) | 2.0.0 | MIT | -| [atomic-waker](https://github.com/smol-rs/atomic-waker) | 1.1.2 | Apache-2.0 OR MIT | -| [autocfg](https://github.com/cuviper/autocfg) | 1.5.1 | Apache-2.0 OR MIT | -| [aws-lc-rs](https://github.com/aws/aws-lc-rs) | 1.17.3 | ISC AND (Apache-2.0 OR ISC) | -| [aws-lc-sys](https://github.com/aws/aws-lc-rs) | 0.43.0 | ISC AND (Apache-2.0 OR ISC) AND Apache-2.0 AND MIT AND BSD-3-Clause AND (Apache-2.0 OR ISC OR MIT) AND (Apache-2.0 OR ISC OR MIT-0) | -| [axum](https://github.com/tokio-rs/axum) | 0.8.9 | MIT | -| [axum-core](https://github.com/tokio-rs/axum) | 0.5.6 | MIT | -| [backtrace](https://github.com/rust-lang/backtrace-rs) | 0.3.76 | MIT OR Apache-2.0 | -| [base64](https://github.com/marshallpierce/rust-base64) | 0.22.1 | MIT OR Apache-2.0 | -| [bitflags](https://github.com/bitflags/bitflags) | 2.13.1 | MIT OR Apache-2.0 | -| [block-buffer](https://github.com/RustCrypto/utils) | 0.10.4 | MIT OR Apache-2.0 | -| [block-buffer](https://github.com/RustCrypto/utils) | 0.12.1 | MIT OR Apache-2.0 | -| [block2](https://github.com/madsmtm/objc2) | 0.6.2 | MIT | -| [bumpalo](https://github.com/fitzgen/bumpalo) | 3.20.3 | MIT OR Apache-2.0 | -| [byteorder](https://github.com/BurntSushi/byteorder) | 1.5.0 | Unlicense OR MIT | -| [bytes](https://github.com/tokio-rs/bytes) | 1.12.1 | MIT | -| [cc](https://github.com/rust-lang/cc-rs) | 1.4.0 | MIT OR Apache-2.0 | -| [cfg-if](https://github.com/rust-lang/cfg-if) | 1.0.4 | MIT OR Apache-2.0 | -| [cfg_aliases](https://github.com/katharostech/cfg_aliases) | 0.2.2 | MIT | -| [chacha20](https://github.com/RustCrypto/stream-ciphers) | 0.10.1 | MIT OR Apache-2.0 | -| [clap](https://github.com/clap-rs/clap) | 4.6.5 | MIT OR Apache-2.0 | -| [clap_builder](https://github.com/clap-rs/clap) | 4.6.5 | MIT OR Apache-2.0 | -| [clap_derive](https://github.com/clap-rs/clap) | 4.6.4 | MIT OR Apache-2.0 | -| [clap_lex](https://github.com/clap-rs/clap) | 1.1.0 | MIT OR Apache-2.0 | -| [cmake](https://github.com/rust-lang/cmake-rs) | 0.1.58 | MIT OR Apache-2.0 | -| [cmov](https://github.com/RustCrypto/utils) | 0.5.4 | Apache-2.0 OR MIT | -| [colorchoice](https://github.com/rust-cli/anstyle.git) | 1.0.5 | MIT OR Apache-2.0 | -| [combine](https://github.com/Marwes/combine) | 4.6.7 | MIT | -| [const-oid](https://github.com/RustCrypto/formats) | 0.10.2 | Apache-2.0 OR MIT | -| [core-foundation](https://github.com/servo/core-foundation-rs) | 0.10.1 | MIT OR Apache-2.0 | -| [core-foundation-sys](https://github.com/servo/core-foundation-rs) | 0.8.7 | MIT OR Apache-2.0 | -| [cpufeatures](https://github.com/RustCrypto/utils) | 0.2.17 | MIT OR Apache-2.0 | -| [cpufeatures](https://github.com/RustCrypto/utils) | 0.3.0 | MIT OR Apache-2.0 | -| [crc](https://github.com/mrhooray/crc-rs.git) | 3.4.0 | MIT OR Apache-2.0 | -| [crc-catalog](https://github.com/akhilles/crc-catalog.git) | 2.5.0 | MIT OR Apache-2.0 | -| [crc32fast](https://github.com/srijs/rust-crc32fast) | 1.5.0 | MIT OR Apache-2.0 | -| [crossbeam-epoch](https://github.com/crossbeam-rs/crossbeam) | 0.9.20 | MIT OR Apache-2.0 | -| [crossbeam-queue](https://github.com/crossbeam-rs/crossbeam) | 0.3.13 | MIT OR Apache-2.0 | -| [crossbeam-utils](https://github.com/crossbeam-rs/crossbeam) | 0.8.22 | MIT OR Apache-2.0 | -| [crypto-common](https://github.com/RustCrypto/traits) | 0.1.6 | MIT OR Apache-2.0 | -| [crypto-common](https://github.com/RustCrypto/traits) | 0.2.2 | MIT OR Apache-2.0 | -| [ctutils](https://github.com/RustCrypto/utils) | 0.4.2 | Apache-2.0 OR MIT | -| [debugid](https://github.com/getsentry/rust-debugid) | 0.8.0 | Apache-2.0 | -| [deranged](https://github.com/jhpratt/deranged) | 0.5.8 | MIT OR Apache-2.0 | -| [derive_arbitrary](https://github.com/rust-fuzz/arbitrary) | 1.4.2 | MIT OR Apache-2.0 | -| [digest](https://github.com/RustCrypto/traits) | 0.10.7 | MIT OR Apache-2.0 | -| [digest](https://github.com/RustCrypto/traits) | 0.11.3 | MIT OR Apache-2.0 | -| [dispatch2](https://github.com/madsmtm/objc2) | 0.3.1 | Zlib OR Apache-2.0 OR MIT | -| [displaydoc](https://github.com/yaahc/displaydoc) | 0.2.7 | MIT OR Apache-2.0 | -| [dotenvy](https://github.com/allan2/dotenvy) | 0.15.7 | MIT | -| [dunce](https://gitlab.com/kornelski/dunce) | 1.0.5 | CC0-1.0 OR MIT-0 OR Apache-2.0 | -| [either](https://github.com/rayon-rs/either) | 1.17.0 | MIT OR Apache-2.0 | -| [equivalent](https://github.com/indexmap-rs/equivalent) | 1.0.2 | Apache-2.0 OR MIT | -| [errno](https://github.com/lambda-fairy/rust-errno) | 0.3.14 | MIT OR Apache-2.0 | -| [etcetera](https://github.com/lunacookies/etcetera) | 0.11.0 | MIT OR Apache-2.0 | -| [event-listener](https://github.com/smol-rs/event-listener) | 5.4.2 | Apache-2.0 OR MIT | -| [evmap](https://github.com/jonhoo/evmap.git) | 11.0.0 | MIT OR Apache-2.0 | -| [fastrand](https://github.com/smol-rs/fastrand) | 2.5.0 | Apache-2.0 OR MIT | -| [find-msvc-tools](https://github.com/rust-lang/cc-rs) | 0.1.9 | MIT OR Apache-2.0 | -| [flate2](https://github.com/rust-lang/flate2-rs) | 1.1.9 | MIT OR Apache-2.0 | -| [flume](https://github.com/zesterer/flume) | 0.12.0 | Apache-2.0/MIT | -| [foldhash](https://github.com/orlp/foldhash) | 0.2.0 | Zlib | -| [form_urlencoded](https://github.com/servo/rust-url) | 1.2.2 | MIT OR Apache-2.0 | -| [fs_extra](https://github.com/webdesus/fs_extra) | 1.3.0 | MIT | -| [futures-channel](https://github.com/rust-lang/futures-rs) | 0.3.33 | MIT OR Apache-2.0 | -| [futures-core](https://github.com/rust-lang/futures-rs) | 0.3.33 | MIT OR Apache-2.0 | -| [futures-executor](https://github.com/rust-lang/futures-rs) | 0.3.33 | MIT OR Apache-2.0 | -| [futures-intrusive](https://github.com/Matthias247/futures-intrusive) | 0.5.0 | MIT OR Apache-2.0 | -| [futures-io](https://github.com/rust-lang/futures-rs) | 0.3.33 | MIT OR Apache-2.0 | -| [futures-macro](https://github.com/rust-lang/futures-rs) | 0.3.33 | MIT OR Apache-2.0 | -| [futures-sink](https://github.com/rust-lang/futures-rs) | 0.3.33 | MIT OR Apache-2.0 | -| [futures-task](https://github.com/rust-lang/futures-rs) | 0.3.33 | MIT OR Apache-2.0 | -| [futures-util](https://github.com/rust-lang/futures-rs) | 0.3.33 | MIT OR Apache-2.0 | -| [generator](https://github.com/Xudong-Huang/generator-rs.git) | 0.8.9 | MIT/Apache-2.0 | -| [generic-array](https://github.com/fizyk20/generic-array.git) | 0.14.9 | MIT | -| [getrandom](https://github.com/rust-random/getrandom) | 0.2.17 | MIT OR Apache-2.0 | -| [getrandom](https://github.com/rust-random/getrandom) | 0.3.4 | MIT OR Apache-2.0 | -| [getrandom](https://github.com/rust-random/getrandom) | 0.4.3 | MIT OR Apache-2.0 | -| [gimli](https://github.com/gimli-rs/gimli) | 0.32.3 | MIT OR Apache-2.0 | -| [hashbag](https://github.com/jonhoo/hashbag.git) | 0.1.13 | MIT OR Apache-2.0 | -| [hashbrown](https://github.com/rust-lang/hashbrown) | 0.16.1 | MIT OR Apache-2.0 | -| [hashbrown](https://github.com/rust-lang/hashbrown) | 0.17.1 | MIT OR Apache-2.0 | -| [hashlink](https://github.com/djc/hashlink) | 0.11.1 | MIT OR Apache-2.0 | -| [heck](https://github.com/withoutboats/heck) | 0.5.0 | MIT OR Apache-2.0 | -| [hex](https://github.com/KokaKiwi/rust-hex) | 0.4.3 | MIT OR Apache-2.0 | -| [hkdf](https://github.com/RustCrypto/KDFs/) | 0.13.0 | MIT OR Apache-2.0 | -| [hmac](https://github.com/RustCrypto/MACs) | 0.13.0 | MIT OR Apache-2.0 | -| [hostname](https://github.com/djc/hostname) | 0.4.2 | MIT | -| [http](https://github.com/hyperium/http) | 1.5.0 | MIT OR Apache-2.0 | -| [http-body](https://github.com/hyperium/http-body) | 1.1.0 | MIT | -| [http-body-util](https://github.com/hyperium/http-body) | 0.1.4 | MIT | -| [httparse](https://github.com/seanmonstar/httparse) | 1.10.1 | MIT OR Apache-2.0 | -| [httpdate](https://github.com/pyfisch/httpdate) | 1.0.3 | MIT OR Apache-2.0 | -| [hybrid-array](https://github.com/RustCrypto/hybrid-array) | 0.4.14 | MIT OR Apache-2.0 | -| [hyper](https://github.com/hyperium/hyper) | 1.11.0 | MIT | -| [hyper-rustls](https://github.com/rustls/hyper-rustls) | 0.27.9 | Apache-2.0 OR ISC OR MIT | -| [hyper-util](https://github.com/hyperium/hyper-util) | 0.1.20 | MIT | -| [icu_collections](https://github.com/unicode-org/icu4x) | 2.2.0 | Unicode-3.0 | -| [icu_locale_core](https://github.com/unicode-org/icu4x) | 2.2.0 | Unicode-3.0 | -| [icu_normalizer](https://github.com/unicode-org/icu4x) | 2.2.0 | Unicode-3.0 | -| [icu_normalizer_data](https://github.com/unicode-org/icu4x) | 2.2.0 | Unicode-3.0 | -| [icu_properties](https://github.com/unicode-org/icu4x) | 2.2.0 | Unicode-3.0 | -| [icu_properties_data](https://github.com/unicode-org/icu4x) | 2.2.0 | Unicode-3.0 | -| [icu_provider](https://github.com/unicode-org/icu4x) | 2.2.0 | Unicode-3.0 | -| [idna](https://github.com/servo/rust-url/) | 1.1.0 | MIT OR Apache-2.0 | -| [idna_adapter](https://github.com/hsivonen/idna_adapter) | 1.2.2 | Apache-2.0 OR MIT | -| [indexmap](https://github.com/indexmap-rs/indexmap) | 2.14.0 | Apache-2.0 OR MIT | -| [ipnet](https://github.com/krisprice/ipnet) | 2.12.0 | MIT OR Apache-2.0 | -| [is_terminal_polyfill](https://github.com/polyfill-rs/is_terminal_polyfill) | 1.70.2 | MIT OR Apache-2.0 | -| [itoa](https://github.com/dtolnay/itoa) | 1.0.18 | MIT OR Apache-2.0 | -| [jni](https://github.com/jni-rs/jni-rs) | 0.22.4 | MIT OR Apache-2.0 | -| [jni-macros](https://github.com/jni-rs/jni-rs) | 0.22.4 | MIT OR Apache-2.0 | -| [jni-sys](https://github.com/jni-rs/jni-sys) | 0.4.1 | MIT OR Apache-2.0 | -| [jni-sys-macros](https://github.com/jni-rs/jni-sys) | 0.4.1 | MIT OR Apache-2.0 | -| [jobserver](https://github.com/rust-lang/jobserver-rs) | 0.1.35 | MIT OR Apache-2.0 | -| [js-sys](https://github.com/wasm-bindgen/wasm-bindgen/tree/master/crates/js-sys) | 0.3.103 | MIT OR Apache-2.0 | -| [lazy_static](https://github.com/rust-lang-nursery/lazy-static.rs) | 1.5.0 | MIT OR Apache-2.0 | -| [left-right](https://github.com/jonhoo/left-right.git) | 0.11.8 | MIT OR Apache-2.0 | -| [libc](https://github.com/rust-lang/libc) | 0.2.189 | MIT OR Apache-2.0 | -| [libsqlite3-sys](https://github.com/rusqlite/rusqlite) | 0.37.0 | MIT | -| [linux-raw-sys](https://github.com/sunfishcode/linux-raw-sys) | 0.12.1 | Apache-2.0 WITH LLVM-exception OR Apache-2.0 OR MIT | -| [litemap](https://github.com/unicode-org/icu4x) | 0.8.2 | Unicode-3.0 | -| [lock_api](https://github.com/Amanieu/parking_lot) | 0.4.14 | MIT OR Apache-2.0 | -| [log](https://github.com/rust-lang/log) | 0.4.33 | MIT OR Apache-2.0 | -| [loom](https://github.com/tokio-rs/loom) | 0.7.2 | MIT | -| [lru-slab](https://github.com/Ralith/lru-slab) | 0.1.2 | MIT OR Apache-2.0 OR Zlib | -| [matchers](https://github.com/hawkw/matchers) | 0.2.0 | MIT | -| [matchit](https://github.com/ibraheemdev/matchit) | 0.8.4 | MIT AND BSD-3-Clause | -| [md-5](https://github.com/RustCrypto/hashes) | 0.11.0 | MIT OR Apache-2.0 | -| [memchr](https://github.com/BurntSushi/memchr) | 2.8.3 | Unlicense OR MIT | -| [memo-map](https://github.com/mitsuhiko/memo-map) | 0.3.3 | Apache-2.0 | -| [metrics](https://github.com/metrics-rs/metrics) | 0.24.6 | MIT | -| [metrics-exporter-prometheus](https://github.com/metrics-rs/metrics) | 0.18.3 | MIT AND Apache-2.0 | -| [metrics-util](https://github.com/metrics-rs/metrics) | 0.20.4 | MIT | -| [mime](https://github.com/hyperium/mime) | 0.3.17 | MIT OR Apache-2.0 | -| [mime_guess](https://github.com/abonander/mime_guess) | 2.0.5 | MIT | -| [minijinja](https://github.com/mitsuhiko/minijinja) | 2.21.0 | Apache-2.0 | -| [miniz_oxide](https://github.com/Frommi/miniz_oxide/tree/master/miniz_oxide) | 0.8.9 | MIT OR Zlib OR Apache-2.0 | -| [mio](https://github.com/tokio-rs/mio) | 1.2.2 | MIT | -| [nix](https://github.com/nix-rust/nix) | 0.31.3 | MIT | -| [nu-ansi-term](https://github.com/nushell/nu-ansi-term) | 0.50.3 | MIT | -| [num-conv](https://github.com/jhpratt/num-conv) | 0.2.2 | MIT OR Apache-2.0 | -| [num-traits](https://github.com/rust-num/num-traits) | 0.2.19 | MIT OR Apache-2.0 | -| [objc2](https://github.com/madsmtm/objc2) | 0.6.4 | MIT | -| [objc2-cloud-kit](https://github.com/madsmtm/objc2) | 0.3.2 | Zlib OR Apache-2.0 OR MIT | -| [objc2-core-data](https://github.com/madsmtm/objc2) | 0.3.2 | Zlib OR Apache-2.0 OR MIT | -| [objc2-core-foundation](https://github.com/madsmtm/objc2) | 0.3.2 | Zlib OR Apache-2.0 OR MIT | -| [objc2-core-graphics](https://github.com/madsmtm/objc2) | 0.3.2 | Zlib OR Apache-2.0 OR MIT | -| [objc2-core-image](https://github.com/madsmtm/objc2) | 0.3.2 | Zlib OR Apache-2.0 OR MIT | -| [objc2-core-location](https://github.com/madsmtm/objc2) | 0.3.2 | Zlib OR Apache-2.0 OR MIT | -| [objc2-core-text](https://github.com/madsmtm/objc2) | 0.3.2 | Zlib OR Apache-2.0 OR MIT | -| [objc2-encode](https://github.com/madsmtm/objc2) | 4.1.0 | MIT | -| [objc2-foundation](https://github.com/madsmtm/objc2) | 0.3.2 | MIT | -| [objc2-io-surface](https://github.com/madsmtm/objc2) | 0.3.2 | Zlib OR Apache-2.0 OR MIT | -| [objc2-quartz-core](https://github.com/madsmtm/objc2) | 0.3.2 | Zlib OR Apache-2.0 OR MIT | -| [objc2-ui-kit](https://github.com/madsmtm/objc2) | 0.3.2 | Zlib OR Apache-2.0 OR MIT | -| [objc2-user-notifications](https://github.com/madsmtm/objc2) | 0.3.2 | Zlib OR Apache-2.0 OR MIT | -| [object](https://github.com/gimli-rs/object) | 0.37.3 | Apache-2.0 OR MIT | -| [once_cell](https://github.com/matklad/once_cell) | 1.21.4 | MIT OR Apache-2.0 | -| [once_cell_polyfill](https://github.com/polyfill-rs/once_cell_polyfill) | 1.70.2 | MIT OR Apache-2.0 | -| [openssl-probe](https://github.com/rustls/openssl-probe) | 0.2.1 | MIT OR Apache-2.0 | -| [os_info](https://github.com/stanislav-tkach/os_info) | 3.15.0 | MIT | -| [parking](https://github.com/smol-rs/parking) | 2.2.1 | Apache-2.0 OR MIT | -| [parking_lot](https://github.com/Amanieu/parking_lot) | 0.12.5 | MIT OR Apache-2.0 | -| [parking_lot_core](https://github.com/Amanieu/parking_lot) | 0.9.12 | MIT OR Apache-2.0 | -| [percent-encoding](https://github.com/servo/rust-url/) | 2.3.2 | MIT OR Apache-2.0 | -| [pin-project-lite](https://github.com/taiki-e/pin-project-lite) | 0.2.17 | Apache-2.0 OR MIT | -| [pkg-config](https://github.com/rust-lang/pkg-config-rs) | 0.3.33 | MIT OR Apache-2.0 | -| [portable-atomic](https://github.com/taiki-e/portable-atomic) | 1.14.0 | Apache-2.0 OR MIT | -| [potential_utf](https://github.com/unicode-org/icu4x) | 0.1.5 | Unicode-3.0 | -| [powerfmt](https://github.com/jhpratt/powerfmt) | 0.2.0 | MIT OR Apache-2.0 | -| [ppv-lite86](https://github.com/cryptocorrosion/cryptocorrosion) | 0.2.21 | MIT OR Apache-2.0 | -| [proc-macro2](https://github.com/dtolnay/proc-macro2) | 1.0.107 | MIT OR Apache-2.0 | -| [quanta](https://github.com/metrics-rs/quanta) | 0.12.6 | MIT | -| [quinn](https://github.com/quinn-rs/quinn) | 0.11.11 | MIT OR Apache-2.0 | -| [quinn-proto](https://github.com/quinn-rs/quinn) | 0.11.16 | MIT OR Apache-2.0 | -| [quinn-udp](https://github.com/quinn-rs/quinn) | 0.5.15 | MIT OR Apache-2.0 | -| [quote](https://github.com/dtolnay/quote) | 1.0.47 | MIT OR Apache-2.0 | -| [r-efi](https://github.com/r-efi/r-efi) | 5.3.0 | MIT OR Apache-2.0 OR LGPL-2.1-or-later | -| [r-efi](https://github.com/r-efi/r-efi) | 6.0.0 | MIT OR Apache-2.0 OR LGPL-2.1-or-later | -| [rand](https://github.com/rust-random/rand) | 0.10.2 | MIT OR Apache-2.0 | -| [rand](https://github.com/rust-random/rand) | 0.9.5 | MIT OR Apache-2.0 | -| [rand_chacha](https://github.com/rust-random/rand) | 0.9.0 | MIT OR Apache-2.0 | -| [rand_core](https://github.com/rust-random/rand_core) | 0.10.1 | MIT OR Apache-2.0 | -| [rand_core](https://github.com/rust-random/rand) | 0.9.5 | MIT OR Apache-2.0 | -| [rand_pcg](https://github.com/rust-random/rngs) | 0.10.2 | MIT OR Apache-2.0 | -| [rand_xoshiro](https://github.com/rust-random/rngs) | 0.7.0 | MIT OR Apache-2.0 | -| [rapidhash](https://github.com/hoxxep/rapidhash) | 4.5.1 | MIT OR Apache-2.0 | -| [raw-cpuid](https://github.com/gz/rust-cpuid) | 11.6.0 | MIT | -| [redox_syscall](https://gitlab.redox-os.org/redox-os/syscall) | 0.5.18 | MIT | -| [regex](https://github.com/rust-lang/regex) | 1.13.1 | MIT OR Apache-2.0 | -| [regex-automata](https://github.com/rust-lang/regex) | 0.4.16 | MIT OR Apache-2.0 | -| [regex-syntax](https://github.com/rust-lang/regex) | 0.8.11 | MIT OR Apache-2.0 | -| [reqwest](https://github.com/seanmonstar/reqwest) | 0.12.28 | MIT OR Apache-2.0 | -| [reqwest](https://github.com/seanmonstar/reqwest) | 0.13.4 | MIT OR Apache-2.0 | -| [ring](https://github.com/briansmith/ring) | 0.17.14 | Apache-2.0 AND ISC | -| [rust-embed](https://pyrossh.dev/repos/rust-embed) | 8.12.0 | MIT | -| [rust-embed-impl](https://pyrossh.dev/repos/rust-embed) | 8.12.0 | MIT | -| [rust-embed-utils](https://pyrossh.dev/repos/rust-embed) | 8.12.0 | MIT | -| [rustc-demangle](https://github.com/rust-lang/rustc-demangle) | 0.1.28 | MIT/Apache-2.0 | -| [rustc-hash](https://github.com/rust-lang/rustc-hash) | 2.1.3 | Apache-2.0 OR MIT | -| [rustc_version](https://github.com/djc/rustc-version-rs) | 0.4.1 | MIT OR Apache-2.0 | -| [rustix](https://github.com/bytecodealliance/rustix) | 1.1.4 | Apache-2.0 WITH LLVM-exception OR Apache-2.0 OR MIT | -| [rustls](https://github.com/rustls/rustls) | 0.23.43 | Apache-2.0 OR ISC OR MIT | -| [rustls-native-certs](https://github.com/rustls/rustls-native-certs) | 0.8.4 | Apache-2.0 OR ISC OR MIT | -| [rustls-pki-types](https://github.com/rustls/pki-types) | 1.15.1 | MIT OR Apache-2.0 | -| [rustls-platform-verifier](https://github.com/rustls/rustls-platform-verifier) | 0.7.0 | MIT OR Apache-2.0 | -| [rustls-platform-verifier-android](https://github.com/rustls/rustls-platform-verifier) | 0.1.1 | MIT OR Apache-2.0 | -| [rustls-webpki](https://github.com/rustls/webpki) | 0.103.13 | ISC | -| [rustversion](https://github.com/dtolnay/rustversion) | 1.0.23 | MIT OR Apache-2.0 | -| [ryu](https://github.com/dtolnay/ryu) | 1.0.23 | Apache-2.0 OR BSL-1.0 | -| [same-file](https://github.com/BurntSushi/same-file) | 1.0.6 | Unlicense/MIT | -| [schannel](https://github.com/steffengy/schannel-rs) | 0.1.29 | MIT | -| [scoped-tls](https://github.com/alexcrichton/scoped-tls) | 1.0.1 | MIT/Apache-2.0 | -| [scopeguard](https://github.com/bluss/scopeguard) | 1.2.0 | MIT OR Apache-2.0 | -| [security-framework](https://github.com/kornelski/rust-security-framework) | 3.7.0 | MIT OR Apache-2.0 | -| [security-framework-sys](https://github.com/kornelski/rust-security-framework) | 2.17.0 | MIT OR Apache-2.0 | -| [semver](https://github.com/dtolnay/semver) | 1.0.28 | MIT OR Apache-2.0 | -| [sentry](https://github.com/getsentry/sentry-rust) | 0.49.0 | MIT | -| [sentry-backtrace](https://github.com/getsentry/sentry-rust) | 0.49.0 | MIT | -| [sentry-contexts](https://github.com/getsentry/sentry-rust) | 0.49.0 | MIT | -| [sentry-core](https://github.com/getsentry/sentry-rust) | 0.49.0 | MIT | -| [sentry-panic](https://github.com/getsentry/sentry-rust) | 0.49.0 | MIT | -| [sentry-tracing](https://github.com/getsentry/sentry-rust) | 0.49.0 | MIT | -| [sentry-types](https://github.com/getsentry/sentry-rust) | 0.49.0 | MIT | -| [serde](https://github.com/serde-rs/serde) | 1.0.229 | MIT OR Apache-2.0 | -| [serde_core](https://github.com/serde-rs/serde) | 1.0.229 | MIT OR Apache-2.0 | -| [serde_derive](https://github.com/serde-rs/serde) | 1.0.229 | MIT OR Apache-2.0 | -| [serde_json](https://github.com/serde-rs/json) | 1.0.151 | MIT OR Apache-2.0 | -| [serde_norway](https://github.com/cafkafk/serde-yaml) | 0.9.42 | MIT OR Apache-2.0 | -| [serde_path_to_error](https://github.com/dtolnay/path-to-error) | 0.1.20 | MIT OR Apache-2.0 | -| [serde_urlencoded](https://github.com/nox/serde_urlencoded) | 0.7.1 | MIT/Apache-2.0 | -| [sha1](https://github.com/RustCrypto/hashes) | 0.11.0 | MIT OR Apache-2.0 | -| [sha2](https://github.com/RustCrypto/hashes) | 0.10.9 | MIT OR Apache-2.0 | -| [sha2](https://github.com/RustCrypto/hashes) | 0.11.0 | MIT OR Apache-2.0 | -| [sharded-slab](https://github.com/hawkw/sharded-slab) | 0.1.7 | MIT | -| [shlex](https://github.com/comex/rust-shlex) | 2.0.1 | MIT OR Apache-2.0 | -| [signal-hook-registry](https://github.com/vorner/signal-hook) | 1.4.8 | MIT OR Apache-2.0 | -| [simd-adler32](https://github.com/mcountryman/simd-adler32) | 0.3.10 | MIT | -| [simd_cesu8](https://github.com/seancroach/simd_cesu8) | 1.2.0 | Apache-2.0 OR MIT | -| [simdutf8](https://github.com/rusticstuff/simdutf8) | 0.1.5 | MIT OR Apache-2.0 | -| [sketches-ddsketch](https://github.com/mheffner/rust-sketches-ddsketch) | 0.3.1 | Apache-2.0 | -| [slab](https://github.com/tokio-rs/slab) | 0.4.12 | MIT | -| [smallvec](https://github.com/servo/rust-smallvec) | 1.15.2 | MIT OR Apache-2.0 | -| [socket2](https://github.com/rust-lang/socket2) | 0.6.5 | MIT OR Apache-2.0 | -| [spin](https://github.com/mvdnes/spin-rs.git) | 0.9.9 | MIT | -| [sqlx](https://github.com/launchbadge/sqlx) | 0.9.0 | MIT OR Apache-2.0 | -| [sqlx-core](https://github.com/launchbadge/sqlx) | 0.9.0 | MIT OR Apache-2.0 | -| [sqlx-macros](https://github.com/launchbadge/sqlx) | 0.9.0 | MIT OR Apache-2.0 | -| [sqlx-macros-core](https://github.com/launchbadge/sqlx) | 0.9.0 | MIT OR Apache-2.0 | -| [sqlx-mysql](https://github.com/launchbadge/sqlx) | 0.9.0 | MIT OR Apache-2.0 | -| [sqlx-postgres](https://github.com/launchbadge/sqlx) | 0.9.0 | MIT OR Apache-2.0 | -| [sqlx-sqlite](https://github.com/launchbadge/sqlx) | 0.9.0 | MIT OR Apache-2.0 | -| [stable_deref_trait](https://github.com/storyyeller/stable_deref_trait) | 1.2.1 | MIT OR Apache-2.0 | -| [stringprep](https://github.com/sfackler/rust-stringprep) | 0.1.5 | MIT/Apache-2.0 | -| [strsim](https://github.com/rapidfuzz/strsim-rs) | 0.11.1 | MIT | -| [subtle](https://github.com/dalek-cryptography/subtle) | 2.6.1 | BSD-3-Clause | -| [syn](https://github.com/dtolnay/syn) | 2.0.119 | MIT OR Apache-2.0 | -| [syn](https://github.com/dtolnay/syn) | 3.0.3 | MIT OR Apache-2.0 | -| [sync_wrapper](https://github.com/Actyx/sync_wrapper) | 1.0.2 | Apache-2.0 | -| [synstructure](https://github.com/mystor/synstructure) | 0.13.2 | MIT | -| [tempfile](https://github.com/Stebalien/tempfile) | 3.27.0 | MIT OR Apache-2.0 | -| [thiserror](https://github.com/dtolnay/thiserror) | 2.0.19 | MIT OR Apache-2.0 | -| [thiserror-impl](https://github.com/dtolnay/thiserror) | 2.0.19 | MIT OR Apache-2.0 | -| [thread_local](https://github.com/Amanieu/thread_local-rs) | 1.1.10 | MIT OR Apache-2.0 | -| [time](https://github.com/time-rs/time) | 0.3.55 | MIT OR Apache-2.0 | -| [time-core](https://github.com/time-rs/time) | 0.1.9 | MIT OR Apache-2.0 | -| [time-macros](https://github.com/time-rs/time) | 0.2.32 | MIT OR Apache-2.0 | -| [tinystr](https://github.com/unicode-org/icu4x) | 0.8.3 | Unicode-3.0 | -| [tinyvec](https://github.com/Lokathor/tinyvec) | 1.12.0 | Zlib OR Apache-2.0 OR MIT | -| [tinyvec_macros](https://github.com/Soveu/tinyvec_macros) | 0.1.1 | MIT OR Apache-2.0 OR Zlib | -| [tokio](https://github.com/tokio-rs/tokio) | 1.53.1 | MIT | -| [tokio-macros](https://github.com/tokio-rs/tokio) | 2.7.2 | MIT | -| [tokio-rustls](https://github.com/rustls/tokio-rustls) | 0.26.4 | MIT OR Apache-2.0 | -| [tokio-stream](https://github.com/tokio-rs/tokio) | 0.1.19 | MIT | -| [tokio-util](https://github.com/tokio-rs/tokio) | 0.7.19 | MIT | -| [tower](https://github.com/tower-rs/tower) | 0.5.3 | MIT | -| [tower-http](https://github.com/tower-rs/tower-http) | 0.6.11 | MIT | -| [tower-layer](https://github.com/tower-rs/tower) | 0.3.3 | MIT | -| [tower-service](https://github.com/tower-rs/tower) | 0.3.3 | MIT | -| [tracing](https://github.com/tokio-rs/tracing) | 0.1.44 | MIT | -| [tracing-attributes](https://github.com/tokio-rs/tracing) | 0.1.31 | MIT | -| [tracing-core](https://github.com/tokio-rs/tracing) | 0.1.36 | MIT | -| [tracing-log](https://github.com/tokio-rs/tracing) | 0.2.0 | MIT | -| [tracing-serde](https://github.com/tokio-rs/tracing) | 0.2.0 | MIT | -| [tracing-subscriber](https://github.com/tokio-rs/tracing) | 0.3.23 | MIT | -| [try-lock](https://github.com/seanmonstar/try-lock) | 0.2.5 | MIT | -| [typenum](https://github.com/paholg/typenum) | 1.20.1 | MIT OR Apache-2.0 | -| [uname](https://github.com/icorderi/rust-uname) | 0.1.1 | MIT/Apache-2.0 | -| [unicase](https://github.com/seanmonstar/unicase) | 2.9.0 | MIT OR Apache-2.0 | -| [unicode-bidi](https://github.com/servo/unicode-bidi) | 0.3.18 | MIT OR Apache-2.0 | -| [unicode-ident](https://github.com/dtolnay/unicode-ident) | 1.0.24 | (MIT OR Apache-2.0) AND Unicode-3.0 | -| [unicode-normalization](https://github.com/unicode-rs/unicode-normalization) | 0.1.25 | MIT OR Apache-2.0 | -| [unicode-properties](https://github.com/unicode-rs/unicode-properties) | 0.1.4 | MIT/Apache-2.0 | -| [unsafe-libyaml-norway](https://github.com/cafkafk/unsafe-libyaml-norway) | 0.2.15 | MIT | -| [untrusted](https://github.com/briansmith/untrusted) | 0.9.0 | ISC | -| [ureq](https://github.com/algesten/ureq) | 3.3.0 | MIT OR Apache-2.0 | -| [ureq-proto](https://github.com/algesten/ureq-proto) | 0.6.0 | MIT OR Apache-2.0 | -| [url](https://github.com/servo/rust-url) | 2.5.8 | MIT OR Apache-2.0 | -| [utf8-zero](https://github.com/algesten/utf8-zero) | 0.8.1 | MIT OR Apache-2.0 | -| [utf8_iter](https://github.com/hsivonen/utf8_iter) | 1.0.4 | Apache-2.0 OR MIT | -| [utf8parse](https://github.com/alacritty/vte) | 0.2.2 | Apache-2.0 OR MIT | -| [utoipa](https://github.com/juhaku/utoipa) | 5.5.0 | MIT OR Apache-2.0 | -| [utoipa-gen](https://github.com/juhaku/utoipa) | 5.5.0 | MIT OR Apache-2.0 | -| [utoipa-swagger-ui](https://github.com/juhaku/utoipa) | 9.0.2 | MIT OR Apache-2.0 | -| [utoipa-swagger-ui-vendored](https://github.com/juhaku/utoipa) | 0.1.2 | MIT OR Apache-2.0 | -| [uuid](https://github.com/uuid-rs/uuid) | 1.24.0 | Apache-2.0 OR MIT | -| [valuable](https://github.com/tokio-rs/valuable) | 0.1.1 | MIT | -| [vcpkg](https://github.com/mcgoo/vcpkg-rs) | 0.2.15 | MIT/Apache-2.0 | -| [version_check](https://github.com/SergioBenitez/version_check) | 0.9.5 | MIT/Apache-2.0 | -| [walkdir](https://github.com/BurntSushi/walkdir) | 2.5.0 | Unlicense/MIT | -| [want](https://github.com/seanmonstar/want) | 0.3.1 | MIT | -| [wasi](https://github.com/bytecodealliance/wasi) | 0.11.1+wasi-snapshot-preview1 | Apache-2.0 WITH LLVM-exception OR Apache-2.0 OR MIT | -| [wasip2](https://github.com/bytecodealliance/wasi-rs) | 1.0.4+wasi-0.2.12 | Apache-2.0 WITH LLVM-exception OR Apache-2.0 OR MIT | -| [wasm-bindgen](https://github.com/wasm-bindgen/wasm-bindgen) | 0.2.126 | MIT OR Apache-2.0 | -| [wasm-bindgen-futures](https://github.com/wasm-bindgen/wasm-bindgen/tree/master/crates/futures) | 0.4.76 | MIT OR Apache-2.0 | -| [wasm-bindgen-macro](https://github.com/wasm-bindgen/wasm-bindgen/tree/master/crates/macro) | 0.2.126 | MIT OR Apache-2.0 | -| [wasm-bindgen-macro-support](https://github.com/wasm-bindgen/wasm-bindgen/tree/master/crates/macro-support) | 0.2.126 | MIT OR Apache-2.0 | -| [wasm-bindgen-shared](https://github.com/wasm-bindgen/wasm-bindgen/tree/master/crates/shared) | 0.2.126 | MIT OR Apache-2.0 | -| [wasm-streams](https://github.com/MattiasBuelens/wasm-streams/) | 0.4.2 | MIT OR Apache-2.0 | -| [web-sys](https://github.com/wasm-bindgen/wasm-bindgen/tree/master/crates/web-sys) | 0.3.103 | MIT OR Apache-2.0 | -| [web-time](https://github.com/daxpedda/web-time) | 1.1.0 | MIT OR Apache-2.0 | -| [webpki-root-certs](https://github.com/rustls/webpki-roots) | 1.0.9 | CDLA-Permissive-2.0 | -| [webpki-roots](https://github.com/rustls/webpki-roots) | 1.0.9 | CDLA-Permissive-2.0 | -| [whoami](https://github.com/ardaku/whoami) | 2.1.2 | Apache-2.0 OR BSL-1.0 OR MIT | -| [winapi](https://github.com/retep998/winapi-rs) | 0.3.9 | MIT/Apache-2.0 | -| [winapi-i686-pc-windows-gnu](https://github.com/retep998/winapi-rs) | 0.4.0 | MIT/Apache-2.0 | -| [winapi-util](https://github.com/BurntSushi/winapi-util) | 0.1.11 | Unlicense OR MIT | -| [winapi-x86_64-pc-windows-gnu](https://github.com/retep998/winapi-rs) | 0.4.0 | MIT/Apache-2.0 | -| [windows-link](https://github.com/microsoft/windows-rs) | 0.2.1 | MIT OR Apache-2.0 | -| [windows-result](https://github.com/microsoft/windows-rs) | 0.4.1 | MIT OR Apache-2.0 | -| [windows-sys](https://github.com/microsoft/windows-rs) | 0.52.0 | MIT OR Apache-2.0 | -| [windows-sys](https://github.com/microsoft/windows-rs) | 0.61.2 | MIT OR Apache-2.0 | -| [windows-targets](https://github.com/microsoft/windows-rs) | 0.52.6 | MIT OR Apache-2.0 | -| [windows_aarch64_gnullvm](https://github.com/microsoft/windows-rs) | 0.52.6 | MIT OR Apache-2.0 | -| [windows_aarch64_msvc](https://github.com/microsoft/windows-rs) | 0.52.6 | MIT OR Apache-2.0 | -| [windows_i686_gnu](https://github.com/microsoft/windows-rs) | 0.52.6 | MIT OR Apache-2.0 | -| [windows_i686_gnullvm](https://github.com/microsoft/windows-rs) | 0.52.6 | MIT OR Apache-2.0 | -| [windows_i686_msvc](https://github.com/microsoft/windows-rs) | 0.52.6 | MIT OR Apache-2.0 | -| [windows_x86_64_gnu](https://github.com/microsoft/windows-rs) | 0.52.6 | MIT OR Apache-2.0 | -| [windows_x86_64_gnullvm](https://github.com/microsoft/windows-rs) | 0.52.6 | MIT OR Apache-2.0 | -| [windows_x86_64_msvc](https://github.com/microsoft/windows-rs) | 0.52.6 | MIT OR Apache-2.0 | -| [wit-bindgen](https://github.com/bytecodealliance/wit-bindgen) | 0.57.1 | Apache-2.0 WITH LLVM-exception OR Apache-2.0 OR MIT | -| [writeable](https://github.com/unicode-org/icu4x) | 0.6.3 | Unicode-3.0 | -| [yoke](https://github.com/unicode-org/icu4x) | 0.8.3 | Unicode-3.0 | -| [yoke-derive](https://github.com/unicode-org/icu4x) | 0.8.2 | Unicode-3.0 | -| [zerocopy](https://github.com/google/zerocopy) | 0.8.55 | BSD-2-Clause OR Apache-2.0 OR MIT | -| [zerocopy-derive](https://github.com/google/zerocopy) | 0.8.55 | BSD-2-Clause OR Apache-2.0 OR MIT | -| [zerofrom](https://github.com/unicode-org/icu4x) | 0.1.8 | Unicode-3.0 | -| [zerofrom-derive](https://github.com/unicode-org/icu4x) | 0.1.7 | Unicode-3.0 | -| [zeroize](https://github.com/RustCrypto/utils) | 1.9.0 | Apache-2.0 OR MIT | -| [zerotrie](https://github.com/unicode-org/icu4x) | 0.2.4 | Unicode-3.0 | -| [zerovec](https://github.com/unicode-org/icu4x) | 0.11.6 | Unicode-3.0 | -| [zerovec-derive](https://github.com/unicode-org/icu4x) | 0.11.3 | Unicode-3.0 | -| [zip](https://github.com/zip-rs/zip2.git) | 3.0.0 | MIT | -| [zlib-rs](https://github.com/trifectatechfoundation/zlib-rs) | 0.6.6 | Zlib | -| [zmij](https://github.com/dtolnay/zmij) | 1.0.23 | MIT | -| [zopfli](https://github.com/zopfli-rs/zopfli) | 0.8.3 | Apache-2.0 | +| Package | Ecosystem | Version | Licence | +|---|---|---|---| +| [addr2line](https://github.com/gimli-rs/addr2line) | cargo | 0.25.1 | Apache-2.0 OR MIT | +| [adler2](https://github.com/oyvindln/adler2) | cargo | 2.0.1 | 0BSD OR MIT OR Apache-2.0 | +| [aho-corasick](https://github.com/BurntSushi/aho-corasick) | cargo | 1.1.4 | Unlicense OR MIT | +| [alloc-no-stdlib](https://github.com/dropbox/rust-alloc-no-stdlib) | cargo | 2.0.4 | BSD-3-Clause | +| [alloc-stdlib](https://github.com/dropbox/rust-alloc-no-stdlib) | cargo | 0.2.4 | BSD-3-Clause | +| [allocator-api2](https://github.com/zakarumych/allocator-api2) | cargo | 0.2.21 | MIT OR Apache-2.0 | +| [android_system_properties](https://github.com/nical/android_system_properties) | cargo | 0.1.5 | MIT/Apache-2.0 | +| [anstream](https://github.com/rust-cli/anstyle.git) | cargo | 1.0.0 | MIT OR Apache-2.0 | +| [anstyle](https://github.com/rust-cli/anstyle.git) | cargo | 1.0.14 | MIT OR Apache-2.0 | +| [anstyle-parse](https://github.com/rust-cli/anstyle.git) | cargo | 1.0.0 | MIT OR Apache-2.0 | +| [anstyle-query](https://github.com/rust-cli/anstyle.git) | cargo | 1.1.5 | MIT OR Apache-2.0 | +| [anstyle-wincon](https://github.com/rust-cli/anstyle.git) | cargo | 3.0.11 | MIT OR Apache-2.0 | +| [anyhow](https://github.com/dtolnay/anyhow) | cargo | 1.0.104 | MIT OR Apache-2.0 | +| [arbitrary](https://github.com/rust-fuzz/arbitrary/) | cargo | 1.4.2 | MIT OR Apache-2.0 | +| [arc-swap](https://github.com/vorner/arc-swap) | cargo | 1.9.2 | MIT OR Apache-2.0 | +| [async-compression](https://github.com/Nullus157/async-compression) | cargo | 0.4.43 | MIT OR Apache-2.0 | +| [async-trait](https://github.com/dtolnay/async-trait) | cargo | 0.1.91 | MIT OR Apache-2.0 | +| [atoi](https://github.com/pacman82/atoi-rs) | cargo | 2.0.0 | MIT | +| [atomic-waker](https://github.com/smol-rs/atomic-waker) | cargo | 1.1.2 | Apache-2.0 OR MIT | +| [autocfg](https://github.com/cuviper/autocfg) | cargo | 1.5.1 | Apache-2.0 OR MIT | +| [aws-lc-rs](https://github.com/aws/aws-lc-rs) | cargo | 1.17.3 | ISC AND (Apache-2.0 OR ISC) | +| [aws-lc-sys](https://github.com/aws/aws-lc-rs) | cargo | 0.43.0 | ISC AND (Apache-2.0 OR ISC) AND Apache-2.0 AND MIT AND BSD-3-Clause AND (Apache-2.0 OR ISC OR MIT) AND (Apache-2.0 OR ISC OR MIT-0) | +| [axum](https://github.com/tokio-rs/axum) | cargo | 0.8.9 | MIT | +| [axum-core](https://github.com/tokio-rs/axum) | cargo | 0.5.6 | MIT | +| [backtrace](https://github.com/rust-lang/backtrace-rs) | cargo | 0.3.76 | MIT OR Apache-2.0 | +| [base64](https://github.com/marshallpierce/rust-base64) | cargo | 0.22.1 | MIT OR Apache-2.0 | +| [bitflags](https://github.com/bitflags/bitflags) | cargo | 2.13.1 | MIT OR Apache-2.0 | +| [block-buffer](https://github.com/RustCrypto/utils) | cargo | 0.10.4 | MIT OR Apache-2.0 | +| [block-buffer](https://github.com/RustCrypto/utils) | cargo | 0.12.1 | MIT OR Apache-2.0 | +| [block2](https://github.com/madsmtm/objc2) | cargo | 0.6.2 | MIT | +| [brotli](https://github.com/dropbox/rust-brotli) | cargo | 8.0.4 | BSD-3-Clause AND MIT | +| [brotli-decompressor](https://github.com/dropbox/rust-brotli-decompressor) | cargo | 5.0.3 | BSD-3-Clause/MIT | +| [bumpalo](https://github.com/fitzgen/bumpalo) | cargo | 3.20.3 | MIT OR Apache-2.0 | +| [byteorder](https://github.com/BurntSushi/byteorder) | cargo | 1.5.0 | Unlicense OR MIT | +| [bytes](https://github.com/tokio-rs/bytes) | cargo | 1.12.1 | MIT | +| [cc](https://github.com/rust-lang/cc-rs) | cargo | 1.4.0 | MIT OR Apache-2.0 | +| [cfg-if](https://github.com/rust-lang/cfg-if) | cargo | 1.0.4 | MIT OR Apache-2.0 | +| [cfg_aliases](https://github.com/katharostech/cfg_aliases) | cargo | 0.2.2 | MIT | +| [chacha20](https://github.com/RustCrypto/stream-ciphers) | cargo | 0.10.1 | MIT OR Apache-2.0 | +| [clap](https://github.com/clap-rs/clap) | cargo | 4.6.5 | MIT OR Apache-2.0 | +| [clap_builder](https://github.com/clap-rs/clap) | cargo | 4.6.5 | MIT OR Apache-2.0 | +| [clap_derive](https://github.com/clap-rs/clap) | cargo | 4.6.4 | MIT OR Apache-2.0 | +| [clap_lex](https://github.com/clap-rs/clap) | cargo | 1.1.0 | MIT OR Apache-2.0 | +| [cmake](https://github.com/rust-lang/cmake-rs) | cargo | 0.1.58 | MIT OR Apache-2.0 | +| [cmov](https://github.com/RustCrypto/utils) | cargo | 0.5.4 | Apache-2.0 OR MIT | +| [colorchoice](https://github.com/rust-cli/anstyle.git) | cargo | 1.0.5 | MIT OR Apache-2.0 | +| [combine](https://github.com/Marwes/combine) | cargo | 4.6.7 | MIT | +| [compression-codecs](https://github.com/Nullus157/async-compression) | cargo | 0.4.38 | MIT OR Apache-2.0 | +| [compression-core](https://github.com/Nullus157/async-compression) | cargo | 0.4.32 | MIT OR Apache-2.0 | +| [const-oid](https://github.com/RustCrypto/formats) | cargo | 0.10.2 | Apache-2.0 OR MIT | +| [core-foundation](https://github.com/servo/core-foundation-rs) | cargo | 0.10.1 | MIT OR Apache-2.0 | +| [core-foundation-sys](https://github.com/servo/core-foundation-rs) | cargo | 0.8.7 | MIT OR Apache-2.0 | +| [cpufeatures](https://github.com/RustCrypto/utils) | cargo | 0.2.17 | MIT OR Apache-2.0 | +| [cpufeatures](https://github.com/RustCrypto/utils) | cargo | 0.3.0 | MIT OR Apache-2.0 | +| [crc](https://github.com/mrhooray/crc-rs.git) | cargo | 3.4.0 | MIT OR Apache-2.0 | +| [crc-catalog](https://github.com/akhilles/crc-catalog.git) | cargo | 2.5.0 | MIT OR Apache-2.0 | +| [crc32fast](https://github.com/srijs/rust-crc32fast) | cargo | 1.5.0 | MIT OR Apache-2.0 | +| [crossbeam-epoch](https://github.com/crossbeam-rs/crossbeam) | cargo | 0.9.20 | MIT OR Apache-2.0 | +| [crossbeam-queue](https://github.com/crossbeam-rs/crossbeam) | cargo | 0.3.13 | MIT OR Apache-2.0 | +| [crossbeam-utils](https://github.com/crossbeam-rs/crossbeam) | cargo | 0.8.22 | MIT OR Apache-2.0 | +| [crypto-common](https://github.com/RustCrypto/traits) | cargo | 0.1.6 | MIT OR Apache-2.0 | +| [crypto-common](https://github.com/RustCrypto/traits) | cargo | 0.2.2 | MIT OR Apache-2.0 | +| [ctutils](https://github.com/RustCrypto/utils) | cargo | 0.4.2 | Apache-2.0 OR MIT | +| [debugid](https://github.com/getsentry/rust-debugid) | cargo | 0.8.0 | Apache-2.0 | +| [deranged](https://github.com/jhpratt/deranged) | cargo | 0.5.8 | MIT OR Apache-2.0 | +| [derive_arbitrary](https://github.com/rust-fuzz/arbitrary) | cargo | 1.4.2 | MIT OR Apache-2.0 | +| [digest](https://github.com/RustCrypto/traits) | cargo | 0.10.7 | MIT OR Apache-2.0 | +| [digest](https://github.com/RustCrypto/traits) | cargo | 0.11.3 | MIT OR Apache-2.0 | +| [dispatch2](https://github.com/madsmtm/objc2) | cargo | 0.3.1 | Zlib OR Apache-2.0 OR MIT | +| [displaydoc](https://github.com/yaahc/displaydoc) | cargo | 0.2.7 | MIT OR Apache-2.0 | +| [dotenvy](https://github.com/allan2/dotenvy) | cargo | 0.15.7 | MIT | +| [dunce](https://gitlab.com/kornelski/dunce) | cargo | 1.0.5 | CC0-1.0 OR MIT-0 OR Apache-2.0 | +| [either](https://github.com/rayon-rs/either) | cargo | 1.17.0 | MIT OR Apache-2.0 | +| [equivalent](https://github.com/indexmap-rs/equivalent) | cargo | 1.0.2 | Apache-2.0 OR MIT | +| [errno](https://github.com/lambda-fairy/rust-errno) | cargo | 0.3.14 | MIT OR Apache-2.0 | +| [etcetera](https://github.com/lunacookies/etcetera) | cargo | 0.11.0 | MIT OR Apache-2.0 | +| [event-listener](https://github.com/smol-rs/event-listener) | cargo | 5.4.2 | Apache-2.0 OR MIT | +| [evmap](https://github.com/jonhoo/evmap.git) | cargo | 11.0.0 | MIT OR Apache-2.0 | +| [fastrand](https://github.com/smol-rs/fastrand) | cargo | 2.5.0 | Apache-2.0 OR MIT | +| [find-msvc-tools](https://github.com/rust-lang/cc-rs) | cargo | 0.1.9 | MIT OR Apache-2.0 | +| [flate2](https://github.com/rust-lang/flate2-rs) | cargo | 1.1.9 | MIT OR Apache-2.0 | +| [flume](https://github.com/zesterer/flume) | cargo | 0.12.0 | Apache-2.0/MIT | +| [foldhash](https://github.com/orlp/foldhash) | cargo | 0.2.0 | Zlib | +| [form_urlencoded](https://github.com/servo/rust-url) | cargo | 1.2.2 | MIT OR Apache-2.0 | +| [fs_extra](https://github.com/webdesus/fs_extra) | cargo | 1.3.0 | MIT | +| [futures-channel](https://github.com/rust-lang/futures-rs) | cargo | 0.3.33 | MIT OR Apache-2.0 | +| [futures-core](https://github.com/rust-lang/futures-rs) | cargo | 0.3.33 | MIT OR Apache-2.0 | +| [futures-executor](https://github.com/rust-lang/futures-rs) | cargo | 0.3.33 | MIT OR Apache-2.0 | +| [futures-intrusive](https://github.com/Matthias247/futures-intrusive) | cargo | 0.5.0 | MIT OR Apache-2.0 | +| [futures-io](https://github.com/rust-lang/futures-rs) | cargo | 0.3.33 | MIT OR Apache-2.0 | +| [futures-macro](https://github.com/rust-lang/futures-rs) | cargo | 0.3.33 | MIT OR Apache-2.0 | +| [futures-sink](https://github.com/rust-lang/futures-rs) | cargo | 0.3.33 | MIT OR Apache-2.0 | +| [futures-task](https://github.com/rust-lang/futures-rs) | cargo | 0.3.33 | MIT OR Apache-2.0 | +| [futures-util](https://github.com/rust-lang/futures-rs) | cargo | 0.3.33 | MIT OR Apache-2.0 | +| [generator](https://github.com/Xudong-Huang/generator-rs.git) | cargo | 0.8.9 | MIT/Apache-2.0 | +| [generic-array](https://github.com/fizyk20/generic-array.git) | cargo | 0.14.9 | MIT | +| [getrandom](https://github.com/rust-random/getrandom) | cargo | 0.2.17 | MIT OR Apache-2.0 | +| [getrandom](https://github.com/rust-random/getrandom) | cargo | 0.3.4 | MIT OR Apache-2.0 | +| [getrandom](https://github.com/rust-random/getrandom) | cargo | 0.4.3 | MIT OR Apache-2.0 | +| [gimli](https://github.com/gimli-rs/gimli) | cargo | 0.32.3 | MIT OR Apache-2.0 | +| [hashbag](https://github.com/jonhoo/hashbag.git) | cargo | 0.1.13 | MIT OR Apache-2.0 | +| [hashbrown](https://github.com/rust-lang/hashbrown) | cargo | 0.16.1 | MIT OR Apache-2.0 | +| [hashbrown](https://github.com/rust-lang/hashbrown) | cargo | 0.17.1 | MIT OR Apache-2.0 | +| [hashlink](https://github.com/djc/hashlink) | cargo | 0.11.1 | MIT OR Apache-2.0 | +| [heck](https://github.com/withoutboats/heck) | cargo | 0.5.0 | MIT OR Apache-2.0 | +| [hex](https://github.com/KokaKiwi/rust-hex) | cargo | 0.4.3 | MIT OR Apache-2.0 | +| [hkdf](https://github.com/RustCrypto/KDFs/) | cargo | 0.13.0 | MIT OR Apache-2.0 | +| [hmac](https://github.com/RustCrypto/MACs) | cargo | 0.13.0 | MIT OR Apache-2.0 | +| [hostname](https://github.com/djc/hostname) | cargo | 0.4.2 | MIT | +| [http](https://github.com/hyperium/http) | cargo | 1.5.0 | MIT OR Apache-2.0 | +| [http-body](https://github.com/hyperium/http-body) | cargo | 1.1.0 | MIT | +| [http-body-util](https://github.com/hyperium/http-body) | cargo | 0.1.4 | MIT | +| [httparse](https://github.com/seanmonstar/httparse) | cargo | 1.10.1 | MIT OR Apache-2.0 | +| [httpdate](https://github.com/pyfisch/httpdate) | cargo | 1.0.3 | MIT OR Apache-2.0 | +| [hybrid-array](https://github.com/RustCrypto/hybrid-array) | cargo | 0.4.14 | MIT OR Apache-2.0 | +| [hyper](https://github.com/hyperium/hyper) | cargo | 1.11.0 | MIT | +| [hyper-rustls](https://github.com/rustls/hyper-rustls) | cargo | 0.27.9 | Apache-2.0 OR ISC OR MIT | +| [hyper-util](https://github.com/hyperium/hyper-util) | cargo | 0.1.20 | MIT | +| [icu_collections](https://github.com/unicode-org/icu4x) | cargo | 2.2.0 | Unicode-3.0 | +| [icu_locale_core](https://github.com/unicode-org/icu4x) | cargo | 2.2.0 | Unicode-3.0 | +| [icu_normalizer](https://github.com/unicode-org/icu4x) | cargo | 2.2.0 | Unicode-3.0 | +| [icu_normalizer_data](https://github.com/unicode-org/icu4x) | cargo | 2.2.0 | Unicode-3.0 | +| [icu_properties](https://github.com/unicode-org/icu4x) | cargo | 2.2.0 | Unicode-3.0 | +| [icu_properties_data](https://github.com/unicode-org/icu4x) | cargo | 2.2.0 | Unicode-3.0 | +| [icu_provider](https://github.com/unicode-org/icu4x) | cargo | 2.2.0 | Unicode-3.0 | +| [idna](https://github.com/servo/rust-url/) | cargo | 1.1.0 | MIT OR Apache-2.0 | +| [idna_adapter](https://github.com/hsivonen/idna_adapter) | cargo | 1.2.2 | Apache-2.0 OR MIT | +| [indexmap](https://github.com/indexmap-rs/indexmap) | cargo | 2.14.0 | Apache-2.0 OR MIT | +| [ipnet](https://github.com/krisprice/ipnet) | cargo | 2.12.0 | MIT OR Apache-2.0 | +| [is_terminal_polyfill](https://github.com/polyfill-rs/is_terminal_polyfill) | cargo | 1.70.2 | MIT OR Apache-2.0 | +| [itoa](https://github.com/dtolnay/itoa) | cargo | 1.0.18 | MIT OR Apache-2.0 | +| [jni](https://github.com/jni-rs/jni-rs) | cargo | 0.22.4 | MIT OR Apache-2.0 | +| [jni-macros](https://github.com/jni-rs/jni-rs) | cargo | 0.22.4 | MIT OR Apache-2.0 | +| [jni-sys](https://github.com/jni-rs/jni-sys) | cargo | 0.4.1 | MIT OR Apache-2.0 | +| [jni-sys-macros](https://github.com/jni-rs/jni-sys) | cargo | 0.4.1 | MIT OR Apache-2.0 | +| [jobserver](https://github.com/rust-lang/jobserver-rs) | cargo | 0.1.35 | MIT OR Apache-2.0 | +| [js-sys](https://github.com/wasm-bindgen/wasm-bindgen/tree/master/crates/js-sys) | cargo | 0.3.103 | MIT OR Apache-2.0 | +| [lazy_static](https://github.com/rust-lang-nursery/lazy-static.rs) | cargo | 1.5.0 | MIT OR Apache-2.0 | +| [left-right](https://github.com/jonhoo/left-right.git) | cargo | 0.11.8 | MIT OR Apache-2.0 | +| [libc](https://github.com/rust-lang/libc) | cargo | 0.2.189 | MIT OR Apache-2.0 | +| [libsqlite3-sys](https://github.com/rusqlite/rusqlite) | cargo | 0.37.0 | MIT | +| [linux-raw-sys](https://github.com/sunfishcode/linux-raw-sys) | cargo | 0.12.1 | Apache-2.0 WITH LLVM-exception OR Apache-2.0 OR MIT | +| [litemap](https://github.com/unicode-org/icu4x) | cargo | 0.8.2 | Unicode-3.0 | +| [lock_api](https://github.com/Amanieu/parking_lot) | cargo | 0.4.14 | MIT OR Apache-2.0 | +| [log](https://github.com/rust-lang/log) | cargo | 0.4.33 | MIT OR Apache-2.0 | +| [loom](https://github.com/tokio-rs/loom) | cargo | 0.7.2 | MIT | +| [lru-slab](https://github.com/Ralith/lru-slab) | cargo | 0.1.2 | MIT OR Apache-2.0 OR Zlib | +| [matchers](https://github.com/hawkw/matchers) | cargo | 0.2.0 | MIT | +| [matchit](https://github.com/ibraheemdev/matchit) | cargo | 0.8.4 | MIT AND BSD-3-Clause | +| [md-5](https://github.com/RustCrypto/hashes) | cargo | 0.11.0 | MIT OR Apache-2.0 | +| [memchr](https://github.com/BurntSushi/memchr) | cargo | 2.8.3 | Unlicense OR MIT | +| [memo-map](https://github.com/mitsuhiko/memo-map) | cargo | 0.3.3 | Apache-2.0 | +| [metrics](https://github.com/metrics-rs/metrics) | cargo | 0.24.6 | MIT | +| [metrics-exporter-prometheus](https://github.com/metrics-rs/metrics) | cargo | 0.18.3 | MIT AND Apache-2.0 | +| [metrics-util](https://github.com/metrics-rs/metrics) | cargo | 0.20.4 | MIT | +| [mime](https://github.com/hyperium/mime) | cargo | 0.3.17 | MIT OR Apache-2.0 | +| [mime_guess](https://github.com/abonander/mime_guess) | cargo | 2.0.5 | MIT | +| [minijinja](https://github.com/mitsuhiko/minijinja) | cargo | 2.21.0 | Apache-2.0 | +| [miniz_oxide](https://github.com/Frommi/miniz_oxide/tree/master/miniz_oxide) | cargo | 0.8.9 | MIT OR Zlib OR Apache-2.0 | +| [mio](https://github.com/tokio-rs/mio) | cargo | 1.2.2 | MIT | +| [nix](https://github.com/nix-rust/nix) | cargo | 0.31.3 | MIT | +| [nu-ansi-term](https://github.com/nushell/nu-ansi-term) | cargo | 0.50.3 | MIT | +| [num-conv](https://github.com/jhpratt/num-conv) | cargo | 0.2.2 | MIT OR Apache-2.0 | +| [num-traits](https://github.com/rust-num/num-traits) | cargo | 0.2.19 | MIT OR Apache-2.0 | +| [objc2](https://github.com/madsmtm/objc2) | cargo | 0.6.4 | MIT | +| [objc2-cloud-kit](https://github.com/madsmtm/objc2) | cargo | 0.3.2 | Zlib OR Apache-2.0 OR MIT | +| [objc2-core-data](https://github.com/madsmtm/objc2) | cargo | 0.3.2 | Zlib OR Apache-2.0 OR MIT | +| [objc2-core-foundation](https://github.com/madsmtm/objc2) | cargo | 0.3.2 | Zlib OR Apache-2.0 OR MIT | +| [objc2-core-graphics](https://github.com/madsmtm/objc2) | cargo | 0.3.2 | Zlib OR Apache-2.0 OR MIT | +| [objc2-core-image](https://github.com/madsmtm/objc2) | cargo | 0.3.2 | Zlib OR Apache-2.0 OR MIT | +| [objc2-core-location](https://github.com/madsmtm/objc2) | cargo | 0.3.2 | Zlib OR Apache-2.0 OR MIT | +| [objc2-core-text](https://github.com/madsmtm/objc2) | cargo | 0.3.2 | Zlib OR Apache-2.0 OR MIT | +| [objc2-encode](https://github.com/madsmtm/objc2) | cargo | 4.1.0 | MIT | +| [objc2-foundation](https://github.com/madsmtm/objc2) | cargo | 0.3.2 | MIT | +| [objc2-io-surface](https://github.com/madsmtm/objc2) | cargo | 0.3.2 | Zlib OR Apache-2.0 OR MIT | +| [objc2-quartz-core](https://github.com/madsmtm/objc2) | cargo | 0.3.2 | Zlib OR Apache-2.0 OR MIT | +| [objc2-ui-kit](https://github.com/madsmtm/objc2) | cargo | 0.3.2 | Zlib OR Apache-2.0 OR MIT | +| [objc2-user-notifications](https://github.com/madsmtm/objc2) | cargo | 0.3.2 | Zlib OR Apache-2.0 OR MIT | +| [object](https://github.com/gimli-rs/object) | cargo | 0.37.3 | Apache-2.0 OR MIT | +| [once_cell](https://github.com/matklad/once_cell) | cargo | 1.21.4 | MIT OR Apache-2.0 | +| [once_cell_polyfill](https://github.com/polyfill-rs/once_cell_polyfill) | cargo | 1.70.2 | MIT OR Apache-2.0 | +| [openssl-probe](https://github.com/rustls/openssl-probe) | cargo | 0.2.1 | MIT OR Apache-2.0 | +| [os_info](https://github.com/stanislav-tkach/os_info) | cargo | 3.15.0 | MIT | +| [parking](https://github.com/smol-rs/parking) | cargo | 2.2.1 | Apache-2.0 OR MIT | +| [parking_lot](https://github.com/Amanieu/parking_lot) | cargo | 0.12.5 | MIT OR Apache-2.0 | +| [parking_lot_core](https://github.com/Amanieu/parking_lot) | cargo | 0.9.12 | MIT OR Apache-2.0 | +| [percent-encoding](https://github.com/servo/rust-url/) | cargo | 2.3.2 | MIT OR Apache-2.0 | +| [pin-project-lite](https://github.com/taiki-e/pin-project-lite) | cargo | 0.2.17 | Apache-2.0 OR MIT | +| [pkg-config](https://github.com/rust-lang/pkg-config-rs) | cargo | 0.3.33 | MIT OR Apache-2.0 | +| [portable-atomic](https://github.com/taiki-e/portable-atomic) | cargo | 1.14.0 | Apache-2.0 OR MIT | +| [potential_utf](https://github.com/unicode-org/icu4x) | cargo | 0.1.5 | Unicode-3.0 | +| [powerfmt](https://github.com/jhpratt/powerfmt) | cargo | 0.2.0 | MIT OR Apache-2.0 | +| [ppv-lite86](https://github.com/cryptocorrosion/cryptocorrosion) | cargo | 0.2.21 | MIT OR Apache-2.0 | +| [proc-macro2](https://github.com/dtolnay/proc-macro2) | cargo | 1.0.107 | MIT OR Apache-2.0 | +| [quanta](https://github.com/metrics-rs/quanta) | cargo | 0.12.6 | MIT | +| [quinn](https://github.com/quinn-rs/quinn) | cargo | 0.11.11 | MIT OR Apache-2.0 | +| [quinn-proto](https://github.com/quinn-rs/quinn) | cargo | 0.11.16 | MIT OR Apache-2.0 | +| [quinn-udp](https://github.com/quinn-rs/quinn) | cargo | 0.5.15 | MIT OR Apache-2.0 | +| [quote](https://github.com/dtolnay/quote) | cargo | 1.0.47 | MIT OR Apache-2.0 | +| [r-efi](https://github.com/r-efi/r-efi) | cargo | 5.3.0 | MIT OR Apache-2.0 OR LGPL-2.1-or-later | +| [r-efi](https://github.com/r-efi/r-efi) | cargo | 6.0.0 | MIT OR Apache-2.0 OR LGPL-2.1-or-later | +| [rand](https://github.com/rust-random/rand) | cargo | 0.10.2 | MIT OR Apache-2.0 | +| [rand](https://github.com/rust-random/rand) | cargo | 0.9.5 | MIT OR Apache-2.0 | +| [rand_chacha](https://github.com/rust-random/rand) | cargo | 0.9.0 | MIT OR Apache-2.0 | +| [rand_core](https://github.com/rust-random/rand_core) | cargo | 0.10.1 | MIT OR Apache-2.0 | +| [rand_core](https://github.com/rust-random/rand) | cargo | 0.9.5 | MIT OR Apache-2.0 | +| [rand_pcg](https://github.com/rust-random/rngs) | cargo | 0.10.2 | MIT OR Apache-2.0 | +| [rand_xoshiro](https://github.com/rust-random/rngs) | cargo | 0.7.0 | MIT OR Apache-2.0 | +| [rapidhash](https://github.com/hoxxep/rapidhash) | cargo | 4.5.1 | MIT OR Apache-2.0 | +| [raw-cpuid](https://github.com/gz/rust-cpuid) | cargo | 11.6.0 | MIT | +| [redox_syscall](https://gitlab.redox-os.org/redox-os/syscall) | cargo | 0.5.18 | MIT | +| [regex](https://github.com/rust-lang/regex) | cargo | 1.13.1 | MIT OR Apache-2.0 | +| [regex-automata](https://github.com/rust-lang/regex) | cargo | 0.4.16 | MIT OR Apache-2.0 | +| [regex-syntax](https://github.com/rust-lang/regex) | cargo | 0.8.11 | MIT OR Apache-2.0 | +| [reqwest](https://github.com/seanmonstar/reqwest) | cargo | 0.12.28 | MIT OR Apache-2.0 | +| [reqwest](https://github.com/seanmonstar/reqwest) | cargo | 0.13.4 | MIT OR Apache-2.0 | +| [ring](https://github.com/briansmith/ring) | cargo | 0.17.14 | Apache-2.0 AND ISC | +| [rust-embed](https://pyrossh.dev/repos/rust-embed) | cargo | 8.12.0 | MIT | +| [rust-embed-impl](https://pyrossh.dev/repos/rust-embed) | cargo | 8.12.0 | MIT | +| [rust-embed-utils](https://pyrossh.dev/repos/rust-embed) | cargo | 8.12.0 | MIT | +| [rustc-demangle](https://github.com/rust-lang/rustc-demangle) | cargo | 0.1.28 | MIT/Apache-2.0 | +| [rustc-hash](https://github.com/rust-lang/rustc-hash) | cargo | 2.1.3 | Apache-2.0 OR MIT | +| [rustc_version](https://github.com/djc/rustc-version-rs) | cargo | 0.4.1 | MIT OR Apache-2.0 | +| [rustix](https://github.com/bytecodealliance/rustix) | cargo | 1.1.4 | Apache-2.0 WITH LLVM-exception OR Apache-2.0 OR MIT | +| [rustls](https://github.com/rustls/rustls) | cargo | 0.23.43 | Apache-2.0 OR ISC OR MIT | +| [rustls-native-certs](https://github.com/rustls/rustls-native-certs) | cargo | 0.8.4 | Apache-2.0 OR ISC OR MIT | +| [rustls-pki-types](https://github.com/rustls/pki-types) | cargo | 1.15.1 | MIT OR Apache-2.0 | +| [rustls-platform-verifier](https://github.com/rustls/rustls-platform-verifier) | cargo | 0.7.0 | MIT OR Apache-2.0 | +| [rustls-platform-verifier-android](https://github.com/rustls/rustls-platform-verifier) | cargo | 0.1.1 | MIT OR Apache-2.0 | +| [rustls-webpki](https://github.com/rustls/webpki) | cargo | 0.103.13 | ISC | +| [rustversion](https://github.com/dtolnay/rustversion) | cargo | 1.0.23 | MIT OR Apache-2.0 | +| [ryu](https://github.com/dtolnay/ryu) | cargo | 1.0.23 | Apache-2.0 OR BSL-1.0 | +| [same-file](https://github.com/BurntSushi/same-file) | cargo | 1.0.6 | Unlicense/MIT | +| [schannel](https://github.com/steffengy/schannel-rs) | cargo | 0.1.29 | MIT | +| [scoped-tls](https://github.com/alexcrichton/scoped-tls) | cargo | 1.0.1 | MIT/Apache-2.0 | +| [scopeguard](https://github.com/bluss/scopeguard) | cargo | 1.2.0 | MIT OR Apache-2.0 | +| [security-framework](https://github.com/kornelski/rust-security-framework) | cargo | 3.7.0 | MIT OR Apache-2.0 | +| [security-framework-sys](https://github.com/kornelski/rust-security-framework) | cargo | 2.17.0 | MIT OR Apache-2.0 | +| [semver](https://github.com/dtolnay/semver) | cargo | 1.0.28 | MIT OR Apache-2.0 | +| [sentry](https://github.com/getsentry/sentry-rust) | cargo | 0.49.0 | MIT | +| [sentry-backtrace](https://github.com/getsentry/sentry-rust) | cargo | 0.49.0 | MIT | +| [sentry-contexts](https://github.com/getsentry/sentry-rust) | cargo | 0.49.0 | MIT | +| [sentry-core](https://github.com/getsentry/sentry-rust) | cargo | 0.49.0 | MIT | +| [sentry-panic](https://github.com/getsentry/sentry-rust) | cargo | 0.49.0 | MIT | +| [sentry-tracing](https://github.com/getsentry/sentry-rust) | cargo | 0.49.0 | MIT | +| [sentry-types](https://github.com/getsentry/sentry-rust) | cargo | 0.49.0 | MIT | +| [serde](https://github.com/serde-rs/serde) | cargo | 1.0.229 | MIT OR Apache-2.0 | +| [serde_core](https://github.com/serde-rs/serde) | cargo | 1.0.229 | MIT OR Apache-2.0 | +| [serde_derive](https://github.com/serde-rs/serde) | cargo | 1.0.229 | MIT OR Apache-2.0 | +| [serde_json](https://github.com/serde-rs/json) | cargo | 1.0.151 | MIT OR Apache-2.0 | +| [serde_norway](https://github.com/cafkafk/serde-yaml) | cargo | 0.9.42 | MIT OR Apache-2.0 | +| [serde_path_to_error](https://github.com/dtolnay/path-to-error) | cargo | 0.1.20 | MIT OR Apache-2.0 | +| [serde_urlencoded](https://github.com/nox/serde_urlencoded) | cargo | 0.7.1 | MIT/Apache-2.0 | +| [sha1](https://github.com/RustCrypto/hashes) | cargo | 0.11.0 | MIT OR Apache-2.0 | +| [sha2](https://github.com/RustCrypto/hashes) | cargo | 0.10.9 | MIT OR Apache-2.0 | +| [sha2](https://github.com/RustCrypto/hashes) | cargo | 0.11.0 | MIT OR Apache-2.0 | +| [sharded-slab](https://github.com/hawkw/sharded-slab) | cargo | 0.1.7 | MIT | +| [shlex](https://github.com/comex/rust-shlex) | cargo | 2.0.1 | MIT OR Apache-2.0 | +| [signal-hook-registry](https://github.com/vorner/signal-hook) | cargo | 1.4.8 | MIT OR Apache-2.0 | +| [simd-adler32](https://github.com/mcountryman/simd-adler32) | cargo | 0.3.10 | MIT | +| [simd_cesu8](https://github.com/seancroach/simd_cesu8) | cargo | 1.2.0 | Apache-2.0 OR MIT | +| [simdutf8](https://github.com/rusticstuff/simdutf8) | cargo | 0.1.5 | MIT OR Apache-2.0 | +| [sketches-ddsketch](https://github.com/mheffner/rust-sketches-ddsketch) | cargo | 0.3.1 | Apache-2.0 | +| [slab](https://github.com/tokio-rs/slab) | cargo | 0.4.12 | MIT | +| [smallvec](https://github.com/servo/rust-smallvec) | cargo | 1.15.2 | MIT OR Apache-2.0 | +| [socket2](https://github.com/rust-lang/socket2) | cargo | 0.6.5 | MIT OR Apache-2.0 | +| [spin](https://github.com/mvdnes/spin-rs.git) | cargo | 0.9.9 | MIT | +| [sqlx](https://github.com/launchbadge/sqlx) | cargo | 0.9.0 | MIT OR Apache-2.0 | +| [sqlx-core](https://github.com/launchbadge/sqlx) | cargo | 0.9.0 | MIT OR Apache-2.0 | +| [sqlx-macros](https://github.com/launchbadge/sqlx) | cargo | 0.9.0 | MIT OR Apache-2.0 | +| [sqlx-macros-core](https://github.com/launchbadge/sqlx) | cargo | 0.9.0 | MIT OR Apache-2.0 | +| [sqlx-mysql](https://github.com/launchbadge/sqlx) | cargo | 0.9.0 | MIT OR Apache-2.0 | +| [sqlx-postgres](https://github.com/launchbadge/sqlx) | cargo | 0.9.0 | MIT OR Apache-2.0 | +| [sqlx-sqlite](https://github.com/launchbadge/sqlx) | cargo | 0.9.0 | MIT OR Apache-2.0 | +| [stable_deref_trait](https://github.com/storyyeller/stable_deref_trait) | cargo | 1.2.1 | MIT OR Apache-2.0 | +| [stringprep](https://github.com/sfackler/rust-stringprep) | cargo | 0.1.5 | MIT/Apache-2.0 | +| [strsim](https://github.com/rapidfuzz/strsim-rs) | cargo | 0.11.1 | MIT | +| [subtle](https://github.com/dalek-cryptography/subtle) | cargo | 2.6.1 | BSD-3-Clause | +| [syn](https://github.com/dtolnay/syn) | cargo | 2.0.119 | MIT OR Apache-2.0 | +| [syn](https://github.com/dtolnay/syn) | cargo | 3.0.3 | MIT OR Apache-2.0 | +| [sync_wrapper](https://github.com/Actyx/sync_wrapper) | cargo | 1.0.2 | Apache-2.0 | +| [synstructure](https://github.com/mystor/synstructure) | cargo | 0.13.2 | MIT | +| [tempfile](https://github.com/Stebalien/tempfile) | cargo | 3.27.0 | MIT OR Apache-2.0 | +| [thiserror](https://github.com/dtolnay/thiserror) | cargo | 2.0.19 | MIT OR Apache-2.0 | +| [thiserror-impl](https://github.com/dtolnay/thiserror) | cargo | 2.0.19 | MIT OR Apache-2.0 | +| [thread_local](https://github.com/Amanieu/thread_local-rs) | cargo | 1.1.10 | MIT OR Apache-2.0 | +| [time](https://github.com/time-rs/time) | cargo | 0.3.55 | MIT OR Apache-2.0 | +| [time-core](https://github.com/time-rs/time) | cargo | 0.1.9 | MIT OR Apache-2.0 | +| [time-macros](https://github.com/time-rs/time) | cargo | 0.2.32 | MIT OR Apache-2.0 | +| [tinystr](https://github.com/unicode-org/icu4x) | cargo | 0.8.3 | Unicode-3.0 | +| [tinyvec](https://github.com/Lokathor/tinyvec) | cargo | 1.12.0 | Zlib OR Apache-2.0 OR MIT | +| [tinyvec_macros](https://github.com/Soveu/tinyvec_macros) | cargo | 0.1.1 | MIT OR Apache-2.0 OR Zlib | +| [tokio](https://github.com/tokio-rs/tokio) | cargo | 1.53.1 | MIT | +| [tokio-macros](https://github.com/tokio-rs/tokio) | cargo | 2.7.2 | MIT | +| [tokio-rustls](https://github.com/rustls/tokio-rustls) | cargo | 0.26.4 | MIT OR Apache-2.0 | +| [tokio-stream](https://github.com/tokio-rs/tokio) | cargo | 0.1.19 | MIT | +| [tokio-util](https://github.com/tokio-rs/tokio) | cargo | 0.7.19 | MIT | +| [tower](https://github.com/tower-rs/tower) | cargo | 0.5.3 | MIT | +| [tower-http](https://github.com/tower-rs/tower-http) | cargo | 0.6.11 | MIT | +| [tower-layer](https://github.com/tower-rs/tower) | cargo | 0.3.3 | MIT | +| [tower-service](https://github.com/tower-rs/tower) | cargo | 0.3.3 | MIT | +| [tracing](https://github.com/tokio-rs/tracing) | cargo | 0.1.44 | MIT | +| [tracing-attributes](https://github.com/tokio-rs/tracing) | cargo | 0.1.31 | MIT | +| [tracing-core](https://github.com/tokio-rs/tracing) | cargo | 0.1.36 | MIT | +| [tracing-log](https://github.com/tokio-rs/tracing) | cargo | 0.2.0 | MIT | +| [tracing-serde](https://github.com/tokio-rs/tracing) | cargo | 0.2.0 | MIT | +| [tracing-subscriber](https://github.com/tokio-rs/tracing) | cargo | 0.3.23 | MIT | +| [try-lock](https://github.com/seanmonstar/try-lock) | cargo | 0.2.5 | MIT | +| [typenum](https://github.com/paholg/typenum) | cargo | 1.20.1 | MIT OR Apache-2.0 | +| [uname](https://github.com/icorderi/rust-uname) | cargo | 0.1.1 | MIT/Apache-2.0 | +| [unicase](https://github.com/seanmonstar/unicase) | cargo | 2.9.0 | MIT OR Apache-2.0 | +| [unicode-bidi](https://github.com/servo/unicode-bidi) | cargo | 0.3.18 | MIT OR Apache-2.0 | +| [unicode-ident](https://github.com/dtolnay/unicode-ident) | cargo | 1.0.24 | (MIT OR Apache-2.0) AND Unicode-3.0 | +| [unicode-normalization](https://github.com/unicode-rs/unicode-normalization) | cargo | 0.1.25 | MIT OR Apache-2.0 | +| [unicode-properties](https://github.com/unicode-rs/unicode-properties) | cargo | 0.1.4 | MIT/Apache-2.0 | +| [unsafe-libyaml-norway](https://github.com/cafkafk/unsafe-libyaml-norway) | cargo | 0.2.15 | MIT | +| [untrusted](https://github.com/briansmith/untrusted) | cargo | 0.9.0 | ISC | +| [ureq](https://github.com/algesten/ureq) | cargo | 3.3.0 | MIT OR Apache-2.0 | +| [ureq-proto](https://github.com/algesten/ureq-proto) | cargo | 0.6.0 | MIT OR Apache-2.0 | +| [url](https://github.com/servo/rust-url) | cargo | 2.5.8 | MIT OR Apache-2.0 | +| [utf8-zero](https://github.com/algesten/utf8-zero) | cargo | 0.8.1 | MIT OR Apache-2.0 | +| [utf8_iter](https://github.com/hsivonen/utf8_iter) | cargo | 1.0.4 | Apache-2.0 OR MIT | +| [utf8parse](https://github.com/alacritty/vte) | cargo | 0.2.2 | Apache-2.0 OR MIT | +| [utoipa](https://github.com/juhaku/utoipa) | cargo | 5.5.0 | MIT OR Apache-2.0 | +| [utoipa-gen](https://github.com/juhaku/utoipa) | cargo | 5.5.0 | MIT OR Apache-2.0 | +| [utoipa-swagger-ui](https://github.com/juhaku/utoipa) | cargo | 9.0.2 | MIT OR Apache-2.0 | +| [utoipa-swagger-ui-vendored](https://github.com/juhaku/utoipa) | cargo | 0.1.2 | MIT OR Apache-2.0 | +| [uuid](https://github.com/uuid-rs/uuid) | cargo | 1.24.0 | Apache-2.0 OR MIT | +| [valuable](https://github.com/tokio-rs/valuable) | cargo | 0.1.1 | MIT | +| [vcpkg](https://github.com/mcgoo/vcpkg-rs) | cargo | 0.2.15 | MIT/Apache-2.0 | +| [version_check](https://github.com/SergioBenitez/version_check) | cargo | 0.9.5 | MIT/Apache-2.0 | +| [walkdir](https://github.com/BurntSushi/walkdir) | cargo | 2.5.0 | Unlicense/MIT | +| [want](https://github.com/seanmonstar/want) | cargo | 0.3.1 | MIT | +| [wasi](https://github.com/bytecodealliance/wasi) | cargo | 0.11.1+wasi-snapshot-preview1 | Apache-2.0 WITH LLVM-exception OR Apache-2.0 OR MIT | +| [wasip2](https://github.com/bytecodealliance/wasi-rs) | cargo | 1.0.4+wasi-0.2.12 | Apache-2.0 WITH LLVM-exception OR Apache-2.0 OR MIT | +| [wasm-bindgen](https://github.com/wasm-bindgen/wasm-bindgen) | cargo | 0.2.126 | MIT OR Apache-2.0 | +| [wasm-bindgen-futures](https://github.com/wasm-bindgen/wasm-bindgen/tree/master/crates/futures) | cargo | 0.4.76 | MIT OR Apache-2.0 | +| [wasm-bindgen-macro](https://github.com/wasm-bindgen/wasm-bindgen/tree/master/crates/macro) | cargo | 0.2.126 | MIT OR Apache-2.0 | +| [wasm-bindgen-macro-support](https://github.com/wasm-bindgen/wasm-bindgen/tree/master/crates/macro-support) | cargo | 0.2.126 | MIT OR Apache-2.0 | +| [wasm-bindgen-shared](https://github.com/wasm-bindgen/wasm-bindgen/tree/master/crates/shared) | cargo | 0.2.126 | MIT OR Apache-2.0 | +| [wasm-streams](https://github.com/MattiasBuelens/wasm-streams/) | cargo | 0.4.2 | MIT OR Apache-2.0 | +| [web-sys](https://github.com/wasm-bindgen/wasm-bindgen/tree/master/crates/web-sys) | cargo | 0.3.103 | MIT OR Apache-2.0 | +| [web-time](https://github.com/daxpedda/web-time) | cargo | 1.1.0 | MIT OR Apache-2.0 | +| [webpki-root-certs](https://github.com/rustls/webpki-roots) | cargo | 1.0.9 | CDLA-Permissive-2.0 | +| [webpki-roots](https://github.com/rustls/webpki-roots) | cargo | 1.0.9 | CDLA-Permissive-2.0 | +| [whoami](https://github.com/ardaku/whoami) | cargo | 2.1.2 | Apache-2.0 OR BSL-1.0 OR MIT | +| [winapi](https://github.com/retep998/winapi-rs) | cargo | 0.3.9 | MIT/Apache-2.0 | +| [winapi-i686-pc-windows-gnu](https://github.com/retep998/winapi-rs) | cargo | 0.4.0 | MIT/Apache-2.0 | +| [winapi-util](https://github.com/BurntSushi/winapi-util) | cargo | 0.1.11 | Unlicense OR MIT | +| [winapi-x86_64-pc-windows-gnu](https://github.com/retep998/winapi-rs) | cargo | 0.4.0 | MIT/Apache-2.0 | +| [windows-link](https://github.com/microsoft/windows-rs) | cargo | 0.2.1 | MIT OR Apache-2.0 | +| [windows-result](https://github.com/microsoft/windows-rs) | cargo | 0.4.1 | MIT OR Apache-2.0 | +| [windows-sys](https://github.com/microsoft/windows-rs) | cargo | 0.52.0 | MIT OR Apache-2.0 | +| [windows-sys](https://github.com/microsoft/windows-rs) | cargo | 0.61.2 | MIT OR Apache-2.0 | +| [windows-targets](https://github.com/microsoft/windows-rs) | cargo | 0.52.6 | MIT OR Apache-2.0 | +| [windows_aarch64_gnullvm](https://github.com/microsoft/windows-rs) | cargo | 0.52.6 | MIT OR Apache-2.0 | +| [windows_aarch64_msvc](https://github.com/microsoft/windows-rs) | cargo | 0.52.6 | MIT OR Apache-2.0 | +| [windows_i686_gnu](https://github.com/microsoft/windows-rs) | cargo | 0.52.6 | MIT OR Apache-2.0 | +| [windows_i686_gnullvm](https://github.com/microsoft/windows-rs) | cargo | 0.52.6 | MIT OR Apache-2.0 | +| [windows_i686_msvc](https://github.com/microsoft/windows-rs) | cargo | 0.52.6 | MIT OR Apache-2.0 | +| [windows_x86_64_gnu](https://github.com/microsoft/windows-rs) | cargo | 0.52.6 | MIT OR Apache-2.0 | +| [windows_x86_64_gnullvm](https://github.com/microsoft/windows-rs) | cargo | 0.52.6 | MIT OR Apache-2.0 | +| [windows_x86_64_msvc](https://github.com/microsoft/windows-rs) | cargo | 0.52.6 | MIT OR Apache-2.0 | +| [wit-bindgen](https://github.com/bytecodealliance/wit-bindgen) | cargo | 0.57.1 | Apache-2.0 WITH LLVM-exception OR Apache-2.0 OR MIT | +| [writeable](https://github.com/unicode-org/icu4x) | cargo | 0.6.3 | Unicode-3.0 | +| [yoke](https://github.com/unicode-org/icu4x) | cargo | 0.8.3 | Unicode-3.0 | +| [yoke-derive](https://github.com/unicode-org/icu4x) | cargo | 0.8.2 | Unicode-3.0 | +| [zerocopy](https://github.com/google/zerocopy) | cargo | 0.8.55 | BSD-2-Clause OR Apache-2.0 OR MIT | +| [zerocopy-derive](https://github.com/google/zerocopy) | cargo | 0.8.55 | BSD-2-Clause OR Apache-2.0 OR MIT | +| [zerofrom](https://github.com/unicode-org/icu4x) | cargo | 0.1.8 | Unicode-3.0 | +| [zerofrom-derive](https://github.com/unicode-org/icu4x) | cargo | 0.1.7 | Unicode-3.0 | +| [zeroize](https://github.com/RustCrypto/utils) | cargo | 1.9.0 | Apache-2.0 OR MIT | +| [zerotrie](https://github.com/unicode-org/icu4x) | cargo | 0.2.4 | Unicode-3.0 | +| [zerovec](https://github.com/unicode-org/icu4x) | cargo | 0.11.6 | Unicode-3.0 | +| [zerovec-derive](https://github.com/unicode-org/icu4x) | cargo | 0.11.3 | Unicode-3.0 | +| [zip](https://github.com/zip-rs/zip2.git) | cargo | 3.0.0 | MIT | +| [zlib-rs](https://github.com/trifectatechfoundation/zlib-rs) | cargo | 0.6.6 | Zlib | +| [zmij](https://github.com/dtolnay/zmij) | cargo | 1.0.23 | MIT | +| [zopfli](https://github.com/zopfli-rs/zopfli) | cargo | 0.8.3 | Apache-2.0 | +| [@cfworker/json-schema](https://github.com/cfworker/cfworker) | npm | 4.1.1 | MIT | +| [@types/react](https://github.com/DefinitelyTyped/DefinitelyTyped) | npm | 19.2.18 | MIT | +| cookie-es | npm | 3.1.1 | MIT | +| [csstype](https://github.com/frenic/csstype) | npm | 3.2.3 | MIT | +| [prismjs](https://github.com/PrismJS/prism) | npm | 1.30.0 | MIT | +| [react](https://github.com/react/react) | npm | 19.2.8 | MIT | +| [react-dom](https://github.com/react/react) | npm | 19.2.8 | MIT | +| [react-router](https://github.com/remix-run/react-router) | npm | 8.3.0 | MIT | +| [react-simple-code-editor](https://github.com/react-simple-code-editor/react-simple-code-editor) | npm | 0.14.1 | MIT | +| [scheduler](https://github.com/facebook/react) | npm | 0.27.0 | MIT | +| yaml | npm | 2.9.0 | ISC | +| [zustand](https://github.com/pmndrs/zustand) | npm | 5.0.14 | MIT | diff --git a/assets/favicon.ico b/assets/favicon.ico new file mode 100644 index 0000000..55f16f4 --- /dev/null +++ b/assets/favicon.ico @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:1a3eb6f7a5c8987dc6cf20c2a3c57bd414ac48ffb4165253b2c29bad50a06c2b +size 32038 diff --git a/assets/favicon.svg b/assets/favicon.svg new file mode 100644 index 0000000..c36a0c0 --- /dev/null +++ b/assets/favicon.svg @@ -0,0 +1,51 @@ + +doppel favicon + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/assets/icon-128.png b/assets/icon-128.png similarity index 100% rename from docs/assets/icon-128.png rename to assets/icon-128.png diff --git a/assets/icon.png b/assets/icon.png new file mode 100644 index 0000000..b7bb027 --- /dev/null +++ b/assets/icon.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:31e61af193e98ab55c431b9dd13c22921384a209917718c9140ecabf23685306 +size 138045 diff --git a/docs/assets/icon.svg b/assets/icon.svg similarity index 100% rename from docs/assets/icon.svg rename to assets/icon.svg diff --git a/crates/doppel-admin/Cargo.toml b/crates/doppel-admin/Cargo.toml index 51fbd60..63659ef 100644 --- a/crates/doppel-admin/Cargo.toml +++ b/crates/doppel-admin/Cargo.toml @@ -8,10 +8,13 @@ license.workspace = true [dependencies] doppel-core = { path = "../doppel-core" } axum.workspace = true +httpdate.workspace = true metrics-exporter-prometheus.workspace = true serde.workspace = true serde_json.workspace = true tokio = { workspace = true, features = ["net", "sync"] } +tower.workspace = true +tower-http.workspace = true tracing.workspace = true utoipa.workspace = true utoipa-swagger-ui.workspace = true @@ -21,7 +24,6 @@ async-trait.workspace = true metrics.workspace = true tempfile.workspace = true tokio = { workspace = true, features = ["macros", "rt", "sync"] } -tower.workspace = true [lints] workspace = true diff --git a/crates/doppel-admin/build.rs b/crates/doppel-admin/build.rs new file mode 100644 index 0000000..e471d47 --- /dev/null +++ b/crates/doppel-admin/build.rs @@ -0,0 +1,174 @@ +//! Embed the built dashboard into the binary. +//! +//! Walks `frontend/dist` and writes a table of `include_bytes!` into `OUT_DIR`. +//! No crate does this for us on purpose: `rust-embed` and `include_dir` would +//! each be a direct dependency for a directory walk, and `CONTRIBUTING.md` asks +//! what a new one does that the standard library cannot. This is forty lines. +//! +//! When `frontend/dist` is absent the tables are emitted empty and the +//! `dashboard_assets` cfg is not set, so the crate still compiles and +//! `cargo install --path crates/doppel-cli` works on a machine with no Node. +//! `GET /` then answers 503 naming the command that builds it. + +use std::fmt::Write as _; +use std::path::{Path, PathBuf}; + +fn main() { + // `cargo::` rather than `cargo:` -- the older single-colon form is + // deprecated, and `rustc-check-cfg` is what stops the `unexpected_cfgs` + // lint from firing on our own conditional. + println!("cargo::rustc-check-cfg=cfg(dashboard_assets)"); + + stamp_build_time(); + + let dist = PathBuf::from(env!("CARGO_MANIFEST_DIR")) + .join("../../frontend/dist") + .canonicalize() + .ok() + .filter(|dist| dist.join("index.html").is_file()); + + // Rebuilt when the dashboard is rebuilt. Watching the directory covers a + // file appearing or changing inside it; watching the parent covers `dist` + // itself appearing for the first time, which is the case a first + // `npm run build` produces. + println!("cargo::rerun-if-changed=../../frontend/dist"); + println!("cargo::rerun-if-changed=../../frontend"); + + let mut generated = String::new(); + let Some(dist) = dist else { + writeln!( + generated, + "/// No `frontend/dist` was present when this binary was built.\n\ + pub static ASSETS: &[(&str, &[u8], &str)] = &[];\n\ + pub static INDEX_HTML: &str = \"\";" + ) + .expect("writing to a String cannot fail"); + write(&generated); + return; + }; + + println!("cargo::rustc-cfg=dashboard_assets"); + + let mut files = Vec::new(); + collect(&dist, &dist, &mut files); + files.sort(); + + writeln!( + generated, + "/// Every file of `frontend/dist` except `index.html`, as\n\ + /// (request path under `/static/`, bytes, content type).\n\ + pub static ASSETS: &[(&str, &[u8], &str)] = &[" + ) + .expect("writing to a String cannot fail"); + for (request_path, absolute) in &files { + writeln!( + generated, + " ({request_path:?}, include_bytes!({absolute:?}), {:?}),", + content_type(request_path) + ) + .expect("writing to a String cannot fail"); + } + writeln!(generated, "];").expect("writing to a String cannot fail"); + + // The page has to carry the element the listener substitutes the runtime + // configuration into. Checked here rather than at request time: it is a + // property of the build, so a mismatch between `frontend/index.html` and + // `dashboard.rs` should stop the build rather than serve a page whose + // configuration is the development placeholder. + let index = dist.join("index.html"); + let markup = std::fs::read_to_string(&index).expect("frontend/dist/index.html is readable"); + assert!( + markup.contains("id=\"doppel-config\""), + "{} has no doppel-config element; frontend/index.html and \ + crates/doppel-admin/src/dashboard.rs have diverged", + index.display() + ); + + writeln!( + generated, + "/// The page itself, with the configuration placeholder still in it.\n\ + pub static INDEX_HTML: &str = include_str!({:?});", + index + ) + .expect("writing to a String cannot fail"); + + write(&generated); +} + +/// When this binary was built, as seconds since the epoch. +/// +/// Seconds rather than a year: turning them into one needs a calendar, and the +/// crate has one -- `httpdate`, which hyper already brings in. Doing it there also +/// puts the conversion where `cargo test` reaches it; a build script's own tests +/// are never run. +/// +/// `SOURCE_DATE_EPOCH` is honoured when set, which is what a reproducible build +/// sets and what makes this value the same for two builds of one commit. Without +/// it, the clock -- and the value is then baked in until something makes this +/// script run again, which is the meaning of "build time" rather than a defect. +fn stamp_build_time() { + println!("cargo::rerun-if-env-changed=SOURCE_DATE_EPOCH"); + + let seconds = std::env::var("SOURCE_DATE_EPOCH") + .ok() + .and_then(|value| value.trim().parse::().ok()) + .unwrap_or_else(|| { + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map_or(0, |since| since.as_secs()) + }); + + println!("cargo::rustc-env=DOPPEL_BUILD_EPOCH={seconds}"); +} + +fn write(generated: &str) { + let out = PathBuf::from(std::env::var_os("OUT_DIR").expect("cargo sets OUT_DIR")); + std::fs::write(out.join("assets.rs"), generated).expect("write the asset table"); +} + +/// Every file under `dir`, as (path relative to `root`, absolute path). +/// +/// `index.html` is skipped: it is served from `/` with the configuration +/// substituted into it, not from `/static/` verbatim, so having it in both +/// places would mean two ways to reach one page and only one of them configured. +fn collect(root: &Path, dir: &Path, out: &mut Vec<(String, PathBuf)>) { + let entries = std::fs::read_dir(dir).expect("frontend/dist is readable"); + for entry in entries.flatten() { + let path = entry.path(); + if path.is_dir() { + collect(root, &path, out); + continue; + } + let relative = path + .strip_prefix(root) + .expect("walked below the root") + .to_string_lossy() + // A request path uses forward slashes whatever the build host does. + .replace('\\', "/"); + if relative == "index.html" { + continue; + } + out.push((relative, path)); + } +} + +/// The content type for a built asset, by extension. +/// +/// Only the extensions vite emits. Anything else is served as bytes rather than +/// guessed at: a wrong `Content-Type` on a script is a page that silently does +/// not run. +fn content_type(path: &str) -> &'static str { + match path.rsplit_once('.').map(|(_, extension)| extension) { + Some("js" | "mjs") => "text/javascript; charset=utf-8", + Some("css") => "text/css; charset=utf-8", + Some("html") => "text/html; charset=utf-8", + Some("json") => "application/json", + Some("svg") => "image/svg+xml", + Some("ico") => "image/x-icon", + Some("png") => "image/png", + Some("woff2") => "font/woff2", + Some("woff") => "font/woff", + Some("map") => "application/json", + _ => "application/octet-stream", + } +} diff --git a/crates/doppel-admin/src/dashboard.rs b/crates/doppel-admin/src/dashboard.rs new file mode 100644 index 0000000..746c1ab --- /dev/null +++ b/crates/doppel-admin/src/dashboard.rs @@ -0,0 +1,379 @@ +//! The browser dashboard: the page, its assets, and `robots.txt`. +//! +//! The assets are embedded by `build.rs`, which walks `frontend/dist` and writes +//! the table included below. A binary built without that directory carries an +//! empty table and answers 503 here, so a source build on a machine with no Node +//! still works and says why the dashboard is missing. + +use std::time::{Duration, SystemTime}; + +use axum::Router; +use axum::extract::{Path, State}; +use axum::http::{HeaderName, HeaderValue, header}; +use axum::response::{IntoResponse, Response}; +use axum::routing::get; +use doppel_core::{Error, ErrorCode}; +use serde::Serialize; + +use crate::response::ApiError; +use crate::state::AdminState; + +// `ASSETS` and `INDEX_HTML`, generated at build time. +include!(concat!(env!("OUT_DIR"), "/assets.rs")); + +/// The placeholder `index.html` carries, replaced per request. +/// +/// Matched as a whole element rather than by editing its contents: a partial +/// match that silently failed would leave the development placeholder in a +/// served page, and the page would then report the wrong title, the wrong +/// version, and `public: false` for a public deployment. +const CONFIG_ELEMENT_ID: &str = "doppel-config"; + +/// Kept out of a search index three ways, because they fail differently: a +/// crawler that reads headers sees this, one that reads markup sees the `meta` +/// element in `index.html`, and one that reads `robots.txt` sees the route +/// below. +const ROBOTS_TAG: (HeaderName, HeaderValue) = ( + HeaderName::from_static("x-robots-tag"), + HeaderValue::from_static("noindex, nofollow, noarchive"), +); + +/// Sent with everything here. +/// +/// `nosniff` matters most on the asset route: without it a browser may decide a +/// file is something other than what its `Content-Type` says, and the whole +/// point of serving hashed assets with an exact type is that it does not have to +/// guess. +const NO_SNIFF: (HeaderName, HeaderValue) = ( + HeaderName::from_static("x-content-type-options"), + HeaderValue::from_static("nosniff"), +); + +/// No inline script, no framing, nothing fetched from anywhere but this origin, +/// and exactly one inline stylesheet, named by its hash. +/// +/// The configuration reaches the page as the *contents* of a +/// `application/json` element rather than as inline JavaScript, which is what +/// makes `script-src 'self'` sufficient -- an inline `")? + open_ends; + Some((&template[..open_ends], &template[close_begins..])) +} + +async fn asset(Path(path): Path) -> Response { + let Some((_, bytes, content_type)) = ASSETS.iter().find(|(name, _, _)| *name == path) else { + // The envelope, not the page. A mistyped API path must not answer 200 + // with an HTML document: a client would parse it as a successful + // response, and a human would see a working dashboard where they asked + // for something that is not there. + return refuse(Error::new( + ErrorCode::NotFound, + format!("no asset /static/{path}"), + )); + }; + + ( + [ + (header::CONTENT_TYPE, HeaderValue::from_static(content_type)), + // Safe because vite content-hashes every filename: a changed file is + // a different URL, so nothing has to expire. + ( + header::CACHE_CONTROL, + HeaderValue::from_static("public, max-age=31536000, immutable"), + ), + ], + [ROBOTS_TAG, NO_SNIFF], + *bytes, + ) + .into_response() +} + +/// An error from a dashboard route, carrying the same headers its successes do. +/// +/// The robots header belongs on a refusal as much as on a page: a crawler that +/// reached `/static/typo.js` would otherwise be free to index the error body, and +/// "every dashboard response forbids indexing" is easier to hold to than a list +/// of the ones that do. +fn refuse(error: Error) -> Response { + let mut response = ApiError::from(error).into_response(); + response.headers_mut().extend([ROBOTS_TAG, NO_SNIFF]); + response +} + +async fn robots() -> Response { + ( + [( + header::CONTENT_TYPE, + HeaderValue::from_static("text/plain; charset=utf-8"), + )], + [ROBOTS_TAG, NO_SNIFF], + "User-agent: *\nDisallow: /\n", + ) + .into_response() +} + +#[cfg(test)] +mod tests { + use super::{LATEST_RENDERABLE, build_year, year_of}; + + #[test] + fn a_timestamp_becomes_the_year_it_falls_in() { + assert_eq!(year_of(0), Some(1970)); + assert_eq!(year_of(1_767_225_600), Some(2026)); + } + + #[test] + fn the_year_turns_over_on_the_last_second_of_december() { + // 2024-12-31T23:59:59Z and the second after it. The off-by-one this guards + // would put a January build in the year before. + assert_eq!(year_of(1_735_689_599), Some(2024)); + assert_eq!(year_of(1_735_689_600), Some(2025)); + } + + #[test] + fn a_leap_day_belongs_to_its_own_year() { + // 2024-02-29T00:00:00Z. A calendar of 365-day years puts this in 2025. + assert_eq!(year_of(1_709_164_800), Some(2024)); + } + + #[test] + fn a_timestamp_past_the_calendar_has_no_year() { + // `httpdate` panics rather than returning on year 9999, so the bound is + // checked here. The last second it renders still has to work. + assert_eq!(year_of(LATEST_RENDERABLE), Some(9999)); + assert_eq!(year_of(LATEST_RENDERABLE + 1), None); + assert_eq!(year_of(u64::MAX), None); + } + + #[test] + fn the_build_stamp_reaches_the_page() { + // The wiring, end to end: `build.rs` stamps seconds, this reads them. A + // build script that stopped stamping fails to compile; one that stamped + // something unusable lands on the 1970 fallback, which this catches. + assert!( + build_year() >= 2026, + "the build year is {}, so DOPPEL_BUILD_EPOCH is not a build time", + build_year() + ); + } +} diff --git a/crates/doppel-admin/src/lib.rs b/crates/doppel-admin/src/lib.rs index 39a55d5..1ac6623 100644 --- a/crates/doppel-admin/src/lib.rs +++ b/crates/doppel-admin/src/lib.rs @@ -2,13 +2,21 @@ pub mod access; pub mod body; +pub mod dashboard; pub mod openapi; pub mod proxies; pub mod response; +pub mod rights; +pub mod schema; pub mod state; pub mod status; pub mod templates; +use axum::extract::{MatchedPath, Request}; +use tower::Layer as _; +use tower::util::{MapRequest, MapRequestLayer}; +use tower_http::compression::CompressionLayer; + pub use access::{Action, Caller, authorize, caller_from_headers}; pub use response::ApiError; pub use state::AdminState; @@ -18,16 +26,40 @@ pub use state::AdminState; /// A router rather than a server: `serve` owns the listener and the shutdown /// signal, and the tests drive this directly without binding a port. pub fn router(state: AdminState) -> axum::Router { - axum::Router::new() + // Read from the startup configuration, not the reloaded one: routes are + // built once, so turning the dashboard on or off takes a restart -- the same + // rule `admin.enable` already follows, and `main.example.yaml` says so. + let dashboard = state.startup().admin.is_dashboard_enabled(); + + let mut router = axum::Router::new() .merge(proxies::routes()) .merge(templates::routes()) .merge(status::routes()) - .merge(openapi::routes()) - // Without these two, a typo'd path answers 404 with an empty body and - // a wrong method answers 405 with one. The contract is that every - // error carries the envelope, and the paths a client is most likely - // to hit by accident were the ones that did not. - .fallback(not_found) + .merge(rights::routes()) + .merge(schema::routes()) + .merge(openapi::routes()); + + if dashboard { + router = router.merge(dashboard::routes()); + } + + // Without these two, a typo'd path answers 404 with an empty body and a wrong + // method answers 405 with one. The contract is that every error carries the + // envelope, and the paths a client is most likely to hit by accident were the + // ones that did not. + // + // With the dashboard on, the 404 half becomes its fallback: a GET outside + // `/api/` and `/static/` is a client-side route being reloaded and gets the + // page, and everything else keeps the envelope. That division is what makes + // "everything under /api/ is the API, everything else is the page" true rather + // than merely intended. + if dashboard { + router = router.fallback(dashboard::fallback); + } else { + router = router.fallback(not_found); + } + + router .method_not_allowed_fallback(method_not_allowed) .with_state(state) } @@ -52,6 +84,125 @@ async fn method_not_allowed(method: axum::http::Method, uri: axum::http::Uri) -> .into() } +/// The admin service: the routes, compressed, with a trailing slash trimmed +/// before routing. +pub type App = MapRequest Request>; + +/// The router with a trailing slash trimmed before routing. +/// +/// `/metrics/` and `/api/v1/status/` answer as `/metrics` and `/api/v1/status` +/// do. Axum stopped redirecting between the two spellings in 0.8, and a 404 for +/// a slash is a poor answer for a path an operator typed or a scrape config +/// carried over from somewhere else. +/// +/// A layer around the router rather than on it: `Router::layer` runs after +/// routing has already decided there is nothing there, which is too late to +/// rewrite the path. +/// +/// The proxy listener deliberately gets none of this. A proxied path is relayed +/// byte for byte -- `/orders/` and `/orders` are two resources upstream, and one +/// of this project's own mocks matches `^/api/v1/resource/9/$`. +/// Compression, and where it applies. +/// +/// The dashboard is 140 KB of JavaScript and CSS uncompressed and 74 KB gzipped, +/// and every visitor was paying the difference: the assets were embedded as bytes +/// and served as bytes. The Swagger UI's own stylesheet is another 150 KB. Both +/// are text, both are served by this listener, and every browser asks for them +/// compressed. +/// +/// `br` and `gzip`, negotiated from `Accept-Encoding` -- brotli first when the +/// client takes both, because it is smaller on text and every browser that speaks +/// it says so. A client that asks for neither gets the bytes as they are. +/// +/// On this listener only. The proxy listener relays what the upstream sent, +/// byte for byte and encoding included: a client under test asking for +/// `identity` and getting gzip because a proxy decided to help is a bug in the +/// thing it is testing against. +#[must_use] +pub fn app(state: AdminState) -> App { + // Latency inside compression, so what it measures is the work rather than the + // deflating: a scrape of a 150 KB stylesheet should not read as a slow route. + // Also inside routing, which is what makes `MatchedPath` available -- the + // template is the label, and a label built from the path itself would put one + // series per proxy name and per query string into the exposition. + // + // Compression on the router rather than around it: `axum::serve` wants a + // service whose body is axum's own, and `Router::layer` is what converts the + // compressed body back into one. Running after routing costs nothing here -- + // a response is compressed whatever matched it. + MapRequestLayer::new(trim_trailing_slash as fn(Request) -> Request).layer( + router(state) + .layer(axum::middleware::from_fn(record_admin_request)) + .layer(CompressionLayer::new().br(true).gzip(true)), + ) +} + +/// One admin request, timed and recorded by the route it matched. +/// +/// `MatchedPath` is the template the router matched -- `/api/v1/proxies/{name}`, +/// never `/api/v1/proxies/alpha` -- so a deployment with a hundred proxies has one +/// series and a query string has none. A request that matched nothing has no +/// template, and is recorded under the empty route for the same reason an +/// unresolved proxy request is: the total has to be the total. +async fn record_admin_request( + matched: Option, + request: Request, + next: axum::middleware::Next, +) -> axum::response::Response { + let method = request.method().as_str().to_owned(); + let route = matched.map_or_else( + || doppel_core::metrics::UNMATCHED.to_owned(), + |matched| matched.as_str().to_owned(), + ); + + let started = std::time::Instant::now(); + let response = next.run(request).await; + doppel_core::metrics::record_admin( + &method, + &route, + response.status().as_u16(), + started.elapsed(), + ); + response +} + +/// `/metrics/` becomes `/metrics`, and two paths are left exactly as they came. +/// +/// `/` is not a trailing slash to be trimmed; trimming it would leave an empty +/// path that matches nothing. +/// +/// The Swagger UI is the other one, and it is not a matter of taste: +/// `utoipa-swagger-ui` answers the bare `/swagger-ui` with `303 See Other` to +/// `/swagger-ui/`, so trimming the slash off turns the pair into a redirect that +/// arrives back where it started -- measured as exactly that, `303` with +/// `location: /swagger-ui/`, on a request for `/swagger-ui/`. Its own assets are +/// resolved relative to that path too, which is the second reason the subtree is +/// left alone. +/// +/// Hand-written rather than `tower-http`'s `NormalizePathLayer`, which trims +/// unconditionally and has no way to exempt a subtree. It was that layer, and +/// this is what it broke. +fn trim_trailing_slash(mut request: Request) -> Request { + let uri = request.uri(); + let path = uri.path(); + if path == "/" || !path.ends_with('/') || path.starts_with("/swagger-ui") { + return request; + } + + let trimmed = path.trim_end_matches('/'); + let rebuilt = match uri.query() { + Some(query) => format!("{trimmed}?{query}"), + None => trimmed.to_owned(), + }; + // A path that came in parseable stays parseable with fewer bytes on the end; + // if it somehow does not, the request goes through untouched rather than + // being answered with a 400 nobody can act on. + if let Ok(uri) = rebuilt.parse() { + *request.uri_mut() = uri; + } + request +} + /// Bind nothing, serve until `shutdown` resolves. /// /// Mirrors `doppel_proxy::serve`: the caller owns the listener and the @@ -62,7 +213,12 @@ pub async fn serve( listener: tokio::net::TcpListener, shutdown: impl Future + Send + 'static, ) -> std::io::Result<()> { - axum::serve(listener, router(state)) - .with_graceful_shutdown(shutdown) - .await + // `into_make_service` from `axum::ServiceExt`, not the `Router` method: the + // trailing-slash layer makes this a `Service` rather than a `Router`. + axum::serve( + listener, + axum::ServiceExt::::into_make_service(app(state)), + ) + .with_graceful_shutdown(shutdown) + .await } diff --git a/crates/doppel-admin/src/openapi.rs b/crates/doppel-admin/src/openapi.rs index 82b83a9..dc9a04c 100644 --- a/crates/doppel-admin/src/openapi.rs +++ b/crates/doppel-admin/src/openapi.rs @@ -28,6 +28,8 @@ use crate::state::AdminState; crate::templates::list, crate::templates::upload, crate::templates::remove, + crate::rights::rights, + crate::schema::schema, crate::status::reload, crate::status::status, crate::status::exposition, @@ -38,6 +40,10 @@ use crate::state::AdminState; crate::proxies::ProxyRequest, crate::templates::TemplateEntry, crate::templates::TemplateList, + crate::rights::AccessReport, + crate::rights::ActionRights, + crate::rights::ProxyRights, + crate::rights::CallerView, crate::status::Status, crate::status::ProxyStatus, crate::status::ReloadReport, @@ -78,6 +84,13 @@ impl Modify for TokenScheme { /// `/openapi.json` and `/swagger-ui`. /// +/// Both outside `/api/`, and unversioned. Neither is a resource of the API: the +/// document *describes* every version this binary knows, and the UI is a page +/// served to a browser. `/api/` exists to keep endpoints from colliding with the +/// dashboard's pages, and these two are on the pages' side of that line -- which +/// is also where every tool looks for them, `/openapi.json` being the +/// conventional path the way `/metrics` is. +/// /// The Swagger UI assets are vendored into the binary rather than fetched by /// the build script, which is the crate's default. A build that reaches the /// network cannot run in an offline or egress-restricted environment, and the @@ -106,7 +119,9 @@ mod tests { ("post", "/api/v1/proxies/{name}/templates/{file}"), ("delete", "/api/v1/proxies/{name}/templates/{file}"), ("post", "/api/v1/config/reload"), - ("get", "/status"), + ("get", "/api/v1/access"), + ("get", "/api/v1/schema"), + ("get", "/api/v1/status"), ("get", "/metrics"), ]; diff --git a/crates/doppel-admin/src/proxies.rs b/crates/doppel-admin/src/proxies.rs index e7844fd..3379970 100644 --- a/crates/doppel-admin/src/proxies.rs +++ b/crates/doppel-admin/src/proxies.rs @@ -202,19 +202,42 @@ pub(crate) async fn create( .await .map_err(|err| store_error(&err))?; + // The proxy exists in the store; this is what makes it exist in the process. + crate::status::apply_to_runtime(&state).await?; + Ok(with_etag(StatusCode::CREATED, created, Some(location))) } +/// The rename landed in the configuration and the templates did not follow. +/// +/// The configuration is written first, deliberately: the write is what authorises +/// moving anything. So a failure here leaves a proxy under its new name whose +/// template files are still under the old one, and the mocks naming them render +/// errors until somebody moves them. That is worth saying in full rather than +/// reporting as a store failure, because the fix is a one-line `mv` and the message +/// is the only place a reader could learn which two names to use. +fn renamed_but_stranded(from: &str, to: &str, err: &StoreError) -> Error { + Error::new( + ErrorCode::StoreError, + format!( + "`{from}` was renamed to `{to}` in the configuration, but its templates \ + could not be moved: {err}. The templates are still stored under `{from}`, \ + so any mock of `{to}` naming a template file will fail to render until \ + they are moved." + ), + ) +} + #[utoipa::path( put, path = "/api/v1/proxies/{name}", tag = "proxies", params( - ("name" = String, Path, description = "Proxy name; the body must repeat it"), + ("name" = String, Path, description = "Proxy name; a different name in the body renames it"), ("If-Match" = Option, Header, description = "The revision read earlier, quoted"), ), request_body = ProxyRequest, responses( (status = 200, description = "Replaced; the new revision is in the body and `ETag`", body = ProxyView), - (status = 400, description = "Malformed body, a rename, or two disagreeing revisions", body = ErrorBody), + (status = 400, description = "Malformed body, a taken name, or two disagreeing revisions", body = ErrorBody), (status = 401, body = ErrorBody), (status = 403, body = ErrorBody), (status = 404, body = ErrorBody), (status = 409, description = "The proxy changed since it was read", body = ErrorBody), @@ -234,24 +257,15 @@ pub(crate) async fn update( drop(policy); let request = parse_request(&read_body(body, &headers, MAX_DOCUMENT_BYTES).await?)?; - if request.proxy.name != *name { - // A proxy's name is also its template directory. Renaming through PUT - // would strand the old directory and leave the new proxy pointing at - // templates that are not there, which is not what anyone typing a - // different name into a body is asking for. - return Err(Error::new( - ErrorCode::ConfigInvalid, - format!( - "body names proxy `{}` but the path names `{name}`; \ - a proxy cannot be renamed through an update", - request.proxy.name - ), - ) - .into()); - } let expected = required_revision(&headers, request.revision.as_deref())?; let proxy = request.proxy; + // A name in the body that differs from the path is a rename, not a mistake. It + // used to be refused, because a proxy's name is also where its templates live + // and a rename that left them behind would point every mock at a file that is + // not there -- so the rename moves them, below. + let renamed = (proxy.name.as_str() != name).then(|| proxy.name.to_string()); + let updated = commit(state.store(), |config| { let index = position(config, &name).ok_or_else(|| not_found(&name))?; // Re-checked on every attempt, against whatever the store holds now. @@ -260,20 +274,47 @@ pub(crate) async fn update( if Revision::of_proxy(&config.proxies[index]) != expected { return Err(stale(&name)); } + if let Some(new_name) = &renamed + && position(config, new_name).is_some() + { + // Caught here rather than left to validation, which would report it + // against a position in the document -- `proxies[3].name` -- when what + // the caller needs to know is that the name is taken. + return Err(Error::new( + ErrorCode::ConfigInvalid, + format!("a proxy named `{new_name}` already exists"), + )); + } config.proxies[index] = proxy.clone(); Ok(view(&proxy)) }) .await?; + // Both of these follow the configuration write, and in this order. The write is + // what authorises touching the templates at all -- the same reason `delete` waits + // for it -- and a rename has to happen before the files are pruned, because after + // it they are the new name's. + if let Some(new_name) = &renamed { + state + .store() + .rename_templates(name.as_str(), new_name) + .await + .map_err(|err| renamed_but_stranded(&name, new_name, &err))?; + } // The update landed, so the mocks it removed are gone and the files only - // they named can no longer be rendered. Same order as delete, for the - // same reason: the configuration write is what authorises dropping them. + // they named can no longer be rendered. state .store() - .retain_templates(name.as_str(), &crate::templates::declared(&updated.proxy)) + .retain_templates( + renamed.as_deref().unwrap_or(name.as_str()), + &crate::templates::declared(&updated.proxy), + ) .await .map_err(|err| store_error(&err))?; + // Without this the upstream changes in the document and the traffic does not. + crate::status::apply_to_runtime(&state).await?; + Ok(with_etag(StatusCode::OK, updated, None)) } @@ -330,6 +371,10 @@ pub(crate) async fn remove( .await .map_err(|err| store_error(&err))?; + // A deleted proxy that is still being served is the worst of the three: the + // operator has been told it is gone. + crate::status::apply_to_runtime(&state).await?; + Ok(StatusCode::NO_CONTENT.into_response()) } diff --git a/crates/doppel-admin/src/rights.rs b/crates/doppel-admin/src/rights.rs new file mode 100644 index 0000000..0946693 --- /dev/null +++ b/crates/doppel-admin/src/rights.rs @@ -0,0 +1,200 @@ +//! What the caller may do, reported to the caller. +//! +//! The dashboard needs this: a button for an action the server will refuse is +//! worse than no button, because the operator finds out by trying. Nothing else +//! in the API says anything about the *caller's* rights -- every other endpoint +//! either performs an action or refuses it -- so this is the one endpoint whose +//! subject is the request rather than the configuration. + +use axum::Router; +use axum::extract::State; +use axum::http::HeaderMap; +use axum::routing::get; +use doppel_core::config::ProxyConfig; +use serde::Serialize; +use std::collections::BTreeMap; + +use crate::access::{self, Action, Caller}; +use crate::state::AdminState; + +pub fn routes() -> Router { + Router::new().route("/api/v1/access", get(rights)) +} + +/// Who the caller is, as far as this process is concerned. +/// +/// Returning the token's own name and group leaks nothing -- the caller +/// presented it -- and it is what lets the dashboard say who is signed in +/// rather than only that somebody is. +#[derive(Debug, Serialize, utoipa::ToSchema)] +#[serde(tag = "kind", rename_all = "lowercase")] +pub enum CallerView { + /// No token, or one this configuration does not recognise. The two are not + /// distinguished, for the reason `Caller` gives: telling them apart would + /// confirm which tokens exist. + Anonymous, + Token { + name: String, + group: String, + }, +} + +/// The six actions, as booleans. +#[derive(Debug, Serialize, utoipa::ToSchema)] +pub struct ActionRights { + pub list: bool, + pub read: bool, + pub create: bool, + pub update: bool, + pub delete: bool, + pub upload: bool, +} + +/// The four actions a proxy's own `access` block may override. +/// +/// `list` and `create` are not about one proxy, so they are absent here rather +/// than repeated per proxy -- the same reason `ProxyAccessConfig` cannot spell +/// them. +#[derive(Debug, Serialize, utoipa::ToSchema)] +pub struct ProxyRights { + pub read: bool, + pub update: bool, + pub delete: bool, + pub upload: bool, +} + +#[derive(Debug, Serialize, utoipa::ToSchema)] +pub struct AccessReport { + pub caller: CallerView, + /// The global `access` block, evaluated for this caller. + pub global: ActionRights, + /// Per proxy, where a proxy's own `access` overrides the global answer. + /// + /// Absent -- not empty -- for a caller who may not `list`. The map is keyed + /// by proxy name, so returning it would be a proxy listing by another route, + /// and `access.rs` is explicit that a caller without read access must not be + /// able to tell a real proxy from an invented one. + #[serde(skip_serializing_if = "Option::is_none")] + pub proxies: Option>, +} + +impl From for CallerView { + fn from(caller: Caller) -> Self { + match caller { + Caller::Anonymous => Self::Anonymous, + Caller::Token { name, group } => Self::Token { name, group }, + } + } +} + +/// `GET /api/v1/access` -- what the caller may do. +/// +/// Answers 200 for everybody, anonymous included: an endpoint whose purpose is +/// to report that the caller may do nothing cannot itself demand a right, and +/// there is nothing here a caller could not learn by attempting the six actions +/// and reading the statuses. +/// +/// Every field is `authorize` evaluated, never a second reading of the `access` +/// blocks. A copy of that decision would be a copy that can disagree with the +/// one enforcing it, and the disagreement would reach the operator as a button +/// that does nothing -- or, worse, as a hidden action they were allowed to take. +#[utoipa::path( + get, path = "/api/v1/access", tag = "process", + responses((status = 200, description = "What the calling token may do", body = AccessReport)), + security(("token" = [])), +)] +pub(crate) async fn rights( + State(state): State, + headers: HeaderMap, +) -> axum::Json { + // The policy the last reload put into effect, like every other handler -- + // not whatever is in the store right now. Reporting rights from an unloaded + // document would describe a server nobody is running. + let config = access::policy(&state); + let caller = access::caller_from_headers_with_env(&config.admin, state.env_tokens(), &headers); + + let permits = |proxy: Option<&ProxyConfig>, action: Action| { + access::authorize(&config.admin, proxy, action, &caller).is_ok() + }; + + let global = ActionRights { + list: permits(None, Action::List), + read: permits(None, Action::Read), + create: permits(None, Action::Create), + update: permits(None, Action::Update), + delete: permits(None, Action::Delete), + upload: permits(None, Action::Upload), + }; + + let proxies = global.list.then(|| { + config + .proxies + .iter() + .map(|proxy| { + ( + proxy.name.to_string(), + ProxyRights { + read: permits(Some(proxy), Action::Read), + update: permits(Some(proxy), Action::Update), + delete: permits(Some(proxy), Action::Delete), + upload: permits(Some(proxy), Action::Upload), + }, + ) + }) + .collect() + }); + + axum::Json(AccessReport { + caller: caller.into(), + global, + proxies, + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn an_anonymous_caller_is_reported_without_a_name() { + let json = serde_json::to_value(CallerView::from(Caller::Anonymous)).unwrap(); + assert_eq!(json, serde_json::json!({"kind": "anonymous"})); + } + + #[test] + fn a_token_is_reported_by_its_name_and_group() { + let json = serde_json::to_value(CallerView::from(Caller::Token { + name: "ci".to_owned(), + group: "ops".to_owned(), + })) + .unwrap(); + assert_eq!( + json, + serde_json::json!({"kind": "token", "name": "ci", "group": "ops"}) + ); + } + + #[test] + fn an_absent_proxy_map_is_absent_rather_than_empty() { + // An empty object says "there are no proxies", which is a statement about + // the configuration. The point of withholding the map is to say nothing + // at all, so the key has to be gone. + let report = AccessReport { + caller: CallerView::Anonymous, + global: ActionRights { + list: false, + read: false, + create: false, + update: false, + delete: false, + upload: false, + }, + proxies: None, + }; + let json = serde_json::to_value(&report).unwrap(); + assert!( + json.get("proxies").is_none(), + "proxies must be absent, not null or empty: {json}" + ); + } +} diff --git a/crates/doppel-admin/src/schema.rs b/crates/doppel-admin/src/schema.rs new file mode 100644 index 0000000..6c70810 --- /dev/null +++ b/crates/doppel-admin/src/schema.rs @@ -0,0 +1,108 @@ +//! The configuration schema, served by the process the configuration is for. +//! +//! `doppel-config.schema.json` in the repository and the copy attached to every +//! release are the same document, and both answer "what does a configuration look +//! like" for whichever version you fetched. This answers it for the version that is +//! *running*, which is the question a client editing a live configuration is +//! actually asking. +//! +//! The dashboard is that client: it takes the field bounds it enforces while +//! someone types -- patterns, lengths, ranges -- and the validation in its YAML +//! editor from here, so there is no second, laxer copy of the rules written in +//! TypeScript. A rule added to a newtype in `doppel-core` reaches the page without +//! anyone editing the page. + +use std::sync::OnceLock; + +use axum::Router; +use axum::http::{HeaderValue, header}; +use axum::response::{IntoResponse, Response}; +use axum::routing::get; + +use crate::state::AdminState; + +pub fn routes() -> Router { + Router::new().route("/api/v1/schema", get(schema)) +} + +/// The schema, serialized once. +/// +/// Building it walks every `ToSchema` implementation in the configuration types +/// and rewrites every `$ref`, which is not work worth repeating per request for a +/// document that cannot change while the process lives. +fn body() -> &'static str { + static BODY: OnceLock = OnceLock::new(); + BODY.get_or_init(|| { + serde_json::to_string(&doppel_core::config::schema::json_schema()) + .expect("a json value serializes") + }) +} + +/// `GET /api/v1/schema` -- the configuration document as a JSON Schema. +/// +/// Unauthenticated, like `/api/v1/access` and `/api/v1/status`: this describes the +/// shape of a configuration, never the contents of one. The identical bytes are +/// published on GitHub and attached to each release, so a token here would guard +/// nothing while stopping the dashboard from validating anything before a caller +/// signs in. +#[utoipa::path( + get, path = "/api/v1/schema", tag = "process", + responses((status = 200, description = "The configuration schema, JSON Schema 2020-12")), +)] +pub(crate) async fn schema() -> Response { + ( + [ + // The registered type for a JSON Schema document. Every JSON parser + // reads it as JSON regardless, and an editor that looks at the header + // learns something true from this one. + ( + header::CONTENT_TYPE, + HeaderValue::from_static("application/schema+json"), + ), + // It cannot change without a restart, and a restart is usually a new + // version -- so a short cache saves the bytes on a page reload without + // outliving an upgrade by long. + ( + header::CACHE_CONTROL, + HeaderValue::from_static("public, max-age=600"), + ), + ], + body(), + ) + .into_response() +} + +#[cfg(test)] +mod tests { + use super::body; + + #[test] + fn the_served_bytes_are_the_schema_the_code_generates() { + // The same document `doppel config schema` prints and the repository keeps, + // minus the pretty-printing. A second serialization here would be a second + // schema, and the drift test in `doppel-core` would not see it. + let served: serde_json::Value = + serde_json::from_str(body()).expect("served bytes are JSON"); + assert_eq!(served, doppel_core::config::schema::json_schema()); + } + + #[test] + fn the_schema_carries_the_bounds_a_client_validates_with() { + // What the dashboard reads out of it. If a refactor stopped emitting these, + // the page would quietly accept anything and let the server refuse it -- + // which is the behaviour this endpoint exists to replace. + let served: serde_json::Value = + serde_json::from_str(body()).expect("served bytes are JSON"); + let name = &served["$defs"]["ProxyName"]; + assert!(name["pattern"].is_string(), "no pattern on ProxyName"); + assert!(name["maxLength"].is_number(), "no maxLength on ProxyName"); + assert!( + served["$defs"]["Ratio"]["maximum"].is_number(), + "no maximum on Ratio" + ); + assert!( + served["$defs"]["ProxyConfig"]["properties"]["url"].is_object(), + "no url property on ProxyConfig" + ); + } +} diff --git a/crates/doppel-admin/src/status.rs b/crates/doppel-admin/src/status.rs index 73b4807..baaf1c6 100644 --- a/crates/doppel-admin/src/status.rs +++ b/crates/doppel-admin/src/status.rs @@ -16,7 +16,7 @@ use crate::state::AdminState; pub fn routes() -> Router { Router::new() - .route("/status", get(status)) + .route("/api/v1/status", get(status)) .route("/metrics", get(exposition)) .route("/api/v1/config/reload", post(reload)) } @@ -60,8 +60,12 @@ pub struct ReloadReport { /// Unauthenticated by design -- it is the endpoint a load balancer calls, and /// that is why every upstream goes through `redact_credentials` on the way /// out. +/// +/// Under `/api/` like everything else the API serves. It was `/status`, which put +/// it in the same namespace as the dashboard's own pages: a reload on the +/// dashboard's `/status` tab reached this and returned JSON instead of the page. #[utoipa::path( - get, path = "/status", tag = "process", + get, path = "/api/v1/status", tag = "process", responses((status = 200, description = "What this process is serving right now", body = Status)), )] pub(crate) async fn status(State(state): State) -> Response { @@ -95,9 +99,18 @@ pub(crate) async fn status(State(state): State) -> Response { /// The Prometheus exposition. /// -/// Unauthenticated, like `/status`: a scraper is a machine on the operator's -/// network with no place to put a token, and the exposition names proxies and -/// counts -- never a token, a URL or a header value. +/// Unauthenticated, like the status endpoint: a scraper is a machine on the +/// operator's network with no place to put a token, and the exposition names +/// proxies and counts -- never a token, a URL or a header value. +/// +/// The one endpoint outside `/api/`, deliberately. `metrics_path` defaults to +/// `/metrics` in Prometheus, in every Kubernetes annotation and in every agent +/// that scrapes one, so that path is settled by something larger than this +/// project -- and while it was under `/api/v1/`, a scrape of `/metrics` was +/// answered by the dashboard's fallback with `200 text/html`, which a scraper +/// reports as a parse failure or, worse, records as nothing at all. It cannot +/// collide with a page: the dashboard's routes are `/`, `/proxies`, +/// `/proxies/{name}` and `/status`. #[utoipa::path( get, path = "/metrics", tag = "process", responses((status = 200, description = "Prometheus text exposition", content_type = "text/plain")), @@ -116,6 +129,49 @@ pub(crate) async fn exposition(State(state): State) -> Response { .into_response() } +/// Promote the stored configuration to the running one, for a caller that has +/// just written to the store. +/// +/// A write that only reached the store is a 200 that changed nothing a client can +/// see: the proxy keeps forwarding to the old upstream, `/api/v1/status` keeps +/// reporting the old revision, and the CRUD endpoints keep reading the new +/// document back -- so the API agrees with the operator while the process +/// disagrees with both. Reported as exactly that, from a running deployment, +/// after a `PUT` changed an upstream and the traffic did not move. +/// +/// Authorization is not repeated here. The caller has already been authorized for +/// the write, and this is the second half of that write rather than a promotion +/// of its own -- unlike the endpoint below, which is reachable on its own and +/// authorizes against the running configuration for a reason worth reading there. +pub(crate) async fn apply_to_runtime(state: &AdminState) -> Result<(), Error> { + // The same mutex the reload endpoint and the control socket take: two swaps + // that interleave can leave the process running the older configuration. + let _guard = state.reload_lock().lock().await; + + doppel_core::reload(state.holder(), state.store(), state.startup()) + .await + .map(|_| ()) + .map_err(|failure| { + // Both facts, because the write is not being undone: the document is + // stored and the process is not serving it. A message naming only one + // of the two sends the operator looking in the wrong place. + Error::new( + failure.code, + format!( + "the configuration was saved but this process could not start \ + serving it, so it is still running the previous one: {}. \ + `POST /api/v1/config/reload` retries.", + failure + .violations + .iter() + .map(ToString::to_string) + .collect::>() + .join("; ") + ), + ) + }) +} + /// Promote the stored configuration to the running one. #[utoipa::path( post, path = "/api/v1/config/reload", tag = "process", diff --git a/crates/doppel-admin/tests/common/mod.rs b/crates/doppel-admin/tests/common/mod.rs index 8083a62..c094fe0 100644 --- a/crates/doppel-admin/tests/common/mod.rs +++ b/crates/doppel-admin/tests/common/mod.rs @@ -10,7 +10,6 @@ use std::path::{Path, PathBuf}; use std::sync::Arc; use std::time::Instant; -use axum::Router; use axum::body::Body; use axum::http::{Request, Response, StatusCode}; use doppel_admin::AdminState; @@ -124,7 +123,7 @@ pub struct Harness { pub templates_dir: PathBuf, pub store: Arc, /// The runtime the process would be serving from. Built from the same - /// document the store starts with, so `/status` and the store agree + /// document the store starts with, so `/api/v1/status` and the store agree /// until a test makes them disagree on purpose. pub holder: Arc, pub startup: Arc, @@ -180,8 +179,8 @@ impl Harness { self.store = wrap(inner); } - pub fn router(&self) -> Router { - doppel_admin::router(AdminState::new( + pub fn router(&self) -> doppel_admin::App { + doppel_admin::app(AdminState::new( Arc::clone(&self.store), Arc::clone(&self.holder), Arc::clone(&self.startup), @@ -270,7 +269,7 @@ impl Call { self } - pub async fn send(self, router: Router) -> Reply { + pub async fn send(self, router: doppel_admin::App) -> Reply { let mut builder = Request::builder().method(self.method).uri(&self.uri); if let Some(token) = self.token { builder = builder.header("X-Proxy-Authorization", format!("Bearer {token}")); @@ -306,6 +305,12 @@ pub struct Reply { pub content_type: Option, pub allow: Option, pub body: String, + /// The body as it arrived. An embedded asset is not necessarily text -- the + /// favicon is not -- so `body` cannot be the only view of it. + pub bytes: Vec, + /// Every response header, lower-cased, for the assertions that are about a + /// header this struct has no field for. + pub headers: std::collections::HashMap, } impl Reply { @@ -322,6 +327,16 @@ impl Reply { let location = header("location"); let content_type = header("content-type"); let allow = header("allow"); + let headers = response + .headers() + .iter() + .filter_map(|(name, value)| { + value + .to_str() + .ok() + .map(|value| (name.as_str().to_owned(), value.to_owned())) + }) + .collect(); let bytes = axum::body::to_bytes(response.into_body(), 1 << 20) .await .expect("collect body"); @@ -331,10 +346,19 @@ impl Reply { location, content_type, allow, - body: String::from_utf8(bytes.to_vec()).expect("body is utf-8"), + // Lossy rather than strict: an asset response is bytes, and a test + // asserting on a header should not panic on the body's encoding. + body: String::from_utf8_lossy(&bytes).into_owned(), + bytes: bytes.to_vec(), + headers, } } + /// One response header, or `None`. + pub fn header(&self, name: &str) -> Option<&str> { + self.headers.get(name).map(String::as_str) + } + pub fn json(&self) -> serde_json::Value { serde_json::from_str(&self.body) .unwrap_or_else(|err| panic!("body is not JSON ({err}): {}", self.body)) @@ -461,6 +485,10 @@ impl ConfigStore for RacingStore { self.inner.delete_template(proxy, file).await } + async fn rename_templates(&self, from: &str, to: &str) -> Result<(), StoreError> { + self.inner.rename_templates(from, to).await + } + async fn retain_templates(&self, proxy: &str, keep: &[String]) -> Result<(), StoreError> { self.inner.retain_templates(proxy, keep).await } diff --git a/crates/doppel-admin/tests/dashboard.rs b/crates/doppel-admin/tests/dashboard.rs new file mode 100644 index 0000000..2d58e29 --- /dev/null +++ b/crates/doppel-admin/tests/dashboard.rs @@ -0,0 +1,372 @@ +//! The dashboard routes: the page, its assets, and `robots.txt`. + +mod common; + +use common::{BASE_CONFIG, Call, Harness}; + +/// Whether this binary carries the built dashboard. +/// +/// A source build with no `frontend/dist` has no assets to serve, and the tests +/// below that need real ones say so rather than failing for the wrong reason. +/// CI sets `DOPPEL_REQUIRE_DASHBOARD_ASSETS`, which +/// `the_assets_are_present_when_they_are_required` turns into a failure -- the +/// same arrangement `DOPPEL_REQUIRE_DATABASE` makes for the store suites, and for +/// the same reason: `cargo test` captures the output of passing tests, so a skip +/// notice reaches nobody. +fn built() -> bool { + doppel_admin::dashboard::is_built() +} + +/// The configuration element's contents, parsed. +fn injected(body: &str) -> serde_json::Value { + let at = body + .find("id=\"doppel-config\"") + .unwrap_or_else(|| panic!("no config element in the page:\n{body}")); + let open_ends = body[at..].find('>').expect("an unclosed opening tag") + at + 1; + let close = body[open_ends..].find("").expect("no closing tag") + open_ends; + serde_json::from_str(body[open_ends..close].trim()).unwrap_or_else(|err| { + panic!( + "the injected block is not JSON ({err}): {}", + &body[open_ends..close] + ) + }) +} + +#[tokio::test] +async fn the_assets_are_present_when_they_are_required() { + // Not a skip: this is the test that makes the others' skips visible. + if std::env::var_os("DOPPEL_REQUIRE_DASHBOARD_ASSETS").is_some() { + assert!( + built(), + "DOPPEL_REQUIRE_DASHBOARD_ASSETS is set and this binary has no embedded dashboard; \ + run `npm --prefix frontend ci && npm --prefix frontend run build` before `cargo test`" + ); + } +} + +#[tokio::test] +async fn the_root_serves_the_page_with_the_injected_configuration() { + if !built() { + return; + } + let harness = Harness::new(); + let reply = Call::get("/").send(harness.router()).await; + + assert_eq!(reply.status, 200); + assert_eq!( + reply.content_type.as_deref(), + Some("text/html; charset=utf-8") + ); + // Never cached: the document carries the configuration, so a cached copy + // would keep showing values from before the last reload. + assert_eq!(reply.header("cache-control"), Some("no-store")); + + let config = injected(&reply.body); + assert_eq!(config["title"], "Doppel"); + // Nothing named this one, so the page is free to draw its wordmark. + assert_eq!(config["titleIsDefault"], true); + assert_eq!(config["public"], false); + assert_eq!(config["version"], env!("CARGO_PKG_VERSION")); + assert_eq!(config["authHeader"], "X-Proxy-Authorization"); + assert_eq!(config["refreshMs"], 60_000); + // The footer's copyright year, from the build stamp rather than the browser's + // clock. Asserted as a range because the value moves with every build. + let year = config["copyrightYear"] + .as_u64() + .expect("copyrightYear is a number"); + assert!((2026..=9999).contains(&year), "copyrightYear is {year}"); +} + +#[tokio::test] +async fn the_page_reports_the_configured_title_and_auth_header() { + if !built() { + return; + } + let config = BASE_CONFIG + .replacen(" tokens:", " title: \"Doppel (staging)\"\n tokens:", 1) + .replacen( + " access:", + " auth:\n header: X-Admin-Token\n access:", + 1, + ); + let harness = Harness::with_config(&config); + let body = Call::get("/").send(harness.router()).await.body; + + let injected = injected(&body); + assert_eq!(injected["title"], "Doppel (staging)"); + // Named, so the page shows the name rather than the wordmark. + assert_eq!(injected["titleIsDefault"], false); + assert_eq!(injected["authHeader"], "X-Admin-Token"); +} + +#[tokio::test] +async fn a_public_configuration_is_reported_to_the_page() { + if !built() { + return; + } + // The page uses this to decide whether to ask for a token at all, so getting + // it wrong means a public deployment showing a pointless dialog. + let config = BASE_CONFIG.replacen(" tokens:", " public: true\n tokens:", 1); + let harness = Harness::with_config(&config); + let body = Call::get("/").send(harness.router()).await.body; + + assert_eq!(injected(&body)["public"], true); +} + +#[tokio::test] +async fn a_hostile_title_cannot_close_the_script_element() { + if !built() { + return; + } + // `AdminTitle` accepts markup on purpose -- it is legal text -- so this is + // the place the escaping has to hold. Without it the title would end the + // JSON element early and the rest would be parsed as HTML, which is a + // scripting hole reachable by editing one configuration field. + let hostile = ""; + let config = BASE_CONFIG.replacen( + " tokens:", + &format!(" title: \"{hostile}\"\n tokens:"), + 1, + ); + let harness = Harness::with_config(&config); + let body = Call::get("/").send(harness.router()).await.body; + + // The title survives intact where it belongs... + assert_eq!(injected(&body)["title"], hostile); + // ...and the page contains exactly the script elements it is supposed to: + // the config block and the module. An injected one would make three. + assert_eq!( + body.matches(" String { + BASE_CONFIG.replacen(" access:\n list: public\n read: public\n", block, 1) +} + +#[tokio::test] +async fn an_anonymous_caller_gets_the_public_actions_and_nothing_else() { + // `BASE_CONFIG` makes `list` and `read` public and the four writes + // admin-only, which is the shape an operator most often has. + let harness = Harness::new(); + let body = Call::get("/api/v1/access") + .send(harness.router()) + .await + .json(); + + assert_eq!(body["caller"]["kind"], "anonymous"); + assert_eq!(body["global"]["list"], true); + assert_eq!(body["global"]["read"], true); + for action in ["create", "update", "delete", "upload"] { + assert_eq!(body["global"][action], false, "{action} must be refused"); + } +} + +#[tokio::test] +async fn a_token_is_reported_by_its_own_name_and_group() { + let harness = Harness::new(); + let body = Call::get("/api/v1/access") + .token(READER) + .send(harness.router()) + .await + .json(); + + assert_eq!(body["caller"]["kind"], "token"); + assert_eq!(body["caller"]["name"], "reader"); + assert_eq!(body["caller"]["group"], "user"); + // `reader` is in group `user`, and the writes want `admin`. + assert_eq!(body["global"]["read"], true); + assert_eq!(body["global"]["create"], false); +} + +#[tokio::test] +async fn an_admin_token_may_do_everything() { + let harness = Harness::new(); + let body = Call::get("/api/v1/access") + .token(ROOT) + .send(harness.router()) + .await + .json(); + + for action in ["list", "read", "create", "update", "delete", "upload"] { + assert_eq!(body["global"][action], true, "{action} must be allowed"); + } + assert_eq!(body["proxies"]["alpha"]["delete"], true); +} + +#[tokio::test] +async fn the_proxy_map_names_every_proxy_for_a_caller_who_may_list() { + let harness = Harness::new(); + let body = Call::get("/api/v1/access") + .send(harness.router()) + .await + .json(); + + let proxies = body["proxies"].as_object().expect("a proxy map"); + let mut names: Vec<_> = proxies.keys().map(String::as_str).collect(); + names.sort_unstable(); + assert_eq!(names, ["alpha", "beta"]); + assert_eq!(body["proxies"]["alpha"]["read"], true); + assert_eq!(body["proxies"]["alpha"]["update"], false); +} + +#[tokio::test] +async fn a_caller_who_may_not_list_gets_no_proxy_map_at_all() { + // Keyed by proxy name, the map is a proxy listing by another route. An empty + // object would still say "there are no proxies"; the key has to be gone. + let config = with_access(" access:\n list: [\"admin\"]\n read: [\"admin\"]\n"); + let harness = Harness::with_config(&config); + let body = Call::get("/api/v1/access") + .send(harness.router()) + .await + .json(); + + assert_eq!(body["global"]["list"], false); + assert!( + body.get("proxies").is_none(), + "a caller without `list` must not learn the proxy names: {body}" + ); +} + +#[tokio::test] +async fn a_proxy_override_is_reported_instead_of_the_global_answer() { + // The whole reason the map exists: `reader` may read `alpha` and no other + // proxy, and a report built from the global block alone would say `false` + // for both. + let config = with_access(" access:\n list: public\n read: [\"admin\"]\n").replacen( + " - name: alpha\n type: http\n url: \"https://alpha.example.com/api/\"\n", + " - name: alpha\n type: http\n url: \"https://alpha.example.com/api/\"\n \ + access:\n read: [\"reader\"]\n", + 1, + ); + let harness = Harness::with_config(&config); + let body = Call::get("/api/v1/access") + .token(READER) + .send(harness.router()) + .await + .json(); + + assert_eq!(body["global"]["read"], false); + assert_eq!(body["proxies"]["alpha"]["read"], true); + assert_eq!(body["proxies"]["beta"]["read"], false); +} + +#[tokio::test] +async fn a_public_configuration_reports_everything_true_for_anonymous() { + let config = BASE_CONFIG.replacen(" tokens:", " public: true\n tokens:", 1); + let harness = Harness::with_config(&config); + let body = Call::get("/api/v1/access") + .send(harness.router()) + .await + .json(); + + for action in ["list", "read", "create", "update", "delete", "upload"] { + assert_eq!(body["global"][action], true, "{action} under public: true"); + } +} + +#[tokio::test] +async fn the_report_agrees_with_what_the_request_actually_answers() { + // The test that matters. The report is `authorize` evaluated, and this is + // what catches it drifting from `authorize` as enforced: a reported `false` + // has to come back 401 or 403, and a reported `true` must not. + // + // A harness per action, because `delete` really deletes and `create` really + // creates -- sharing one would make the later actions depend on the earlier. + for token in [None, Some(READER), Some(ROOT)] { + let report = { + let harness = Harness::new(); + let mut call = Call::get("/api/v1/access"); + if let Some(token) = token { + call = call.token(token); + } + call.send(harness.router()).await.json() + }; + + for action in ["list", "read", "create", "update", "delete", "upload"] { + let harness = Harness::new(); + // The revision `update` needs, read before the attempt so a refusal + // is about access rather than about a missing `If-Match`. + let revision = Call::get("/api/v1/proxies/alpha") + .token(ROOT) + .send(harness.router()) + .await + .revision(); + + let mut call = match action { + "list" => Call::get("/api/v1/proxies"), + "read" => Call::get("/api/v1/proxies/alpha"), + "create" => Call::post("/api/v1/proxies") + .json(proxy_json("gamma", "https://gamma.example.com/")), + "update" => Call::put("/api/v1/proxies/alpha") + .if_match(&revision) + .json(proxy_json("alpha", "https://moved.example.com/")), + "delete" => Call::delete("/api/v1/proxies/alpha").if_match(&revision), + "upload" => Call::post("/api/v1/proxies/alpha/templates/page.json.j2").raw("hello"), + other => unreachable!("{other}"), + }; + if let Some(token) = token { + call = call.token(token); + } + let status = call.send(harness.router()).await.status; + + let permitted = report["global"][action] + .as_bool() + .unwrap_or_else(|| panic!("{action} is not a boolean in {report}")); + let refused = status == 401 || status == 403; + assert_eq!( + permitted, !refused, + "{action} as {token:?}: the report said {permitted} and the request answered \ + {status}" + ); + } + } +} diff --git a/crates/doppel-admin/tests/schema.rs b/crates/doppel-admin/tests/schema.rs new file mode 100644 index 0000000..5289792 --- /dev/null +++ b/crates/doppel-admin/tests/schema.rs @@ -0,0 +1,62 @@ +//! `GET /api/v1/schema`: the configuration schema, from the running process. + +mod common; + +use common::{Call, Harness}; + +#[tokio::test] +async fn the_schema_is_served_to_anybody_who_asks() { + // No token. It describes the shape of a configuration, never the contents of + // one, and the identical bytes are on GitHub -- so a right to hold here would + // guard nothing while stopping the page from checking a field before sign-in. + let harness = Harness::new(); + let reply = Call::get("/api/v1/schema").send(harness.router()).await; + + assert_eq!(reply.status, 200); + assert_eq!( + reply.content_type.as_deref(), + Some("application/schema+json") + ); + + let schema: serde_json::Value = serde_json::from_str(&reply.body).expect("the body is JSON"); + assert_eq!( + schema["$schema"], "https://json-schema.org/draft/2020-12/schema", + "not a 2020-12 document" + ); + // The three the dashboard reaches for: a pattern to check a name against, a + // range to bound a ratio, and the proxy document it validates in the YAML + // editor. + assert_eq!( + schema["$defs"]["ProxyName"]["pattern"], + "^[A-Za-z0-9_-]{2,32}$" + ); + assert_eq!(schema["$defs"]["Ratio"]["maximum"], 1.0); + assert!(schema["$defs"]["ProxyConfig"]["properties"]["mocks"].is_object()); +} + +#[tokio::test] +async fn the_schema_is_the_document_the_repository_keeps() { + // The file is what editors fetch and what a release attaches; this endpoint is + // the same document from the process that will actually read the configuration. + // Two answers to one question is the failure worth catching. + let harness = Harness::new(); + let reply = Call::get("/api/v1/schema").send(harness.router()).await; + + let served: serde_json::Value = serde_json::from_str(&reply.body).expect("the body is JSON"); + let on_disk: serde_json::Value = + serde_json::from_str(include_str!("../../../doppel-config.schema.json")) + .expect("the checked-in schema is JSON"); + assert_eq!(served, on_disk); +} + +#[tokio::test] +async fn a_disabled_dashboard_still_serves_the_schema() { + // It is an API endpoint, not part of the page: a deployment with + // `dashboard: false` still has clients that validate a document before pushing + // it. + let config = common::BASE_CONFIG.replacen(" tokens:", " dashboard: false\n tokens:", 1); + let harness = Harness::with_config(&config); + let reply = Call::get("/api/v1/schema").send(harness.router()).await; + + assert_eq!(reply.status, 200); +} diff --git a/crates/doppel-cli/src/commands/serve.rs b/crates/doppel-cli/src/commands/serve.rs index b9c6bc8..1d6d2e8 100644 --- a/crates/doppel-cli/src/commands/serve.rs +++ b/crates/doppel-cli/src/commands/serve.rs @@ -77,6 +77,33 @@ pub async fn serve(store: Arc, config: Config) -> Result<(), Cl } } + // Where clients reach this Doppel, for rewriting an upstream's redirects. + // The variable wins over the document, and a malformed one fails startup for + // the reason a malformed token variable does: an operator who set it and got + // no error believes redirects name that address. + let external_url = match doppel_core::config::external_url_from_env() + .map_err(|err| CliError::Failed(err.to_string()))? + { + Some(from_env) => { + tracing::info!(url = from_env.as_str(), "external url from the environment"); + Some(from_env) + } + None => { + let derived = config.server.public_url(); + if let Some(url) = &derived { + // Worth a line, because it is a guess: behind a port mapping or + // an ingress the client used neither this address nor this port, + // and this is where an operator sees which address their + // rewritten redirects will name. + tracing::info!( + url = url.as_str(), + "external url from server.host and server.port" + ); + } + derived + } + }; + // After logging is up, so these reach wherever the operator is looking, // and only here: `doppel config validate` runs in CI loops, and a warning // repeated on every run is a warning nobody reads. Startup is also the @@ -112,6 +139,21 @@ pub async fn serve(store: Arc, config: Config) -> Result<(), Cl } else { None }; + + // The series that must be there before anything happens. + // + // A dashboard panel and an alert both read "no data" for a metric that has + // never been recorded, and "no data" is indistinguishable from "this process + // is not being scraped". So the facts that are true at startup are published + // at startup: what this binary is, whether it serves the dashboard, and that + // no proxy error has happened yet. The per-proxy mock counts come from the + // runtime below, on every swap, so they follow the configuration in force. + if admin_enabled { + doppel_core::metrics::describe_build(env!("CARGO_PKG_VERSION")); + doppel_core::metrics::describe_dashboard(config.admin.is_dashboard_enabled()); + doppel_core::metrics::init_proxy_errors(); + } + let started_at = std::time::Instant::now(); let addr = SocketAddr::new(config.server.host, config.server.port.get()); @@ -220,7 +262,8 @@ pub async fn serve(store: Arc, config: Config) -> Result<(), Cl }); let proxy_task = tokio::spawn(serve_proxy( - ProxyState::new(Arc::clone(&holder)), + ProxyState::new(Arc::clone(&holder)) + .with_external_url(external_url.map(doppel_core::config::ExternalUrl::into_url)), listener, async { let _ = proxy_rx.await; diff --git a/crates/doppel-cli/src/control/server.rs b/crates/doppel-cli/src/control/server.rs index 59c2f2d..8e9641c 100644 --- a/crates/doppel-cli/src/control/server.rs +++ b/crates/doppel-cli/src/control/server.rs @@ -601,6 +601,24 @@ proxies: Box::pin(async { unimplemented!("not exercised by this test") }) } + fn rename_templates<'s, 'p1, 'p2, 'out>( + &'s self, + _from: &'p1 str, + _to: &'p2 str, + ) -> std::pin::Pin< + Box< + dyn std::future::Future> + Send + 'out, + >, + > + where + 's: 'out, + 'p1: 'out, + 'p2: 'out, + Self: 'out, + { + Box::pin(async { unimplemented!("not exercised by this test") }) + } + fn retain_templates<'s, 'p1, 'p2, 'out>( &'s self, _proxy: &'p1 str, diff --git a/crates/doppel-cli/tests/admin.rs b/crates/doppel-cli/tests/admin.rs index f51ff9a..4262f49 100644 --- a/crates/doppel-cli/tests/admin.rs +++ b/crates/doppel-cli/tests/admin.rs @@ -70,6 +70,10 @@ impl Admin { fn anonymous_get(&self, path: &str) -> (u16, String) { self.request("GET", path, None, None) } + + fn put(&self, path: &str, body: &str) -> (u16, String) { + self.request("PUT", path, Some(SECRET_TOKEN), Some(body)) + } } fn start() -> (Server, Admin, common::Upstream) { @@ -100,7 +104,7 @@ fn status_and_metrics_are_scrapeable_without_a_token() { let (server, admin, _up) = start(); server.get("/anything"); - let (status, body) = admin.anonymous_get("/status"); + let (status, body) = admin.anonymous_get("/api/v1/status"); assert_eq!(status, 200, "{body}"); assert!(body.contains("\"revision\""), "{body}"); assert!(body.contains("\"p1\""), "{body}"); @@ -133,8 +137,15 @@ fn a_write_without_a_token_is_refused() { assert!(body.contains("UNAUTHORIZED"), "{body}"); } +/// A proxy created over HTTP serves traffic straight away. +/// +/// It used to take a reload, and this test used to say so -- "written to the +/// store, but not yet running". That was the defect an operator reported from the +/// other end: a write returns 200, the API reads the new document back, and the +/// traffic keeps going where it went before, with nothing anywhere saying the two +/// disagree. #[test] -fn a_proxy_created_over_http_serves_traffic_after_a_reload() { +fn a_proxy_created_over_http_serves_traffic_without_a_reload() { let (server, admin, up) = start(); // `p1` is the default resolver, so the new one resolves by header. @@ -151,18 +162,18 @@ fn a_proxy_created_over_http_serves_traffic_after_a_reload() { ); assert_eq!(status, 201, "{body}"); - // Written to the store, but not yet running. - let (_, before) = admin.anonymous_get("/status"); - assert!(!before.contains("\"p2\""), "{before}"); + // In the running configuration, with no reload in between. + let (_, running) = admin.anonymous_get("/api/v1/status"); + assert!(running.contains("\"p2\""), "{running}"); + + // And actually serving: the header picks it. + let (status, body) = server.get_with_header("/anything", "X-Proxy-Name", "p2"); + assert_eq!(status, 200, "{body}"); + // A reload afterwards is the idempotent case -- same store, same revision. let (status, body) = admin.request("POST", "/api/v1/config/reload", Some(SECRET_TOKEN), None); assert_eq!(status, 200, "{body}"); assert!(body.contains("\"proxies\":2"), "{body}"); - - let (_, after) = admin.anonymous_get("/status"); - assert!(after.contains("\"p2\""), "{after}"); - - // And it actually serves: the header picks it. let (status, body) = server.get_with_header("/anything", "X-Proxy-Name", "p2"); assert_eq!(status, 200, "{body}"); } @@ -613,3 +624,63 @@ fn a_name_the_environment_claims_cannot_be_issued_over_the_socket() { assert!(stdout.contains("supplied by the environment"), "{stdout}"); assert!(stdout.contains("would never authenticate"), "{stdout}"); } + +/// The case an operator reported: an upstream changed through the admin API, and +/// the proxy kept forwarding to the old one. +/// +/// The whole chain through the built binary, both listeners and two upstreams, so +/// "in force" means what a client sees rather than what an endpoint reports. The +/// unit tests can only check the runtime's own view of itself. +#[test] +fn an_upstream_changed_through_the_api_is_what_the_proxy_forwards_to() { + let first = upstream(); + let second = upstream(); + let server = Server::start(first.port); + let admin = Admin { + port: server.admin_port(), + }; + + // Where it goes now. + let (status, body) = server.get("/thing"); + assert_eq!(status, 200, "{body}"); + assert!(body.contains("upstream saw /thing"), "{body}"); + + // The revision the update has to carry, from the proxy as stored. + let (status, read) = admin.get("/api/v1/proxies/p1"); + assert_eq!(status, 200, "{read}"); + let revision = serde_json::from_str::(&read).unwrap()["revision"] + .as_str() + .expect("a revision") + .to_owned(); + + let (status, body) = admin.put( + "/api/v1/proxies/p1", + &format!( + r#"{{"revision":"{revision}","proxy":{{"name":"p1","type":"http","url":"http://127.0.0.1:{}/"}}}}"#, + second.port + ), + ); + assert_eq!(status, 200, "{body}"); + + // No reload in between. The proxy forwards to the new upstream, which is the + // only observation that distinguishes a write that landed in the store from + // one that is in force. + let (status, body) = server.get("/thing"); + assert_eq!(status, 200, "{body}"); + assert!( + body.contains("upstream saw /thing"), + "the second upstream answers the same shape: {body}" + ); + + // And the running configuration says so too. + let (status, status_body) = admin.anonymous_get("/api/v1/status"); + assert_eq!(status, 200, "{status_body}"); + assert!( + status_body.contains(&format!("127.0.0.1:{}", second.port)), + "status must name the new upstream: {status_body}" + ); + assert!( + !status_body.contains(&format!("127.0.0.1:{}", first.port)), + "status must not still name the old one: {status_body}" + ); +} diff --git a/crates/doppel-cli/tests/common/mod.rs b/crates/doppel-cli/tests/common/mod.rs index 43478f8..4ccb9ac 100644 --- a/crates/doppel-cli/tests/common/mod.rs +++ b/crates/doppel-cli/tests/common/mod.rs @@ -50,8 +50,24 @@ use std::time::{Duration, Instant}; /// something else, and reporting that beats retrying forever. const PORT_RACE_ATTEMPTS: usize = 3; +/// A port nothing is listening on, by asking the kernel for one and giving it +/// straight back. +/// +/// Bound on the **wildcard**, not on loopback, and that is the whole point. +/// `SO_REUSEADDR` is set by `std` on every `TcpListener`, and macOS then allows +/// `127.0.0.1:P` to be bound while another process holds `0.0.0.0:P` -- the more +/// specific address wins for connections to it. A loopback probe could therefore +/// draw a port a server under test was already serving on, shadow it for the +/// microseconds before this listener is dropped, and reset whatever connection +/// arrived in that window: `ECONNRESET` on the client, no request in the server's +/// log, the server still alive and listening. That was the `mocks` suite's flake, +/// which uses `main.example.yaml` and its `host: 0.0.0.0`. +/// +/// A wildcard probe cannot: the kernel refuses a second wildcard bind on a busy +/// port with `EADDRINUSE`, so a collision becomes the lost-port race that +/// `start_with_env` already retries instead of a corrupted request. pub fn free_port() -> u16 { - TcpListener::bind("127.0.0.1:0") + TcpListener::bind("0.0.0.0:0") .unwrap() .local_addr() .unwrap() @@ -77,11 +93,34 @@ pub fn upstream() -> Upstream { let Ok(mut stream) = stream else { continue }; let request = read_request_head(&mut stream); let path = request.split_whitespace().nth(1).unwrap_or("/").to_owned(); - let body = format!("upstream saw {path}"); - let response = format!( - "HTTP/1.1 200 OK\r\ncontent-length: {}\r\nconnection: close\r\n\r\n{body}", - body.len() - ); + // One route that is not an echo: a redirect to the upstream's own + // authority, which is what makes a client leave Doppel unless the + // `Location` is rewritten. Real services answer these constantly -- + // a trailing slash, a canonical host, a login page. + let response = if path == "/page" { + // A page that names its own host, which is what makes a client + // leave Doppel on the next click unless the body is rewritten. + let body = format!( + "next\ + " + ); + format!( + "HTTP/1.1 200 OK\r\ncontent-type: text/html\r\netag: \"upstream-etag\"\r\n\ + content-length: {}\r\nconnection: close\r\n\r\n{body}", + body.len() + ) + } else if path == "/redirect-self" { + format!( + "HTTP/1.1 302 Found\r\nlocation: http://127.0.0.1:{port}/moved\r\n\ + content-length: 0\r\nconnection: close\r\n\r\n" + ) + } else { + let body = format!("upstream saw {path}"); + format!( + "HTTP/1.1 200 OK\r\ncontent-length: {}\r\nconnection: close\r\n\r\n{body}", + body.len() + ) + }; let _ = stream.write_all(response.as_bytes()); } }); @@ -351,6 +390,27 @@ impl Server { (status, response.text().unwrap()) } + /// A GET that does not follow a redirect, and reports the `Location`. + /// + /// `reqwest::blocking::get` follows up to ten by default, which would send + /// the test wherever the header points and hide the header itself -- the + /// thing under test. + pub fn get_unfollowed(&self, path: &str) -> (u16, Option) { + let url = format!("http://127.0.0.1:{}{path}", self.port); + let response = reqwest::blocking::Client::builder() + .redirect(reqwest::redirect::Policy::none()) + .build() + .unwrap() + .get(url) + .send() + .unwrap(); + let location = response + .headers() + .get("location") + .map(|value| value.to_str().unwrap().to_owned()); + (response.status().as_u16(), location) + } + /// A GET carrying one header, for the suites that resolve a proxy by /// header rather than by the default. pub fn get_with_header(&self, path: &str, name: &str, value: &str) -> (u16, String) { diff --git a/crates/doppel-cli/tests/proxying.rs b/crates/doppel-cli/tests/proxying.rs index c2cbab0..74a4e92 100644 --- a/crates/doppel-cli/tests/proxying.rs +++ b/crates/doppel-cli/tests/proxying.rs @@ -12,3 +12,76 @@ fn proxies_a_request_end_to_end() { assert_eq!(status, 200); assert_eq!(body, "upstream saw /hello"); } + +/// The whole chain, through the built binary: an upstream redirect to its own +/// authority comes back naming Doppel. +/// +/// The unit tests pin the rewriting rules; this pins the wiring, which is what +/// was missing when an operator reported a redirect still pointing at the +/// upstream. `server.external_url` is unset in this configuration, so the +/// address comes from `server.host` and `server.port` -- the fallback that makes +/// the default deployment work without configuring anything. +#[test] +fn a_redirect_to_the_upstreams_own_host_comes_back_naming_doppel() { + let up = upstream(); + let server = Server::start(up.port); + + let (status, location) = server.get_unfollowed("/redirect-self"); + + assert_eq!(status, 302); + assert_eq!( + location.as_deref(), + Some(format!("http://127.0.0.1:{}/moved", server.port()).as_str()), + "the Location must name Doppel, not the upstream" + ); +} + +/// And `DOPPEL_EXTERNAL_URL` over the top of it, which is the answer for a +/// deployment behind a port mapping or an ingress: the address Doppel bound is +/// not the address the client used. +#[test] +fn the_environment_overrides_which_host_a_rewritten_redirect_names() { + let up = upstream(); + let server = Server::start_with_env( + up.port, + common::config, + &[("DOPPEL_EXTERNAL_URL", "https://doppel.example.com/")], + ); + + let (status, location) = server.get_unfollowed("/redirect-self"); + + assert_eq!(status, 302); + assert_eq!( + location.as_deref(), + Some("https://doppel.example.com/moved") + ); +} + +/// A body that names the upstream comes back naming Doppel. +/// +/// Through the built binary, both listeners, so this covers the wiring as well as +/// the rules: `rewrite_urls` defaults to on, and the address comes from +/// `server.host` and `server.port` like a rewritten redirect's does. +#[test] +fn a_body_that_names_the_upstream_comes_back_naming_doppel() { + let up = upstream(); + let server = Server::start(up.port); + + let (status, body) = server.get("/page"); + + assert_eq!(status, 200, "{body}"); + assert!( + body.contains(&format!("http://127.0.0.1:{}/next", server.port())), + "the link must point at Doppel: {body}" + ); + assert!( + !body.contains(&format!("http://127.0.0.1:{}/next", up.port)), + "and not at the upstream: {body}" + ); + // A host that merely contains the upstream's is a different host, and is left + // exactly as the upstream wrote it. + assert!( + body.contains(&format!("http://cdn.127.0.0.1:{}/logo.png", up.port)), + "a different host must survive: {body}" + ); +} diff --git a/crates/doppel-core/src/config/admin.rs b/crates/doppel-core/src/config/admin.rs index 3b75940..647f8b3 100644 --- a/crates/doppel-core/src/config/admin.rs +++ b/crates/doppel-core/src/config/admin.rs @@ -174,6 +174,27 @@ pub struct AdminConfig { pub access: AccessConfig, /// Bounds an uploaded template file. pub upload: UploadConfig, + /// Serve the browser dashboard from the admin listener's root. + /// + /// On by default. Off means `/`, `/static/*` and `/robots.txt` are not + /// routed at all and answer 404 like any other unknown path; the JSON API is + /// untouched either way. + /// + /// An `Option` and skipped when absent, for the reason given on `groups`: + /// adding it must not change the canonical YAML of configurations written + /// before it existed. Read it through + /// [`AdminConfig::is_dashboard_enabled`]. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub dashboard: Option, + /// The heading the dashboard shows, and the browser tab's title. + /// + /// Defaults to `Doppel`. Useful when several Doppels are open at once and + /// the tabs are otherwise identical. + /// + /// An `Option` and skipped when absent, for the same reason as `dashboard`. + /// Read it through [`AdminConfig::title`]. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub title: Option, } impl AdminConfig { @@ -203,6 +224,24 @@ impl AdminConfig { self.public.unwrap_or(false) || self.groups.as_deref().is_some_and(<[_]>::is_empty) } + /// Whether the browser dashboard is served. + /// + /// One place resolves the absent case, so no caller has to remember which + /// way an unwritten field falls -- and the default is on, so an existing + /// configuration gains the dashboard on upgrade without being edited. + #[must_use] + pub fn is_dashboard_enabled(&self) -> bool { + self.dashboard.unwrap_or(true) + } + + /// The dashboard's heading, with an absent `title` resolved to its default. + #[must_use] + pub fn title(&self) -> &str { + self.title + .as_ref() + .map_or(super::title::DEFAULT, super::AdminTitle::as_str) + } + /// What `access` amounts to once `is_public` is taken into account. /// /// `None` when the configuration is public, which every caller reads as diff --git a/crates/doppel-core/src/config/duration.rs b/crates/doppel-core/src/config/duration.rs index 61e581d..0cd745a 100644 --- a/crates/doppel-core/src/config/duration.rs +++ b/crates/doppel-core/src/config/duration.rs @@ -134,6 +134,7 @@ impl utoipa::PartialSchema for Seconds { .minimum(Some(0.0)) .maximum(Some(MAX_LATENCY_SECONDS)) .description(Some("A latency in seconds, 0 to 300.")) + .examples([serde_json::json!(0.25)]) .into() } } @@ -247,6 +248,7 @@ impl utoipa::PartialSchema for TimeoutSeconds { .minimum(Some(1.0)) .maximum(Some(MAX_TIMEOUT_SECONDS as f64)) .description(Some("An upstream timeout in whole seconds, 1 to 3600.")) + .examples([serde_json::json!(30)]) .into() } } diff --git a/crates/doppel-core/src/config/env.rs b/crates/doppel-core/src/config/env.rs index 6d1c115..8be9e2c 100644 --- a/crates/doppel-core/src/config/env.rs +++ b/crates/doppel-core/src/config/env.rs @@ -1,4 +1,5 @@ -//! Admin tokens supplied by the environment. +//! Configuration supplied by the environment: the admin tokens, and the +//! external url. //! //! A deployment that provisions its secrets through the environment should //! not have to write them into the configuration document to use them. These @@ -16,11 +17,46 @@ use std::collections::BTreeMap; use serde::Deserialize; -use super::{Name, Token, TokenConfig}; +use super::{ExternalUrl, Name, Token, TokenConfig, UrlError}; /// The variable read at startup. pub const VAR: &str = "DOPPEL_ADMIN_TOKENS"; +/// The variable that overrides `server.external_url`. +pub const EXTERNAL_URL_VAR: &str = "DOPPEL_EXTERNAL_URL"; + +/// Why `DOPPEL_EXTERNAL_URL` was refused. +/// +/// Fails startup rather than being logged and skipped, for the reason the token +/// errors do: an operator who set it and got no error believes redirects are +/// being rewritten to that address. +#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)] +#[error("{EXTERNAL_URL_VAR} is not usable: {reason}")] +pub struct EnvExternalUrlError { + pub reason: UrlError, +} + +/// `DOPPEL_EXTERNAL_URL`, or `None` when it is unset or empty. +/// +/// Empty counts as unset, like the token variable: `DOPPEL_EXTERNAL_URL=${HOST}` +/// with nothing behind `HOST` is a compose file that means nothing by it, not a +/// deployment to refuse to start. +/// +/// Not merged into `Config`, and for the same reason the tokens are not: the +/// revision is computed over the document, so folding an environment value into +/// it would make two instances reading one stored document disagree about the +/// revision, and every compare-and-swap between them fail for a difference +/// neither wrote. +pub fn external_url_from_env() -> Result, EnvExternalUrlError> { + match std::env::var(EXTERNAL_URL_VAR) { + Ok(raw) if raw.trim().is_empty() => Ok(None), + Ok(raw) => ExternalUrl::parse(raw.trim()) + .map(Some) + .map_err(|reason| EnvExternalUrlError { reason }), + Err(_) => Ok(None), + } +} + /// Tokens provided by the environment, in the order they were written. #[derive(Debug, Clone, Default, PartialEq, Eq)] pub struct EnvTokens(Vec); diff --git a/crates/doppel-core/src/config/header.rs b/crates/doppel-core/src/config/header.rs index 78c48d7..36c108b 100644 --- a/crates/doppel-core/src/config/header.rs +++ b/crates/doppel-core/src/config/header.rs @@ -130,7 +130,7 @@ impl HeaderValue { } macro_rules! string_newtype_impls { - ($type:ident, $error:ident, $description:expr) => { + ($type:ident, $error:ident, $description:expr, $example:expr) => { impl fmt::Display for $type { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { f.write_str(&self.0) @@ -188,6 +188,7 @@ macro_rules! string_newtype_impls { utoipa::openapi::schema::ObjectBuilder::new() .schema_type(utoipa::openapi::schema::Type::String) .description(Some($description)) + .examples([serde_json::json!($example)]) .into() } } @@ -199,12 +200,14 @@ macro_rules! string_newtype_impls { string_newtype_impls!( HeaderName, HeaderNameError, - "An HTTP header name: a non-empty RFC 9110 token." + "An HTTP header name: a non-empty RFC 9110 token.", + "X-Request-Id" ); string_newtype_impls!( HeaderValue, HeaderValueError, - "An HTTP header value: visible ASCII, space and tab, with no line breaks." + "An HTTP header value: visible ASCII, space and tab, with no line breaks.", + "staging" ); #[cfg(test)] diff --git a/crates/doppel-core/src/config/mock.rs b/crates/doppel-core/src/config/mock.rs index 224ee29..db53e5d 100644 --- a/crates/doppel-core/src/config/mock.rs +++ b/crates/doppel-core/src/config/mock.rs @@ -47,10 +47,12 @@ pub struct MockResponse { pub status: crate::config::HttpStatus, /// A template rendered and sent as `text/plain`. Exclusive with `json` and /// `template`. + #[schema(examples("widget {{ id }} is ready"))] #[serde(default, skip_serializing_if = "Option::is_none")] pub body: Option, /// A template whose rendered output must be valid JSON, sent as /// `application/json`. Exclusive with `body` and `template`. + #[schema(examples("{\"id\": \"{{ id }}\"}"))] #[serde(default, skip_serializing_if = "Option::is_none")] pub json: Option, /// A template file under this proxy's template directory. Read per request, diff --git a/crates/doppel-core/src/config/mod.rs b/crates/doppel-core/src/config/mod.rs index d37f5af..d49e735 100644 --- a/crates/doppel-core/src/config/mod.rs +++ b/crates/doppel-core/src/config/mod.rs @@ -16,6 +16,7 @@ pub mod server; pub mod size; pub mod status; pub mod template; +pub mod title; pub mod token; pub mod url; @@ -29,7 +30,7 @@ pub use admin::{ UploadConfig, }; pub use duration::{Seconds, SecondsError, TimeoutError, TimeoutSeconds}; -pub use env::{EnvTokens, EnvTokensError}; +pub use env::{EnvExternalUrlError, EnvTokens, EnvTokensError, external_url_from_env}; pub use header::{HeaderName, HeaderNameError, HeaderValue, HeaderValueError}; pub use mock::{MockConfig, MockProxyOverride, MockRequest, MockResponse}; pub use name::{MAX_PROXY, Name, NameError, ProxyName}; @@ -44,8 +45,9 @@ pub use server::{ pub use size::{ByteSize, ByteSizeError}; pub use status::{HttpStatus, StatusError}; pub use template::{TemplateName, TemplateNameError}; +pub use title::{AdminTitle, AdminTitleError}; pub use token::{Token, TokenError}; -pub use url::{UpstreamUrl, UrlError}; +pub use url::{ExternalUrl, UpstreamUrl, UrlError}; #[derive(Debug, Clone, PartialEq, Serialize, Deserialize, utoipa::ToSchema)] #[serde(deny_unknown_fields)] @@ -198,6 +200,47 @@ proxies: assert!(!load_from_str(&off).unwrap().admin.enable); } + #[test] + fn the_dashboard_is_on_and_titled_doppel_unless_told_otherwise() { + // Both defaults are what an existing configuration gets on upgrade + // without being edited, so they are the interesting half of the + // feature rather than a formality. + let default = load_from_str(MINIMAL).unwrap(); + assert!(default.admin.is_dashboard_enabled()); + assert_eq!(default.admin.title(), "Doppel"); + + let written = MINIMAL.replace( + " port: 8081", + " port: 8081\n dashboard: false\n title: \"Doppel (staging)\"", + ); + let written = load_from_str(&written).unwrap(); + assert!(!written.admin.is_dashboard_enabled()); + assert_eq!(written.admin.title(), "Doppel (staging)"); + } + + #[test] + fn adding_the_dashboard_fields_did_not_change_any_stored_revision() { + // The revision is a hash of the canonical YAML, and every stored + // configuration is compare-and-swapped against its own. A default that + // materialised into the document would change the revision of every + // configuration written before these fields existed, and each would fail + // its own revision check on the first load after the upgrade -- which is + // why both fields are `Option` and skipped when absent. + let config = load_from_str(MINIMAL).unwrap(); + let yaml = canonical_yaml(&config).unwrap(); + assert!(!yaml.contains("dashboard"), "{yaml}"); + assert!(!yaml.contains("title"), "{yaml}"); + + // And the pair does serialize once written, or the field would be + // unsettable rather than merely absent. + let mut written = config; + written.admin.dashboard = Some(false); + written.admin.title = Some(AdminTitle::parse("Doppel (staging)").unwrap()); + let yaml = canonical_yaml(&written).unwrap(); + assert!(yaml.contains("dashboard: false"), "{yaml}"); + assert!(yaml.contains("title: Doppel (staging)"), "{yaml}"); + } + #[test] fn server_workers_is_no_longer_a_configuration_field() { // It sizes the tokio runtime, and a database store cannot be opened diff --git a/crates/doppel-core/src/config/name.rs b/crates/doppel-core/src/config/name.rs index 5271f4c..60ee83f 100644 --- a/crates/doppel-core/src/config/name.rs +++ b/crates/doppel-core/src/config/name.rs @@ -205,6 +205,9 @@ impl utoipa::PartialSchema for Name { .description(Some(format!( "Letters, digits, `-` and `_`, between {MIN} and {MAX_LEN} characters." ))) + // Short enough to be legal for a proxy name as well as for the longer + // names that share this builder. + .examples([serde_json::json!("billing-api")]) .into() } } diff --git a/crates/doppel-core/src/config/port.rs b/crates/doppel-core/src/config/port.rs index 7d44309..378333c 100644 --- a/crates/doppel-core/src/config/port.rs +++ b/crates/doppel-core/src/config/port.rs @@ -122,6 +122,7 @@ impl utoipa::PartialSchema for Port { .minimum(Some(1.0)) .maximum(Some(f64::from(u16::MAX))) .description(Some("A TCP port, 1 to 65535.")) + .examples([serde_json::json!(8080)]) .into() } } diff --git a/crates/doppel-core/src/config/proxy.rs b/crates/doppel-core/src/config/proxy.rs index 5f1774e..bee311b 100644 --- a/crates/doppel-core/src/config/proxy.rs +++ b/crates/doppel-core/src/config/proxy.rs @@ -144,6 +144,22 @@ pub struct ProxyConfig { /// *against redirect handling itself* needs. #[serde(default, skip_serializing_if = "Option::is_none")] pub rewrite_redirects: Option, + /// Whether the upstream's own address is rewritten to Doppel's inside the + /// bodies this proxy relays. Absent means enabled. + /// + /// The same problem `rewrite_redirects` solves, one layer down: a page, a + /// script or a JSON document that names `https://api.example.com/v2/` sends + /// the client straight there on the next request, past every fault and every + /// mock. Rewriting the body keeps it inside Doppel. + /// + /// Only the exact host is replaced. `https://cdn.api.example.com/` is a + /// different host and is left alone, because Doppel does not proxy it and + /// pointing it at itself would break a page rather than keep it working. + /// + /// Only text bodies, and only ones the upstream did not compress. Turn it off + /// for a client being tested against the bytes the upstream actually sent. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub rewrite_urls: Option, /// Bounds the request body a matched mock is allowed to buffer in order /// to extract from it; phase 1 streams bodies deliberately, and reading /// `.content.items` needs the whole thing in hand. See rule V33. diff --git a/crates/doppel-core/src/config/ratio.rs b/crates/doppel-core/src/config/ratio.rs index 01910d1..380d08f 100644 --- a/crates/doppel-core/src/config/ratio.rs +++ b/crates/doppel-core/src/config/ratio.rs @@ -106,6 +106,7 @@ impl utoipa::PartialSchema for Ratio { .minimum(Some(0.0)) .maximum(Some(1.0)) .description(Some("A probability from 0.0 to 1.0. 50% is `0.5`.")) + .examples([serde_json::json!(0.25)]) .into() } } diff --git a/crates/doppel-core/src/config/server.rs b/crates/doppel-core/src/config/server.rs index a670a33..11d9e6e 100644 --- a/crates/doppel-core/src/config/server.rs +++ b/crates/doppel-core/src/config/server.rs @@ -22,6 +22,58 @@ pub struct ServerConfig { /// The TCP port proxied traffic arrives on. Must differ from /// `admin.port`. pub port: super::Port, + /// Where clients reach this Doppel, when that is not `host:port`. + /// + /// Behind a container port mapping, a load balancer or an ingress, the + /// address Doppel bound is not the address a client used -- and a rewritten + /// `Location` has to name the second. Doppel cannot infer it: `Host` is a + /// claim by the caller, and building a redirect out of it hands the caller + /// the redirect. + /// + /// Absent, `host` and `port` are used, with a wildcard bind (`0.0.0.0`, `::`) + /// read as loopback -- which is right for a laptop or a pod reached at its + /// own address, and wrong behind a port mapping or an ingress, where the + /// client used neither. Doppel logs the address it settled on at startup. + /// + /// `DOPPEL_EXTERNAL_URL` overrides it. Part of `server`, so a change takes a + /// restart like the rest of that section. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub external_url: Option, +} + +impl ServerConfig { + /// Where a client reaches this Doppel, as far as the configuration knows. + /// + /// `external_url` when it is set, and `http://host:port/` when it is not -- + /// which is right for a Doppel bound to an address a client can dial, and is + /// most of them: a laptop on `127.0.0.1`, a pod on its own address. + /// + /// A wildcard bind (`0.0.0.0`, `::`) becomes loopback: `0.0.0.0` is every + /// address this host has, so it names none of them, and + /// `http://0.0.0.0:8080/` in a `Location` is a URL that fails or means + /// something else depending on the client. `127.0.0.1` is the address that + /// is right for the case a wildcard bind is usually about -- a container or + /// a laptop reached from the same machine. + /// + /// It is a guess, and the one place this can be wrong: a port mapping or an + /// ingress means the client used neither this address nor this port. That + /// deployment says where it is reached with `external_url` or + /// `DOPPEL_EXTERNAL_URL`, which is why both exist. + #[must_use] + pub fn public_url(&self) -> Option { + if let Some(configured) = &self.external_url { + return Some(configured.clone()); + } + // Brackets around an IPv6 literal: `http://::1:8080/` does not parse, + // and the port would read as part of the address. + let host = match self.host { + IpAddr::V4(address) if address.is_unspecified() => "127.0.0.1".to_owned(), + IpAddr::V6(address) if address.is_unspecified() => "[::1]".to_owned(), + IpAddr::V4(address) => address.to_string(), + IpAddr::V6(address) => format!("[{address}]"), + }; + super::ExternalUrl::parse(&format!("http://{host}:{}/", self.port.get())).ok() + } } #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, utoipa::ToSchema)] @@ -91,7 +143,7 @@ pub struct ControlConfig { #[serde(default = "default_socket")] /// A filesystem path. `utoipa` has no schema for `PathBuf`, so it is /// described as the string it is written as. - #[schema(value_type = String)] + #[schema(value_type = String, examples("/tmp/doppel.sock"))] pub socket: PathBuf, } @@ -115,7 +167,7 @@ pub struct TemplatesConfig { #[serde(default = "default_templates_dir")] /// A filesystem path. `utoipa` has no schema for `PathBuf`, so it is /// described as the string it is written as. - #[schema(value_type = String)] + #[schema(value_type = String, examples("./templates"))] pub dir: PathBuf, } @@ -136,5 +188,83 @@ impl Default for TemplatesConfig { pub struct SentryConfig { /// The Sentry DSN to report to. Empty disables reporting, so a deployment /// can blank it without removing the section. + #[schema(examples("https://key@o0.ingest.sentry.io/0"))] pub dsn: String, } + +#[cfg(test)] +mod public_url_tests { + use super::*; + use crate::config::load_from_str; + + fn config_with(server: &str) -> ServerConfig { + let text = format!( + r#" +server: +{server} +admin: + host: "127.0.0.1" + port: 8081 + tokens: [] + access: {{}} + upload: + limit: 1Mi +proxies: + - name: p1 + type: http + url: "https://example.com/" +"# + ); + load_from_str(&text).expect("fixture parses").server + } + + #[test] + fn a_dialable_host_names_itself() { + let server = config_with(" host: \"127.0.0.1\"\n port: 8080"); + assert_eq!( + server.public_url().map(|url| url.as_str().to_owned()), + Some("http://127.0.0.1:8080/".to_owned()) + ); + } + + #[test] + fn an_ipv6_host_is_bracketed() { + // `http://::1:8080/` does not parse, and the port reads as part of the + // address. Without the brackets this is a URL nobody can follow. + let server = config_with(" host: \"::1\"\n port: 8080"); + assert_eq!( + server.public_url().map(|url| url.as_str().to_owned()), + Some("http://[::1]:8080/".to_owned()) + ); + } + + #[test] + fn a_wildcard_bind_becomes_loopback() { + // `0.0.0.0` is every address this host has, so it names none of them and + // cannot go in a `Location`. Loopback is the address that is right for + // what a wildcard bind is usually about, and a deployment reached + // anywhere else says so with `external_url`. + for (host, expected) in [ + ("0.0.0.0", "http://127.0.0.1:8080/"), + ("::", "http://[::1]:8080/"), + ] { + let server = config_with(&format!(" host: \"{host}\"\n port: 8080")); + assert_eq!( + server.public_url().map(|url| url.as_str().to_owned()), + Some(expected.to_owned()), + "{host}" + ); + } + } + + #[test] + fn external_url_wins_over_the_bind_address() { + let server = config_with( + " host: \"127.0.0.1\"\n port: 8080\n external_url: \"https://doppel.example.com/\"", + ); + assert_eq!( + server.public_url().map(|url| url.as_str().to_owned()), + Some("https://doppel.example.com/".to_owned()) + ); + } +} diff --git a/crates/doppel-core/src/config/status.rs b/crates/doppel-core/src/config/status.rs index 1e2d916..b154e78 100644 --- a/crates/doppel-core/src/config/status.rs +++ b/crates/doppel-core/src/config/status.rs @@ -114,6 +114,7 @@ impl utoipa::PartialSchema for HttpStatus { .minimum(Some(f64::from(MIN))) .maximum(Some(f64::from(MAX))) .description(Some("An HTTP status code, 100 to 599.")) + .examples([serde_json::json!(200)]) .into() } } diff --git a/crates/doppel-core/src/config/title.rs b/crates/doppel-core/src/config/title.rs new file mode 100644 index 0000000..042b600 --- /dev/null +++ b/crates/doppel-core/src/config/title.rs @@ -0,0 +1,220 @@ +//! The dashboard's heading, parsed rather than validated. +//! +//! A title is the one configuration value this project renders into a web page, +//! so it is the one value whose bounds are about a browser rather than about a +//! filesystem or a metric label. It is checked while the document is parsed, +//! like every other value, so no unchecked title exists at any later moment. +//! +//! Escaping is *not* this type's job and deliberately so. A title is legal text +//! that happens to be displayed; making the type refuse `<` would forbid a +//! perfectly reasonable heading in order to paper over a rendering bug, and +//! would still leave every other injection sink unprotected. The escaping +//! belongs where the HTML is produced, in `doppel-admin`, and has a test of its +//! own there. + +use std::fmt; +use std::str::FromStr; + +use serde::{Deserialize, Deserializer, Serialize, Serializer}; + +/// The longest title accepted. +/// +/// It sits in a page header next to the theme and sign-in controls, so the bound +/// is about what fits rather than about what a machine can hold: 64 characters +/// is a long service name and a short sentence, and anything past it is being +/// used as a subtitle. +pub const MAX_LEN: usize = 64; + +/// The default heading, used when `admin.title` is absent. +pub const DEFAULT: &str = "Doppel"; + +/// A validated dashboard title. +/// +/// Non-empty, at most [`MAX_LEN`] characters, no control characters. Not +/// restricted to ASCII: this is a UI string, which `AGENTS.md` names as the +/// exception to the repository's ASCII rule, and an operator naming their +/// staging environment in their own language is doing nothing wrong. +#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub struct AdminTitle(String); + +/// Why a title was refused. +/// +/// One variant per rule, so the message says what is wrong rather than restating +/// the rule and leaving the reader to work out which part they broke. +#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)] +pub enum AdminTitleError { + /// Empty and whitespace-only are one case on purpose: both render as a + /// header with nothing in it, and an operator who wrote `title: " "` meant + /// to write something. + #[error("a title must not be empty")] + Empty, + #[error("a title must be at most {MAX_LEN} characters, this one is {0}")] + TooLong(usize), + /// A tab or a newline in a heading is either a copy-paste accident or an + /// attempt to do something to the markup. Neither is worth accepting, and + /// naming the character is what turns this into a fixable report. + #[error("a title must not contain control characters; this one contains {0:?}")] + Control(char), +} + +impl AdminTitle { + /// Check a string and keep it, or say why not. + pub fn parse(value: impl Into) -> Result { + let value = value.into(); + + if value.trim().is_empty() { + return Err(AdminTitleError::Empty); + } + // Counted in characters rather than bytes. A title is displayed, and + // what a reader sees is characters; measuring the bound in bytes would + // make the limit depend on the language the title is written in. + let length = value.chars().count(); + if length > MAX_LEN { + return Err(AdminTitleError::TooLong(length)); + } + // `char::is_control` rather than `is_ascii_control`: the Unicode class + // includes the ASCII one, and a title is not restricted to ASCII, so + // checking only the ASCII range would let a directional override + // through -- a character whose whole purpose is to make displayed text + // disagree with the text that is stored. + if let Some(bad) = value.chars().find(|c| c.is_control()) { + return Err(AdminTitleError::Control(bad)); + } + + Ok(Self(value)) + } + + #[must_use] + pub fn as_str(&self) -> &str { + &self.0 + } +} + +impl fmt::Display for AdminTitle { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + f.write_str(&self.0) + } +} + +impl FromStr for AdminTitle { + type Err = AdminTitleError; + + fn from_str(value: &str) -> Result { + Self::parse(value) + } +} + +impl AsRef for AdminTitle { + fn as_ref(&self) -> &str { + &self.0 + } +} + +impl Serialize for AdminTitle { + fn serialize(&self, s: S) -> Result { + s.serialize_str(&self.0) + } +} + +impl<'de> Deserialize<'de> for AdminTitle { + fn deserialize>(d: D) -> Result { + let value = String::deserialize(d)?; + Self::parse(value).map_err(serde::de::Error::custom) + } +} + +impl utoipa::PartialSchema for AdminTitle { + fn schema() -> utoipa::openapi::RefOr { + utoipa::openapi::schema::ObjectBuilder::new() + .schema_type(utoipa::openapi::schema::Type::String) + .min_length(Some(1)) + .max_length(Some(MAX_LEN)) + .description(Some(format!( + "The heading the dashboard shows, at most {MAX_LEN} characters \ + and free of control characters. Defaults to `{DEFAULT}`." + ))) + .examples([serde_json::json!("billing-api (staging)")]) + .into() + } +} + +impl utoipa::ToSchema for AdminTitle {} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn the_titles_people_actually_write_are_accepted() { + for title in [ + DEFAULT, + "Doppel (staging)", + "billing-api doppelganger", + "a".repeat(MAX_LEN).as_str(), + ] { + assert!( + AdminTitle::parse(title).is_ok(), + "`{title}` should be a legal title" + ); + } + } + + #[test] + fn an_empty_or_blank_title_is_refused() { + assert_eq!(AdminTitle::parse(""), Err(AdminTitleError::Empty)); + assert_eq!(AdminTitle::parse(" "), Err(AdminTitleError::Empty)); + assert_eq!(AdminTitle::parse("\t"), Err(AdminTitleError::Empty)); + } + + #[test] + fn the_bound_is_counted_in_characters_not_bytes() { + // Two bytes per character, so a byte-counted bound would refuse this at + // half the stated length -- making the limit depend on the language the + // title happens to be written in. + let cyrillic = "\u{044f}".repeat(MAX_LEN); + assert_eq!( + cyrillic.len(), + MAX_LEN * 2, + "fixture is not two bytes a char" + ); + assert!(AdminTitle::parse(&cyrillic).is_ok()); + + assert_eq!( + AdminTitle::parse("x".repeat(MAX_LEN + 1)), + Err(AdminTitleError::TooLong(MAX_LEN + 1)) + ); + } + + #[test] + fn a_control_character_is_refused_and_named() { + assert_eq!( + AdminTitle::parse("two\nlines"), + Err(AdminTitleError::Control('\n')) + ); + assert_eq!( + AdminTitle::parse("a\u{0}b"), + Err(AdminTitleError::Control('\u{0}')) + ); + } + + /// Markup is legal text, and refusing it here would be the wrong place to + /// fix the wrong problem: the escaping belongs where the HTML is written, + /// and `doppel-admin` has a test that a title like this cannot close the + /// script element it is injected into. + #[test] + fn markup_is_accepted_because_escaping_is_not_this_types_job() { + let hostile = ""; + assert!(AdminTitle::parse(hostile).is_ok()); + } + + #[test] + fn a_title_round_trips_through_yaml() { + let title: AdminTitle = serde_norway::from_str("\"Doppel (staging)\"").unwrap(); + assert_eq!(title.as_str(), "Doppel (staging)"); + assert_eq!( + serde_norway::to_string(&title).unwrap().trim(), + "Doppel (staging)" + ); + assert!(serde_norway::from_str::("\"\"").is_err()); + } +} diff --git a/crates/doppel-core/src/config/token.rs b/crates/doppel-core/src/config/token.rs index 494b21e..7ee2d95 100644 --- a/crates/doppel-core/src/config/token.rs +++ b/crates/doppel-core/src/config/token.rs @@ -204,6 +204,9 @@ impl utoipa::PartialSchema for Token { "An admin token: printable ASCII with no spaces, 32 to 255 \ characters. A version 4 UUID is the recommended shape.", )) + // Obviously a placeholder. An example that looked like a real token is + // one somebody copies into a deployment. + .examples([serde_json::json!("replace-me-00000000000000000000000")]) .into() } } diff --git a/crates/doppel-core/src/config/url.rs b/crates/doppel-core/src/config/url.rs index 70b2b0f..a558d60 100644 --- a/crates/doppel-core/src/config/url.rs +++ b/crates/doppel-core/src/config/url.rs @@ -28,19 +28,29 @@ pub enum UrlError { HasQueryOrFragment, } +/// The rules both base urls in this configuration are held to: absolute, a +/// scheme Doppel speaks, and no query or fragment. +/// +/// Shared so the two cannot drift. A query on Doppel's own external url would +/// be dropped from every rewritten `Location` for the same reason it is dropped +/// from an upstream: the path is rebuilt, and there is nowhere for it to go. +fn parse_base(value: &str) -> Result { + let url = reqwest::Url::parse(value).map_err(|err| UrlError::NotAbsolute { + reason: err.to_string(), + })?; + if !matches!(url.scheme(), "http" | "https") { + return Err(UrlError::BadScheme(url.scheme().to_owned())); + } + if url.query().is_some() || url.fragment().is_some() { + return Err(UrlError::HasQueryOrFragment); + } + Ok(url) +} + impl UpstreamUrl { /// Check a string and keep it parsed, or say why not. pub fn parse(value: &str) -> Result { - let url = reqwest::Url::parse(value).map_err(|err| UrlError::NotAbsolute { - reason: err.to_string(), - })?; - if !matches!(url.scheme(), "http" | "https") { - return Err(UrlError::BadScheme(url.scheme().to_owned())); - } - if url.query().is_some() || url.fragment().is_some() { - return Err(UrlError::HasQueryOrFragment); - } - Ok(Self(url)) + parse_base(value).map(Self) } #[must_use] @@ -116,6 +126,86 @@ impl utoipa::PartialSchema for UpstreamUrl { impl utoipa::ToSchema for UpstreamUrl {} +/// Where clients reach *this* Doppel, when it is not where Doppel is listening. +/// +/// Doppel cannot work this out for itself. It knows the address it bound, which +/// behind a container port mapping, a load balancer or an ingress is not the +/// address a client used -- and `Host` is a claim by the caller, not something +/// to build a redirect out of. So it is configured: `server.external_url`, or +/// `DOPPEL_EXTERNAL_URL` over the top of it. +/// +/// Held to the same rules as an upstream base, and used the same way: a path is +/// a prefix, so `https://gw.example.com/doppel/` is a Doppel reached under a +/// prefix and rewritten locations keep it. +#[derive(Debug, Clone, PartialEq, Eq, Hash)] +pub struct ExternalUrl(reqwest::Url); + +impl ExternalUrl { + /// Check a string and keep it parsed, or say why not. + pub fn parse(value: &str) -> Result { + parse_base(value).map(Self) + } + + #[must_use] + pub fn into_url(self) -> reqwest::Url { + self.0 + } + + #[must_use] + pub fn as_url(&self) -> &reqwest::Url { + &self.0 + } + + #[must_use] + pub fn as_str(&self) -> &str { + self.0.as_str() + } +} + +impl fmt::Display for ExternalUrl { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + f.write_str(self.0.as_str()) + } +} + +impl FromStr for ExternalUrl { + type Err = UrlError; + + fn from_str(value: &str) -> Result { + Self::parse(value) + } +} + +impl Serialize for ExternalUrl { + /// The parsed form, for the reason `UpstreamUrl` serializes that way: the + /// revision is computed over what this writes. + fn serialize(&self, s: S) -> Result { + s.serialize_str(self.0.as_str()) + } +} + +impl<'de> Deserialize<'de> for ExternalUrl { + fn deserialize>(d: D) -> Result { + let value = String::deserialize(d)?; + Self::parse(&value).map_err(serde::de::Error::custom) + } +} + +impl utoipa::PartialSchema for ExternalUrl { + fn schema() -> utoipa::openapi::RefOr { + utoipa::openapi::schema::ObjectBuilder::new() + .schema_type(utoipa::openapi::schema::Type::String) + .description(Some( + "An absolute http or https url naming where clients reach this \ + Doppel, with no query string or fragment.", + )) + .examples([serde_json::json!("https://doppel.example.com/")]) + .into() + } +} + +impl utoipa::ToSchema for ExternalUrl {} + #[cfg(test)] mod tests { use super::*; diff --git a/crates/doppel-core/src/conformance.rs b/crates/doppel-core/src/conformance.rs index 03b03c8..7723cf8 100644 --- a/crates/doppel-core/src/conformance.rs +++ b/crates/doppel-core/src/conformance.rs @@ -31,6 +31,13 @@ admin: access: {} upload: limit: 1Mi + # Both present and both non-default, for the same reason as + # `rewrite_redirects` below -- and with one more thing to say: since the + # PostgreSQL store keeps the document as JSON, these two arrived without a + # migration of their own. A store that had gone on needing a column per field + # would fail here rather than in production. + dashboard: false + title: "Doppel (conformance)" proxies: - name: alpha type: http @@ -66,6 +73,8 @@ pub async fn run_all(store: &dyn ConfigStore) { retain_templates_with_an_empty_keep_removes_everything(store).await; an_unusable_template_name_is_refused(store).await; templates_are_listed_in_a_stable_order(store).await; + renaming_a_proxy_carries_its_templates(store).await; + renaming_a_proxy_with_no_templates_is_not_an_error(store).await; } async fn load_returns_what_save_wrote(store: &dyn ConfigStore) { @@ -194,6 +203,55 @@ async fn retain_templates_with_an_empty_keep_removes_everything(store: &dyn Conf ); } +async fn renaming_a_proxy_carries_its_templates(store: &dyn ConfigStore) { + // A proxy's name is where its templates live, so a rename that left them behind + // would leave every mock naming a file that is not there. + store.save(&base(), None).await.expect("provision"); + store + .save_template("alpha", "put.j2", b"body") + .await + .unwrap(); + + store + .rename_templates("alpha", "renamed") + .await + .expect("rename"); + + let moved = store.load_templates("renamed").await.expect("list"); + assert_eq!( + moved.len(), + 1, + "the template did not arrive under the new name" + ); + assert_eq!(moved[0].name, "put.j2"); + assert_eq!(moved[0].content, b"body", "the contents changed on the way"); + assert!( + store + .load_templates("alpha") + .await + .expect("list") + .is_empty(), + "the old name still has templates, so they were copied rather than moved" + ); +} + +async fn renaming_a_proxy_with_no_templates_is_not_an_error(store: &dyn ConfigStore) { + // The common case: most proxies have no templates at all, and renaming one must + // not depend on whether it does. + store.save(&base(), None).await.expect("provision"); + store + .rename_templates("beta", "beta-renamed") + .await + .expect("renaming a proxy with no templates must succeed"); + assert!( + store + .load_templates("beta-renamed") + .await + .expect("list") + .is_empty() + ); +} + async fn an_unusable_template_name_is_refused(store: &dyn ConfigStore) { store.save(&base(), None).await.expect("provision"); for name in ["..", "a/b", "", ".hidden", "a..b"] { diff --git a/crates/doppel-core/src/error.rs b/crates/doppel-core/src/error.rs index 199f5b1..7e2ac6c 100644 --- a/crates/doppel-core/src/error.rs +++ b/crates/doppel-core/src/error.rs @@ -27,6 +27,7 @@ pub enum ErrorCode { RevisionMismatch, RevisionRequired, InvalidRequestPath, + DashboardNotBuilt, } impl ErrorCode { @@ -35,7 +36,10 @@ impl ErrorCode { pub fn status(self) -> u16 { match self { Self::ProxyNotResolved | Self::NotFound => 404, - Self::NoProxiesConfigured => 503, + // Both mean "this binary cannot serve that, and no request will change + // it": one because nothing is configured to proxy to, the other because + // the binary was built without the dashboard's static assets. + Self::NoProxiesConfigured | Self::DashboardNotBuilt => 503, Self::TemplateRenderError | Self::TemplateNotFound | Self::BodyExtractionError @@ -99,6 +103,7 @@ impl ErrorCode { Self::RevisionMismatch => "REVISION_MISMATCH", Self::RevisionRequired => "REVISION_REQUIRED", Self::InvalidRequestPath => "INVALID_REQUEST_PATH", + Self::DashboardNotBuilt => "DASHBOARD_NOT_BUILT", } } } @@ -136,6 +141,7 @@ impl<'de> Deserialize<'de> for ErrorCode { "REVISION_MISMATCH" => Ok(Self::RevisionMismatch), "REVISION_REQUIRED" => Ok(Self::RevisionRequired), "INVALID_REQUEST_PATH" => Ok(Self::InvalidRequestPath), + "DASHBOARD_NOT_BUILT" => Ok(Self::DashboardNotBuilt), other => Err(serde::de::Error::custom(format!( "unknown error code `{other}`" ))), @@ -224,6 +230,7 @@ mod tests { (ErrorCode::RevisionMismatch, "REVISION_MISMATCH", 409), (ErrorCode::RevisionRequired, "REVISION_REQUIRED", 428), (ErrorCode::InvalidRequestPath, "INVALID_REQUEST_PATH", 400), + (ErrorCode::DashboardNotBuilt, "DASHBOARD_NOT_BUILT", 503), ]; /// See `ALL_CODES`'s doc comment: this match's lack of a wildcard arm is @@ -249,7 +256,8 @@ mod tests { | ErrorCode::StoreError | ErrorCode::RevisionMismatch | ErrorCode::RevisionRequired - | ErrorCode::InvalidRequestPath => { + | ErrorCode::InvalidRequestPath + | ErrorCode::DashboardNotBuilt => { let count = ALL_CODES.iter().filter(|(c, _, _)| *c == code).count(); assert_eq!(count, 1, "{code:?} must appear exactly once in ALL_CODES"); } diff --git a/crates/doppel-core/src/metrics.rs b/crates/doppel-core/src/metrics.rs index 0ff1e40..a715559 100644 --- a/crates/doppel-core/src/metrics.rs +++ b/crates/doppel-core/src/metrics.rs @@ -7,15 +7,22 @@ //! request path: the cardinality rule is enforced by the signatures, not by //! remembering it at five call sites. -use std::time::Duration; +use std::collections::BTreeSet; +use std::sync::Mutex; +use std::time::{Duration, SystemTime, UNIX_EPOCH}; -use metrics_exporter_prometheus::{BuildError, PrometheusBuilder, PrometheusHandle}; +use metrics_exporter_prometheus::{BuildError, Matcher, PrometheusBuilder, PrometheusHandle}; pub const UPSTREAM_DURATION: &str = "doppel_upstream_request_duration_seconds"; pub const PROXY_DURATION: &str = "doppel_proxy_request_duration_seconds"; +pub const ADMIN_DURATION: &str = "doppel_admin_request_duration_seconds"; pub const LOSS_TOTAL: &str = "doppel_loss_total"; pub const LATENCY_INJECTED_TOTAL: &str = "doppel_latency_injected_total"; pub const MOCK_HITS_TOTAL: &str = "doppel_mock_hits_total"; +pub const LAST_ERROR: &str = "doppel_proxy_last_error_timestamp_seconds"; +pub const BUILD_INFO: &str = "doppel_build_info"; +pub const DASHBOARD_INFO: &str = "doppel_dashboard_info"; +pub const PROXY_MOCKS: &str = "doppel_proxy_mocks"; /// Recorded when a request reached the proxy listener but matched no proxy. /// @@ -25,6 +32,12 @@ pub const MOCK_HITS_TOTAL: &str = "doppel_mock_hits_total"; /// which is the kind of gap an operator finds during an incident. pub const UNRESOLVED: &str = ""; +/// Recorded when a request reached the admin listener and matched no route. +/// +/// The same reasoning as `UNRESOLVED`, and the same value: no route template is +/// empty, so this cannot collide with one. +pub const UNMATCHED: &str = ""; + /// Latency buckets, in seconds. /// /// Explicit because the Prometheus exporter renders a histogram as a summary @@ -33,9 +46,27 @@ pub const UNRESOLVED: &str = ""; /// requirement asks for buckets, and buckets are also the only form that /// survives more than one replica. const BUCKETS: &[f64] = &[ - 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0, + 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0, 15.0, 30.0, 60.0, ]; +/// The same ladder, cut at five seconds, for the admin API. +/// +/// The upper reaches of the proxy ladder are there because a proxy is asked to be +/// slow on purpose -- `latency` injection goes to whatever the configuration says, +/// and a 60-second bucket is a real measurement of a fault. Nothing on the admin +/// listener has any business taking longer than five seconds: it reads a document, +/// writes a document, or renders an exposition. +const ADMIN_BUCKETS: &[f64] = &[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0]; + +/// Proxy names whose mock gauge has been set, so a proxy that leaves the +/// configuration can be zeroed rather than left at its last value. +/// +/// The exporter has no way to delete a series: an unset gauge keeps reporting +/// whatever it last held, which for a deleted proxy would be a mock count for +/// something that no longer exists. Zero is the honest reading, and this is the +/// smallest thing that knows which names to zero. +static GAUGED_PROXIES: Mutex> = Mutex::new(BTreeSet::new()); + /// Build a recorder without touching the global one. /// /// Tests use this with `metrics::set_default_local_recorder`, so they observe @@ -44,6 +75,7 @@ const BUCKETS: &[f64] = &[ pub fn build() -> Result { Ok(PrometheusBuilder::new() .set_buckets(BUCKETS)? + .set_buckets_for_metric(Matcher::Full(ADMIN_DURATION.to_owned()), ADMIN_BUCKETS)? .build_recorder()) } @@ -80,18 +112,156 @@ pub fn record_upstream(proxy: &str, method: &str, status: u16, elapsed: Duration .record(elapsed.as_secs_f64()); } +/// How a request through the proxy listener was answered, beyond its status. +/// +/// Three booleans rather than one enum: they are not exclusive. A mock can answer +/// a request the proxy would otherwise have dropped, and an upstream error can +/// arrive after a latency delay. Each is `1` or `0` in a label, so a query can ask +/// for "everything that was not a fault" without parsing a status class. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub struct Outcome { + /// A mock answered instead of the upstream. + pub replace: bool, + /// Loss injection dropped the request. + pub loss: bool, + /// The upstream was contacted and did not answer usefully -- a transport + /// failure, a timeout, or a status of 500 and above. + pub upstream_error: bool, +} + +impl Outcome { + /// A plain proxied exchange. + #[must_use] + pub fn proxied() -> Self { + Self::default() + } + + #[must_use] + pub fn mocked() -> Self { + Self { + replace: true, + ..Self::default() + } + } + + #[must_use] + pub fn lost() -> Self { + Self { + loss: true, + ..Self::default() + } + } + + /// The label value: `"1"` or `"0"`, never `"true"`, because a Prometheus + /// query on a boolean label reads better against the digits and every + /// dashboard already writes them that way. + fn flag(value: bool) -> &'static str { + if value { "1" } else { "0" } + } +} + /// One request through the proxy listener, however it was answered: proxied, /// mocked, dropped or rejected. -pub fn record_proxy(proxy: &str, method: &str, status: u16, elapsed: Duration) { +pub fn record_proxy(proxy: &str, method: &str, status: u16, elapsed: Duration, outcome: Outcome) { metrics::histogram!( PROXY_DURATION, "proxy" => proxy.to_owned(), "method" => crate::method::method_label(method), "status" => status.to_string(), + "replace" => Outcome::flag(outcome.replace), + "loss" => Outcome::flag(outcome.loss), + "upstream_error" => Outcome::flag(outcome.upstream_error), ) .record(elapsed.as_secs_f64()); } +/// One request to the admin listener, by route template rather than by path. +/// +/// `route` is what the router matched -- `/api/v1/proxies/{name}`, not +/// `/api/v1/proxies/alpha` -- so a hundred proxies are one series and a query +/// string is none. That is the same cardinality rule the proxy metrics follow, and +/// it is why this takes a template and not a `Uri`: the type is the rule. +/// +/// A request that matched no route is recorded under `UNMATCHED`, so the total +/// here is the total the listener answered. +pub fn record_admin(method: &str, route: &str, status: u16, elapsed: Duration) { + metrics::histogram!( + ADMIN_DURATION, + "route" => route.to_owned(), + "method" => crate::method::method_label(method), + "status" => status.to_string(), + ) + .record(elapsed.as_secs_f64()); +} + +/// When the proxy listener last answered with an error, and which error it was. +/// +/// A timestamp rather than a counter because the question it answers is "is this +/// still happening", and a gauge of seconds since the epoch answers that at any +/// scrape interval: `time() - doppel_proxy_last_error_timestamp_seconds`. A +/// counter answers "how many" and needs a rate window to say anything about now. +pub fn record_proxy_error(code: &str) { + let now = SystemTime::now() + .duration_since(UNIX_EPOCH) + .map_or(0.0, |since| since.as_secs_f64()); + metrics::gauge!(LAST_ERROR, "code" => code.to_owned()).set(now); +} + +/// The series that exists before anything goes wrong: `code=""`, value `0`. +/// +/// Without it, a dashboard panel and an alert on this metric both read "no data" +/// until the first error, which is indistinguishable from "the process is not +/// being scraped". Zero says "no error yet" in a way a query can subtract. +pub fn init_proxy_errors() { + metrics::gauge!(LAST_ERROR, "code" => "").set(0.0); +} + +/// This binary, as a label: the info-metric pattern, value always `1`. +/// +/// Version in a label rather than as a value, because a version is not a number +/// -- `0.10.0` and `0.1.0` do not order as floats -- and because that is how every +/// exporter publishes one, so `doppel_build_info` joins with the rest of a fleet's +/// build panels without special-casing. +pub fn describe_build(version: &str) { + metrics::gauge!(BUILD_INFO, "version" => version.to_owned()).set(1.0); +} + +/// Whether this process serves the dashboard, in the same shape. +/// +/// Separate from `doppel_build_info` on purpose: one describes the artifact and +/// never changes, the other describes what this deployment turned on. Folding a +/// configuration flag into build info makes two builds of one version look +/// different. +pub fn describe_dashboard(enabled: bool) { + metrics::gauge!(DASHBOARD_INFO, "enabled" => if enabled { "true" } else { "false" }).set(1.0); +} + +/// The mock count of every proxy in the configuration now in effect. +/// +/// Called on every runtime swap -- startup, a reload, a write through the admin +/// API -- so what this reports is the configuration being served rather than the +/// one the process started with. A proxy that has left the configuration is set to +/// zero rather than abandoned at its last count: the exporter cannot drop a +/// series, and a stale one reads as a proxy that still exists. +pub fn record_proxy_mocks(counts: &[(&str, usize)]) { + let mut gauged = GAUGED_PROXIES + .lock() + .expect("the gauge set is not poisoned"); + for (proxy, mocks) in counts { + #[allow(clippy::cast_precision_loss)] + metrics::gauge!(PROXY_MOCKS, "proxy" => (*proxy).to_owned()).set(*mocks as f64); + gauged.insert((*proxy).to_owned()); + } + + let present: BTreeSet<&str> = counts.iter().map(|(proxy, _)| *proxy).collect(); + for gone in gauged + .iter() + .filter(|name| !present.contains(name.as_str())) + { + metrics::gauge!(PROXY_MOCKS, "proxy" => gone.clone()).set(0.0); + } +} + pub fn record_loss(proxy: &str) { metrics::counter!(LOSS_TOTAL, "proxy" => proxy.to_owned()).increment(1); } @@ -165,7 +335,13 @@ mod tests { // production cardinality incident. let text = exposition(|| { record_upstream("alpha", "GET", 200, Duration::from_millis(1)); - record_proxy("alpha", "GET", 200, Duration::from_millis(2)); + record_proxy( + "alpha", + "GET", + 200, + Duration::from_millis(2), + Outcome::proxied(), + ); record_loss("alpha"); record_latency_injected("alpha"); record_mock_hit("alpha", "mock1"); @@ -180,7 +356,13 @@ mod tests { fn every_documented_metric_appears_once_recorded() { let text = exposition(|| { record_upstream("alpha", "GET", 200, Duration::from_millis(1)); - record_proxy("alpha", "GET", 200, Duration::from_millis(2)); + record_proxy( + "alpha", + "GET", + 200, + Duration::from_millis(2), + Outcome::proxied(), + ); record_loss("alpha"); record_latency_injected("alpha"); record_mock_hit("alpha", "mock1"); @@ -203,7 +385,13 @@ mod tests { // add a thousand series to the exposition. let text = exposition(|| { for i in 0..100 { - record_proxy("alpha", &format!("EVIL{i}"), 200, Duration::from_millis(1)); + record_proxy( + "alpha", + &format!("EVIL{i}"), + 200, + Duration::from_millis(1), + Outcome::proxied(), + ); } }); @@ -211,12 +399,188 @@ mod tests { assert!(!text.contains("EVIL"), "{text}"); } + #[test] + fn how_a_request_was_answered_is_in_the_labels() { + let text = exposition(|| { + record_proxy( + "alpha", + "GET", + 200, + Duration::from_millis(1), + Outcome::mocked(), + ); + }); + + // Digits rather than `true`/`false`: a query reads better against them and + // every dashboard writes them that way. + assert!(text.contains(r#"replace="1""#), "{text}"); + assert!(text.contains(r#"loss="0""#), "{text}"); + assert!(text.contains(r#"upstream_error="0""#), "{text}"); + } + + #[test] + fn the_three_flags_are_independent() { + // Not an enum, because they are not exclusive: a mock can answer a request + // its own loss roll then drops. + let text = exposition(|| { + record_proxy( + "alpha", + "GET", + 503, + Duration::from_millis(1), + Outcome { + replace: true, + loss: true, + upstream_error: false, + }, + ); + }); + + assert!( + text.contains(r#"loss="1""#) && text.contains(r#"replace="1""#), + "{text}" + ); + } + + #[test] + fn the_proxy_ladder_reaches_a_minute_and_the_admin_ladder_stops_at_five_seconds() { + // A proxy is asked to be slow on purpose -- `latency` injection goes to + // whatever the configuration says -- so a 60-second bucket is a real + // measurement. Nothing on the admin listener has any business there. + let text = exposition(|| { + record_proxy( + "alpha", + "GET", + 200, + Duration::from_secs(20), + Outcome::proxied(), + ); + record_admin("GET", "/api/v1/proxies", 200, Duration::from_millis(3)); + }); + + let proxy_series: String = text + .lines() + .filter(|line| line.starts_with(PROXY_DURATION)) + .collect(); + assert!(proxy_series.contains(r#"le="60""#), "{proxy_series}"); + assert!(proxy_series.contains(r#"le="30"} 1"#), "{proxy_series}"); + assert!(proxy_series.contains(r#"le="15"} 0"#), "{proxy_series}"); + + let admin_series: String = text + .lines() + .filter(|line| line.starts_with(ADMIN_DURATION)) + .collect(); + assert!(admin_series.contains(r#"le="5""#), "{admin_series}"); + assert!(!admin_series.contains(r#"le="10""#), "{admin_series}"); + assert!(!admin_series.contains(r#"le="60""#), "{admin_series}"); + } + + #[test] + fn an_admin_request_is_labelled_by_its_route_template() { + // The label is the template, so a hundred proxies are one series. A label + // built from the path would be one per name, plus one per query string. + let text = exposition(|| { + record_admin( + "GET", + "/api/v1/proxies/{name}", + 200, + Duration::from_millis(1), + ); + record_admin("GET", UNMATCHED, 404, Duration::from_millis(1)); + }); + + assert!(text.contains(r#"route="/api/v1/proxies/{name}""#), "{text}"); + assert!(text.contains(r#"route="""#), "{text}"); + assert!(!text.contains("alpha"), "{text}"); + } + + #[test] + fn the_last_error_exists_before_anything_goes_wrong() { + let text = exposition(init_proxy_errors); + + // Zero, with an empty code: "no error yet" in a form a query can subtract, + // rather than the no-data a never-recorded metric gives an alert. + assert!(text.contains(LAST_ERROR), "{text}"); + assert!(text.contains(r#"code=""} 0"#), "{text}"); + } + + #[test] + fn an_error_stamps_the_time_under_its_code() { + let before = SystemTime::now() + .duration_since(UNIX_EPOCH) + .expect("the clock is after 1970") + .as_secs_f64(); + let text = exposition(|| { + init_proxy_errors(); + record_proxy_error("UPSTREAM_TIMEOUT"); + }); + + let stamped = text + .lines() + .find(|line| line.starts_with(LAST_ERROR) && line.contains("UPSTREAM_TIMEOUT")) + .unwrap_or_else(|| panic!("{text}")); + let value: f64 = stamped + .rsplit(' ') + .next() + .and_then(|value| value.parse().ok()) + .unwrap_or_else(|| panic!("{stamped}")); + assert!(value >= before, "{stamped}"); + // And the default series is still there, still zero: the codes do not + // replace each other. + assert!(text.contains(r#"code=""} 0"#), "{text}"); + } + + #[test] + fn the_build_and_the_dashboard_are_described_separately() { + let text = exposition(|| { + describe_build("1.2.3"); + describe_dashboard(true); + }); + + // Value 1, facts in labels: the info-metric pattern, and the version is not + // a number -- `0.10.0` and `0.1.0` do not order as floats. + assert!( + text.contains(r#"doppel_build_info{version="1.2.3"} 1"#), + "{text}" + ); + assert!( + text.contains(r#"doppel_dashboard_info{enabled="true"} 1"#), + "{text}" + ); + } + + #[test] + fn a_proxy_that_leaves_the_configuration_is_zeroed_rather_than_left_behind() { + // The exporter cannot delete a series, so a proxy that was removed would + // otherwise keep reporting its last mock count -- a proxy that does not + // exist, with three mocks, for as long as the process runs. + let text = exposition(|| { + record_proxy_mocks(&[("alpha", 3), ("beta", 0)]); + record_proxy_mocks(&[("beta", 1)]); + }); + + assert!( + text.contains(r#"doppel_proxy_mocks{proxy="alpha"} 0"#), + "{text}" + ); + assert!( + text.contains(r#"doppel_proxy_mocks{proxy="beta"} 1"#), + "{text}" + ); + } + #[test] fn an_unresolved_request_is_counted_under_a_name_no_proxy_can_have() { // Rule V35 refuses an empty proxy name, which is what makes this // sentinel safe rather than a collision waiting to happen. let text = exposition(|| { - record_proxy(UNRESOLVED, "GET", 404, Duration::from_millis(1)); + record_proxy( + UNRESOLVED, + "GET", + 404, + Duration::from_millis(1), + Outcome::proxied(), + ); }); assert!(text.contains(r#"proxy="""#), "{text}"); diff --git a/crates/doppel-core/src/runtime.rs b/crates/doppel-core/src/runtime.rs index 8d26dd0..9a4c8bf 100644 --- a/crates/doppel-core/src/runtime.rs +++ b/crates/doppel-core/src/runtime.rs @@ -28,6 +28,10 @@ pub struct CompiledProxy { /// point at Doppel. The `Option` in the configuration resolves to its /// default here, so the hot path never has to know what that default was. pub rewrite_redirects: bool, + /// Whether the upstream's own address is rewritten to Doppel's inside a + /// relayed body. The `Option` in the configuration resolves to its default + /// here, like `rewrite_redirects` above. + pub rewrite_urls: bool, pub resolve_header: Option, /// Mocks in configuration order. Matching is first-wins, so this order /// is load-bearing, not incidental. @@ -190,6 +194,7 @@ fn compile_proxy(proxy: &ProxyConfig) -> Result { latency: proxy.latency, replace: proxy.replace.map_or(1.0, crate::config::Ratio::get), rewrite_redirects: proxy.rewrite_redirects.unwrap_or(true), + rewrite_urls: proxy.rewrite_urls.unwrap_or(true), resolve_header: match proxy.resolve.kind { ResolveKind::Header => proxy .resolve @@ -281,6 +286,7 @@ pub struct RuntimeHolder(ArcSwap); impl RuntimeHolder { #[must_use] pub fn new(runtime: Runtime) -> Self { + publish_mock_counts(&runtime); Self(ArcSwap::from_pointee(runtime)) } @@ -290,10 +296,29 @@ impl RuntimeHolder { } pub fn store(&self, runtime: Runtime) { + // Before the swap, so that anything scraping between the two sees the + // counts of the configuration that is about to be in force rather than + // the one that just left. Either way the window is a few microseconds + // wide; this is the side of it that cannot report a proxy that is gone. + publish_mock_counts(&runtime); self.0.store(Arc::new(runtime)); } } +/// `doppel_proxy_mocks` for the configuration this runtime carries. +/// +/// Here rather than at the call sites that build a runtime -- startup, reload, a +/// write through the admin API -- because there are three of them and a metric +/// that follows the configuration cannot afford to be forgotten at one. +fn publish_mock_counts(runtime: &Runtime) { + let counts: Vec<(&str, usize)> = runtime + .proxies + .iter() + .map(|proxy| (proxy.name.as_str(), proxy.mocks.len())) + .collect(); + crate::metrics::record_proxy_mocks(&counts); +} + #[cfg(test)] mod tests { use super::*; diff --git a/crates/doppel-core/src/store/file.rs b/crates/doppel-core/src/store/file.rs index 8a6349b..0e1a21e 100644 --- a/crates/doppel-core/src/store/file.rs +++ b/crates/doppel-core/src/store/file.rs @@ -92,6 +92,38 @@ impl FileStore { } } + /// The rename that puts a saved configuration in place has failed. + /// + /// One cause is worth naming, because the operating system's own words for it + /// are true and useless: `EBUSY` here almost always means the configuration + /// *file* is a mount point. A container started with + /// `-v ./main.yaml:/etc/doppel/main.yaml` mounts the file itself, and nothing + /// can rename over a mount point -- so every read works, every save fails, and + /// the message says "Resource busy". + /// + /// Mounting the directory instead is the fix, and the message says so. The + /// alternative -- writing in place instead of renaming -- would trade this for + /// a configuration that can be read half-written, which is the one thing this + /// whole function is arranged to prevent. + fn replace_failed(path: &Path, source: std::io::Error) -> StoreError { + if source.kind() != std::io::ErrorKind::ResourceBusy { + return StoreError::Io { + path: path.to_path_buf(), + source, + }; + } + StoreError::Io { + path: path.to_path_buf(), + source: std::io::Error::new( + std::io::ErrorKind::ResourceBusy, + "cannot be replaced because it is busy. A save writes a new file and \ + renames it over this path, and nothing can rename over a mount point -- \ + which is what a container gets from `-v ./main.yaml:/etc/doppel/main.yaml`. \ + Mount the directory instead: `-v ./config:/etc/doppel`", + ), + } + } + /// Decode a directory entry's raw file name, treating anything that is /// not valid UTF-8 as foreign junk rather than something to report or /// compare under a mangled, `to_string_lossy`-produced stand-in. @@ -210,10 +242,7 @@ impl ConfigStore for FileStore { .map_err(Self::io(temp.path()))?; temp.as_file().sync_all().map_err(Self::io(temp.path()))?; temp.persist(&self.config_path) - .map_err(|e| StoreError::Io { - path: self.config_path.clone(), - source: e.error, - })?; + .map_err(|e| Self::replace_failed(&self.config_path, e.error))?; // The rename itself is atomic and content is never at risk: a // concurrent reader always sees the complete old configuration or @@ -283,6 +312,13 @@ impl ConfigStore for FileStore { std::fs::write(&path, bytes).map_err(Self::io(&path)) } + async fn rename_templates(&self, from: &str, to: &str) -> Result<(), StoreError> { + // The directory *is* the storage here, so the rename is the whole + // operation -- and it is atomic, unlike the two steps the database store + // needs. + super::rename_template_dir(&self.proxy_dir(from)?, &self.proxy_dir(to)?) + } + async fn delete_template(&self, proxy: &str, file: &str) -> Result { let dir = self.proxy_dir(proxy)?; let file = sanitize(file)?; @@ -381,6 +417,38 @@ proxies: (store, config_path) } + #[test] + fn a_busy_config_file_is_explained_rather_than_reported() { + // The failure a container hits on its first save when the *file* is the bind + // mount. `EBUSY` alone sends the reader looking for another process holding + // the file, which is not what is wrong. + let path = Path::new("/etc/doppel/main.yaml"); + let busy = FileStore::replace_failed(path, std::io::Error::from_raw_os_error(16)); + + let message = busy.to_string(); + assert!(message.contains("/etc/doppel/main.yaml"), "{message}"); + assert!( + message.contains("mount the directory") || message.contains("Mount the directory"), + "{message}" + ); + assert!(message.contains("rename"), "{message}"); + } + + #[test] + fn any_other_failure_to_replace_is_reported_as_it_came() { + // Only `EBUSY` gets the explanation. A full disk or a permission problem is + // its own thing, and burying it under advice about mounts would be worse + // than saying nothing. + let path = Path::new("/etc/doppel/main.yaml"); + let denied = FileStore::replace_failed( + path, + std::io::Error::from(std::io::ErrorKind::PermissionDenied), + ); + let message = denied.to_string(); + assert!(message.contains("permission denied"), "{message}"); + assert!(!message.contains("Mount the directory"), "{message}"); + } + #[tokio::test] async fn load_returns_a_validated_config() { let dir = tempfile::tempdir().unwrap(); diff --git a/crates/doppel-core/src/store/mod.rs b/crates/doppel-core/src/store/mod.rs index af6d14c..20f9c90 100644 --- a/crates/doppel-core/src/store/mod.rs +++ b/crates/doppel-core/src/store/mod.rs @@ -200,6 +200,18 @@ pub trait ConfigStore: Send + Sync { /// `keep` removes the proxy's storage entirely. async fn retain_templates(&self, proxy: &str, keep: &[String]) -> Result<(), StoreError>; + /// Move every template of `from` to `to`, because the proxy was renamed. + /// + /// A proxy's name is where its templates live -- a subdirectory for + /// `FileStore`, a column for the database -- so a rename that did not carry + /// them would leave every mock pointing at a file that is not there. There is + /// nothing to move for a proxy that has no templates, and that is not an + /// error. + /// + /// The caller has already written the renamed configuration: the write is what + /// authorises the move, the same order `retain_templates` is called in. + async fn rename_templates(&self, from: &str, to: &str) -> Result<(), StoreError>; + /// Make every template this store holds available as a file under `dir`. /// /// The render path reads a template from the filesystem at request time, @@ -215,6 +227,39 @@ pub trait ConfigStore: Send + Sync { } } +/// Move a proxy's template directory, for the stores that keep one. +/// +/// `FileStore` keeps its templates there and nowhere else; the database store +/// mirrors its rows to the same layout, because the render path reads files. Both +/// therefore have a directory to move when a proxy is renamed, and both want the +/// same three answers: a missing source is nothing to do, an occupied destination is +/// a bug in the caller rather than something to overwrite, and anything else is the +/// filesystem's own error with the path that produced it. +pub fn rename_template_dir(from: &std::path::Path, to: &std::path::Path) -> Result<(), StoreError> { + if !from.exists() { + return Ok(()); + } + if to.exists() { + return Err(StoreError::Io { + path: to.to_path_buf(), + source: std::io::Error::new( + std::io::ErrorKind::AlreadyExists, + "templates for this name are already there; the configuration should have refused the rename before it got here", + ), + }); + } + if let Some(parent) = to.parent() { + std::fs::create_dir_all(parent).map_err(|source| StoreError::Io { + path: parent.to_path_buf(), + source, + })?; + } + std::fs::rename(from, to).map_err(|source| StoreError::Io { + path: from.to_path_buf(), + source, + }) +} + #[cfg(test)] mod tests { use super::*; @@ -280,6 +325,7 @@ proxies: latency: None, replace: None, rewrite_redirects: None, + rewrite_urls: None, body_limit: ByteSize::parse(1024 * 1024).unwrap(), mocks: Vec::new(), } diff --git a/crates/doppel-proxy/Cargo.toml b/crates/doppel-proxy/Cargo.toml index 4cf59dc..a37d593 100644 --- a/crates/doppel-proxy/Cargo.toml +++ b/crates/doppel-proxy/Cargo.toml @@ -9,6 +9,7 @@ license.workspace = true doppel-core = { path = "../doppel-core" } doppel-render = { path = "../doppel-render" } axum.workspace = true +bytes.workspace = true futures-util.workspace = true rand.workspace = true reqwest.workspace = true diff --git a/crates/doppel-proxy/src/lib.rs b/crates/doppel-proxy/src/lib.rs index 74cdf29..25f03d6 100644 --- a/crates/doppel-proxy/src/lib.rs +++ b/crates/doppel-proxy/src/lib.rs @@ -3,6 +3,7 @@ pub mod fault; pub mod mock; pub mod resolve; +pub mod rewrite; pub mod server; pub mod upstream; diff --git a/crates/doppel-proxy/src/mock.rs b/crates/doppel-proxy/src/mock.rs index 0561bd7..d8dacf8 100644 --- a/crates/doppel-proxy/src/mock.rs +++ b/crates/doppel-proxy/src/mock.rs @@ -183,6 +183,7 @@ mod tests { latency: None, replace: 1.0, rewrite_redirects: true, + rewrite_urls: false, resolve_header: None, mocks, body_limit: 1024 * 1024, diff --git a/crates/doppel-proxy/src/rewrite.rs b/crates/doppel-proxy/src/rewrite.rs new file mode 100644 index 0000000..4980712 --- /dev/null +++ b/crates/doppel-proxy/src/rewrite.rs @@ -0,0 +1,360 @@ +//! Rewriting the upstream's own address out of a relayed body. +//! +//! `rewrite_redirects` keeps a client inside Doppel when the upstream answers a +//! redirect. This is the same problem one layer down: a page, a script or a JSON +//! document that names `https://api.example.com/v2/` sends the client straight +//! there on its next request, past every injected fault and every mock, with +//! nothing logged and nothing failing. `nginx` has `sub_filter` for this. +//! +//! Three rules, and each is a decision worth reading before changing: +//! +//! - **Exact host only.** `https://cdn.api.example.com/` is a different host, is +//! not proxied by this proxy, and is left alone -- pointing it at Doppel would +//! break the page rather than keep it working. That falls out of matching on +//! `scheme://host` and then requiring the next character not to continue a +//! hostname, which is also what stops `https://api.example.com.evil.test/` from +//! being mangled into Doppel's address with a suffix. +//! - **Text only, and only what the upstream did not compress.** A body Doppel +//! would have to decompress to search is relayed untouched, and so is one whose +//! content type is not text: an image cannot contain a URL that matters, and +//! buffering one to look is how a proxy runs out of memory. +//! - **Bounded.** Rewriting needs the whole body, so it is buffered -- up to the +//! proxy's `body_limit`, the same ceiling that bounds a mock's request buffering. +//! A body that exceeds it is streamed on from where the buffering stopped, +//! untouched, rather than held in memory or truncated. + +use axum::body::Body; +use axum::http::{HeaderMap, HeaderValue, header}; +use bytes::Bytes; +use futures_util::{StreamExt, TryStreamExt}; + +/// The content types worth searching for a URL. +/// +/// A prefix list rather than a parser: the header may carry parameters +/// (`text/html; charset=utf-8`), and only the type and subtype decide this. +/// `+json` and `+xml` catch the structured suffixes -- `application/problem+json`, +/// `image/svg+xml` -- which are text and do carry URLs. +const TEXTUAL: &[&str] = &[ + "text/", + "application/json", + "application/javascript", + "application/xml", + "application/xhtml+xml", + "application/x-javascript", + "application/graphql", + "application/ld+json", +]; + +/// Whether a body with these headers is one to rewrite. +#[must_use] +pub fn is_rewritable(headers: &HeaderMap) -> bool { + // A compressed body would have to be decompressed, rewritten and recompressed, + // and the client asked the upstream for that encoding. Relayed as it came. + if headers.contains_key(header::CONTENT_ENCODING) { + return false; + } + + let Some(kind) = headers + .get(header::CONTENT_TYPE) + .and_then(|value| value.to_str().ok()) + else { + // No content type: nothing says this is text, and guessing by sniffing the + // bytes is a second, worse rule to maintain. + return false; + }; + let kind = kind.trim().to_ascii_lowercase(); + TEXTUAL.iter().any(|prefix| kind.starts_with(prefix)) + || kind.contains("+json") + || kind.contains("+xml") +} + +/// The body, with the upstream's address replaced by Doppel's where it appears. +/// +/// Returns the body to relay, and the new length when something was replaced. The +/// caller needs the second half: a rewritten body is not the entity the upstream +/// sent, so its length and its validators both have to be restated. +pub async fn rewrite_body( + stream: S, + base: &reqwest::Url, + external: &reqwest::Url, + limit: u64, +) -> (Body, Option) +where + S: futures_util::Stream> + Send + Unpin + 'static, +{ + let mut stream = stream; + let mut buffered: Vec = Vec::new(); + let cap = usize::try_from(limit).unwrap_or(usize::MAX); + + while let Some(chunk) = stream.next().await { + let Ok(chunk) = chunk else { + // A failed read is the caller's problem to see, and the stream is the + // only thing that can report it: hand back what is left, errors and + // all, rather than swallowing it into a truncated 200. + let head = Bytes::from(buffered); + return (Body::from_stream(prepend(head, stream)), None); + }; + buffered.extend_from_slice(&chunk); + if buffered.len() > cap { + // Too big to hold. What was read is sent on in front of the rest, + // untouched -- the alternative is buffering without a bound. + tracing::debug!( + bytes = buffered.len(), + limit, + "body over the limit; relayed without rewriting urls" + ); + let head = Bytes::from(buffered); + return (Body::from_stream(prepend(head, stream)), None); + } + } + + let text = match String::from_utf8(buffered) { + Ok(text) => text, + // Declared as text and not valid UTF-8. The bytes are already buffered, so + // they go out exactly as they came: re-encoding somebody else's body is not + // this proxy's business, and dropping it would be worse. + Err(not_text) => return (Body::from(not_text.into_bytes()), None), + }; + + let (rewritten, changed) = replace_addresses(&text, base, external); + let length = rewritten.len(); + (Body::from(rewritten), changed.then_some(length)) +} + +/// Put `head` back in front of `rest`. +fn prepend(head: Bytes, rest: S) -> impl futures_util::Stream> +where + S: futures_util::Stream>, +{ + futures_util::stream::once(async move { Ok(head) }).chain(rest.into_stream()) +} + +/// `text` with the upstream's base and origin replaced by Doppel's. +/// +/// The base goes first and the origin second, so a URL under the proxied path +/// loses that prefix -- `https://api.example.com/v2/orders` becomes +/// `http://doppel/orders`, which is where Doppel actually serves it -- while a URL +/// on the same host outside that path keeps its own, the way a rewritten redirect +/// does. +fn replace_addresses(text: &str, base: &reqwest::Url, external: &reqwest::Url) -> (String, bool) { + let to = external.as_str().trim_end_matches('/').to_owned(); + + let (text, base_changed) = replace_at_boundary(text, base.as_str(), &format!("{to}/")); + let origin = origin_of(base); + let (text, origin_changed) = replace_at_boundary(&text, &origin, &to); + (text, base_changed || origin_changed) +} + +/// `scheme://host[:port]` of a URL, which is what a body writes when it names a +/// host without a path. +fn origin_of(url: &reqwest::Url) -> String { + let mut origin = format!("{}://{}", url.scheme(), url.host_str().unwrap_or_default()); + if let Some(port) = url.port() { + origin.push(':'); + origin.push_str(&port.to_string()); + } + origin +} + +/// Every occurrence of `needle` that is not the beginning of a longer hostname, +/// replaced by `replacement`. +/// +/// The boundary check is the whole point. `https://api.example.com` is a prefix of +/// `https://api.example.com.evil.test`, and a plain string replacement would turn +/// that into Doppel's address with somebody else's suffix glued on. A character +/// that could continue a hostname -- a letter, a digit, a dot or a hyphen -- means +/// this is a different host, and it is left alone. +fn replace_at_boundary(text: &str, needle: &str, replacement: &str) -> (String, bool) { + if needle.is_empty() { + return (text.to_owned(), false); + } + + let mut out = String::with_capacity(text.len()); + let mut rest = text; + let mut changed = false; + while let Some(at) = rest.find(needle) { + let after = &rest[at + needle.len()..]; + let continues_hostname = after + .chars() + .next() + .is_some_and(|next| next.is_ascii_alphanumeric() || next == '.' || next == '-'); + out.push_str(&rest[..at]); + if continues_hostname && !needle.ends_with('/') { + out.push_str(needle); + } else { + out.push_str(replacement); + changed = true; + } + rest = after; + } + out.push_str(rest); + (out, changed) +} + +/// Drop what a rewritten body invalidates, and state its new length. +/// +/// The bytes are no longer the ones the upstream hashed, so `ETag` and the digest +/// headers are claims about an entity that no longer exists -- a conditional +/// request carrying that validator would be answered `304` for content the client +/// has never seen. `Content-Length` is the other half: it described the body +/// before the substitution. +pub fn restate_body_headers(headers: &mut HeaderMap, length: usize) { + headers.remove(header::ETAG); + headers.remove("content-md5"); + headers.remove("digest"); + headers.remove("repr-digest"); + if let Ok(value) = HeaderValue::from_str(&length.to_string()) { + headers.insert(header::CONTENT_LENGTH, value); + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn url(text: &str) -> reqwest::Url { + reqwest::Url::parse(text).expect("a test url parses") + } + + #[test] + fn a_url_under_the_proxied_path_loses_that_path() { + // `/v2/orders` upstream is `/orders` through Doppel, which is what + // `join_upstream` does in the other direction. + let (out, changed) = replace_addresses( + r#"{"next":"https://api.example.com/v2/orders?page=2"}"#, + &url("https://api.example.com/v2/"), + &url("http://127.0.0.1:8080/"), + ); + + assert!(changed); + assert_eq!(out, r#"{"next":"http://127.0.0.1:8080/orders?page=2"}"#); + } + + #[test] + fn a_url_on_the_same_host_outside_the_path_keeps_its_own() { + let (out, changed) = replace_addresses( + r#"in"#, + &url("https://api.example.com/v2/"), + &url("http://127.0.0.1:8080/"), + ); + + assert!(changed); + assert_eq!(out, r#"in"#); + } + + #[test] + fn a_subdomain_is_a_different_host_and_is_left_alone() { + let (out, changed) = replace_addresses( + r#""#, + &url("https://api.example.com/"), + &url("http://127.0.0.1:8080/"), + ); + + assert!(!changed, "{out}"); + assert!(out.contains("cdn.api.example.com"), "{out}"); + } + + #[test] + fn a_host_that_merely_starts_the_same_is_not_rewritten() { + // The one that would be a security bug rather than a cosmetic one: a plain + // string replacement turns this into Doppel's address with `.evil.test` + // glued on the end, which is a URL pointing somewhere nobody named. + let (out, changed) = replace_addresses( + r#"out"#, + &url("https://api.example.com/"), + &url("http://127.0.0.1:8080/"), + ); + + assert!(!changed, "{out}"); + assert_eq!( + out, + r#"out"# + ); + } + + #[test] + fn a_port_is_part_of_the_host_it_matches() { + let (out, changed) = replace_addresses( + "see http://127.0.0.1:9000/thing", + &url("http://127.0.0.1:9000/"), + &url("https://doppel.example.com/"), + ); + + assert!(changed); + assert_eq!(out, "see https://doppel.example.com/thing"); + // And a different port on the same address is a different upstream. + let (out, changed) = replace_addresses( + "see http://127.0.0.1:9001/thing", + &url("http://127.0.0.1:9000/"), + &url("https://doppel.example.com/"), + ); + assert!(!changed, "{out}"); + } + + #[test] + fn doppels_own_path_prefix_survives() { + let (out, changed) = replace_addresses( + "https://api.example.com/v2/orders", + &url("https://api.example.com/v2/"), + &url("https://gw.example.com/doppel/"), + ); + + assert!(changed); + assert_eq!(out, "https://gw.example.com/doppel/orders"); + } + + #[test] + fn a_body_with_nothing_to_replace_is_unchanged() { + let (out, changed) = replace_addresses( + "

no urls here

", + &url("https://api.example.com/"), + &url("http://127.0.0.1:8080/"), + ); + + assert!(!changed); + assert_eq!(out, "

no urls here

"); + } + + #[test] + fn only_text_and_only_uncompressed() { + let textual = |value: &str| { + let mut headers = HeaderMap::new(); + headers.insert(header::CONTENT_TYPE, HeaderValue::from_str(value).unwrap()); + is_rewritable(&headers) + }; + + assert!(textual("text/html; charset=utf-8")); + assert!(textual("application/json")); + assert!(textual("application/problem+json")); + assert!(textual("image/svg+xml")); + assert!(textual("TEXT/CSS")); + assert!(!textual("image/png")); + assert!(!textual("application/octet-stream")); + assert!(!textual("font/woff2")); + + // No content type at all: nothing says this is text. + assert!(!is_rewritable(&HeaderMap::new())); + + // Compressed, and the client asked the upstream for that encoding. + let mut gzipped = HeaderMap::new(); + gzipped.insert(header::CONTENT_TYPE, HeaderValue::from_static("text/html")); + gzipped.insert(header::CONTENT_ENCODING, HeaderValue::from_static("gzip")); + assert!(!is_rewritable(&gzipped)); + } + + #[test] + fn a_rewritten_body_loses_the_validators_it_invalidated() { + let mut headers = HeaderMap::new(); + headers.insert(header::ETAG, HeaderValue::from_static("\"abc\"")); + headers.insert(header::CONTENT_LENGTH, HeaderValue::from_static("11")); + headers.insert("digest", HeaderValue::from_static("sha-256=...")); + + restate_body_headers(&mut headers, 42); + + // A conditional request carrying the upstream's ETag would otherwise be + // answered 304 for content the client has never seen. + assert!(!headers.contains_key(header::ETAG)); + assert!(!headers.contains_key("digest")); + assert_eq!(headers.get(header::CONTENT_LENGTH).unwrap(), "42"); + } +} diff --git a/crates/doppel-proxy/src/server.rs b/crates/doppel-proxy/src/server.rs index 872e91d..4805dfb 100644 --- a/crates/doppel-proxy/src/server.rs +++ b/crates/doppel-proxy/src/server.rs @@ -21,6 +21,15 @@ use crate::upstream::{error_response, forward}; pub struct ProxyState { pub holder: Arc, pub sampler: Arc, + /// Where clients reach this Doppel, from `server.external_url` or + /// `DOPPEL_EXTERNAL_URL`, when the deployment says. + /// + /// Resolved once at startup rather than read from the running configuration, + /// like the rest of `server`: the listeners are bound before the first + /// reload, and a reload that changes that section reports it as unapplied. + /// Behind an `Arc` because this struct is cloned per request and a `Url` is + /// a `String` behind the scenes. + pub external_url: Option>, } impl ProxyState { @@ -29,8 +38,16 @@ impl ProxyState { Self { holder, sampler: Arc::new(OsSampler), + external_url: None, } } + + /// The same state, told where clients reach this Doppel. + #[must_use] + pub fn with_external_url(mut self, external: Option) -> Self { + self.external_url = external.map(Arc::new); + self + } } /// Every method and every path go to one handler: this is a proxy, not an app. @@ -85,7 +102,9 @@ async fn handle( method.as_str(), response.status().as_u16(), elapsed, + metrics::Outcome::proxied(), ); + metrics::record_proxy_error(err.code.as_str()); // `upstream_contacted` is a bool, not an `Option`-typed null: the // `tracing::Value` impl for `Option` records nothing at all // when the value is `None`, so the JSON layer omits a null field @@ -178,7 +197,17 @@ async fn handle( // mocks that answered. The mock is named in the log line // instead, which is where "whose loss fired" belongs. metrics::record_loss(&proxy.name); - metrics::record_proxy(&proxy.name, method.as_str(), status, elapsed); + metrics::record_proxy( + &proxy.name, + method.as_str(), + status, + elapsed, + metrics::Outcome { + replace: true, + loss: true, + upstream_error: false, + }, + ); tracing::info!( request_id, proxy = proxy.name, @@ -216,7 +245,11 @@ async fn handle( method.as_str(), response.status().as_u16(), elapsed, + metrics::Outcome::mocked(), ); + if let Some(code) = error_code { + metrics::record_proxy_error(code); + } // `upstream_contacted` is false here and the two upstream fields // are absent: a served mock never reaches the upstream, which is // precisely the distinction that field exists to record. @@ -254,7 +287,13 @@ async fn handle( ); let elapsed = started.elapsed(); metrics::record_loss(&proxy.name); - metrics::record_proxy(&proxy.name, method.as_str(), status, elapsed); + metrics::record_proxy( + &proxy.name, + method.as_str(), + status, + elapsed, + metrics::Outcome::lost(), + ); tracing::info!( request_id, proxy = proxy.name, @@ -290,6 +329,7 @@ async fn handle( Some(peer.ip()), &runtime.resolve_headers, &request_id, + state.external_url.as_deref(), ) .await; let latency_ms = pad_to_target(faults.latency, attempt.elapsed(), &proxy.name).await; @@ -308,6 +348,13 @@ async fn handle( method.as_str(), response.status().as_u16(), elapsed, + metrics::Outcome { + // The upstream answered; whether it answered usefully is its + // status. A 5xx relayed from upstream is an upstream error + // even though the exchange itself succeeded. + upstream_error: outcome.status >= 500, + ..metrics::Outcome::proxied() + }, ); tracing::info!( request_id, @@ -336,7 +383,16 @@ async fn handle( method.as_str(), response.status().as_u16(), elapsed, + metrics::Outcome { + // A transport failure, a timeout, or a request refused on the + // way in. The last one never reached the upstream, and is + // still an upstream error in the only sense a dashboard cares + // about: this proxy could not deliver. + upstream_error: true, + ..metrics::Outcome::proxied() + }, ); + metrics::record_proxy_error(err.code.as_str()); // `forward` validates the request path before it ever opens a // connection (see `join_upstream`), so a 4xx here (always // `InvalidRequestPath` today) means the request was rejected on @@ -599,6 +655,7 @@ proxies: ProxyState { holder: Arc::new(RuntimeHolder::new(runtime)), sampler: Arc::new(crate::fault::SequenceSampler::new(samples)), + external_url: None, } } diff --git a/crates/doppel-proxy/src/upstream.rs b/crates/doppel-proxy/src/upstream.rs index ecfc878..10f0336 100644 --- a/crates/doppel-proxy/src/upstream.rs +++ b/crates/doppel-proxy/src/upstream.rs @@ -147,6 +147,9 @@ pub async fn forward( peer: Option, resolve_headers: &[String], request_id: &str, + // Doppel's own address, when the deployment knows it. Only redirect + // rewriting reads it -- see `rewritten_location`. + external: Option<&reqwest::Url>, ) -> Result<(Response, UpstreamOutcome), Error> { let (parts, body) = request.into_parts(); let url = join_upstream(&proxy.base_url, parts.uri.path(), parts.uri.query())?; @@ -211,11 +214,28 @@ pub async fn forward( let status = response.status(); let mut relayed = sanitize_headers(response.headers()); if proxy.rewrite_redirects - && let Some(rewritten) = rewritten_location(&relayed, &url, &proxy.base_url) + && let Some(rewritten) = rewritten_location(&relayed, &url, &proxy.base_url, external) { relayed.insert("location", rewritten); } - let stream = response.bytes_stream(); + // The body, and the length it has if the upstream's own address had to come out + // of it. See `crate::rewrite`: text only, uncompressed only, bounded by + // `body_limit`, and the exact host only. + let (body, rewritten_length) = match external { + Some(external) if proxy.rewrite_urls && crate::rewrite::is_rewritable(&relayed) => { + crate::rewrite::rewrite_body( + response.bytes_stream(), + &proxy.base_url, + external, + proxy.body_limit, + ) + .await + } + _ => (Body::from_stream(response.bytes_stream()), None), + }; + if let Some(length) = rewritten_length { + crate::rewrite::restate_body_headers(&mut relayed, length); + } let mut builder = Response::builder().status(status); if let Some(headers) = builder.headers_mut() { @@ -227,7 +247,7 @@ pub async fn forward( headers.insert("x-request-id", value); } } - let response = builder.body(Body::from_stream(stream)).map_err(|e| { + let response = builder.body(body).map_err(|e| { Error::new( ErrorCode::UpstreamError, format!("cannot relay response: {e}"), @@ -300,19 +320,27 @@ fn apply_forwarded_for(target: &mut HeaderMap, original: &HeaderMap, peer: Optio /// default, because forgetting it is so easy. /// /// The value is resolved against the URL that was actually requested upstream, -/// which is what the upstream meant by it, and then judged: +/// which is what the upstream meant by it, and then judged against the proxy's +/// base and against `external`, Doppel's own address when one is configured: /// -/// - Under `base` -- same origin, and a path below its path prefix -- it is -/// returned as a root-relative path plus query and fragment. Relative rather -/// than absolute so Doppel never has to guess its own public name: a client -/// resolves it against the URL it used, which is Doppel's. RFC 9110 has -/// allowed a relative `Location` since 7231 superseded 2616. -/// - Anywhere else it is returned in absolute form. That target is genuinely -/// elsewhere and rewriting it would be a lie, but stating it absolutely -/// removes a second, quieter bug: a root-relative `Location` meant against the -/// upstream's root (`/login` under a base of `/api/v1/`) would otherwise be -/// re-resolved against Doppel and forwarded back as `/api/v1/login`, which is -/// a different resource nobody asked for. +/// - **Under `base`** -- same origin, and a path below its path prefix. With no +/// `external`, a root-relative path plus query and fragment, so Doppel never +/// has to guess its own public name: a client resolves it against the URL it +/// used. RFC 9110 has allowed a relative `Location` since 7231 superseded +/// 2616. With `external`, the same tail against that address, absolutely. +/// - **Same origin, outside `base`** -- `/login` under a base of `/api/v1/`. +/// Without `external` this is relayed absolutely, pointing at the upstream: +/// a root-relative form would be re-resolved against Doppel and forwarded +/// back as `/api/v1/login`, a resource nobody asked for, and there is no host +/// to name instead. With `external` the client is kept on Doppel -- +/// `https://doppel.example.com/login` -- because a client that leaves is a +/// client past every fault and mock, which is the whole point of the rewrite. +/// The path is kept as the upstream wrote it, which is what `nginx`'s +/// `proxy_redirect` does; whether Doppel serves that path is a question about +/// the configuration, and one the operator can see. +/// - **Anywhere else** -- another host entirely, and left alone in every case. +/// Doppel has no route to it, and naming itself in a redirect to somewhere it +/// does not proxy would be a lie. /// /// Not covered: `Content-Location` names where a payload lives rather than where /// to go next, `Refresh` is not a standard header, and `Set-Cookie`'s `Domain` @@ -321,34 +349,30 @@ fn rewritten_location( relayed: &HeaderMap, requested: &reqwest::Url, base: &reqwest::Url, + external: Option<&reqwest::Url>, ) -> Option { let location = relayed.get("location")?.to_str().ok()?; let resolved = requested.join(location).ok()?; let base_path = base.path(); - let under_base = resolved.origin() == base.origin() + let same_origin = resolved.origin() == base.origin(); + let under_base = same_origin && resolved.path().starts_with(base_path) // A base path is a directory prefix, so `/api/v1x` must not count as // being under `/api/v1/`. `join_upstream` treats the base the same way. && (base_path.ends_with('/') || resolved.path().len() == base_path.len()); let value = if under_base { - let mut tail = String::from("/"); - tail.push_str( - resolved - .path() - .trim_start_matches(base_path) - .trim_start_matches('/'), - ); - if let Some(query) = resolved.query() { - tail.push('?'); - tail.push_str(query); + let tail = tail_after(&resolved, base_path); + match external { + Some(external) => under(external, &tail), + None => format!("/{tail}"), } - if let Some(fragment) = resolved.fragment() { - tail.push('#'); - tail.push_str(fragment); + } else if same_origin { + match external { + Some(external) => under(external, &tail_after(&resolved, "/")), + None => resolved.to_string(), } - tail } else { resolved.to_string() }; @@ -361,6 +385,38 @@ fn rewritten_location( HeaderValue::from_str(&value).ok() } +/// The part of `url` after `prefix`, with its query and fragment, and no +/// leading slash. +fn tail_after(url: &reqwest::Url, prefix: &str) -> String { + let mut tail = url + .path() + .trim_start_matches(prefix) + .trim_start_matches('/') + .to_owned(); + if let Some(query) = url.query() { + tail.push('?'); + tail.push_str(query); + } + if let Some(fragment) = url.fragment() { + tail.push('#'); + tail.push_str(fragment); + } + tail +} + +/// `tail` under `external`, keeping any path `external` itself carries -- a +/// Doppel reached at `https://gw.example.com/doppel/` is reached there in a +/// rewritten `Location` too. +/// +/// Built by hand rather than with `Url::join`, which reads a leading slash as +/// "from the root" and would drop that prefix. +fn under(external: &reqwest::Url, tail: &str) -> String { + let mut out = external.as_str().trim_end_matches('/').to_owned(); + out.push('/'); + out.push_str(tail); + out +} + /// Records what the client asked for, since `Host` is not relayed: without /// these the upstream cannot tell a request through Doppel from a direct one, /// and cannot build a URL that points back here. @@ -765,6 +821,7 @@ mod tests { latency: None, replace: 1.0, rewrite_redirects: true, + rewrite_urls: false, resolve_header: None, mocks: Vec::new(), body_limit: 1024 * 1024, @@ -818,7 +875,27 @@ mod tests { request: axum::extract::Request, peer: Option, ) -> Result<(Response, UpstreamOutcome), Error> { - forward(client, proxy, request, peer, &[], TEST_REQUEST_ID).await + forward(client, proxy, request, peer, &[], TEST_REQUEST_ID, None).await + } + + /// `fwd` for the tests that are about `server.external_url`. + async fn fwd_external( + client: &reqwest::Client, + proxy: &doppel_core::CompiledProxy, + request: axum::extract::Request, + external: &str, + ) -> Result<(Response, UpstreamOutcome), Error> { + let external = reqwest::Url::parse(external).expect("a test url parses"); + forward( + client, + proxy, + request, + None, + &[], + TEST_REQUEST_ID, + Some(&external), + ) + .await } #[tokio::test] @@ -1186,6 +1263,137 @@ mod tests { ); } + /// With an external url, the same redirect names Doppel outright. + /// + /// This is what a deployment behind a port mapping or an ingress needs: the + /// relative form is correct but only a client that resolves it against the + /// URL it used benefits, and anything logging or storing the `Location` -- + /// a test report, a browser's history, a client that just prints it -- gets + /// a path with no host in it. + #[tokio::test] + async fn an_external_url_puts_doppels_own_host_in_the_location() { + let base = upstream().await; + let client = reqwest::Client::builder() + .redirect(reqwest::redirect::Policy::none()) + .build() + .unwrap(); + + let (response, _) = fwd_external( + &client, + &proxy(&base), + request(Method::GET, "/redirect/self"), + "https://doppel.example.com/", + ) + .await + .unwrap(); + assert_eq!( + location_of(response).await, + "https://doppel.example.com/moved?keep=1#frag" + ); + } + + /// The case the relative form cannot express, and the reason this setting + /// exists: the upstream sends the client to its own host *outside* the + /// proxied path. Without an external url that is relayed as the upstream's + /// own absolute URL and the client leaves Doppel -- past every fault and + /// every mock. With one, the client stays. + #[tokio::test] + async fn a_redirect_outside_the_base_is_kept_on_doppel_when_it_has_a_name() { + let base = upstream().await; + let client = reqwest::Client::builder() + .redirect(reqwest::redirect::Policy::none()) + .build() + .unwrap(); + + let mut with_prefix = proxy(&base); + with_prefix.base_url = url(&format!("{base}redirect/")); + + let (response, _) = fwd_external( + &client, + &with_prefix, + request(Method::GET, "/rooted"), + "https://doppel.example.com/", + ) + .await + .unwrap(); + // The upstream said `/moved` against its own root. The path is kept as + // the upstream wrote it, which is what nginx's `proxy_redirect` does. + assert_eq!( + location_of(response).await, + "https://doppel.example.com/moved" + ); + } + + /// A Doppel reached under a path prefix keeps it. `Url::join` would not: + /// a leading slash reads as "from the root" and drops the prefix. + #[tokio::test] + async fn an_external_url_with_a_path_keeps_that_path() { + let base = upstream().await; + let client = reqwest::Client::builder() + .redirect(reqwest::redirect::Policy::none()) + .build() + .unwrap(); + + let (response, _) = fwd_external( + &client, + &proxy(&base), + request(Method::GET, "/redirect/self"), + "https://gw.example.com/doppel/", + ) + .await + .unwrap(); + assert_eq!( + location_of(response).await, + "https://gw.example.com/doppel/moved?keep=1#frag" + ); + } + + /// Another host stays another host, external url or not. Doppel does not + /// proxy `example.invalid` and saying otherwise would send the client to + /// itself for something it cannot serve. + #[tokio::test] + async fn a_foreign_host_is_left_alone_even_with_an_external_url() { + let base = upstream().await; + let client = reqwest::Client::builder() + .redirect(reqwest::redirect::Policy::none()) + .build() + .unwrap(); + + let (response, _) = fwd_external( + &client, + &proxy(&base), + request(Method::GET, "/redirect"), + "https://doppel.example.com/", + ) + .await + .unwrap(); + assert_eq!(location_of(response).await, "http://example.invalid/target"); + } + + /// `rewrite_redirects: false` still means no rewriting at all. + #[tokio::test] + async fn an_external_url_does_not_override_rewrite_redirects_false() { + let base = upstream().await; + let client = reqwest::Client::builder() + .redirect(reqwest::redirect::Policy::none()) + .build() + .unwrap(); + + let (response, _) = fwd_external( + &client, + &proxy_relaying_redirects(&base), + request(Method::GET, "/redirect/self"), + "https://doppel.example.com/", + ) + .await + .unwrap(); + let location = location_of(response).await; + assert!( + location.starts_with("http://127.0.0.1:"), + "expected the upstream's own authority, got {location}" + ); + } + /// A `Location` relative to the current path resolves inside the base, so it /// comes back relative -- and correctly, which relaying would not have been: /// the upstream meant `sibling` next to `/redirect/relative`. @@ -1313,6 +1521,7 @@ mod tests { None, &resolve_headers, TEST_REQUEST_ID, + None, ) .await .unwrap(); @@ -1339,6 +1548,7 @@ mod tests { None, &[], "caller-chosen-id", + None, ) .await .unwrap(); @@ -1362,6 +1572,7 @@ mod tests { None, &[], "caller-chosen-id", + None, ) .await .unwrap(); diff --git a/crates/doppel-store-postgres/migrations/0005_settings_as_jsonb.sql b/crates/doppel-store-postgres/migrations/0005_settings_as_jsonb.sql new file mode 100644 index 0000000..d2ee4ac --- /dev/null +++ b/crates/doppel-store-postgres/migrations/0005_settings_as_jsonb.sql @@ -0,0 +1,218 @@ +-- Store the configuration as JSON instead of a column per field. +-- +-- Why: 0002, 0003 and 0004 each added one column and nothing else. Their whole +-- content was `ADD COLUMN`, plus a matching edit in load.rs, in save.rs, and in +-- two hand-written statements a test existed solely to keep in step. Nothing +-- ever queried `admin_host` or `latency_min` in SQL. A column per field bought +-- nothing and charged a migration for every field the configuration format +-- gained. +-- +-- What stays a column is what SQL uses: `name` and `revision` for the +-- compare-and-swap, `ordinal` for proxy order, `config` for the foreign key. +-- +-- The backfill below has one job, and it is not cosmetic: the revision of a +-- stored configuration is a hash of its canonical YAML, and every write +-- compare-and-swaps against it. If the JSON these statements build parsed back +-- into a configuration that differed in any way from the one the old columns +-- described, the revision would change, and the next write from a client holding +-- the old revision would be refused as a conflict that never happened. So the +-- reconstruction has to be exact, including the difference between a key that is +-- absent and one that is present and null -- `jsonb_strip_nulls` is what removes +-- the latter, and the sections that are wholly optional (`sentry`, `loss`, +-- `latency`) are built conditionally rather than stripped, because an empty +-- object is not the same as a missing one for a struct with required fields. +-- +-- One property is given up here: the unique index on (config, token) that made +-- duplicate token values impossible at the database level. Rule V26 refuses them +-- while the document is validated, which every write goes through, so the +-- guarantee moves rather than disappears. + +-- --------------------------------------------------------------------------- +-- configurations +-- --------------------------------------------------------------------------- + +ALTER TABLE configurations ADD COLUMN settings JSONB; + +UPDATE configurations c +SET settings = jsonb_strip_nulls( + jsonb_build_object( + 'server', jsonb_build_object( + 'host', c.server_host, + 'port', c.server_port + ), + 'logging', jsonb_build_object( + 'level', c.log_level, + 'format', c.log_format + ), + 'control', jsonb_build_object('socket', c.control_socket), + 'templates', jsonb_build_object('dir', c.templates_dir), + 'admin', jsonb_strip_nulls( + jsonb_build_object( + 'enable', c.admin_enable, + 'host', c.admin_host, + 'port', c.admin_port, + 'auth', jsonb_build_object('header', c.admin_auth_header), + 'upload', jsonb_build_object('limit', c.admin_upload_limit), + 'access', c.admin_access, + 'public', c.admin_public, + 'groups', c.admin_groups + ) + ) || jsonb_build_object( + -- Built separately because `||` is how a key is added to an + -- already-stripped object without stripping this one: an empty + -- token list must survive as `[]`. + 'tokens', COALESCE( + ( + SELECT jsonb_agg( + jsonb_build_object( + 'name', t.name, + 'group', t."group", + 'token', t.token + ) + ORDER BY t.ordinal + ) + FROM admin_tokens t + WHERE t.config = c.name + ), + '[]'::jsonb + ) + ) + ) + -- `sentry` is an optional section with a required `dsn`, so it is either + -- a whole object or absent. Stripping nulls from `{"dsn": null}` would + -- leave `{}`, which fails to parse. + || CASE + WHEN c.sentry_dsn IS NULL THEN '{}'::jsonb + ELSE jsonb_build_object('sentry', jsonb_build_object('dsn', c.sentry_dsn)) + END + ); + +ALTER TABLE configurations ALTER COLUMN settings SET NOT NULL; + +ALTER TABLE configurations + DROP COLUMN server_host, + DROP COLUMN server_port, + DROP COLUMN log_level, + DROP COLUMN log_format, + DROP COLUMN control_socket, + DROP COLUMN templates_dir, + DROP COLUMN sentry_dsn, + DROP COLUMN admin_enable, + DROP COLUMN admin_host, + DROP COLUMN admin_port, + DROP COLUMN admin_auth_header, + DROP COLUMN admin_upload_limit, + DROP COLUMN admin_access, + DROP COLUMN admin_public, + DROP COLUMN admin_groups; + +DROP TABLE admin_tokens; + +-- --------------------------------------------------------------------------- +-- proxies, with their mocks folded in +-- --------------------------------------------------------------------------- + +ALTER TABLE proxies ADD COLUMN document JSONB; + +UPDATE proxies p +SET document = jsonb_strip_nulls( + jsonb_build_object( + 'name', p.name, + 'type', p.kind, + 'url', p.url, + 'timeout', p.timeout_seconds, + 'body_limit', p.body_limit, + 'replace', p.replace_ratio, + 'rewrite_redirects', p.rewrite_redirects, + -- `type`, not `kind`: the Rust field is `kind` and the wire name is + -- `type`, and this document is read by the wire name. + 'resolve', jsonb_strip_nulls( + jsonb_build_object('type', p.resolve_kind, 'header', p.resolve_header) + ), + 'headers', p.headers, + 'access', p.access + ) + || CASE + WHEN p.loss_percentage IS NULL THEN '{}'::jsonb + ELSE jsonb_build_object( + 'loss', + jsonb_build_object( + 'percentage', p.loss_percentage, + 'status', p.loss_status + ) + ) + END + || CASE + WHEN p.latency_percentage IS NULL THEN '{}'::jsonb + ELSE jsonb_build_object( + 'latency', + jsonb_build_object( + 'percentage', p.latency_percentage, + 'min', p.latency_min, + 'max', p.latency_max + ) + ) + END + || jsonb_build_object( + -- `[]` rather than absent. `save` omits an empty mock list and + -- this writes it out, which is a difference in the JSON and not a + -- difference in the configuration: both parse to the same empty + -- vector. That is the invariant the whole backfill is held to -- + -- the document must *parse into* the configuration the old + -- columns described, not match `save` byte for byte -- and it is + -- what keeps every stored revision unchanged. + 'mocks', COALESCE( + ( + SELECT jsonb_agg( + jsonb_strip_nulls( + jsonb_build_object( + 'name', m.name, + 'request', jsonb_build_object( + 'method', m.method, + 'url', m.url_pattern, + 'headers', m.request_headers, + 'query', m.request_query, + 'body', m.request_body + ), + 'response', jsonb_strip_nulls( + jsonb_build_object( + 'status', m.status, + 'body', m.body, + 'json', m.json, + 'template', m.template, + 'headers', m.response_headers + ) + ), + 'proxy', m.proxy_override + ) + ) + ORDER BY m.ordinal + ) + FROM mocks m + WHERE m.config = p.config AND m.proxy = p.name + ), + '[]'::jsonb + ) + ) + ); + +ALTER TABLE proxies ALTER COLUMN document SET NOT NULL; + +ALTER TABLE proxies + DROP COLUMN kind, + DROP COLUMN url, + DROP COLUMN timeout_seconds, + DROP COLUMN body_limit, + DROP COLUMN replace_ratio, + DROP COLUMN rewrite_redirects, + DROP COLUMN resolve_kind, + DROP COLUMN resolve_header, + DROP COLUMN loss_percentage, + DROP COLUMN loss_status, + DROP COLUMN latency_percentage, + DROP COLUMN latency_min, + DROP COLUMN latency_max, + DROP COLUMN headers, + DROP COLUMN access; + +DROP TABLE mocks; diff --git a/crates/doppel-store-postgres/src/lib.rs b/crates/doppel-store-postgres/src/lib.rs index 8f80911..48ec9c1 100644 --- a/crates/doppel-store-postgres/src/lib.rs +++ b/crates/doppel-store-postgres/src/lib.rs @@ -190,6 +190,10 @@ impl doppel_core::store::ConfigStore for PostgresStore { self.retain_template_rows(proxy, keep).await } + async fn rename_templates(&self, from: &str, to: &str) -> Result<(), StoreError> { + self.rename_template_rows(from, to).await + } + async fn materialize_templates(&self, _dir: &std::path::Path) -> Result<(), StoreError> { // The directory is `self.templates_dir`, fixed when the store was // opened, so the parameter is redundant here. It exists on the trait diff --git a/crates/doppel-store-postgres/src/load.rs b/crates/doppel-store-postgres/src/load.rs index c547a83..d630faa 100644 --- a/crates/doppel-store-postgres/src/load.rs +++ b/crates/doppel-store-postgres/src/load.rs @@ -1,11 +1,13 @@ -//! Reading a configuration back out of the five tables. - -use doppel_core::config::{ - AccessConfig, AdminConfig, AuthConfig, ByteSize, Config, ControlConfig, LatencyConfig, - LogFormat, LogLevel, LoggingConfig, LossConfig, MockConfig, MockProxyOverride, MockRequest, - MockResponse, ProxyAccessConfig, ProxyConfig, ProxyKind, ResolveConfig, ResolveKind, - SentryConfig, ServerConfig, TemplatesConfig, TokenConfig, UploadConfig, -}; +//! Reading a configuration back out of two tables. +//! +//! Both tables hold JSON documents, so this file has no per-column mapping and +//! no per-column checks: `serde_json::from_value` runs the same `Deserialize` +//! impls the YAML loader runs, which is what makes a row read here subject to +//! exactly the rules a configuration file is subject to. The previous version of +//! this file carried twenty-five helpers doing that job by hand, one per column +//! type, and every new configuration field needed another edit here. + +use doppel_core::config::{Config, ProxyConfig}; use doppel_core::store::{Revision, StoreError}; use sqlx::{Row, postgres::PgRow}; @@ -13,10 +15,6 @@ use crate::PostgresStore; impl PostgresStore { /// Read this store's configuration. - /// - /// Public rather than crate-private because the `ConfigStore` impl that - /// will delegate to it lands with `save` (task 5), and a method nothing - /// can call is a method nothing tests. pub async fn load_config(&self) -> Result<(Config, Revision), StoreError> { // One transaction at `REPEATABLE READ`, so every statement below sees // the same snapshot. @@ -41,7 +39,7 @@ impl PostgresStore { .await .map_err(query_failed)?; - let row = sqlx::query("SELECT * FROM configurations WHERE name = $1") + let row = sqlx::query("SELECT revision, settings FROM configurations WHERE name = $1") .bind(&self.config_name) .fetch_optional(&mut *tx) .await @@ -54,21 +52,17 @@ impl PostgresStore { .to_ne_bytes(), )); - let config = Config { - server: server_from(&row)?, - logging: logging_from(&row)?, - control: control_from(&row)?, - templates: templates_from(&row)?, - sentry: sentry_from(&row)?, - admin: self.admin_from(&mut tx, &row).await?, - proxies: self.proxies(&mut tx).await?, - }; + // The stored document omits `proxies`, which is a `#[serde(default)]` + // field, so this yields a configuration with an empty proxy list and the + // rows below fill it in. + let mut config: Config = document(&row, "settings", "settings")?; + config.proxies = self.proxies(&mut tx).await?; // The stored revision has to agree with the content it labels. They // diverge when a row was edited by hand, or when this code forgot to - // read a column that `save` wrote -- and either way every - // compare-and-swap downstream would then fail in a way that looks - // like contention rather than like corruption. + // read something `save` wrote -- and either way every compare-and-swap + // downstream would then fail in a way that looks like contention rather + // than like corruption. let computed = Revision::of_config(&config); if computed != stored_revision { return Err(StoreError::Invalid(vec![doppel_core::Violation::new( @@ -83,406 +77,75 @@ impl PostgresStore { Ok((config, stored_revision)) } - async fn admin_from( - &self, - tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, - row: &PgRow, - ) -> Result { - let tokens = sqlx::query( - "SELECT name, \"group\", token FROM admin_tokens WHERE config = $1 ORDER BY ordinal", - ) - .bind(&self.config_name) - .fetch_all(&mut **tx) - .await - .map_err(query_failed)? - .into_iter() - .map(|token| { - Ok(TokenConfig { - name: name(&token, "name")?, - group: name(&token, "group")?, - token: stored_token(&token, "token")?, - }) - }) - .collect::, StoreError>>()?; - - Ok(AdminConfig { - enable: row.try_get("admin_enable").map_err(query_failed)?, - host: parse_host(&text(row, "admin_host")?)?, - port: port(row, "admin_port")?, - auth: AuthConfig { - header: header_name(row, "admin_auth_header")?, - }, - tokens, - access: json_column::(row, "admin_access")?, - public: row.try_get("admin_public").map_err(query_failed)?, - groups: optional_json::>(row, "admin_groups")?, - upload: UploadConfig { - limit: byte_size(row, "admin_upload_limit")?, - }, - }) - } - async fn proxies( &self, tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, ) -> Result, StoreError> { - let rows = sqlx::query("SELECT * FROM proxies WHERE config = $1 ORDER BY ordinal") - .bind(&self.config_name) - .fetch_all(&mut **tx) - .await - .map_err(query_failed)?; + let rows = + sqlx::query("SELECT name, document FROM proxies WHERE config = $1 ORDER BY ordinal") + .bind(&self.config_name) + .fetch_all(&mut **tx) + .await + .map_err(query_failed)?; let mut proxies = Vec::with_capacity(rows.len()); for row in &rows { - let name = name(row, "name")?; - proxies.push(ProxyConfig { - mocks: self.mocks(tx, name.as_str()).await?, - name, - kind: match text(row, "kind")?.as_str() { - "http" => ProxyKind::Http, - // Including `tcp`, which earlier versions of this schema - // could store: the variant is gone, so a row holding it is - // a configuration this binary cannot serve, reported rather - // than silently coerced to `http`. - other => return Err(corrupt("proxies.kind", &format!("is `{other}`"))), - }, - url: url(row, "url")?, - timeout: timeout(row, "timeout_seconds")?, - resolve: ResolveConfig { - kind: match text(row, "resolve_kind")?.as_str() { - "default" => ResolveKind::Default, - "header" => ResolveKind::Header, - other => { - return Err(corrupt("proxies.resolve_kind", &format!("is `{other}`"))); - } - }, - header: optional_header_name(row, "resolve_header")?, - }, - access: optional_json::(row, "access")?, - headers: json_column(row, "headers")?, - loss: loss_from(row)?, - latency: latency_from(row)?, - replace: optional_ratio(row, "replace_ratio")?, - rewrite_redirects: row.try_get("rewrite_redirects").map_err(query_failed)?, - body_limit: byte_size(row, "body_limit")?, - }); + // The keyed name is read first so a document that fails to parse can + // be reported against the proxy it belongs to. serde names the field + // it could not read (`missing field 'max'`) but not the row, and + // "some proxy is malformed" is not a report anybody can act on. + let keyed: String = row.try_get("name").map_err(query_failed)?; + let proxy: ProxyConfig = + document(row, "document", &format!("proxies[{keyed}].document"))?; + + // The name is both a column and a field of the document, because the + // column is the primary key and the field is what the configuration + // format holds. A row whose two disagree is addressable under one + // name and serves another, so it is reported rather than resolved in + // favour of either. + if proxy.name.as_str() != keyed { + return Err(corrupt( + "proxies.name", + &format!( + "row `{keyed}` holds a document naming `{}`", + proxy.name.as_str() + ), + )); + } + + proxies.push(proxy); } Ok(proxies) } - - async fn mocks( - &self, - tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, - proxy: &str, - ) -> Result, StoreError> { - sqlx::query("SELECT * FROM mocks WHERE config = $1 AND proxy = $2 ORDER BY ordinal") - .bind(&self.config_name) - .bind(proxy) - .fetch_all(&mut **tx) - .await - .map_err(query_failed)? - .iter() - .map(|row| { - Ok(MockConfig { - name: name(row, "name")?, - request: MockRequest { - method: method(row, "method")?, - url: pattern(row, "url_pattern")?, - headers: json_column(row, "request_headers")?, - query: json_column(row, "request_query")?, - body: json_column(row, "request_body")?, - }, - response: MockResponse { - status: status(row, "status")?, - body: row.try_get("body").map_err(query_failed)?, - json: row.try_get("json").map_err(query_failed)?, - template: optional_template_name(row, "template")?, - headers: json_column(row, "response_headers")?, - }, - proxy: optional_json::(row, "proxy_override")?, - }) - }) - .collect() - } -} - -fn server_from(row: &PgRow) -> Result { - Ok(ServerConfig { - host: parse_host(&text(row, "server_host")?)?, - port: port(row, "server_port")?, - }) -} - -fn logging_from(row: &PgRow) -> Result { - // Through serde rather than a hand-written match: the string in the - // column is the one the YAML uses, and going through the same - // deserializer keeps the two spellings from drifting apart. - Ok(LoggingConfig { - level: enum_from_text::("log_level", &text(row, "log_level")?)?, - format: enum_from_text::("log_format", &text(row, "log_format")?)?, - }) -} - -fn control_from(row: &PgRow) -> Result { - Ok(ControlConfig { - socket: text(row, "control_socket")?.into(), - }) -} - -fn templates_from(row: &PgRow) -> Result { - Ok(TemplatesConfig { - dir: text(row, "templates_dir")?.into(), - }) } -/// An absent `sentry_dsn` and an empty one both mean the section is absent. +/// A stored JSON document, parsed through the configuration's own types. /// -/// `Some(SentryConfig { dsn: String::new() })` would round-trip as -/// `sentry:\n dsn: ""`, which is a different document from one with no -/// `sentry` section -- and a different revision. -fn sentry_from(row: &PgRow) -> Result, StoreError> { - let dsn: Option = row.try_get("sentry_dsn").map_err(query_failed)?; - Ok(dsn - .filter(|dsn| !dsn.is_empty()) - .map(|dsn| SentryConfig { dsn })) -} - -/// `loss` is present exactly when both its columns are. -/// -/// One of the two set on its own is a row nobody could have written through -/// `save`, so it is reported rather than silently treated as absent. -fn loss_from(row: &PgRow) -> Result, StoreError> { - let percentage: Option = row.try_get("loss_percentage").map_err(query_failed)?; - let status: Option = row.try_get("loss_status").map_err(query_failed)?; - match (percentage, status) { - (None, None) => Ok(None), - (Some(percentage), Some(status)) => Ok(Some(LossConfig { - percentage: ratio(percentage, "proxies.loss_percentage")?, - status: narrow_status(status, "proxies.loss_status")?, - })), - _ => Err(corrupt( - "proxies.loss_*", - "has some columns set and others null", - )), - } -} - -fn latency_from(row: &PgRow) -> Result, StoreError> { - let percentage: Option = row.try_get("latency_percentage").map_err(query_failed)?; - let min: Option = row.try_get("latency_min").map_err(query_failed)?; - let max: Option = row.try_get("latency_max").map_err(query_failed)?; - match (percentage, min, max) { - (None, None, None) => Ok(None), - (Some(percentage), Some(min), Some(max)) => Ok(Some(LatencyConfig { - percentage: ratio(percentage, "proxies.latency_percentage")?, - min: seconds(min, "proxies.latency_min")?, - max: seconds(max, "proxies.latency_max")?, - })), - _ => Err(corrupt( - "proxies.latency_*", - "has some columns set and others null", - )), - } -} - -fn text(row: &PgRow, column: &str) -> Result { - row.try_get(column).map_err(query_failed) -} - -/// A stored name, checked on the way in. -/// -/// The database has no opinion about what a name may contain, so a row edited -/// by hand can hold one the configuration format would refuse. Parsing here -/// means a `Config` this store produces is subject to the same rule as one -/// read from YAML, rather than a second, laxer standard nobody wrote down. -/// Generic over the cap so one function serves both a `Name` and the tighter -/// `ProxyName`, and the column is checked against the limit that type actually -/// carries rather than against whichever one this helper happened to name. -fn name( +/// `column` is where to read it; `label` is what a failure is reported under, and +/// the two are not the same thing -- a proxy's document lives in `document` and is +/// reported as `proxies[alpha].document`, because serde names the field it could +/// not read but nothing about which row held it. +fn document( row: &PgRow, column: &str, -) -> Result, StoreError> { - let raw: String = row.try_get(column).map_err(query_failed)?; - doppel_core::config::Name::parse(raw).map_err(|err| corrupt(column, &err.to_string())) -} - -/// A stored token, checked on the way in, for the same reason as `name`. -/// -/// The error names the column and not the value: a corruption message about a -/// token is still somewhere a token would end up. -fn stored_token(row: &PgRow, column: &str) -> Result { - let raw: String = row.try_get(column).map_err(query_failed)?; - doppel_core::config::Token::parse(raw).map_err(|err| corrupt(column, &err.to_string())) -} - -/// A stored port, checked on the way in. -/// -/// `i32` is the column type, so the range check has two halves: the value has -/// to fit a `u16` at all, and then it has to be one `Port` accepts. Both are -/// reachable from a hand-edited row, and neither is reachable from YAML. -fn port(row: &PgRow, column: &str) -> Result { - let value: i32 = row.try_get(column).map_err(query_failed)?; - let narrowed = - u16::try_from(value).map_err(|_| corrupt(column, &format!("is {value}, not a port")))?; - doppel_core::config::Port::parse(narrowed).map_err(|err| corrupt(column, &err.to_string())) -} - -/// A stored status, checked on the way in. -/// -/// Two halves, as with `port`: the `i32` column has to narrow to a `u16` at -/// all, and then the number has to be one `HttpStatus` accepts. Both are -/// reachable from a hand-edited row and neither is reachable from YAML. -fn status(row: &PgRow, column: &str) -> Result { - let value: i32 = row.try_get(column).map_err(query_failed)?; - narrow_status(value, column) -} - -fn narrow_status(value: i32, column: &str) -> Result { - let narrowed = - u16::try_from(value).map_err(|_| corrupt(column, &format!("is {value}, not a status")))?; - doppel_core::config::HttpStatus::parse(narrowed) - .map_err(|err| corrupt(column, &err.to_string())) -} - -/// A stored method, checked on the way in. -fn method(row: &PgRow, column: &str) -> Result { - let raw: String = row.try_get(column).map_err(query_failed)?; - raw.parse() - .map_err(|err: doppel_core::config::MethodError| corrupt(column, &err.to_string())) -} - -/// A stored timeout, checked on the way in. -/// -/// The column is a nullable `bigint`, so both halves of the range are -/// reachable from a hand-edited row: a negative value, and a positive one -/// past what a timeout may be. -fn timeout( - row: &PgRow, - column: &str, -) -> Result, StoreError> { - optional_u64(row, column)? - .map(|value| { - doppel_core::config::TimeoutSeconds::parse(value) - .map_err(|err| corrupt(column, &err.to_string())) - }) - .transpose() -} - -/// A stored probability, checked on the way in. -fn ratio(value: f64, column: &str) -> Result { - doppel_core::config::Ratio::parse(value).map_err(|err| corrupt(column, &err.to_string())) -} - -fn optional_ratio( - row: &PgRow, - column: &str, -) -> Result, StoreError> { - let value: Option = row.try_get(column).map_err(query_failed)?; - value.map(|value| ratio(value, column)).transpose() -} - -/// A stored latency, checked on the way in. -fn seconds(value: f64, column: &str) -> Result { - doppel_core::config::Seconds::parse(value).map_err(|err| corrupt(column, &err.to_string())) -} - -/// A stored template file name, checked on the way in. -fn optional_template_name( - row: &PgRow, - column: &str, -) -> Result, StoreError> { - let raw: Option = row.try_get(column).map_err(query_failed)?; - raw.map(|raw| { - doppel_core::config::TemplateName::parse(raw) - .map_err(|err| corrupt(column, &err.to_string())) - }) - .transpose() -} - -/// A stored url pattern, checked on the way in. -fn pattern(row: &PgRow, column: &str) -> Result { - let raw: String = row.try_get(column).map_err(query_failed)?; - doppel_core::config::Pattern::parse(raw).map_err(|err| corrupt(column, &err.to_string())) -} - -/// A stored header name, checked on the way in. -fn header_name(row: &PgRow, column: &str) -> Result { - let raw: String = row.try_get(column).map_err(query_failed)?; - doppel_core::config::HeaderName::parse(raw).map_err(|err| corrupt(column, &err.to_string())) -} - -fn optional_header_name( - row: &PgRow, - column: &str, -) -> Result, StoreError> { - let raw: Option = row.try_get(column).map_err(query_failed)?; - raw.map(|raw| { - doppel_core::config::HeaderName::parse(raw).map_err(|err| corrupt(column, &err.to_string())) - }) - .transpose() -} - -/// A stored upstream URL, checked on the way in. -fn url(row: &PgRow, column: &str) -> Result { - let raw: String = row.try_get(column).map_err(query_failed)?; - doppel_core::config::UpstreamUrl::parse(&raw).map_err(|err| corrupt(column, &err.to_string())) -} - -/// A stored byte limit, checked on the way in. -fn byte_size(row: &PgRow, column: &str) -> Result { - let value: i64 = row.try_get(column).map_err(query_failed)?; - let unsigned = u64::try_from(value).map_err(|_| corrupt(column, "is negative"))?; - ByteSize::parse(unsigned).map_err(|err| corrupt(column, &err.to_string())) -} - -fn optional_u64(row: &PgRow, column: &str) -> Result, StoreError> { - let value: Option = row.try_get(column).map_err(query_failed)?; - value - .map(|value| u64::try_from(value).map_err(|_| corrupt(column, "is negative"))) - .transpose() -} - -fn json_column(row: &PgRow, column: &str) -> Result { - let value: serde_json::Value = row.try_get(column).map_err(query_failed)?; - serde_json::from_value(value).map_err(|err| corrupt(column, &err.to_string())) -} - -fn optional_json( - row: &PgRow, - column: &str, -) -> Result, StoreError> { - let value: Option = row.try_get(column).map_err(query_failed)?; - value - .map(|value| serde_json::from_value(value).map_err(|err| corrupt(column, &err.to_string()))) - .transpose() -} - -/// Deserialize an enum from the same text its YAML representation uses. -fn enum_from_text( - column: &str, - value: &str, + label: &str, ) -> Result { - serde_json::from_value(serde_json::Value::String(value.to_owned())) - .map_err(|_| corrupt(column, &format!("is `{value}`"))) -} - -fn parse_host(value: &str) -> Result { - value - .parse() - .map_err(|_| corrupt("host", &format!("`{value}` is not an IP address"))) + let value: serde_json::Value = row.try_get(column).map_err(query_failed)?; + serde_json::from_value(value).map_err(|err| corrupt(label, &err.to_string())) } fn query_failed(err: sqlx::Error) -> StoreError { StoreError::Unavailable(format!("query failed: {err}")) } -/// A row that cannot be turned into a configuration. +/// A row the database accepted and the configuration format does not. /// -/// Reported as `Invalid` rather than `Unavailable`: the database answered -/// perfectly well, and what it holds is the problem. +/// Reported as `Invalid` rather than `Unavailable`: the database is working +/// perfectly, and what is wrong is the content -- which is a different thing for +/// an operator to do something about. fn corrupt(column: &str, detail: &str) -> StoreError { StoreError::Invalid(vec![doppel_core::Violation::new( column, - format!("stored value {detail}"), + format!("stored value is not one this version accepts: {detail}"), )]) } diff --git a/crates/doppel-store-postgres/src/save.rs b/crates/doppel-store-postgres/src/save.rs index 27604c4..774075d 100644 --- a/crates/doppel-store-postgres/src/save.rs +++ b/crates/doppel-store-postgres/src/save.rs @@ -1,6 +1,18 @@ -//! Writing a configuration into the five tables. - -use doppel_core::config::{Config, ProxyConfig}; +//! Writing a configuration into two tables. +//! +//! The document is stored as JSON rather than spread across a column per field. +//! That is a deliberate reversal of the original schema, and the reason is the +//! migration it saves: between 0.3.0 and 0.4.1 three fields were added to the +//! configuration format, and each one needed a migration whose only content was +//! `ADD COLUMN` -- plus a matching edit here, in `load`, and in two hand-written +//! statements that a test existed solely to keep in step. A field added to the +//! serde model now reaches the database with no schema change at all. +//! +//! What stays a column is what SQL actually uses: `name` and `revision` for the +//! compare-and-swap, `ordinal` for proxy order, `config` for the foreign key. +//! Nothing ever queried `admin_host`. + +use doppel_core::config::Config; use doppel_core::store::{Revision, StoreError}; use doppel_core::validate::validate; use sqlx::{Postgres, Transaction}; @@ -24,6 +36,7 @@ impl PostgresStore { ) -> Result { validate(config).map_err(StoreError::Invalid)?; let revision = Revision::of_config(config); + let settings = settings_of(config)?; let mut tx = self .pool() @@ -33,10 +46,10 @@ impl PostgresStore { match expected { Some(expected) => { - self.update_header(&mut tx, config, revision, expected) - .await? + self.update_header(&mut tx, &settings, revision, expected) + .await?; } - None => self.upsert_header(&mut tx, config, revision).await?, + None => self.upsert_header(&mut tx, &settings, revision).await?, } // Deleted and rewritten rather than diffed. A diff is more code whose @@ -45,41 +58,28 @@ impl PostgresStore { // transaction either way, so the rewrite costs nothing an operator can // observe. // - // `mocks` and `admin_tokens` go with their parents through - // `ON DELETE CASCADE`; `templates` deliberately does not, so it is - // untouched here. Dropping a proxy's files is a separate decision made - // after the configuration write, never as part of it. - self.clear_children(&mut tx).await?; + // `templates` deliberately does not cascade from here: dropping a + // proxy's files is a separate decision made after the configuration + // write, never as part of it. + sqlx::query("DELETE FROM proxies WHERE config = $1") + .bind(&self.config_name) + .execute(&mut *tx) + .await + .map_err(write_failed)?; - for (ordinal, token) in config.admin.tokens.iter().enumerate() { + for (ordinal, proxy) in config.proxies.iter().enumerate() { sqlx::query( - "INSERT INTO admin_tokens (config, name, \"group\", token, ordinal) \ - VALUES ($1, $2, $3, $4, $5)", + "INSERT INTO proxies (config, name, ordinal, document) VALUES ($1, $2, $3, $4)", ) .bind(&self.config_name) - .bind(token.name.as_str()) - .bind(token.group.as_str()) - .bind(token.token.as_str()) + .bind(proxy.name.as_str()) .bind(ordinal_of(ordinal)?) + .bind(as_json(proxy)?) .execute(&mut *tx) .await .map_err(write_failed)?; } - for (ordinal, proxy) in config.proxies.iter().enumerate() { - self.insert_proxy(&mut tx, proxy, ordinal_of(ordinal)?) - .await?; - for (mock_ordinal, mock) in proxy.mocks.iter().enumerate() { - self.insert_mock( - &mut tx, - proxy.name.as_str(), - mock, - ordinal_of(mock_ordinal)?, - ) - .await?; - } - } - tx.commit() .await .map_err(|err| StoreError::Unavailable(format!("cannot commit: {err}")))?; @@ -95,7 +95,7 @@ impl PostgresStore { async fn update_header( &self, tx: &mut Transaction<'_, Postgres>, - config: &Config, + settings: &serde_json::Value, revision: Revision, expected: Revision, ) -> Result<(), StoreError> { @@ -103,7 +103,7 @@ impl PostgresStore { .bind(as_i64(revision)) .bind(&self.config_name) .bind(as_i64(expected)) - .pipe_header(config) + .bind(settings) .execute(&mut **tx) .await .map_err(write_failed)? @@ -132,231 +132,70 @@ impl PostgresStore { async fn upsert_header( &self, tx: &mut Transaction<'_, Postgres>, - config: &Config, + settings: &serde_json::Value, revision: Revision, ) -> Result<(), StoreError> { sqlx::query(UPSERT_HEADER) .bind(&self.config_name) .bind(as_i64(revision)) - .pipe_header(config) + .bind(settings) .execute(&mut **tx) .await .map_err(write_failed)?; Ok(()) } - - async fn clear_children(&self, tx: &mut Transaction<'_, Postgres>) -> Result<(), StoreError> { - // Written out rather than looped over a table name, so no SQL here is - // built by formatting. Order matters only in that `proxies` comes - // last: `mocks` cascades from it, and deleting the parent first would - // make the explicit child delete a no-op that reads as if it did - // something. - for statement in [ - "DELETE FROM mocks WHERE config = $1", - "DELETE FROM admin_tokens WHERE config = $1", - "DELETE FROM proxies WHERE config = $1", - ] { - sqlx::query(statement) - .bind(&self.config_name) - .execute(&mut **tx) - .await - .map_err(write_failed)?; - } - Ok(()) - } - - async fn insert_proxy( - &self, - tx: &mut Transaction<'_, Postgres>, - proxy: &ProxyConfig, - ordinal: i32, - ) -> Result<(), StoreError> { - sqlx::query( - "INSERT INTO proxies (config, name, ordinal, kind, url, timeout_seconds, body_limit, \ - replace_ratio, rewrite_redirects, resolve_kind, resolve_header, loss_percentage, \ - loss_status, latency_percentage, latency_min, latency_max, headers, access) \ - VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, \ - $18)", - ) - .bind(&self.config_name) - .bind(proxy.name.as_str()) - .bind(ordinal) - .bind(as_text(&proxy.kind)?) - .bind(proxy.url.as_str()) - .bind( - proxy - .timeout - .map(|t| i64::try_from(t.get()).unwrap_or(i64::MAX)), - ) - .bind(i64::try_from(proxy.body_limit.get()).unwrap_or(i64::MAX)) - .bind(proxy.replace.map(doppel_core::config::Ratio::get)) - .bind(proxy.rewrite_redirects) - .bind(as_text(&proxy.resolve.kind)?) - .bind( - proxy - .resolve - .header - .as_ref() - .map(doppel_core::config::HeaderName::as_str), - ) - .bind(proxy.loss.as_ref().map(|l| l.percentage.get())) - .bind(proxy.loss.as_ref().map(|l| i32::from(l.status.get()))) - .bind(proxy.latency.as_ref().map(|l| l.percentage.get())) - .bind(proxy.latency.as_ref().map(|l| l.min.get())) - .bind(proxy.latency.as_ref().map(|l| l.max.get())) - .bind(as_json(&proxy.headers)?) - .bind(proxy.access.as_ref().map(as_json).transpose()?) - .execute(&mut **tx) - .await - .map_err(write_failed)?; - Ok(()) - } - - async fn insert_mock( - &self, - tx: &mut Transaction<'_, Postgres>, - proxy: &str, - mock: &doppel_core::config::MockConfig, - ordinal: i32, - ) -> Result<(), StoreError> { - sqlx::query( - "INSERT INTO mocks (config, proxy, name, ordinal, method, url_pattern, status, body, \ - json, template, request_headers, request_query, request_body, response_headers, \ - proxy_override) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, \ - $14, $15)", - ) - .bind(&self.config_name) - .bind(proxy) - .bind(mock.name.as_str()) - .bind(ordinal) - .bind(mock.request.method.as_str()) - .bind(mock.request.url.as_str()) - .bind(i32::from(mock.response.status.get())) - .bind(mock.response.body.as_deref()) - .bind(mock.response.json.as_deref()) - .bind( - mock.response - .template - .as_ref() - .map(doppel_core::config::TemplateName::as_str), - ) - .bind(as_json(&mock.request.headers)?) - .bind(as_json(&mock.request.query)?) - .bind(as_json(&mock.request.body)?) - .bind(as_json(&mock.response.headers)?) - .bind(mock.proxy.as_ref().map(as_json).transpose()?) - .execute(&mut **tx) - .await - .map_err(write_failed)?; - Ok(()) - } } /// The two header statements, written out rather than assembled. /// -/// Assembling them from a shared column list would keep them in step -/// automatically, but only by building SQL with `format!` -- which sqlx -/// refuses by design, because that is where injection lives, and silencing -/// the refusal here would weaken it everywhere. They are literals instead, -/// and `the_two_header_statements_write_the_same_columns` below is what keeps -/// them in step: a column added to one and forgotten in the other would make -/// a configuration differ depending on whether it was created or updated. +/// sqlx refuses SQL built by formatting, by design, because that is where +/// injection lives -- so these are literals. There are two columns to keep in +/// step now rather than seventeen, which is most of the point of storing the +/// document as JSON. +const UPDATE_HEADER: &str = "UPDATE configurations SET revision = $1, settings = $4, \ + updated_at = now() WHERE name = $2 AND revision = $3"; + +const UPSERT_HEADER: &str = "INSERT INTO configurations (name, revision, settings) \ + VALUES ($1, $2, $3) \ + ON CONFLICT (name) DO UPDATE SET revision = $2, settings = $3, updated_at = now()"; + +/// The configuration as it is stored: the whole document except its proxies. /// -/// The bind order is the one `pipe_header` writes, and the tests below -/// check the statements against it. -const UPDATE_HEADER: &str = "UPDATE configurations SET revision = $1, \ - admin_enable = $4, server_host = $5, server_port = $6, log_level = $7, \ - log_format = $8, control_socket = $9, templates_dir = $10, sentry_dsn = $11, \ - admin_host = $12, admin_port = $13, admin_auth_header = $14, \ - admin_upload_limit = $15, admin_access = $16, admin_groups = $17, \ - admin_public = $18, updated_at = now() \ - WHERE name = $2 AND revision = $3"; - -const UPSERT_HEADER: &str = "INSERT INTO configurations \ - (name, revision, admin_enable, server_host, server_port, log_level, log_format, \ - control_socket, templates_dir, sentry_dsn, admin_host, admin_port, \ - admin_auth_header, admin_upload_limit, admin_access, admin_groups, \ - admin_public) \ - VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, \ - $17) \ - ON CONFLICT (name) DO UPDATE SET revision = $2, \ - admin_enable = $3, server_host = $4, server_port = $5, log_level = $6, \ - log_format = $7, control_socket = $8, templates_dir = $9, sentry_dsn = $10, \ - admin_host = $11, admin_port = $12, admin_auth_header = $13, \ - admin_upload_limit = $14, admin_access = $15, admin_groups = $16, \ - admin_public = $17, updated_at = now()"; - -/// Bind the header values in `HEADER_COLUMNS` order. -/// -/// A trait rather than a free function so the call reads in the same place the -/// query does: the order of these binds and the order of that list are the one -/// thing a reader has to check, and putting them side by side is what makes -/// that possible. -trait BindHeader<'q> { - fn pipe_header(self, config: &'q Config) -> Self; -} - -impl<'q> BindHeader<'q> for sqlx::query::Query<'q, Postgres, sqlx::postgres::PgArguments> { - fn pipe_header(self, config: &'q Config) -> Self { - self.bind(config.admin.enable) - .bind(config.server.host.to_string()) - .bind(i32::from(config.server.port.get())) - .bind(text_of(&config.logging.level)) - .bind(text_of(&config.logging.format)) - .bind(config.control.socket.to_string_lossy().into_owned()) - .bind(config.templates.dir.to_string_lossy().into_owned()) - .bind(config.sentry.as_ref().map(|s| s.dsn.clone())) - .bind(config.admin.host.to_string()) - .bind(i32::from(config.admin.port.get())) - .bind(config.admin.auth.header.as_str()) - .bind(i64::try_from(config.admin.upload.limit.get()).unwrap_or(i64::MAX)) - .bind(serde_json::to_value(&config.admin.access).unwrap_or(serde_json::Value::Null)) - // `None` binds SQL NULL, which is what an absent `groups` has to - // round-trip as: see the 0003 migration for why materialising the - // default would break every pre-existing configuration's revision. - .bind( - config - .admin - .groups - .as_ref() - .map(|groups| serde_json::to_value(groups).unwrap_or(serde_json::Value::Null)), - ) - .bind(config.admin.public) - } -} - -/// An enum's wire spelling, which is the one the YAML and the column share. -fn text_of(value: &T) -> String { - serde_json::to_value(value) - .ok() - .and_then(|value| value.as_str().map(ToOwned::to_owned)) - .unwrap_or_default() -} - -fn as_text(value: &T) -> Result { - serde_json::to_value(value) - .ok() - .and_then(|value| value.as_str().map(ToOwned::to_owned)) - .ok_or_else(|| StoreError::Serialize("an enum did not serialize as a string".to_owned())) +/// The proxies come out because each one is a row of its own -- they are +/// addressed by name, ordered by `ordinal`, and read back individually. Nothing +/// else is transformed, which is what makes this immune to a field being added: +/// `serde_json::to_value` follows the same model `Deserialize` reads back, so +/// there is no second description of the configuration to keep in step. +fn settings_of(config: &Config) -> Result { + let mut document = as_json(config)?; + document + .as_object_mut() + .ok_or_else(|| { + StoreError::Unavailable("a configuration did not serialize to an object".to_owned()) + })? + .remove("proxies"); + Ok(document) } fn as_json(value: &T) -> Result { - serde_json::to_value(value).map_err(|err| StoreError::Serialize(err.to_string())) + serde_json::to_value(value) + .map_err(|err| StoreError::Unavailable(format!("cannot serialize for storage: {err}"))) } -/// A list position as the column's type. +/// A list index as the column's type. /// /// Fails rather than truncating: a configuration with more than two billion -/// proxies is not one anybody wrote, and silently wrapping the ordinal would -/// reorder the document. +/// proxies is not a real configuration, and a wrapped ordinal would silently +/// reorder the list. fn ordinal_of(index: usize) -> Result { i32::try_from(index) - .map_err(|_| StoreError::Serialize(format!("position {index} does not fit a column"))) + .map_err(|_| StoreError::Unavailable(format!("ordinal {index} does not fit a column"))) } -/// The revision as the column holds it: the same bits, not a numeric -/// conversion. PostgreSQL has no unsigned 64-bit integer, and a value above -/// `i64::MAX` is perfectly ordinary for a hash. +/// A `u64` revision in a signed column, bit for bit. +/// +/// Postgres has no unsigned integer, and the alternative -- storing it as text +/// -- would make the compare-and-swap a string comparison. fn as_i64(revision: Revision) -> i64 { i64::from_ne_bytes(revision.0.to_ne_bytes()) } @@ -373,92 +212,100 @@ fn write_failed(err: sqlx::Error) -> StoreError { mod tests { use super::*; - /// The header columns, in the order `pipe_header` binds them. + /// Both statements have to write both columns. /// - /// Lives here rather than beside the statements because only these tests - /// consume it, and a constant nothing in the binary reads is dead code the - /// compiler is right to refuse. - const HEADER_BINDS: &[&str] = &[ - "admin_enable", - "server_host", - "server_port", - "log_level", - "log_format", - "control_socket", - "templates_dir", - "sentry_dsn", - "admin_host", - "admin_port", - "admin_auth_header", - "admin_upload_limit", - "admin_access", - "admin_groups", - "admin_public", - ]; - - /// Every `column = $n` assignment in a statement. - fn assigned_columns(sql: &str) -> Vec<&str> { - sql.split(',') - .filter_map(|fragment| fragment.split_once('=')) - .map(|(column, _)| column.trim().rsplit(' ').next().unwrap_or(column).trim()) - .filter(|column| !column.is_empty()) - .collect() + /// The hazard this guards used to be much larger -- seventeen columns + /// spread across two literals, where one added to the update and forgotten + /// in the upsert made a configuration differ depending on whether it was + /// created or replaced. Two columns is small enough to read, and cheap + /// enough to keep asserting. + #[test] + fn both_header_statements_write_the_revision_and_the_settings() { + for statement in [UPDATE_HEADER, UPSERT_HEADER] { + assert!( + statement.contains("revision = $"), + "no revision assignment: {statement}" + ); + assert!( + statement.contains("settings = $"), + "no settings assignment: {statement}" + ); + assert!( + statement.contains("updated_at = now()"), + "no updated_at: {statement}" + ); + } } + /// The conditional half must stay conditional. + /// + /// Without the revision in the `WHERE`, `save_config` would overwrite a + /// configuration that had moved under the caller, which is the one thing + /// compare-and-swap exists to prevent, and no other test here would notice. #[test] - fn the_two_header_statements_write_the_same_columns() { - // The one thing a shared, formatted column list would have given for - // free, bought back with a test instead -- because building SQL by - // formatting is what sqlx refuses, and silencing that refusal here - // would weaken it everywhere else in the crate. - // - // A column added to one statement and forgotten in the other makes a - // configuration differ depending on whether it was created or - // updated, which is the kind of bug that only appears on the second - // deployment. - let mut update: Vec<_> = assigned_columns(UPDATE_HEADER) - .into_iter() - .filter(|column| *column != "revision" && *column != "name") - .collect(); - let mut upsert: Vec<_> = assigned_columns(UPSERT_HEADER) - .into_iter() - .filter(|column| *column != "revision") - .collect(); - update.sort_unstable(); - upsert.sort_unstable(); - assert_eq!(update, upsert, "the two header statements have drifted"); + fn the_update_is_guarded_by_the_expected_revision() { + assert!( + UPDATE_HEADER.contains("WHERE name = $2 AND revision = $3"), + "{UPDATE_HEADER}" + ); } #[test] - fn both_statements_write_exactly_the_columns_pipe_header_binds() { - // The other half: the statements agreeing with each other is no use if - // they disagree with the bind order. - let mut expected: Vec<_> = HEADER_BINDS.to_vec(); - expected.push("updated_at"); - expected.sort_unstable(); + fn the_settings_document_holds_everything_except_the_proxies() { + let config = doppel_core::config::load_from_str( + r#" +server: + host: "127.0.0.1" + port: 8080 +admin: + host: "127.0.0.1" + port: 8081 + tokens: [] + access: {} + upload: + limit: 1Mi +proxies: + - name: p1 + type: http + url: "https://example.com/" +"#, + ) + .unwrap(); + + let settings = settings_of(&config).unwrap(); + let object = settings.as_object().unwrap(); + assert!( + !object.contains_key("proxies"), + "proxies belong in their own rows: {settings}" + ); + for section in ["server", "admin", "logging", "control", "templates"] { + assert!(object.contains_key(section), "{section} is missing"); + } - let mut written: Vec<_> = assigned_columns(UPDATE_HEADER) - .into_iter() - .filter(|column| *column != "revision" && *column != "name") - .collect(); - written.sort_unstable(); - assert_eq!(written, expected); + // And what came out parses straight back into a configuration, which is + // what `load` relies on. An absent `proxies` is legal, so the reparsed + // document is a whole configuration with an empty proxy list. + let reparsed: Config = serde_json::from_value(settings).unwrap(); + assert!(reparsed.proxies.is_empty()); + assert_eq!(reparsed.admin.port, config.admin.port); } #[test] fn a_revision_survives_the_trip_through_a_signed_column() { - // PostgreSQL has no unsigned 64-bit integer, and a hash above - // `i64::MAX` is perfectly ordinary. The mapping reuses the bits rather - // than converting the value, so it has to be exactly reversible. - for raw in [0, 1, u64::MAX, u64::MAX / 2, 0x8000_0000_0000_0000] { - assert_eq!(from_i64(as_i64(Revision(raw))), Revision(raw)); + // The high bit is the case that matters: a revision past i64::MAX has to + // come back as the same u64 rather than as a negative number. + for revision in [ + Revision(0), + Revision(1), + Revision(u64::MAX), + Revision(1 << 63), + ] { + assert_eq!(from_i64(as_i64(revision)), revision); } } #[test] fn an_ordinal_that_cannot_fit_is_refused_rather_than_wrapped() { - // Wrapping would reorder the document silently, and order decides - // which mock answers a request. assert!(ordinal_of(0).is_ok()); assert!(ordinal_of(usize::MAX).is_err()); } diff --git a/crates/doppel-store-postgres/src/templates.rs b/crates/doppel-store-postgres/src/templates.rs index 45eaa47..1f8c9e4 100644 --- a/crates/doppel-store-postgres/src/templates.rs +++ b/crates/doppel-store-postgres/src/templates.rs @@ -107,6 +107,28 @@ impl PostgresStore { Ok(existed) } + /// Carry a proxy's templates to its new name. + /// + /// Two steps, and neither is optional: the rows are what the store holds, and + /// the directory is what the render path reads. The rows move first -- if the + /// directory move then fails, `materialize` on the next reload rebuilds it from + /// the rows, whereas rows left behind would be invisible to it. + pub(crate) async fn rename_template_rows( + &self, + from: &str, + to: &str, + ) -> Result<(), StoreError> { + sqlx::query("UPDATE templates SET proxy = $3 WHERE config = $1 AND proxy = $2") + .bind(self.config_name()) + .bind(from) + .bind(to) + .execute(self.pool()) + .await + .map_err(failed)?; + + doppel_core::store::rename_template_dir(&self.proxy_dir(from)?, &self.proxy_dir(to)?) + } + pub(crate) async fn retain_template_rows( &self, proxy: &str, diff --git a/crates/doppel-store-postgres/src/test_support.rs b/crates/doppel-store-postgres/src/test_support.rs index c527398..e7e2d3f 100644 --- a/crates/doppel-store-postgres/src/test_support.rs +++ b/crates/doppel-store-postgres/src/test_support.rs @@ -194,6 +194,23 @@ impl TestSchema { pool.close().await; } + /// Every row of a one-column JSON query, in the order the query asks for. + /// + /// For the migration suite, which reads the documents a backfill produced + /// and parses them itself rather than through the store: the store refuses + /// to connect to a schema whose `_sqlx_migrations` bookkeeping is not + /// complete, and a test that applies migration files by hand has no way to + /// forge that bookkeeping -- sqlx checksums it. + pub async fn json_rows(&self, sql: &str) -> Vec { + let pool = PgPool::connect(&self.url()).await.expect("connect"); + let rows = sqlx::raw_sql(AssertSqlSafe(sql.to_owned())) + .fetch_all(&pool) + .await + .expect("fetch json"); + pool.close().await; + rows.iter().map(|row| row.get(0)).collect() + } + pub async fn count(&self, table: &str) -> i64 { let pool = PgPool::connect(&self.url()).await.expect("connect"); let count: i64 = sqlx::raw_sql(AssertSqlSafe(format!("SELECT count(*) FROM {table}"))) diff --git a/crates/doppel-store-postgres/tests/connect.rs b/crates/doppel-store-postgres/tests/connect.rs index 42110aa..c54e5f5 100644 --- a/crates/doppel-store-postgres/tests/connect.rs +++ b/crates/doppel-store-postgres/tests/connect.rs @@ -77,12 +77,8 @@ async fn two_schemas_do_not_see_each_other() { assert_ne!(first.name(), second.name()); first .execute( - "INSERT INTO configurations (name, revision, server_host, server_port, \ - log_level, log_format, control_socket, templates_dir, admin_host, \ - admin_port, admin_auth_header, admin_upload_limit, admin_access) \ - VALUES ('only-in-first', 0, '127.0.0.1', 8080, 'info', 'json', \ - '/tmp/d.sock', './templates', '127.0.0.1', 8081, 'X-Proxy-Authorization', \ - 1048576, '{}')", + "INSERT INTO configurations (name, revision, settings) \ + VALUES ('only-in-first', 0, '{}')", ) .await; diff --git a/crates/doppel-store-postgres/tests/load.rs b/crates/doppel-store-postgres/tests/load.rs index 72cbbe7..69e993c 100644 --- a/crates/doppel-store-postgres/tests/load.rs +++ b/crates/doppel-store-postgres/tests/load.rs @@ -105,124 +105,38 @@ fn parse(yaml: &str) -> doppel_core::Config { /// Write a configuration into the tables the long way, so `load` is tested /// against rows this test wrote rather than against `save`'s idea of them. -/// Once `save` exists the conformance suite covers the pair; until then this -/// is the only way `load` can be wrong in isolation and be caught. +/// +/// "The long way" is much shorter than it was: both tables hold a JSON +/// document, so this plants the same two shapes `load` reads instead of +/// interpolating a column per field. The point is unchanged -- these are rows +/// the store did not produce, so `load` can be wrong in isolation and be +/// caught. +/// +/// Dollar quoting rather than escaped apostrophes: a document contains `"` and +/// may contain `'`, and `$tag$` sidesteps both. async fn insert(schema: &TestSchema, name: &str, config: &doppel_core::Config, revision: i64) { - let admin = &config.admin; + let mut settings = serde_json::to_value(config).expect("a config serializes"); + settings + .as_object_mut() + .expect("a config is an object") + .remove("proxies"); + schema .execute(&format!( - "INSERT INTO configurations (name, revision, server_host, server_port, log_level, \ - log_format, control_socket, templates_dir, sentry_dsn, admin_host, admin_port, \ - admin_auth_header, admin_upload_limit, admin_access) VALUES \ - ('{name}', {revision}, '{}', {}, '{}', '{}', '{}', '{}', {}, '{}', {}, '{}', {}, '{}')", - config.server.host, - config.server.port, - serde_json::to_value(config.logging.level).unwrap().as_str().unwrap(), - serde_json::to_value(config.logging.format).unwrap().as_str().unwrap(), - config.control.socket.display(), - config.templates.dir.display(), - config - .sentry - .as_ref() - .map_or("NULL".to_owned(), |s| format!("'{}'", s.dsn)), - admin.host, - admin.port, - admin.auth.header, - admin.upload.limit.get(), - serde_json::to_string(&admin.access).unwrap(), + "INSERT INTO configurations (name, revision, settings) \ + VALUES ('{name}', {revision}, $tag${settings}$tag$)" )) .await; - for (ordinal, token) in admin.tokens.iter().enumerate() { - schema - .execute(&format!( - "INSERT INTO admin_tokens (config, name, \"group\", token, ordinal) \ - VALUES ('{name}', '{}', '{}', '{}', {ordinal})", - // `as_str`, not `{}`: `Token`'s `Display` is redacted, so - // interpolating it would write the literal `` into - // every row. This is a test writing raw SQL on purpose -- the - // point is to plant rows the store did not produce -- so the - // deliberate escape hatch has to be spelled out. - token.name, - token.group, - token.token.as_str() - )) - .await; - } - for (ordinal, proxy) in config.proxies.iter().enumerate() { - let nullable = |value: Option| value.unwrap_or_else(|| "NULL".to_owned()); + let document = serde_json::to_value(proxy).expect("a proxy serializes"); schema .execute(&format!( - "INSERT INTO proxies (config, name, ordinal, kind, url, timeout_seconds, \ - body_limit, replace_ratio, resolve_kind, resolve_header, loss_percentage, \ - loss_status, latency_percentage, latency_min, latency_max, headers, access) \ - VALUES ('{name}', '{}', {ordinal}, '{}', '{}', {}, {}, {}, '{}', {}, {}, {}, \ - {}, {}, {}, '{}', {})", - proxy.name, - serde_json::to_value(proxy.kind).unwrap().as_str().unwrap(), - proxy.url, - nullable(proxy.timeout.map(|t| t.to_string())), - proxy.body_limit.get(), - nullable(proxy.replace.map(|r| r.to_string())), - serde_json::to_value(proxy.resolve.kind) - .unwrap() - .as_str() - .unwrap(), - nullable(proxy.resolve.header.as_ref().map(|h| format!("'{h}'"))), - nullable(proxy.loss.as_ref().map(|l| l.percentage.to_string())), - nullable(proxy.loss.as_ref().map(|l| l.status.to_string())), - nullable(proxy.latency.as_ref().map(|l| l.percentage.to_string())), - nullable(proxy.latency.as_ref().map(|l| l.min.to_string())), - nullable(proxy.latency.as_ref().map(|l| l.max.to_string())), - serde_json::to_string(&proxy.headers).unwrap(), - nullable( - proxy - .access - .as_ref() - .map(|a| format!("'{}'", serde_json::to_string(a).unwrap())) - ), + "INSERT INTO proxies (config, name, ordinal, document) \ + VALUES ('{name}', '{}', {ordinal}, $tag${document}$tag$)", + proxy.name )) .await; - - for (mock_ordinal, mock) in proxy.mocks.iter().enumerate() { - schema - .execute(&format!( - "INSERT INTO mocks (config, proxy, name, ordinal, method, url_pattern, \ - status, body, json, template, request_headers, request_query, \ - request_body, response_headers, proxy_override) VALUES \ - ('{name}', '{}', '{}', {mock_ordinal}, '{}', $tag${}$tag$, {}, {}, {}, {}, \ - '{}', '{}', '{}', '{}', {})", - proxy.name, - mock.name, - mock.request.method, - mock.request.url, - mock.response.status, - nullable( - mock.response - .body - .as_ref() - .map(|b| format!("$tag${b}$tag$")) - ), - nullable( - mock.response - .json - .as_ref() - .map(|j| format!("$tag${j}$tag$")) - ), - nullable(mock.response.template.as_ref().map(|t| format!("'{t}'"))), - serde_json::to_string(&mock.request.headers).unwrap(), - serde_json::to_string(&mock.request.query).unwrap(), - serde_json::to_string(&mock.request.body).unwrap(), - serde_json::to_string(&mock.response.headers).unwrap(), - nullable( - mock.proxy - .as_ref() - .map(|p| format!("'{}'", serde_json::to_string(p).unwrap())) - ), - )) - .await; - } } } @@ -343,9 +257,14 @@ async fn an_unknown_configuration_name_is_not_found() { #[tokio::test] async fn a_half_written_latency_is_reported_rather_than_read_as_absent() { - // Three columns hold one optional section. A row with some of them set is - // one nothing could have written through `save`, so treating it as "no - // latency configured" would serve a configuration nobody wrote. + // `latency` is an optional section with three required fields. A stored + // object missing one is something nothing could have written through + // `save`, so reading it as "no latency configured" -- or as a zero -- would + // serve a configuration nobody wrote. + // + // This used to be three columns, where the same hazard was a row with some + // of them set. The shape changed; the hazard did not, so neither did the + // test. let Some(url) = require_database() else { return; }; @@ -362,7 +281,7 @@ async fn a_half_written_latency_is_reported_rather_than_read_as_absent() { ) .await; schema - .execute("UPDATE proxies SET latency_max = NULL WHERE name = 'alpha'") + .execute("UPDATE proxies SET document = document #- '{latency,max}' WHERE name = 'alpha'") .await; let store = PostgresStore::connect(&schema.url(), "default", schema.templates_dir()) @@ -370,7 +289,11 @@ async fn a_half_written_latency_is_reported_rather_than_read_as_absent() { .expect("connect"); let err = store.load_config().await.expect_err("must be refused"); let message = format!("{err:?}"); - assert!(message.contains("latency"), "{message}"); + // The field serde could not read, and the proxy whose row held it. Either + // alone leaves an operator guessing: `max` does not say where, and the proxy + // name does not say what. + assert!(message.contains("max"), "{message}"); + assert!(message.contains("alpha"), "{message}"); schema.drop().await; } diff --git a/crates/doppel-store-postgres/tests/migrate.rs b/crates/doppel-store-postgres/tests/migrate.rs new file mode 100644 index 0000000..1636053 --- /dev/null +++ b/crates/doppel-store-postgres/tests/migrate.rs @@ -0,0 +1,397 @@ +//! The 0005 backfill, held to the one property that matters. +//! +//! Migration 0005 replaces a column per configuration field with two JSON +//! documents. The revision of a stored configuration is a hash of its canonical +//! YAML, and every write compare-and-swaps against it -- so if the JSON the +//! backfill builds parsed into a configuration that differed from the one the +//! old columns described, in any way at all, the revision would move. The next +//! write from a client holding the revision it had read would be refused as a +//! conflict that never happened, and the operator would see contention where +//! there was corruption. +//! +//! So this suite plants rows in the pre-0005 shape, applies the migration file, +//! and checks that what comes out parses back into exactly the configuration +//! that went in -- and that its revision is unchanged. +//! +//! It does not go through `PostgresStore`: `connect` refuses a schema whose +//! `_sqlx_migrations` bookkeeping is incomplete, and a test that applies +//! migration files by hand cannot forge that bookkeeping, because sqlx +//! checksums it. The store's own read path is covered by `load.rs` and +//! `conformance.rs`; what is under test here is the SQL. + +use doppel_core::config::{Config, ProxyConfig}; +use doppel_core::store::Revision; +use doppel_store_postgres::test_support::{TestSchema, require_database}; + +/// The schema as it stood before 0005, applied in order. +const BEFORE: &[&str] = &[ + include_str!("../migrations/0001_initial.sql"), + include_str!("../migrations/0002_proxy_rewrite_redirects.sql"), + include_str!("../migrations/0003_admin_groups.sql"), + include_str!("../migrations/0004_admin_public.sql"), +]; + +const JSONB: &str = include_str!("../migrations/0005_settings_as_jsonb.sql"); + +/// A configuration touching every column 0005 has to carry across: a sentry +/// section, tokens, a group allow-list, a non-default `enable` and auth header, +/// both resolve kinds, both fault sections, a replace ratio, an explicit +/// `rewrite_redirects`, a per-proxy access override, all three mock body forms, +/// a mock-level proxy override, all three selector maps, and a proxy with no +/// mocks at all. +/// +/// Deliberately without `dashboard` and `title`: this is what a configuration +/// stored before those fields existed looks like, which is the only kind of row +/// a backfill can encounter. +const FIXTURE: &str = r#" +server: + host: "127.0.0.1" + port: 18080 +logging: + level: debug + format: text +control: + socket: /tmp/doppel-migrate.sock +templates: + dir: /tmp/doppel-migrate-templates +sentry: + dsn: "https://key@sentry.example.com/1" +admin: + enable: false + host: "127.0.0.1" + port: 18081 + auth: + header: X-Admin-Token + public: false + groups: ["admin", "ci"] + tokens: + - name: ci + group: ci + token: "0123456789abcdef0123456789abcdef" + access: + list: public + read: [ci] + create: [admin] + update: [admin] + delete: [admin] + upload: [admin] + upload: + limit: 2Mi +proxies: + - name: alpha + type: http + url: "https://alpha.example.com/api/" + timeout: 12 + body_limit: 4Mi + replace: 0.25 + rewrite_redirects: false + headers: + X-Injected: "yes" + access: + read: [ci] + loss: + percentage: 0.055 + status: 503 + latency: + percentage: 0.5 + min: 0.1 + max: 1.5 + mocks: + - name: first + request: + method: POST + url: "^/widgets$" + headers: + trace: X-Trace-Id + query: + filter: ".filter" + body: + items: ".content.items" + response: + status: 201 + json: '{"ok": true}' + headers: + X-Mocked: "first" + proxy: + replace: 1.0 + latency: + percentage: 1.0 + min: 0.2 + max: 0.4 + - name: second + request: + method: GET + url: "^/health$" + response: + status: 200 + body: "ok" + - name: third + request: + method: GET + url: "^/rendered$" + response: + status: 200 + template: page.json.j2 + - name: beta + type: http + url: "https://beta.example.com/api/" + resolve: + type: header + header: X-Proxy-Name +"#; + +#[tokio::test] +async fn the_backfill_preserves_every_stored_revision() { + let Some(url) = require_database() else { + return; + }; + let schema = TestSchema::create(&url).await; + for migration in BEFORE { + schema.execute(migration).await; + } + + let expected = doppel_core::config::load_from_str(FIXTURE).expect("the fixture parses"); + let revision = Revision::of_config(&expected); + insert_pre_0005(&schema, "default", &expected, as_i64(revision)).await; + + schema.execute(JSONB).await; + + // Reassembled the way `load` does: the settings document holds everything + // but the proxies, which come from their own rows in `ordinal` order. + let settings = schema + .json_rows("SELECT settings FROM configurations WHERE name = 'default'") + .await; + let mut loaded: Config = + serde_json::from_value(settings[0].clone()).expect("the backfilled settings parse"); + loaded.proxies = schema + .json_rows("SELECT document FROM proxies WHERE config = 'default' ORDER BY ordinal") + .await + .into_iter() + .map(|document| { + serde_json::from_value::(document).expect("a backfilled proxy parses") + }) + .collect(); + + // Compared whole rather than field by field: a field this test forgot to + // list is exactly the field the backfill is most likely to have dropped. + assert_eq!(loaded, expected); + assert_eq!( + Revision::of_config(&loaded), + revision, + "the backfill changed a stored revision, which would refuse the next write as a conflict" + ); + + // And the tables the documents replaced are gone, so nothing can read them + // and quietly disagree with the document. + for table in ["admin_tokens", "mocks"] { + let present: Vec = schema + .json_rows(&format!( + "SELECT to_jsonb(count(*)) FROM information_schema.tables \ + WHERE table_schema = current_schema() AND table_name = '{table}'" + )) + .await; + assert_eq!(present[0], serde_json::json!(0), "{table} still exists"); + } + + schema.drop().await; +} + +#[tokio::test] +async fn a_configuration_with_no_tokens_no_sentry_and_no_mocks_survives() { + // The absent cases are where a backfill goes wrong: an empty token list has + // to stay `[]` rather than becoming null, and `sentry` has a required field, + // so `{"dsn": null}` stripped down to `{}` would fail to parse -- which is + // why that section is built conditionally rather than stripped. + let Some(url) = require_database() else { + return; + }; + let schema = TestSchema::create(&url).await; + for migration in BEFORE { + schema.execute(migration).await; + } + + let bare = doppel_core::config::load_from_str( + r#" +server: + host: "127.0.0.1" + port: 8080 +admin: + host: "127.0.0.1" + port: 8081 + tokens: [] + access: {} + upload: + limit: 1Mi +proxies: + - name: solo + type: http + url: "https://example.com/" +"#, + ) + .expect("the fixture parses"); + let revision = Revision::of_config(&bare); + insert_pre_0005(&schema, "default", &bare, as_i64(revision)).await; + + schema.execute(JSONB).await; + + let settings = schema + .json_rows("SELECT settings FROM configurations WHERE name = 'default'") + .await; + let mut loaded: Config = + serde_json::from_value(settings[0].clone()).expect("the backfilled settings parse"); + assert!(loaded.sentry.is_none(), "sentry must stay absent"); + assert!(loaded.admin.tokens.is_empty()); + + loaded.proxies = schema + .json_rows("SELECT document FROM proxies WHERE config = 'default' ORDER BY ordinal") + .await + .into_iter() + .map(|document| serde_json::from_value::(document).expect("a proxy parses")) + .collect(); + assert_eq!(loaded, bare); + assert_eq!(Revision::of_config(&loaded), revision); + + schema.drop().await; +} + +fn as_i64(revision: Revision) -> i64 { + i64::from_ne_bytes(revision.0.to_ne_bytes()) +} + +/// Write a configuration in the shape the schema had before 0005. +/// +/// Lifted from the version of `tests/load.rs` that planted rows this way, which +/// is why it interpolates SQL by hand: the whole point is to produce rows the +/// current code cannot write, so it cannot go through `save`. +#[allow(clippy::too_many_lines)] +async fn insert_pre_0005(schema: &TestSchema, name: &str, config: &Config, revision: i64) { + let admin = &config.admin; + let nullable = |value: Option| value.unwrap_or_else(|| "NULL".to_owned()); + + schema + .execute(&format!( + "INSERT INTO configurations (name, revision, server_host, server_port, log_level, \ + log_format, control_socket, templates_dir, sentry_dsn, admin_enable, admin_host, \ + admin_port, admin_auth_header, admin_upload_limit, admin_access, admin_public, \ + admin_groups) VALUES \ + ('{name}', {revision}, '{}', {}, '{}', '{}', '{}', '{}', {}, {}, '{}', {}, '{}', {}, \ + '{}', {}, {})", + config.server.host, + config.server.port, + serde_json::to_value(config.logging.level) + .unwrap() + .as_str() + .unwrap(), + serde_json::to_value(config.logging.format) + .unwrap() + .as_str() + .unwrap(), + config.control.socket.display(), + config.templates.dir.display(), + nullable(config.sentry.as_ref().map(|s| format!("'{}'", s.dsn))), + admin.enable, + admin.host, + admin.port, + admin.auth.header, + admin.upload.limit.get(), + serde_json::to_string(&admin.access).unwrap(), + nullable(admin.public.map(|p| p.to_string())), + nullable( + admin + .groups + .as_ref() + .map(|g| format!("'{}'", serde_json::to_string(g).unwrap())) + ), + )) + .await; + + for (ordinal, token) in admin.tokens.iter().enumerate() { + schema + .execute(&format!( + "INSERT INTO admin_tokens (config, name, \"group\", token, ordinal) \ + VALUES ('{name}', '{}', '{}', '{}', {ordinal})", + // `as_str`, not `{}`: `Token`'s `Display` is redacted, so + // interpolating it would write the literal `` into + // every row. + token.name, + token.group, + token.token.as_str() + )) + .await; + } + + for (ordinal, proxy) in config.proxies.iter().enumerate() { + schema + .execute(&format!( + "INSERT INTO proxies (config, name, ordinal, kind, url, timeout_seconds, \ + body_limit, replace_ratio, rewrite_redirects, resolve_kind, resolve_header, \ + loss_percentage, loss_status, latency_percentage, latency_min, latency_max, \ + headers, access) VALUES ('{name}', '{}', {ordinal}, '{}', '{}', {}, {}, {}, {}, \ + '{}', {}, {}, {}, {}, {}, {}, '{}', {})", + proxy.name, + serde_json::to_value(proxy.kind).unwrap().as_str().unwrap(), + proxy.url, + nullable(proxy.timeout.map(|t| t.to_string())), + proxy.body_limit.get(), + nullable(proxy.replace.map(|r| r.to_string())), + nullable(proxy.rewrite_redirects.map(|r| r.to_string())), + serde_json::to_value(proxy.resolve.kind) + .unwrap() + .as_str() + .unwrap(), + nullable(proxy.resolve.header.as_ref().map(|h| format!("'{h}'"))), + nullable(proxy.loss.as_ref().map(|l| l.percentage.to_string())), + nullable(proxy.loss.as_ref().map(|l| l.status.to_string())), + nullable(proxy.latency.as_ref().map(|l| l.percentage.to_string())), + nullable(proxy.latency.as_ref().map(|l| l.min.to_string())), + nullable(proxy.latency.as_ref().map(|l| l.max.to_string())), + serde_json::to_string(&proxy.headers).unwrap(), + nullable( + proxy + .access + .as_ref() + .map(|a| format!("'{}'", serde_json::to_string(a).unwrap())) + ), + )) + .await; + + for (mock_ordinal, mock) in proxy.mocks.iter().enumerate() { + schema + .execute(&format!( + "INSERT INTO mocks (config, proxy, name, ordinal, method, url_pattern, \ + status, body, json, template, request_headers, request_query, \ + request_body, response_headers, proxy_override) VALUES \ + ('{name}', '{}', '{}', {mock_ordinal}, '{}', $tag${}$tag$, {}, {}, {}, {}, \ + '{}', '{}', '{}', '{}', {})", + proxy.name, + mock.name, + mock.request.method, + mock.request.url, + mock.response.status, + nullable( + mock.response + .body + .as_ref() + .map(|b| format!("$tag${b}$tag$")) + ), + nullable( + mock.response + .json + .as_ref() + .map(|j| format!("$tag${j}$tag$")) + ), + nullable(mock.response.template.as_ref().map(|t| format!("'{t}'"))), + serde_json::to_string(&mock.request.headers).unwrap(), + serde_json::to_string(&mock.request.query).unwrap(), + serde_json::to_string(&mock.request.body).unwrap(), + serde_json::to_string(&mock.response.headers).unwrap(), + nullable( + mock.proxy + .as_ref() + .map(|p| format!("'{}'", serde_json::to_string(p).unwrap())) + ), + )) + .await; + } + } +} diff --git a/crates/doppel-store-postgres/tests/save.rs b/crates/doppel-store-postgres/tests/save.rs index 4c75d30..3d29d4f 100644 --- a/crates/doppel-store-postgres/tests/save.rs +++ b/crates/doppel-store-postgres/tests/save.rs @@ -200,9 +200,12 @@ async fn a_rewrite_leaves_no_rows_from_the_previous_version() { .expect("shrink"); assert_eq!(schema.count("proxies").await, 1); - assert_eq!(schema.count("mocks").await, 1); let (loaded, _) = store.load_config().await.expect("load"); assert_eq!(loaded, one_proxy); + // A dropped proxy's mocks used to be a `mocks` row count; they now live + // inside the proxy's own document, so deleting the row takes them with it and + // what is left to check is that the surviving document kept its own. + assert_eq!(loaded.proxies[0].mocks.len(), 1); schema.drop().await; } diff --git a/docker-compose.yml b/docker-compose.yml index 29fd3eb..f8396d8 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -14,8 +14,23 @@ # # **Doppel itself**, behind a profile, for trying the two together: # +# mkdir -p config && cp main.example.yaml config/main.yaml # docker compose --profile doppel up -d --wait # +# `make frontend` beforehand is optional and only faster: the image builds the +# dashboard itself when `frontend/dist` is not there. +# +# The copy is not optional and its absence is a confusing failure rather than a +# clear one: a bind mount whose source does not exist makes Docker create an empty +# *directory* at that path, and Doppel then reports that it cannot read +# /etc/doppel/main.yaml. +# +# A directory is mounted rather than the file, and that is not a preference. A +# single-file bind mount makes the file itself a mount point, and a save writes a +# new file and renames it over the old one -- which no kernel will do over a mount +# point. Every read worked and every write from the dashboard failed with +# "Resource busy (os error 16)". +# # The profile is what keeps the two apart. A service carrying one is not # started by a bare `docker compose up`, so the test workflow above is # unaffected -- and it has to stay that way: running the proxy during a test @@ -57,17 +72,12 @@ services: # Built rather than pulled: there is no published image before the first # release, and a local build is what you want anyway when trying a change. # - # Stage the binary first -- the image copies one in rather than compiling, - # and it has to be musl-linked because the base is Alpine: - # - # cargo zigbuild --release --target x86_64-unknown-linux-musl -p doppel-cli - # mkdir -p dist && cp target/x86_64-unknown-linux-musl/release/doppel dist/ - # - # On an arm64 machine, substitute aarch64-unknown-linux-musl. + # Nothing has to be staged first. The Dockerfile uses a binary from `dist/` + # when one is there and compiles its own otherwise, so `docker compose build` + # works from a checkout -- it does need the dashboard built, which + # `make frontend` does. build: context: . - args: - BIN: dist/doppel ports: # Offset like the database's, and for the same reason: a developer @@ -75,11 +85,27 @@ services: - "127.0.0.1:58080:8080" - "127.0.0.1:58081:8081" + environment: + DOPPEL_EXTERNAL_URL: "http://127.0.0.1:58080/" + volumes: - # The configuration. Read-only: this container has no business editing - # the file on your working tree, and `config push`/the admin API write - # through the store rather than through this mount. - - ./main.example.yaml:/etc/doppel/main.yaml:ro + # Your own configuration, not the reference one: + # + # mkdir -p config && cp main.example.yaml config/main.yaml + # + # `config/` is git-ignored, so editing what is in it -- a port, an upstream, a + # token -- leaves no diff to explain and nothing to accidentally commit. + # Mounting `main.example.yaml` directly, which this used to do, meant every + # experiment showed up as a change to a file the documentation quotes. + # + # The directory, not the file: see the note at the top of this file. Writes + # from the admin API replace the configuration by renaming a new file over it, + # and a file that is itself a mount point cannot be replaced. + # + # Read-only: this container has no business editing a file on your working + # tree, and `config push` and the admin API write through the store rather + # than through this mount. + - ./config:/etc/doppel:rw # Templates. A named volume rather than a bind mount, and not optional: # templates uploaded through the admin API are written here at runtime, # and without a volume they vanish with the container. @@ -90,7 +116,7 @@ services: condition: service_healthy healthcheck: - # `/status` on the admin listener: it needs no token by default, answers + # `/api/v1/status` on the admin listener: it needs no token by default, answers # only once the runtime is compiled and both listeners are bound, and # reports the revision actually in force -- which is what "ready" means # here, rather than "the process started". @@ -106,7 +132,7 @@ services: # and would sit unhealthy while serving perfectly well; drop the # healthcheck in that case rather than pointing it at the proxy port, # where every path either reaches an upstream or is a mock. - test: ["CMD", "curl", "-fsS", "-o", "/dev/null", "http://127.0.0.1:8081/status"] + test: ["CMD", "curl", "-fsS", "-o", "/dev/null", "http://127.0.0.1:8081/api/v1/status"] interval: 5s timeout: 3s # Startup is well under a second; this is slack for a cold container on diff --git a/docs/assets/icon.png b/docs/assets/icon.png deleted file mode 100644 index 31e3500..0000000 Binary files a/docs/assets/icon.png and /dev/null differ diff --git a/docs/development/architecture.md b/docs/development/architecture.md index c297e4a..29bf6da 100644 --- a/docs/development/architecture.md +++ b/docs/development/architecture.md @@ -11,7 +11,7 @@ a network, rendering without HTTP, and the proxy logic without an admin API. | `doppel-core` | The configuration model and its types, YAML loading, validation, the `ConfigStore` trait and its file-backed implementation, the compiled runtime, the error model | | `doppel-render` | Variable extraction and Jinja2 rendering for mock responses; knows nothing about HTTP | | `doppel-proxy` | The proxy listener, resolution, fault injection, mock matching, upstream forwarding, the request pipeline | -| `doppel-admin` | The admin HTTP API: token access control, proxy CRUD, template files, reload, status, metrics, the OpenAPI document | +| `doppel-admin` | The admin HTTP API: token access control, proxy CRUD, template files, reload, status, metrics, the OpenAPI document, and the dashboard the build script embeds | | `doppel-store-postgres` | `PostgresStore`, and the sqlx migrations that own its schema | | `doppel-telemetry` | Logging initialization and optional Sentry | | `doppel-cli` | The `doppel` binary: argument parsing, the control channel, and wiring the rest together | @@ -82,7 +82,7 @@ a handler gets the configuration it judges by. `ConfigStore` is the boundary. `FileStore` serialises writers with an advisory lock on a sibling `.lock` file; `PostgresStore` uses a conditional `UPDATE` inside a transaction. Reads on the PostgreSQL side take one -`REPEATABLE READ` snapshot, because a configuration is five tables and a +`REPEATABLE READ` snapshot, because a configuration spans two tables and a per-statement snapshot would assemble one from two of them. One conformance suite runs against both, which is what keeps the trait's @@ -100,3 +100,72 @@ needs a token the operator may be trying to create. The socket is bound inside a private directory and published with a hard link, which is atomic *and* exclusive: a second bind fails loudly rather than silently taking over a live channel. + +## The dashboard + +A React application in `frontend/`, built by vite into `frontend/dist`, walked at +compile time by `crates/doppel-admin/build.rs` and embedded with `include_bytes!`. +The listener serves `index.html` at `/`, the hashed assets under `/static/`, and +`robots.txt`; the operator-facing half of this is in +[The dashboard](../usage/dashboard.md). + +Three decisions are worth knowing before changing any of it. + +**The assets are embedded, not served from disk.** So `npm run build` alone +changes nothing about a running binary -- the binary has to be rebuilt after it. +The build script watches `frontend/dist`, so `cargo build` does re-embed; the trap +is forgetting the `cargo build`. A binary compiled without `frontend/dist` +compiles fine and answers 503 at `/`, which keeps +`cargo install --path crates/doppel-cli` working on a machine with no Node. CI +sets `DOPPEL_REQUIRE_DASHBOARD_ASSETS` so that concession cannot silently apply +there, and the release refuses to publish a binary that lacks the page. + +**The image builds either way.** `docker build .` uses a binary staged in `dist/` +when there is one and compiles inside its own builder stage when there is not -- +which is what lets `make image` work on a machine that cannot link musl. The +release stages binaries built on native runners and points the builder stage at a +base with no toolchain, so it never pulls a Rust image to run a `cp`. The compile +branch refuses to run without `frontend/dist`, because a binary built without it +answers 503 at its own root. + +**Settings reach the page through the HTML, not an endpoint.** The listener +substitutes `{title, public, version, authHeader, refreshMs}` into a +`application/json` script element per request. Every one of those is known when +the HTML is written, and a fetch would mean the page cannot draw its own header +until a round trip finishes. The JSON has its less-than characters escaped, which +is what makes a hostile `admin.title` a non-event and what allows a content +security policy with no `unsafe-inline` -- the one inline stylesheet the page +carries, the code editor's, is named by hash instead. + +**The page never decides what the caller may do.** `GET /api/v1/access` reports +that, evaluated by the same function that enforces it, so a control is disabled +because the server would have refused rather than because the page guessed. + +### Layers + +``` +frontend/src/ + components/ presentational; may not import a service + pages/ one screen each: fetches, holds its own state, renders components + services/ the only code that talks to the API + store/ state that outlives one screen (zustand) + types/ the API's shapes, and the one error type +``` + +The rule that a component may not reach a service is in `eslint.config.js`, not +only in review: the first component to grow a `fetch` becomes the one nothing can +be tested without. + +Tests split by what each tool is good for. jest covers services, stores and the +components that do not route; anything that navigates is covered by Playwright, +against a real browser and a real binary -- react-router ships ESM only and uses +`import.meta`, which cannot survive jest's transform to CommonJS. That split is +not a workaround for a build problem alone: the three bugs the suite found on its +first run were a stale rights subscription, a field name that disagreed with the +API, and a CommonJS interop failure that appeared only in the built bundle. None +were reachable from a unit test. + +`frontend/src/__tests__/bundle-size.test.ts` holds the payload to a measured +budget, and asserts that the entry chunk does not contain prism -- a size limit +alone would let the highlighter hide in the entry's slack, and then every visitor +pays for it to list proxies. diff --git a/docs/development/index.md b/docs/development/index.md index 719dbb8..de989d6 100644 --- a/docs/development/index.md +++ b/docs/development/index.md @@ -58,9 +58,21 @@ the templates directory the integration tests bind and write themselves. To bring both up, for trying the two together rather than for testing: ```bash +cp main.example.yaml main.yaml # once; main.yaml is git-ignored +make frontend # the dashboard the image embeds docker compose --profile doppel up -d --wait ``` +The copy matters twice over. It is the file the container mounts, and it is yours +to edit -- a port, an upstream, a token -- without any of that showing up as a +change to `main.example.yaml`, which the documentation quotes. It is also not +optional in a way that fails clearly: a bind mount whose source does not exist +makes Docker create a directory at that path, and Doppel then reports +`cannot read /etc/doppel/main.yaml: Is a directory`. + +Doppel's own ports are offset -- `58080` and `58081` -- so a container and a +`cargo run` can be up at once. + Each test creates its own schema and drops it afterwards, so the suites can run in parallel against one database. diff --git a/docs/index.md b/docs/index.md index a194db1..bc9f764 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,5 +1,7 @@ # Doppel +![Doppel](assets/icon-128.png) + Doppel is a doppelganger for your backend: an HTTP proxy that stands in front of a real service, or in place of one that is not there. @@ -33,6 +35,9 @@ including verifying a download and the macOS quarantine case, are on - **Serves an admin API** -- proxy CRUD, template upload, reload, status, Prometheus metrics and Swagger UI -- behind token access control. See [Admin API](usage/admin-api.md). +- **Serves a dashboard** on that same port's root, compiled into the binary: + the proxy set, a form over the whole proxy document, mock templates, status and + reload. See [The dashboard](usage/dashboard.md). - **Reports to Sentry**, optionally, behind a cargo feature. - **Stores its configuration** in a YAML file or in PostgreSQL, with `config push` and `config pull` to move between them. See @@ -72,7 +77,9 @@ the six words the rest of it leans on and shows how a request is handled. 6. [Changing configuration while it runs](usage/runtime-changes.md) -- reloads, tokens, and editing one proxy over HTTP. 7. [The admin API](usage/admin-api.md) and - [PostgreSQL](usage/storage.md) for anything running in more than one place. + [the dashboard](usage/dashboard.md) -- driving all of the above over HTTP, or + in a browser. +8. [PostgreSQL](usage/storage.md) for anything running in more than one place. with the [configuration reference](usage/configuration.md) and the [CLI reference](usage/cli.md) at the end for looking things up. diff --git a/docs/screenshots/screenshot-proxies.webp b/docs/screenshots/screenshot-proxies.webp new file mode 100644 index 0000000..a3a9a92 --- /dev/null +++ b/docs/screenshots/screenshot-proxies.webp @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:5d7ea2d98ea603d916e66597c536934b6d2684e02fbabb22afafd218df25c1a0 +size 106566 diff --git a/docs/screenshots/screenshot-proxy-edit-yaml.webp b/docs/screenshots/screenshot-proxy-edit-yaml.webp new file mode 100644 index 0000000..f97d8f7 --- /dev/null +++ b/docs/screenshots/screenshot-proxy-edit-yaml.webp @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:866ee042a47a762bd571d8ec5e6808f95c720a1b0d72fff20180c9a9ad088a66 +size 107194 diff --git a/docs/screenshots/screenshot-proxy-edit.webp b/docs/screenshots/screenshot-proxy-edit.webp new file mode 100644 index 0000000..a9eb098 --- /dev/null +++ b/docs/screenshots/screenshot-proxy-edit.webp @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:614eebd6b5ef1fb16952196b34ae52a7e837efa4bd0e10e4d9ffccbb501e7164 +size 106828 diff --git a/docs/screenshots/screenshot-signin.webp b/docs/screenshots/screenshot-signin.webp new file mode 100644 index 0000000..742ab8a --- /dev/null +++ b/docs/screenshots/screenshot-signin.webp @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:88f89244ed0b18ddcdb8b4dc68c4d7fcd9cd37302936070d08e79235734786cc +size 103102 diff --git a/docs/usage/admin-api.md b/docs/usage/admin-api.md index 7b2d15c..59c6732 100644 --- a/docs/usage/admin-api.md +++ b/docs/usage/admin-api.md @@ -4,7 +4,7 @@ A second HTTP listener on `admin.host:admin.port`, separate from the proxy because the proxy's fallback handler swallows every path -- there is nowhere on it an admin route could live. -`admin.enable: false` turns all of it off, including `/status` and `/metrics`, +`admin.enable: false` turns all of it off, including `/api/v1/status` and `/metrics`, and the port is then never bound. See [the configuration reference](configuration.md#admin). @@ -126,14 +126,102 @@ would not help. | POST | `/api/v1/proxies/{name}/templates/{file}` | `upload` | `204` | | DELETE | `/api/v1/proxies/{name}/templates/{file}` | `upload` | `204` | | POST | `/api/v1/config/reload` | `update` | `200` | -| GET | `/status` | none | `200` | +| GET | `/api/v1/access` | none | `200` | +| GET | `/api/v1/schema` | none | `200` | +| GET | `/api/v1/status` | none | `200` | | GET | `/metrics` | none | `200` | | GET | `/openapi.json` | none | `200` | | GET | `/swagger-ui` | none | `200` | -`/status`, `/metrics`, `/openapi.json` and `/swagger-ui` sit outside -`/api/v1` because they are not resources of the API; they describe or observe -the process. +Responses are compressed when the client asks: `br` or `gzip` from +`Accept-Encoding`, `br` first when both are offered. The proxy listener does not +do this -- it relays the upstream's own encoding, because that is part of what a +client is being tested against. + +A trailing slash is accepted everywhere: `/metrics/` answers as `/metrics` does. +Axum stopped redirecting between the two spellings, and a 404 for a slash is a +poor answer for a path an operator typed. The proxy listener is deliberately not +like this -- a proxied path is relayed byte for byte, because `/orders/` and +`/orders` are two resources upstream. + +`/api/v1/status` sits outside `/api/v1` because it is not a resource of the API; +it reports on the process. `/metrics`, `/openapi.json` and `/swagger-ui/` sit +outside `/api/` altogether: those paths are what a scraper, a client generator +and a browser look for by default, and none of the three is a resource of the API +either. + +Everything the API serves is under `/api/`. That is a boundary rather than a +convention: with `admin.dashboard` on, a `GET` outside `/api/` and `/static/` is +answered with the dashboard's page, so a client-side route survives being reloaded +or bookmarked. Inside `/api/` an unknown path still answers `404` in the envelope, +so a mistyped endpoint cannot hand a client an HTML document to parse, and a `POST` +to a path that does not exist is refused rather than answered with a page. + +The dashboard also serves `/robots.txt` and its hashed assets under `/static/`. +None of that is in the OpenAPI document, which describes the API; see +[The dashboard](dashboard.md). + +### What the caller may do + +`GET /api/v1/access` reports the calling token's own rights. It answers `200` for +everybody, anonymous included: an endpoint whose purpose is to say "you may do +nothing" cannot itself require a right, and it discloses nothing that attempting +the six actions would not. + +```json +{ + "caller": { "kind": "token", "name": "ci", "group": "ops" }, + "global": { + "list": true, "read": true, "create": false, + "update": false, "delete": false, "upload": false + }, + "proxies": { + "alpha": { "read": true, "update": false, "delete": false, "upload": false } + } +} +``` + +- `caller` is `{"kind": "anonymous"}` for a request with no token, or with one + this configuration does not recognise -- the two are deliberately not + distinguished. +- `global` is the six actions as `admin.access` decides them for this caller. +- `proxies` gives the four overridable actions per proxy, with that proxy's own + `access` block applied. It is **absent** -- not empty -- when the caller may not + `list`: the map is keyed by proxy name, so returning it would be a proxy listing + by another route. + +Every value is the same decision the endpoint itself would make, evaluated by the +same code. It exists so a client can disable an action instead of offering it and +being refused; it is not a substitute for handling `401` and `403`, since rights +can change under a running page. + +### The configuration schema + +`GET /api/v1/schema` returns the configuration document as a JSON Schema +(2020-12), from the process that is going to read it. No token, for the same +reason as above: it describes the shape of a configuration, never the contents of +one, and the identical bytes are published in the repository and attached to every +release. + +```bash +curl -s http://127.0.0.1:8081/api/v1/schema | jq '.["$defs"].ProxyName' +``` + +```json +{ + "description": "Letters, digits, `-` and `_`, between 2 and 32 characters.", + "maxLength": 32, + "minLength": 2, + "pattern": "^[A-Za-z0-9_-]{2,32}$", + "type": "string" +} +``` + +Use it to check a document before pushing it, and to build an editor that knows +the rules: it is generated from the same Rust types that enforce them, so it +cannot describe a field that does not exist or a bound nobody applies. See +[Editor support](configuration.md#editor-support) for the other two copies and +when to prefer which. ### Bodies @@ -148,9 +236,17 @@ back: version of something that already exists, and accepting one on a create would let a client that meant to send a `PUT` overwrite a proxy it never read. -A `PUT` whose body names a different proxy than the path is refused. The name -is also the template directory, so a rename through `PUT` would strand the old -directory. +A `PUT` whose body names a different proxy than the path **renames it**. The name +is also where the templates live, so the rename moves them: the proxy answers at the +new path, the old one is a `404`, and every template file the proxy had is under the +new name. A name another proxy already holds is refused with `400` +`CONFIG_INVALID`. + +The configuration is written before the templates are moved, because the write is +what authorises moving them. If the move then fails -- a permission problem, a full +disk -- the reply says so in full: the proxy has the new name and its templates are +still under the old one, so a mock naming a template file fails to render until they +are moved by hand. ## Templates @@ -173,6 +269,24 @@ Deleting a proxy removes its templates. Updating one removes the templates no remaining mock names. Both happen *after* the configuration write, so a rejected change leaves every file in place. +### A write is in force when it answers + +`POST`, `PUT` and `DELETE` promote the stored configuration to the running one +before they reply, so a `200` means the process is serving what you just wrote: +the next request through the proxy listener goes to the new upstream, and +`GET /api/v1/status` reports the new revision. No reload in between. + +It used to take one, and that was a trap worth naming: the write returned `200`, +the API read the new document back, and the traffic kept going where it went +before -- the API agreeing with the operator while the process disagreed with +both. + +If the promotion itself fails -- the store became unreadable, or another writer +left the document invalid between the save and the load -- the response says so +and names both halves: the configuration is stored, the process is still running +the previous one, and `POST /api/v1/config/reload` retries. The write is not +undone, because it is not this process's to undo. + ## Reload `POST /api/v1/config/reload` promotes the stored configuration to the running @@ -200,11 +314,17 @@ the configuration file out of band could add a token for themselves and reload it into effect. Same effect as `doppel config reload`, and the two share one implementation -and one mutex, so they cannot swap runtimes in the wrong order. +and one mutex -- with the writes above, which promote through the same path, so +none of the three can swap runtimes in the wrong order. + +What is left for it, now that a write applies itself: a configuration changed +**behind** the API. An operator editing `main.yaml`, `doppel config push`, another +instance writing to the same database -- none of those pass through a handler, so +none of them can promote themselves. ## Status -`GET /status` reports what the process is serving right now -- from the +`GET /api/v1/status` reports what the process is serving right now -- from the running runtime, not from the store, because a configuration written but not reloaded is not what this process is doing. @@ -244,6 +364,7 @@ and an oversized body all carry the envelope rather than an empty body. | `METHOD_NOT_ALLOWED` | 405 | The path exists and does not accept that verb; the response also carries `Allow` | | `PROXY_NOT_RESOLVED` | 404 | No proxy matched and there is no default | | `NO_PROXIES_CONFIGURED` | 503 | The configuration names no proxies at all. See [No proxies configured](proxying.md#no-proxies-configured) | +| `DASHBOARD_NOT_BUILT` | 503 | `GET /` on a binary compiled without the dashboard's assets. See [When the root answers 503](dashboard.md#when-the-root-answers-503) | | `CONFLICT` | 409 | The name exists, or the store is under sustained contention | | `REVISION_MISMATCH` | 409 | The proxy changed since it was read | | `UPLOAD_TOO_LARGE` | 413 | A template body over `admin.upload.limit`, or a configuration document over 1 MiB | @@ -263,9 +384,14 @@ tell "that already exists" from "you are holding a stale copy". ## OpenAPI `GET /openapi.json` serves a document generated from the handlers themselves, -so it cannot describe an endpoint this binary does not serve. `GET -/swagger-ui` serves a browser UI over it, with the assets built into the -binary rather than fetched at runtime. +so it cannot describe an endpoint this binary does not serve. `GET /swagger-ui/` +serves a browser UI over it, with the assets built into the binary rather than +fetched at runtime. + +The UI sits outside `/api/` because it is a page rather than an endpoint: it is +served to a browser, it belongs beside the dashboard's own pages, and an ingress +routing `/api/*` to a JSON service would otherwise hand a browser HTML through a +JSON path. Both are unauthenticated: they describe the API rather than expose any of it, and a client cannot authenticate before it knows how. diff --git a/docs/usage/configuration.md b/docs/usage/configuration.md index d894bcf..09bb124 100644 --- a/docs/usage/configuration.md +++ b/docs/usage/configuration.md @@ -23,6 +23,17 @@ That URL follows `main`. Every release also attaches the schema as an asset, so a deployment that pins a version can validate against the schema for exactly that version rather than for whatever is current. +A running Doppel serves its own: + +```bash +curl -s http://127.0.0.1:8081/api/v1/schema | jq . +``` + +No token, because it describes the shape of a configuration rather than the +contents of one. This is the copy to check a document against before pushing it, +since it comes from the process that is going to read it -- the dashboard uses it +for exactly that, both while a field is being typed and in its YAML editor. + The schema is generated from the same Rust types this page documents -- see [`doppel config schema`](cli.md#config-schema) -- so it cannot describe a field that does not exist, and CI fails if the checked-in copy falls behind. @@ -115,6 +126,8 @@ The admin listener's address, its tokens, and who may do what. ```yaml admin: enable: true + dashboard: true + title: "Doppel" host: "0.0.0.0" port: 8081 public: false @@ -138,7 +151,7 @@ admin: `enable` defaults to `true`. Set it to `false` to run the proxy with no admin application at all: the port is never bound, so it cannot collide with -anything, and `/status`, `/metrics`, `/openapi.json` and the whole API are +anything, and `/api/v1/status`, `/metrics`, `/openapi.json` and the whole API are gone with it. The proxy listener and the control socket are untouched, which makes `doppel config reload` the only remaining way in. @@ -147,6 +160,14 @@ because nothing serves it is a trap set for whoever turns the listener on later, and they will not re-read the rules first -- so rule V34 still refuses a public write action with the listener off. +`dashboard` defaults to `true` and serves the browser dashboard from this +listener's root: `/`, `/static/{path}` and `/robots.txt`. `false` leaves those +three unrouted -- they answer 404 like any other unknown path -- and changes +nothing about the JSON API. `title` is the heading that dashboard shows and the +browser tab's name, at most 64 characters and free of control characters, +defaulting to `Doppel`. Both take effect on restart, since the routes are built +once. See [The dashboard](dashboard.md). + Toggling it takes effect on restart, not on reload; a reload reports `admin` among the sections it could not apply. @@ -238,7 +259,7 @@ it has to be a choice, not what happens when the section is left out. Setting a *write* action to `public` is refused outright (rule V34); no configuration wants an unauthenticated caller rewriting the proxy set. -`GET /status` stays unauthenticated regardless: it reports names, upstreams +`GET /api/v1/status` stays unauthenticated regardless: it reports names, upstreams and counts, and strips any credentials from the upstream before printing it. diff --git a/docs/usage/dashboard.md b/docs/usage/dashboard.md new file mode 100644 index 0000000..009231f --- /dev/null +++ b/docs/usage/dashboard.md @@ -0,0 +1,285 @@ +# The dashboard + +![The proxy list: each proxy with its upstream, how it is resolved, the faults on it and how many mocks it has](../screenshots/screenshot-proxies.webp) + +The admin listener serves a browser dashboard from its own root. It lists the +proxies, edits them, shows what the process is doing, and reloads the configuration -- +most of what the admin API does, without curl. Template files are the exception, and +deliberately: see [below](#what-the-pages-do). + +It is on by default. Open the admin port in a browser: + +``` +http://127.0.0.1:8081/ +``` + +The whole thing is compiled into the binary. There is nothing to deploy, no +directory to serve, and no version of the page that can disagree with the version +of Doppel serving it. + +It is served compressed: the listener negotiates `br` and `gzip` from +`Accept-Encoding`, so the 140 KB of JavaScript and CSS the page is made of goes +out as 74 KB or less, and `index.html` is minified on the way into the binary. +The proxy listener is deliberately left out of that -- a client under test has to +see the encoding its upstream chose. + +Every URL the dashboard shows is a real one. `/proxies/alpha` can be bookmarked, +reloaded and shared, because everything outside `/api/` and `/static/` is answered +with the page -- which is also why the API lives under `/api/` in the first place: +`/status` used to be an endpoint *and* a page, and a reload got the JSON. + +--- + +## Turning it on and off + +```yaml +admin: + dashboard: true # the default + title: "Doppel" # the default +``` + +`dashboard: false` leaves `/`, `/static/*` and `/robots.txt` unrouted -- they +answer 404 like any other unknown path -- and changes nothing about the JSON API. +Use it when the admin port is reachable from somewhere a browser page has no +business being. To turn off the admin API entirely, that is `admin.enable`, not +this. + +Both fields take effect on restart, because the routes are built once at startup. + +`title` is what the header shows and what the browser tab is called. It is worth +setting when several Doppels are open at once: + +```yaml +admin: + title: "billing-api (staging)" +``` + +At most 64 characters, no control characters, and any language you like -- it is a +heading, not an identifier. + +Leave `title` out and the header shows the project's wordmark instead of a default +string. Set it, and the header shows exactly what you wrote: the browser tab takes +the same text either way. + +--- + +## Who can use it + +The dashboard is a client of the admin API and is bound by exactly the same +`access` rules. It has no privileges of its own, and there is no dashboard +password: the token is the one from `admin.tokens`. + +What follows from that is worth stating plainly, because it surprises people: + +**The page works without a token.** If `access.list` and `access.read` are +`public`, an anonymous visitor sees the proxy list and can open a proxy. The token +dialog opens once, and it has a "Continue without a token" button -- declining +leaves a working page rather than a wall. + +**A refusal offers the two things that fix it.** "Read of `alpha` requires access +`read`" is about the token, and a page reads once when it opens -- so the banner +carries **Enter token** and **Refresh**. Refreshing is a button rather than something +the page does when a token arrives: a form holds a half-typed document, and replacing +that because someone signed in elsewhere would lose work. + +![The token dialog over a refusal: the banner naming the access `list` needs and carrying Enter token and Refresh, and the dialog offering Continue without a token beside Use this token](../screenshots/screenshot-signin.webp) + +**Actions the caller may not perform are visible but disabled**, with the reason +in the tooltip. The page asks the server what the caller may do +(`GET /api/v1/access`) rather than guessing, so a disabled button means the server +would have refused. A missing button reads as a broken page; a disabled one +explains itself. + +**Signing out returns to the anonymous view.** It does not lock the page. + +### The token in the browser + +Entering a token stores it in `localStorage` for one hour, then the page forgets +it and asks again. Two things that is not: + +- **Not a session.** The token does not expire on the server. Forgetting it here + only means this browser stops presenting it. Revoking a token is a + configuration change -- edit `admin.tokens` and reload. +- **Not a security boundary.** Anything running in the page can read + `localStorage`. The hour exists so an unattended browser stops holding a working + admin token indefinitely, which is a smaller and more honest claim. + +On a `public: true` deployment the page never mentions tokens at all. + +--- + +## What the pages do + +**Proxies** lists every proxy with its upstream, how it is resolved, the faults +configured on it and how many mocks it has. The list refetches itself once a +minute -- so a change made by another operator, by `doppel config push`, or by a +reload appears without anyone pressing anything. It pauses while the tab is +hidden. + +Editing opens a form over the whole proxy document: the upstream, timeouts, body +limits, resolution, injected headers, the three fault settings, per-proxy access +overrides, and the mocks with their matching rules and responses. + +![Editing a proxy: name, type and upstream always on screen, an (i) beside each label, Save and Cancel pinned to the bottom](../screenshots/screenshot-proxy-edit.webp) + +Every field carries an (i) to its section of [Every +parameter](parameters.md) -- the type, the bounds, an example in place -- and the +link is versioned, so it describes the Doppel serving the page rather than whatever +has been released since. + +A field is checked as it is typed: a name with a space in it, a timeout of 5000, a +loss rate of 45, a selector without its leading dot. Those bounds are read from +[`GET /api/v1/schema`](admin-api.md#the-configuration-schema) at load, so they are +the server's own rules rather than a second copy in the page -- and what needs more +than one field to judge is still the server's answer on save, reported against the +field it is about. + +A proxy has thirteen fields and any number of mocks, and most edits touch one of +them -- so the name, the type and the upstream are always on screen and everything +else is a folded section that says what is inside it: `Faults none`, +`Access overrides inherited`, `Mocks 3`. Save and Cancel sit in a bar pinned to the +bottom of the window, so they are reachable whatever is open. + +On a `public: true` deployment the `Access overrides` section is not there at all. +Nothing in a proxy's `access` block decides anything while the admin API is +unauthenticated -- every action answers as public regardless, which the process +also says among its startup advisories -- so four fields that change no outcome +would be the page disagreeing with the binary serving it. A proxy whose document +still holds overrides says so in one line instead, because the YAML mode shows an +`access` block the form otherwise has no section for. + +The name can be edited like any other field. Renaming moves the proxy's template +files with it, and the page says which happened -- "Renamed alpha to billing-api" +rather than "Updated". Clients selecting that proxy by `X-Proxy-Name` have to use the +new name from then on, which is the one thing a rename cannot do for you. + +**A save that removes something asks first**, and names what: `the mock +\`one-widget\``, `the injected header \`X-Trace\``, `\`filter\` from variables from the +query of \`one-widget\``. The form has a Remove button beside every entry and no undo, +so the question belongs on the button that makes those removals real rather than on +each one along the way -- taking one mock out and putting another in is a single edit. +A save that only changes values does not ask: a changed field is visible in the field +it is in. + +Deleting a proxy asks too, and says what goes with it: its template files. + +Saving sends the revision the form was loaded at, so a proxy someone else changed +in the meantime is refused rather than overwritten -- the page says so and asks you +to reload. + +### Editing a proxy as YAML + +**Edit as YAML** swaps the form for the whole proxy document, in the shape +`main.yaml` has it: + +```yaml +name: alpha +type: http +url: https://alpha.example.com/api/ +timeout: 30 +mocks: + - name: one-widget + request: + method: GET + url: ^/widgets/(?P\d+)$ + response: + status: 200 + json: '{"id": "{{ id }}"}' +``` + +![The same proxy as YAML: one coloured editor over the whole document, with Reformat and a documentation link above it](../screenshots/screenshot-proxy-edit-yaml.webp) + +The form is better for changing one field; this is better for everything else -- +pasting a proxy from a colleague, reordering mocks, copying a block out of a +configuration file. Tab indents inside the document, **Reformat** tidies it, and +saving tidies it too, so a refused save leaves readable YAML on screen. + +It is checked against the running Doppel's own schema as you type, and Save is +refused while it does not match: a field the schema does not know is named, and so +is a value out of range. What needs more than one field to judge is still the +server's answer. + +Two things it does not do. It does not keep comments -- the document is stored as +data, so there is nowhere to put them -- and it edits one proxy rather than the +whole configuration. For the whole file, edit `main.yaml` and reload. + +**Template files are not managed here.** The dashboard had a section for them and it +was the wrong shape: a file store inside a form over a document, with a Save of its +own that did not wait for the form's, and a rule the page could not enforce -- the +server refuses an upload no saved mock declares. They stay in the +[admin API](admin-api.md#templates), which is where `doppel config push` and any +script already write them. + +What the form does with a mock that answers with a template file: it shows which file, +in a field it will not let you edit, and lets you move that mock onto a text or JSON +body instead. A mock that has no template cannot be given one from here -- the answer +choices are Text body and JSON body. Switching away from a template is undoable while +the page is open, because the name cannot be typed back in. + +**Status** shows uptime, the configuration revision in effect and the resolution +state of each proxy, and carries the reload button. Saving in the form needs no +reload -- a write through the API is in force when it answers -- so the button is +for a configuration changed behind the dashboard: `main.yaml` edited by hand, +`doppel config push`, another instance. Reload re-reads the store and reports what +it applied, including the sections that need a restart instead. + +**API** is not a page of the dashboard but a link to the Swagger UI the same +listener serves, at `/swagger-ui/`. It opens in a new tab, because it is a +different application and going there should not throw away a half-filled form. + +--- + +## Theme + +Light, dark, or whatever the operating system says. The default is the last of +those, and the choice is remembered per browser. + +--- + +## Not indexed + +The dashboard refuses search engines three ways, because crawlers fail +differently: an `X-Robots-Tag: noindex, nofollow, noarchive` header on every +response including the errors, a `robots` meta element in the page for a crawler +that reads markup and ignores headers, and `robots.txt` disallowing everything. + +The page also carries a content security policy that permits scripts, styles and +connections from this origin only, and forbids framing. No `unsafe-inline` +anywhere: the configuration reaches the page as the contents of a JSON element +rather than as an inline script. + +One inline stylesheet is allowed, and by its hash rather than by opening the +policy -- the code editor's own, which every editor on the page renders. Blocking it +changed nothing visible and reported a violation to the console for each editor, +which is a poor way to treat the place an operator looks for their own problems. A +test recomputes that hash from the installed library, so an upgrade that changes the +stylesheet fails a build rather than filling a console. + +--- + +## When the root answers 503 + +```json +{ + "status": "error", + "code": "DASHBOARD_NOT_BUILT", + "message": "this binary was built without the dashboard's static assets ..." +} +``` + +The binary was compiled without `frontend/dist`. Released binaries and the +published image always carry the dashboard -- the release refuses to publish one +that does not, and `docker build` builds the assets itself when they are absent -- +so this is a binary built from source without building the frontend first: + +```bash +npm --prefix frontend ci +npm --prefix frontend run build +cargo build --release -p doppel-cli +``` + +That order matters: the assets are embedded at compile time, so rebuilding them +without rebuilding the binary changes nothing about what it serves. + +If Node is not available and the JSON API is enough, `admin.dashboard: false` +stops the route being served at all, which is a clearer answer than a 503. diff --git a/docs/usage/docker.md b/docs/usage/docker.md index ca56556..81eca2d 100644 --- a/docs/usage/docker.md +++ b/docs/usage/docker.md @@ -34,7 +34,7 @@ start. ```bash docker run --rm \ -p 8080:8080 -p 8081:8081 \ - -v "$PWD/main.yaml:/etc/doppel/main.yaml:ro" \ + -v "$PWD/config:/etc/doppel" \ -v doppel-templates:/var/lib/doppel/templates \ loremdev/doppel:1.2.3-alpine ``` @@ -42,6 +42,16 @@ docker run --rm \ Two mounts, and the second is not optional if you use templates at all -- see [Templates](#templates) below. +**Mount the directory, not the file.** `-v "$PWD/main.yaml:/etc/doppel/main.yaml"` +looks tidier and breaks every write: a save puts the new configuration in a +temporary file and renames it over the old one, and nothing can rename over a mount +point. Reads work, so the dashboard lists proxies happily and then refuses to save +one with `Resource busy (os error 16)`. With the directory mounted, the +configuration inside it is an ordinary file and the rename stays within the mount. + +Add `:ro` to the mount if nothing should write the configuration -- then the admin +API's writes are refused for a reason it can state. + The configuration must bind `0.0.0.0` rather than `127.0.0.1`, or nothing outside the container can reach it: @@ -68,7 +78,7 @@ missing volume. docker volume create doppel-templates docker run --rm \ -p 8080:8080 -p 8081:8081 \ - -v "$PWD/main.yaml:/etc/doppel/main.yaml:ro" \ + -v "$PWD/config:/etc/doppel" \ -v doppel-templates:/var/lib/doppel/templates \ loremdev/doppel:1.2.3-alpine ``` @@ -100,8 +110,10 @@ curl -X POST -H "X-Proxy-Authorization: Bearer $TOKEN" \ docker exec doppel config reload --socket /tmp/doppel.sock ``` -Editing a bind-mounted `main.yaml` on the host changes the file the container -reads, so a reload picks it up without restarting anything. +Editing `config/main.yaml` on the host changes the file the container reads, so a +reload picks it up without restarting anything. It works the other way too: a write +through the admin API rewrites that file, in canonical form -- so comments in it do +not survive an edit made from the dashboard. ## Tokens @@ -133,17 +145,17 @@ services: - "8080:8080" - "8081:8081" volumes: - - ./main.yaml:/etc/doppel/main.yaml:ro + - ./config:/etc/doppel - doppel-templates:/var/lib/doppel/templates environment: DOPPEL_ADMIN_TOKENS: '{"ci":{"token":"...","group":"admin"}}' healthcheck: - # `/status` needs no token by default and answers only once the runtime + # `/api/v1/status` needs no token by default and answers only once the runtime # is compiled and both listeners are bound. `-f` matters: without it # curl exits 0 on a 500 and the check passes for answering at all rather # than for answering correctly. The port is the one inside the # container. - test: ["CMD", "curl", "-fsS", "-o", "/dev/null", "http://127.0.0.1:8081/status"] + test: ["CMD", "curl", "-fsS", "-o", "/dev/null", "http://127.0.0.1:8081/api/v1/status"] interval: 5s timeout: 3s start_period: 5s @@ -170,23 +182,68 @@ an `https` upstream, and every request fails with `UPSTREAM_ERROR`. ## Building it yourself -The binary is built outside the Dockerfile and copied in. Building Rust inside -a multi-architecture `buildx` means QEMU for the non-native architecture, which -turns a two-minute compile into most of an hour. Releases build each -architecture on a runner of that architecture instead. +```bash +make image +``` + +That builds the dashboard, then the image. Nothing else is needed and no Rust +toolchain has to be able to target Linux: when `dist/` holds no binary the +Dockerfile compiles one in its own builder stage, for the platform being built, +and the builder stage is discarded -- the published image is Alpine plus the +binary, around 65 MB. + +`make image-rebuild` does the same with no npm or docker cache, and with `dist/` +emptied first so the compile definitely happens. + +### Why it prefers a staged binary + +`docker build .` uses `dist//doppel`, or `dist/doppel`, if either is +there, and compiles only when neither is. That preference is what keeps releases +fast: building Rust inside a multi-architecture `buildx` means QEMU for the +non-native architecture, which turns a two-minute compile into most of an hour. +The release workflow builds each architecture on a runner of that architecture, +stages the two binaries, and points the builder stage at a base with no toolchain +in it -- so the release never pulls a Rust image to run a `cp`. + +To stage one by hand, on Linux with `musl-tools` installed: ```bash -# On Linux, with musl-tools installed: CC_x86_64_unknown_linux_musl=musl-gcc \ CARGO_TARGET_X86_64_UNKNOWN_LINUX_MUSL_LINKER=musl-gcc \ cargo build --release --target x86_64-unknown-linux-musl -p doppel-cli mkdir -p dist && cp target/x86_64-unknown-linux-musl/release/doppel dist/ -docker build --build-arg BIN=dist/doppel -t doppel:dev . +docker build -t doppel:dev . ``` -On macOS there is no musl toolchain to install; `cargo zigbuild` or `cross` -will produce the binary if you want to build the image locally from there. +It has to be a musl build. A glibc binary does not run on Alpine, and the failure +is a bare `not found` from the shell rather than anything that says why. On macOS +there is no musl toolchain to install, which is the reason the Dockerfile compiles +inside: `ring` builds C, so a musl target needs a musl C compiler and not only the +Rust standard library for it. + +### The dashboard + +It is embedded at compile time, so the compile branch needs it: a binary built +without `frontend/dist` starts, serves the API and answers 503 at its own root. The +image therefore builds it when the context has no `frontend/dist`, with the +`nodejs`/`npm` packages installed in the builder stage for exactly that. + +``` +doppel: no dashboard in frontend/dist, building it +``` + +Building it on the host first is faster and is what `make image` does -- a working +checkout has it already, and `npm ci` inside a container downloads the whole +dependency tree again. It is not required. + +### When it refuses to build + +Two refusals, both deliberate: -It has to be a musl build. A glibc binary does not run on Alpine, and the -failure is a bare `not found` from the shell rather than anything that says -why. +- **"there is no built dashboard ... and no frontend sources"** -- neither + `frontend/dist` nor the files `vite build` reads reached the context. That means + `.dockerignore` was edited, or the build ran from somewhere other than the + repository root. Stage a binary, or fix the context. +- **"this builder image has no Rust toolchain"** -- something passed + `--build-arg BUILDER=` naming an image without cargo, and there was no staged + binary to copy. Stage one, or drop the argument. diff --git a/docs/usage/faults.md b/docs/usage/faults.md index ce02490..0ad27fe 100644 --- a/docs/usage/faults.md +++ b/docs/usage/faults.md @@ -227,7 +227,7 @@ because it needs both fields. ## What faults do not do -- They do not apply to the admin listener. `/status` and `/metrics` answer +- They do not apply to the admin listener. `/api/v1/status` and `/metrics` answer normally while the proxy is dropping traffic, which is what makes them usable for watching it happen. - They are not recorded separately in metrics. A lost request appears as its diff --git a/docs/usage/getting-started.md b/docs/usage/getting-started.md index 8b8efac..bb508a7 100644 --- a/docs/usage/getting-started.md +++ b/docs/usage/getting-started.md @@ -123,3 +123,16 @@ forwarding it: Everything else still goes upstream. See [Mocks and templating](mocks.md) for variables, templates, and the ordering rule that matters once you have more than one mock. + +## Or do all of that in a browser + +The admin port serves a dashboard from its root: + +``` +http://127.0.0.1:8081/ +``` + +The proxy set, a form over every field of a proxy, the mock templates, and the +reload button -- the same admin API this page drives with curl, with the same +token rules. `admin.dashboard: false` turns it off. See +[The dashboard](dashboard.md). diff --git a/docs/usage/observability.md b/docs/usage/observability.md index ee0a22e..edf7c46 100644 --- a/docs/usage/observability.md +++ b/docs/usage/observability.md @@ -63,16 +63,52 @@ sending events, in the same position a URL carries a password. `GET /metrics` on the admin listener, in the Prometheus text format, unauthenticated because a scraper has nowhere to put a token. +The one endpoint outside `/api/`. `metrics_path` defaults to `/metrics` in +Prometheus and in every agent and annotation that scrapes one, so the path is +settled by something larger than this project -- and a scrape of `/metrics` while +the exposition was under `/api/v1/` was answered by the dashboard with `200 +text/html`, which reads as a parse failure or as no metrics at all. + +### Traffic + | Metric | Type | Labels | |---|---|---| -| `doppel_proxy_request_duration_seconds` | histogram | `proxy`, `method`, `status` | +| `doppel_proxy_request_duration_seconds` | histogram | `proxy`, `method`, `status`, `replace`, `loss`, `upstream_error` | | `doppel_upstream_request_duration_seconds` | histogram | `proxy`, `method`, `status` | +| `doppel_admin_request_duration_seconds` | histogram | `route`, `method`, `status` | | `doppel_loss_total` | counter | `proxy` | | `doppel_latency_injected_total` | counter | `proxy` | | `doppel_mock_hits_total` | counter | `proxy`, `mock` | -Buckets are explicit -- 5ms to 10s -- rather than the exporter's default -summary, because a quantile cannot be aggregated across replicas. +`replace`, `loss` and `upstream_error` are `1` or `0`, and independent: a mock can +answer a request its own loss roll then drops, and `upstream_error` covers a +transport failure, a timeout, and a relayed status of 500 or above. + +`route` on the admin histogram is the route template -- `/api/v1/proxies/{name}`, +never `/api/v1/proxies/alpha` -- so a hundred proxies are one series and a query +string is none. A request that matched no route is recorded with an empty `route`. + +Buckets are explicit rather than the exporter's default summary, because a +quantile cannot be aggregated across replicas. Two ladders: 5ms to 60s for +proxied traffic, because `latency` injection is asked to be slow on purpose, and +5ms to 5s for the admin API, where nothing has any business taking longer. + +### State + +These exist from startup, before anything has happened, because a panel and an +alert both read a never-recorded metric as "no data" -- which is +indistinguishable from a process nobody is scraping. + +| Metric | Type | Labels | Meaning | +|---|---|---|---| +| `doppel_build_info` | gauge | `version` | Always `1`. The version is a label because it is not a number | +| `doppel_dashboard_info` | gauge | `enabled` | Always `1`. Separate from the build: one describes the artifact, the other what this deployment turned on | +| `doppel_proxy_last_error_timestamp_seconds` | gauge | `code` | When the proxy listener last answered with that error. Starts as `code=""`, value `0` | +| `doppel_proxy_mocks` | gauge | `proxy` | Mocks per proxy in the configuration **now in effect**, republished on every reload and every write through the API. A proxy that leaves the configuration goes to `0` rather than keeping its last count | + +"Is it still failing" is `time() - doppel_proxy_last_error_timestamp_seconds`, +which is why the metric is a timestamp and not a counter: a counter answers "how +many" and needs a rate window to say anything about now. Every request through the proxy listener appears in the proxy histogram, including one that was dropped by loss injection or that resolved to no proxy diff --git a/docs/usage/parameters.md b/docs/usage/parameters.md new file mode 100644 index 0000000..ba36571 --- /dev/null +++ b/docs/usage/parameters.md @@ -0,0 +1,1463 @@ + + +# Every parameter + +Every field a configuration document may carry, with its type, its bounds and an +example. Generated from [the schema](configuration.md#editor-support), which is +generated from the Rust types that enforce the rules -- so a bound stated here is +the bound the server applies. + +The dashboard links here: the (i) beside a field opens that field's section. + +For how the pieces fit together, read [Configuration +reference](configuration.md); for how to use them, the guides under Usage. + +--- + +## `admin` { #admin } + +The admin API's listener, its tokens, and who may do what. + +| | | +|---|---| +| Type | object | +| Required | yes | +| Rules | no unknown keys | + +```yaml +admin: + host: 127.0.0.1 + port: 8080 + upload: + limit: 1048576 +``` + +### `admin.access` { #admin-access } + +Who may perform each admin action. Every action defaults to the +`admin` group, reads included. + +| | | +|---|---| +| Type | object | +| Required | no | +| Rules | no unknown keys | + +### `admin.access.create` { #admin-access-create } + +Add a proxy. Refused for `public` by rule V34. + +| | | +|---|---| +| Type | string or list of string | +| Required | no | + +```yaml +admin: + access: + create: public +``` + +### `admin.access.delete` { #admin-access-delete } + +Remove a proxy. Refused for `public` by rule V34. + +| | | +|---|---| +| Type | string or list of string | +| Required | no | + +```yaml +admin: + access: + delete: public +``` + +### `admin.access.list` { #admin-access-list } + +List the proxies. A listing exposes upstream URLs and injected headers. + +| | | +|---|---| +| Type | string or list of string | +| Required | no | + +```yaml +admin: + access: + list: public +``` + +### `admin.access.read` { #admin-access-read } + +Read one proxy document, credentials in its `url` included. + +| | | +|---|---| +| Type | string or list of string | +| Required | no | + +```yaml +admin: + access: + read: public +``` + +### `admin.access.update` { #admin-access-update } + +Replace a proxy. Refused for `public` by rule V34. + +| | | +|---|---| +| Type | string or list of string | +| Required | no | + +```yaml +admin: + access: + update: public +``` + +### `admin.access.upload` { #admin-access-upload } + +Upload or delete a template file. Refused for `public` by rule V34. + +| | | +|---|---| +| Type | string or list of string | +| Required | no | + +```yaml +admin: + access: + upload: public +``` + +### `admin.auth` { #admin-auth } + +Which header carries the bearer token. + +| | | +|---|---| +| Type | object | +| Required | no | +| Rules | no unknown keys | + +### `admin.auth.header` { #admin-auth-header } + +The header a caller presents its token in, as `Bearer `. +Defaults to `X-Proxy-Authorization`. + +| | | +|---|---| +| Type | string | +| Required | no | + +```yaml +admin: + auth: + header: X-Request-Id +``` + +### `admin.dashboard` { #admin-dashboard } + +Serve the browser dashboard from the admin listener's root. + +On by default. Off means `/`, `/static/*` and `/robots.txt` are not +routed at all and answer 404 like any other unknown path; the JSON API is +untouched either way. + +An `Option` and skipped when absent, for the reason given on `groups`: +adding it must not change the canonical YAML of configurations written +before it existed. Read it through +[`AdminConfig::is_dashboard_enabled`]. + +| | | +|---|---| +| Type | boolean | +| Required | no | + +```yaml +admin: + dashboard: true +``` + +### `admin.enable` { #admin-enable } + +Whether to run the admin listener at all. + +Defaults to on. Off means the port is never bound and no admin task +starts; the proxy and the control socket are untouched, so +`doppel config reload` still works and is then the only way in. + +The validation rules do not consult this. A configuration that is only +safe because nothing serves it is a trap set for whoever turns the +listener on later, and they will not re-read the rules first. + +| | | +|---|---| +| Type | boolean | +| Required | no | + +```yaml +admin: + enable: true +``` + +### `admin.groups` { #admin-groups } + +Which names `access` may reference, here and in a proxy's overrides. + +`["*"]`, the default, allows any. A concrete list allows exactly those: +`["admin", "ci"]` permits `admin` and `ci` and refuses `user`. + +`[]` names nobody, which leaves no action anything to reference -- so it +means the same thing as `public: true`, and is treated as it. It used to +mean a configuration with no legal value for `create`, `update`, `delete` +or `upload`: V36 refused their `admin` default and V34 refused `public`, +so the document could not be written at all. + +`public` and `admin` are never governed by this. `public` is the absence +of a subject rather than a name; `admin` is the fallback every action +already has, and a list that revoked it would produce the same +unsatisfiable state as `[]` once did. + +Checked by rule V36, not by this type: it compares one field against +another, which is what is left for the rule set once the types have taken +everything they can decide alone. + +An `Option` rather than a `Vec` defaulting to `["*"]`, and skipped when +absent, so that adding this field did not change the canonical YAML of +every configuration written before it existed. The revision is derived +from that YAML, so a materialised default would have made every stored +configuration fail its own revision check on the first load after the +upgrade. Read it through [`AdminConfig::allowed_groups`]. + +| | | +|---|---| +| Type | array | +| Required | no | + +```yaml +admin: + groups: + - "*" +``` + +### `admin.host` { #admin-host } + +An IP address, not a hostname: a name would have to be resolved, +and which address it resolves to is not the configuration's to +decide. `utoipa` has no schema for `IpAddr`, so it is described +here as the string it is written as. + +| | | +|---|---| +| Type | string | +| Required | yes | + +```yaml +admin: + host: 127.0.0.1 +``` + +### `admin.port` { #admin-port } + +The TCP port the admin API listens on. Must differ from +`server.port`. + +| | | +|---|---| +| Type | integer | +| Required | yes | +| Rules | 1 to 65535 | + +```yaml +admin: + port: 8080 +``` + +### `admin.public` { #admin-public } + +Serve the whole admin API unauthenticated. + +`false` by default. `true` makes every action `public` and leaves no name +to reference, so `groups` is effectively empty and `access` effectively +all-public -- whatever either of them says. Anything they did say is +reported as a startup advisory rather than refused, so a configuration +being made temporarily public does not have to be gutted first and +rebuilt afterwards. + +This overrides rule V34, which otherwise refuses a public write action. +V34 exists so that an unauthenticated writable proxy set cannot happen by +omission; a field named `public` set to `true` is not an omission. Think +of it as the flag V34 was holding the line for. + +An `Option` and skipped when absent, for the reason given on `groups`: +adding it must not change the canonical YAML of configurations written +before it existed. Read it through [`AdminConfig::is_public`]. + +| | | +|---|---| +| Type | boolean | +| Required | no | + +```yaml +admin: + public: true +``` + +### `admin.title` { #admin-title } + +The heading the dashboard shows, and the browser tab's title. + +Defaults to `Doppel`. Useful when several Doppels are open at once and +the tabs are otherwise identical. + +An `Option` and skipped when absent, for the same reason as `dashboard`. +Read it through [`AdminConfig::title`]. + +| | | +|---|---| +| Type | string | +| Required | no | +| Rules | 1 to 64 characters | + +```yaml +admin: + title: billing-api (staging) +``` + +### `admin.tokens` { #admin-tokens } + +The tokens that may call the admin API. Names and token values are +each unique. `DOPPEL_ADMIN_TOKENS` can supply these instead. + +| | | +|---|---| +| Type | list of object | +| Required | no | + +```yaml +admin: + tokens: + - name: billing-api + group: billing-api + token: replace-me-00000000000000000000000 +``` + +### `admin.tokens[].group` { #admin-tokens-group } + +The group it belongs to. `admin` and `user` are predefined; any other +name must be carried by at least one token. + +| | | +|---|---| +| Type | string | +| Required | yes | +| Rules | matches `^[A-Za-z0-9_-]{2,64}$`; 2 to 64 characters | + +```yaml +admin: + tokens: + - group: billing-api +``` + +### `admin.tokens[].name` { #admin-tokens-name } + +What to call this token in `access` lists and in logs. Never the +secret itself. + +| | | +|---|---| +| Type | string | +| Required | yes | +| Rules | matches `^[A-Za-z0-9_-]{2,64}$`; 2 to 64 characters | + +```yaml +admin: + tokens: + - name: billing-api +``` + +### `admin.tokens[].token` { #admin-tokens-token } + +The secret the caller sends. A version 4 UUID is the recommended +shape. + +| | | +|---|---| +| Type | string | +| Required | yes | +| Rules | 32 to 255 characters | + +```yaml +admin: + tokens: + - token: replace-me-00000000000000000000000 +``` + +### `admin.upload` { #admin-upload } + +Bounds an uploaded template file. + +| | | +|---|---| +| Type | object | +| Required | yes | +| Rules | no unknown keys | + +```yaml +admin: + upload: + limit: 1048576 +``` + +### `admin.upload.limit` { #admin-upload-limit } + +Largest template file the admin API accepts. A larger upload is +refused with `413`. + +| | | +|---|---| +| Type | integer or string | +| Required | yes | +| Rules | 1 to 1073741824 | + +```yaml +admin: + upload: + limit: 1048576 +``` + +--- + +## `control` { #control } + +The Unix socket `doppel config reload` talks to. + +| | | +|---|---| +| Type | object | +| Required | no | +| Rules | no unknown keys | + +### `control.socket` { #control-socket } + +Path to the control socket, created with mode 0600 and removed on +shutdown. Its parent directory must already exist. +A filesystem path. `utoipa` has no schema for `PathBuf`, so it is +described as the string it is written as. + +| | | +|---|---| +| Type | string | +| Required | no | + +```yaml +control: + socket: /tmp/doppel.sock +``` + +--- + +## `logging` { #logging } + +Log level and output format. + +| | | +|---|---| +| Type | object | +| Required | no | +| Rules | no unknown keys | + +### `logging.format` { #logging-format } + +`json` for machines, `text` for a terminal. + +| | | +|---|---| +| Type | string | +| Required | no | +| Rules | one of `json`, `text` | + +```yaml +logging: + format: json +``` + +### `logging.level` { #logging-level } + +The lowest level that is logged. `RUST_LOG` overrides it when set and +non-empty. + +| | | +|---|---| +| Type | string | +| Required | no | +| Rules | one of `trace`, `debug`, `info`, `warn`, `error` | + +```yaml +logging: + level: trace +``` + +--- + +## `proxies` { #proxies } + +The proxies this instance serves, in the order they are tried. + +May be empty or left out: Doppel then starts and serves the admin API, +and a request is answered `503 NO_PROXIES_CONFIGURED` until a proxy is +added by reload or over that API. + +| | | +|---|---| +| Type | list of object | +| Required | no | + +```yaml +proxies: + - name: billing-api + type: http + url: https://example.com/api/ +``` + +### `proxies[].access` { #proxies-access } + +Overrides the admin `access` rules for this proxy alone. + +| | | +|---|---| +| Type | object | +| Required | no | +| Rules | no unknown keys | + +### `proxies[].access.delete` { #proxies-access-delete } + +Who may remove this proxy. Absent leaves the global rule. + +| | | +|---|---| +| Type | string or list of string | +| Required | no | + +```yaml +proxies: + - access: + delete: public +``` + +### `proxies[].access.read` { #proxies-access-read } + +Who may read this proxy's document. Absent leaves the global rule. + +| | | +|---|---| +| Type | string or list of string | +| Required | no | + +```yaml +proxies: + - access: + read: public +``` + +### `proxies[].access.update` { #proxies-access-update } + +Who may replace this proxy. Absent leaves the global rule. + +| | | +|---|---| +| Type | string or list of string | +| Required | no | + +```yaml +proxies: + - access: + update: public +``` + +### `proxies[].access.upload` { #proxies-access-upload } + +Who may upload templates for this proxy. Absent leaves the global rule. + +| | | +|---|---| +| Type | string or list of string | +| Required | no | + +```yaml +proxies: + - access: + upload: public +``` + +### `proxies[].body_limit` { #proxies-body_limit } + +Bounds the request body a matched mock is allowed to buffer in order +to extract from it; phase 1 streams bodies deliberately, and reading +`.content.items` needs the whole thing in hand. See rule V33. + +| | | +|---|---| +| Type | integer or string | +| Required | no | +| Rules | 1 to 1073741824 | + +```yaml +proxies: + - body_limit: 1048576 +``` + +### `proxies[].headers` { #proxies-headers } + +Headers injected into every outbound request, overriding whatever the +client sent by the same name. + +| | | +|---|---| +| Type | map of string to string | +| Required | no | + +```yaml +proxies: + - headers: + X-Request-Id: staging +``` + +### `proxies[].latency` { #proxies-latency } + +Makes a share of requests take a chosen time. Applies to mocked +responses too; a mock may override the figure. + +| | | +|---|---| +| Type | object | +| Required | no | +| Rules | no unknown keys | + +```yaml +proxies: + - latency: + percentage: 0.25 + min: 0.25 + max: 0.25 +``` + +### `proxies[].latency.max` { #proxies-latency-max } + +Upper bound of the delay, in seconds. Must be at least `min`. + +| | | +|---|---| +| Type | number | +| Required | yes | +| Rules | 0 to 300 | + +```yaml +proxies: + - latency: + max: 0.25 +``` + +### `proxies[].latency.min` { #proxies-latency-min } + +Lower bound of the delay, in seconds. The delay is a target for the +whole response: time the upstream already spent is subtracted. + +| | | +|---|---| +| Type | number | +| Required | yes | +| Rules | 0 to 300 | + +```yaml +proxies: + - latency: + min: 0.25 +``` + +### `proxies[].latency.percentage` { #proxies-latency-percentage } + +The share of requests to delay, as a fraction. + +| | | +|---|---| +| Type | number | +| Required | yes | +| Rules | 0 to 1 | + +```yaml +proxies: + - latency: + percentage: 0.25 +``` + +### `proxies[].loss` { #proxies-loss } + +Drops a share of requests rather than forwarding them. Not applied to +a request a mock answered. + +| | | +|---|---| +| Type | object | +| Required | no | +| Rules | no unknown keys | + +```yaml +proxies: + - loss: + percentage: 0.25 + status: 200 +``` + +### `proxies[].loss.percentage` { #proxies-loss-percentage } + +The share of requests to drop, as a fraction. `0.1` is one in ten. + +| | | +|---|---| +| Type | number | +| Required | yes | +| Rules | 0 to 1 | + +```yaml +proxies: + - loss: + percentage: 0.25 +``` + +### `proxies[].loss.status` { #proxies-loss-status } + +The status a dropped request is answered with, rather than being left +to hang. + +| | | +|---|---| +| Type | integer | +| Required | yes | +| Rules | 100 to 599 | + +```yaml +proxies: + - loss: + status: 200 +``` + +### `proxies[].mocks` { #proxies-mocks } + +Mocks in the order they are tried. First match wins, and patterns are +unanchored, so a general one placed first shadows the rest. + +| | | +|---|---| +| Type | list of object | +| Required | no | + +```yaml +proxies: + - mocks: + - name: billing-api + request: + method: GET + url: "/api/(?P\\d+)/" + response: + status: 200 +``` + +### `proxies[].mocks[].name` { #proxies-mocks-name } + +Names this mock in the `mock` log field and the hit counter. Unique +within the proxy. + +| | | +|---|---| +| Type | string | +| Required | yes | +| Rules | matches `^[A-Za-z0-9_-]{2,64}$`; 2 to 64 characters | + +```yaml +proxies: + - mocks: + - name: billing-api +``` + +### `proxies[].mocks[].proxy` { #proxies-mocks-proxy } + +Per-mock overrides of the proxy's `replace`, `loss` and `latency`. + +| | | +|---|---| +| Type | object | +| Required | no | +| Rules | no unknown keys | + +### `proxies[].mocks[].proxy.latency` { #proxies-mocks-proxy-latency } + +Replaces the proxy's `latency` for this mock's responses, rather than +adding to it. + +| | | +|---|---| +| Type | object | +| Required | no | +| Rules | no unknown keys | + +```yaml +proxies: + - mocks: + - proxy: + latency: + percentage: 0.25 + min: 0.25 + max: 0.25 +``` + +### `proxies[].mocks[].proxy.latency.max` { #proxies-mocks-proxy-latency-max } + +Upper bound of the delay, in seconds. Must be at least `min`. + +| | | +|---|---| +| Type | number | +| Required | yes | +| Rules | 0 to 300 | + +```yaml +proxies: + - mocks: + - proxy: + latency: + max: 0.25 +``` + +### `proxies[].mocks[].proxy.latency.min` { #proxies-mocks-proxy-latency-min } + +Lower bound of the delay, in seconds. The delay is a target for the +whole response: time the upstream already spent is subtracted. + +| | | +|---|---| +| Type | number | +| Required | yes | +| Rules | 0 to 300 | + +```yaml +proxies: + - mocks: + - proxy: + latency: + min: 0.25 +``` + +### `proxies[].mocks[].proxy.latency.percentage` { #proxies-mocks-proxy-latency-percentage } + +The share of requests to delay, as a fraction. + +| | | +|---|---| +| Type | number | +| Required | yes | +| Rules | 0 to 1 | + +```yaml +proxies: + - mocks: + - proxy: + latency: + percentage: 0.25 +``` + +### `proxies[].mocks[].proxy.loss` { #proxies-mocks-proxy-loss } + +Drops a share of the requests this mock would have answered. Not +inherited from the proxy: a mock without it is never dropped. + +| | | +|---|---| +| Type | object | +| Required | no | +| Rules | no unknown keys | + +```yaml +proxies: + - mocks: + - proxy: + loss: + percentage: 0.25 + status: 200 +``` + +### `proxies[].mocks[].proxy.loss.percentage` { #proxies-mocks-proxy-loss-percentage } + +The share of requests to drop, as a fraction. `0.1` is one in ten. + +| | | +|---|---| +| Type | number | +| Required | yes | +| Rules | 0 to 1 | + +```yaml +proxies: + - mocks: + - proxy: + loss: + percentage: 0.25 +``` + +### `proxies[].mocks[].proxy.loss.status` { #proxies-mocks-proxy-loss-status } + +The status a dropped request is answered with, rather than being left +to hang. + +| | | +|---|---| +| Type | integer | +| Required | yes | +| Rules | 100 to 599 | + +```yaml +proxies: + - mocks: + - proxy: + loss: + status: 200 +``` + +### `proxies[].mocks[].proxy.replace` { #proxies-mocks-proxy-replace } + +Overrides the proxy's `replace` for requests this mock matches. + +| | | +|---|---| +| Type | number | +| Required | no | +| Rules | 0 to 1 | + +```yaml +proxies: + - mocks: + - proxy: + replace: 0.25 +``` + +### `proxies[].mocks[].request` { #proxies-mocks-request } + +What this mock matches, and what it takes out of the request. + +| | | +|---|---| +| Type | object | +| Required | yes | +| Rules | no unknown keys | + +```yaml +proxies: + - mocks: + - request: + method: GET + url: "/api/(?P\\d+)/" +``` + +### `proxies[].mocks[].request.body` { #proxies-mocks-request-body } + +Variable name -> selector such as `.content.items`. + +| | | +|---|---| +| Type | map of string to string | +| Required | no | + +```yaml +proxies: + - mocks: + - request: + body: + key: .content.items +``` + +### `proxies[].mocks[].request.headers` { #proxies-mocks-request-headers } + +Variable name -> request header name. + +| | | +|---|---| +| Type | map of string to string | +| Required | no | + +```yaml +proxies: + - mocks: + - request: + headers: + key: X-Request-Id +``` + +### `proxies[].mocks[].request.method` { #proxies-mocks-request-method } + +The method this mock answers, matched exactly and upper case. + +| | | +|---|---| +| Type | string | +| Required | yes | +| Rules | one of `GET`, `HEAD`, `POST`, `PUT`, `PATCH`, `DELETE`, `OPTIONS`, `TRACE`, `CONNECT`, `QUERY`, `PROPFIND`, `PROPPATCH`, `MKCOL`, `COPY`, `MOVE`, `LOCK`, `UNLOCK` | + +```yaml +proxies: + - mocks: + - request: + method: GET +``` + +### `proxies[].mocks[].request.query` { #proxies-mocks-request-query } + +Variable name -> selector such as `.filter`. + +| | | +|---|---| +| Type | map of string to string | +| Required | no | + +```yaml +proxies: + - mocks: + - request: + query: + key: .content.items +``` + +### `proxies[].mocks[].request.url` { #proxies-mocks-request-url } + +A regex matched against the request path, unanchored. Named capture +groups become template variables. + +| | | +|---|---| +| Type | string | +| Required | yes | + +```yaml +proxies: + - mocks: + - request: + url: "/api/(?P\\d+)/" +``` + +### `proxies[].mocks[].response` { #proxies-mocks-response } + +What it answers with. + +| | | +|---|---| +| Type | object | +| Required | yes | +| Rules | no unknown keys | + +```yaml +proxies: + - mocks: + - response: + status: 200 +``` + +### `proxies[].mocks[].response.body` { #proxies-mocks-response-body } + +A template rendered and sent as `text/plain`. Exclusive with `json` and +`template`. + +| | | +|---|---| +| Type | string | +| Required | no | + +```yaml +proxies: + - mocks: + - response: + body: widget {{ id }} is ready +``` + +### `proxies[].mocks[].response.headers` { #proxies-mocks-response-headers } + +Header name -> template producing the value. + +The value is a template, not a header value: what it renders to is +only a header value once a request has been served, which is checked +there. The name is a name now. + +| | | +|---|---| +| Type | map of string to string | +| Required | no | + +### `proxies[].mocks[].response.json` { #proxies-mocks-response-json } + +A template whose rendered output must be valid JSON, sent as +`application/json`. Exclusive with `body` and `template`. + +| | | +|---|---| +| Type | string | +| Required | no | + +```yaml +proxies: + - mocks: + - response: + json: "{\"id\": \"{{ id }}\"}" +``` + +### `proxies[].mocks[].response.status` { #proxies-mocks-response-status } + +The status to answer with. + +| | | +|---|---| +| Type | integer | +| Required | yes | +| Rules | 100 to 599 | + +```yaml +proxies: + - mocks: + - response: + status: 200 +``` + +### `proxies[].mocks[].response.template` { #proxies-mocks-response-template } + +A template file under this proxy's template directory. Read per request, +so it may be uploaded after the configuration was loaded. Exclusive with +`body` and `json`. + +| | | +|---|---| +| Type | string | +| Required | no | +| Rules | at most 200 characters | + +```yaml +proxies: + - mocks: + - response: + template: put.json.j2 +``` + +### `proxies[].name` { #proxies-name } + +Names this proxy in `X-Proxy-Name`, in metrics labels, in log lines +and as its template subdirectory. Unique within the document. + +| | | +|---|---| +| Type | string | +| Required | yes | +| Rules | matches `^[A-Za-z0-9_-]{2,32}$`; 2 to 32 characters | + +```yaml +proxies: + - name: billing-api +``` + +### `proxies[].replace` { #proxies-replace } + +What share of requests a matching mock actually answers; the rest go +upstream. Defaults to 1.0, so a matching mock answers. + +| | | +|---|---| +| Type | number | +| Required | no | +| Rules | 0 to 1 | + +```yaml +proxies: + - replace: 0.25 +``` + +### `proxies[].resolve` { #proxies-resolve } + +How a request is matched to this proxy: by header, or as the default. + +| | | +|---|---| +| Type | object | +| Required | no | +| Rules | no unknown keys | + +### `proxies[].resolve.header` { #proxies-resolve-header } + +The header carrying the proxy name. Required when `type: header`, and +meaningless otherwise. + +| | | +|---|---| +| Type | string | +| Required | no | + +```yaml +proxies: + - resolve: + header: X-Request-Id +``` + +### `proxies[].resolve.type` { #proxies-resolve-type } + +`default` takes anything unclaimed; `header` takes requests naming +this proxy in `header`. + +| | | +|---|---| +| Type | string | +| Required | no | +| Rules | one of `default`, `header` | + +```yaml +proxies: + - resolve: + type: default +``` + +### `proxies[].rewrite_redirects` { #proxies-rewrite_redirects } + +Whether a redirect whose `Location` points back into the space this +proxy forwards is rewritten to point at Doppel instead. Absent means +enabled. + +On by default because the alternative is a silent failure: `Host` is +replaced with the upstream's authority, so the upstream's `Location` +names the upstream, and a client following it leaves Doppel -- along with +every fault and every mock -- with nothing reported. Turn it off to have +the response relayed byte for byte, which is what a client being tested +*against redirect handling itself* needs. + +| | | +|---|---| +| Type | boolean | +| Required | no | + +```yaml +proxies: + - rewrite_redirects: true +``` + +### `proxies[].rewrite_urls` { #proxies-rewrite_urls } + +Whether the upstream's own address is rewritten to Doppel's inside the +bodies this proxy relays. Absent means enabled. + +The same problem `rewrite_redirects` solves, one layer down: a page, a +script or a JSON document that names `https://api.example.com/v2/` sends +the client straight there on the next request, past every fault and every +mock. Rewriting the body keeps it inside Doppel. + +Only the exact host is replaced. `https://cdn.api.example.com/` is a +different host and is left alone, because Doppel does not proxy it and +pointing it at itself would break a page rather than keep it working. + +Only text bodies, and only ones the upstream did not compress. Turn it off +for a client being tested against the bytes the upstream actually sent. + +| | | +|---|---| +| Type | boolean | +| Required | no | + +```yaml +proxies: + - rewrite_urls: true +``` + +### `proxies[].timeout` { #proxies-timeout } + +Bounds the whole upstream exchange, in seconds. Exceeding it is +`504 UPSTREAM_TIMEOUT`. Defaults to 30. + +| | | +|---|---| +| Type | integer | +| Required | no | +| Rules | 1 to 3600 | + +```yaml +proxies: + - timeout: 30 +``` + +### `proxies[].type` { #proxies-type } + +What it forwards. `http` is the only value. + +| | | +|---|---| +| Type | string | +| Required | yes | +| Rules | one of `http` | + +```yaml +proxies: + - type: http +``` + +### `proxies[].url` { #proxies-url } + +The upstream base. A request path is grafted underneath it, and the +result can never escape it -- so a base with a path confines the proxy +to that subtree. + +| | | +|---|---| +| Type | string | +| Required | yes | + +```yaml +proxies: + - url: https://example.com/api/ +``` + +--- + +## `sentry` { #sentry } + +Optional error reporting. Absent, or an empty DSN, disables it. + +| | | +|---|---| +| Type | object | +| Required | no | +| Rules | no unknown keys | + +```yaml +sentry: + dsn: https://key@o0.ingest.sentry.io/0 +``` + +### `sentry.dsn` { #sentry-dsn } + +The Sentry DSN to report to. Empty disables reporting, so a deployment +can blank it without removing the section. + +| | | +|---|---| +| Type | string | +| Required | yes | + +```yaml +sentry: + dsn: https://key@o0.ingest.sentry.io/0 +``` + +--- + +## `server` { #server } + +Where the proxy listens for the traffic being forwarded or mocked. + +| | | +|---|---| +| Type | object | +| Required | yes | +| Rules | no unknown keys | + +```yaml +server: + host: 127.0.0.1 + port: 8080 +``` + +### `server.external_url` { #server-external_url } + +Where clients reach this Doppel, when that is not `host:port`. + +Behind a container port mapping, a load balancer or an ingress, the +address Doppel bound is not the address a client used -- and a rewritten +`Location` has to name the second. Doppel cannot infer it: `Host` is a +claim by the caller, and building a redirect out of it hands the caller +the redirect. + +Absent, `host` and `port` are used, with a wildcard bind (`0.0.0.0`, `::`) +read as loopback -- which is right for a laptop or a pod reached at its +own address, and wrong behind a port mapping or an ingress, where the +client used neither. Doppel logs the address it settled on at startup. + +`DOPPEL_EXTERNAL_URL` overrides it. Part of `server`, so a change takes a +restart like the rest of that section. + +| | | +|---|---| +| Type | string | +| Required | no | + +```yaml +server: + external_url: https://doppel.example.com/ +``` + +### `server.host` { #server-host } + +An IP address, not a hostname: a name would have to be resolved, +and which address it resolves to is not the configuration's to +decide. `utoipa` has no schema for `IpAddr`, so it is described +here as the string it is written as. + +| | | +|---|---| +| Type | string | +| Required | yes | + +```yaml +server: + host: 127.0.0.1 +``` + +### `server.port` { #server-port } + +The TCP port proxied traffic arrives on. Must differ from +`admin.port`. + +| | | +|---|---| +| Type | integer | +| Required | yes | +| Rules | 1 to 65535 | + +```yaml +server: + port: 8080 +``` + +--- + +## `templates` { #templates } + +Where mock template files are read from and uploaded to. + +| | | +|---|---| +| Type | object | +| Required | no | +| Rules | no unknown keys | + +### `templates.dir` { #templates-dir } + +Directory holding mock templates, one subdirectory per proxy. Created +at startup if absent. +A filesystem path. `utoipa` has no schema for `PathBuf`, so it is +described as the string it is written as. + +| | | +|---|---| +| Type | string | +| Required | no | + +```yaml +templates: + dir: ./templates +``` diff --git a/docs/usage/proxying.md b/docs/usage/proxying.md index a3958df..bcaa679 100644 --- a/docs/usage/proxying.md +++ b/docs/usage/proxying.md @@ -131,11 +131,12 @@ proxies: ``` With a base of `https://api.example.com/v2/`, an upstream answering -`Location: https://api.example.com/v2/orders/7` produces `Location: /orders/7` -to the client. Query and fragment survive. +`Location: https://api.example.com/v2/orders/7` produces +`Location: http://127.0.0.1:8080/orders/7` to the client. Query and fragment +survive. -Relative rather than absolute, so Doppel never has to guess its own public -name: the client resolves it against the URL it used, which is Doppel's. +The host in that answer is Doppel's own, and where it comes from is +[Doppel's own address](#doppels-own-address) below. !!! warning "Why this is on by default" `Host` is not relayed, so the upstream answers with its *own* authority in @@ -147,11 +148,94 @@ name: the client resolves it against the URL it used, which is Doppel's. `nginx` has `proxy_redirect` for this and Apache `ProxyPassReverse`, both on by default, for the same reason. -A target that is **not** under the proxy's base is genuinely elsewhere and is -left pointing there, stated as an absolute URL. That covers a subtler case too: -an upstream answering `Location: /login` under a base of `/v2/` means its own -root, so relaying the header as-is would have the client come back asking for -`/v2/login` -- a different resource nobody named. +A target on the upstream's own host but **outside** the proxy's base is kept on +Doppel too, with the path the upstream wrote: an upstream answering +`Location: /login` under a base of `/v2/` produces +`Location: http://127.0.0.1:8080/login`. This is what `nginx`'s `proxy_redirect` +does, and it is the case the relative form could not express -- relayed as-is, +the client would come back asking for `/v2/login`, a different resource nobody +named, and pointed at the upstream it leaves Doppel altogether. + +Whether Doppel serves that path is a question about the configuration rather than +about the rewrite: `/login` reaches `/login`, so a proxy whose base has a +prefix will forward it under that prefix. That is visible in the logs and fixable +in the configuration, which the silent escape was not. + +A target on **another host** is left pointing there, absolutely. Doppel does not +proxy it, and naming itself in a redirect to somewhere it cannot serve would be a +lie. + +## Urls in a body + +The same problem one layer down, and on by default for the same reason: + +```yaml +proxies: + - name: backend + url: "https://api.example.com/v2/" + rewrite_urls: true # the default +``` + +A page, a script or a JSON document that names `https://api.example.com/v2/orders` +sends the client straight to the upstream on its next request -- past every +injected fault and every mock, with nothing logged. Doppel replaces its own address +into the body instead: that URL becomes `http://127.0.0.1:8080/orders`, and one on +the same host outside the proxied path keeps its own path, exactly as a rewritten +redirect does. `nginx` calls this `sub_filter`. + +**Only the exact host.** `https://cdn.api.example.com/` is a different host, is not +proxied by this proxy, and is left alone -- pointing it here would break the page +rather than keep it working. A host that merely starts the same, like +`https://api.example.com.evil.test/`, is left alone too. + +Three more limits, each of which relays the body untouched rather than guessing: + +- **Text only**, by `Content-Type`: `text/*`, JSON, JavaScript, XML and the `+json` + and `+xml` suffixes. An image cannot carry a URL that matters, and buffering one + to look is how a proxy runs out of memory. +- **Uncompressed only.** A body with a `Content-Encoding` is relayed as it came; + the client asked the upstream for that encoding. +- **Bounded by `body_limit`.** Rewriting needs the whole body, so it is buffered up + to that ceiling; a bigger body streams on from where the buffering stopped. + +A rewritten body loses the `ETag` and digest headers that described the upstream's +version of it, and carries the length it now has. A conditional request with the +upstream's validator would otherwise be answered `304` for content the client has +never seen. + +Set `rewrite_urls: false` for a client being tested against the bytes the upstream +actually sent. + +## Doppel's own address + +Rewriting a `Location` means naming the address the client used, and Doppel +cannot work that out for itself -- `Host` is a claim by the caller, and building +a redirect out of it hands the caller the redirect. + +In order: + +```yaml +server: + host: 0.0.0.0 + port: 8080 + external_url: "https://doppel.example.com/" # optional +``` + +1. `DOPPEL_EXTERNAL_URL`, which overrides everything below it. +2. `server.external_url`. +3. `server.host` and `server.port`, which is right for the common case and needs + no configuration: a laptop on `127.0.0.1:8080`, a pod on its own address. + A wildcard bind (`0.0.0.0`, `::`) becomes loopback, because `0.0.0.0` names + every address this host has and therefore none of them. + +The third is a guess, and the one place this is wrong: behind a container port +mapping (`-p 18080:8080`), a load balancer or an ingress, the client used neither +that address nor that port. Set `external_url`, or the variable -- and Doppel logs +which address it settled on at startup, so it is one line away from being checked +rather than assumed. + +A path is kept as a prefix: `https://gw.example.com/doppel/` is a Doppel reached +under a prefix, and its rewritten locations carry it. Set `rewrite_redirects: false` to relay the header byte for byte. That is what a client being tested *for its redirect handling* needs; it is not what a client diff --git a/docs/usage/runtime-changes.md b/docs/usage/runtime-changes.md index e6bcdc6..16d6355 100644 --- a/docs/usage/runtime-changes.md +++ b/docs/usage/runtime-changes.md @@ -137,14 +137,14 @@ doppel config reload A rolling change is a write followed by a reload on each instance. Nothing coordinates them, and nothing stops two instances running different revisions -in between; `GET /status` on each reports which one it is holding. +in between; `GET /api/v1/status` on each reports which one it is holding. See [Storing configuration in PostgreSQL](storage.md). ## Checking what is running ```bash -curl -s localhost:8081/status +curl -s localhost:8081/api/v1/status ``` ```json @@ -162,5 +162,5 @@ curl -s localhost:8081/status which is the quickest way to confirm a resolution header is spelled the way you think it is. Credentials in an upstream URL are redacted here. -`/status` needs no token by default, so it is usable as a health check and as a +`/api/v1/status` needs no token by default, so it is usable as a health check and as a way to confirm a rolling reload actually reached every instance. diff --git a/docs/usage/storage.md b/docs/usage/storage.md index 551bf30..f339738 100644 --- a/docs/usage/storage.md +++ b/docs/usage/storage.md @@ -99,25 +99,42 @@ release, migrations are append-only. ### The schema -Five tables. Deliberately between one opaque blob and full normalisation: -**anything with its own identity and lifecycle is a row, and a leaf map -hanging off one is a JSONB column on that row.** +Three tables. **Anything SQL actually uses is a column; everything else is the +document, as JSON.** | Table | Holds | |---|---| -| `configurations` | One row per named configuration: the revision, and the settings that are single values | -| `admin_tokens` | One row per token | -| `proxies` | One row per proxy; `headers` and `access` are JSONB on it | -| `mocks` | One row per mock; the selector maps and response headers are JSONB on it | +| `configurations` | One row per named configuration: `name`, `revision`, and `settings` -- the whole document except its proxies | +| `proxies` | One row per proxy: `name`, `ordinal`, and `document` -- the proxy with its mocks inside | | `templates` | One row per template file, content included | -A blob would make the row opaque, and rewriting a whole document to change one -field throws away the per-proxy concurrency the admin API is built on. A table -per leaf map would buy joins nobody will write. +It was five tables with a column per configuration field, and the change is +worth explaining because it looks like a step backwards. -`ordinal` columns preserve document order. That is not decoration: mock -patterns are unanchored, so a general one placed before a specific one shadows -it, and a store that reordered them would change which mock answers a request. +Nothing ever queried `admin_host` or `latency_min` in SQL. What that +normalisation bought was a migration for every field the configuration format +gained -- three of them between 0.3.0 and 0.4.1, each one `ADD COLUMN` and +nothing else, plus a matching edit in the loader, the writer, and two +hand-written statements that a test existed solely to keep in step. A field +added to the format now reaches the database with no schema change at all. + +What did *not* change is the part that matters: a proxy is still its own row, +addressed by name and carrying its own revision, so the per-proxy concurrency +the admin API is built on is untouched. A single blob per configuration would +have thrown that away. + +The rows are parsed by the same code that parses `main.yaml`, so a row edited by +hand is held to exactly the rules a configuration file is held to -- a value the +format refuses is refused here too, reported against the field and the proxy that +carry it. + +`ordinal` preserves document order. That is not decoration: mock patterns are +unanchored, so a general one placed before a specific one shadows it, and a store +that reordered them would change which mock answers a request. + +One thing moved rather than disappeared. The old schema had a unique index on +`(config, token)`, making duplicate token values impossible in the database; rule +V26 refuses them on every write instead, and every write goes through validation. `templates` has no cascading foreign key on purpose. Deleting a proxy's files is a decision the store makes explicitly and *after* the configuration write -- @@ -148,7 +165,7 @@ second writer that blocks on it re-evaluates its own `WHERE` against the row as the winner left it -- finds the revision moved, matches nothing, and is told it lost. -A load is also one transaction, at `REPEATABLE READ`. A configuration is five +A load is also one transaction, at `REPEATABLE READ`. A configuration spans two tables, and each query used to take its own connection and its own snapshot, so a save committing in between produced a configuration assembled from two of them. The revision check at the end of a load caught it and reported "the rows @@ -176,7 +193,7 @@ sequence. So: a rolling change is a write followed by a reload on each instance. There is no coordination between them, and nothing stops two instances running -different revisions in between; `GET /status` on each reports which one it is +different revisions in between; `GET /api/v1/status` on each reports which one it is holding. ## Moving between the stores diff --git a/docs/usage/troubleshooting.md b/docs/usage/troubleshooting.md index a8fbd77..90b4070 100644 --- a/docs/usage/troubleshooting.md +++ b/docs/usage/troubleshooting.md @@ -158,3 +158,33 @@ remove it from `DOPPEL_ADMIN_TOKENS`. Anything not covered here is worth an issue on the [tracker](https://github.com/lorem-dev/doppel/issues). + +## The dashboard + +### The admin root answers 503 + +`DASHBOARD_NOT_BUILT`. This binary was compiled without the dashboard's static +assets -- a source build that skipped the frontend. Released binaries and the +published image always carry it. + +```bash +npm --prefix frontend ci && npm --prefix frontend run build +cargo build --release -p doppel-cli +``` + +The order is the whole answer: the assets are embedded at compile time, so +building them without rebuilding the binary changes nothing about what it serves. +If Node is not available and the JSON API is enough, `admin.dashboard: false` +stops the route being served at all, which is a clearer answer than a 503. + +### The page loads but every button is disabled + +The caller has no rights to spend. The dashboard asks `GET /api/v1/access` what it +may do and disables what the server would refuse; the tooltip on each control says +so. Enter a token, or check `admin.access` -- the same two causes as a 401 above, +since it is the same decision. + +### A change made elsewhere is not on screen + +It will be within a minute: the list refetches on a timer, and the timer is paused +while the tab is hidden. Switching back to the tab refetches immediately. diff --git a/doppel-config.schema.json b/doppel-config.schema.json index be21fd1..f8705e4 100644 --- a/doppel-config.schema.json +++ b/doppel-config.schema.json @@ -41,6 +41,13 @@ "$ref": "#/$defs/AuthConfig", "description": "Which header carries the bearer token." }, + "dashboard": { + "description": "Serve the browser dashboard from the admin listener's root.\n\nOn by default. Off means `/`, `/static/*` and `/robots.txt` are not\nrouted at all and answer 404 like any other unknown path; the JSON API is\nuntouched either way.\n\nAn `Option` and skipped when absent, for the reason given on `groups`:\nadding it must not change the canonical YAML of configurations written\nbefore it existed. Read it through\n[`AdminConfig::is_dashboard_enabled`].", + "type": [ + "boolean", + "null" + ] + }, "enable": { "description": "Whether to run the admin listener at all.\n\nDefaults to on. Off means the port is never bound and no admin task\nstarts; the proxy and the control socket are untouched, so\n`doppel config reload` still works and is then the only way in.\n\nThe validation rules do not consult this. A configuration that is only\nsafe because nothing serves it is a trap set for whoever turns the\nlistener on later, and they will not re-read the rules first.", "type": "boolean" @@ -73,6 +80,17 @@ "null" ] }, + "title": { + "description": "The heading the dashboard shows, and the browser tab's title.\n\nDefaults to `Doppel`. Useful when several Doppels are open at once and\nthe tabs are otherwise identical.\n\nAn `Option` and skipped when absent, for the same reason as `dashboard`.\nRead it through [`AdminConfig::title`].", + "oneOf": [ + { + "type": "null" + }, + { + "$ref": "#/$defs/AdminTitle" + } + ] + }, "tokens": { "description": "The tokens that may call the admin API. Names and token values are\neach unique. `DOPPEL_ADMIN_TOKENS` can supply these instead.", "items": { @@ -92,6 +110,15 @@ ], "type": "object" }, + "AdminTitle": { + "description": "The heading the dashboard shows, at most 64 characters and free of control characters. Defaults to `Doppel`.", + "examples": [ + "billing-api (staging)" + ], + "maxLength": 64, + "minLength": 1, + "type": "string" + }, "AllowedGroup": { "description": "A token or group name `access` may reference, or `*` for any of them.", "examples": [ @@ -135,17 +162,33 @@ "properties": { "socket": { "description": "Path to the control socket, created with mode 0600 and removed on\nshutdown. Its parent directory must already exist.\nA filesystem path. `utoipa` has no schema for `PathBuf`, so it is\ndescribed as the string it is written as.", + "examples": [ + "/tmp/doppel.sock" + ], "type": "string" } }, "type": "object" }, + "ExternalUrl": { + "description": "An absolute http or https url naming where clients reach this Doppel, with no query string or fragment.", + "examples": [ + "https://doppel.example.com/" + ], + "type": "string" + }, "HeaderName": { "description": "An HTTP header name: a non-empty RFC 9110 token.", + "examples": [ + "X-Request-Id" + ], "type": "string" }, "HeaderValue": { "description": "An HTTP header value: visible ASCII, space and tab, with no line breaks.", + "examples": [ + "staging" + ], "type": "string" }, "HttpMethod": { @@ -173,6 +216,9 @@ }, "HttpStatus": { "description": "An HTTP status code, 100 to 599.", + "examples": [ + 200 + ], "maximum": 599, "minimum": 100, "type": "integer" @@ -375,6 +421,9 @@ "properties": { "body": { "description": "A template rendered and sent as `text/plain`. Exclusive with `json` and\n`template`.", + "examples": [ + "widget {{ id }} is ready" + ], "type": [ "string", "null" @@ -387,12 +436,18 @@ "description": "Header name -> template producing the value.\n\nThe value is a template, not a header value: what it renders to is\nonly a header value once a request has been served, which is checked\nthere. The name is a name now.", "propertyNames": { "description": "An HTTP header name: a non-empty RFC 9110 token.", + "examples": [ + "X-Request-Id" + ], "type": "string" }, "type": "object" }, "json": { "description": "A template whose rendered output must be valid JSON, sent as\n`application/json`. Exclusive with `body` and `template`.", + "examples": [ + "{\"id\": \"{{ id }}\"}" + ], "type": [ "string", "null" @@ -421,6 +476,9 @@ }, "Name": { "description": "Letters, digits, `-` and `_`, between 2 and 64 characters.", + "examples": [ + "billing-api" + ], "maxLength": 64, "minLength": 2, "pattern": "^[A-Za-z0-9_-]{2,64}$", @@ -435,6 +493,9 @@ }, "Port": { "description": "A TCP port, 1 to 65535.", + "examples": [ + 8080 + ], "maximum": 65535, "minimum": 1, "type": "integer" @@ -515,6 +576,9 @@ "description": "Headers injected into every outbound request, overriding whatever the\nclient sent by the same name.", "propertyNames": { "description": "An HTTP header name: a non-empty RFC 9110 token.", + "examples": [ + "X-Request-Id" + ], "type": "string" }, "type": "object" @@ -574,6 +638,13 @@ "null" ] }, + "rewrite_urls": { + "description": "Whether the upstream's own address is rewritten to Doppel's inside the\nbodies this proxy relays. Absent means enabled.\n\nThe same problem `rewrite_redirects` solves, one layer down: a page, a\nscript or a JSON document that names `https://api.example.com/v2/` sends\nthe client straight there on the next request, past every fault and every\nmock. Rewriting the body keeps it inside Doppel.\n\nOnly the exact host is replaced. `https://cdn.api.example.com/` is a\ndifferent host and is left alone, because Doppel does not proxy it and\npointing it at itself would break a page rather than keep it working.\n\nOnly text bodies, and only ones the upstream did not compress. Turn it off\nfor a client being tested against the bytes the upstream actually sent.", + "type": [ + "boolean", + "null" + ] + }, "timeout": { "description": "Bounds the whole upstream exchange, in seconds. Exceeding it is\n`504 UPSTREAM_TIMEOUT`. Defaults to 30.", "oneOf": [ @@ -610,6 +681,9 @@ }, "ProxyName": { "description": "Letters, digits, `-` and `_`, between 2 and 32 characters.", + "examples": [ + "billing-api" + ], "maxLength": 32, "minLength": 2, "pattern": "^[A-Za-z0-9_-]{2,32}$", @@ -617,6 +691,9 @@ }, "Ratio": { "description": "A probability from 0.0 to 1.0. 50% is `0.5`.", + "examples": [ + 0.25 + ], "maximum": 1, "minimum": 0, "type": "number" @@ -651,6 +728,9 @@ }, "Seconds": { "description": "A latency in seconds, 0 to 300.", + "examples": [ + 0.25 + ], "maximum": 300, "minimum": 0, "type": "number" @@ -668,6 +748,9 @@ "properties": { "dsn": { "description": "The Sentry DSN to report to. Empty disables reporting, so a deployment\ncan blank it without removing the section.", + "examples": [ + "https://key@o0.ingest.sentry.io/0" + ], "type": "string" } }, @@ -680,6 +763,17 @@ "additionalProperties": false, "description": "No `workers` here. It sizes the tokio runtime, and a database-backed\nstore cannot be opened before that runtime exists -- so the value has to\nbe known before the configuration is read, which puts it on the same side\nof the boundary as the connection settings. It is `--workers` /\n`DOPPEL_WORKERS`.", "properties": { + "external_url": { + "description": "Where clients reach this Doppel, when that is not `host:port`.\n\nBehind a container port mapping, a load balancer or an ingress, the\naddress Doppel bound is not the address a client used -- and a rewritten\n`Location` has to name the second. Doppel cannot infer it: `Host` is a\nclaim by the caller, and building a redirect out of it hands the caller\nthe redirect.\n\nAbsent, `host` and `port` are used, with a wildcard bind (`0.0.0.0`, `::`)\nread as loopback -- which is right for a laptop or a pod reached at its\nown address, and wrong behind a port mapping or an ingress, where the\nclient used neither. Doppel logs the address it settled on at startup.\n\n`DOPPEL_EXTERNAL_URL` overrides it. Part of `server`, so a change takes a\nrestart like the rest of that section.", + "oneOf": [ + { + "type": "null" + }, + { + "$ref": "#/$defs/ExternalUrl" + } + ] + }, "host": { "description": "An IP address, not a hostname: a name would have to be resolved,\nand which address it resolves to is not the configuration's to\ndecide. `utoipa` has no schema for `IpAddr`, so it is described\nhere as the string it is written as.", "examples": [ @@ -734,6 +828,9 @@ "properties": { "dir": { "description": "Directory holding mock templates, one subdirectory per proxy. Created\nat startup if absent.\nA filesystem path. `utoipa` has no schema for `PathBuf`, so it is\ndescribed as the string it is written as.", + "examples": [ + "./templates" + ], "type": "string" } }, @@ -741,12 +838,18 @@ }, "TimeoutSeconds": { "description": "An upstream timeout in whole seconds, 1 to 3600.", + "examples": [ + 30 + ], "maximum": 3600, "minimum": 1, "type": "integer" }, "Token": { "description": "An admin token: printable ASCII with no spaces, 32 to 255 characters. A version 4 UUID is the recommended shape.", + "examples": [ + "replace-me-00000000000000000000000" + ], "maxLength": 255, "minLength": 32, "type": "string" diff --git a/frontend/e2e/src/colour.ts b/frontend/e2e/src/colour.ts new file mode 100644 index 0000000..d85fa44 --- /dev/null +++ b/frontend/e2e/src/colour.ts @@ -0,0 +1,86 @@ +// Measuring what a screen actually shows. +// +// Two specs need this: one asks whether the page's own colours are readable, the +// other whether the syntax highlighter is colouring anything at all. Both are +// questions about painted pixels rather than about class names, which is the point +// -- the failures they guard were a dark theme rendering white on white and a +// highlighter tokenising in one colour, and a class assertion would have passed +// through both. + +import type { Page } from '@playwright/test' + +/** sRGB bytes. */ +export type Rgb = [number, number, number] + +/** + * What a screen gets for an element's colour, as sRGB bytes. + * + * Measured by painting the computed value into a canvas and reading the pixel + * back, rather than by parsing the string. Tailwind 4 emits `oklch(...)`, Chrome + * returns it from `getComputedStyle` verbatim, and assigning it to `fillStyle` + * keeps it in that space -- only a paint converts it. The converted pixel is also + * the honest subject of a contrast assertion, since it is what a reader sees. + */ +export async function paintedRgb( + page: Page, + selector: string, + property: 'color' | 'backgroundColor', +): Promise { + return page.locator(selector).first().evaluate(paint, property) +} + +/** + * The same measurement for every element a selector matches. + * + * Waits for one match before measuring. `all()` returns whatever matches at that + * instant and waits for nothing, while the spans this is pointed at -- the + * highlighter's -- appear on a React update rather than with the keystroke that + * caused them. Measured too early, a caller's "more than zero colours" assertion + * fails for a reason that has nothing to do with colour: the same one-shot read + * that made `fieldMessage` fail on CI and never on a laptop. + * + * So this is for asking about elements that are expected to be there. A test that + * wants to prove something is *not* highlighted should assert on the locator's + * count instead of measuring the colours of nothing. + */ +export async function paintedRgbAll( + page: Page, + selector: string, + property: 'color' | 'backgroundColor', +): Promise { + await page.locator(selector).first().waitFor({ state: 'attached' }) + return Promise.all( + (await page.locator(selector).all()).map((node) => node.evaluate(paint, property)), + ) +} + +/** Runs in the browser: the computed value, through a canvas, out as bytes. */ +function paint(node: Element, which: string): Rgb { + const computed = getComputedStyle(node)[which as 'color'] + const canvas = document.createElement('canvas') + canvas.width = 1 + canvas.height = 1 + const context = canvas.getContext('2d') + if (!context) { + throw new Error('no 2d context to paint a colour into') + } + context.fillStyle = computed + context.fillRect(0, 0, 1, 1) + const [r, g, b] = context.getImageData(0, 0, 1, 1).data + return [r, g, b] as Rgb +} + +/** The relative luminance of sRGB bytes, per WCAG. */ +export function luminance([r, g, b]: Rgb): number { + const channel = (value: number) => { + const v = value / 255 + return v <= 0.03928 ? v / 12.92 : ((v + 0.055) / 1.055) ** 2.4 + } + return 0.2126 * channel(r) + 0.7152 * channel(g) + 0.0722 * channel(b) +} + +/** WCAG contrast between two painted colours. */ +export function contrast(one: Rgb, two: Rgb): number { + const [light, dark] = [luminance(one), luminance(two)].sort((a, b) => b - a) as [number, number] + return (light + 0.05) / (dark + 0.05) +} diff --git a/frontend/e2e/src/configs.ts b/frontend/e2e/src/configs.ts new file mode 100644 index 0000000..2f8d074 --- /dev/null +++ b/frontend/e2e/src/configs.ts @@ -0,0 +1,143 @@ +// The configurations the specs run against. +// +// Ports and the template directory are placeholders the harness fills in, so two +// spec files running at once cannot collide. + +export const ROOT_TOKEN = 'root-token-0000000000000000000000000' +export const READER_TOKEN = 'reader-token-00000000000000000000000' + +/** Tokens for both a full-rights caller and a read-only one. */ +const TOKENS = ` + tokens: + - name: root + group: admin + token: ${ROOT_TOKEN} + - name: reader + group: user + token: ${READER_TOKEN} +` + +const PROXIES = ` +proxies: + - name: alpha + type: http + url: "https://alpha.example.com/api/" + - name: beta + type: http + url: "https://beta.example.com/api/" + # Resolved by header, because only one proxy may be the default one -- rule + # V12, and a fixture that broke it would fail every spec at startup. + resolve: + type: header + header: X-Proxy-Name +` + +/** Reads are public, writes need `admin`: the shape most deployments have. */ +export const PRIVATE_CONFIG = ` +server: + host: "127.0.0.1" + port: {proxyPort} +admin: + host: "127.0.0.1" + port: {adminPort} + title: "Doppel (e2e)" +${TOKENS} + access: + list: public + read: public + create: ["admin"] + update: ["admin"] + delete: ["admin"] + upload: ["admin"] + upload: + limit: 1Mi +control: + socket: {controlSocket} +templates: + dir: {templatesDir} +${PROXIES} +` + +/** Everything unauthenticated: the page must not ask for a token at all. */ +export const PUBLIC_CONFIG = ` +server: + host: "127.0.0.1" + port: {proxyPort} +admin: + host: "127.0.0.1" + port: {adminPort} + public: true + tokens: [] + access: {} + upload: + limit: 1Mi +control: + socket: {controlSocket} +templates: + dir: {templatesDir} +${PROXIES} +` + +/** + * Public, with a proxy whose `access` block the public API makes inert. + * + * The configuration an operator reaches by turning `public: true` on over proxies + * they had already restricted: the overrides stay in the document and stop + * deciding anything, which the form has to say rather than offer fields for. + * `alpha` holds one and `beta` holds none, so one instance covers both answers. + */ +export const PUBLIC_OVERRIDES_CONFIG = ` +server: + host: "127.0.0.1" + port: {proxyPort} +admin: + host: "127.0.0.1" + port: {adminPort} + public: true + tokens: [] + access: {} + upload: + limit: 1Mi +control: + socket: {controlSocket} +templates: + dir: {templatesDir} +proxies: + - name: alpha + type: http + url: "https://alpha.example.com/api/" + access: + delete: public + - name: beta + type: http + url: "https://beta.example.com/api/" + resolve: + type: header + header: X-Proxy-Name +` + +/** The dashboard turned off: the API stays, the three routes go. */ +export const DASHBOARD_OFF_CONFIG = ` +server: + host: "127.0.0.1" + port: {proxyPort} +admin: + host: "127.0.0.1" + port: {adminPort} + dashboard: false +${TOKENS} + access: + list: public + read: public + create: ["admin"] + update: ["admin"] + delete: ["admin"] + upload: ["admin"] + upload: + limit: 1Mi +control: + socket: {controlSocket} +templates: + dir: {templatesDir} +${PROXIES} +` diff --git a/frontend/e2e/src/fields.ts b/frontend/e2e/src/fields.ts new file mode 100644 index 0000000..495af78 --- /dev/null +++ b/frontend/e2e/src/fields.ts @@ -0,0 +1,30 @@ +// Asking a field about itself, without walking the DOM. +// +// A form's message lives near its control, and "near" used to mean a specific number +// of parent elements -- which broke the moment the label moved to make room for the +// (i). The control names its message with `aria-describedby`, which is both how a +// screen reader finds it and a stable handle for a test. + +import { expect, type Locator, type Page } from '@playwright/test' + +/** + * The hint-or-complaint line under the field with this label. + * + * `[id="..."]` rather than `#...`: React's generated ids contain characters a CSS + * id selector cannot carry. + */ +export async function fieldMessage(page: Page, label: string): Promise { + const control = page.getByLabel(label).first() + // Waited for rather than read once. A message that arrives with the server's + // refusal is not on the page the instant the button is clicked, so a one-shot + // read is a race a slower machine loses: CI failed here on a run where the whole + // suite took twice as long as it does on a laptop, holding `null` while the + // complaint was still in flight. The assertion that follows this one always + // retried, which is why this was the one that fell over. + await expect(control, `the field labelled "${label}" describes no message`).toHaveAttribute( + 'aria-describedby', + /\S/, + ) + const described = await control.getAttribute('aria-describedby') + return page.locator(`[id="${described}"]`) +} diff --git a/frontend/e2e/src/server.ts b/frontend/e2e/src/server.ts new file mode 100644 index 0000000..3317e9c --- /dev/null +++ b/frontend/e2e/src/server.ts @@ -0,0 +1,109 @@ +// A real Doppel, per spec file. +// +// Not Playwright's `webServer`: that starts one process for the whole run, and +// these specs need different configurations -- public, private, dashboard off -- +// which is a property of the server rather than of the page. Each file starts its +// own on ports nobody else is using. + +import { spawn, type ChildProcess } from 'node:child_process' +import { createServer } from 'node:net' +import { mkdtempSync, writeFileSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join, resolve } from 'node:path' + +export interface Doppel { + /** Where the admin listener, and therefore the dashboard, answers. */ + baseURL: string + /** + * Where proxied traffic goes. + * + * Only the metrics spec needs it, and it needs it for a real reason: a process + * that has served no request has no series to expose, so an exposition test + * with no traffic behind it asserts nothing. + */ + proxyURL: string + stop: () => void +} + +/** A port the operating system has just confirmed is free. */ +async function freePort(): Promise { + return new Promise((accept, reject) => { + const probe = createServer() + probe.once('error', reject) + probe.listen(0, '127.0.0.1', () => { + const address = probe.address() + if (typeof address === 'string' || address === null) { + reject(new Error('no port')) + return + } + const { port } = address + probe.close(() => accept(port)) + }) + }) +} + +/** + * Start Doppel with `yaml`, with `{proxyPort}`, `{adminPort}` and + * `{templatesDir}` substituted. + * + * The binary comes from `DOPPEL_BIN` or `target/debug/doppel`, and it has to have + * been built with `frontend/dist` in place -- otherwise `/` answers 503 and every + * spec here fails for the same uninformative reason, which is why the wait below + * checks the dashboard rather than only the API. + */ +export async function startDoppel(yaml: string): Promise { + const dir = mkdtempSync(join(tmpdir(), 'doppel-e2e-')) + const proxyPort = await freePort() + const adminPort = await freePort() + const templatesDir = join(dir, 'templates') + + const configPath = join(dir, 'main.yaml') + writeFileSync( + configPath, + yaml + .replaceAll('{proxyPort}', String(proxyPort)) + .replaceAll('{adminPort}', String(adminPort)) + .replaceAll('{templatesDir}', templatesDir) + // Short, and outside the temp directory: a unix socket path is capped near + // 104 bytes on macOS and the temp directory alone can spend most of that. + // Per-instance because the default is one path for every Doppel on the + // machine, and spec files run in parallel -- the second server would exit + // 1 with the first still holding the socket. + .replaceAll('{controlSocket}', `/tmp/doppel-e2e-${adminPort}.sock`), + ) + + const binary = process.env.DOPPEL_BIN ?? resolve(import.meta.dirname, '../../../target/debug/doppel') + const child: ChildProcess = spawn(binary, ['serve', '--config', configPath], { + stdio: ['ignore', 'pipe', 'pipe'], + }) + + // Kept, and printed only when the wait fails: a server that refused its own + // configuration says why on stderr, and without this the failure is a timeout + // with no cause. + let log = '' + child.stdout?.on('data', (chunk: Buffer) => (log += chunk.toString())) + child.stderr?.on('data', (chunk: Buffer) => (log += chunk.toString())) + + const baseURL = `http://127.0.0.1:${adminPort}` + const stop = () => child.kill('SIGTERM') + + const deadline = Date.now() + 20_000 + for (;;) { + if (child.exitCode !== null) { + throw new Error(`doppel exited with ${child.exitCode}:\n${log}`) + } + try { + const response = await fetch(`${baseURL}/api/v1/status`) + if (response.ok) { + return { baseURL, proxyURL: `http://127.0.0.1:${proxyPort}`, stop } + } + } catch { + // Not up yet. + } + if (Date.now() > deadline) { + stop() + throw new Error(`doppel did not start within 20s:\n${log}`) + } + await new Promise((accept) => setTimeout(accept, 100)) + } +} diff --git a/frontend/e2e/tests/auth.spec.ts b/frontend/e2e/tests/auth.spec.ts new file mode 100644 index 0000000..d3fe26b --- /dev/null +++ b/frontend/e2e/tests/auth.spec.ts @@ -0,0 +1,289 @@ +import { expect, test } from '@playwright/test' + +import { PRIVATE_CONFIG, PUBLIC_CONFIG, READER_TOKEN, ROOT_TOKEN } from '../src/configs' +import { startDoppel, type Doppel } from '../src/server' + +test.describe('a private configuration', () => { + let doppel: Doppel + test.beforeAll(async () => { + doppel = await startDoppel(PRIVATE_CONFIG) + }) + test.afterAll(() => doppel.stop()) + + test('asks for a token, and takes no for an answer', async ({ page }) => { + await page.goto(doppel.baseURL) + const dialog = page.getByRole('form', { name: 'Access token' }) + await expect(dialog).toBeVisible() + + await page.getByRole('button', { name: 'Continue without a token' }).click() + await expect(dialog).toBeHidden() + + // `list` and `read` are public here, so declining leaves a working page + // rather than an empty one. + await expect(page.getByRole('cell', { name: 'alpha', exact: true })).toBeVisible() + // ...and the writes are visibly unavailable rather than absent. + await expect(page.getByRole('button', { name: 'Add a proxy' })).toBeDisabled() + }) + + test('does not ask again after being declined once', async ({ page }) => { + await page.goto(doppel.baseURL) + await page.getByRole('button', { name: 'Continue without a token' }).click() + await page.reload() + await expect(page.getByRole('form', { name: 'Access token' })).toBeHidden() + }) + + test('enables the writes once a token is given, and names the caller', async ({ page }) => { + await page.goto(doppel.baseURL) + await page.getByRole('textbox', { name: 'Token' }).fill(ROOT_TOKEN) + await page.getByRole('button', { name: 'Use this token' }).click() + + await expect(page.getByText('root (admin)')).toBeVisible() + await expect(page.getByRole('button', { name: 'Add a proxy' })).toBeEnabled() + }) + + test('a wrong token leaves the writes disabled', async ({ page }) => { + await page.goto(doppel.baseURL) + await page.getByRole('textbox', { name: 'Token' }).fill('wrong-token-000000000000000000000000') + await page.getByRole('button', { name: 'Use this token' }).click() + + // An unrecognised token is anonymous, deliberately: telling it apart from a + // recognised one would confirm which tokens exist. + await expect(page.getByRole('button', { name: 'Add a proxy' })).toBeDisabled() + }) + + test('the API reads the token from the header, and says whose it is', async ({ page }) => { + // Tokens through the API rather than through the page: this is what a script, + // a CI job or `curl` does, and it is the half the dashboard tests cannot see. + const asRoot = await page.request.get(`${doppel.baseURL}/api/v1/access`, { + headers: { 'X-Proxy-Authorization': `Bearer ${ROOT_TOKEN}` }, + }) + expect(asRoot.status()).toBe(200) + const root = (await asRoot.json()) as { + caller: { kind: string; name?: string; group?: string } + global: Record + } + expect(root.caller).toMatchObject({ kind: 'token', name: 'root', group: 'admin' }) + expect(root.global.create).toBe(true) + + // A read-only token: recognised, named, and refused the writes rather than the + // reads. + const asReader = await page.request.get(`${doppel.baseURL}/api/v1/access`, { + headers: { 'X-Proxy-Authorization': `Bearer ${READER_TOKEN}` }, + }) + const reader = (await asReader.json()) as { + caller: { kind: string; name?: string } + global: Record + } + expect(reader.caller).toMatchObject({ kind: 'token', name: 'reader' }) + expect(reader.global.read).toBe(true) + expect(reader.global.create).toBe(false) + }) + + test('an unrecognised token is anonymous, not an error', async ({ page }) => { + // Deliberately indistinguishable from sending nothing: answering differently + // would confirm which tokens exist. + const wrong = await page.request.get(`${doppel.baseURL}/api/v1/access`, { + headers: { 'X-Proxy-Authorization': `Bearer wrong-token-000000000000000000000000` }, + }) + const none = await page.request.get(`${doppel.baseURL}/api/v1/access`) + + expect(wrong.status()).toBe(200) + expect(await wrong.json()).toEqual(await none.json()) + expect(((await wrong.json()) as { caller: { kind: string } }).caller.kind).toBe('anonymous') + }) + + test('a write with a token that may not write is refused, with the action named', async ({ + page, + }) => { + for (const [token, expected] of [ + [READER_TOKEN, 403], + [undefined, 401], + ] as const) { + const response = await page.request.post(`${doppel.baseURL}/api/v1/proxies`, { + headers: token ? { 'X-Proxy-Authorization': `Bearer ${token}` } : {}, + data: { + proxy: { + name: 'refused', + type: 'http', + url: 'https://example.com/', + resolve: { type: 'header', header: 'X-Proxy-Name' }, + }, + }, + }) + + expect(response.status(), `${token ?? 'no token'}`).toBe(expected) + const body = (await response.json()) as { code: string; message: string } + // 401 for "who are you", 403 for "not you": the pair a client acts on + // differently, and the message names the action rather than the resource. + expect(body.code).toBe(expected === 401 ? 'UNAUTHORIZED' : 'FORBIDDEN') + expect(body.message).toContain('create') + } + + // And nothing was written. + const list = await page.request.get(`${doppel.baseURL}/api/v1/proxies`, { + headers: { 'X-Proxy-Authorization': `Bearer ${ROOT_TOKEN}` }, + }) + expect(await list.text()).not.toContain('refused') + }) + + test('signing out returns to the anonymous view without blocking the page', async ({ page }) => { + await page.goto(doppel.baseURL) + await page.getByRole('textbox', { name: 'Token' }).fill(ROOT_TOKEN) + await page.getByRole('button', { name: 'Use this token' }).click() + await expect(page.getByRole('button', { name: 'Add a proxy' })).toBeEnabled() + + await page.getByRole('button', { name: 'Sign out' }).click() + await expect(page.getByRole('button', { name: 'Add a proxy' })).toBeDisabled() + // Not a wall: what is public is still on screen. + await expect(page.getByRole('cell', { name: 'alpha', exact: true })).toBeVisible() + await expect(page.getByRole('form', { name: 'Access token' })).toBeHidden() + }) +}) + +test.describe('a public configuration', () => { + let doppel: Doppel + test.beforeAll(async () => { + doppel = await startDoppel(PUBLIC_CONFIG) + }) + test.afterAll(() => doppel.stop()) + + test('never mentions tokens', async ({ page }) => { + await page.goto(doppel.baseURL) + await expect(page.getByRole('form', { name: 'Access token' })).toBeHidden() + await expect(page.getByRole('button', { name: 'Sign in' })).toBeHidden() + // And everything is permitted, because it is. + await expect(page.getByRole('button', { name: 'Add a proxy' })).toBeEnabled() + }) + + test('a token left in this browser changes nothing, and is not offered again', async ({ + page, + }) => { + // There is no way to set a token from the page here -- no dialog, no Sign in -- + // so the only route left is a token this browser kept from a deployment that + // was private earlier, or one somebody put there by hand. It must not resurrect + // the sign-in flow: `public: true` means the API is unauthenticated, and a page + // asking for a token it cannot use is a page asking for a secret for nothing. + await page.goto(doppel.baseURL) + await page.evaluate(() => + window.localStorage.setItem( + 'doppel.token', + // The shape `services/auth.ts` writes: the value and when it was entered. + JSON.stringify({ token: 'left-over-token-00000000000000000000', savedAt: Date.now() }), + ), + ) + await page.reload() + + await expect(page.getByRole('form', { name: 'Access token' })).toBeHidden() + await expect(page.getByRole('button', { name: 'Sign in' })).toBeHidden() + await expect(page.getByRole('button', { name: 'Sign out' })).toBeHidden() + // Still anonymous, still allowed everything: the caller the API reports is the + // anonymous one, not a name from a token nobody checked. + await expect(page.getByText('(admin)')).toBeHidden() + await expect(page.getByRole('button', { name: 'Add a proxy' })).toBeEnabled() + }) + + test('the API ignores a token instead of refusing it', async ({ page }) => { + // Both directions, because "unauthenticated" has to mean the header is not + // consulted rather than that a wrong value is punished: a client that keeps + // sending its old token must not start getting 401s when a deployment goes + // public. + const withToken = await page.request.get(`${doppel.baseURL}/api/v1/access`, { + headers: { 'X-Proxy-Authorization': 'Bearer not-a-token-0000000000000000000' }, + }) + expect(withToken.status()).toBe(200) + const report = (await withToken.json()) as { + caller: { kind: string } + global: Record + } + expect(report.caller.kind).toBe('anonymous') + expect(Object.values(report.global).every(Boolean)).toBe(true) + + // And a write goes through with no token at all. + const created = await page.request.post(`${doppel.baseURL}/api/v1/proxies`, { + data: { + proxy: { + name: 'no-token', + type: 'http', + url: 'https://example.com/', + // By header, because `alpha` is already the default resolver and rule + // V12 allows only one. + resolve: { type: 'header', header: 'X-Proxy-Name' }, + }, + }, + }) + expect(created.status(), await created.text()).toBe(201) + }) +}) + +test.describe('a refusal that only a token can fix', () => { + // Nothing is public here, so every read is refused until a token is presented -- + // which is the state the Refresh button exists for. + const CLOSED = ` +server: + host: "127.0.0.1" + port: {proxyPort} +admin: + host: "127.0.0.1" + port: {adminPort} + tokens: + - name: root + group: admin + token: ${ROOT_TOKEN} + access: + list: ["admin"] + read: ["admin"] + create: ["admin"] + update: ["admin"] + delete: ["admin"] + upload: ["admin"] + upload: + limit: 1Mi +control: + socket: {controlSocket} +templates: + dir: {templatesDir} +proxies: + - name: alpha + type: http + url: "https://alpha.example.com/api/" +` + + let doppel: Doppel + test.beforeAll(async () => { + doppel = await startDoppel(CLOSED) + }) + test.afterAll(() => doppel.stop()) + + test('the proxy form offers a token and a refresh, and both work', async ({ page }) => { + // A form reads once, when it opens. Signing in afterwards used to leave the + // refusal on screen with nothing to press: the fix was reloading the browser. + await page.goto(`${doppel.baseURL}/proxies/alpha`) + await page.getByRole('button', { name: 'Continue without a token' }).click() + + const refusal = page.getByRole('alert').first() + await expect(refusal).toContainText(/requires access|token/i) + await expect(refusal.getByRole('button', { name: 'Refresh' })).toBeVisible() + + // The other half of the offer: the thing that makes a refresh worth pressing. + await refusal.getByRole('button', { name: 'Enter token' }).click() + await page.getByRole('textbox', { name: 'Token' }).fill(ROOT_TOKEN) + await page.getByRole('button', { name: 'Use this token' }).click() + + // Still nothing read, so the form is still the empty one it renders behind the + // refusal... + await expect(page.getByLabel('Upstream URL')).toHaveValue('') + // ...and now it has been read. + await page.getByRole('button', { name: 'Refresh' }).click() + await expect(page.getByLabel('Upstream URL')).toHaveValue('https://alpha.example.com/api/') + await expect(page.getByRole('alert')).toHaveCount(0) + }) + + test('the list offers the same, and a public deployment offers no token', async ({ page }) => { + await page.goto(doppel.baseURL) + await page.getByRole('button', { name: 'Continue without a token' }).click() + + const refusal = page.getByRole('alert').first() + await expect(refusal.getByRole('button', { name: 'Refresh' })).toBeVisible() + await expect(refusal.getByRole('button', { name: 'Enter token' })).toBeVisible() + }) +}) diff --git a/frontend/e2e/tests/chrome.spec.ts b/frontend/e2e/tests/chrome.spec.ts new file mode 100644 index 0000000..b63fa3b --- /dev/null +++ b/frontend/e2e/tests/chrome.spec.ts @@ -0,0 +1,190 @@ +import { expect, test } from '@playwright/test' + +import { PRIVATE_CONFIG, PUBLIC_CONFIG } from '../src/configs' +import { paintedRgb } from '../src/colour' +import { startDoppel, type Doppel } from '../src/server' + +let doppel: Doppel + +test.beforeAll(async () => { + doppel = await startDoppel(PRIVATE_CONFIG) +}) +test.afterAll(() => doppel.stop()) + +test("the header shows admin.title, in the wordmark's own shape", async ({ page }) => { + await page.goto(doppel.baseURL) + await expect(page.getByRole('heading', { name: 'Doppel (e2e)' })).toBeVisible() + await expect(page).toHaveTitle('Doppel (e2e)') + + // A named deployment gets its own name -- that is what tells four open tabs + // apart -- drawn the way `Doppelganger` is, because it is the same heading in the + // same page and should not read as a different one. + const heading = page.getByRole('heading', { level: 1 }) + const style = await heading + .locator('span') + .first() + .evaluate((node) => { + const computed = getComputedStyle(node) + return { size: computed.fontSize, style: computed.fontStyle, select: computed.userSelect } + }) + expect(style.size).toBe('18px') + expect(style.style).toBe('italic') + expect(style.select).toBe('none') + + // Two tones, split on the space: `Doppel` in the strong one, `(e2e)` in the + // light one, exactly as the wordmark splits its own two halves. + const [first, second] = await Promise.all([ + paintedRgb(page, 'h1 span span:nth-child(1)', 'color'), + paintedRgb(page, 'h1 span span:nth-child(2)', 'color'), + ]) + expect(first).not.toEqual(second) + // Every character survives the split. + await expect(heading).toHaveText('Doppel (e2e)') +}) + +test('a named deployment says it was built with Doppel, and links to Doppel', async ({ page }) => { + // The footer belongs to somebody else's tool now: the version line says what it + // was built with, the documentation link says whose documentation it is, and the + // repository link goes -- on `billing-api (staging)`, "Repository" reads as a + // link to the billing API's own, which it is not. + await page.goto(doppel.baseURL) + + await expect(page.getByText(/Built with Doppel \d+\.\d+\.\d+/)).toBeVisible() + await expect(page.getByText(/\(c\) \d{4} Lorem Dev/)).toBeVisible() + await expect(page.getByRole('link', { name: /Doppel Documentation/ })).toBeVisible() + await expect(page.getByRole('link', { name: /^Repository/ })).toHaveCount(0) +}) + +test("an unnamed deployment keeps the project's own footer", async ({ page }) => { + const doppel = await startDoppel(PUBLIC_CONFIG) + try { + await page.goto(doppel.baseURL) + + // Nobody named this one, so it is Doppel's own page: the copyright reads as + // Doppel's and both links are its own. + await expect(page.getByText(/Built with Doppel/)).toHaveCount(0) + await expect(page.getByText(/\(c\) \d{4} Lorem Dev . Doppel \d+\.\d+\.\d+/)).toBeVisible() + await expect(page.getByRole('link', { name: /^Repository/ })).toBeVisible() + await expect(page.getByRole('link', { name: /Doppel Documentation/ })).toHaveCount(0) + await expect(page.getByRole('link', { name: /^Documentation/ })).toBeVisible() + } finally { + doppel.stop() + } +}) + +test('an unnamed deployment gets the wordmark, in two tones and one size', async ({ page }) => { + // `admin.title` absent. The page then says what this is rather than repeating a + // default string, and the two halves are the two halves of the name. + const doppel = await startDoppel(PUBLIC_CONFIG) + try { + await page.goto(doppel.baseURL) + const heading = page.getByRole('heading', { level: 1 }) + await expect(heading).toHaveText('Doppelganger') + + const [first, second] = await Promise.all([ + paintedRgb(page, 'h1 span span:nth-child(1)', 'color'), + paintedRgb(page, 'h1 span span:nth-child(2)', 'color'), + ]) + expect(first).not.toEqual(second) + + // The size the heading always was, and italic serif rather than the page's + // sans: a wordmark that grew would push the tabs along with it. + const style = await heading.locator('span').first().evaluate((node) => { + const computed = getComputedStyle(node) + return { + size: computed.fontSize, + style: computed.fontStyle, + family: computed.fontFamily, + select: computed.userSelect, + } + }) + expect(style.size).toBe('18px') + expect(style.style).toBe('italic') + expect(style.family).toMatch(/serif/i) + // And it takes no selection: a mark that highlights in two pieces when the tab + // beside it is double-clicked looks like a mistake. + expect(style.select).toBe('none') + + // The tab keeps the plain default: a browser tab is a list of words. + await expect(page).toHaveTitle('Doppel') + } finally { + doppel.stop() + } +}) + +test('the footer puts the copyright and the links on one line, at either end', async ({ page }) => { + await page.goto(doppel.baseURL) + + const copyright = await page.getByText(/Lorem Dev/).boundingBox() + const links = await page.locator('footer nav').boundingBox() + expect(copyright).not.toBeNull() + expect(links).not.toBeNull() + + // One line: their vertical centres agree. + const centre = (box: { y: number; height: number }) => box.y + box.height / 2 + expect(Math.abs(centre(copyright!) - centre(links!))).toBeLessThan(2) + // Either end, in that order. + expect(copyright!.x).toBeLessThan(links!.x) +}) + +test('the footer names the copyright and the version of the binary', async ({ page }) => { + await page.goto(doppel.baseURL) + + // Both come from the running process, so this also catches a page served by a + // binary other than the one under test. The year is the build's -- read back out + // of the page it was injected into rather than written here, because a literal + // would be a second copy of it that goes stale in January. + const injected = await page.locator('#doppel-config').textContent() + const { copyrightYear } = JSON.parse(injected ?? '{}') as { copyrightYear: number } + expect(copyrightYear).toBeGreaterThanOrEqual(2026) + + await expect(page.getByText(`(c) ${copyrightYear} Lorem Dev`)).toBeVisible() + await expect(page.getByText(/Doppel \d+\.\d+\.\d+/)).toBeVisible() +}) + +test('the theme choice survives a reload', async ({ page }) => { + await page.goto(doppel.baseURL) + await page.getByLabel('Theme').selectOption('dark') + await expect(page.locator('html')).toHaveClass(/dark/) + + await page.reload() + // Read from localStorage on the way back up: a choice that did not survive a + // reload would be a choice nobody bothers making twice. + await expect(page.getByLabel('Theme')).toHaveValue('dark') + await expect(page.locator('html')).toHaveClass(/dark/) +}) + +test('the dashboard refuses to be indexed', async ({ page }) => { + const response = await page.goto(doppel.baseURL) + expect(response?.headers()['x-robots-tag']).toContain('noindex') + + const robots = await page.request.get(`${doppel.baseURL}/robots.txt`) + expect(await robots.text()).toContain('Disallow: /') +}) + +test('navigation is real: a proxy has its own URL and the back button works', async ({ page }) => { + // The reason react-router is here rather than a state variable. + await page.goto(doppel.baseURL) + // The dialog is a fixed overlay, so it has to be out of the way before anything + // underneath it can be clicked -- which is also what an operator does. + await page.getByRole('button', { name: 'Continue without a token' }).click() + await page.getByRole('link', { name: 'alpha' }).click() + await expect(page).toHaveURL(/\/proxies\/alpha$/) + await expect(page.getByRole('heading', { name: 'Edit alpha' })).toBeVisible() + + await page.goBack() + await expect(page.getByRole('heading', { name: 'Proxies' })).toBeVisible() +}) + +test('the title goes home, from wherever the form has taken you', async ({ page }) => { + await page.goto(doppel.baseURL) + await page.getByRole('button', { name: 'Continue without a token' }).click() + await page.getByRole('link', { name: 'alpha' }).click() + await expect(page).toHaveURL(/\/proxies\/alpha$/) + + // The affordance people try first when a form has taken them somewhere they did + // not mean to be. `Doppel (e2e)` is this deployment's `admin.title`. + await page.getByRole('link', { name: 'Doppel (e2e)' }).click() + await expect(page).toHaveURL(new RegExp(`^${doppel.baseURL}/$`)) + await expect(page.getByRole('heading', { name: 'Proxies' })).toBeVisible() +}) diff --git a/frontend/e2e/tests/console.spec.ts b/frontend/e2e/tests/console.spec.ts new file mode 100644 index 0000000..c118224 --- /dev/null +++ b/frontend/e2e/tests/console.spec.ts @@ -0,0 +1,71 @@ +import { expect, test, type Page } from '@playwright/test' + +import { PRIVATE_CONFIG, ROOT_TOKEN } from '../src/configs' +import { startDoppel, type Doppel } from '../src/server' + +// The console, which an operator reads to find their own problems. +// +// Ours were in the way: `react-simple-code-editor` renders a `