From 694473725ea1c101d7e9b2d7206c0b649aa249a4 Mon Sep 17 00:00:00 2001 From: Brandur Date: Thu, 8 Oct 2026 01:17:45 -0500 Subject: [PATCH] More consistent language READMEs We have a little problem where the various language READMEs aren't all doing the same thing. Some contain user instructions, others development instructions. Here, make them explicitly intended for user-facing docs and move any internal development content to `docs/development.md` in each directory. Changes: * Java: Mostly usage examples already. Moved contributor instructions to `java/docs/development.md`. * JavaScript: Similar. Moved API-doc generation command. * Ruby: Mostly links and contributor commands. Added installation instructions and example. Moved contributor commands. * Rust: Most workspace/development workflow. Added installation instructions, example, features, and crate links. Move other content. --- java/README.md | 143 +++++---------------------------- java/docs/development.md | 121 ++++++++++++++++++++++++++++ js/README.md | 17 +++- js/docs/development.md | 3 +- ruby/README.md | 97 ++++++++++++++++++---- ruby/docs/development.md | 11 +++ rust/README.md | 169 +++++++++++++++++++++++++-------------- rust/docs/development.md | 54 +++++++++++++ 8 files changed, 418 insertions(+), 197 deletions(-) create mode 100644 java/docs/development.md diff --git a/java/README.md b/java/README.md index eefaffc21..4381192ed 100644 --- a/java/README.md +++ b/java/README.md @@ -4,6 +4,25 @@ Prerelease Java 21 client and worker runtime for River's Postgres and SQLite schemas. Jobs are ordinary River jobs: another language can insert, cancel, retry, or work them using the same database. +## Installation + +Java 21 or newer is required. Add `com.riverqueue:river:0.48.0-alpha.1` and a JDBC driver to your application's dependencies. Jackson 3 is included transitively. Applications do not need Go. + +```xml + + com.riverqueue + river + 0.48.0-alpha.1 + + + org.postgresql + postgresql + 42.7.12 + +``` + +For SQLite, replace the Postgres JDBC dependency with `org.xerial:sqlite-jdbc:3.53.4.0`. To install a local copy of River from source, see the [development guide](docs/development.md#building-from-source). + ## Quick start Define job arguments as a record, give the job a stable kind, and register a @@ -47,23 +66,6 @@ service, keep `Workers` open until the application stops. Each active job runs on a virtual thread; queue limits bound concurrent jobs. In production, run migrations as a deployment step using the CLI below. -## Installation - -The Maven artifact is `com.riverqueue:river:0.48.0-alpha.1`. To build from source, -run `make generate/fixtures` from the repository root, then `mvn install` from -`java/`, with `JAVA_HOME` pointing to JDK 21 or 25. Source tests need Go to generate -their reference fixtures; applications do not need Go. The pinned formatter does -not run on JDK 27. Add the JDBC driver for your database to your application. -Jackson 3 is included transitively. - -```xml - - com.riverqueue - river - 0.48.0-alpha.1 - -``` - Job kinds are explicit, stable wire names independent of Java class names. Record properties use snake_case in JSON by default, so `accountId` becomes `account_id`. Match kinds and JSON fields across languages sharing a job. @@ -78,10 +80,10 @@ for each operation. River never closes an application-owned data source. An The `river-cli` executable JAR includes Postgres and SQLite JDBC drivers and runs with Java 21 or newer. It needs no Go installation or application classpath. -Build it as described under [Installation](#installation), then run: +For a local build, see the [development guide](docs/development.md#building-from-source). Set `RIVER_CLI` to the executable JAR, then run: ```sh -RIVER_CLI=cli/target/river-cli-0.48.0-alpha.1-all.jar +RIVER_CLI=/path/to/river-cli-0.48.0-alpha.1-all.jar export DATABASE_URL=postgres://localhost/myapp # For SQLite: export DATABASE_URL=jdbc:sqlite:/absolute/path/myapp.db @@ -684,105 +686,4 @@ their queue's peers when due. ## Development -From the repository root, with Go (the version in `go.work`), JDK 21 or 25, -and Maven installed: - -```sh -make test/java/conformance -make test/java/sqlite -RIVER_TEST_DATABASE_URL=postgres://localhost/river_test make test/java/postgres -make lint/java -make check/java/package -make verify/java-migrations -make check/modzip -``` - -`test/java` runs library and executable CLI tests and checks formatting. Client -and worker tests use Postgres when `RIVER_TEST_DATABASE_URL` is set, with a -fresh schema for each test that is removed afterward. Otherwise they use SQLite. -`test/java/postgres` requires that URL; `test/java/sqlite` explicitly uses SQLite -and excludes Postgres-only tests, even if a URL is set in your environment. -`test/java/conformance` needs no external database: it runs the JUnit tests tagged -`conformance`, using temporary SQLite databases for storage checks even if a -Postgres URL is set. The full SQLite and Postgres targets run these checks -against their selected backend. These targets first generate fresh fixtures -directly from this checkout's Go implementation, just like -`test/js/conformance` and `test/rust/conformance`. - -Java reads all four files in `conformance/testdata`: uniqueness hashes, cron -schedules, snooze counters, and protocol values (states, metadata keys, -notification payloads, attempt errors, and retry bounds). These files are ignored -by Git and read at test time. Missing fixtures fail with instructions to run -`make generate/fixtures`; there are no bundled fallback goldens. To use Maven or -an IDE directly, generate the fixtures first. `make test` from `java/` delegates -to the root target; `mvn spotless:apply` formats Java sources. - -Notification tests check both encoding and dispatch: targeted cancellation, queue -wakeups, leadership signals, and ignoring a client's own resignation. -Database-backed tests also verify topic routing, recovery after malformed -payloads, and immediate leadership wakeup after a peer resigns. - -Uniqueness tests include the raw JSON fixtures for duplicate keys and integer-like -map keys. Protocol tests exercise both retry jitter boundaries, decode stored -states and uniqueness bits, and verify periodic IDs, insert nonces, resumable -checkpoints, output, and rescue counters using Go's metadata keys. - -`make generate/java-migrations` syncs SQL and the migration catalog from the -canonical Go drivers. `verify/java-migrations` detects changed, missing, or extra -migrations. `check/java/package` checks the library and CLI archives, including -source JARs, for development content and verifies bundled runtime resources. -The Go maintenance tools live in `java/bin/`. `make test/java` and `make lint/java` -include their tests and lint checks; `make test/java/tools` and -`make lint/java/tools` run only the tool checks. From `java/`, use `make test/tools` -and `make lint/tools`. These checks run in Java CI, separately from the root -Go-only `make test` and `make lint` targets. -The legacy adapter is not installed or deployed as a Maven artifact. -`java/go.mod` excludes this directory from Go module archives and package -discovery. The Make targets invoke the Go tools by file from the root workspace; -the module is not part of `go.work` and must not be tagged as a Go module. - -The Java CI workflow follows the Rust and JavaScript layout: a quality/package -job, a JDK 21/25 matrix running unit and SQLite tests, and a Postgres 14–18 -matrix running client, worker, and migration tests on JDK 25. All jobs use the -shared Java setup action and Maven dependency caching. The whole workflow is -filtered to changes in Java, its build configuration, fixtures, and canonical -migrations. Go changes run the smaller fixture suite in the shared Conformance -workflow alongside Rust and JavaScript. - -### Legacy cross-process harness - -The original interoperability adapter remains available for broader storage, -runtime, and multi-engine scenarios. It uses the historical harness pinned in -`java/conformance/reference-revision`, which is separate from the current -Go-generated fixtures and is not part of `master`'s conformance module. - -From `java/`, using only disposable databases (the harness resets job tables): - -```sh -RIVER_CONFORMANCE_DATABASE_URL=postgres://localhost/river_java_conformance \ - python3 conformance/bin/run.py postgres -python3 conformance/bin/run.py sqlite -``` - -The runner extracts that revision into ignored build storage and registers the -Java candidate there. `--reference /path/to/checkout` uses an existing harness. -`--refresh` opts into the old reference branch's current head; review its -contract and profiles before updating the pin. - -See [intentional differences](DIFFERENCES.md) for the Java API and lifecycle -choices. Passing a conformance profile demonstrates the scenarios in that -profile; it is not a claim about untested behavior or performance. - -The full peer matrix uses `multi` with `RIVER_CONFORMANCE_PEER_FILE` set to a -colon-separated list of Rust and JS candidate descriptors, and -`RIVERQUEUE_JS_ROOT` pointing to the JS checkout. `multi-soak` also requires -`RIVER_CONFORMANCE_MULTI_ENGINE_SOAK_DURATION=5m` (or longer). The upstream -harness currently has a Postgres soak; SQLite endurance validation repeats -`TestMultiEngineSQLiteConformance` with `-count=5`. - -`python3 bin/import-reference.py --check /path/to/reference` verifies the legacy -adapter contract. Omit `--check` to import it and record its hash. This command -does not overwrite the current migrations or generated fixtures. - -See [validation results](VALIDATION.md) for the tested revisions and remaining -limitations. +See [developing River for Java](docs/development.md). diff --git a/java/docs/development.md b/java/docs/development.md new file mode 100644 index 000000000..fe24afa65 --- /dev/null +++ b/java/docs/development.md @@ -0,0 +1,121 @@ +# River Java development + +Run commands from the repository root unless a section says otherwise. See the [Java README](../README.md) for application usage. + +## Building from source + +Install the Go version specified by [`go.work`](../../go.work), Maven, and JDK 21 or 25. Set `JAVA_HOME` and `PATH` to that JDK; the pinned formatter does not run on JDK 27. Go is needed to generate test fixtures, but applications using River do not need Go. + +Generate the fixtures and install the Java artifacts into your local Maven repository: + +```sh +make generate/fixtures +mvn --batch-mode --no-transfer-progress -f java/pom.xml install +``` + +The executable migration CLI is built at `java/cli/target/river-cli-0.48.0-alpha.1-all.jar`. For a build without running tests, use `make build/java`. + +## Tests and checks + +From the repository root, with Go (the version in `go.work`), JDK 21 or 25, +and Maven installed: + +```sh +make test/java/conformance +make test/java/sqlite +RIVER_TEST_DATABASE_URL=postgres://localhost/river_test make test/java/postgres +make lint/java +make check/java/package +make verify/java-migrations +make check/modzip +``` + +`test/java` runs library and executable CLI tests and checks formatting. Client +and worker tests use Postgres when `RIVER_TEST_DATABASE_URL` is set, with a +fresh schema for each test that is removed afterward. Otherwise they use SQLite. +`test/java/postgres` requires that URL; `test/java/sqlite` explicitly uses SQLite +and excludes Postgres-only tests, even if a URL is set in your environment. +`test/java/conformance` needs no external database: it runs the JUnit tests tagged +`conformance`, using temporary SQLite databases for storage checks even if a +Postgres URL is set. The full SQLite and Postgres targets run these checks +against their selected backend. These targets first generate fresh fixtures +directly from this checkout's Go implementation, just like +`test/js/conformance` and `test/rust/conformance`. + +Java reads all four files in `conformance/testdata`: uniqueness hashes, cron +schedules, snooze counters, and protocol values (states, metadata keys, +notification payloads, attempt errors, and retry bounds). These files are ignored +by Git and read at test time. Missing fixtures fail with instructions to run +`make generate/fixtures`; there are no bundled fallback goldens. To use Maven or +an IDE directly, generate the fixtures first. `make test` from `java/` delegates +to the root target; `mvn spotless:apply` formats Java sources. + +Notification tests check both encoding and dispatch: targeted cancellation, queue +wakeups, leadership signals, and ignoring a client's own resignation. +Database-backed tests also verify topic routing, recovery after malformed +payloads, and immediate leadership wakeup after a peer resigns. + +Uniqueness tests include the raw JSON fixtures for duplicate keys and integer-like +map keys. Protocol tests exercise both retry jitter boundaries, decode stored +states and uniqueness bits, and verify periodic IDs, insert nonces, resumable +checkpoints, output, and rescue counters using Go's metadata keys. + +`make generate/java-migrations` syncs SQL and the migration catalog from the +canonical Go drivers. `verify/java-migrations` detects changed, missing, or extra +migrations. `check/java/package` checks the library and CLI archives, including +source JARs, for development content and verifies bundled runtime resources. +The Go maintenance tools live in `java/bin/`. `make test/java` and `make lint/java` +include their tests and lint checks; `make test/java/tools` and +`make lint/java/tools` run only the tool checks. From `java/`, use `make test/tools` +and `make lint/tools`. These checks run in Java CI, separately from the root +Go-only `make test` and `make lint` targets. +The legacy adapter is not installed or deployed as a Maven artifact. +`java/go.mod` excludes this directory from Go module archives and package +discovery. The Make targets invoke the Go tools by file from the root workspace; +the module is not part of `go.work` and must not be tagged as a Go module. + +The Java CI workflow follows the Rust and JavaScript layout: a quality/package +job, a JDK 21/25 matrix running unit and SQLite tests, and a Postgres 14–18 +matrix running client, worker, and migration tests on JDK 25. All jobs use the +shared Java setup action and Maven dependency caching. The whole workflow is +filtered to changes in Java, its build configuration, fixtures, and canonical +migrations. Go changes run the smaller fixture suite in the shared Conformance +workflow alongside Rust and JavaScript. + +## Legacy cross-process harness + +The original interoperability adapter remains available for broader storage, +runtime, and multi-engine scenarios. It uses the historical harness pinned in +`java/conformance/reference-revision`, which is separate from the current +Go-generated fixtures and is not part of `master`'s conformance module. + +From `java/`, using only disposable databases (the harness resets job tables): + +```sh +RIVER_CONFORMANCE_DATABASE_URL=postgres://localhost/river_java_conformance \ + python3 conformance/bin/run.py postgres +python3 conformance/bin/run.py sqlite +``` + +The runner extracts that revision into ignored build storage and registers the +Java candidate there. `--reference /path/to/checkout` uses an existing harness. +`--refresh` opts into the old reference branch's current head; review its +contract and profiles before updating the pin. + +See [intentional differences](../DIFFERENCES.md) for the Java API and lifecycle +choices. Passing a conformance profile demonstrates the scenarios in that +profile; it is not a claim about untested behavior or performance. + +The full peer matrix uses `multi` with `RIVER_CONFORMANCE_PEER_FILE` set to a +colon-separated list of Rust and JS candidate descriptors, and +`RIVERQUEUE_JS_ROOT` pointing to the JS checkout. `multi-soak` also requires +`RIVER_CONFORMANCE_MULTI_ENGINE_SOAK_DURATION=5m` (or longer). The upstream +harness currently has a Postgres soak; SQLite endurance validation repeats +`TestMultiEngineSQLiteConformance` with `-count=5`. + +`python3 bin/import-reference.py --check /path/to/reference` verifies the legacy +adapter contract. Omit `--check` to import it and record its hash. This command +does not overwrite the current migrations or generated fixtures. + +See [validation results](../VALIDATION.md) for the tested revisions and remaining +limitations. diff --git a/js/README.md b/js/README.md index 7b42e5fa1..7d8f0773c 100644 --- a/js/README.md +++ b/js/README.md @@ -2,7 +2,7 @@ River is a fast, reliable background job system backed by Postgres or SQLite. This implementation runs on Node.js and shares River's database protocol with -River for Go and Rust, so services in all three languages can insert and work +River's other language implementations, so services can insert and work the same jobs in the same database. ## Requirements @@ -93,6 +93,17 @@ For SQLite, install `@riverqueue/driver-sqlite` instead of the Postgres packages; it uses Node's built-in `node:sqlite`. See the [SQLite driver](./driver/sqlite/README.md). +## Transactions and other features + +[Insert jobs in an application transaction](./docs/README.md#transactions) so the job and its related database changes commit or roll back together. River supports transactions with node-postgres, Prisma, and SQLite. + +- [Bulk insertion](./docs/README.md#batches) and [unique jobs](./docs/README.md#unique-jobs). +- [Retries, timeouts, cancellation, and snoozing](./docs/errors-and-retries.md). +- [Periodic jobs](./docs/periodic-jobs.md) and [resumable jobs](./docs/resumable-jobs.md). +- [Job and queue administration](./docs/README.md#query-and-control-jobs). +- [Logging, events, and metrics](./docs/observability.md). +- [Worker threads](./worker-threads/README.md) for CPU-bound handlers. + ## Packages | Package | Purpose | @@ -120,7 +131,9 @@ Start with the [guide](./docs/README.md), then the topic guides: - [Running alongside Go and Rust](./docs/deployment.md) - [Migrating from `riverqueue` 0.1](./docs/migrating-from-0.1.md) -API reference documentation is generated with `pnpm run docs:api`. +## Development + +See [developing River for JavaScript and TypeScript](./docs/development.md). ## License diff --git a/js/docs/development.md b/js/docs/development.md index 7de378c24..d0b951162 100644 --- a/js/docs/development.md +++ b/js/docs/development.md @@ -1,4 +1,4 @@ -# River TypeScript development +# River JavaScript and TypeScript development ## Setup @@ -37,6 +37,7 @@ pnpm run test:coverage # Run unit tests with line/branch coverage pnpm run test:integration # Run integration tests (requires database) pnpm run verify:migrations # Verify generated migration files and hashes pnpm run migration:legacy # Verify and compile the pinned 0.1.0 fixture +pnpm run docs:api # Generate the API reference with TypeDoc pnpm run docs:snippets # Typecheck package README examples pnpm run api:report # Regenerate the etc/*.api.md declaration reports pnpm run package:check # Validate tarballs, consumers, and examples diff --git a/ruby/README.md b/ruby/README.md index 82f30b933..af52500cc 100644 --- a/ruby/README.md +++ b/ruby/README.md @@ -1,22 +1,91 @@ # River for Ruby -A Postgres and SQLite job queue that shares River's schema with the Go, Rust, -and JavaScript clients. Includes Active Record and Sequel drivers, plus Rails -and Active Job integration. +River is a fast, reliable background job system backed by Postgres or SQLite. The Ruby client inserts and works jobs using the same database schema and job protocol as River's other languages. It includes Active Record and Sequel drivers, plus Rails and Active Job integration. -- [Usage and configuration](docs/README.md) -- [Development and releases](docs/development.md) -- [Conformance coverage](docs/conformance.md) -- [Migrations](docs/migrations.md) +## Installation -From the repository root: +Ruby 3.2 or newer is required. Add a River driver and its database adapter to your application's `Gemfile`; the driver brings in the core `riverqueue` gem: + +```ruby +gem "riverqueue-sequel" +gem "pg" +``` + +Use `riverqueue-activerecord` for Active Record, or `sqlite3` instead of `pg` for SQLite. Install the gems with `bundle install`. Rails applications can use [`riverqueue-rails`](rails/riverqueue-rails/README.md) to configure River as their Active Job backend. + +## Quick start + +Set `DATABASE_URL` to an existing application database and apply River's migrations before inserting or working jobs: ```sh -make -C ruby install -createdb river_test -RIVER_REQUIRE_DATABASES=1 make test/ruby -make lint/ruby typecheck/ruby +export DATABASE_URL=postgres://localhost/my_app +bundle exec river migrate-up +``` + +Define job arguments and a worker with the same stable kind, register the worker, and insert a job. Save this as `worker.rb`: + +```ruby +require "riverqueue-sequel" + +class SendEmailArgs + def initialize(address:) + @address = address + end + + def kind = "send_email" + def to_json = JSON.generate(address: @address) +end + +class SendEmailWorker + def self.kind = "send_email" + + def work(job) + puts "Sending email to #{job.args.fetch("address")}" + job.output = {delivered: true} + end +end + +db = Sequel.connect(ENV.fetch("DATABASE_URL"), max_connections: 12) +client = River::Client.new( + River::Driver::Sequel.new(db), + config: River::Config.new( + queues: {default: 10}, + workers: River::Workers.new.add(SendEmailWorker) + ) +) + +begin + client.start + result = client.insert(SendEmailArgs.new(address: "person@example.com")) + puts "Inserted job #{result.job.id}; press Ctrl-C to stop" + sleep +rescue Interrupt + # Stop fetching and wait for active jobs before closing the database pool. +ensure + client.stop + db.disconnect +end ``` -Ruby 3.2 or later is required. `make test/ruby/conformance` runs only the -Go-generated fixture checks and needs no Postgres server. +Run `bundle exec ruby worker.rb`. Replace the worker body with your mail service. Job arguments implement `kind` and `to_json`; workers read decoded JSON keys as strings. Services sharing a job must agree on its kind and JSON fields. For SQLite and alternate Postgres schemas, see [migrations](docs/migrations.md). + +A client without configured queues can insert jobs without starting worker threads. Applications own their database pools and should close them only after stopping River. + +## Transactions and other features + +Inserts join a transaction opened on the same Sequel database or Active Record connection, so jobs commit or roll back with the application writes that caused them. See [transactional enqueueing](docs/README.md#transactional-enqueueing) for examples. + +- [Bulk insertion](docs/README.md#bulk-insertion), [scheduled jobs](docs/README.md#scheduled-jobs), and [unique jobs](docs/README.md#unique-jobs). +- [Retries](docs/README.md#job-retries), [cancellation](docs/README.md#cancelling-jobs), and [snoozing](docs/README.md#snoozing-jobs). +- [Multiple queues](docs/README.md#multiple-queues), [periodic jobs](docs/README.md#periodic-jobs), and [resumable jobs](docs/README.md#resumable-jobs). +- [Event subscriptions](docs/README.md#subscriptions) and [job administration](docs/README.md#job-administration). +- [Graceful stopping](docs/README.md#stopping-gracefully) and a foreground [worker command](docs/workers.md). +- [Testing helpers](docs/testing.md) for application jobs and workers. + +## Documentation + +See the [usage and configuration guide](docs/README.md), [migration guide](docs/migrations.md), and [Rails and Active Job integration](rails/riverqueue-rails/README.md). The [Sidekiq migration guide](docs/migrating_from_sidekiq.md) covers moving an existing application. + +## Development + +See [developing River for Ruby](docs/development.md). diff --git a/ruby/docs/development.md b/ruby/docs/development.md index 62dbd6766..368724cd7 100644 --- a/ruby/docs/development.md +++ b/ruby/docs/development.md @@ -2,6 +2,17 @@ All commands on this page run from `ruby/`, unless specified otherwise. +From the repository root, the equivalent setup and checks are: + +```sh +make -C ruby install +createdb river_test +RIVER_REQUIRE_DATABASES=1 make test/ruby +make lint/ruby typecheck/ruby +``` + +`make test/ruby/conformance` runs only the Go-generated fixture checks and needs no Postgres server. + ## Install dependencies ```shell diff --git a/rust/README.md b/rust/README.md index 6843bd8ed..39b8b77b1 100644 --- a/rust/README.md +++ b/rust/README.md @@ -1,77 +1,128 @@ # River for Rust -This workspace contains River's Rust implementation. It shares River's database schema and job protocol with River for Go on Postgres and SQLite, with an API designed for Rust and Tokio. The Rust crates are versioned independently of River for Go. +River is a fast, reliable background job system backed by Postgres or SQLite. Its Rust implementation uses Tokio and shares River's database schema and job protocol with the other languages, so services can insert and work jobs in the same database. -## Workspace crates +## Installation -- `riverqueue`: typed client, worker runtime, CRUD, queues, events, extensions, - periodic/resumable jobs, and maintenance. -- `riverqueue-macros`: `#[derive(JobArgs)]`. -- `riverqueue-migrate`: canonical River migration lines. -- `riverqueue-cli`: the `riverqueue` command-line program for migrations and - benchmarks. -- `riverqueue-test`: typed fixtures and worker-test helpers. +```toml +[dependencies] +riverqueue = "0.3.0" +serde = { version = "1", features = ["derive"] } +serde_json = "1" +tokio = { version = "1", features = ["macros", "rt-multi-thread", "signal"] } +``` -The API uses a caller-owned SQLx pool, Tokio, typed workers, and -`CancellationToken`. `Client` isn't generic over the database: it accepts a -Postgres or SQLite pool, and there's no driver trait to implement. +The quick start below uses these dependencies. River runs on Tokio, and job +arguments derive Serde's `Serialize` and `Deserialize`. The minimum supported +Rust version is 1.95. + +| Feature | Default | Enables | +|---|---|---| +| `postgres` | yes | Postgres through SQLx | +| `sqlite` | no | SQLite 3.45 or newer through SQLx | +| `chrono-tz` | no | IANA zone names such as `America/New_York` in cron `CRON_TZ=` and `TZ=` prefixes | + +For SQLite alone, use +`riverqueue = { version = "0.3.0", default-features = false, features = ["sqlite"] }`. + +River's API uses types from SQLx (pools and transactions), Chrono +(timestamps), `serde_json` (metadata, outputs, and other JSON values), and +`tokio-util` (the worker's `CancellationToken`). The crate re-exports each one +as `riverqueue::sqlx`, `riverqueue::chrono`, `riverqueue::serde_json`, and +`riverqueue::tokio_util`. Use the re-exports, or depend on versions +compatible with River's (SQLx 0.9, Chrono 0.4, `serde_json` 1, and +`tokio-util` 0.7), so the types match. River doesn't choose a TLS +implementation for SQLx; enable one of SQLx's TLS features in your own SQLx +dependency if your database connections use TLS. ## Quick start -The [`riverqueue` crate README](riverqueue/README.md) walks through defining -a job, registering a worker, inserting, and starting a client. +Set `DATABASE_URL` to your application's Postgres database. Define serializable job arguments, register a worker, apply migrations, and start a client: + +```rust,no_run +use riverqueue::migrate::PostgresMigrator; +use riverqueue::sqlx::PgPool; +use riverqueue::{ + BoxError, Client, Job, JobArgs, QueueConfig, WorkContext, WorkOutcome, Workers, +}; +use serde::{Deserialize, Serialize}; + +#[derive(Clone, Debug, Deserialize, JobArgs, Serialize)] +#[river(kind = "send_email")] +struct SendEmail { + address: String, +} + +async fn send_email( + context: WorkContext, + job: Job, +) -> Result { + println!("sending email to {}", job.args.address); + context.record_output(serde_json::json!({"delivered": true}))?; + Ok(WorkOutcome::Complete) +} + +#[tokio::main] +async fn main() -> Result<(), Box> { + let pool = PgPool::connect(&std::env::var("DATABASE_URL")?).await?; + // Create or upgrade River's tables. Applications often run + // `riverqueue migrate-up` from `riverqueue-cli` at deploy time instead. + PostgresMigrator::new(pool.clone()).migrate_up().await?; + + let mut workers = Workers::new(); + workers.add_fn(send_email)?; + + let client = Client::builder(pool) + .workers(workers) + .queue("default", QueueConfig::new(10)) + .build()?; + // Work jobs until Ctrl-C, then stop fetching and let running jobs finish. + let mut run = client.start_with_graceful_stop(async { + let _ = tokio::signal::ctrl_c().await; + })?; + + client + .insert(SendEmail { + address: "person@example.com".to_owned(), + }) + .await?; + + run.wait().await?; + Ok(()) +} +``` -To run Rust clients alongside River Go against one database, including schema and protocol compatibility, queue and kind layout, unique jobs, and rolling deployment and rollback, see the [mixed deployment guide](riverqueue/docs/mixed-deployments.md), also published as `riverqueue::guide::mixed_deployments`. +Run the application and press Ctrl-C to stop fetching jobs and let active workers finish. Apply migrations before starting any clients; production deployments can use the [`riverqueue` CLI](riverqueue-cli/README.md) as a separate deployment step. -Runnable examples in `riverqueue/examples` cover workers and graceful -shutdown, cancellation, transactional completion, unique and periodic jobs, -event subscriptions, custom schemas, SQLite, and a mixed Go and Rust -deployment; `riverqueue-migrate/examples` covers migrations. +A client without queues or workers can insert jobs without being started. Job kinds and serialized JSON fields must agree between languages sharing a job. -Run the Rust suite from the repository root: +## Transactions and other features -```sh -make lint/rust -make test/rust -make doc/rust -make check/rust/package -``` +Insert jobs in the same transaction as application data by chaining `.tx(&mut transaction)` onto `client.insert(args)`. The job becomes available only when that transaction commits. The [insertion guide](riverqueue/README.md#inserting-jobs) covers transaction helpers and error handling. -For basic end-to-end performance figures, the `riverqueue` binary from -`riverqueue-cli` has the Rust equivalent of `river bench`. It truncates the selected River job table, -so use a disposable database: +- [Job and queue administration](riverqueue/README.md#managing-jobs-and-queues), including cancellation, retries, and pausing queues. +- [Worker outcomes and cancellation](riverqueue/README.md#worker-outcomes-and-cancellation), including snoozing and graceful stopping. +- [Unique, periodic, and resumable jobs](riverqueue/README.md#reliability-features). +- [Event subscriptions](riverqueue/README.md#events) for logging and metrics. +- [Leader election and maintenance](riverqueue/README.md#leadership-and-maintenance) for scheduling, rescue, and cleanup. +- [Postgres and SQLite support](riverqueue/README.md#database-support) with caller-owned SQLx pools. -```sh -make bench/rust DATABASE_URL=postgres://localhost/river_bench \ - RUST_BENCH_ARGS='--duration 30s' -``` +## Crates -The command supports continuous burn, fixed `--num-total-jobs` burn-down, -custom schemas, tunable worker/pool/batch sizes, periodic jobs/sec output, and a -final jobs/sec plus p95 end-to-end latency summary. Use `riverqueue bench ---help` for all options. +| Crate | Purpose | +| --- | --- | +| [`riverqueue`](riverqueue/README.md) | Typed client, workers, job and queue administration, and maintenance | +| [`riverqueue-macros`](riverqueue-macros/README.md) | `#[derive(JobArgs)]` | +| [`riverqueue-migrate`](riverqueue-migrate/README.md) | River's database migrations | +| [`riverqueue-cli`](riverqueue-cli/README.md) | The `riverqueue` command for migrations and benchmarks | +| [`riverqueue-test`](riverqueue-test/README.md) | Fixtures and worker test helpers | -Postgres integration tests require a disposable database. They build only -with `--cfg river_postgres_tests`, which the Makefile targets pass to rustc -and rustdoc, building into `target/postgres-tests`: +The Rust crates are versioned together, independently of River for Go. -```sh -RIVER_RUST_DATABASE_URL=postgres://localhost/river_rust_test \ - make test/rust/postgres -``` +## Documentation + +See the [client guide](riverqueue/README.md), [API reference](https://docs.rs/riverqueue), and [mixed deployment guide](riverqueue/docs/mixed-deployments.md). The [examples](riverqueue/examples) cover workers, graceful stopping, transactions, unique and periodic jobs, events, custom schemas, and SQLite. + +## Development -CI runs unit, doc, and SQLite tests on each supported Rust version, and -Postgres tests against versions 14 through 18. Rust tests check unique -keys, retry bounds, cron schedules, and snooze counts against fixtures that -River's Go implementation generates into `conformance/testdata`, which isn't -committed. The `make test/rust` targets generate them first, so Go is needed -to run the tests; when running `cargo test` directly, run `make -generate/fixtures` beforehand. A missing fixture fails its test. - -`make check/rust/package` builds the five publishable crate archives and -verifies that each one builds from its packaged sources, resolving the -exact-version workspace dependencies from the other archives. It does not -publish anything. Release tags use `rust/vX.Y.Z`, independently of Go -module tags. - -See [Rust development](docs/development.md#releasing-a-new-version) for the release procedure. +See [developing River for Rust](docs/development.md). diff --git a/rust/docs/development.md b/rust/docs/development.md index cc7f42e9d..e30ce8eb0 100644 --- a/rust/docs/development.md +++ b/rust/docs/development.md @@ -1,5 +1,59 @@ # River Rust development +Run commands from the repository root. See the [Rust README](../README.md) for application usage. + +## Setup + +Install Rust 1.95 or newer and the Go version specified by [`go.work`](../../go.work). Go generates the shared conformance fixtures used by the tests. Postgres integration tests also need a disposable database. + +## Tests and checks + +```sh +make lint/rust +make test/rust +make doc/rust +make check/rust/package +``` + +Postgres integration tests require a disposable database. They build only +with `--cfg river_postgres_tests`, which the Makefile targets pass to rustc +and rustdoc, building into `target/postgres-tests`: + +```sh +RIVER_RUST_DATABASE_URL=postgres://localhost/river_rust_test \ + make test/rust/postgres +``` + +CI runs unit, doc, and SQLite tests on each supported Rust version, and +Postgres tests against versions 14 through 18. Rust tests check unique +keys, retry bounds, cron schedules, and snooze counts against fixtures that +River's Go implementation generates into `conformance/testdata`, which isn't +committed. The `make test/rust` targets generate them first, so Go is needed +to run the tests; when running `cargo test` directly, run `make +generate/fixtures` beforehand. A missing fixture fails its test. + +`make check/rust/package` builds the five publishable crate archives and +verifies that each one builds from its packaged sources, resolving the +exact-version workspace dependencies from the other archives. It does not +publish anything. Release tags use `rust/vX.Y.Z`, independently of Go +module tags. + +## Benchmarking + +For basic end-to-end performance figures, the `riverqueue` binary from +`riverqueue-cli` has the Rust equivalent of `river bench`. It truncates the selected River job table, +so use a disposable database: + +```sh +make bench/rust DATABASE_URL=postgres://localhost/river_bench \ + RUST_BENCH_ARGS='--duration 30s' +``` + +The command supports continuous burn, fixed `--num-total-jobs` burn-down, +custom schemas, tunable worker/pool/batch sizes, periodic jobs/sec output, and a +final jobs/sec plus p95 end-to-end latency summary. Use `riverqueue bench +--help` for all options. + ## Releasing a new version Run these commands from the repository root. All five Rust crates are versioned and released together. `VERSION` has no leading `v`; Git tags use `rust/vX.Y.Z`, independently of Go module tags.