Skip to content

Commit d5649aa

Browse files
bgentrybgentry
committed
run the conformance harness in CI
The harness proves River Go, Rust, and JavaScript agree when they share a database, but nothing runs it, so a change to any of them can break the agreement unnoticed. Each pull request runs one job per candidate, only when that candidate's files or the conformance module change: Go against itself in the `Conformance` workflow, and Rust and JavaScript against Go in their own workflows, which already filter on `rust/**`, `js/**`, and `conformance/**`. A job runs every scenario on PostgreSQL 18 and then SQLite, so the pull request tier adds three runners, each a few minutes, and no matrix. A new `Conformance nightly` workflow, on a schedule and by hand, runs what's too slow or noisy for pull requests: - The nightly tier (process kills, database faults, rolling deploys, completion batching, pool pressure) for each candidate on PostgreSQL 14 through 18, with SQLite only on 18. - A three-engine fleet of Go, Rust, and JavaScript, and the ordinary suite with JavaScript against Rust as the reference. - Throughput and p95 latency of Rust and JavaScript relative to Go, reported without failing the run on shared runners. - A one-hour soak through all three engines. Its `release_candidate` input soaks for 5.5 hours instead, the most a GitHub-hosted job's six-hour limit leaves room for, and makes a throughput or latency regression fail the run.
1 parent b448958 commit d5649aa

4 files changed

Lines changed: 448 additions & 4 deletions

File tree

‎.github/workflows/conformance-nightly.yaml‎

Lines changed: 315 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,315 @@
1+
name: Conformance nightly
2+
3+
# The pull request jobs run each candidate against River Go on PostgreSQL 18
4+
# and SQLite. This workflow adds what's too slow or too noisy for them: the
5+
# nightly tier (process kills, database faults, rolling deploys, connection
6+
# and batching checks) on every supported PostgreSQL version, three-engine
7+
# fleets, Rust and JavaScript paired without Go, throughput comparisons, and
8+
# a soak. Run it by hand with `release_candidate` before a release to soak
9+
# for longer and fail on performance regressions.
10+
on:
11+
schedule:
12+
- cron: "23 7 * * *"
13+
workflow_dispatch:
14+
inputs:
15+
release_candidate:
16+
default: false
17+
description: Soak for 5.5 hours and fail on throughput or latency regressions
18+
type: boolean
19+
20+
permissions:
21+
contents: read
22+
23+
env:
24+
# Keep the cross-run cache small; Cargo still reuses compiled dependencies.
25+
CARGO_INCREMENTAL: "0"
26+
RIVER_CONFORMANCE_NIGHTLY: "1"
27+
TEST_DATABASE_URL: postgres://postgres:postgres@localhost:5432/river_test?sslmode=disable
28+
29+
jobs:
30+
nightly:
31+
name: Nightly tier (${{ matrix.candidate }}, PostgreSQL ${{ matrix.postgres-version }})
32+
runs-on: ubuntu-latest
33+
timeout-minutes: 50
34+
35+
strategy:
36+
fail-fast: false
37+
matrix:
38+
candidate: [go, js, rust]
39+
postgres-version: [14, 15, 16, 17, 18]
40+
41+
services:
42+
postgres:
43+
image: postgres:${{ matrix.postgres-version }}
44+
env:
45+
POSTGRES_DB: river_test
46+
POSTGRES_PASSWORD: postgres
47+
options: >-
48+
--health-cmd pg_isready
49+
--health-interval 2s
50+
--health-timeout 5s
51+
--health-retries 5
52+
ports:
53+
- 5432:5432
54+
55+
env:
56+
RIVER_CONFORMANCE: ${{ matrix.candidate }}
57+
# SQLite doesn't vary with the PostgreSQL version, so only the newest
58+
# version's job runs it.
59+
RIVER_CONFORMANCE_DRIVERS: ${{ matrix.postgres-version == 18 && 'postgres,sqlite' || 'postgres' }}
60+
61+
steps:
62+
- uses: actions/checkout@v6
63+
with:
64+
persist-credentials: false
65+
66+
- uses: actions/setup-go@v6
67+
with:
68+
cache-dependency-path: conformance/go.sum
69+
go-version-file: go.work
70+
71+
- uses: dtolnay/rust-toolchain@stable
72+
if: matrix.candidate == 'rust'
73+
id: rust
74+
75+
- name: Cache Rust dependencies and build artifacts
76+
if: matrix.candidate == 'rust'
77+
uses: actions/cache@v5
78+
with:
79+
path: |
80+
~/.cargo/registry/index
81+
~/.cargo/registry/cache
82+
~/.cargo/git/db
83+
rust/target
84+
key: rust-v1-${{ runner.os }}-${{ runner.arch }}-conformance-${{ steps.rust.outputs.cachekey }}-${{ hashFiles('rust/**/Cargo.toml', 'rust/Cargo.lock') }}
85+
restore-keys: rust-v1-${{ runner.os }}-${{ runner.arch }}-conformance-${{ steps.rust.outputs.cachekey }}-
86+
87+
- uses: ./.github/actions/setup-js
88+
if: matrix.candidate == 'js'
89+
90+
- name: Build the Rust adapter
91+
if: matrix.candidate == 'rust'
92+
run: make build/rust/conformance
93+
94+
- name: Build the JavaScript adapter
95+
if: matrix.candidate == 'js'
96+
run: make build/js/conformance
97+
98+
# Throughput comparisons run once, in the performance job.
99+
- name: Conformance scenarios and the nightly tier
100+
run: go test ./harness -count=1 -timeout 40m -skip '^TestPerformance$/^Throughput$'
101+
working-directory: conformance
102+
103+
# Throughput and p95 latency relative to River Go, on PostgreSQL 18. Shared
104+
# runners are noisy, so a scheduled run only reports a regression; a
105+
# release candidate run fails on one.
106+
performance:
107+
name: Performance (${{ matrix.candidate }})
108+
runs-on: ubuntu-latest
109+
timeout-minutes: 30
110+
continue-on-error: ${{ !inputs.release_candidate }}
111+
112+
strategy:
113+
fail-fast: false
114+
matrix:
115+
candidate: [js, rust]
116+
117+
services:
118+
postgres:
119+
image: postgres:18
120+
env:
121+
POSTGRES_DB: river_test
122+
POSTGRES_PASSWORD: postgres
123+
options: >-
124+
--health-cmd pg_isready
125+
--health-interval 2s
126+
--health-timeout 5s
127+
--health-retries 5
128+
ports:
129+
- 5432:5432
130+
131+
env:
132+
RIVER_CONFORMANCE: ${{ matrix.candidate }}
133+
134+
steps:
135+
- uses: actions/checkout@v6
136+
with:
137+
persist-credentials: false
138+
139+
- uses: actions/setup-go@v6
140+
with:
141+
cache-dependency-path: conformance/go.sum
142+
go-version-file: go.work
143+
144+
- uses: dtolnay/rust-toolchain@stable
145+
if: matrix.candidate == 'rust'
146+
id: rust
147+
148+
- name: Cache Rust dependencies and build artifacts
149+
if: matrix.candidate == 'rust'
150+
uses: actions/cache@v5
151+
with:
152+
path: |
153+
~/.cargo/registry/index
154+
~/.cargo/registry/cache
155+
~/.cargo/git/db
156+
rust/target
157+
key: rust-v1-${{ runner.os }}-${{ runner.arch }}-conformance-${{ steps.rust.outputs.cachekey }}-${{ hashFiles('rust/**/Cargo.toml', 'rust/Cargo.lock') }}
158+
restore-keys: rust-v1-${{ runner.os }}-${{ runner.arch }}-conformance-${{ steps.rust.outputs.cachekey }}-
159+
160+
- uses: ./.github/actions/setup-js
161+
if: matrix.candidate == 'js'
162+
163+
- name: Build the Rust adapter
164+
if: matrix.candidate == 'rust'
165+
run: make build/rust/conformance
166+
167+
- name: Build the JavaScript adapter
168+
if: matrix.candidate == 'js'
169+
run: make build/js/conformance
170+
171+
- name: Throughput and latency
172+
run: go test ./harness -count=1 -timeout 25m -run '^TestPerformance$/^Throughput$'
173+
working-directory: conformance
174+
175+
# Every job above pairs a candidate with River Go. These pair the other
176+
# implementations: Rust, JavaScript, and Go in one fleet, and JavaScript
177+
# against Rust as the reference, on PostgreSQL 18 and SQLite.
178+
multi_engine:
179+
name: ${{ matrix.name }}
180+
runs-on: ubuntu-latest
181+
timeout-minutes: 50
182+
183+
strategy:
184+
fail-fast: false
185+
matrix:
186+
include:
187+
- name: Three-engine fleet (Go, Rust, and JavaScript)
188+
candidate: rust
189+
peer: js
190+
reference: go
191+
run: "^TestMultiEngine$"
192+
- name: JavaScript against Rust
193+
candidate: js
194+
peer: ""
195+
reference: rust
196+
run: ""
197+
198+
services:
199+
postgres:
200+
image: postgres:18
201+
env:
202+
POSTGRES_DB: river_test
203+
POSTGRES_PASSWORD: postgres
204+
options: >-
205+
--health-cmd pg_isready
206+
--health-interval 2s
207+
--health-timeout 5s
208+
--health-retries 5
209+
ports:
210+
- 5432:5432
211+
212+
env:
213+
RIVER_CONFORMANCE: ${{ matrix.candidate }}
214+
RIVER_CONFORMANCE_PEER: ${{ matrix.peer }}
215+
RIVER_CONFORMANCE_REFERENCE: ${{ matrix.reference }}
216+
217+
steps:
218+
- uses: actions/checkout@v6
219+
with:
220+
persist-credentials: false
221+
222+
- uses: actions/setup-go@v6
223+
with:
224+
cache-dependency-path: conformance/go.sum
225+
go-version-file: go.work
226+
227+
- uses: dtolnay/rust-toolchain@stable
228+
id: rust
229+
230+
- name: Cache Rust dependencies and build artifacts
231+
uses: actions/cache@v5
232+
with:
233+
path: |
234+
~/.cargo/registry/index
235+
~/.cargo/registry/cache
236+
~/.cargo/git/db
237+
rust/target
238+
key: rust-v1-${{ runner.os }}-${{ runner.arch }}-conformance-${{ steps.rust.outputs.cachekey }}-${{ hashFiles('rust/**/Cargo.toml', 'rust/Cargo.lock') }}
239+
restore-keys: rust-v1-${{ runner.os }}-${{ runner.arch }}-conformance-${{ steps.rust.outputs.cachekey }}-
240+
241+
- uses: ./.github/actions/setup-js
242+
243+
- name: Build the Rust and JavaScript adapters
244+
run: make build/rust/conformance build/js/conformance
245+
246+
# Throughput bounds are relative to River Go, so they don't apply with
247+
# Rust as the reference.
248+
- name: Conformance scenarios and the nightly tier
249+
run: go test ./harness -count=1 -timeout 40m -run "$RUN" -skip '^TestPerformance$/^Throughput$'
250+
working-directory: conformance
251+
env:
252+
RUN: ${{ matrix.run }}
253+
254+
# Mixed traffic through Go, Rust, and JavaScript on PostgreSQL 18, with the
255+
# leader restarted regularly: every job completes exactly once and
256+
# connections stay bounded. GitHub-hosted jobs stop after six hours, so a
257+
# release candidate's soak is 5.5 hours.
258+
soak:
259+
name: Soak (Go, Rust, and JavaScript)
260+
runs-on: ubuntu-latest
261+
timeout-minutes: ${{ inputs.release_candidate && 355 || 90 }}
262+
263+
services:
264+
postgres:
265+
image: postgres:18
266+
env:
267+
POSTGRES_DB: river_test
268+
POSTGRES_PASSWORD: postgres
269+
options: >-
270+
--health-cmd pg_isready
271+
--health-interval 2s
272+
--health-timeout 5s
273+
--health-retries 5
274+
ports:
275+
- 5432:5432
276+
277+
env:
278+
RIVER_CONFORMANCE: rust
279+
RIVER_CONFORMANCE_PEER: js
280+
RIVER_CONFORMANCE_SOAK: ${{ inputs.release_candidate && '5h30m' || '1h' }}
281+
282+
steps:
283+
- uses: actions/checkout@v6
284+
with:
285+
persist-credentials: false
286+
287+
- uses: actions/setup-go@v6
288+
with:
289+
cache-dependency-path: conformance/go.sum
290+
go-version-file: go.work
291+
292+
- uses: dtolnay/rust-toolchain@stable
293+
id: rust
294+
295+
- name: Cache Rust dependencies and build artifacts
296+
uses: actions/cache@v5
297+
with:
298+
path: |
299+
~/.cargo/registry/index
300+
~/.cargo/registry/cache
301+
~/.cargo/git/db
302+
rust/target
303+
key: rust-v1-${{ runner.os }}-${{ runner.arch }}-conformance-${{ steps.rust.outputs.cachekey }}-${{ hashFiles('rust/**/Cargo.toml', 'rust/Cargo.lock') }}
304+
restore-keys: rust-v1-${{ runner.os }}-${{ runner.arch }}-conformance-${{ steps.rust.outputs.cachekey }}-
305+
306+
- uses: ./.github/actions/setup-js
307+
308+
- name: Build the Rust and JavaScript adapters
309+
run: make build/rust/conformance build/js/conformance
310+
311+
- name: Soak
312+
run: go test ./harness -count=1 -timeout "$TIMEOUT" -run '^TestSoak$'
313+
working-directory: conformance
314+
env:
315+
TIMEOUT: ${{ inputs.release_candidate && '5h45m' || '70m' }}

‎.github/workflows/conformance.yaml‎

Lines changed: 45 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,13 @@
11
name: Conformance
22

33
# The Rust and JavaScript workflows run only when their own files change, but
4-
# their fixture tests compare against values generated from River's Go code.
5-
# This job closes that gap: when Go code, SQL queries the generator reads, or
6-
# the generator itself changes, it regenerates the fixtures and runs only the
7-
# port tests that read them. Keep both events' paths in sync.
4+
# their fixture tests compare against values generated from River's Go code,
5+
# and their conformance jobs run against River Go. This workflow closes that
6+
# gap for Go changes: when Go code, SQL queries, or the conformance module
7+
# change, it regenerates the fixtures and runs only the port tests that read
8+
# them, and runs the conformance harness with Go as the candidate. The Rust
9+
# and JavaScript workflows run the harness with their own candidate. Keep both
10+
# events' paths in sync.
811
on:
912
push:
1013
branches:
@@ -39,6 +42,44 @@ env:
3942
CARGO_INCREMENTAL: "0"
4043

4144
jobs:
45+
# Go against itself exercises the harness and River Go's adapter. Every
46+
# scenario runs on PostgreSQL and SQLite in one job; the nightly workflow
47+
# covers older PostgreSQL versions.
48+
harness:
49+
name: Conformance harness (Go)
50+
runs-on: ubuntu-latest
51+
timeout-minutes: 15
52+
53+
services:
54+
postgres:
55+
image: postgres:18
56+
env:
57+
POSTGRES_DB: river_test
58+
POSTGRES_PASSWORD: postgres
59+
options: >-
60+
--health-cmd pg_isready
61+
--health-interval 2s
62+
--health-timeout 5s
63+
--health-retries 5
64+
ports:
65+
- 5432:5432
66+
67+
env:
68+
TEST_DATABASE_URL: postgres://postgres:postgres@localhost:5432/river_test?sslmode=disable
69+
70+
steps:
71+
- uses: actions/checkout@v6
72+
with:
73+
persist-credentials: false
74+
75+
- uses: actions/setup-go@v6
76+
with:
77+
cache-dependency-path: conformance/go.sum
78+
go-version-file: go.work
79+
80+
- name: Conformance scenarios
81+
run: make test/conformance CANDIDATE=go
82+
4283
port_fixtures:
4384
name: Port fixture tests
4485
runs-on: ubuntu-latest

0 commit comments

Comments
 (0)