Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 76 additions & 0 deletions .github/workflows/publish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
name: Publish approved npm release
on:
release:
types: [published]
permissions:
contents: read
concurrency:
group: npm-release-${{ github.event.release.tag_name }}
cancel-in-progress: false
jobs:
candidate:
# Disabled until a maintainer separately approves/configures OIDC and protections.
if: github.repository == 'agent-axiom/streamsplit' && vars.NPM_PUBLISH_ENABLED == 'true' && !github.event.release.prerelease
runs-on: ubuntu-latest
timeout-minutes: 15
outputs:
source-sha: ${{ steps.source.outputs.sha }}
env:
RELEASE_TAG: ${{ github.event.release.tag_name }}
steps:
- uses: actions/checkout@v4
with:
ref: ${{ github.event.release.tag_name }}
fetch-depth: 0
persist-credentials: false
- uses: actions/setup-node@v4
with:
node-version: 24
# Deliberately no dependency cache in release jobs.
- name: Verify tag, version, clean source, and main ancestry
id: source
run: |
node scripts/verify-release.mjs source
echo "sha=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"
- run: npm ci --ignore-scripts
- run: npm run release:check
- run: node scripts/verify-release.mjs candidate
- uses: actions/upload-artifact@v4
with:
name: npm-candidate-${{ steps.source.outputs.sha }}
path: release/
if-no-files-found: error
retention-days: 14
publish:
needs: candidate
runs-on: ubuntu-latest
timeout-minutes: 10
# Configure required reviewers, prevent self-review, and restrict tags BEFORE enabling.
environment: npm
permissions:
contents: read
id-token: write
env:
RELEASE_TAG: ${{ github.event.release.tag_name }}
steps:
- uses: actions/checkout@v4
with:
ref: ${{ needs.candidate.outputs.source-sha }}
persist-credentials: false
- uses: actions/setup-node@v4
with:
node-version: 24
registry-url: https://registry.npmjs.org
- uses: actions/download-artifact@v4
with:
name: npm-candidate-${{ needs.candidate.outputs.source-sha }}
path: release/
- name: Check OIDC-capable npm and exact candidate
run: |
node -e 'const [a,b,c] = process.argv[1].split(".").map(Number); if (a < 11 || (a === 11 && (b < 5 || (b === 5 && c < 1)))) process.exit(1)' "$(npm --version)"
node scripts/verify-release.mjs candidate
- name: Publish verified tarball through OIDC
# Uses only the verified archive; package lifecycle scripts are disabled.
run: npm publish "release/agent-axiom-streamsplit-$(node -p 'JSON.parse(require("fs").readFileSync("package.json")).version').tgz" --access public --ignore-scripts --registry=https://registry.npmjs.org
- name: Verify registry integrity and provenance metadata
run: node scripts/verify-release.mjs registry
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,9 @@ npm ci
npm run demo:short
```

The demo catches a broken UTF-8 decoder in SSE, reproduces a two-chunk failure, then checks fixed SSE and NDJSON parsers.
The demo catches a broken UTF-8 decoder in SSE, reproduces a two-chunk failure, then replays it with the fix.

Try three real parser integrations: `npm run example:sse`, `npm run example:ndjson`, and `npm run example:json`. [Adapters, edge cases, and tested versions →](docs/INTEGRATIONS.md)

## One assertion

Expand Down
16 changes: 15 additions & 1 deletion docs/INTEGRATIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

[Home](../README.md) · [Usage](USAGE.md) · [API](API.md) · [Safety](SAFETY.md)

These are compatibility tests maintained in this repository, not claims of adoption or endorsement by upstream projects. Both packages are development-only dependencies; StreamSplit's published runtime still has none.
These are compatibility tests maintained in this repository, not claims of adoption or endorsement by upstream projects. All three packages are development-only dependencies; StreamSplit's published runtime still has none.

## Verified targets

Expand All @@ -11,12 +11,16 @@ These are compatibility tests maintained in this repository, not claims of adopt
| [eventsource-parser](https://github.com/rexxars/eventsource-parser) | 3.0.6 on Node 20/22/24 | `createParser({onEvent})`, `feed(text)`, `reset({consume: true})` |
| [eventsource-parser](https://github.com/rexxars/eventsource-parser) | 4.1.1 on Node 22 | Same adapter; v4 requires Node >=22.12 |
| [ndjson](https://github.com/ndjson/ndjson.js) | 2.0.0 on Node 20/22/24 | Transform stream, awaited writes, readable completion |
| [@streamparser/json](https://github.com/juanjoDiaz/streamparser-json) | 0.0.26 on Node 20/22/24 | `JSONParser`, byte `write`, `onValue`, guarded `end` |

Versions are pinned in `package-lock.json` and the explicit v4 CI job. Current-source APIs and supported engines are documented by [eventsource-parser](https://github.com/rexxars/eventsource-parser/blob/main/package.json) and [ndjson](https://github.com/ndjson/ndjson.js/blob/master/package.json). This is fixture-specific compatibility evidence, not a conformance suite for either project.

```sh
npm ci
npm run test:integrations
npm run example:sse # broken decoding → exact replay → fixed adapter
npm run example:ndjson # real Transform, CRLF, Unicode, final record at EOF
npm run example:json # real JSONParser, selected array elements
```

To repeat the v4 check on Node 22.12 or newer:
Expand All @@ -43,6 +47,16 @@ Feed byte chunks directly to `ndjson.parse()`. Its decoder handles UTF-8. Await

The tests assert the actual expected Unicode records, split CRLF handling, blank lines, and the final record without a trailing newline. An invalid record fixture uses explicit `errorPolicy: 'compare'` to check the error and emitted prefix together.

## Streaming JSON: choose complete values and preserve EOF

[Complete adapter source](https://github.com/agent-axiom/streamsplit/blob/main/examples/integrations/stream-json.mjs) · [Runnable examples](https://github.com/agent-axiom/streamsplit/blob/main/examples/integrations/demo.mjs)

The pinned `@streamparser/json` accepts `Uint8Array` directly. A fresh `JSONParser` uses `paths: ['$']` for complete roots or `paths: ['$.*']` for array elements. Emit the callback's `value`, not its live `parent`/`stack` structures. `keepStack: false` avoids retaining already-emitted siblings. Partial-preview callbacks are deliberately disabled: their timing and count may depend on chunk boundaries.

Call `end()` unless `isEnded` is already true. A root number needs EOF to finish, while a complete object can end the parser earlier. `separator: ''` explicitly enables concatenated roots. See the [upstream API and lifecycle](https://github.com/juanjoDiaz/streamparser-json/tree/main/packages/plainjs).

Tests assert exact values for UTF-8, escaped quotes/newlines, escaped surrogate pairs, exponent numbers, arrays, booleans, null, selected elements, and concatenated documents. A truncated array checks both the completed prefix and EOF error using `errorPolicy: 'compare'`; harness emission errors must still propagate. These are integration examples, not a claim that every possible input or partition was tested.

## A useful boundary: diagnostic callbacks can differ intentionally

For eventsource-parser 4.1.1, `unknown: value\n` emits an `unknown-field` diagnostic when fed whole. Feeding the same 15 bytes as sizes `[1, 14]` omits that diagnostic. StreamSplit reproduces the difference when the adapter's `diagnostics: true` option includes diagnostic callbacks in the event trace.
Expand Down
39 changes: 26 additions & 13 deletions docs/RELEASING.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

[Home](../README.md) · [Usage](USAGE.md) · [Integrations](INTEGRATIONS.md) · [Safety](SAFETY.md)

This repository can build and verify a release candidate without registry credentials. It does not automatically publish to npm, create tokens, configure a trusted publisher, or claim a registry release exists.
This repository can build and verify a release candidate without registry credentials. The read-only readiness workflow does not publish. A separate opt-in publishing workflow is prepared below; it stays disabled until explicitly configured.

## Build the exact candidate

Expand All @@ -15,28 +15,41 @@ The command runs the full test/type/packed-consumer checks and short demo, then

The manually triggered **Release readiness** GitHub Actions workflow runs the same checks and stores the candidate for 14 days. It has read-only repository permissions and no OIDC permission or registry-publishing step.

## Maintainer decisions before publication
## Maintainer decisions before the next publication

1. Confirm the npm account and its right to publish under `@agent-axiom`. Owning the GitHub organization does not confer npm scope ownership. Confirm 2FA and the exact intended package name/version.
2. Check the current registry state at release time. An unauthenticated registry E404 only means the package is not publicly accessible; it does not establish ownership or availability of its scope.
3. Review the exact tarball, its contents/hash, the clean source commit, and that commit's CI result. Confirm public publication of this version. A version cannot simply be overwritten with a different tarball.
4. Perform the authorized first publication using the owner's interactive npm authentication. Do not copy long-lived tokens into the repository, chat, or CI. First publication and creating persistent publisher access are separate decisions.
5. Verify that the installed registry version, exported API, metadata, and tarball match the reviewed candidate. Only then replace the README's not-yet-published notice with the verified single-command install.
4. Perform only the separately authorized publication, using interactive npm authentication or the configured workflow below. Do not copy long-lived tokens into the repository, chat, or CI. Publication and creating persistent publisher access are separate decisions.
5. Verify that the installed registry version, exported API, metadata, and tarball match the reviewed candidate. Version 0.1.0 is already published; do not overwrite it or imply these repository changes are in that version.

The package metadata sets public access explicitly, but metadata alone does not authorize or perform publication. `prepublishOnly` runs the complete verification suite as an additional guard.

## Trusted publishing for later releases
## Prepared automatic release workflow

Requirements checked against official documentation on **2026-10-06**; recheck them when configuring the publisher:
[Publish approved npm release](https://github.com/agent-axiom/streamsplit/blob/main/.github/workflows/publish.yml) reacts to a published, non-prerelease GitHub release. It does nothing unless `NPM_PUBLISH_ENABLED` is exactly `true`. This PR neither enables that variable nor creates an npm publisher, environment protection, tag, or release.

- The package must already exist, and the npm account needs package write access and 2FA.
- GitHub OIDC publication requires a GitHub-hosted runner, Node >=22.14, npm >=11.5.1, and a publishing job with `id-token: write` plus `contents: read`.
- npm's publisher configuration must match repository owner `agent-axiom`, repository `streamsplit`, the exact workflow basename, and its environment when configured.
- Publisher setup creates persistent publishing access. Have the owner approve and perform that setup; the current readiness workflow intentionally cannot publish.
- Current npm publisher configurations default to stage-only publication; direct publishing must be explicitly permitted if desired. New configurations must successfully publish within two days, so configure close to the actual release.
- Use owner-approved release/environment protections. GitHub OIDC provenance is automatic for a public package from a public repository; verify the published version's provenance rather than assuming it exists.
Once enabled, the workflow:

npm also supports staged publication, including a public placeholder for a new package. Staging is still an external publication action and is not part of this readiness check.
1. Requires a stable `vX.Y.Z` tag matching `package.json`, a clean checkout, and a tagged commit in `main` history. Bump both package manifests in a reviewed PR before a new release.
2. Installs locked dependencies without install scripts, runs `release:check`, and uploads the tested archive with its source commit, SHA-256, and npm integrity. Release jobs do not restore dependency caches.
3. Waits at the `npm` environment. Only this job has OIDC permission. It downloads the same candidate, verifies its source/hash, and publishes that tarball with lifecycle scripts disabled. It neither rebuilds nor installs dependencies while holding publishing permission.
4. Checks the registry's name, version, integrity, and provenance metadata. This metadata check does not independently cryptographically verify the attestation. If a post-publish check fails, inspect the registry before rerunning; an already-published version cannot be overwritten.

The Node 20/22/24 PR checks must pass for the exact release commit before a maintainer approves the release. The workflow itself verifies on Node 24; it does not query or bypass branch protections.

## Separate approvals and setup

Official requirements rechecked **2026-10-08**:

- npm publisher setup needs package write access and account 2FA. The package must exist. `npm trust` setup commands require npm >=11.15.0; publishing itself requires npm >=11.5.1 and Node >=22.14.
- Approve persistent publishing access for `@agent-axiom/streamsplit` from GitHub owner `agent-axiom`, repository `streamsplit`, workflow filename `publish.yml`, environment `npm`. The filename is case-sensitive and excludes `.github/workflows/`.
- This workflow uses direct `npm publish`, which must be explicitly allowed. New publishers otherwise default to staging; granting stage-only access will not enable this workflow. Dist-tag management is not needed.
- Before enabling, approve/configure the GitHub `npm` environment with required reviewers, prevent self-review, and restrict deployment tags. Protect release tags and `main` as appropriate. A YAML environment name alone does not establish these protections.
- Configure near the approved next release: a new publisher expires unless it successfully publishes within two days. Use GitHub-hosted runners. Public GitHub/public npm OIDC releases receive automatic provenance.
- After verifying protections and npm configuration, separately enable `NPM_PUBLISH_ENABLED=true`. Publishing the next GitHub release starts the workflow; environment approval releases the reviewed candidate. No npm token secret is required.

Creating persistent trust, enabling automatic publication, and approving a specific release are separate actions. None has been performed by preparing these files. Changing existing npm publishing-access settings or revoking credentials also requires its own authorization.

## Official references

Expand Down
37 changes: 37 additions & 0 deletions examples/integrations/demo.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
import assert from 'node:assert/strict';
import { assertChunkInvariant, checkChunkInvariant, replayChunkInvariant } from '../../dist/index.js';
import { eventsourceAdapter } from './eventsource.mjs';
import { ndjsonAdapter } from './ndjson.mjs';
import { streamJsonAdapter } from './stream-json.mjs';

const cases = {
sse: { input: 'data: Hello 🌍\r\n\r\n', createParser: eventsourceAdapter, expected: [{ kind: 'event', data: 'Hello 🌍', event: null, id: null }] },
ndjson: { input: '{"city":"東京"}\r\n{"emoji":"🌍"}', createParser: ndjsonAdapter, expected: [{ city: '東京' }, { emoji: '🌍' }] },
json: { input: '[{"city":"東京"},{"value":-1.25e+3}]', createParser: emit => streamJsonAdapter(emit, { paths: ['$.*'] }), expected: [{ city: '東京' }, { value: -1250 }] },
};
const name = process.argv[2] ?? 'sse';
assert.ok(Object.hasOwn(cases, name), 'Choose sse, ndjson, or json.');
const { input: text, createParser, expected } = cases[name];
const input = new TextEncoder().encode(text);
const events = [];
const parser = createParser(value => events.push(value));
await parser.write(input);
await parser.end();
assert.deepEqual(events, expected);
console.log(`${name}: expected semantic output ${JSON.stringify(events)}`);

if (name === 'sse') {
// Deliberately break the decoding adapter, not the upstream parser.
const broken = emit => eventsourceAdapter(emit, { streaming: false });
const result = await checkChunkInvariant({ input, createParser: broken, seed: 42 });
assert.equal(result.ok, false);
console.log(`BROKEN: split UTF-8 corrupts ${JSON.stringify(result.failure.actual.events)}`);
console.log(`REPLAY: exact chunk sizes ${JSON.stringify(result.failure.fixture.chunkSizes)}`);
assert.equal((await replayChunkInvariant({ fixture: result.failure.fixture, createParser: broken })).ok, false);
assert.equal((await replayChunkInvariant({ fixture: result.failure.fixture, createParser })).ok, true);
console.log('FIX: decoder.decode(chunk, { stream: true }); exact fixture now passes');
}
const coverage = await assertChunkInvariant({ input, createParser, seed: 42 });
assert.equal(coverage.allSingleCutsChecked, true);
assert.equal(coverage.bytewiseChecked, true);
console.log(`PASS: ${coverage.schedulesChecked} schedules; every single cut and bytewise reads checked`);
13 changes: 13 additions & 0 deletions examples/integrations/stream-json.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
import { JSONParser } from '@streamparser/json';

/** Compare complete selected values, not chunk-dependent partial previews. */
export function streamJsonAdapter(emit, { paths = ['$'], separator } = {}) {
const parser = new JSONParser({ paths, separator, keepStack: false });
// parent/stack are live parser internals; emit only each completed value.
parser.onValue = ({ value }) => emit(value);
return {
write(chunk) { parser.write(chunk); },
// A complete single root can end the parser before the source reaches EOF.
end() { if (!parser.isEnded) parser.end(); },
};
}
8 changes: 8 additions & 0 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 5 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -50,9 +50,13 @@
"demo:short": "npm run build && node examples/demo-short.mjs",
"test:integrations": "npm run build && node --test test/integrations.test.mjs",
"release:check": "node scripts/release-check.mjs",
"prepublishOnly": "npm run check"
"prepublishOnly": "npm run check",
"example:sse": "npm run build && node examples/integrations/demo.mjs sse",
"example:ndjson": "npm run build && node examples/integrations/demo.mjs ndjson",
"example:json": "npm run build && node examples/integrations/demo.mjs json"
},
"devDependencies": {
"@streamparser/json": "0.0.26",
"eventsource-parser": "3.0.6",
"ndjson": "2.0.0",
"typescript": "5.9.3"
Expand Down
Loading
Loading